Manager Can View Subordinates Doc

Hi,
A manager / supervisor of an employee can see all his doc types such as salary slips, time sheets, etc. Is there a way to restrict the manager from seeing the documents of people reporting to them.

Can anyone advise please?