Automatic and secure file transfer over two servers

the problem is how to automatically transfer backups from server A to server B in the best possible way in terms of security?

Here is what comes to my mind:

• authorize server A pub key to access server B
• create user on server B only for backups
• run ssh-agent on server A, enter password manually after restart
• automatically copy files with scp from server A to server B

but if bad person gets access to server A, he will basically have access to server B also. So I have to restrict server B user? How can I do that, and is this the best way of doing this whole backup over network?

Regards