[surreality]

I think you can use iptables to block other IP addresses from accessing your server through SSH. I'm not sure of the actual method, though. Perhaps someone else here knows more about it.