When using agent forwarding with SSH, your private key never leaves your computer. Signing/authentication requests are instead forwarded to your local agent (hey, that’s why it’s called agent forwarding! isn’t that cool?).
But obviously, if somebody on a system you’re connected to uses your agent (such as a bad guy), you don’t really know.
For gnupg agent (which supports ssh agent protocol), get and compile (note, you only need the resulting gpg-agent binary):
For ssh agent on OSX, notifications via Growl, get and compile (note, you only need the resulting ssh-agent binary):
Copy the agent in your path (like, /usr/local/bin), and make sure it’s used instead of the old agent. For OSX, you’ll want to have a script starting ssh agent instead of the OSX builtin agent emulation. The internet is full of “how to” do this.
On Linux, the notification UI generally allows you to have very customizable notifications that aren’t poping up a huge message every time the agent is used. Instead, you can get a small blink. This is entirely dependent on the notification agent. KDE’s agent is pretty good for example.
On OSX, your mileage may vary.