The SSH Agent is a little program which typically runs on your desktop
which allows the Secure Shell program to use your private secret key
for logging into remote servers. The idea with the agent is that it,
and it alone, holds onto the secret key. Each time you use
ssh(1) to connect to a server, it checks with the agent
to see if any of the keys that the agent is holding can be used to
login to the server you are trying to log into. If there is, the
remote server comes up with a challenge, which can only be solved with
the private key. The challenge is passed to the agent, and the agent
responds with the solution, proving to the remote server that you have
the private key. Neither the remote server, nor the
ever see the private key.
I came across this post about SSH Agent forwarding, which is a great piece on SSH authentication agent forwarding.
I’ve seen on multiple occasions, instructions like this:
- Set up access to the bastion host; put this in your
ForwardAgent yes host bastion hostname bastion.prod.pnw.company.com
- Now, when you want to log in to a server, first login to the bastion, then ssh to the server you want to go to:
$ ssh bastion ...bastion presents challenges... bastion:~$ ssh appserver1 appserver1:~$
The connection from
appserver1 was able to work
without entering a password because the SSH agent connection was
bastion. If you log in to a host with forwarding,
you can see the socket in
$ ls -l $SSH_AUTH_SOCK srw-rw-rw- 1 samv wheel 0 Nov 18 12:39 /private/tmp/com.apple.launchd.P1lT8bBS1b/Listeners
s in the very first column indicates that the file is
a unix domain socket, which is essentially a little access panel to
the program that created it. It can be connected to as if it were an
IP address and port number, but only from somewhere that shares a
filesystem (ie, on the same system or kubernetes pod).
So what happened was that the
ssh running on the
connected to that socket, asked it for the keys, and then the program
listening on it - the SSH daemon process (
sshd) forwarded the
request to your desktop. Once it discovers that the
you are trying to log into accepts a key it has, it will happily
provide solutions to the challenges required to prove that the key is
The Gaping Security Hole
This agent forwarding is somewhat pragmatic for single hosts which act
as gatekeepers to other hosts, and on which nobody is granted
If people are allowed
root access, then you’ve just opened up a
little security hole: if somebody can access that file in
and send messages from wherever they are, then they can login as you
to wherever you are allowed access. And to make things worse,
you’ll never know your key was used.
If you enable this all the time, then every system you login to is able to use your agent while you are logged in to login anywhere else you are allowed to log in. Let that sink in.
So, at the very least, only put
ForwardAgent Yes on those few
hosts that you expect this to work with:
host bastion hostname bastion.prod.pnw.company.com ForwardAgent yes
It’s also possible to make the agent request confirmation each time it is used to login to a remote server (see the post linked earlier); however, the version which ships with MacOS X 10.11 (El Capitan) does not support that. The built-in keyring system seems to silently carry out authentications, as well.
It’s much better and more convenient to follow the instructions in the
post and use
ProxyCommand. And it works even better with
SSH ControlMaster and ProxyCommand
First, let’s set up SSH ControlMaster, so that you don’t have to keep re-authenticating to hosts you’re already connected to.
ControlMaster auto ControlPath ~/.ssh/control-%r@%h:%p.sock
This command opens another one of those magic unix domain sockets,
ssh commands you run go right through.
Be warned that if your local machine gets compromised, this does make
it a little easier for the attacker to get into the host you’re trying
to get into. That said, it’s not really that much harder to replace
ssh command with a malicious version which achieves the same
access tunnel even if you disable
ControlMaster. If you like, set
ask, so that you at least get a pop up window
each time this is used (unless the person owning you is cleverer than
Next, set up a wildcard rule. This will be based on your company’s
hostname convention: let’s assume it’s always something
host *.prod hostname %h.pnw.company.com proxyCommand ssh -ax bastion.prod.pnw.company.com nc %h %p
nc is an alternative to the
ssh -W command in the original
post; if that works, then go ahead and use it, because it’s slightly
more resource efficient)
Now, you can complete both of these steps with a single command:
$ ssh appserver1.prod ...bastion presents challenges... appserver1:~$
You skipped the intermediate host!
Not only that, but now you’ll also be able to use commands like
scp to send files directly to and from the server behind the
bastion you’re interested in. Without entering a password!
$ scp appserver1.prod:/tmp/data_file.json . data_file.json 200kb $
Once the original session is closed, the
closes and all other sessions piggy-backing over it will be shut down.
Personally, I find the ability to use
rsync etc through
the bastions is indispensible and set up something like this so that I
don’t have to use