The SSH Agent is a little program which typically runs on your desktop
which allows the Secure Shell program to use your private secret key
for logging into remote servers. The idea with the agent is that it,
and it alone, holds onto the secret key. Each time you use
ssh(1) to connect to a server, it checks with the agent
to see if any of the keys that the agent is holding can be used to
login to the server you are trying to log into. If there is, the
remote server comes up with a challenge, which can only be solved with
the private key. The challenge is passed to the agent, and the agent
responds with the solution, proving to the remote server that you have
the private key. Neither the remote server, nor the ssh program
ever see the private key.
I came across this post about SSH Agent forwarding, which is a great piece on SSH authentication agent forwarding.
I’ve seen on multiple occasions, instructions like this:
- Set up access to the bastion host; put this in your
~/.ssh/config:
ForwardAgent yes
host bastion
hostname bastion.prod.pnw.company.com
- Now, when you want to log in to a server, first login to the bastion, then ssh to the server you want to go to:
$ ssh bastion
...bastion presents challenges...
bastion:~$ ssh appserver1
appserver1:~$
The connection from bastion to appserver1 was able to work
without entering a password because the SSH agent connection was
forwarded to bastion. If you log in to a host with forwarding,
you can see the socket in /tmp:
$ ls -l $SSH_AUTH_SOCK
srw-rw-rw- 1 samv wheel 0 Nov 18 12:39 /private/tmp/com.apple.launchd.P1lT8bBS1b/Listeners
That little s in the very first column indicates that the file is
a unix domain socket, which is essentially a little access panel to
the program that created it. It can be connected to as if it were an
IP address and port number, but only from somewhere that shares a
filesystem (ie, on the same system or kubernetes pod).
So what happened was that the ssh running on the bastion host
connected to that socket, asked it for the keys, and then the program
listening on it - the SSH daemon process (sshd) forwarded the
request to your desktop. Once it discovers that the appserver1
you are trying to log into accepts a key it has, it will happily
provide solutions to the challenges required to prove that the key is
available.
The Gaping Security Hole
This agent forwarding is somewhat pragmatic for single hosts which act
as gatekeepers to other hosts, and on which nobody is granted root
access.
If people are allowed root access, then you’ve just opened up a
little security hole: if somebody can access that file in /tmp,
and send messages from wherever they are, then they can login as you
to wherever you are allowed access. And to make things worse,
you’ll never know your key was used.
If you enable this all the time, then every system you login to is able to use your agent while you are logged in to login anywhere else you are allowed to log in. Let that sink in.
So, at the very least, only put ForwardAgent Yes on those few
hosts that you expect this to work with:
host bastion
hostname bastion.prod.pnw.company.com
ForwardAgent yes
It’s also possible to make the agent request confirmation each time it is used to login to a remote server (see the post linked earlier); however, the version which ships with MacOS X 10.11 (El Capitan) does not support that. The built-in keyring system seems to silently carry out authentications, as well.
It’s much better and more convenient to follow the instructions in the
post and use ProxyCommand. And it works even better with
ControlMaster!
SSH ControlMaster and ProxyCommand
First, let’s set up SSH ControlMaster, so that you don’t have to keep re-authenticating to hosts you’re already connected to.
ControlMaster auto
ControlPath ~/.ssh/control-%r@%h:%p.sock
This command opens another one of those magic unix domain sockets,
which lets ssh commands you run go right through.
Be warned that if your local machine gets compromised, this does make
it a little easier for the attacker to get into the host you’re trying
to get into. That said, it’s not really that much harder to replace
your ssh command with a malicious version which achieves the same
access tunnel even if you disable ControlMaster. If you like, set
ControlMaster to ask, so that you at least get a pop up window
each time this is used (unless the person owning you is cleverer than
that).
Next, set up a wildcard rule. This will be based on your company’s
hostname convention: let’s assume it’s always something .prod:
host *.prod
hostname %h.pnw.company.com
proxyCommand ssh -ax bastion.prod.pnw.company.com nc %h %p
(nc is an alternative to the ssh -W command in the original
post; if that works, then go ahead and use it, because it’s slightly
more resource efficient)
Now, you can complete both of these steps with a single command:
$ ssh appserver1.prod
...bastion presents challenges...
appserver1:~$
You skipped the intermediate host!
Not only that, but now you’ll also be able to use commands like
scp to send files directly to and from the server behind the
bastion you’re interested in. Without entering a password!
$ scp appserver1.prod:/tmp/data_file.json .
data_file.json 200kb
$
Once the original session is closed, the ControlMaster socket
closes and all other sessions piggy-backing over it will be shut down.
Personally, I find the ability to use scp, rsync etc through
the bastions is indispensible and set up something like this so that I
don’t have to use ForwardAgent.