GNU/Linux Crypto: Agents

This entry is part 5 of 10 in the series GNU/Linux Crypto.

Now that we have both GnuPG and SSH securely set up, we’re able to encrypt, decrypt, sign, and verify messages, and securely authenticate to remote servers without any risk of exposing passwords and with effectively zero possibility of a brute-force attack. This is all great, but there’s still one more weak link in the chain with which to deal — our passphrases.

If you use these tools often, typing your passphrase for most operations can get annoying. It may be tempting to either include a means of automating the passphrase entry, or not to use a passphrase at all, leaving your private key unencrypted. As security-conscious users, we definitely want to avoid the latter in case our private key file ever gets stolen, which is where the concepts of agents for both SSH and GnuPG comes into play.

An agent is a daemon designed to streamline the process of using decrypted private keys by storing the details securely in memory, ideally for a limited period of time. What this allows you do with both SSH and GnuPG is to type your passphrase just once, and subsequent uses that require the unencrypted private key are managed by the agent.

In this article, we’ll go through the basics of agent setup for both SSH and GnuPG. Once we know how they work, we’ll then introduce a convenient tool to start both of them and manage them for us easily.

SSH agents

The ssh-agent(1) program comes as part of the OpenSSH suite. It can be run in two modes, either as a parent process, or daemonized into the background. We’ll discuss the latter method, as it’s more commonly used and more flexible.


When we run ssh-agent(1) for the first time, its behavior is curious; it appears to do nothing except spit out some cryptic shell script:

$ ssh-agent 
SSH_AUTH_SOCK=/tmp/ssh-EYqoH3qwfvbe/agent.28881; export SSH_AUTH_SOCK;
echo Agent pid 28882;

However, we can see that the daemon is running with the PID it mentions:

$ ps 28882
28882 ?        Ss     0:00 ssh-agent

So if it’s running fine, what’s with all the shell script it outputs? Why doesn’t it just run that for us?

The answer is an interesting workaround to a stricture of the Unix process model; specifically, a process cannot modify its parent environment. The variables SSH_AUTH_SOCK and SSH_AGENT_PID are designed to allow programs like ssh(1) to find the agent so it can communicate with it, so we definitely need them set. However, if ssh-agent(1) were to set these variables itself, it would only apply for its own process, not the shell where we called it.

Therefore, not only do we need to run ssh-agent(1), we need to execute the code it outputs so the variables get assigned in our shell. A good method of doing this in Bash is using eval and command substitution with $(...):

$ eval "$(ssh-agent)"
Agent 3954

If we run this, we can see that not only is ssh-agent(1) running, we have two new variables in our environment identifying its socket path and process ID:

$ pgrep ssh-agent
$ env | grep ^SSH

With this done, the agent is ready, and we can start using it to manage our keys for us.


The next step is to load our keys into the agent with ssh-add(1). Pass this program the full path to the private key you would like to use with the agent. This is likely either ~/.ssh/id_rsa or ~/.ssh/id_dsa:

$ ssh-add ~/.ssh/id_rsa
Enter passphrase for /home/tom/.ssh/id_rsa:
Identity added: /home/tom/.ssh/id_rsa (/home/tom/.ssh/id_rsa)

You can leave out the filename argument if you want ssh-add to add any or all of the default key types in ~/.ssh if they exist (id_dsa, id_rsa, and id_ecdsa):

$ ssh-add

Either way, you should be prompted for your passphrase; this is expected, and you should go ahead and type it in.

If we then ask ssh-add(1) to list the keys it’s managing, we see the key we just added:

$ ssh-add -l
4096 87:ec:57:8b:ea:24:56:0e:f1:54:2f:6b:ab:c0:e8:56 /home/tom/.ssh/id_rsa (RSA)

With this done, if we try to use this key to connect to another server, we no longer need to provide the passphrase; we’re just logged straight in:

tom@local:~$ ssh remote
Welcome to, running GNU/Linux!

The default is to maintain the keys permanently, until the agent is stopped or the keys are explicitly removed one-by-one with ssh-add -d <keyfile> or all at once with ssh-add -D. For the cautious, you can set a time limit in seconds with ssh-add -t. For example, to have ssh-add forget about your keys after two hours, you might use:

$ ssh-add -t 7200 ~/.ssh/id_rsa

To kill the agent completely, you can use ssh-agent -k, again with an eval $(...) wrapper:

$ eval "$(ssh-agent -k)"
Agent pid 4501 killed

You may like to consider adding this to ~/.bash_logout or a similar script to get rid of the running agent after you’re done with your session.

Permanent setup

If you like this and find it makes your key management more convenient, it makes sense to put it into a startup script like ~/.bash_profile. This way, the agent will be started for each login shell, and we will be able to communicate with it from any subshell (xterm, screen, or an appropriately configured tmux):

eval "$(ssh-agent)"
ssh-add ~/.ssh/id_rsa

On our next TTY login, we should be prompted for a passphrase, and from there be able to connect to any machine using the keys managed by the agent:

tom@local:~$ ssh remote
Welcome to, running GNU/Linux!

If you want this to work for a desktop manager like GDM or XDM, you can add a variable pointing to the ssh-askpass(1) program:

eval $(ssh-agent)
export SSH_ASKPASS=/usr/bin/ssh-askpass
ssh-add ~/.ssh/id_rsa

If SSH_ASKPASS is set like this and DISPLAY refers to a working display, then a simple graphical prompt will appear asking for your passphrase:

ssh-askpass prompting for a passphrase

This program may need to be installed separately. Under Debian-derived systems, its package name is ssh-askpass.

All child processes and subshells of the login shell will inherit the agent’s variables, since they were exported with export:

tom@local:~$ screen
tom@local:~$ tmux bash
tom@local:~$ bash
tom@local:~$ ssh remote
Welcome to, running GNU/Linux!

We thus have to type our passphrase only once per login session, and can connect to all of the servers to which our keys confer access … very convenient!

GnuPG Agents

Just like ssh-agent(1), there exists an agent for managing GnuPG keys too, called gpg-agent(1). Its behavior is very similar. On Debian-derived systems, it can be installed as the gnupg-agent package. You should also install a pinentry program; as we’re focussing on learning the nuts and bolts on the command line here, we’ll use pinentry-curses(1) for a console-based passphrase prompt:

# apt-get install gnupg-agent pinentry-curses


We’ll start the agent using the same eval $(...) trick we learned with ssh-agent:

$ eval "$(gpg-agent --daemon)"

We can verify that the agent is running in the background with the given PID, and that we have a new environment variable:

$ pgrep gpg-agent
$ env | grep ^GPG

We’ll also set GPG_TTY, which will help the pinentry program know on which terminal to draw its passphrase request screen:

$ export GPG_TTY=$(tty)
$ echo $GPG_TTY

Finally, to prod gpg(1) into actually using the agent, we need to add a line to ~/.gnupg/gpg.conf. You can create this file if it doesn’t exist.



With this done, if we try to do anything requiring our private key, we should be prompted for a passphrase not directly on the command line, but by our PIN entry program:

$ gpg --armor --sign message1.txt

│ You need a passphrase to unlock the secret key for user: │
│ "Thomas Ryder (tyrmored, tejr) <>"    │
│ 4096-bit RSA key, ID 25926609, created 2013-03-12        │
│ (main key ID 77BB8872)                                   │
│                                                          │
│                                                          │
│ Passphrase ***__________________________________________ │
│                                                          │
│       <OK>                                 <Cancel>      │

When we enter the passphrase, our operation is performed:

$ ls message1*

Afterwards, if we perform another option requiring the private key, we see that we are not prompted:

$ gpg --armor --sign message2.txt
$ ls message2*

The agent has thus cached the private key for us, making it much easier to perform a series of operations with it. The default timeout is 10 minutes, but you can change this with the default-cache-ttl and max-cache-ttl settings in ~/.gnupg/gpg-agent.conf. For example, to retain any private key for one hour after its last use and a maximum of two hours from its first use, we could write:

default-cache-ttl 3600
max-cache-ttl 7200

Changing these values will require prompting the agent to reload:

$ gpg-connect-agent <<<RELOADAGENT

Permanent setup

Just like ssh-agent(1), an ideal place for gpg-agent(1)‘s startup lines is in a login shell setup script like ~/.bash_profile:

eval "$(gpg-agent --daemon)"

The agent will be started, and all of its environment variables will be set and inherited by all subshells, just as with ssh-agent.

If you’re using the console PIN entry tool, you should also add this to end of your interactive shell startup script. This should be ~/.bashrc for Bash on Linux; you may need to put it in ~/.bashrc on Mac OS X.

export GPG_TTY=$(tty)


To manage both ssh-agent(1) and gpg-agent(1) effectively, a tool called keychain(1) is available. It provides a simple way to start both agents with one command, including loading keys at startup, and also prevents running either agent twice, picking up on agents started elsewhere on the system. Because desktop environments are often configured to start one or both agents for users, it makes sense to re-use them where possible, at which keychain(1) excels.

On Debian-derived systems, the program is available in the keychain package:

# apt-get install keychain

With keychain installed, we can start both agents with just one command in ~/.bash_profile:

eval "$(keychain --eval)"

We can optionally include the filenames of SSH keys in ~/.ssh or the hex IDs of GnuPG keys as arguments to prompt loading the private key (including requesting the passphrase) at startup:

eval "$(keychain --eval id_rsa 0x77BB8872)"

If this program is available to you, then I highly recommend this; managing agents and environments can be fiddly, and keychain(1) does all the hard work for you in this regard so you don’t have to worry about whether an agent is available to you in your particular context. Check out the project’s homepage for more information about the tool.

GNU/Linux Crypto: SSH keys

This entry is part 4 of 10 in the series GNU/Linux Crypto.

The usual method of authenticating to an OpenSSH server is to type your shell password for the remote machine:

tom@local:~$ ssh remote
The authenticity of host 'remote (' can't be established.
RSA key fingerprint is d1:35:45:a6:d1:b2:e4:08:f8:67:b1:19:fe:04:ca:1c.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'remote,' (RSA) to the list of known hosts.
tom@remote's password:


This is appropriate for first contact with a machine, and the authentication method is supported out of the box for most OpenSSH sshd(8) installations.

sshd(8) is a very common target for attacks, particularly automated ones; malicious bots attempt to connect to servers listening on the standard SSH destination port of tcp/22, as well as some common alternatives like tcp/2222. If you enforce a strong password policy on your system, this generally isn’t too much of a problem, particularly if only appropriate users have shells, or if you restrict SSH connections only to certain usernames or groups.

There are other measures to defeat automated attacks, such as employing systems like fail2ban to reject clients who make too many spurious connection attempts, but perhaps the most effective way of short-circuiting automated attacks is to bypass passwords completely and instead use SSH keys, allowing this as the only connection method to the relevant machines.

How it works

Similar to the GnuPG keys setup in the first two articles in this series, SSH keypairs are comprised of one private key and one public key, two cryptographically linked files. The basis of keys for authentication is that if someone has your public key, they’re able to authenticate you by requesting operations that you would only be able to perform with the corresponding private key; it works similarly to cryptographic signing.

The reason this is so effective is because if you require a valid public key to authenticate, with sufficient key length it’s effectively impossible for an attacker to guess your authentication details; there’s no such thing as a “common” private key to guess, so they would need to run through every possible private key, which is not even remotely practical.

Your system’s sshd(8) may still be attacked, but if you use only public key authentication, then you can be comfortably certain it’s effectively impossible to brute-force your credentials. Note that this doesn’t necessarily protect you from security problems in sshd(8) itself, however, and you will still need to protect your private key from being hijacked or compromised, hence the necessity of a passphrase.

All of the below assumes you have OpenSSH installed as both the client and the server on the appropriate systems. On Debian-derived systems, these can be installed with:

# apt-get install ssh
# apt-get install openssh-server

Both the client and server often come standard with systems (e.g. their native OpenBSD).

Generating keys

Similar to the GnuPG setup process, we start by generating a keypair on the machine from which we’d like to connect, using ssh-keygen(1). I’m using 4096-bit RSA here, as it’s widely supported even on very old systems, and should be relatively future-proof, although generating new keys if RSA ever becomes unsafe is not hard. If you’d prefer to use the newer ECDSA algorithm that’s the default in recent versions of OpenSSH, all of this will still work. I’m also applying a comment for the key as an unencrypted identifier to distinguish multiple keys if I have them. I find email addresses work well.

$ ssh-keygen -t rsa -b 4096 -C
Generating public/private rsa key pair.

First, we’re prompted for a location to which the key files should be saved. I recommend accepting the default by pressing Enter, as using the default location makes the next few steps easier:

Enter file in which to save the key (/home/tom/.ssh/id_rsa):

Next, we’re prompted for a passphrase, which should definitely be added to keep the key from being used if it’s ever compromised. The same guidelines for passphrases apply to SSH here, and you should choose a different passphrase:

Enter passphrase (empty for no passphrase):
Enter same passphrase again:

This done, the key is generated, including a pictorial representation to recognise keys at a glance. I’ve never found this very useful, but the key fingerprint is helpful:

Your identification has been saved in /home/tom/.ssh/id_rsa.
Your public key has been saved in /home/tom/.ssh/
The key fingerprint is:
d5:81:8c:eb:c6:c5:a2:b9:6a:ae:32:cc:20:bf:cf:66 tom@local
The key's randomart image is:
+--[ RSA 4096]----+
|         o ..    |
|        . o. .   |
|         o. .    |
|        o.o      |
|       =So       |
|o     o +        |
|=.     o         |
|oo..E .          |
| ooO=.           |

The key files should now be available in ~/.ssh:

$ ls -l .ssh
-rw-------  1 tom  tom  3326 Apr  2 22:47 id_rsa
-rw-r--r--  1 tom  tom   754 Apr  2 22:47

The id_rsa file contains the encrypted private key, and should be kept locked down and confidential. The file, however, contains the public key, which can be safely distributed, in the same way as a PGP public key.

Key-based authentication

We can now arrange to use the newly generated public key for authentication in lieu of a password. Start by ensuring you can connect to the remote machine with your username and password:

$ ssh remote
tom@remote's password:

Once connected, ensure that the ~/.ssh directory exists on the remote machine, and that you don’t already have keys listed in ~/.ssh/authorized_keys, as we’re about to overwrite them:

$ mkdir -p ~/.ssh
$ chmod 0700 ~/.ssh

If this worked, close the connection (exit or Ctrl-D) to return to your local machine’s shell, and copy your public key onto the remote machine with scp(1):

$ scp ~/.ssh/ remote:.ssh/authorized_keys
tom@remote's password:    100%    754    0.7KB/s    00:00

Note that there’s a tool included in recent versions of OpenSSH that does this for you called ssh-copy-id(1), but it’s good to have some idea of what it’s doing in the background.

With this done, your next connection attempt to the remote host should prompt you for your passphrase, rather than your password:

$ ssh remote
Enter passphrase for key '/home/tom/.ssh/id_rsa':


At first, it may not seem like you’ve done much useful here. After all, you still have to type in something to connect each time. From a security perspective, the first major advantage to this method is that neither your password, nor your passphrase, nor your private key are ever transmitted to the server to which you’re connecting; authentication is done purely based on the public-private key pair, decrypted by your passphrase.

This means that if the machine you’re connecting to were compromised, or your DNS had been poisoned, or some similar attack tricked you into connecting to a fake SSH daemon designed to collect credentials, your private key and your password remain safe.

The second advantage comes with turning off password authentication entirely on the host machine, once all its users have switched to public key authentication only. This is done with the following settings in sshd_config(5), usually in /etc/ssh/sshd_config on the remote server:

PubkeyAuthentication yes
ChallengeResponseAuthentication no
PasswordAuthentication no

Restart the SSH server after these are applied:

$ sudo /etc/init.d/ssh restart

You should then no longer be able to connect via passwords at all, only by private keys, which as mentioned above are effectively (though not literally) impossible to brute-force. In order to connect to the server as you, an attacker would not only need to know your passphrase, but also have access to your private key, making things significantly harder.

Using public key authentication also allows sshd(8) some finer-grained control over authentication, such as which hosts can connect with which keys, whether they can execute TCP or X11 tunnels, and (to an extent) which commands they can run once connected. See the manual page for authorized_keys(5) to take a look at some examples.

Finally, there’s a major usability advantage in using SSH keys for authentication with agents, which we’ll discuss in the next article.

Host keys and fingerprints

SSH connection should ideally be a two-way authentication process. Just as the server to which you’re connecting needs to be sure who you are, you need to be sure that the host you’re connecting to is the one you expect. With tunnelling, firewalls, DNS poisoning, NAT, hacked systems, and various other tricks, it’s appropriate to be careful that you’re connecting to the right systems. This is where OpenSSH’s host key system comes into play.

The first time you connect to a new server, you should see prompts like this:

$ ssh newremote
The authenticity of host 'newremote (' can't be established.
RSA key fingerprint is f4:4b:f4:8c:c5:50:f6:c8:d3:b2:e9:14:68:86:b5:7b.
Are you sure you want to continue connecting (yes/no)?

A lot of administrators turn these off; don’t! They are very important.

The key fingerprint is a relatively short hash for the host key used by OpenSSH on that server. It’s verified by your SSH client, and can’t easily be faked. If you’re connecting to a new server, it’s appropriate to check the host key fingerprint matches the one you see on first connection attempt, or to ask the system’s administrator to do so for you.

The host key’s fingerprint can be checked on the SSH server with a call to ssh-keygen(1):

$ ssh-keygen -lf /etc/ssh/ssh_host_rsa_key
2048 f4:4b:f4:8c:c5:50:f6:c8:d3:b2:e9:14:68:86:b5:7b /etc/ssh/ (RSA)

If you want, you can check the key without making a connection attempt with a similar call on the client side:

$ ssh-keygen -lF newremote
# Host found: line 1 type RSA
2048 f4:4b:f4:8c:c5:50:f6:c8:d3:b2:e9:14:68:86:b5:7b newremote (RSA)

If you can’t check the host key yourself, have the administrator send it to you over a secure, trusted channel, such as in person or via a PGP signed message. If the colon-delimited SSH fingerprint is not exactly the same, then you might be the victim of someone attempting to spoof your connection!

This is definitely overkill for new virtual machines and probably new machines on a trusted LAN, but for machines accessed over the public internet, it’s a very prudent practice.

Similarly, ssh(1) by default keeps a record of the host keys for hosts, which is why when a different host key is presented on a connection attempt, it warns you:

$ ssh newremote
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
Please contact your system administrator.
Add correct host key in /home/tom/.ssh/known_hosts to get rid of this message.
Offending RSA key in /home/tom/.ssh/known_hosts:22
RSA host key for newremote has changed and you have requested strict checking.

Again, this is something ssh(1) users often turn off, which is pretty risky, especially if you are using password authentication and hence might send your password to a malicious or compromised server!

SSH agents

Public key authentication has a lot of advantages for connecting to servers, particularly if it’s the only allowed means of authentication, reducing the chances of a brute force password attack to zero. However, it doesn’t solve the problem of having to type in a password or passphrase on each connection, unless you’re using a private key with no passphrase, which is quite risky if the private key is compromised.

Thankfully, there’s a nice supplement to a well-secured SSH key setup in the use of agents on trusted boxes to securely store decrypted keys per-session, per-user. Judicious use of an SSH agent program on a trusted machine allows you to connect to any server for which your public key is authorised by typing your passphrase to decrypt your private key only once.

SSH agent setup

The ssh-agent program is designed as a wrapper for a shell. If you have a private and public key setup ready, and you have remote machines for which your key is authorised, you can get an idea of how the agent works by typing:

$ ssh-agent bash

This will prompt you for your passphrase, and once entered, within the context of that subshell, you will be able to connect to authorised remote servers without typing in the passphrase again. Once loaded, you can examine the identities you have by using ssh-add -l to see the fingerprints, and ssh-add -L for the public keys:

$ ssh-agent bash
Enter passphrase for /home/user/.ssh/id_rsa:
Identity added: /home/user/.ssh/id_rsa (/home/user/.ssh/id_rsa)
$ ssh-add -l
2048 07:1e:7d:c4:8a:0e:bc:b0:74:40:71:49:7c:70:9c /home/user/.ssh/id_rsa (RSA)
$ ssh-add -L

You can set up your .bashrc file to automatically search for accessible SSH agents to use for the credentials for new connections, and to prompt you for a passphrase to open a new one if it need be. There are very workable instructions on GitHub for setting this up.

If you want to shut down the agent at any time, you can use ssh-agent -k.

$ ssh-agent -k
echo Agent pid 790 killed;

SSH agent forwarding

Where the configuration of the remote machine allows it, you can forward authentication requests made from the remote machine back to the agent on your workstation. This is handy for working with semi-trusted gateway machines that you trust to forward your authentication requests correctly, but on which you’d prefer not to put your private key.

This means that if you connect to a remote machine from your workstation running an SSH agent with the following, using the -A parameter:

user@workstation:~$ ssh -A

You can then connect to another machine from using your private key on workstation:

user@remote:~$ ssh

SSH agent authentication via PAM

It’s also possible to use SSH agent authentication as a PAM method for general authentication, such as for sudo, using pam_ssh_agent_auth.

Shortcut for adding SSH keys

If you’ve dabbled with SSH much, for example by following the excellent tutorial a few years ago, you’ll know about adding keys to allow passwordless login (or, if you prefer, a passphrase) using public key authentication. Specifically, you copy the public key ~/.ssh/ or ~/.ssh/ off the machine from which you wish to connect into the /.ssh/authorized_keys file on the target machine. That will allow you to open an SSH session with the machine from the user account on the local machine to the one on the remote machine, without having to type in a password.

tom@conan:~$ scp ~/.ssh/ crom:.ssh/conan.pubkey
tom@conan:~$ ssh crom
tom@crom:~$ cd .ssh
tom@crom:~$ cat .ssh/conan.pubkey >>~/.ssh/authorized_keys

However, there’s a nice shortcut that I didn’t know about when I first learned how to do this, which has since been added to that tutorial too — specifically, the ssh-copy-id tool, which is available in most modern OpenSSH distributions and combines this all into one less error-prone step. If you have it available to you, it’s definitely a much better way to add authorized keys onto a remote machine.

tom@conan:~$ ssh-copy-id crom

Incidentally, this isn’t just good for convenience or for automated processes; strong security policies for publically accessible servers might disallow logging in via passwords completely, as usernames and passwords can be guessed. It’s a lot harder to guess an entire SSH key, so forcing this login method will reduce your risk of script kiddies or automated attacks brute-forcing your OpenSSH server to zero. You can arrange this by setting ChallengeResponseAuthentication to no in your sshd_config, but if that’s a remote server, be careful not to lock yourself out!