Prompt directory shortening

The common default of some variant of \h:\w\$ for a Bash prompt PS1 string includes the \w escape character, so that the user’s current working directory appears in the prompt, but with $HOME shortened to a tilde:

tom@sanctum:~$
tom@sanctum:~/Documents$
tom@sanctum:/usr/local/nagios$

This is normally very helpful, particularly if you leave your shell for a time and forget where you are, though of course you can always call the pwd shell builtin. However it can get annoying for very deep directory hierarchies, particularly if you’re using a smaller terminal window:

tom@sanctum:/chroot/apache/usr/local/perl/app-library/lib/App/Library/Class:~$

If you’re using Bash version 4.0 or above (bash --version), you can save a bit of terminal space by setting the PROMPT_DIRTRIM variable for the shell. This limits the length of the tail end of the \w and \W expansions to that number of path elements:

tom@sanctum:/chroot/apache/usr/local/app-library/lib/App/Library/Class$ PROMPT_DIRTRIM=3
tom@sanctum:.../App/Library/Class$

This is a good thing to include in your ~/.bashrc file if you often find yourself deep in directory trees where the upper end of the hierarchy isn’t of immediate interest to you. You can remove the effect again by unsetting the variable:

tom@sanctum:.../App/Library/Class$ unset PROMPT_DIRTRIM
tom@sanctum:/chroot/apache/usr/local/app-library/lib/App/Library/Class$

Testing exit values in Bash

In Bash scripting (and shell scripting in general), we often want to check the exit value of a command to decide an action to take after it completes, likely for the purpose of error handling. For example, to determine whether a particular regular expression regex was present somewhere in a file options, we might apply grep(1) with its POSIX -q option to suppress output and just use the exit value:

grep -q regex options

An approach sometimes taken is then to test the exit value with the $? parameter, using if to check if it’s non-zero, which is not very elegant and a bit hard to read:

# Bad practice
grep -q regex options
if (($? > 0)); then
    printf '%s\n' 'myscript: Pattern not found!' >&2
    exit 1
fi

Because the if construct by design tests the exit value of commands, it’s better to test the command directly, making the expansion of $? unnecessary:

# Better
if grep -q regex options; then
    # Do nothing
    :
else
    printf '%s\n' 'myscript: Pattern not found!\n' >&2
    exit 1
fi

We can precede the command to be tested with ! to negate the test as well, to prevent us having to use else as well:

# Best
if ! grep -q regex options; then
    printf '%s\n' 'myscript: Pattern not found!' >&2
    exit 1
fi

An alternative syntax is to use && and || to perform if and else tests with grouped commands between braces, but these tend to be harder to read:

# Alternative
grep -q regex options || {
    printf '%s\n' 'myscript: Pattern not found!' >&2
    exit 1
}

With this syntax, the two commands in the block are only executed if the grep(1) call exits with a non-zero status. We can apply && instead to execute commands if it does exit with zero.

That syntax can be convenient for quickly short-circuiting failures in scripts, for example due to nonexistent commands, particularly if the command being tested already outputs its own error message. This therefore cuts the script off if the given command fails, likely due to ffmpeg(1) being unavailable on the system:

hash ffmpeg || exit 1

Note that the braces for a grouped command are not needed here, as there’s only one command to be run in case of failure, the exit call.

Calls to cd are another good use case here, as running a script in the wrong directory if a call to cd fails could have really nasty effects:

cd wherever || exit 1

In general, you’ll probably only want to test $? when you have specific non-zero error conditions to catch. For example, if we were using the --max-delete option for rsync(1), we could check a call’s return value to see whether rsync(1) hit the threshold for deleted file count and write a message to a logfile appropriately:

rsync --archive --delete --max-delete=5 source destination
if (($? == 25)); then
    printf '%s\n' 'Deletion limit was reached' >"$logfile"
fi

It may be tempting to use the errexit feature in the hopes of stopping a script as soon as it encounters any error, but there are some problems with its usage that make it a bit error-prone. It’s generally more straightforward to simply write your own error handling using the methods above.

For a really thorough breakdown of dealing with conditionals in Bash, take a look at the relevant chapter of the Bash Guide.

Linux Crypto: Importance

While this series was being written, from June 2013, Edward Snowden began leaking top-secret documents from the United States National Security Agency, showing that the agency was capable of Internet surveillance on a massive scale with the PRISM surveillance system and with the XKeyscore interface into their amassed data. The fact that covert government surveillance was possible and was taking place does not come as particularly surprising news to network engineers and conspiracy theorists, but the revelations have finally given the general, non-technical public an idea of how badly the proprietary systems around which they have built much of their digital lives can be used to harm them and compromise their privacy.

Concerned people in the United States will be only too aware of how the secret abuse of power to exercise this surveillance and the failed motions to curtail it by the United States Congress has dented their trust in their own government. However, the leaks’ implications are international as well. The foreign intelligence agency in my own country of New Zealand, the Government Communications Security Bureau, was earlier this year accused of illegally spying on New Zealand citizens, and diplomatic cables from WikiLeaks show the GCSB is potentially already cooperating with the NSA. In spite of this, new legislation is set to extend the GCSB’s powers, despite independent reviews condemning the bill from both a legal and human rights perspective, even after amendments. The scandal and the anger over surveillance abuse extends to the United Kingdom, Germany, Sweden, and many other countries.

I do hold out some hope for the efforts such as the Electronic Frontier Foundation’s class action suit to curtail the surveillance or at the very least to register the public’s anger about this unwarranted intrusion into private lives. However I am concerned not just by the possibility of the rise of a global surveillance state, but by the implications this has for the right to secure communications using cryptography for authentication and encryption.

It’s no secret that cryptography and encryption presents a problem to the NSA’s surveillance systems, and that they expend a great deal of effort in attempting to circumvent it, including demanding private keys from businesses for applications like HTTPS. My concern is this: If it becomes publically accepted that governments spy warrantlessly on international networks and that this is justified or necessary, then we may reach a point where the legality of the general public’s use of cryptography itself may again be called into question.

Computing professionals of my generation likely did not begin their careers until after the United States’ cryptographic export controls were relaxed in 1999, perhaps prompting us to take for granted the availability of algorithms like RSA and AES with high key sizes for cryptographic purposes. A world where a government agency would actively attempt to curtail the use of such technology may seem very far-fetched to us — perhaps less so to those who remember that Pretty Good Privacy was a radical new idea that caused its activist creator Phil Zimmermann real legal trouble.

I believe that computing enthusiasts and users of free software operating systems, not just cryptographic experts, are in a special position to assist their concerned friends and family with defending their online privacy and securing their communications, and that if we value both freedom and security of information as good hackers do, then we in fact have a responsibility to do so. I believe that people need to be aware of not just the implications of massive surveillance on a global scale, but also how to exercise their rights to fight against it. If the legality of cryptography is ever called into question again as the result of its impeding warrantless surveillance, then its pervasiveness and the public’s insistence on its free availability should make restricting its use not just impractical, but unthinkable.

This is why I support the Electronic Frontier Foundation, the Free Software Foundation, and anyone else who supports the freedoms and rights of everyone to use technology safely and privately. I hope anyone reading this will consider doing the same.

Linux Crypto: Disks

GnuPG provides us with a means to securely encrypt individual files on a filesystem, but for really high-security information or environments, it may be appropriate to encrypt an entire disk, to mitigate problems such as caching sensitive files in plaintext. Perhaps the most well-known means of doing this is with the cross-platform solution TrueCrypt, but the Linux kernel includes its own disk encryption solution in the kernel, dm-crypt. This can be leveraged with a low-end tool called cryptsetup, or more easily with LUKS, the Linux Unified Key Setup, implementing strong cryptography with passphrases or keyfiles.

In this example, we’ll demonstrate how this can work to encrypt a USB drive, which is a good method for securely storing really sensitive data such as PGP master keys that’s only needed occasionally, rather than leaving it always mounted on a networked device. Be aware that this erases any existing files on the drive.

Installation

The cryptographic tools used by dm-crypt and LUKS are built-in to Linux kernels after 2.6, but you may have to install a package to get access to the cryptsetup frontend. On Debian-derived systems, it’s available in cryptsetup:

# apt-get install cryptsetup

On RPM-based systems like Fedora or CentOS, the package has the same name, cryptsetup:

# yum install cryptsetup

Creating the volume

After identifying the block device on which we want the encrypted filesystem, for example /dev/sdc1, we can erase any existing contents using a call to wipefs:

# wipefs -a /dev/sdc1

Alternatively, we can zero out the whole disk, if we want to completely overwrite any trace of the data previously on the disk; this can take a long time for large volumes:

# cat /dev/zero >/dev/sdc1

If you don’t have a USB drive to hand, but would still like to try this out, you can use a loop block device in a file. For example, to create a 100MB loop device:

# dd if=/dev/zero of=/loopdev bs=1k count=102400
102400+0 records in
102400+0 records out
104857600 bytes (105 MB) copied, 0.331452 s, 316 MB/s
# losetup -f
/dev/loop0
# losetup /dev/loop0 /loopdev

You can then follow the rest of this guide using /dev/loop0 as the raw block device in place of /dev/sdc1. In the above output, losetup -f returns the first available loop device for use.

Setting up a LUKS container on the block device is then done as follows, providing a passphrase of decent strength; as always, the longer the better. Ideally, you should not use the same passphrase as your GnuPG or SSH keys.

# cryptsetup luksFormat /dev/sdc1

WARNING!
========
This will overwrite data on /dev/sdc1 irrevocably.

Are you sure? (Type uppercase yes): YES
Enter passphrase:
Verify passphrase:

This creates an abstracted encryption container on the disk, which can be opened by providing the appropriate passphrase. A virtual mapped device is then provided that encrypts all data written to it transparently, with the encrypted data written to the disk.

Using the mapped device

We can open the mapped device using cryptsetup luksOpen, which will prompt us for the passphrase:

# cryptsetup luksOpen /dev/sdc1 secret

The last argument here is the filename for the block device to appear under /dev/mapper; this example provides /dev/mapper/secret.

With this done, the block device on /dev/mapper/secret can now be used in the same way as any other block device; all of the disk operations are abstracted through encryption operations. You’ll probably want to create a filesystem on it; in this case, I’m creating an ext4 filesystem:

# mkfs.ext4 /dev/mapper/secret
mke2fs 1.42.8 (20-Jun-2013)
Filesystem label=
OS type: Linux
Block size=1024 (log=0)
Fragment size=1024 (log=0)
Stride=0 blocks, Stripe width=0 blocks
25168 inodes, 100352 blocks
5017 blocks (5.00%) reserved for the super user
First data block=1
Maximum filesystem blocks=67371008
13 block groups
8192 blocks per group, 8192 fragments per group
1936 inodes per group
Superblock backups stored on blocks:
        8193, 24577, 40961, 57345, 73729

Allocating group tables: done
Writing inode tables: done
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: done

We can then mount the device as normal, and data put into the newly created filesytem will be transparently encrypted:

# mkdir -p /mnt/secret
# mount /dev/mapper/secret /mnt/secret

For example, we could store a private GnuPG key on it:

# cp -prv /home/tom/.gnupg/secring.gpg /mnt/secret

Information about the device

We can get some information about the LUKS container and the specifics of its encryption using luksDump on the underlying block device. This shows us the encryption method used, in this case aes-xts-plain64.

# cryptsetup luksDump /dev/sdc1
LUKS header information for /dev/sdc1

Version:        1
Cipher name:    aes
Cipher mode:    xts-plain64
Hash spec:      sha1
Payload offset: 4096
MK bits:        256
MK digest:      87 6d 08 59 b2 f0 c6 6e ca ec 5f 72 2c e0 35 33 c2 9e cb 8e
MK salt:        7f a5 38 4c 14 85 61 cb 6c 22 65 48 87 21 60 8f
                fa 40 2a ab ae 7d cc df c9 9b a4 e3 3c 64 b6 bb
MK iterations:  49375
UUID:           f4e5f28c-3b34-4003-9bcd-dbb2352042ba

Key Slot 0: ENABLED
        Iterations:             197530
        Salt:                   2d 57 f6 2b 44 a6 61 ee d6 ee e4 7d 64 f0 71 d6
                                55 16 09 83 b4 f0 94 ca 19 17 11 a9 34 84 02 96
        Key material offset:    8
        AF stripes:             4000
Key Slot 1: DISABLED
Key Slot 2: DISABLED
Key Slot 3: DISABLED
Key Slot 4: DISABLED
Key Slot 5: DISABLED
Key Slot 6: DISABLED
Key Slot 7: DISABLED

Unmounting the device

When finished with the data on the device, we should both unmount any filesystem on it, and also close the mapped device so that the passphrase is required to re-open it:

# umount /mnt/secret
# cryptsetup luksClose /dev/mapper/secret

If the data is a removable device, you should also consider physically removing the media from the machine and placing it in some secure location.

This post only scratches the surface of LUKS functionality; many more things are possible with the system, including automatic mounting of encrypted filesystems and the use of stored keyfiles instead of typed passphrases. The FAQ for cryptsetup contains a great deal of information, including some treatment of data recovery, and the Arch Wiki has an exhaustive page on various ways of using LUKS securely.

Linux Crypto: Backups

While having local backups for quick restores is important, such as on a USB disk or spare hard drive, it’s equally important to have a backup offsite from which you can restore your important documents if, for example, your office was burgled or burned down, losing both your workstation and backup media.

The easiest way to do this for most people is with a storage provider, offering convenient access to bulk storage of suitable size maintained on another company’s systems for a relatively modest price or even for free, such as the Ubuntu One service, or Microsoft’s offering, Skydrive. The best storage providers will also encrypt the data on their own servers, whether or not they have access.

Trusting a company with all your data and the encryption thereof is risky, particularly given recent revelations of corporate collusion with the NSA, and privacy-conscious users should prefer the security of encrypting the backups before they go up onto the provider’s servers. The provider may implement closed and/or symmetric encryption mechanisms of their own, which may or may not be trustworthy. For very strong personal encryption, as established, we can use our GnuPG setup to encrypt files before we put them up there:

$ tar -cf docsbackup-"$(date +%Y-%m-%d)".tar $HOME/Documents
$ gpg --encrypt docsbackup-2013-07-27.tar
$ scp docsbackup-2013-07-27.tar.gpg user@backup.example.com:docsbackup

The problem with encrypting whole files before we put them up for storage is that for even modestly sized data, performing entire backups and uploading all of the files together every time can cost a lot of bandwidth. Similarly, we’d like to be able to restore our personal files as they were on a specific date, in case of bad backups or accidental deletion, but without storing every file on every backup day, which may end up requiring far too much space.

Incremental backups

Normally, the solution is to use an incremental backup system, meaning after first uploading your files in their entirety to the backup system, successive backups upload only the changes, storing them in a retrievable and space-efficient format. Systems like Dirvish, a free Perl frontend to rsync(1), allow this.

Unfortunately, Dirvish doesn’t encrypt the files or changesets it stores. What’s needed is an incremental backup solution that efficiently calculates and stores changes in files on a remote server, and also encrypts them. Duplicity, a Python tool built around librsync, excels at this, and can use our GnuPG asymmetric key setup for the file encryption. It’s available in Debian-derived systems in the duplicity package. Note that, as before, a GnuPG key setup with an agent is required for this to work.

Usage

We can get an idea of how duplicity(1) works by asking it to start a backup vault on our local machine. It uses much the same source destination argument as tools like rsync or scp:

$ cd
$ duplicity --encrypt-key tom@sanctum.geek.nz Documents file://docsbackup

It’s important to specify --encrypt-key, because otherwise duplicity(1) will use symmetric encryption with a passphrase rather than a public key, which is considerably less secure. Specify the email address corresponding to the public keypair you would like to use for the encryption.

This performs a full encrypted backup of the directory, returning the following output:

Local and Remote metadata are synchronized, no sync needed.
Last full backup date: none
No signatures found, switching to full backup.
--------------[ Backup Statistics ]--------------
StartTime 1374903081.74 (Sat Jul 27 17:31:21 2013)
EndTime 1374903081.75 (Sat Jul 27 17:31:21 2013)
ElapsedTime 0.01 (0.01 seconds)
SourceFiles 4
SourceFileSize 142251 (139 KB)
NewFiles 4
NewFileSize 142251 (139 KB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 4
RawDeltaSize 138155 (135 KB)
TotalDestinationSizeChange 138461 (135 KB)
Errors 0
-------------------------------------------------

You’ll note you were not prompted for your passphrase to do this. Remember, encrypting files with your public key does not require a passphrase; the whole idea is that anyone can encrypt using your key without needing your permission.

Checking the created directory docsbackup, we find three new files within it, all three of them encrypted:

$ ls -1 docsbackup
duplicity-full.20130727T053121Z.manifest.gpg
duplicity-full.20130727T053121Z.vol1.difftar.gpg
duplicity-full-signatures.20130727T053121Z.sigtar.gpg

The vol1.difftar.gpg file contains the actual data stored; the other two files contain metadata about the backup’s contents, for use to calculate differences the next time the backup runs.

If we make a small change to a file in the directory being backed up, and run the same command again, we note that the backup has been performed incrementally, and only the changes (the new file) have been saved:

$ duplicity --encrypt-key tom@sanctum.geek.nz Documents file://docsbackup
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Sat Jul 27 17:34:33 2013
--------------[ Backup Statistics ]--------------
StartTime 1374903396.52 (Sat Jul 27 17:36:36 2013)
EndTime 1374903396.52 (Sat Jul 27 17:36:36 2013)
ElapsedTime 0.01 (0.01 seconds)
SourceFiles 5
SourceFileSize 142255 (139 KB)
NewFiles 2
NewFileSize 4100 (4.00 KB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 2
RawDeltaSize 4 (4 bytes)
TotalDestinationSizeChange 753 (753 bytes)
Errors 0
-------------------------------------------------

We also find three new files in the docsbackup directory containing the new data:

$ ls -1 docsbackup
duplicity-full.20130727T053433Z.manifest.gpg
duplicity-full.20130727T053433Z.vol1.difftar.gpg
duplicity-full-signatures.20130727T053433Z.sigtar.gpg
duplicity-inc.20130727T053433Z.to.20130727T053636Z.manifest.gpg
duplicity-inc.20130727T053433Z.to.20130727T053636Z.vol1.difftar.gpg
duplicity-new-signatures.20130727T053433Z.to.20130727T053636Z.sigtar.gpg

Note that the new files have the prefix duplicity-inc- or duplicity-new-, denoting them as incremental backups and not full ones.

Note that in order to keep track of what files have already been backed up, duplicity(1) stores metadata in ~/.cache/duplicity, as well as storing them along with the backup. This allows us to let our backup processes run unattended, rather than having to put in our passphrase to read the metadata on the remote server before performing an incremental backup. Of course, if we lose our cached files, that’s OK; we can read the ones out of the backup vault by supplying our passphrase on request for decryption.

Remote backups

If you have SSH or even just SCP/SFTP access to your storage provider’s servers, not much has to change to make duplicity(1) store the files up there instead:

$ duplicity --encrypt-key tom@sanctum.geek.nz Documents sftp://user@backup.example.com:docsbackup

Your backups will then be sent over an SSH link to the directory docsbackup on the system backup.example.com, with username user. In this way, not only is all the data protected in transmission, it’s stored encrypted on the remote server; it never sees your plaintext data. All anyone with access to your backups can see is their approximate size, the dates they were made, and (if you publish your public key) the user ID on the GnuPG key used to encrypt them.

If you’re using the ssh-agent(1) program to store your decrypted private keys, you won’t even have to enter a passphrase for that.

The duplicity(1) frontend supports other methods of uploading to different servers, too, including the boto backend for S3 Amazon Web Services, the gdocs backend for Google Docs, and httplib2 or oauthlib for Ubuntu One.

If you like, you can also sign your backups to make sure they haven’t been tampered with at the time of restoration, by changing --encrypt-key to --encrypt-sign-key. Note that this will require your passphrase.

Restoring

Restoring from a duplicity(1) backup volume is much the same, but with the arguments reversed:

$ duplicity sftp://user@backup.example.com:docsbackup docsrestore
Synchronizing remote metadata to local cache...
GnuPG passphrase:
Copying duplicity-full-signatures.20130727T053433Z.sigtar.gpg to local cache.
Copying duplicity-full.20130727T053433Z.manifest.gpg to local cache.
Copying duplicity-inc.20130727T053433Z.to.20130727T053636Z.manifest.gpg to local cache.
Copying duplicity-new-signatures.20130727T053433Z.to.20130727T053636Z.sigtar.gpg to local cache.
Last full backup date: Sat Jul 27 17:34:33 2013

Note that this time you are asked for your passphrase. This is because restoring the backup requires decrypting the data and possibly the signatures in the backup vault. After doing this, the complete set of documents from the time of your most recent incremental backup will be available in docsrestore.

Using this incremental system also allows you to restore your data in the state in the last backup before a given time. For example, to retrieve my ~/Documents directory as it was three days ago, I might run this:

$ duplicity --time 3D \
    sftp://user@backup.example.com:docsbackup \
    docsrestore

You can extend this to only restore particular files for large vaults, if you only need a particular file from the vault:

$ duplicity --time 3D \
    --file-to-restore private/eff.txt \
    sftp://user@backup.example.com:docsbackup \
    docsrestore

Automating

You should run your first full backup interactively to make sure it’s doing exactly what you need, but once you’re confident that everything is working correctly, you can set up a simple Bash script to run incremental backups for you. Here’s an example script, saved in $HOME/.local/bin/backup-remote:

#!/usr/bin/env bash

# Run keychain to recognise any agents holding decrypted keys we might need
# (optional, depending on your SSH key setup)
eval "$(keychain --eval --quiet)"

# Specify directory to back up, GnuPG key ID, and remote username and
# hostname
keyid=tom@sanctum.geek.nz
local=/home/tom/Documents
remote=sftp://user@backup.example.com/docbackups

# Run backup with duplicity
/usr/bin/duplicity --encrypt-key "$keyid" -- "$local" "$remote"

The line with keychain is optional, but will be necessary if you’re using an SSH key with a passphrase on it; you’ll also need to have authenticated with ssh-agent at least once. See the earlier article on SSH/GPG agents for details on this setup.

Don’t forget to make the script executable:

$ chmod +x ~/.local/bin/backup-remote

You can then have cron(8) call this for you every week, running it as your user, by editing your user crontab(5) file:

$ crontab -e

The following line would run this script every morning, beginning at 6.00am:

0 6 * * *   ~/.local/bin/backup-remote

Tips

A few general best practices apply to this, consistent with the Tao of Backup:

  • Check that your backups completed; either have the output of the cron script mailed to you, or log it to a file that you check at least occasionally to make sure your backups are working. I highly recommend using an email message, and including error output:

    MAILTO=you@example.com
    0 6 * * *   ~/.local/bin/backup-remote 2>&1
    
  • Run backups to your local servers too; this might prevent your backup provider from reading your files, but it won’t save them from being accidentally deleted.

  • Don’t forget to occasionally test-restore your backups to make sure they’re working correctly. It’s also wise to use duplicity verify on them occasionally, particularly if you don’t back up every day:

    $ duplicity verify sftp://user@remote.example.com/docbackups Documents
    Local and Remote metadata are synchronized, no sync needed.
    Last full backup date: Sat Jul 27 17:34:33 2013
    GnuPG passphrase:
    Verify complete: 2195 files compared, 0 differences found.
    
  • This incremental system means that you’ll likely only have to make full backups once, so you should back up too much data rather than too little; if you can spare the bandwidth and have the space, backing up your entire computer isn’t really that extreme.

  • Try not to depend too much on your remote backups; see them as a last resort, and work securely and with local backups as much as you can. Certainly, never rely on backups as a version control system; use Git for that.

Linux Crypto: Email

An encrypted password storage is well and good, but now that we have a working GnuPG setup, we should consider using PGP for what it was originally designed: email messages. To do this, we’ll be using Mutt.

Mutt is a console-based mail user agent, or MUA, designed chiefly for managing and reading mail. Unlike mailer programs like Thunderbird, it was not designed to be a POP3/IMAP client, or an SMTP agent, although versions in recent years do include this functionality; these are tasks done by programs like Getmail and MSMTP.

If like many people, you’re using Gmail, this works very well with POP3/IMAP and SMTP, enabling you to compose email messages in plain text with your choice editor in a terminal window, in a highly configurable environment, and doing your own email encryption for any sensitive communications in such a way that even your email provider can’t read it.

General usage of Mutt and setup for Gmail users is not covered in detail here, although it may be the subject of a later article. For now, there are many excellent articles on the basics of a Mutt setup. If you’re interested in the setup for other Linux mail clients like Claws or Thunderbird, Cory Sadowski has a very good article walking you through that, among other privacy settings relevant to both GNU/Linux and Windows.

All of the below is assuming you already have a GnuPG keypair ready, with gpg-agent(1) running in the background to manage your keys.

Background

Most of the PGP setup guides for Mutt you can find online are quite old, and they usually suggest a lot of lines of .muttrc configuration to interface directly with the gpg command, with a myriad of options and some byzantine variable substitution:

set pgp_clearsign_command="gpg --no-verbose --batch --output - ...
set pgp_decode_command="gpg %?p?--passphrase-fd 0? --no-verbose ...
set pgp_decrypt_command="gpg --passphrase-fd 0 --no-verbose --batch ...
set pgp_encrypt_only_command="pgpewrap gpg --batch --quiet ...
set pgp_encrypt_sign_command="pgpewrap gpg --passphrase-fd 0 ...
set pgp_export_command="gpg --no-verbose --export --armor %r"
set pgp_import_command="gpg --no-verbose --import -v %f"
set pgp_list_pubring_command="gpg --no-verbose --batch --with-colons ...
set pgp_list_secring_command="gpg --no-verbose --batch --with-colons ...
set pgp_sign_command="gpg --no-verbose --batch --output - ...
set pgp_verify_command="gpg --no-verbose --batch --output - --verify %s %f"
set pgp_verify_key_command="gpg --no-verbose --batch --fingerprint ...

I’m all for the Unix philosophy of using programs together, but this is just too much. It’s a fickle setup that’s very hard to work with, and it requires too much understanding of the gpg(1) frontend to use and edit sensibly. After all, we want to end up with a setup that we understand reasonably well.

So, throw all that away; we’re going to use GPGME instead. The above is exactly the problem that this library is designed to solve; it’s a library to which applications can link to streamline the usage of GnuPG functions, including interfacing with agents. We can replace all of the above with this:

set crypt_use_gpgme = yes

Installation

If you have Mutt installed, odds are it already has a GPGME interface. You can check if your current version of Mutt has GPGME powers by looking at the mutt -v version output. Here’s the output of mine, using the packaged Mutt from Debian GNU/Linux, which does have GPGME support:

$ mutt -v | grep -i gpgme
+CRYPT_BACKEND_CLASSIC_PGP  +CRYPT_BACKEND_CLASSIC_SMIME  +CRYPT_BACKEND_GPGME
upstream/548577-gpgme-1.2.patch

If you don’t have a version of Mutt with GPGME, you can build your own by downloading the source and building it with --enable-gpgme:

$ ./configure --enable-gpgme
$ make
# make install

You may need to make sure you have the GPGME library and headers installed first:

# apt-get install libgpgme11 libgpgme11-dev

Setup

Add the following lines to your .muttrc file; remove anything else beginning with crypt_* or pgp_*:

# Use GPGME
set crypt_use_gpgme = yes

# Sign replies to signed email
set crypt_replysign = yes

# Encrypt replies to encrypted email
set crypt_replyencrypt = yes

# Encrypt and sign replies to encrypted and signed email
set crypt_replysignencrypted = yes

# Attempt to verify email signatures automatically
set crypt_verify_sig = yes

Restart Mutt, and you should be ready to go.

Usage

First of all, check that you have the public key for your intended recipient available in your GnuPG keychain:

$ gpg --list-keys joe@example.com

If you’re able to download it from somewhere, a useful formula is to download it with curl(1) and import it directly into gpg(1):

$ curl http://www.example.com/joe-somebody.asc | gpg --import
gpg: key 1234ABCD: public key "Joe Somebody <joe@example.com>" imported
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)
gpg: no ultimately trusted keys found

Remember, it’s your responsibility to decide how much you trust this public key; normally it’s best practice that you actually meet someone in person to exchange written key fingerprints in order to be completely sure that the key corresponds to that user.

If you don’t have anyone else you know using PGP to communicate with, you can send me a message encrypted with my public key 0x77BB8872 to tom@sanctum.geek.nz. If you send or link me your public key in your message, then I’ll reply to you with a message encrypted with your public key so you can check everything’s working.

$ curl http://static.sanctum.geek.nz/thomas-ryder.asc | gpg --import

Back in Mutt, begin composing a message with m. Enter the recipient and subject as normal, and compose your message. When you’ve finished writing and save and quit $EDITOR, and your message is in the Compose screen waiting to send, press p to bring up the PGP menu at the bottom:

PGP (e)ncrypt, (s)ign, sign (a)s, (b)oth, s/(m)ime or (c)lear?

We’ll press b to both sign and encrypt the message.

If you want to be able to read the message after sending, then you’ll need to arrange to encrypt it with your key as well as the recipient’s. I find the cleanest way to do this is to add your address to the Bcc: header with b. You can also set this as a default with the following line in ~/.gnupg/gpg.conf, where 0x1234ABCD is the short ID of your own key:

encrypt-to 0x1234ABCD

Mutt compose screen

When you send the message with y, you might need to specify which key you want to use for each recipient, if you don’t have a unique key on your keychain with your recipient’s email address.

When you send, you should be prompted for your passphrase by your PIN entry program, unless your agent is already holding the key for you. This is needed in order to sign the message. When you’ve provided this, the message will be sent, and if you included yourself in the Bcc: field, you should be able to read it in your sent mail, with some headers showing the PGP information (whether the message was signed, encrypted, or both):

Mutt sent message

Your recipient will be able to decrypt the message in their mail user agent with their private key, and nobody else but the two of you will be able to read it. Note that this works for any number of recipients, as long as you have a public key for each of them.

Keep in mind that the metadata of the message, such as the sender and recipient name and address, date and time it was sent, and (importantly) the subject, are sent in plain text. Only the body of the message (including attachments) is encrypted.

Useful extras

With GPGME, Mutt tries to use the first secret key available to it in its private keychain. If you want to use some other specific keypair for signing messages, you can specify that with with the pgp_sign_as option in .muttrc:

set pgp_sign_as = 0x9876FEDC

If you’d like to automatically sign all of your outgoing mail, you can set the crypt_autosign option:

set crypt_autosign = yes

The first batch of options we set earlier will already automatically sign and/or encrypt messages in responses to messages doing either/both.

If you’d like to include a link to your PGP key in the headers to each message, you can add a custom header with my_hdr:

my_hdr X-PGP-Key: http://static.sanctum.geek.nz/thomas-ryder.asc

All of this combines with Mutt’s extensive speed and high-powered configuration to make Mutt a very capable and convenient PGP mail client. As always, the more people you know using PGP, and the more public keys you have, the more useful this will be.

Linux Crypto: Passwords

It’s now becoming more widely known that using guessable passwords or using the same password for more than one account is a serious security risk, because an attacker able to control one account (such as an email account) can do a lot of damage. If an attacker gets the hash of your password from some web service, you want to be assured that the hash will be very difficult to reverse, and even if it can be reversed, that it’s unique and won’t give them access to any of your other accounts.

This growing awareness has contributed to the popularity of password managers, tools designed to securely generate, store, and retrieve passwords, encrypted with a master password or passphrase. In some cases these are locally stored, such as KeePass, and in others they are stored on a web service, such as LastPass. Both are good tools, and work well with Linux. I personally have some reservations about LastPass as I don’t want my passwords stored on a third party service, and I don’t trust JavaScript encryption.

Interestingly, because we now have a tidy GnuPG setup to handle the encryption ourselves, another option is the pass(1) tool, billing itself as “the standard UNIX password manager”. It’s little more than a shell script and some bash(1) completions wrapped around existing tools like git(1), gpg2(1), pwgen(1), tree(1), and xclip(1), and your choice of $EDITOR. If you’re not already invested in an existing password management method, you might find this a good first application of your new cryptography setup, and a great minimal approach to secure password storage accessible from the command line (and therefore SSH).

On Debian-derived systems, it’s available as part of the pass package:

# apt-get install pass

This includes a manual:

$ man pass

Instructions for installing on other operating systems are also available on the site. Releases are also available for download, and a link to the development repository. If you use this, make sure you have the required tools outlined above installed as well, although xclip(1) is only needed if you run the X Windows system.

Setup

We can get an overview of what pass(1) can do by invoking it with no arguments:

$ pass

To start, we’ll initialize our password store. For your own passwords, you will want to do this as your own user rather than root. Because pass(1) uses GnuPG for its encryption, we also need to tell it the ID of the appropriate key to use. Remember, you can find this eight-digit hex code by typing gpg --list-secret-keys. A unique string identifying your private key such as your name or email address may also work.

$ pass init 0x77BB8872
mkdir: created directory ‘/home/tom/.password-store’
Password store initialized for 0x77BB8872.

Indeed, we note the directory ~/.password-store has been created, although it’s presently empty except for the .gpg-id file recording our key ID:

$ find .password-store
.password-store
.password-store/.gpg-id

Inserting

We’ll insert an existing password of ours with pass insert, giving it a descriptive hierarchical name:

$ pass insert google.com/gmail/example@gmail.com
mkdir: created directory ‘/home/tom/.password-store/google.com’
mkdir: created directory ‘/home/tom/.password-store/google.com/gmail’
Enter password for google.com/gmail/example@gmail.com:
Retype password for google.com/gmail/example@gmail.com:

The password is read from the command line, encrypted, and placed in ~/.password-store:

$ find .password-store
.password-store
.password-store/google.com
.password-store/google.com/gmail
.password-store/google.com/gmail/example@gmail.com.gpg
.password-store/.gpg-id

Notice that pass(1) creates a directory structure for us automatically. We can get a nice view of the password store with pass with no arguments:

$ pass
Password Store
└── google.com
    └── gmail
            └── example@gmail.com

Generating

If you’d like it to generate a new secure random password for you, you can use generate instead, including a password length as the last argument:

$ pass generate google.com/gmail/example@gmail.com 16
The generated password to google.com/gmail/example@gmail.com is:
!Q%i$$&q1+JJi-|X

If you have some service that doesn’t cooperate with symbols in passwords, you can add the -n option to this call:

$ pass generate -n google.com/gmail/example@gmail.com 16
The generated password to google.com/gmail/example@gmail.com is:
pJeF18CrZEZzI59D

pass(1) uses pwgen(1) for this password generation. In each case, the password is automatically inserted into the password store for you.

If we need to change an existing password, we can either overwrite it with insert again, or use the edit operation to invoke our choice of $EDITOR:

$ pass edit google.com/gmail/example@gmail.com

If you do this, you may like to be careful that your editor is not configured to keep backups or swap files in plain text of documents it edits in temporary directories or memory filesystems. If you’re using Vim, I wrote some configuration in an attempt to solve this problem.

Note that adding or overwriting passwords does not require your passphrase; only retrieval and editing does, consistent with how GnuPG normally works.

Retrieval

This password can now be retrieved and echoed onto the command line given the appropriate passphrase:

$ pass google.com/gmail/example@gmail.com
(...gpg-agent pinentry prompt...)
Tr0ub4dor&3

If you’re using X windows and have xclip(1) installed, you can put the password on the clipboard temporarily to paste into web forms:

$ pass -c google.com/gmail/example@gmail.com
Copied google.com/gmail/example@gmail.com to clipboard. Will clear in 45 seconds.

In each case, note that if you have the bash completion installed and working, you should be able to complete the full path to the passwords with Tab, just as if you were directly browsing a directory hierarchy.

Deletion

If we no longer need the password, we can remove it with pass rm:

$ pass rm google.com/gmail/example@gmail.com
Are you sure you would like to delete google.com/gmail/example@gmail.com? [y/N] y
removed ‘/home/tom/.password-store/google.com/gmail/example@gmail.com.gpg’

We can delete whole directories of passwords with pass rm -r:

$ pass rm -r google.com
Are you sure you would like to delete google.com? [y/N] y
removed ‘/home/tom/.password-store/google.com/gmail/example@gmail.com.gpg’
removed directory: ‘/home/tom/.password-store/google.com/gmail’
removed directory: ‘/home/tom/.password-store/google.com’

Version control

To keep historical passwords, including deleted ones if we find we do need them again one day, we can set up some automatic version control on the directory with pass git init:

$ pass git init
Initialized empty Git repository in /home/tom/.password-store/.git/
[master (root-commit) 0ebb933] Added current contents of password store.
 1 file changed, 1 insertion(+)
 create mode 100644 .gpg-id

This will update the repository every time the password store is changed, meaning we can be confident we’ll be able to retrieve old passwords we’ve replaced or deleted:

$ pass insert google.com/gmail/newexample@gmail.com
mkdir: created directory ‘/home/tom/.password-store/google.com’
mkdir: created directory ‘/home/tom/.password-store/google.com/gmail’
Enter password for google.com/gmail/newexample@gmail.com:
Retype password for google.com/gmail/newexample@gmail.com:
[master 00971b6] Added given password for google.com/gmail/newexample@gmail.com to store.
 1 file changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 google.com/gmail/newexample@gmail.com.gpg

Backups

Because the password files are all encrypted only to your GnuPG key, you can relatively safely back up the store on remote and third-party sites simply by copying the ~/.password-store directory. If the filenames themselves contain sensitive information, such as private usernames or sites, you might like to back up an encrypted tarball of the store instead:

$ tar -cz .password-store \
    | gpg --sign --encrypt -r 0x77BB8872 \
    > password-store-backup.tar.gz.gpg

This directory can be restored in a similar way:

$ gpg --decrypt \
    < password-store-backup.tar.gz.gpg \
    | tar -xz 

Linux Crypto: Agents

Now that we have both GnuPG and SSH securely set up, we’re able to encrypt, decrypt, sign, and verify messages, and securely authenticate to remote servers without any risk of exposing passwords and with effectively zero possibility of a brute-force attack. This is all great, but there’s still one more weak link in the chain with which to deal — our passphrases.

If you use these tools often, typing your passphrase for most operations can get annoying. It may be tempting to either include a means of automating the passphrase entry, or not to use a passphrase at all, leaving your private key unencrypted. As security-conscious users, we definitely want to avoid the latter in case our private key file ever gets stolen, which is where the concepts of agents for both SSH and GnuPG comes into play.

An agent is a daemon designed to streamline the process of using decrypted private keys by storing the details securely in memory, ideally for a limited period of time. What this allows you do with both SSH and GnuPG is to type your passphrase just once, and subsequent uses that require the unencrypted private key are managed by the agent.

In this article, we’ll go through the basics of agent setup for both SSH and GnuPG. Once we know how they work, we’ll then introduce a convenient tool to start both of them and manage them for us easily.

SSH agents

The ssh-agent(1) program comes as part of the OpenSSH suite. It can be run in two modes, either as a parent process, or daemonized into the background. We’ll discuss the latter method, as it’s more commonly used and more flexible.

Setup

When we run ssh-agent(1) for the first time, its behavior is curious; it appears to do nothing except spit out some cryptic shell script:

$ ssh-agent 
SSH_AUTH_SOCK=/tmp/ssh-EYqoH3qwfvbe/agent.28881; export SSH_AUTH_SOCK;
SSH_AGENT_PID=28882; export SSH_AGENT_PID;
echo Agent pid 28882;

However, we can see that the daemon is running with the PID it mentions:

$ ps 28882
  PID TTY      STAT   TIME COMMAND
28882 ?        Ss     0:00 ssh-agent

So if it’s running fine, what’s with all the shell script it outputs? Why doesn’t it just run that for us?

The answer is an interesting workaround to a stricture of the Unix process model; specifically, a process cannot modify its parent environment. The variables SSH_AUTH_SOCK and SSH_AGENT_PID are designed to allow programs like ssh(1) to find the agent so it can communicate with it, so we definitely need them set. However, if ssh-agent(1) were to set these variables itself, it would only apply for its own process, not the shell where we called it.

Therefore, not only do we need to run ssh-agent(1), we need to execute the code it outputs so the variables get assigned in our shell. A good method of doing this in Bash is using eval and command substitution with $(...):

$ eval $(ssh-agent)
Agent 3954

If we run this, we can see that not only is ssh-agent(1) running, we have two new variables in our environment identifying its socket path and process ID:

$ pgrep ssh-agent
3954
$ env | grep ^SSH
SSH_AUTH_SOCK=/tmp/ssh-oF1sg154ygSt/agent.3953
SSH_AGENT_PID=3954

With this done, the agent is ready, and we can start using it to manage our keys for us.

Usage

The next step is to load our keys into the agent with ssh-add(1). Pass this program the full path to the private key you would like to use with the agent. This is likely either ~/.ssh/id_rsa or ~/.ssh/id_dsa:

$ ssh-add ~/.ssh/id_rsa
Enter passphrase for /home/tom/.ssh/id_rsa:
Identity added: /home/tom/.ssh/id_rsa (/home/tom/.ssh/id_rsa)

You can leave out the filename argument if you want ssh-add to add any or all of the default key types in ~/.ssh if they exist (id_dsa, id_rsa, and id_ecdsa):

$ ssh-add

Either way, you should be prompted for your passphrase; this is expected, and you should go ahead and type it in.

If we then ask ssh-add(1) to list the keys it’s managing, we see the key we just added:

$ ssh-add -l
4096 87:ec:57:8b:ea:24:56:0e:f1:54:2f:6b:ab:c0:e8:56 /home/tom/.ssh/id_rsa (RSA)

With this done, if we try to use this key to connect to another server, we no longer need to provide the passphrase; we’re just logged straight in:

tom@local:~$ ssh remote
Welcome to remote.sanctum.geek.nz, running GNU/Linux!
tom@remote:~$

The default is to maintain the keys permanently, until the agent is stopped or the keys are explicitly removed one-by-one with ssh-add -d <keyfile> or all at once with ssh-add -D. For the cautious, you can set a time limit in seconds with ssh-add -t. For example, to have ssh-add forget about your keys after two hours, you might use:

$ ssh-add -t 7200 ~/.ssh/id_rsa

To kill the agent completely, you can use ssh-agent -k, again with an eval $(...) wrapper:

$ eval $(ssh-agent -k)
Agent pid 4501 killed

You may like to consider adding this to ~/.bash_logout or a similar script to get rid of the running agent after you’re done with your session.

Permanent setup

If you like this and find it makes your key management more convenient, it makes sense to put it into a startup script like ~/.bash_profile. This way, the agent will be started for each login shell, and we will be able to communicate with it from any subshell (xterm, screen, or an appropriately configured tmux):

eval $(ssh-agent)
ssh-add ~/.ssh/id_rsa

On our next TTY login, we should be prompted for a passphrase, and from there be able to connect to any machine using the keys managed by the agent:

tom@local:~$ ssh remote
Welcome to remote.sanctum.geek.nz, running GNU/Linux!

If you want this to work for a desktop manager like GDM or XDM, you can add a variable pointing to the ssh-askpass(1) program:

eval $(ssh-agent)
export SSH_ASKPASS=/usr/bin/ssh-askpass
ssh-add ~/.ssh/id_rsa

If SSH_ASKPASS is set like this and DISPLAY refers to a working display, then a simple graphical prompt will appear asking for your passphrase:

ssh-askpass prompting for a passphrase

This program may need to be installed separately. Under Debian-derived systems, its package name is ssh-askpass.

All child processes and subshells of the login shell will inherit the agent’s variables, since they were exported with export:

tom@local:~$ screen
tom@local:~$ tmux bash
tom@local:~$ bash
tom@local:~$ ssh remote
Welcome to remote.sanctum.geek.nz, running GNU/Linux!
tom@remote:~$

We thus have to type our passphrase only once per login session, and can connect to all of the servers to which our keys confer access … very convenient!

GnuPG Agents

Just like ssh-agent(1), there exists an agent for managing GnuPG keys too, called gpg-agent(1). Its behavior is very similar. On Debian-derived systems, it can be installed as the gnupg-agent package. You should also install a pinentry program; as we’re focussing on learning the nuts and bolts on the command line here, we’ll use pinentry-curses(1) for a console-based passphrase prompt:

# apt-get install gnupg-agent pinentry-curses

Setup

We’ll start the agent using the same eval $(...) trick we learned with ssh-agent:

$ eval $(gpg-agent --daemon)

We can verify that the agent is running in the background with the given PID, and that we have a new environment variable:

$ pgrep gpg-agent
5131
$ env | grep ^GPG
GPG_AGENT_INFO=/tmp/gpg-hbro8r/S.gpg-agent:5131:1

We’ll also set GPG_TTY, which will help the pinentry program know on which terminal to draw its passphrase request screen:

$ export GPG_TTY=$(tty)
$ echo $GPG_TTY
/dev/pts/2

Finally, to prod gpg(1) into actually using the agent, we need to add a line to ~/.gnupg/gpg.conf. You can create this file if it doesn’t exist.

use-agent

Usage

With this done, if we try to do anything requiring our private key, we should be prompted for a passphrase not directly on the command line, but by our PIN entry program:

$ gpg --armor --sign message1.txt

┌──────────────────────────────────────────────────────────┐
│ You need a passphrase to unlock the secret key for user: │
│ "Thomas Ryder (tyrmored, tejr) <tom@sanctum.geek.nz>"    │
│ 4096-bit RSA key, ID 25926609, created 2013-03-12        │
│ (main key ID 77BB8872)                                   │
│                                                          │
│                                                          │
│ Passphrase ***__________________________________________ │
│                                                          │
│       <OK>                                 <Cancel>      │
└──────────────────────────────────────────────────────────┘

When we enter the passphrase, our operation is performed:

$ ls message1*
message1.txt
message1.txt.asc

Afterwards, if we perform another option requiring the private key, we see that we are not prompted:

$ gpg --armor --sign message2.txt
$ ls message2*
message2.txt
message2.txt.asc

The agent has thus cached the private key for us, making it much easier to perform a series of operations with it. The default timeout is 10 minutes, but you can change this with the default-cache-ttl and max-cache-ttl settings in ~/.gnupg/gpg-agent.conf. For example, to retain any private key for one hour after its last use and a maximum of two hours from its first use, we could write:

default-cache-ttl 3600
max-cache-ttl 7200

Changing these values will require prompting the agent to reload:

$ gpg-connect-agent <<<RELOADAGENT
OK

Permanent setup

Just like ssh-agent(1), an ideal place for gpg-agent(1)‘s startup lines is in a login shell setup script like ~/.bash_profile:

eval $(gpg-agent --daemon)

The agent will be started, and all of its environment variables will be set and inherited by all subshells, just as with ssh-agent.

If you’re using the console PIN entry tool, you should also add this to end of your interactive shell startup script. This should be ~/.bashrc for Bash on Linux; you may need to put it in ~/.bashrc on Mac OS X.

export GPG_TTY=$(tty)

Keychain

To manage both ssh-agent(1) and gpg-agent(1) effectively, a tool called keychain(1) is available. It provides a simple way to start both agents with one command, including loading keys at startup, and also prevents running either agent twice, picking up on agents started elsewhere on the system. Because desktop environments are often configured to start one or both agents for users, it makes sense to re-use them where possible, at which keychain(1) excels.

On Debian-derived systems, the program is available in the keychain package:

# apt-get install keychain

With keychain installed, we can start both agents with just one command in ~/.bash_profile:

eval $(keychain --eval)

We can optionally include the filenames of SSH keys in ~/.ssh or the hex IDs of GnuPG keys as arguments to prompt loading the private key (including requesting the passphrase) at startup:

eval $(keychain --eval id_rsa 0x77BB8872)

If this program is available to you, then I highly recommend this; managing agents and environments can be fiddly, and keychain(1) does all the hard work for you in this regard so you don’t have to worry about whether an agent is available to you in your particular context. Check out the project’s homepage for more information about the tool.

Linux Crypto: SSH keys

The usual method of authenticating to an OpenSSH server is to type your shell password for the remote machine:

tom@local:~$ ssh remote
The authenticity of host 'remote (192.168.0.64)' can't be established.
RSA key fingerprint is d1:35:45:a6:d1:b2:e4:08:f8:67:b1:19:fe:04:ca:1c.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'remote,192.168.0.64' (RSA) to the list of known hosts.
tom@remote's password:

tom@remote:~$

This is appropriate for first contact with a machine, and the authentication method is supported out of the box for most OpenSSH sshd(8) installations.

sshd(8) is a very common target for attacks, particularly automated ones; malicious bots attempt to connect to servers listening on the standard SSH destination port of tcp/22, as well as some common alternatives like tcp/2222. If you enforce a strong password policy on your system, this generally isn’t too much of a problem, particularly if only appropriate users have shells, or if you restrict SSH connections only to certain usernames or groups.

There are other measures to defeat automated attacks, such as employing systems like fail2ban to reject clients who make too many spurious connection attempts, but perhaps the most effective way of short-circuiting automated attacks is to bypass passwords completely and instead use SSH keys, allowing this as the only connection method to the relevant machines.

How it works

Similar to the GnuPG keys setup in the first two articles in this series, SSH keypairs are comprised of one private key and one public key, two cryptographically linked files. The basis of keys for authentication is that if someone has your public key, they’re able to authenticate you by requesting operations that you would only be able to perform with the corresponding private key; it works similarly to cryptographic signing.

The reason this is so effective is because if you require a valid public key to authenticate, with sufficient key length it’s effectively impossible for an attacker to guess your authentication details; there’s no such thing as a “common” private key to guess, so they would need to run through every possible private key, which is not even remotely practical.

Your system’s sshd(8) may still be attacked, but if you use only public key authentication, then you can be comfortably certain it’s effectively impossible to brute-force your credentials. Note that this doesn’t necessarily protect you from security problems in sshd(8) itself, however, and you will still need to protect your private key from being hijacked or compromised, hence the necessity of a passphrase.

All of the below assumes you have OpenSSH installed as both the client and the server on the appropriate systems. On Debian-derived systems, these can be installed with:

# apt-get install ssh
# apt-get install openssh-server

Both the client and server often come standard with systems (e.g. their native OpenBSD).

Generating keys

Similar to the GnuPG setup process, we start by generating a keypair on the machine from which we’d like to connect, using ssh-keygen(1). I’m using 4096-bit RSA here, as it’s widely supported even on very old systems, and should be relatively future-proof, although generating new keys if RSA ever becomes unsafe is not hard. If you’d prefer to use the newer ECDSA algorithm that’s the default in recent versions of OpenSSH, all of this will still work. I’m also applying a comment for the key as an unencrypted identifier to distinguish multiple keys if I have them. I find email addresses work well.

$ ssh-keygen -t rsa -b 4096 -C tom@sanctum.geek.nz
Generating public/private rsa key pair.

First, we’re prompted for a location to which the key files should be saved. I recommend accepting the default by pressing Enter, as using the default location makes the next few steps easier:

Enter file in which to save the key (/home/tom/.ssh/id_rsa):

Next, we’re prompted for a passphrase, which should definitely be added to keep the key from being used if it’s ever compromised. The same guidelines for passphrases apply to SSH here, and you should choose a different passphrase:

Enter passphrase (empty for no passphrase):
Enter same passphrase again:

This done, the key is generated, including a pictorial representation to recognise keys at a glance. I’ve never found this very useful, but the key fingerprint is helpful:

Your identification has been saved in /home/tom/.ssh/id_rsa.
Your public key has been saved in /home/tom/.ssh/id_rsa.pub.
The key fingerprint is:
d5:81:8c:eb:c6:c5:a2:b9:6a:ae:32:cc:20:bf:cf:66 tom@local
The key's randomart image is:
+--[ RSA 4096]----+
|         o ..    |
|        . o. .   |
|         o. .    |
|        o.o      |
|       =So       |
|o     o +        |
|=.     o         |
|oo..E .          |
| ooO=.           |
+-----------------+

The key files should now be available in ~/.ssh:

$ ls -l .ssh
-rw-------  1 tom  tom  3326 Apr  2 22:47 id_rsa
-rw-r--r--  1 tom  tom   754 Apr  2 22:47 id_rsa.pub

The id_rsa file contains the encrypted private key, and should be kept locked down and confidential. The id_rsa.pub file, however, contains the public key, which can be safely distributed, in the same way as a PGP public key.

Key-based authentication

We can now arrange to use the newly generated public key for authentication in lieu of a password. Start by ensuring you can connect to the remote machine with your username and password:

$ ssh remote
tom@remote's password:

Once connected, ensure that the ~/.ssh directory exists on the remote machine, and that you don’t already have keys listed in ~/.ssh/authorized_keys, as we’re about to overwrite them:

$ mkdir -p ~/.ssh
$ chmod 0700 ~/.ssh

If this worked, close the connection (exit or Ctrl-D) to return to your local machine’s shell, and copy your public key onto the remote machine with scp(1):

$ scp ~/.ssh/id_rsa.pub remote:.ssh/authorized_keys
tom@remote's password:
id_rsa.pub    100%    754    0.7KB/s    00:00

Note that there’s a tool included in recent versions of OpenSSH that does this for you called ssh-copy-id(1), but it’s good to have some idea of what it’s doing in the background.

With this done, your next connection attempt to the remote host should prompt you for your passphrase, rather than your password:

$ ssh remote
Enter passphrase for key '/home/tom/.ssh/id_rsa':

Advantages

At first, it may not seem like you’ve done much useful here. After all, you still have to type in something to connect each time. From a security perspective, the first major advantage to this method is that neither your password, nor your passphrase, nor your private key are ever transmitted to the server to which you’re connecting; authentication is done purely based on the public-private key pair, decrypted by your passphrase.

This means that if the machine you’re connecting to were compromised, or your DNS had been poisoned, or some similar attack tricked you into connecting to a fake SSH daemon designed to collect credentials, your private key and your password remain safe.

The second advantage comes with turning off password authentication entirely on the host machine, once all its users have switched to public key authentication only. This is done with the following settings in sshd_config(5), usually in /etc/ssh/sshd_config on the remote server:

PubkeyAuthentication yes
ChallengeResponseAuthentication no
PasswordAuthentication no

Restart the SSH server after these are applied:

$ sudo /etc/init.d/ssh restart

You should then no longer be able to connect via passwords at all, only by private keys, which as mentioned above are effectively (though not literally) impossible to brute-force. In order to connect to the server as you, an attacker would not only need to know your passphrase, but also have access to your private key, making things significantly harder.

Using public key authentication also allows sshd(8) some finer-grained control over authentication, such as which hosts can connect with which keys, whether they can execute TCP or X11 tunnels, and (to an extent) which commands they can run once connected. See the manual page for authorized_keys(5) to take a look at some examples.

Finally, there’s a major usability advantage in using SSH keys for authentication with agents, which we’ll discuss in the next article.

Host keys and fingerprints

SSH connection should ideally be a two-way authentication process. Just as the server to which you’re connecting needs to be sure who you are, you need to be sure that the host you’re connecting to is the one you expect. With tunnelling, firewalls, DNS poisoning, NAT, hacked systems, and various other tricks, it’s appropriate to be careful that you’re connecting to the right systems. This is where OpenSSH’s host key system comes into play.

The first time you connect to a new server, you should see prompts like this:

$ ssh newremote
The authenticity of host 'newremote (192.168.0.65)' can't be established.
RSA key fingerprint is f4:4b:f4:8c:c5:50:f6:c8:d3:b2:e9:14:68:86:b5:7b.
Are you sure you want to continue connecting (yes/no)?

A lot of administrators turn these off; don’t! They are very important.

The key fingerprint is a relatively short hash for the host key used by OpenSSH on that server. It’s verified by your SSH client, and can’t easily be faked. If you’re connecting to a new server, it’s appropriate to check the host key fingerprint matches the one you see on first connection attempt, or to ask the system’s administrator to do so for you.

The host key’s fingerprint can be checked on the SSH server with a call to ssh-keygen(1):

$ ssh-keygen -lf /etc/ssh/ssh_host_rsa_key
2048 f4:4b:f4:8c:c5:50:f6:c8:d3:b2:e9:14:68:86:b5:7b /etc/ssh/ssh_host_rsa_key.pub (RSA)

If you want, you can check the key without making a connection attempt with a similar call on the client side:

$ ssh-keygen -lF newremote
# Host 192.168.0.65 found: line 1 type RSA
2048 f4:4b:f4:8c:c5:50:f6:c8:d3:b2:e9:14:68:86:b5:7b newremote (RSA)

If you can’t check the host key yourself, have the administrator send it to you over a secure, trusted channel, such as in person or via a PGP signed message. If the colon-delimited SSH fingerprint is not exactly the same, then you might be the victim of someone attempting to spoof your connection!

This is definitely overkill for new virtual machines and probably new machines on a trusted LAN, but for machines accessed over the public internet, it’s a very prudent practice.

Similarly, ssh(1) by default keeps a record of the host keys for hosts, which is why when a different host key is presented on a connection attempt, it warns you:

$ ssh newremote
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
d7:06:51:16:80:f6:32:b4:35:7c:53:8d:5a:49:69:ec
Please contact your system administrator.
Add correct host key in /home/tom/.ssh/known_hosts to get rid of this message.
Offending RSA key in /home/tom/.ssh/known_hosts:22
RSA host key for newremote has changed and you have requested strict checking.

Again, this is something ssh(1) users often turn off, which is pretty risky, especially if you are using password authentication and hence might send your password to a malicious or compromised server!

Linux Crypto: GnuPG Usage

With our private and public key generated and stored, we can start using a few of GnuPG’s features to sign, verify, encrypt, and decrypt files and messages for distribution over untrusted channels like the internet.

Signing a message or text file

We’ll start by signing a simple text file, using the --clearsign option. This includes the signature in the message, which we can then distribute to people to read. Here’s the contents of message.txt:

This is a public message from Tom Ryder.

We’ll sign that with our new private key like so:

$ gpg --clearsign message.txt

We’re prompted for our passphrase for the private key:

You need a passphrase to unlock the secret key for
user: "Tom Ryder (Test Key Only) <tom@sanctum.geek.nz>"
4096-bit RSA key, ID 040FE79B, created 2013-03-23

Having provided that, the file message.txt.asc is created, with PGP sections and a plaintext ASCII signature:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

This is a public message from Tom Ryder.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)

iQIcBAEBCAAGBQJRTQcTAAoJEGIHQ4oED+ebtjoP/19PlndkGhR46BA6YZmDVdC1
Snk9aXe4Eo42kRpW13bjo8xg+pb+U26ylHkH52OBB7fJ3/BR5eZMug/RXJLGIf+U
aiylbGVz4dGjkeTCSxtg1TvcyEtzhm1ETfIWLarboj9PPBJfO1QIh4uPDkD7kb6+
+OnpdxPURbiJ03osu2Mj2fFq5wYT0La+I9BdKUAJmS8zt+CaTirKw+xF6l0sEAVv
lqslWjEwF9JCfumKAj8aMBeZMndoKRqW18ZHoYJdP0x3g1SUKAjZ/NRRUGug6Eg+
JTJ82ETCRKGKmYFLkHJ6iCaucrmhLTd9IYyEQZE/weUuClKqtshollhNHF1dD3SF
fWMPq0+29DInjlXwIkXyzVDln1wULNzbd5zv5Wg5b6lSCZlwH0xCrNjiYO7f413c
Ty4q6SqtRFUommnpMA5XcmX7ebbUpMfqfqqzoLqeTpA15Yuhh3DDR6NoMN82oLyF
FFt7UZh/JlYMc8GOnEqyZfT7d57FbKSLn3vpZbH9QXNFWG6/oZabFFyRm8r7k8F2
FzYTdyp59O0dW4T5OJ6a/xo/OnZutUN1RqW6ZJS19Xb4/5eEohFAFL9cDVLuC6zo
HmlA2m5zwv8aYbJad6Rk6+vpQAAdxHgNq/VYdcOfLOtcJAE6Jnm3alVyeXb1PbMB
WBxaM998Z6R8VmeMB7gQ
=WMzO
-----END PGP SIGNATURE-----

Note that the message itself is plainly readable; this message isn’t encrypted, it’s just verified as having been written by a particular person, and not altered since it was written.

Now anyone who has our public key on their keyring (as we ourselves do) can verify that it was actually us who wrote this message:

$ gpg --verify message.txt.asc
gpg: Signature made Sat 23 Mar 2013 14:32:17 NZDT using RSA key ID 040FE79B
gpg: Good signature from "Tom Ryder (Test Key Only) <tom@sanctum.geek.nz>"

If anybody tampers with the message, even something like removing a period from the end of a sentence, the verification will fail, suggesting the message was tampered with:

$ gpg --verify message.txt.asc
gpg: Signature made Sat 23 Mar 2013 14:32:17 NZDT using RSA key ID 040FE79B
gpg: BAD signature from "Tom Ryder (Test Key Only) <tom@sanctum.geek.nz>"

Signing and verifying binary files

For all other files, we likely need to make the signature file separate with a detached signature:

$ gpg --armor --detach-sign archive.tar.gz

This produces a file archive.tar.gz.asc in the same directory, containing the signature. We use --armor to make the signature in ASCII, which makes for a longer file but easier distribution online.

In this case, both the file and the signature are required for verification; put the signature file first when you check this:

$ gpg --verify archive.tar.gz.asc archive.tar.gz

You can use this method to verify software downloads from trusted sources, such as the Apache HTTPD development team. First, we would download and import all their public keys at the URL they nominate:

$ wget http://www.apache.org/dist/httpd/KEYS
$ gpg --import KEYS

We could then download an Apache HTTPD release, along with its key, from an arbitrary mirror:

$ wget http://www.example.com/apache/httpd/httpd-2.4.4.tar.gz
$ wget https://www.apache.org/dist/httpd/httpd-2.4.4.tar.gz.asc

We can then use the key and signature to verify that it’s an uncompromised copy of the original file signed by the developers:

$ gpg --verify httpd-2.4.4.tar.gz.asc httpd-2.4.4.tar.gz
gpg: Signature made Tue 19 Feb 2013 09:28:39 NZDT using RSA key ID 791485A8
gpg: Good signature from "Jim Jagielski (Release Signing Key) <jim@apache.org>"
gpg:                 aka "Jim Jagielski <jim@jaguNET.com>"
gpg:                 aka "Jim Jagielski <jim@jimjag.com>"
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: A93D 62EC C3C8 EA12 DB22  0EC9 34EA 76E6 7914 85A8

Note that the gpg output cautions that this is still not perfect assurance that the release actually came from Jim Jagielski, because we’ve never met him and can’t absolutely, definitely say that this is his public key. Looking him up on public keyservers, we can however see a lot of other Apache developers have signed his key, which looks promising, but do we know who they are?

Despite the lack of absolute certainty, when downloading from mirrors this is a lot better (and harder to exploit) than simply downloading without validating or checksumming at all, given that the signature and the KEYS file were downloaded from Apache’s own site.

You will need to decide for yourself how sure you need to be whether a person’s public key really corresponds to them. This might extend to the point of arranging to meet them with government-issued identification!

Encrypting a file

We can encrypt a file so that only nominated people can decrypt and read it. In this case, we encrypt it not with our own private key, but with the recipient’s public key. This means that they will be able to decrypt it using their own private key.

Here’s the contents of secret-message.txt:

This is a secret message from Tom Ryder.

Now we need at least one recipient. Let’s say this message was intended for my friend John Public. He’s given me his public key in a file called john-public.asc on a USB drive in person; he even brought along his birth certificate and driver’s license (which is weird, because I’ve known him since I was four).

To start with, I’ll import his key into my keychain:

$ gpg --import john-public.asc
gpg: key 695195A5: public key "John Public (Main key) <johnpublic@example.com>" imported
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)

Now I can encrypt the message for only John to read. I like to use the 8-digit hex code for the key for --recipient, to make sure I’ve got the right person. You can see it in the output above, or in the output of gpg --list-keys.

$ gpg --armor --recipient 695195A5 --encrypt secret-message.txt

The encrypted message is written to secret-message.txt.asc:

-----BEGIN PGP MESSAGE-----
Version: GnuPG v1.4.10 (GNU/Linux)

hQEMAxiBb8eWSupuAQgAgOUQvqbTh60N6RQhDtP/bY9l+gjm4Grx5XcuhgQqK6pn
YtyPTKcpHdPK679lhbv0vE0RYe7pL+nBOngU1hCQYuGbRDZDxIXTIZW/rBvXbtHA
jgeSxrquad2totfh2nc7upePVCqXncPrLraJyDJBLLMrBHVvmOZymDabJbemOFuq
A/NbcmT3+osptvaEPFdlbgAW+J3vGxXMuQQYkT8GSnuutfEhZRb7SEL1ktaXwaMc
AA6NAan5ak7nCyDDHhDSDFMS9SQQHd8TDvQPF6OzRXlq26EOFD8HvlbDcgc51lbS
+N5nWaHM/CiuPh9dIOEV0H4Y8WDBdgkxp6kXKQfqb9JzAdwQ047r82SJAA7MSqCS
HRVtCRf5SNM12HqTRzF9XXum4uG+HXT6Bpy+K/lYpLgmHcHoUVKh8c2OcGaCHWQh
UC9B+aaThKdkxUfD/9tVIRmugjutgj7KdtDTGm+qLeCoJqp6HK5z5SX8Ha+P6/P5
hxinyw==
=kqUG
-----END PGP MESSAGE-----

Note that even I can’t read it, because I didn’t list myself as a recipient, and I don’t have access to John’s private key:

tom@tombox:~$ gpg --decrypt secret-message.txt.asc
gpg: encrypted with 2048-bit RSA key, ID 964AEA6E, created 2013-03-10
    "John Public (Main key) <johnpublic@example.com>"
gpg: decryption failed: secret key not available

However, on John’s computer, using his private key, he can decrypt and read it:

john@johnbox:~$ gpg --decrypt secret-message.txt.asc
gpg: encrypted with 2048-bit RSA key, ID 964AEA6E, created 2013-03-10
    "John Public (Main key) <johnpublic@example.com>"
This is a private, secret message from Tom Ryder.

If I wanted to make sure I could read the message too, I’d add my own public key to identify myself as a recipient when I encrypt it. Then either of us will be able to read it with our private keys (independently of the other):

$ gpg --recipient 695195A5 --recipient 040FE79B \
    --armor --encrypt secret-message.txt

Just to be thorough, we can sign the message as well to prove it came from us:

$ gpg --recipient 695195A5 --recipient 040FE79B \
    --armor --sign --encrypt secret-message.txt

Then when John runs the --decrypt, gpg will automatically verify the signature for us too, provided he has my public key in his keyring:

$ gpg --decrypt secret-message.txt.asc
gpg: encrypted with 2048-bit RSA key, ID 964AEA6E, created 2013-03-10
    "John Public (Main key) <johnpublic@example.com>"
gpg: encrypted with 4096-bit RSA key, ID AA159E5B, created 2013-03-23
    "Tom Ryder (Test Key Only) <tom@sanctum.geek.nz>"
This is a private, secret message from Tom Ryder.
gpg: Signature made Sat 23 Mar 2013 17:23:20 NZDT using RSA key ID 040FE79B
gpg: Good signature from "Tom Ryder (Test Key Only) <tom@sanctum.geek.nz>"

These are all the basic functions of GnuPG that will be useful to most people. We haven’t considered here submitting keys to public servers, or participating in the web of trust; you should only look into this once you’re happy with how your key setup is working, and are ready to publish your key for public use.