GNU/Linux Crypto: Backups

This entry is part 8 of 10 in the series GNU/Linux Crypto.

While having local backups for quick restores is important, such as on a USB disk or spare hard drive, it’s equally important to have a backup offsite from which you can restore your important documents if, for example, your office was burgled or burned down, losing both your workstation and backup media.

The easiest way to do this for most people is with a storage provider, offering convenient access to bulk storage of suitable size maintained on another company’s systems for a relatively modest price or even for free, such as the Ubuntu One service, or Microsoft’s offering, Skydrive. The best storage providers will also encrypt the data on their own servers, whether or not they have access.

Trusting a company with all your data and the encryption thereof is risky, particularly given recent revelations of corporate collusion with the NSA, and privacy-conscious users should prefer the security of encrypting the backups before they go up onto the provider’s servers. The provider may implement closed and/or symmetric encryption mechanisms of their own, which may or may not be trustworthy. For very strong personal encryption, as established, we can use our GnuPG setup to encrypt files before we put them up there:

$ tar -cf docsbackup-"$(date +%Y-%m-%d)".tar $HOME/Documents
$ gpg --encrypt docsbackup-2013-07-27.tar
$ scp docsbackup-2013-07-27.tar.gpg user@backup.example.com:docsbackup

The problem with encrypting whole files before we put them up for storage is that for even modestly sized data, performing entire backups and uploading all of the files together every time can cost a lot of bandwidth. Similarly, we’d like to be able to restore our personal files as they were on a specific date, in case of bad backups or accidental deletion, but without storing every file on every backup day, which may end up requiring far too much space.

Incremental backups

Normally, the solution is to use an incremental backup system, meaning after first uploading your files in their entirety to the backup system, successive backups upload only the changes, storing them in a retrievable and space-efficient format. Systems like Dirvish, a free Perl frontend to rsync(1), allow this.

Unfortunately, Dirvish doesn’t encrypt the files or changesets it stores. What’s needed is an incremental backup solution that efficiently calculates and stores changes in files on a remote server, and also encrypts them. Duplicity, a Python tool built around librsync, excels at this, and can use our GnuPG asymmetric key setup for the file encryption. It’s available in Debian-derived systems in the duplicity package. Note that, as before, a GnuPG key setup with an agent is required for this to work.

Usage

We can get an idea of how duplicity(1) works by asking it to start a backup vault on our local machine. It uses much the same source destination argument as tools like rsync or scp:

$ cd
$ duplicity --encrypt-key tom@sanctum.geek.nz Documents file://docsbackup

It’s important to specify --encrypt-key, because otherwise duplicity(1) will use symmetric encryption with a passphrase rather than a public key, which is considerably less secure. Specify the email address corresponding to the public keypair you would like to use for the encryption.

This performs a full encrypted backup of the directory, returning the following output:

Local and Remote metadata are synchronized, no sync needed.
Last full backup date: none
No signatures found, switching to full backup.
--------------[ Backup Statistics ]--------------
StartTime 1374903081.74 (Sat Jul 27 17:31:21 2013)
EndTime 1374903081.75 (Sat Jul 27 17:31:21 2013)
ElapsedTime 0.01 (0.01 seconds)
SourceFiles 4
SourceFileSize 142251 (139 KB)
NewFiles 4
NewFileSize 142251 (139 KB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 4
RawDeltaSize 138155 (135 KB)
TotalDestinationSizeChange 138461 (135 KB)
Errors 0
-------------------------------------------------

You’ll note you were not prompted for your passphrase to do this. Remember, encrypting files with your public key does not require a passphrase; the whole idea is that anyone can encrypt using your key without needing your permission.

Checking the created directory docsbackup, we find three new files within it, all three of them encrypted:

$ ls -1 docsbackup
duplicity-full.20130727T053121Z.manifest.gpg
duplicity-full.20130727T053121Z.vol1.difftar.gpg
duplicity-full-signatures.20130727T053121Z.sigtar.gpg

The vol1.difftar.gpg file contains the actual data stored; the other two files contain metadata about the backup’s contents, for use to calculate differences the next time the backup runs.

If we make a small change to a file in the directory being backed up, and run the same command again, we note that the backup has been performed incrementally, and only the changes (the new file) have been saved:

$ duplicity --encrypt-key tom@sanctum.geek.nz Documents file://docsbackup
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Sat Jul 27 17:34:33 2013
--------------[ Backup Statistics ]--------------
StartTime 1374903396.52 (Sat Jul 27 17:36:36 2013)
EndTime 1374903396.52 (Sat Jul 27 17:36:36 2013)
ElapsedTime 0.01 (0.01 seconds)
SourceFiles 5
SourceFileSize 142255 (139 KB)
NewFiles 2
NewFileSize 4100 (4.00 KB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 2
RawDeltaSize 4 (4 bytes)
TotalDestinationSizeChange 753 (753 bytes)
Errors 0
-------------------------------------------------

We also find three new files in the docsbackup directory containing the new data:

$ ls -1 docsbackup
duplicity-full.20130727T053433Z.manifest.gpg
duplicity-full.20130727T053433Z.vol1.difftar.gpg
duplicity-full-signatures.20130727T053433Z.sigtar.gpg
duplicity-inc.20130727T053433Z.to.20130727T053636Z.manifest.gpg
duplicity-inc.20130727T053433Z.to.20130727T053636Z.vol1.difftar.gpg
duplicity-new-signatures.20130727T053433Z.to.20130727T053636Z.sigtar.gpg

Note that the new files have the prefix duplicity-inc- or duplicity-new-, denoting them as incremental backups and not full ones.

Note that in order to keep track of what files have already been backed up, duplicity(1) stores metadata in ~/.cache/duplicity, as well as storing them along with the backup. This allows us to let our backup processes run unattended, rather than having to put in our passphrase to read the metadata on the remote server before performing an incremental backup. Of course, if we lose our cached files, that’s OK; we can read the ones out of the backup vault by supplying our passphrase on request for decryption.

Remote backups

If you have SSH or even just SCP/SFTP access to your storage provider’s servers, not much has to change to make duplicity(1) store the files up there instead:

$ duplicity --encrypt-key tom@sanctum.geek.nz Documents sftp://user@backup.example.com:docsbackup

Your backups will then be sent over an SSH link to the directory docsbackup on the system backup.example.com, with username user. In this way, not only is all the data protected in transmission, it’s stored encrypted on the remote server; it never sees your plaintext data. All anyone with access to your backups can see is their approximate size, the dates they were made, and (if you publish your public key) the user ID on the GnuPG key used to encrypt them.

If you’re using the ssh-agent(1) program to store your decrypted private keys, you won’t even have to enter a passphrase for that.

The duplicity(1) frontend supports other methods of uploading to different servers, too, including the boto backend for S3 Amazon Web Services, the gdocs backend for Google Docs, and httplib2 or oauthlib for Ubuntu One.

If you like, you can also sign your backups to make sure they haven’t been tampered with at the time of restoration, by changing --encrypt-key to --encrypt-sign-key. Note that this will require your passphrase.

Restoring

Restoring from a duplicity(1) backup volume is much the same, but with the arguments reversed:

$ duplicity sftp://user@backup.example.com:docsbackup docsrestore
Synchronizing remote metadata to local cache...
GnuPG passphrase:
Copying duplicity-full-signatures.20130727T053433Z.sigtar.gpg to local cache.
Copying duplicity-full.20130727T053433Z.manifest.gpg to local cache.
Copying duplicity-inc.20130727T053433Z.to.20130727T053636Z.manifest.gpg to local cache.
Copying duplicity-new-signatures.20130727T053433Z.to.20130727T053636Z.sigtar.gpg to local cache.
Last full backup date: Sat Jul 27 17:34:33 2013

Note that this time you are asked for your passphrase. This is because restoring the backup requires decrypting the data and possibly the signatures in the backup vault. After doing this, the complete set of documents from the time of your most recent incremental backup will be available in docsrestore.

Using this incremental system also allows you to restore your data in the state in the last backup before a given time. For example, to retrieve my ~/Documents directory as it was three days ago, I might run this:

$ duplicity --time 3D \
    sftp://user@backup.example.com:docsbackup \
    docsrestore

You can extend this to only restore particular files for large vaults, if you only need a particular file from the vault:

$ duplicity --time 3D \
    --file-to-restore private/eff.txt \
    sftp://user@backup.example.com:docsbackup \
    docsrestore

Automating

You should run your first full backup interactively to make sure it’s doing exactly what you need, but once you’re confident that everything is working correctly, you can set up a simple Bash script to run incremental backups for you. Here’s an example script, saved in $HOME/.local/bin/backup-remote:

#!/usr/bin/env bash

# Run keychain to recognise any agents holding decrypted keys we might need
# (optional, depending on your SSH key setup)
eval "$(keychain --eval --quiet)"

# Specify directory to back up, GnuPG key ID, and remote username and
# hostname
keyid=tom@sanctum.geek.nz
local=/home/tom/Documents
remote=sftp://user@backup.example.com/docbackups

# Run backup with duplicity
/usr/bin/duplicity --encrypt-key "$keyid" -- "$local" "$remote"

The line with keychain is optional, but will be necessary if you’re using an SSH key with a passphrase on it; you’ll also need to have authenticated with ssh-agent at least once. See the earlier article on SSH/GPG agents for details on this setup.

Don’t forget to make the script executable:

$ chmod +x ~/.local/bin/backup-remote

You can then have cron(8) call this for you every week, running it as your user, by editing your user crontab(5) file:

$ crontab -e

The following line would run this script every morning, beginning at 6.00am:

0 6 * * *   ~/.local/bin/backup-remote

Tips

A few general best practices apply to this, consistent with the Tao of Backup:

  • Check that your backups completed; either have the output of the cron script mailed to you, or log it to a file that you check at least occasionally to make sure your backups are working. I highly recommend using an email message, and including error output:

    MAILTO=you@example.com
    0 6 * * *   ~/.local/bin/backup-remote 2>&1
    
  • Run backups to your local servers too; this might prevent your backup provider from reading your files, but it won’t save them from being accidentally deleted.

  • Don’t forget to occasionally test-restore your backups to make sure they’re working correctly. It’s also wise to use duplicity verify on them occasionally, particularly if you don’t back up every day:

    $ duplicity verify sftp://user@remote.example.com/docbackups Documents
    Local and Remote metadata are synchronized, no sync needed.
    Last full backup date: Sat Jul 27 17:34:33 2013
    GnuPG passphrase:
    Verify complete: 2195 files compared, 0 differences found.
    
  • This incremental system means that you’ll likely only have to make full backups once, so you should back up too much data rather than too little; if you can spare the bandwidth and have the space, backing up your entire computer isn’t really that extreme.

  • Try not to depend too much on your remote backups; see them as a last resort, and work securely and with local backups as much as you can. Certainly, never rely on backups as a version control system; use Git for that.

GNU/Linux Crypto: Email

This entry is part 7 of 10 in the series GNU/Linux Crypto.

An encrypted password storage is well and good, but now that we have a working GnuPG setup, we should consider using PGP for what it was originally designed: email messages. To do this, we’ll be using Mutt.

Mutt is a console-based mail user agent, or MUA, designed chiefly for managing and reading mail. Unlike mailer programs like Thunderbird, it was not designed to be a POP3/IMAP client, or an SMTP agent, although versions in recent years do include this functionality; these are tasks done by programs like Getmail and MSMTP.

If like many people, you’re using Gmail, this works very well with POP3/IMAP and SMTP, enabling you to compose email messages in plain text with your choice editor in a terminal window, in a highly configurable environment, and doing your own email encryption for any sensitive communications in such a way that even your email provider can’t read it.

General usage of Mutt and setup for Gmail users is not covered in detail here, although it may be the subject of a later article. For now, there are many excellent articles on the basics of a Mutt setup. If you’re interested in the setup for other GNU/Linux mail clients like Claws or Thunderbird, Cory Sadowski has a very good article walking you through that, among other privacy settings relevant to both GNU/Linux and Windows.

All of the below is assuming you already have a GnuPG keypair ready, with gpg-agent(1) running in the background to manage your keys.

Background

Most of the PGP setup guides for Mutt you can find online are quite old, and they usually suggest a lot of lines of .muttrc configuration to interface directly with the gpg command, with a myriad of options and some byzantine variable substitution:

set pgp_clearsign_command="gpg --no-verbose --batch --output - ...
set pgp_decode_command="gpg %?p?--passphrase-fd 0? --no-verbose ...
set pgp_decrypt_command="gpg --passphrase-fd 0 --no-verbose --batch ...
set pgp_encrypt_only_command="pgpewrap gpg --batch --quiet ...
set pgp_encrypt_sign_command="pgpewrap gpg --passphrase-fd 0 ...
set pgp_export_command="gpg --no-verbose --export --armor %r"
set pgp_import_command="gpg --no-verbose --import -v %f"
set pgp_list_pubring_command="gpg --no-verbose --batch --with-colons ...
set pgp_list_secring_command="gpg --no-verbose --batch --with-colons ...
set pgp_sign_command="gpg --no-verbose --batch --output - ...
set pgp_verify_command="gpg --no-verbose --batch --output - --verify %s %f"
set pgp_verify_key_command="gpg --no-verbose --batch --fingerprint ...

I’m all for the Unix philosophy of using programs together, but this is just too much. It’s a fickle setup that’s very hard to work with, and it requires too much understanding of the gpg(1) frontend to use and edit sensibly. After all, we want to end up with a setup that we understand reasonably well.

So, throw all that away; we’re going to use GPGME instead. The above is exactly the problem that this library is designed to solve; it’s a library to which applications can link to streamline the usage of GnuPG functions, including interfacing with agents. We can replace all of the above with this:

set crypt_use_gpgme = yes

Installation

If you have Mutt installed, odds are it already has a GPGME interface. You can check if your current version of Mutt has GPGME powers by looking at the mutt -v version output. Here’s the output of mine, using the packaged Mutt from Debian GNU/Linux, which does have GPGME support:

$ mutt -v | grep -i gpgme
+CRYPT_BACKEND_CLASSIC_PGP  +CRYPT_BACKEND_CLASSIC_SMIME  +CRYPT_BACKEND_GPGME
upstream/548577-gpgme-1.2.patch

If you don’t have a version of Mutt with GPGME, you can build your own by downloading the source and building it with --enable-gpgme:

$ ./configure --enable-gpgme
$ make
# make install

You may need to make sure you have the GPGME library and headers installed first:

# apt-get install libgpgme11 libgpgme11-dev

Setup

Add the following lines to your .muttrc file; remove anything else beginning with crypt_* or pgp_*:

# Use GPGME
set crypt_use_gpgme = yes

# Sign replies to signed email
set crypt_replysign = yes

# Encrypt replies to encrypted email
set crypt_replyencrypt = yes

# Encrypt and sign replies to encrypted and signed email
set crypt_replysignencrypted = yes

# Attempt to verify email signatures automatically
set crypt_verify_sig = yes

Restart Mutt, and you should be ready to go.

Usage

First of all, check that you have the public key for your intended recipient available in your GnuPG keychain:

$ gpg --list-keys joe@example.com

If you’re able to download it from somewhere, a useful formula is to download it with curl(1) and import it directly into gpg(1):

$ curl http://www.example.com/joe-somebody.asc | gpg --import
gpg: key 1234ABCD1234ABCD: public key "Joe Somebody <joe@example.com>" imported
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)
gpg: no ultimately trusted keys found

Remember, it’s your responsibility to decide how much you trust this public key; normally it’s best practice that you actually meet someone in person to exchange written key fingerprints in order to be completely sure that the key corresponds to that user.

If you don’t have anyone else you know using PGP to communicate with, you can send me a message encrypted with my public key 0xC14286EA77BB8872 to tom@sanctum.geek.nz. If you send or link me your public key in your message, then I’ll reply to you with a message encrypted with your public key so you can check everything’s working.

$ curl https://sanctum.geek.nz/keys/pgp/tom@sanctum.geek.nz.pub.asc | gpg --import

EDIT 2021: Yes, this offer is still good; you can still email me if you want to test your new setup; I will still reply to you!

Back in Mutt, begin composing a message with m. Enter the recipient and subject as normal, and compose your message. When you’ve finished writing and save and quit $EDITOR, and your message is in the Compose screen waiting to send, press p to bring up the PGP menu at the bottom:

PGP (e)ncrypt, (s)ign, sign (a)s, (b)oth, s/(m)ime or (c)lear?

We’ll press b to both sign and encrypt the message.

If you want to be able to read the message after sending, then you’ll need to arrange to encrypt it with your key as well as the recipient’s. I find the cleanest way to do this is to add your address to the Bcc: header with b. You can also set this as a default with the following line in ~/.gnupg/gpg.conf, where 0x1234ABCD1234ABCD is the short ID of your own key:

encrypt-to 0x1234ABCD1234ABCD

Mutt compose screen

When you send the message with y, you might need to specify which key you want to use for each recipient, if you don’t have a unique key on your keychain with your recipient’s email address.

When you send, you should be prompted for your passphrase by your PIN entry program, unless your agent is already holding the key for you. This is needed in order to sign the message. When you’ve provided this, the message will be sent, and if you included yourself in the Bcc: field, you should be able to read it in your sent mail, with some headers showing the PGP information (whether the message was signed, encrypted, or both):

Mutt sent message

Your recipient will be able to decrypt the message in their mail user agent with their private key, and nobody else but the two of you will be able to read it. Note that this works for any number of recipients, as long as you have a public key for each of them.

Keep in mind that the metadata of the message, such as the sender and recipient name and address, date and time it was sent, and (importantly) the subject, are sent in plain text. Only the body of the message (including attachments) is encrypted.

Useful extras

With GPGME, Mutt tries to use the first secret key available to it in its private keychain. If you want to use some other specific keypair for signing messages, you can specify that with with the pgp_sign_as option in .muttrc:

set pgp_sign_as = 0x9876FEDC9876FEDC

If you’d like to automatically sign all of your outgoing mail, you can set the crypt_autosign option:

set crypt_autosign = yes

The first batch of options we set earlier will already automatically sign and/or encrypt messages in responses to messages doing either/both.

If you’d like to include a link to your PGP key in the headers to each message, you can add a custom header with my_hdr:

my_hdr X-PGP-Key: https://sanctum.geek.nz/keys/pgp/tom@sanctum.geek.nz.pub.asc

All of this combines with Mutt’s extensive speed and high-powered configuration to make Mutt a very capable and convenient PGP mail client. As always, the more people you know using PGP, and the more public keys you have, the more useful this will be.

GNU/Linux Crypto: GnuPG Usage

This entry is part 3 of 10 in the series GNU/Linux Crypto.

With our private and public key generated and stored, we can start using a few of GnuPG’s features to sign, verify, encrypt, and decrypt files and messages for distribution over untrusted channels like the internet.

Signing a message or text file

We’ll start by signing a simple text file, using the --clearsign option. This includes the signature in the message, which we can then distribute to people to read. Here’s the contents of message.txt:

This is a public message from Tom Ryder.

We’ll sign that with our new private key like so:

$ gpg --clearsign message.txt

We’re prompted for our passphrase for the private key:

You need a passphrase to unlock the secret key for
user: "Tom Ryder (Test Key Only) <tom@sanctum.geek.nz>"
4096-bit RSA key, ID 040FE79B, created 2013-03-23

Having provided that, the file message.txt.asc is created, with PGP sections and a plaintext ASCII signature:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

This is a public message from Tom Ryder.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)

iQIcBAEBCAAGBQJRTQcTAAoJEGIHQ4oED+ebtjoP/19PlndkGhR46BA6YZmDVdC1
Snk9aXe4Eo42kRpW13bjo8xg+pb+U26ylHkH52OBB7fJ3/BR5eZMug/RXJLGIf+U
aiylbGVz4dGjkeTCSxtg1TvcyEtzhm1ETfIWLarboj9PPBJfO1QIh4uPDkD7kb6+
+OnpdxPURbiJ03osu2Mj2fFq5wYT0La+I9BdKUAJmS8zt+CaTirKw+xF6l0sEAVv
lqslWjEwF9JCfumKAj8aMBeZMndoKRqW18ZHoYJdP0x3g1SUKAjZ/NRRUGug6Eg+
JTJ82ETCRKGKmYFLkHJ6iCaucrmhLTd9IYyEQZE/weUuClKqtshollhNHF1dD3SF
fWMPq0+29DInjlXwIkXyzVDln1wULNzbd5zv5Wg5b6lSCZlwH0xCrNjiYO7f413c
Ty4q6SqtRFUommnpMA5XcmX7ebbUpMfqfqqzoLqeTpA15Yuhh3DDR6NoMN82oLyF
FFt7UZh/JlYMc8GOnEqyZfT7d57FbKSLn3vpZbH9QXNFWG6/oZabFFyRm8r7k8F2
FzYTdyp59O0dW4T5OJ6a/xo/OnZutUN1RqW6ZJS19Xb4/5eEohFAFL9cDVLuC6zo
HmlA2m5zwv8aYbJad6Rk6+vpQAAdxHgNq/VYdcOfLOtcJAE6Jnm3alVyeXb1PbMB
WBxaM998Z6R8VmeMB7gQ
=WMzO
-----END PGP SIGNATURE-----

Note that the message itself is plainly readable; this message isn’t encrypted, it’s just verified as having been written by a particular person, and not altered since it was written.

Now anyone who has our public key on their keyring (as we ourselves do) can verify that it was actually us who wrote this message:

$ gpg --verify message.txt.asc
gpg: Signature made Sat 23 Mar 2013 14:32:17 NZDT using RSA key ID 040FE79B
gpg: Good signature from "Tom Ryder (Test Key Only) <tom@sanctum.geek.nz>"

If anybody tampers with the message, even something like removing a period from the end of a sentence, the verification will fail, suggesting the message was tampered with:

$ gpg --verify message.txt.asc
gpg: Signature made Sat 23 Mar 2013 14:32:17 NZDT using RSA key ID 040FE79B
gpg: BAD signature from "Tom Ryder (Test Key Only) <tom@sanctum.geek.nz>"

Signing and verifying binary files

For all other files, we likely need to make the signature file separate with a detached signature:

$ gpg --armor --detach-sign archive.tar.gz

This produces a file archive.tar.gz.asc in the same directory, containing the signature. We use --armor to make the signature in ASCII, which makes for a longer file but easier distribution online.

In this case, both the file and the signature are required for verification; put the signature file first when you check this:

$ gpg --verify archive.tar.gz.asc archive.tar.gz

You can use this method to verify software downloads from trusted sources, such as the Apache HTTPD development team. First, we would download and import all their public keys at the URL they nominate:

$ wget http://www.apache.org/dist/httpd/KEYS
$ gpg --import KEYS

We could then download an Apache HTTPD release, along with its key, from an arbitrary mirror:

$ wget http://www.example.com/apache/httpd/httpd-2.4.4.tar.gz
$ wget https://www.apache.org/dist/httpd/httpd-2.4.4.tar.gz.asc

We can then use the key and signature to verify that it’s an uncompromised copy of the original file signed by the developers:

$ gpg --verify httpd-2.4.4.tar.gz.asc httpd-2.4.4.tar.gz
gpg: Signature made Tue 19 Feb 2013 09:28:39 NZDT using RSA key ID 791485A8
gpg: Good signature from "Jim Jagielski (Release Signing Key) <jim@apache.org>"
gpg:                 aka "Jim Jagielski <jim@jaguNET.com>"
gpg:                 aka "Jim Jagielski <jim@jimjag.com>"
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: A93D 62EC C3C8 EA12 DB22  0EC9 34EA 76E6 7914 85A8

Note that the gpg output cautions that this is still not perfect assurance that the release actually came from Jim Jagielski, because we’ve never met him and can’t absolutely, definitely say that this is his public key. Looking him up on public keyservers, we can however see a lot of other Apache developers have signed his key, which looks promising, but do we know who they are?

Despite the lack of absolute certainty, when downloading from mirrors this is a lot better (and harder to exploit) than simply downloading without validating or checksumming at all, given that the signature and the KEYS file were downloaded from Apache’s own site.

You will need to decide for yourself how sure you need to be whether a person’s public key really corresponds to them. This might extend to the point of arranging to meet them with government-issued identification!

Encrypting a file

We can encrypt a file so that only nominated people can decrypt and read it. In this case, we encrypt it not with our own private key, but with the recipient’s public key. This means that they will be able to decrypt it using their own private key.

Here’s the contents of secret-message.txt:

This is a secret message from Tom Ryder.

Now we need at least one recipient. Let’s say this message was intended for my friend John Public. He’s given me his public key in a file called john-public.asc on a USB drive in person; he even brought along his birth certificate and driver’s license (which is weird, because I’ve known him since I was four).

To start with, I’ll import his key into my keychain:

$ gpg --import john-public.asc
gpg: key 695195A5: public key "John Public (Main key) <johnpublic@example.com>" imported
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)

Now I can encrypt the message for only John to read. I like to use the 8-digit hex code for the key for --recipient, to make sure I’ve got the right person. You can see it in the output above, or in the output of gpg --list-keys.

$ gpg --armor --recipient 695195A5 --encrypt secret-message.txt

The encrypted message is written to secret-message.txt.asc:

-----BEGIN PGP MESSAGE-----
Version: GnuPG v1.4.10 (GNU/Linux)

hQEMAxiBb8eWSupuAQgAgOUQvqbTh60N6RQhDtP/bY9l+gjm4Grx5XcuhgQqK6pn
YtyPTKcpHdPK679lhbv0vE0RYe7pL+nBOngU1hCQYuGbRDZDxIXTIZW/rBvXbtHA
jgeSxrquad2totfh2nc7upePVCqXncPrLraJyDJBLLMrBHVvmOZymDabJbemOFuq
A/NbcmT3+osptvaEPFdlbgAW+J3vGxXMuQQYkT8GSnuutfEhZRb7SEL1ktaXwaMc
AA6NAan5ak7nCyDDHhDSDFMS9SQQHd8TDvQPF6OzRXlq26EOFD8HvlbDcgc51lbS
+N5nWaHM/CiuPh9dIOEV0H4Y8WDBdgkxp6kXKQfqb9JzAdwQ047r82SJAA7MSqCS
HRVtCRf5SNM12HqTRzF9XXum4uG+HXT6Bpy+K/lYpLgmHcHoUVKh8c2OcGaCHWQh
UC9B+aaThKdkxUfD/9tVIRmugjutgj7KdtDTGm+qLeCoJqp6HK5z5SX8Ha+P6/P5
hxinyw==
=kqUG
-----END PGP MESSAGE-----

Note that even I can’t read it, because I didn’t list myself as a recipient, and I don’t have access to John’s private key:

tom@tombox:~$ gpg --decrypt secret-message.txt.asc
gpg: encrypted with 2048-bit RSA key, ID 964AEA6E, created 2013-03-10
    "John Public (Main key) <johnpublic@example.com>"
gpg: decryption failed: secret key not available

However, on John’s computer, using his private key, he can decrypt and read it:

john@johnbox:~$ gpg --decrypt secret-message.txt.asc
gpg: encrypted with 2048-bit RSA key, ID 964AEA6E, created 2013-03-10
    "John Public (Main key) <johnpublic@example.com>"
This is a private, secret message from Tom Ryder.

If I wanted to make sure I could read the message too, I’d add my own public key to identify myself as a recipient when I encrypt it. Then either of us will be able to read it with our private keys (independently of the other):

$ gpg --recipient 695195A5 --recipient 040FE79B \
    --armor --encrypt secret-message.txt

Just to be thorough, we can sign the message as well to prove it came from us:

$ gpg --recipient 695195A5 --recipient 040FE79B \
    --armor --sign --encrypt secret-message.txt

Then when John runs the --decrypt, gpg will automatically verify the signature for us too, provided he has my public key in his keyring:

$ gpg --decrypt secret-message.txt.asc
gpg: encrypted with 2048-bit RSA key, ID 964AEA6E, created 2013-03-10
    "John Public (Main key) <johnpublic@example.com>"
gpg: encrypted with 4096-bit RSA key, ID AA159E5B, created 2013-03-23
    "Tom Ryder (Test Key Only) <tom@sanctum.geek.nz>"
This is a private, secret message from Tom Ryder.
gpg: Signature made Sat 23 Mar 2013 17:23:20 NZDT using RSA key ID 040FE79B
gpg: Good signature from "Tom Ryder (Test Key Only) <tom@sanctum.geek.nz>"

These are all the basic functions of GnuPG that will be useful to most people. We haven’t considered here submitting keys to public servers, or participating in the web of trust; you should only look into this once you’re happy with how your key setup is working, and are ready to publish your key for public use.

GNU/Linux Crypto: GnuPG Keys

This entry is part 2 of 10 in the series GNU/Linux Crypto.

Many tools that use cryptography on GNU/Linux and the internet revolve around the Pretty Good Privacy software standard (OpenPGP). The GNU Privacy Guard (GnuPG or GPG) is a popular free software implementation of this standard.

You can install GnuPG with its gpg(1) frontend on Debian like so:

# apt-get install gnupg

You can do a lot of very cool things with GPG, but it boils down to four central ideas:

  • Generation of keypairs, randomly-generated and mathematically linked pairs of files, one of which is kept permanently secret (the private key) and one of which is published (the public key). This is the basis of asymmetric key cryptography.
  • Managing keys, both your own public and private key, along with other people’s public keys, so that you can verify others’ messages and files, or encrypt them so that only those people can read them. This might include publishing your public key to online keyservers, and getting people to sign it to confirm that the key is really yours.
  • Signing files and messages with your private key to enable others to verify that a file or message was authored or sighted by you, and not edited in transmission over untrusted channels like the internet. The message itself remains readable to everybody.
  • Encrypting files and messages with other people’s public keys, so that only those people can decrypt and read them with their private keys. You can also sign such messages with your own private key so that people can verify that it was sent by you.

We’ll run through the fundamentals of each of these. We won’t concern ourselves too much with the mathematics or algorithms behind these operations; the Wikipedia article for asymmetric key cryptography explains this very well for those curious for further details.

Generating a keypair

Let’s start by generating a 4096-bit RSA keypair, which should be more than sufficient for almost everyone at the time of writing. We’ll observe a few of the best practices recommended for the Debian developers.

Doing this on a private, up-to-date desktop machine is best, as it’s easier to generate entropy this way. It’s still possible on an SSH-only headless server, but you may have to resort to less cryptographically sound methods to generate proper randomness.

Create or edit the file ~/.gnupg/gpg.conf on your system, and add the following lines:

personal-digest-preferences SHA256
cert-digest-algo SHA256
default-preference-list SHA512 SHA384 SHA256 SHA224 AES256 AES192 AES CAST5 ZLIB BZIP2 ZIP Uncompressed

These lines tell GnuPG to use the cryptographically stronger SHA256 hashing algorithm for signatures in preference to the long-broken SHA1 algorithm.

With that done, we can get down to generating some keys:

$ gpg --gen-key

You will be prompted to choose the type of keypair you want. The default ought to be RSA and RSA, which means we’ll generate one master key for signing, and one subkey for encryption:

Please select what kind of key you want:
   (1) RSA and RSA (default)
   (2) DSA and Elgamal
   (3) DSA (sign only)
   (4) RSA (sign only)
Your selection? 1

For the key length, choose the maximum 4096 bit RSA:

What keysize do you want? (2048) 4096
Requested keysize is 4096 bits

The expiry date is up to you. Good practice is to set an expiry date about a year out, because as long as you have access to the private key material, you can update the expiry date indefinitely, even if it’s already expired. For this particular example, we’ll set an expiry date one year out:

Please specify how long the key should be valid.
         0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0) 1y
Key expires at Wed 21 Jan 2015 12:24:57 NZDT
Is this correct? (y/N) y

Next, we’re prompted for some basic information to name the key. In almost all circumstances you should use your real name, as without a real-world means to actually verify your identity, public keys are much less useful long-term. For the comment, you can include the key’s purpose, or your public aliases, or any other information relevant to the key:

Real name: Tom Ryder
Email address: tom@sanctum.geek.nz
Comment: Test Key Only
You selected this USER-ID:
    "Tom Ryder (Test Key Only) <tom@sanctum.geek.nz>"

Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O

Key passphrase

Next, we’re prompted for a passphrase to encrypt the key, so that if it ever falls into the wrong hands, nobody will be able to use it without knowing the passphrase.

You need a Passphrase to protect your secret key.

Choose a sequence of random words, or possibly a unique sentence you can easily memorise in any language, the longer the better. Don’t choose anything that might be feasibly guessable, like proverbs or movie quotes. You will also need to remember how you typed the passphrase exactly; I recommend using all-lowercase and no punctuation. Wikipedia has a few guidelines here.

You’ll need to type the passphrase twice to confirm it, and it won’t echo on your terminal, much as if you were typing a password.

Entropy generation

Finally, the system will prompt us to generate some entropy:

We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.

Not enough random bytes available.  Please do some other work to give
the OS a chance to collect more entropy! (Need 283 more bytes)

This step is necessary for the computer to generate sufficient random information to ensure that the private key being generated could not feasibly be reproduced. Moving the mouse around and using the keyboard on a desktop system is ideal, but generating any kind of hardware activity (including spinning disks up) should do the trick. Running expensive find(1) operations over a filesystem (with contents that couldn’t be reasonably predicted or guessed) helps too.

This step benefits from patience. You might find discussion online about forcing the use of the non-blocking PRNG random device /dev/urandom instead, using a tool like rngd(1). This definitely speeds up the process, but if you’re going to be using your key for anything serious, I recommend actually interacting with the computer and using hardware noise to seed the randomness adequately, if you can.

When adequate entropy is read and the key generation is done, you’ll be presented with some details for your master signing key pair and its encrypting subkey pair, and the private and public keys for each are automatically added to your keyring for use:

gpg: /home/tom/.gnupg/trustdb.gpg: trustdb created
gpg: key 040FE79B marked as ultimately trusted
public and secret key created and signed.
gpg: checking the trustdb
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
pub   4096R/040FE79B 2013-03-23
    Key fingerprint = 7A28 5ADA 7680 6813 48DF  401B 6207 438A 040F E79B
uid                  Tom Ryder (Test Key Only) <tom@sanctum.geek.nz>
sub   4096R/AA159E5B 2013-03-23

Managing keys

With this done, we have our own keys added to the private and public keychain:

$ gpg --list-secret-keys
/home/tom/.gnupg/secring.gpg
----------------------------
sec   4096R/040FE79B 2013-03-23
uid                  Tom Ryder (Test Key Only) <tom@sanctum.geek.nz>
ssb   4096R/AA159E5B 2013-03-23

$ gpg --list-public-keys
/home/tom/.gnupg/pubring.gpg
----------------------------
pub   4096R/040FE79B 2013-03-23
uid                  Tom Ryder (Test Key Only) <tom@sanctum.geek.nz>
sub   4096R/AA159E5B 2013-03-23

The directory ~/.gnupg contains the managed keys. It’s very, very important to keep this directory private and to back it up securely, preferably to removable media that you keep in some physically secure place. Don’t lose it!

In most contexts in GnuPG, you can refer to a key by the name of its owner, or by its eight-digit hex ID. I prefer the latter method. Here, the short ID of my main key is 040FE79B. While you shouldn’t use this for any actual verification, it’s sufficiently unique that you can use it to identify a specific key on your keyring with which you want to work.

For example, if we want to provide someone with a copy of our public key, a friendly way to do so is to export it in ASCII format with --armor, providing the appropriate key’s short ID:

$ gpg --armor --export 040FE79B > tom-ryder.public.asc

While you can export private keys the same way with --export-secret-key, you should never, ever provide anyone with your private key, so this shouldn’t be necessary.

Revocation certificate

After generating your keys, you should generate a revocation certificate:

$ gpg --output revoke.asc --gen-revoke 040FE79B

sec  4096R/040FE79B 2013-03-23 Tom Ryder (Test Key Only) <tom@sanctum.geek.nz>

Create a revocation certificate for this key? (y/N) y
Please select the reason for the revocation:
0 = No reason specified
1 = Key has been compromised
2 = Key is superseded
3 = Key is no longer used
Q = Cancel
(Probably you want to select 1 here)
Your decision? 1
Enter an optional description; end it with an empty line:
>
Reason for revocation: Key has been compromised
(No description given)
Is this okay? (y/N) y

You need a passphrase to unlock the secret key for
user: "Tom Ryder (Test Key Only) <tom@sanctum.geek.nz>"
4096-bit RSA key, ID 040FE79B, created 2013-03-23

ASCII armored output forced.
Revocation certificate created.

Please move it to a medium which you can hide away; if Mallory gets
access to this certificate he can use it to make your key unusable.
It is smart to print this certificate and store it away, just in case
your media become unreadable.  But have some caution:  The print system of
your machine might store the data and make it available to others!

You should store the resulting revoke.asc file somewhere safe. You can use this certificate to revoke your key later on if the private key is ever compromised, so that people know the key should no longer be used or trusted. You may even like to print it out and keep a hard copy, as the output of gpg suggests.

With the above setup done, we can proceed with some basic usage of GnuPG, as discussed in the next article.

Subkeys

In the output of both commands, you’ll note we actually have two private and two public keys. The sub line refers to the encryption subkey automatically generated for you. The master key is used for cryptographic signing, and the subkey for encryption; this is how GnuPG does things by default with RSA keypairs.

For extra security, it might be appropriate to physically remove the master private key from your computer, and instead use a second generated subkey for signing files as well. This is desirable because it allows you to keep the master key secure on some removable media (preferably with a backup), and not loaded on your main computer in case you get compromised.

This means you can sign and encrypt files as normal with your signing subkey and encryption subkey. If those keys ever get compromised, you can simply revoke them and generate new ones with your uncompromised master key; everyone who has signed your public master key or otherwise indicated they trust it will not have to do that all over again.

For details on how to do this, I suggest reading the Debian Wiki article on subkey management. However, it’s not necessary for performing basic GPG operations.

Thanks to commenter coldtobi for recommending setting a key expiry.

GNU/Linux Crypto: Introduction

This entry is part 1 of 10 in the series GNU/Linux Crypto.

Most of this series has been independently translated into Portuguese by Rafael Beraldo. Thanks very much, Rafael!

With the growing popularity of operating systems with Linux kernels that do not primarily use GNU components in the base system, this series was retitled to “GNU/Linux Crypto” in May 2017 for less ambiguity and to give proper credit to the GNU Project.

Cryptography for authentication and encryption is a complex and frequently changing field, and for somebody new to using it, it can be hard to know where to start. If you’re a GNU/Linux user comfortable with the terminal, but unfamiliar with the cryptographic tools available to you on open source UNIX-like operating systems, this series of posts aims at getting you set up with some basic tools that will allow you to keep your own information secure, to authenticate conveniently and safely with remote servers, and to work with signed and encrypted files online.

I’ll be working on Debian GNU/Linux, but most of these tools should adapt well to other open source UNIX-likes, including BSD. Please feel free to comment on the articles with details relevant to your own implementations, or with extra security considerations for interested readers.

As a disclaimer, I’m not myself an expert on cryptographic algorithms or key security. If you are, and you find an error or security problem with any of my explanations or suggestions, please let me know and I will correct it and credit you.

I’ll be covering the following topics:

If you already know about a specific topic, feel free to skip around through the other articles.