Linux Crypto: Passwords

It’s now becoming more widely known that using guessable passwords or using the same password for more than one account is a serious security risk, because an attacker able to control one account (such as an email account) can do a lot of damage. If an attacker gets the hash of your password from some web service, you want to be assured that the hash will be very difficult to reverse, and even if it can be reversed, that it’s unique and won’t give them access to any of your other accounts.

This growing awareness has contributed to the popularity of password managers, tools designed to securely generate, store, and retrieve passwords, encrypted with a master password or passphrase. In some cases these are locally stored, such as KeePass, and in others they are stored on a web service, such as LastPass. Both are good tools, and work well with Linux. I personally have some reservations about LastPass as I don’t want my passwords stored on a third party service, and I don’t trust JavaScript encryption.

Interestingly, because we now have a tidy GnuPG setup to handle the encryption ourselves, another option is the pass(1) tool, billing itself as “the standard UNIX password manager”. It’s little more than a shell script and some bash(1) completions wrapped around existing tools like git(1), gpg2(1), pwgen(1), tree(1), and xclip(1), and your choice of $EDITOR. If you’re not already invested in an existing password management method, you might find this a good first application of your new cryptography setup, and a great minimal approach to secure password storage accessible from the command line (and therefore SSH).

On Debian-derived systems, it’s available as part of the pass package:

# apt-get install pass

This includes a manual:

$ man pass

Instructions for installing on other operating systems are also available on the site. Releases are also available for download, and a link to the development repository. If you use this, make sure you have the required tools outlined above installed as well, although xclip(1) is only needed if you run the X Windows system.

Setup

We can get an overview of what pass(1) can do by invoking it with no arguments:

$ pass

To start, we’ll initialize our password store. For your own passwords, you will want to do this as your own user rather than root. Because pass(1) uses GnuPG for its encryption, we also need to tell it the ID of the appropriate key to use. Remember, you can find this eight-digit hex code by typing gpg --list-secret-keys. A unique string identifying your private key such as your name or email address may also work.

$ pass init 0x77BB8872
mkdir: created directory ‘/home/tom/.password-store’
Password store initialized for 0x77BB8872.

Indeed, we note the directory ~/.password-store has been created, although it’s presently empty except for the .gpg-id file recording our key ID:

$ find .password-store
.password-store
.password-store/.gpg-id

Inserting

We’ll insert an existing password of ours with pass insert, giving it a descriptive hierarchical name:

$ pass insert google.com/gmail/example@gmail.com
mkdir: created directory ‘/home/tom/.password-store/google.com’
mkdir: created directory ‘/home/tom/.password-store/google.com/gmail’
Enter password for google.com/gmail/example@gmail.com:
Retype password for google.com/gmail/example@gmail.com:

The password is read from the command line, encrypted, and placed in ~/.password-store:

$ find .password-store
.password-store
.password-store/google.com
.password-store/google.com/gmail
.password-store/google.com/gmail/example@gmail.com.gpg
.password-store/.gpg-id

Notice that pass(1) creates a directory structure for us automatically. We can get a nice view of the password store with pass with no arguments:

$ pass
Password Store
└── google.com
    └── gmail
            └── example@gmail.com

Generating

If you’d like it to generate a new secure random password for you, you can use generate instead, including a password length as the last argument:

$ pass generate google.com/gmail/example@gmail.com 16
The generated password to google.com/gmail/example@gmail.com is:
!Q%i$$&q1+JJi-|X

If you have some service that doesn’t cooperate with symbols in passwords, you can add the -n option to this call:

$ pass generate -n google.com/gmail/example@gmail.com 16
The generated password to google.com/gmail/example@gmail.com is:
pJeF18CrZEZzI59D

pass(1) uses pwgen(1) for this password generation. In each case, the password is automatically inserted into the password store for you.

If we need to change an existing password, we can either overwrite it with insert again, or use the edit operation to invoke our choice of $EDITOR:

$ pass edit google.com/gmail/example@gmail.com

If you do this, you may like to be careful that your editor is not configured to keep backups or swap files in plain text of documents it edits in temporary directories or memory filesystems. If you’re using Vim, I wrote some configuration in an attempt to solve this problem.

Note that adding or overwriting passwords does not require your passphrase; only retrieval and editing does, consistent with how GnuPG normally works.

Retrieval

This password can now be retrieved and echoed onto the command line given the appropriate passphrase:

$ pass google.com/gmail/example@gmail.com
(...gpg-agent pinentry prompt...)
Tr0ub4dor&3

If you’re using X windows and have xclip(1) installed, you can put the password on the clipboard temporarily to paste into web forms:

$ pass -c google.com/gmail/example@gmail.com
Copied google.com/gmail/example@gmail.com to clipboard. Will clear in 45 seconds.

In each case, note that if you have the bash completion installed and working, you should be able to complete the full path to the passwords with Tab, just as if you were directly browsing a directory hierarchy.

Deletion

If we no longer need the password, we can remove it with pass rm:

$ pass rm google.com/gmail/example@gmail.com
Are you sure you would like to delete google.com/gmail/example@gmail.com? [y/N] y
removed ‘/home/tom/.password-store/google.com/gmail/example@gmail.com.gpg’

We can delete whole directories of passwords with pass rm -r:

$ pass rm -r google.com
Are you sure you would like to delete google.com? [y/N] y
removed ‘/home/tom/.password-store/google.com/gmail/example@gmail.com.gpg’
removed directory: ‘/home/tom/.password-store/google.com/gmail’
removed directory: ‘/home/tom/.password-store/google.com’

Version control

To keep historical passwords, including deleted ones if we find we do need them again one day, we can set up some automatic version control on the directory with pass git init:

$ pass git init
Initialized empty Git repository in /home/tom/.password-store/.git/
[master (root-commit) 0ebb933] Added current contents of password store.
 1 file changed, 1 insertion(+)
 create mode 100644 .gpg-id

This will update the repository every time the password store is changed, meaning we can be confident we’ll be able to retrieve old passwords we’ve replaced or deleted:

$ pass insert google.com/gmail/newexample@gmail.com
mkdir: created directory ‘/home/tom/.password-store/google.com’
mkdir: created directory ‘/home/tom/.password-store/google.com/gmail’
Enter password for google.com/gmail/newexample@gmail.com:
Retype password for google.com/gmail/newexample@gmail.com:
[master 00971b6] Added given password for google.com/gmail/newexample@gmail.com to store.
 1 file changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 google.com/gmail/newexample@gmail.com.gpg

Backups

Because the password files are all encrypted only to your GnuPG key, you can relatively safely back up the store on remote and third-party sites simply by copying the ~/.password-store directory. If the filenames themselves contain sensitive information, such as private usernames or sites, you might like to back up an encrypted tarball of the store instead:

$ tar -cz .password-store \
    | gpg --sign --encrypt -r 0x77BB8872 \
    > password-store-backup.tar.gz.gpg

This directory can be restored in a similar way:

$ gpg --decrypt \
    < password-store-backup.tar.gz.gpg \
    | tar -xz 
This entry is part 6 of 10 in the series Linux Crypto.

Unix as IDE: Files

One prominent feature of an IDE is a built-in system for managing files, both the elementary functions like moving, renaming, and deleting, and ones more specific to development, like compiling and checking syntax. It may also be useful to have operations on sets of files, such as finding files of a certain extension or size, or searching files for specific patterns. In this first article, I’ll explore some useful ways to use tools that will be familiar to most Linux users for the purposes of working with sets of files in a project.

Listing files

Using ls is probably one of the first commands an administrator will learn for getting a simple list of the contents of the directory. Most administrators will also know about the -a and -l switches, to show all files including dot files and to show more detailed data about files in columns, respectively.

There are other switches to GNU ls which are less frequently used, some of which turn out to be very useful for programming:

  • -t — List files in order of last modification date, newest first. This is useful for very large directories when you want to get a quick list of the most recent files changed, maybe piped through head or sed 10q. Probably most useful combined with -l. If you want the oldest files, you can add -r to reverse the list.
  • -X — Group files by extension; handy for polyglot code, to group header files and source files separately, or to separate source files from directories or build files.
  • -v — Naturally sort version numbers in filenames.
  • -S — Sort by filesize.
  • -R — List files recursively. This one is good combined with -l and pipedthrough a pager like less.

Since the listing is text like anything else, you could, for example, pipe the output of this command into a vim process, so you could add explanations of what each file is for and save it as an inventory file or add it to a README:

$ ls -XR | vim -

This kind of stuff can even be automated by make with a little work, which I’ll cover in another article later in the series.

Finding files

Funnily enough, you can get a complete list of files including relative paths with no decoration by simply typing find with no arguments, though it’s usually a good idea to pipe it through sort:

$ find | sort
.
./Makefile
./README
./build
./client.c
./client.h
./common.h
./project.c
./server.c
./server.h
./tests
./tests/suite1.pl
./tests/suite2.pl
./tests/suite3.pl
./tests/suite4.pl

If you want an ls -l style listing, you can add -ls as the action to find results:

$ find -ls | sort -k 11
1155096    4 drwxr-xr-x   4 tom      tom          4096 Feb 10 09:37 .
1155152    4 drwxr-xr-x   2 tom      tom          4096 Feb 10 09:17 ./build
1155155    4 -rw-r--r--   1 tom      tom          2290 Jan 11 07:21 ./client.c
1155157    4 -rw-r--r--   1 tom      tom          1871 Jan 11 16:41 ./client.h
1155159   32 -rw-r--r--   1 tom      tom         30390 Jan 10 15:29 ./common.h
1155153   24 -rw-r--r--   1 tom      tom         21170 Jan 11 05:43 ./Makefile
1155154   16 -rw-r--r--   1 tom      tom         13966 Jan 14 07:39 ./project.c
1155080   28 -rw-r--r--   1 tom      tom         25840 Jan 15 22:28 ./README
1155156   32 -rw-r--r--   1 tom      tom         31124 Jan 11 02:34 ./server.c
1155158    4 -rw-r--r--   1 tom      tom          3599 Jan 16 05:27 ./server.h
1155160    4 drwxr-xr-x   2 tom      tom          4096 Feb 10 09:29 ./tests
1155161    4 -rw-r--r--   1 tom      tom           288 Jan 13 03:04 ./tests/suite1.pl
1155162    4 -rw-r--r--   1 tom      tom          1792 Jan 13 10:06 ./tests/suite2.pl
1155163    4 -rw-r--r--   1 tom      tom           112 Jan  9 23:42 ./tests/suite3.pl
1155164    4 -rw-r--r--   1 tom      tom           144 Jan 15 02:10 ./tests/suite4.pl

Note that in this case I have to specify to sort that it should sort by the 11th column of output, the filenames; this is done with the -k option.

find has a complex filtering syntax all of its own; the following examples show some of the most useful filters you can apply to retrieve lists of certain files:

  • find -name '*.c' — Find files with names matching a shell-style pattern. Use -iname for a case-insensitive search.
  • find -path '*test*' — Find files with paths matching a shell-style pattern. Use -ipath for a case-insensitive search.
  • find -mtime -5 — Find files edited within the last five days. You can use +5 instead to find files edited before five days ago.
  • find -newer server.c — Find files more recently modified than server.c.
  • find -type d — Find directories. For files, use -type f; for symbolic links, use -type l.

Note, in particular, that all of these can be combined, for example to find C source files edited in the last two days:

$ find -name '*.c' -mtime -2

By default, the action find takes for search results is simply to list them on standard output, but there are several other useful actions:

  • -ls — Provide an ls -l style listing, as above
  • -delete — Delete matching files
  • -exec — Run an arbitrary command line on each file, replacing {} with the appropriate filename, and terminated by \;; for example:

    $ find -name '*.pl' -exec perl -c {} \;
    

    You can use + as the terminating character instead if you want to put all of the results on one invocation of the command. One trick I find myself using often is using find to generate lists of files that I then edit in vertically split Vim windows:

    $ find -name '*.c' -exec vim {} +
    

Earlier versions of Unix as IDE suggested the use of xargs with find results. In most cases this should not really be necessary, and it’s more robust to handle filenames with whitespace using -exec or a while read -r loop.

Searching files

More often than attributes of a set of files, however, you want to find files based on their contents, and it’s no surprise that grep, in particular grep -R, is useful here. This searches the current directory tree recursively for anything matching ‘someVar’:

$ grep -FR someVar .

Don’t forget the case insensitivity flag either, since by default grep works with fixed case:

$ grep -iR somevar .

Also, you can print a list of files that match without printing the matches themselves with grep -l:

$ grep -lR someVar .

If you write scripts or batch jobs using the output of the above, use a while loop with read to handle spaces and other special characters in filenames:

grep -lR someVar | while IFS= read -r file; do
    head "$file"
done

If you’re using version control for your project, this often includes metadata in the .svn, .git, or .hg directories. This is dealt with easily enough by excluding (grep -v) anything matching an appropriate fixed (grep -F) string:

$ grep -R someVar . | grep -vF .svn

Some versions of grep include --exclude and --exclude-dir options, which may be tidier.

With all this said, there’s a very popular alternative to grep called ack, which excludes this sort of stuff for you by default. It also allows you to use Perl-compatible regular expressions (PCRE), which are a favourite for many hackers. It has a lot of utilities that are generally useful for working with source code, so while there’s nothing wrong with good old grep since you know it will always be there, if you can install ack I highly recommend it. There’s a Debian package called ack-grep, and being a Perl script it’s otherwise very simple to install.

Unix purists might be displeased with my even mentioning a relatively new Perl script alternative to classic grep, but I don’t believe that the Unix philosophy or using Unix as an IDE is dependent on sticking to the same classic tools when alternatives with the same spirit that solve new problems are available.

File metadata

The file tool gives you a one-line summary of what kind of file you’re looking at, based on its extension, headers and other cues. This is very handy used with find when examining a set of unfamiliar files:

$ find -exec file {} \;
.:            directory
./hanoi:      Perl script, ASCII text executable
./.hanoi.swp: Vim swap file, version 7.3
./factorial:  Perl script, ASCII text executable
./bits.c:     C source, ASCII text
./bits:       ELF 32-bit LSB executable, Intel 80386, version ...

Matching files

As a final tip for this section, I’d suggest learning a bit about pattern matching and brace expansion in Bash, which you can do in my earlier post entitled Bash shell expansion.

All of the above make the classic UNIX shell into a pretty powerful means of managing files in programming projects.

This entry is part 2 of 7 in the series Unix as IDE.