Bash history expansion

Setting the Bash option histexpand allows some convenient typing shortcuts using Bash history expansion. The option can be set with either of these:

$ set -H
$ set -o histexpand

It’s likely that this option is already set for all interactive shells, as it’s on by default. The manual, man bash, describes these features as follows:

-H  Enable ! style history substitution. This option is on
    by default when the shell is interactive.

You may have come across this before, perhaps to your annoyance, in the following error message that comes up whenever ! is used in a double-quoted string, or without being escaped with a backslash:

$ echo "Hi, this is Tom!"
bash: !": event not found

If you don’t want the feature and thereby make ! into a normal character, it can be disabled with either of these:

$ set +H
$ set +o histexpand

History expansion is actually a very old feature of shells, having been available in csh before Bash usage became common.

This article is a good followup to Better Bash history, which among other things explains how to include dates and times in history output, as these examples do.

Basic history expansion

Perhaps the best known and most useful of these expansions is using !! to refer to the previous command. This allows repeating commands quickly, perhaps to monitor the progress of a long process, such as disk space being freed while deleting a large file:

$ rm big_file &
[1] 23608
$ du -sh .
3.9G    .
$ !!
du -sh .
3.3G    .

It can also be useful to specify the full filesystem path to programs that aren’t in your $PATH:

$ hdparm
-bash: hdparm: command not found
$ /sbin/!!
/sbin/hdparm

In each case, note that the command itself is printed as expanded, and then run to print the output on the following line.

History by absolute index

However, !! is actually a specific example of a more general form of history expansion. For example, you can supply the history item number of a specific command to repeat it, after looking it up with history:

$ history | grep expand
 3951  2012-08-16 15:58:53  set -o histexpand
$ !3951
set -o histexpand

You needn’t enter the !3951 on a line by itself; it can be included as any part of the command, for example to add a prefix like sudo:

$ sudo !3850

If you include the escape string \! as part of your Bash prompt, you can include the current command number in the prompt before the command, making repeating commands by index a lot easier as long as they’re still visible on the screen.

History by relative index

It’s also possible to refer to commands relative to the current command. To subtitute the second-to-last command, we can type !-2. For example, to check whether truncating a file with sed worked correctly:

$ wc -l bigfile.txt
267 bigfile.txt
$ printf '%s\n' '11,$d' w | ed -s bigfile.txt
$ !-2
wc -l bigfile.txt
10 bigfile.txt

This works further back into history, with !-3, !-4, and so on.

Expanding for historical arguments

In each of the above cases, we’re substituting for the whole command line. There are also ways to get specific tokens, or words, from the command if we want that. To get the first argument of a particular command in the history, use the !^ token:

$ touch a.txt b.txt c.txt
$ ls !^
ls a.txt
a.txt

To get the last argument, add !$:

$ touch a.txt b.txt c.txt
$ ls !$
ls c.txt
c.txt

To get all arguments (but not the command itself), use !*:

$ touch a.txt b.txt c.txt
$ ls !*
ls a.txt b.txt c.txt
a.txt  b.txt  c.txt

This last one is particularly handy when performing several operations on a group of files; we could run du and wc over them to get their size and character count, and then perhaps decide to delete them based on the output:

$ du a.txt b.txt c.txt
4164    a.txt
5184    b.txt
8356    c.txt
$ wc !*
wc a.txt b.txt c.txt
16689    94038  4250112 a.txt
20749   117100  5294592 b.txt
33190   188557  8539136 c.txt
70628   399695 18083840 total
$ rm !*
rm a.txt b.txt c.txt

These work not just for the preceding command in history, but also absolute and relative command numbers:

$ history 3
 3989  2012-08-16 16:30:59  wc -l b.txt
 3990  2012-08-16 16:31:05  du -sh c.txt
 3991  2012-08-16 16:31:12  history 3
$ echo !3989^
echo -l
-l
$ echo !3990$
echo c.txt
c.txt
$ echo !-1*
echo c.txt
c.txt

More generally, you can use the syntax !n:w to refer to any specific argument in a history item by number. In this case, the first word, usually a command or builtin, is word 0:

$ history | grep bash
 4073  2012-08-16 20:24:53  man bash
$ !4073:0
man
What manual page do you want?
$ !4073:1
bash

You can even select ranges of words by separating their indices with a hyphen:

$ history | grep apt-get
 3663  2012-08-15 17:01:30  sudo apt-get install gnome
$ !3663:0-1 purge !3663:3
sudo apt-get purge gnome

You can include ^ and $ as start and endpoints for these ranges, too. 3* is a shorthand for 3-$, meaning “all arguments from the third to the last.”

Expanding history by string

You can also refer to a previous command in the history that starts with a specific string with the syntax !string:

$ !echo
echo c.txt
c.txt
$ !history
history 3
 4011  2012-08-16 16:38:28  rm a.txt b.txt c.txt
 4012  2012-08-16 16:42:48  echo c.txt
 4013  2012-08-16 16:42:51  history 3

If you want to match any part of the command line, not just the start, you can use !?string?:

$ !?bash?
man bash

Be careful when using these, if you use them at all. By default it will run the most recent command matching the string immediately, with no prompting, so it might be a problem if it doesn’t match the command you expect.

Checking history expansions before running

If you’re paranoid about this, Bash allows you to audit the command as expanded before you enter it, with the histverify option:

$ shopt -s histverify
$ !rm
$ rm a.txt b.txt c.txt

This option works for any history expansion, and may be a good choice for more cautious administrators. It’s a good thing to add to one’s .bashrc if so.

If you don’t need this set all the time, but you do have reservations at some point about running a history command, you can arrange to print the command without running it by adding a :p suffix:

$ !rm:p
rm important-file

In this instance, the command was expanded, but thankfully not actually run.

Substituting strings in history expansions

To get really in-depth, you can also perform substitutions on arbitrary commands from the history with !!:gs/pattern/replacement/. This is getting pretty baroque even for Bash, but it’s possible you may find it useful at some point:

$ !!:gs/txt/mp3/
rm a.mp3 b.mp3 c.mp3

If you only want to replace the first occurrence, you can omit the g:

$ !!:s/txt/mp3/
rm a.mp3 b.txt c.txt

Stripping leading directories or trailing files

If you want to chop a filename off a long argument to work with the directory, you can do this by adding an :h suffix, kind of like a dirname call in Perl:

$ du -sh /home/tom/work/doc.txt
$ cd !$:h
cd /home/tom/work

To do the opposite, like a basename call in Perl, use :t:

$ ls /home/tom/work/doc.txt
$ document=!$:t
document=doc.txt

Stripping extensions or base names

A bit more esoteric, but still possibly useful; to strip a file’s extension, use :r:

$ vi /home/tom/work/doc.txt
$ stripext=!$:r
stripext=/home/tom/work/doc

To do the opposite, to get only the extension, use :e:

$ vi /home/tom/work/doc.txt
$ extonly=!$:e
extonly=.txt

Quoting history

If you’re performing substitution not to execute a command or fragment but to use it as a string, it’s likely you’ll want to quote it. For example, if you’ve just found through experiment and trial and error an ideal ffmpeg command line to accomplish some task, you might want to save it for later use by writing it to a script:

$ ffmpeg -f alsa -ac 2 -i hw:0,0 -f x11grab -r 30 -s 1600x900 \
> -i :0.0+1600,0 -acodec pcm_s16le -vcodec libx264 -preset ultrafast \
> -crf 0 -threads 0 "$(date +%Y%m%d%H%M%S)".mkv 

To make sure all the escaping is done correctly, you can write the command into the file with the :q modifier:

$ echo '#!/usr/bin/env bash' >ffmpeg.sh
$ echo !ffmpeg:q >>ffmpeg.sh

In this case, this will prevent Bash from executing the command expansion "$(date ... )", instead writing it literally to the file as desired. If you build a lot of complex commands interactively that you later write to scripts once completed, this feature is really helpful and saves a lot of cutting and pasting.

Thanks to commenter Mihai Maruseac for pointing out a bug in the examples.

Calculating with Bash

If you’re at a terminal or writing a script and need to do some quick integer calculations, Bash offers arithmetic expansion with the $((...)) syntax.

The usual arithmetic operations of addition, subtraction, multiplication, and division are supported, but the name of arithmetic expansion is a bit misleading as it also supports operations well beyond the elementary arithmetic set, including exponentiation, equality and inequality checks, and logical and bitwise NOT/AND/OR/XOR:

  • VAR++ VAR-- — post-increment, post-decrement
  • ++VAR --VAR — pre-increment, pre-decrement
  • + - — addition, subtraction
  • ** — exponentiation
  • * / % — multiplication, division, remainder
  • == != — equality, inequality
  • < > <= >= — comparison
  • ! && || — logical NOT, AND, OR
  • ~ & | ^ — bitwise NOT, AND, OR, XOR

These operators and their precedence are all the same as used in most C-style languages. Pivotally, the expansion can include variables defined elsewhere in the command or script:

$ number=1234
$ printf '%s\n' "$((number * 2))"
2468

One place this can be very useful is in calculating filesizes, since most of the time shell commands work in either bytes or potentially arbitrary disk block sizes. If you work with SI decimal prefixes, you can use powers of 10 to calculate the sizes:

kilobytes=$((bytes / 10**3))
megabytes=$((bytes / 10**6))
gigabytes=$((bytes / 10**9))

If you prefer to work with IEC binary prefixes, i.e. multiples of 1024:

kilobytes=$((bytes / 2**10))
megabytes=$((bytes / 2**20))
gigabytes=$((bytes / 2**30))

Or you could use bit shifting:

kilobytes=$((bytes >> 10))
megabytes=$((bytes >> 20))
gigabytes=$((bytes >> 30))

Bash shell expansion

Operations in the shell very often involve repeating yourself or working with sets or ranges of information. This is where the various kinds of expansion in the shell are most useful, for generating a large number of shell tokens using a compact syntax. I’ll discuss two types here: filename expansions, which will be familiar to most shell users; and brace expansions, which seem to be in less common usage.

Filename expansions (globs)

The most commonly seen type of filename expansion in Bash is the * wildcard, which can be used to match lists of files in any directory, whether through complete listings or partial matches:

$ ls *
a.txt b.txt c.txt d.txt extra.txt README install.sh
$ ls *.txt
a.txt b.txt c.txt d.txt extra.txt

However, there are two other types of filename expansions. Firstly, you can match single characters rather than ranges of characters with a question mark:

$ ls ?.txt
a.txt b.txt c.txt d.txt

You can further restrict this by matching only a single character in a specified group of characters, or within a range:

$ ls [ac].txt
a.txt c.txt
$ ls [a-c].txt
a.txt b.txt c.txt

These can be useful in very large directories to only view files starting with a nominated letter or number, or range of letters or numbers.

Brace expansion

Brace expansion is a little different, as it generates shell tokens that don’t necessarily correspond to existing files. This allows you to expand a single general form into a lot of space-delimited specific tokens. This is probably best explained with examples. First of all, you can define the expansions you want with comma separation:

$ echo example{test,show,define,declare}
exampletest exampleshow exampledefine exampledeclare

If you’re dealing with sequences of numbers or letters, there’s also the .. separator syntax for ranges:

$ echo example{a..d}
examplea exampleb examplec exampled
$ echo example{1..5}
example1 example2 example3 example4 example5

This can be useful for creating, renaming, copying, or moving files:

$ touch file.txt.{1..6}
$ mv file.{txt,html}
$ cp file.txt{,.bak}
$ mv website.co.nz/{testing,production}/index.php

I also find it useful when dealing with long paths in Subversion branching and merging:

$ svn copy svn://server/project/{trunk,branches/experimental}
$ svn merge svn://server/project/{branches/experimental,trunk} .

You can also combine and even nest these expansions, which makes it useful for creating directory trees for new projects or chroot environments:

$ mkdir -p {test,prod}/{,usr/,usr/local/}{{,s}bin,etc,lib,share}

The above rather compact syntax creates all of the following directories:

test
test/bin
test/etc
test/sbin
test/lib
test/usr
test/usr/bin
test/usr/etc
test/usr/sbin
test/usr/lib
test/usr/share
test/usr/local
test/usr/local/bin
test/usr/local/etc
test/usr/local/sbin
test/usr/local/lib
test/usr/local/share
test/share
prod
prod/bin
prod/etc
prod/sbin
prod/lib
prod/usr
prod/usr/bin
prod/usr/etc
prod/usr/sbin
prod/usr/lib
prod/usr/share
prod/usr/local
prod/usr/local/bin
prod/usr/local/etc
prod/usr/local/sbin
prod/usr/local/lib
prod/usr/local/share
prod/share

Thanks to Reddit user silvermoot for corrections/suggestions.