Category Archives: Bash

Typo correction in Bash

Posted on by

If you make a typo in a long command line in Bash, instead of typing it all out again, you can either recall it from your history, or use caret notation to repeat the command with an appropriate substitution. This looks like the following:

# sudo apache2ctl restrat
Action 'restrat' failed.
The Apache error log may have more information.
# ^strat^start
sudo apache2ctl restart

The string after the first caret is the text to match, and the one after the second string is the text with which it should be replaced. This provides a convenient method of not only quickly correcting typos, but to change small parts of the command line in general quickly:

$ touch filea.txt
$ ^filea^fileb
touch fileb.txt
$ ^fileb^filec
touch filec.txt

For the particular case of correcting small errors in long paths for cd calls, it’s helpful to use the cdspell option for Bash, which I discuss in my earlier article on smarter directory navigation.

Bash command existence

Posted on by

If you set environment variables like your EDITOR in your .bashrc file that refer to commands that you expect to be available on the system, it’s prudent to check that an appropriate command actually exists before making the setting.

A common way of approaching this is using which, in a syntax similar to the below, which sets EDITOR to the first executable instance of vi found in PATH, or blank if not found:

EDITOR=$(which vi)

Because the behaviour of which can be unexpected, it’s better practice to use one of Bash’s builtins to do this, either command, type, or hash. I prefer using hash, which searches PATH for a command of the given name, and loads it into Bash’s command hash table if found. An implementation like the below works well:

if hash vi 2>/dev/null; then
    export EDITOR=vi
fi

This ignores any error output from the hash call by redirecting it to the null device, and only defines the value for EDITOR if a matching command is found.

You can compact this syntax into one line:

hash vi 2>/dev/null && export EDITOR=vi

Thanks to commenter Yu-Jie Lin for pointing out the above abbreviation.

Calculating with Bash

Posted on by

If you’re at a terminal or writing a script and need to do some quick integer calculations, Bash offers arithmetic expansion with the $((...)) syntax.

The usual arithmetic operations of addition, subtraction, multiplication, and division are supported, but the name of arithmetic expansion is a bit misleading as it also supports operations well beyond the elementary arithmetic set, including exponentiation, equality and inequality checks, and logical and bitwise NOT/AND/OR/XOR:

  • VAR++ VAR-- — post-increment, post-decrement
  • ++VAR --VAR — pre-increment, pre-decrement
  • + - — addition, subtraction
  • ** — exponentiation
  • * / % — multiplication, division, remainder
  • == != — equality, inequality
  • < > <= >= — comparison
  • ! && || — logical NOT, AND, OR
  • ~ & | ^ — bitwise NOT, AND, OR, XOR

These operators and their precedence are all the same as used in most C-style languages. Pivotally, the expansion can include variables defined elsewhere in the command or script:

$ number=1234
$ printf '%s\n' "$((number * 2))"
2468

One place this can be very useful is in calculating filesizes, since most of the time shell commands work in either bytes or potentially arbitrary disk block sizes. If you work with SI decimal prefixes, you can use powers of 10 to calculate the sizes:

kilobytes=$((bytes / 10**3))
megabytes=$((bytes / 10**6))
gigabytes=$((bytes / 10**9))

If you prefer to work with IEC binary prefixes, i.e. multiples of 1024:

kilobytes=$((bytes / 2**10))
megabytes=$((bytes / 2**20))
gigabytes=$((bytes / 2**30))

Or you could use bit shifting:

kilobytes=$((bytes >> 10))
megabytes=$((bytes >> 20))
gigabytes=$((bytes >> 30))

Lazier Tab completion

Posted on by

Bash completion allows you to complete paths, commands, and filenames, and where implemented can even expand the syntax of commands like git and svn. It’s very convenient by default, but there are a couple of tweaks that can make it much faster and easier to use.

Single Tab press

With Bash completion enabled, the default is to complete the current path to the longest unique string it can on the first press of the Tab key, and then show a list of all the possible completions if the Tab key is then pressed a second time.

If you don’t like pressing Tab twice, you can fix this with a line in .inputrc so that the line is both completed to the longest unambiguous pattern, and then possible values are also printed:

set show-all-if-ambiguous on

Log out and then in again, or re-read your .inputrc file with Ctrl+X then Ctrl+R, and you should find that you now only need to press Tab once on completions to get both behaviours in one hit.

If you want Tab completion of filenames to work the same way in Vim’s command mode, you can include the below in your .vimrc. Note that the separator is a colon, rather than a comma, which means to perform both the command completion and suggestion at the first Tab press:

set wildmode=longest:list

Complete case-insensitively

If you commonly Tab your way through long paths that sometimes include mixed-case names, you can make Readline handle that for you by making the completion case-insensitive, with the following in your .inputrc:

set completion-ignore-case on

You can supplement this to make the completion apply similar insensitivity between hyphens and underscores:

set completion-map-case on

If you were tabbing your way through an ImageMagick directory, for example, and forgot the capital letters in typing imagem, Readline would quietly complete it to ImageMagick for you. Similarly, if you typed lib-undersc when the actual filename was lib_underscore, the shell would quietly fix that for you too.

You can get case-insensitivity for the filename completion in Vim with the following. I suggest wrapping it in a conditional as below, since it’s a reasonably new option:

if exists("&wildignorecase")
    set wildignorecase
endif

Unfortunately, there doesn’t seem to be a way that I can find to reproduce the underscore/hyphen mapping. If you know of one, please comment.

Don’t prompt for many possible completions

If you’re on a fast terminal and you’re not bothered by having the screeds of texts it can sometimes generate being spat at you, you can turn off the sometimes annoying prompt that checks with you whether you want to show more than a hundred results for possible completion, with the following in .inputrc:

set completion-query-items -1

If you do this, you should probably set Readline’s built-in pager to be off, otherwise it’ll attempt to walk you through the possible completions page-by-page:

set page-completions off

If you do happen to deal with directories with more than a thousand files in them now and then (which doesn’t tend to happen much in routine system administration), it might be a bit safer to set it to some number much higher than the default of 100:

set completion-query-items 1000

These three tips combined make tabbing your way through path names much more pleasant and intuitive; you won’t find yourself irritably tapping Tab over and over nearly as much. Even if you’re very much used to playing ball with the stricter idiosyncrasies of the usual form of tab completion, you may find making the shell do the work for you in this way starts to feel very natural very quickly.

Vi mode in Bash

Posted on by

Because the Bash shell uses GNU Readline, you’re able to set options in .bashrc and .inputrc to extensively customise how you enter your command lines, including specifying whether you want the default Emacs-like keybindings (e.g. Ctrl+W to erase a word), or bindings using a modal interface familiar to vi users.

To use vi mode in Bash and any other tool that uses GNU Readline, such as the MySQL command line, you need only put this into your .inputrc file, then log out and in again:

set editing-mode vi

If you only want to use this mode in Bash, an alternative is to use the following in your .bashrc, again with a login/logout or resourcing of the file:

set -o vi

You can check this has applied correctly with the following, which will spit out a list of the currently available bindings for a set of fixed possible actions for text:

$ bind -P

The other possible value for editing-mode is emacs. Both options simply change settings in the output of bind -P above.

This done, you should find that you open a command line in insert mode, but when you press Escape or Ctrl+[, you will enter an emulation of Vi’s normal mode. Enter will run the command in its present state while in either mode. The bindings of most use in this mode are the classic keys for moving to positions on a line:

  • ^ — Move to start of line
  • $ — Move to end of line
  • b — Move back a word
  • w — Move forward a word
  • e — Move to the end of the next word

Deleting, yanking, and pasting all work too. Also note that k and j in normal mode allow you to iterate through your history in the same way that Ctrl+P and Ctrl+N do in Emacs mode. Searching with ? and / doesn’t work by default, but the Ctrl+R and Ctrl+S bindings still seem to work.

If you are reasonably used to working with the modeless Emacs for most of your shell work but do occasionally find yourself in a situation where being able to edit a long command line in vi would be handy, you may prefer to press Ctrl+X Ctrl+E to bring up the command line in your $EDITOR instead, affording you the complete power of Vim to edit rather than the somewhat sparse vi emulation provided by Readline. If you want to edit a command you just submitted, the fc (fix command) Bash builtin works too.

The shell is a very different environment for editing text, being inherently interactive and fixed to one line rather than a static multi-line buffer, which leads some otherwise diehard vi users such as myself to prefer the Emacs mode for editing commands. For one thing, vi mode in Bash trips on the vi anti-pattern of putting you in insert mode by default, and at the start of every command line. But if the basic vi commands are etched indelibly into your memory and have become automatic, you may appreciate being able to edit your command line using these keys instead. It’s certainly worth a try at any rate, and I do still occasionally see set -o vi in the .bashrc files of a few of the pros on GitHub.

Arabesque passed 100,000 visits today, having existed little over a month, with over 60,000 unique visitors. Thanks very much to everyone who reads and comments on the articles.

Bash process substitution

Posted on by

For tools like diff that work with multiple files as parameters, it can be useful to work with not just files on the filesystem, but also potentially with the output of arbitrary commands. Say, for example, you wanted to compare the output of ps and ps -e with diff -u. An obvious way to do this is to write files to compare the output:

$ ps > ps.out
$ ps -e > pse.out
$ diff -u ps.out pse.out

This works just fine, but Bash provides a shortcut in the form of process substitution, allowing you to treat the standard output of commands as files. This is done with the <() and >() operators. In our case, we want to direct the standard output of two commands into place as files:

$ diff -u <(ps) <(ps -e)

This is functionally equivalent, except it’s a little tidier because it doesn’t leave files lying around. This is also very handy for elegantly comparing files across servers, using ssh:

$ diff -u .bashrc <(ssh remote cat .bashrc)

Conversely, you can also use the >() operator to direct from a filename context to the standard input of a command. This is handy for setting up in-place filters for things like logs. In the following example, I’m making a call to rsync, specifying that it should make a log of its actions in log.txt, but filter it through grep -vF .tmp first to remove anything matching the fixed string .tmp:

$ rsync -arv --log-file=>(grep -vF .tmp >log.txt) src/ host::dst/

Combined with tee this syntax is a way of simulating multiple filters for a stdout stream, transforming output from a command in as many ways as you see fit:

$ ps -ef | tee >(awk '$1=="tom"' >toms-procs.txt) \
               >(awk '$1=="root"' >roots-procs.txt) \
               >(awk '$1!="httpd"' >not-apache-procs.txt) \
               >(awk 'NR>1{print $1}' >pids-only.txt)

In general, the idea is that wherever on the command line you could specify a file to be read from or written to, you can instead use this syntax to make an implicit named pipe for the text stream.

Thanks to Reddit user Rhomboid for pointing out an incorrect assertion about this syntax necessarily abstracting mkfifo calls, which I’ve since removed.

Better Bash history

Posted on by

By default, the Bash shell keeps the history of your most recent session in the .bash_history file, and the commands you’ve issued in your current session are also available with a history call. These defaults are useful for keeping track of what you’ve been up to in the shell on any given machine, but with disks much larger and faster than they were when Bash was designed, a little tweaking in your .bashrc file can record history more permanently, consistently, and usefully.

Append history instead of rewriting it

You should start by setting the histappend option, which will mean that when you close a session, your history will be appended to the .bash_history file rather than overwriting what’s in there.

shopt -s histappend

Allow a larger history file

The default maximum number of commands saved into the .bash_history file is a rather meager 500. If you want to keep history further back than a few weeks or so, you may as well bump this up by explicitly setting $HISTSIZE to a much larger number in your .bashrc. We can do the same thing with the $HISTFILESIZE variable.

HISTFILESIZE=1000000
HISTSIZE=1000000

The man page for Bash says that HISTFILESIZE can be unset to stop truncation entirely, but unfortunately this doesn’t work in .bashrc files due to the order in which variables are set; it’s therefore more straightforward to simply set it to a very large number.

If you’re on a machine with resource constraints, it might be a good idea to occasionally archive old .bash_history files to speed up login and reduce memory footprint.

Don’t store specific lines

You can prevent commands that start with a space from going into history by setting $HISTCONTROL to ignorespace. You can also ignore duplicate commands, for example repeated du calls to watch a file grow, by adding ignoredups. There’s a shorthand to set both in ignoreboth.

HISTCONTROL=ignoreboth

You might also want to remove the use of certain commands from your history, whether for privacy or readability reasons. This can be done with the $HISTIGNORE variable. It’s common to use this to exclude ls calls, job control builtins like bg and fg, and calls to history itself:

HISTIGNORE='ls:bg:fg:history'

Record timestamps

If you set $HISTTIMEFORMAT to something useful, Bash will record the timestamp of each command in its history. In this variable you can specify the format in which you want this timestamp displayed when viewed with history. I find the full date and time to be useful, because it can be sorted easily and works well with tools like cut and awk.

HISTTIMEFORMAT='%F %T '

Use one command per line

To make your .bash_history file a little easier to parse, you can force commands that you entered on more than one line to be adjusted to fit on only one with the cmdhist option:

shopt -s cmdhist

Store history immediately

By default, Bash only records a session to the .bash_history file on disk when the session terminates. This means that if you crash or your session terminates improperly, you lose the history up to that point. You can fix this by recording each line of history as you issue it, through the $PROMPT_COMMAND variable:

PROMPT_COMMAND='history -a'

Smarter directory navigation

Posted on by

Shell autocompletion of directory and file names makes navigating through your filesystem quite a bit quicker than if you had to type all of the paths in full, but there are a few other ways to make navigating through and between long pathnames on your filesystem a little bit easier.

cd

As a first note, typing cd with no arguments will take you straight back to your HOME directory; you don’t actually need to type cd ~ or cd $HOME in most cases.

cd ~username

You can move to any particular user’s HOME directory with the general form cd ~username.

cd -

To navigate to the directory you were in before the last cd call, you can use cd -. This will both move you back into that directory and echo the pathname it was on for you.

$ pwd
/home/tom
$ cd projects
$ pwd
/home/tom/projects
$ cd -
/home/tom

pushd/popd

If you are working mostly within one directory and need to change to another briefly for some reason, rather than using cd you can use pushd to move to that directory and add it onto the directory stack. You can move to further directories with pushd successively, and then when you’re done in each one, popd will take you a step backward again. Each time you call pushd or popd, it will print the current contents of the directory stack:

$ pushd projects
~/projects ~
$ pushd /usr/local/bin
/usr/local/bin ~/projects ~
$ popd
~/projects ~
$ popd
~

CDPATH

By default, you can move around relatively in the filesystem tree by typing cd dir for any subdirectory of your working directory, rather than its full path. You can generalise this to provide a list of directories to check successively for matching subdirectories:

$ export CDPATH=.:~

In this case, if I were to type cd projects, if there were no subdirectory of the working directory by the name of project but there was one called /home/tom/project, I would be sent there instead. This probably isn’t a good thing to set in any non-interactive terminal, but it can be a nice shortcut for interactive shells. Note that the first part of the list should always be ., for the working directory.

This has the side effect of printing the complete path to which you just moved on a line before your prompt if it’s used. I happen to find this quite helpful.

Tolerate typos

In an interactive shell, if you make a small mistake in typing a path for cd that means it doesn’t resolve to an actual directory, it’s generally safe to have the shell correct mistakes for you and move you into the directory you most likely intended:

shopt -s cdspell

This will correct things like dropped or swapped characters in the path you typed:

$ cd /home/tmo
/home/tom

Variables

If you know you’re going to be switching back and forth between several directories, it often makes sense to put them into variables for quick changing, or even just editing files directly without having to move around at all:

$ acnf=/usr/local/apache/conf
$ abin=/usr/local/apache/bin
$ sanc=/var/www/sanctum
$ vim "$acnd"/httpd.conf
$ vim "$sanc"/index.html
$ cd "$abin"
$ ./apachectl configtest
$ ./apachectl restart

This last one may seem like a very elementary trick, or something that you’d typically only do within a shell script. However, if you use consistent variable names or even put them into a startup file like .bashrc, it’s actually quite surprising how much time it can save you; you start to realise how much time you were spending typing out paths.

Counting with grep and uniq

Posted on by

A common idiom in Unix is to count the lines of output in a file or pipe with wc -l:

$ wc -l example.txt
43
$ ps -e | wc -l
97

Sometimes you want to count the number of lines of output from a grep call, however. You might do it this way:

$ ps -ef | grep apache | wc -l
6

But grep has built-in counting of its own, with the -c option:

$ ps -ef | grep -c apache
6

The above is more a matter of good style than efficiency, but another tool with a built-in counting option that could save you time is the oft-used uniq. The below example shows a use of uniq to filter a sorted list into unique rows:

$ ps -eo user= | sort | uniq
105
daemon
lp
mysql
nagios
postfix
root
snmp
tom
www-data

If it would be useful to know in this case how many processes were being run by each of these users, you can include the -c option for uniq:

$ ps -eo user= | sort | uniq -c
    1 105
    1 daemon
    1 lp
    1 mysql
    1 nagios
    2 postfix
    78 root
    1 snmp
    7 tom
    5 www-data

You could even sort this output itself to show the users running the most processes first with sort -rn:

$ ps -eo user= | sort | uniq -c | sort -rn
    78 root
    8 tom
    5 www-data
    2 postfix
    1 snmp
    1 nagios
    1 mysql
    1 lp
    1 daemon
    1 105

Incidentally, if you’re not counting results and really do just want a list of unique users, you can leave out the uniq and just add the -u flag to sort:

$ ps -eo user= | sort -u
105
daemon
lp
mysql
nagios
postfix
root
snmp
tom
www-data

The above means I actually find myself using uniq with no options quite seldom.

Bash shell expansion

Posted on by

Operations in the shell very often involve repeating yourself or working with sets or ranges of information. This is where the various kinds of expansion in the shell are most useful, for generating a large number of shell tokens using a compact syntax. I’ll discuss two types here: filename expansions, which will be familiar to most shell users; and brace expansions, which seem to be in less common usage.

Filename expansions (globs)

The most commonly seen type of filename expansion in Bash is the * wildcard, which can be used to match lists of files in any directory, whether through complete listings or partial matches:

$ ls *
a.txt b.txt c.txt d.txt extra.txt README install.sh
$ ls *.txt
a.txt b.txt c.txt d.txt extra.txt

However, there are two other types of filename expansions. Firstly, you can match single characters rather than ranges of characters with a question mark:

$ ls ?.txt
a.txt b.txt c.txt d.txt

You can further restrict this by matching only a single character in a specified group of characters, or within a range:

$ ls [ac].txt
a.txt c.txt
$ ls [a-c].txt
a.txt b.txt c.txt

These can be useful in very large directories to only view files starting with a nominated letter or number, or range of letters or numbers.

Brace expansion

Brace expansion is a little different, as it generates shell tokens that don’t necessarily correspond to existing files. This allows you to expand a single general form into a lot of space-delimited specific tokens. This is probably best explained with examples. First of all, you can define the expansions you want with comma separation:

$ echo example{test,show,define,declare}
exampletest exampleshow exampledefine exampledeclare

If you’re dealing with sequences of numbers or letters, there’s also the .. separator syntax for ranges:

$ echo example{a..d}
examplea exampleb examplec exampled
$ echo example{1..5}
example1 example2 example3 example4 example5

This can be useful for creating, renaming, copying, or moving files:

$ touch file.txt.{1..6}
$ mv file.{txt,html}
$ cp file.txt{,.bak}
$ mv website.co.nz/{testing,production}/index.php

I also find it useful when dealing with long paths in Subversion branching and merging:

$ svn copy svn://server/project/{trunk,branches/experimental}
$ svn merge svn://server/project/{branches/experimental,trunk} .

You can also combine and even nest these expansions, which makes it useful for creating directory trees for new projects or chroot environments:

$ mkdir -p {test,prod}/{,usr/,usr/local/}{{,s}bin,etc,lib,share}

The above rather compact syntax creates all of the following directories:

test
test/bin
test/etc
test/sbin
test/lib
test/usr
test/usr/bin
test/usr/etc
test/usr/sbin
test/usr/lib
test/usr/share
test/usr/local
test/usr/local/bin
test/usr/local/etc
test/usr/local/sbin
test/usr/local/lib
test/usr/local/share
test/share
prod
prod/bin
prod/etc
prod/sbin
prod/lib
prod/usr
prod/usr/bin
prod/usr/etc
prod/usr/sbin
prod/usr/lib
prod/usr/share
prod/usr/local
prod/usr/local/bin
prod/usr/local/etc
prod/usr/local/sbin
prod/usr/local/lib
prod/usr/local/share
prod/share

Thanks to Reddit user silvermoot for corrections/suggestions.