Author Archives: Tom Ryder

About Tom Ryder

sysadmin, dev, writer, IRC nuisance. vi/vim/vimself

Bash command existence

Posted on by

If you set environment variables like your EDITOR in your .bashrc file that refer to commands that you expect to be available on the system, it’s prudent to check that an appropriate command actually exists before making the setting.

A common way of approaching this is using which, in a syntax similar to the below, which sets EDITOR to the first executable instance of vi found in PATH, or blank if not found:

EDITOR=$(which vi)

Because the behaviour of which can be unexpected, it’s better practice to use one of Bash’s builtins to do this, either command, type, or hash. I prefer using hash, which searches PATH for a command of the given name, and loads it into Bash’s command hash table if found. An implementation like the below works well:

if hash vi 2>/dev/null; then
    export EDITOR=vi
fi

This ignores any error output from the hash call by redirecting it to the null device, and only defines the value for EDITOR if a matching command is found.

You can compact this syntax into one line:

hash vi 2>/dev/null && export EDITOR=vi

Thanks to commenter Yu-Jie Lin for pointing out the above abbreviation.

Reloading tmux config

Posted on by

If you have made changes to your tmux configuration file in the ~/.tmux.conf file, it shouldn’t be necessary to start the server up again from scratch with kill-server. Instead, you can prompt the current tmux session to reload the configuration with the source-file command.

This can be done either from within tmux, by pressing Ctrl+B and then : to bring up a command prompt, and typing:

:source-file ~/.tmux.conf

Or simply from a shell:

$ tmux source-file ~/.tmux.conf

This should apply your changes to the running tmux server without affecting the sessions or windows within them.

Learning /bin and /usr/bin

Posted on by

When you have some spare time, something instructive to do that can help fill gaps in your Unix knowledge and to get a better idea of the programs installed on your system and what they can do is a simple whatis call, run over all the executable files in your /bin and /usr/bin directories. This will give you a one-line summary of the file’s function if available from man pages.

tom@conan:/bin$ whatis *
bash (1) - GNU Bourne-Again SHell
bunzip2 (1) - a block-sorting file compressor, v1.0.4
busybox (1) - The Swiss Army Knife of Embedded Linux
bzcat (1) - decompresses files to stdout
...

tom@conan:/usr/bin$ whatis *
[ (1)                - check file types and compare values
2to3 (1)             - Python2 to Python3 converter
2to3-2.7 (1)         - Python2 to Python3 converter
411toppm (1)         - convert Sony Mavica .411 image to ppm
...

It also works on many of the files in other directories, such as /etc:

tom@conan:/etc$ whatis *
acpi (1)             - Shows battery status and other ACPI information
adduser.conf (5)     - configuration file for adduser(8) and addgroup(8)
adjtime (3)          - correct the time to synchronize the system clock
aliases (5)          - Postfix local alias database format
...

Because packages often install more than one binary and you’re only in the habit of using one or two of them, this process can tell you about programs on your system that you may have missed, particularly standard tools that solve common problems. As an example, I first learned about watch this way, having used a clunky solution with for loops with sleep calls to do the same thing many times before.

Searching compressed files

Posted on by

If you need to search a set of log files in /var/log, some of which have been compressed with gzip as part of the logrotate procedure, it can be a pain to deflate them to check them for a specific string, particularly where you want to include the current log which isn’t compressed:

$ gzip -d log.1.gz log.2.gz log.3.gz
$ grep pattern log log.1 log.2 log.3

It turns out to be a little more elegant to use the -c switch for gzip to deflate the files in-place and write the content of the files to standard output, concatenating any uncompressed files you may also want to search in with cat:

$ gzip -dc log.*.gz | cat - log | grep pattern

This and similar operations with compressed files are common enough problems that short scripts in /bin on GNU/Linux systems exist, providing analogues to existing tools that can work with files in both a compressed and uncompressed state. In this case, the zgrep tool is of the most use to us:

$ zgrep pattern log*

Note that this search will also include the uncompressed log file and search it normally. The tools are for possibly compressed files, which makes them particularly well-suited to searching and manipulating logs in mixed compression states. It’s worth noting that most of these are actually reasonably simple shell scripts.

The complete list of tools, most of which do the same thing as their z-less equivalents, can be gleaned with a quick whatis call:

$ pwd
/bin
$ whatis z*
zcat (1)   - compress or expand files
zcmp (1)   - compare compressed files
zdiff (1)  - compare compressed files
zegrep (1) - search possibly compressed files for a regular expression
zfgrep (1) - search possibly compressed files for a regular expression
zforce (1) - force a '.gz' extension on all gzip files
zgrep (1)  - search possibly compressed files for a regular expression
zless (1)  - file perusal filter for crt viewing of compressed text
zmore (1)  - file perusal filter for crt viewing of compressed text
znew (1)   - recompress .Z files to .gz files

If you are dealing with files compressed with bzip, the analogous tools instead begin with “bz”.

Calculating with Bash

Posted on by

If you’re at a terminal or writing a script and need to do some quick integer calculations, Bash offers arithmetic expansion with the $((...)) syntax.

The usual arithmetic operations of addition, subtraction, multiplication, and division are supported, but the name of arithmetic expansion is a bit misleading as it also supports operations well beyond the elementary arithmetic set, including exponentiation, equality and inequality checks, and logical and bitwise NOT/AND/OR/XOR:

  • VAR++ VAR-- — post-increment, post-decrement
  • ++VAR --VAR — pre-increment, pre-decrement
  • + - — addition, subtraction
  • ** — exponentiation
  • * / % — multiplication, division, remainder
  • == != — equality, inequality
  • < > <= >= — comparison
  • ! && || — logical NOT, AND, OR
  • ~ & | ^ — bitwise NOT, AND, OR, XOR

These operators and their precedence are all the same as used in most C-style languages. Pivotally, the expansion can include variables defined elsewhere in the command or script:

$ number=1234
$ printf '%s\n' "$((number * 2))"
2468

One place this can be very useful is in calculating filesizes, since most of the time shell commands work in either bytes or potentially arbitrary disk block sizes. If you work with SI decimal prefixes, you can use powers of 10 to calculate the sizes:

kilobytes=$((bytes / 10**3))
megabytes=$((bytes / 10**6))
gigabytes=$((bytes / 10**9))

If you prefer to work with IEC binary prefixes, i.e. multiples of 1024:

kilobytes=$((bytes / 2**10))
megabytes=$((bytes / 2**20))
gigabytes=$((bytes / 2**30))

Or you could use bit shifting:

kilobytes=$((bytes >> 10))
megabytes=$((bytes >> 20))
gigabytes=$((bytes >> 30))

Restricting public keys

Posted on by

It may be the case that while you’re happy to allow a user or process to have public key authentication access to your server via the ~/.ssh/authorized_keys file, you don’t necessarily want to give them a full shell, or you may want to restrict them from doing things like SSH port forwarding or X11 forwarding.

One method that’s supposed to prevent users from accessing a shell is by defining their shell in /etc/passwd as /bin/false, which does indeed prevent them from logging in with the usual ssh or ssh command syntax. This isn’t a good approach because it still allows port forwarding and other SSH-enabled services.

If you want to restrict the use of logins with a public key, you can prepend option pairs to its line in the authorized_keys file. Some of the most useful options here include:

  • from="<hostname/ip>" — Prepending from="*.example.com" to the key line would only allow public-key authenticated login if the connection was coming from some host with a reverse DNS of example.com. You can also put IP addresses in here. This is particularly useful for setting up automated processes through keys with null passphrases.
  • command="<command>" — Means that once authenticated, the command specified is run, and the connection is closed. Again, this is useful in automated setups for running only a certain script on successful authentication, and nothing else.
  • no-agent-forwarding — Prevents the key user from forwarding authentication requests to an SSH agent on their client, using the -A or ForwardAgent option to ssh.
  • no-port-forwarding — Prevents the key user from forwarding ports using -L and -R.
  • no-X11-forwarding — Prevents the key user from forwarding X11 processes.
  • no-pty — Prevents the key user from being allocated a tty device at all.

So, for example, a public key that is only used to run a script called runscript on the server by the client runscript@client.example:

command="runscript",client="client.example",no-pty,no-agent-forwarding,no-port-forwarding ssh-rsa AAAAB2....19Q runscript@client.example

A public key for a user whom you were happy to allow to log in from anywhere with a full shell, but did not want to allow agent, port, or X11 forwarding:

no-agent-forwarding,no-port-forwarding,no-X11-forwarding ssh-rsa AAAAD3....19Q user@client.example

Use of these options goes a long way to making your public key authentication setup harder to exploit, and is very consistent with the principle of least privilege. To see a complete list of the options available, check out the man page for sshd.

Avoiding svn import

Posted on by

The usual workflow for Subversion when putting an existing project under version control, and the one documented in the popular Version Control with Subversion book, is using svn import:

$ svn import /home/you/yourproject svn://vcs.network/yourproject/trunk -m "Import."

This gets the results you need, but realistically it often happens that you’ve already started your project, and in some cases it may even be live to some extent. This means that if you want the instance of the project that you just imported to also become a working copy of the project, you need to do something like this:

$ cd /home/you
$ mv yourproject yourproject.imported
$ svn checkout svn://vcs.network/yourproject/trunk yourproject

Or possibly this:

$ cd /home/you/yourproject
$ svn checkout --force svn://vcs.network/yourproject/trunk .

There’s a tidier way of managing this that completely avoids the use of the import command, turning the imported copy directly into your first working copy, and also allows you finer-grained control over which files should be included in the repository. Create an empty repository on your server, check its trunk out directly inside the existing project directory, add everything you want to add (including svn:ignore for files you don’t, among other properties), and commit it:

$ svn mkdir svn://vcs.network/yourproject/{branches,tags,trunk} -m "Recommended structure"
$ cd /home/you/yourproject
$ svn checkout svn://vcs.network/yourproject/trunk .
$ svn add file1.c file1.h README.txt
$ svn propset svn:ignore '*.o' .
$ svn commit -m "First commit."

Using this means in most cases you can completely eschew the use of svn import, and you might well find that you don’t miss it.

Temporary files

Posted on by

With judicious use of tricks like pipes, redirects, and process substitution in modern shells, it’s very often possible to avoid using temporary files, doing everything inline and keeping them quite neat. However when manipulating a lot of data into various formats you do find yourself occasionally needing a temporary file, just to hold data temporarily.

A common way to deal with this is to create a temporary file in your home directory, with some arbitrary name, something like test or working:

$ ps -ef >~/test

If you want to save the information indefinitely for later use, this makes sense, although it would be better to give it a slightly more instructive name than just test.

If you really only needed the data temporarily, however, you’re much better to use the temporary files directory. This is usually /tmp, but for good practice’s sake it’s better to check the value of TMPDIR first, and only use /tmp as a default:

$ ps -ef >"${TMPDIR:-/tmp}"/test

This is getting better, but there is still a significant problem: there’s no built-in check that the test file doesn’t already exist, perhaps being used by some other user or program, particularly another running instance of the same script.

To that end, we have the mktemp program, which creates an empty temporary file in the appropriate directory for you without overwriting anything, and prints the filename it created. This allows you to use the file inline in both shell scripts and one-liners, and is much safer than specifying hardcoded paths:

$ mktemp
/tmp/tmp.yezXn0evDf
$ procsfile=$(mktemp)
$ printf '%s\n' "$procsfile"
/tmp/tmp.9rBjzWYaSU
$ ps -ef >"$procsfile"

If you’re going to create several such files for related purposes, you could also create a directory in which to put them using the -d option:

$ procsdir=$(mktemp -d)
$ printf '%s\n' "$procsdir"
/tmp/tmp.HMAhM2RBSO

On GNU/Linux systems, files of a sufficient age in TMPDIR are cleared on boot (controlled in /etc/default/rcS on Debian-derived systems, /etc/cron.daily/tmpwatch on Red Hat ones), making /tmp useful as a general scratchpad as well as for a kind of relatively reliable inter-process communication without cluttering up users’ home directories.

In some cases, there may be additional advantages in using /tmp for its designed purpose as some administrators choose to mount it as a tmpfs filesystem, so it operates in RAM and works very quickly. It’s also common practice to set the noexec flag on the mount to prevent malicious users from executing any code they manage to find or save in the directory.

Lazier Tab completion

Posted on by

Bash completion allows you to complete paths, commands, and filenames, and where implemented can even expand the syntax of commands like git and svn. It’s very convenient by default, but there are a couple of tweaks that can make it much faster and easier to use.

Single Tab press

With Bash completion enabled, the default is to complete the current path to the longest unique string it can on the first press of the Tab key, and then show a list of all the possible completions if the Tab key is then pressed a second time.

If you don’t like pressing Tab twice, you can fix this with a line in .inputrc so that the line is both completed to the longest unambiguous pattern, and then possible values are also printed:

set show-all-if-ambiguous on

Log out and then in again, or re-read your .inputrc file with Ctrl+X then Ctrl+R, and you should find that you now only need to press Tab once on completions to get both behaviours in one hit.

If you want Tab completion of filenames to work the same way in Vim’s command mode, you can include the below in your .vimrc. Note that the separator is a colon, rather than a comma, which means to perform both the command completion and suggestion at the first Tab press:

set wildmode=longest:list

Complete case-insensitively

If you commonly Tab your way through long paths that sometimes include mixed-case names, you can make Readline handle that for you by making the completion case-insensitive, with the following in your .inputrc:

set completion-ignore-case on

You can supplement this to make the completion apply similar insensitivity between hyphens and underscores:

set completion-map-case on

If you were tabbing your way through an ImageMagick directory, for example, and forgot the capital letters in typing imagem, Readline would quietly complete it to ImageMagick for you. Similarly, if you typed lib-undersc when the actual filename was lib_underscore, the shell would quietly fix that for you too.

You can get case-insensitivity for the filename completion in Vim with the following. I suggest wrapping it in a conditional as below, since it’s a reasonably new option:

if exists("&wildignorecase")
    set wildignorecase
endif

Unfortunately, there doesn’t seem to be a way that I can find to reproduce the underscore/hyphen mapping. If you know of one, please comment.

Don’t prompt for many possible completions

If you’re on a fast terminal and you’re not bothered by having the screeds of texts it can sometimes generate being spat at you, you can turn off the sometimes annoying prompt that checks with you whether you want to show more than a hundred results for possible completion, with the following in .inputrc:

set completion-query-items -1

If you do this, you should probably set Readline’s built-in pager to be off, otherwise it’ll attempt to walk you through the possible completions page-by-page:

set page-completions off

If you do happen to deal with directories with more than a thousand files in them now and then (which doesn’t tend to happen much in routine system administration), it might be a bit safer to set it to some number much higher than the default of 100:

set completion-query-items 1000

These three tips combined make tabbing your way through path names much more pleasant and intuitive; you won’t find yourself irritably tapping Tab over and over nearly as much. Even if you’re very much used to playing ball with the stricter idiosyncrasies of the usual form of tab completion, you may find making the shell do the work for you in this way starts to feel very natural very quickly.

Vi mode in tmux

Posted on by

Updated May 2017 to show the new syntax for the copy-mode keys.

tmux offers a set of vi-like bindings for navigating a buffer in a window. These allow you to not only navigate through the buffer beyond what your screen is currently showing, but also to search all the output generated thus far, and to select and copy text that can be pasted in any other window in the tmux session.

You can enable this as a default setting in .tmux.conf with the following:

set-window-option -g mode-keys vi

You can confirm this is working by pressing Ctrl+B and then : in a tmux session to bring up the command line, and typing:

list-keys -T copy-mode-vi

In version 2.3 and below, the syntax is different:

list-keys -t vi-copy

This will bring up a complete list of the vi-like functionality available to you in this mode.

With this done, within a tmux instance, pressing Ctrl+B and then [ will enter copy mode, allowing you to copy text or view the history of the buffer, including searching with / and ?. Most of the basic vi movements work, including screenwise vertical movement commands like Ctrl+F and Ctrl+B. You can leave this mode just by pressing Enter, or you can start a selection by pressing Space on a character, moving to another, and then pressing Enter. If you have text copied like this you can paste it into any tmux window in that session by pressing Ctrl+B and then ].

If you don’t mind artifically introducing a few Vim-only features to the vi mode, you can set things up so that v starts a selection and y finishes it in the same way that Space and Enter do, more like Vim:

bind-key -T copy-mode-vi 'v' send -X begin-selection
bind-key -T copy-mode-vi 'y' send -X copy-selection-and-cancel

In version 2.3 and below, the syntax is very different:

bind-key -t vi-copy 'v' begin-selection
bind-key -t vi-copy 'y' copy-selection

Even if you’re using a terminal emulator in a GUI environment, if you get good at using this you’ll find it’s much faster and more precise than clicking and dragging to select text to copy and paste it. It keeps you from having to move to the mouse, and additionally eliminates the problem of copying whole lines split into more than one panes.

Note that all of the above assumes that your prefix is the default of Ctrl+B, which many tmux users, myself included, seem to prefer to change to the old GNU Screen prefix key of Ctrl+A.