Compiling in $HOME

Posted on by

If you don’t have root access on a particular GNU/Linux system that you use, or if you don’t want to install anything to the system directories and potentially interfere with others’ work on the machine, one option is to build your favourite tools in your $HOME directory.

This can be useful if there’s some particular piece of software that you really need for whatever reason, particularly on legacy systems that you share with other users or developers. The process can include not just applications, but libraries as well; you can link against a mix of your own libraries and the system’s libraries as you need.

Preparation

In most cases this is actually quite a straightforward process, as long as you’re allowed to use the system’s compiler and any relevant build tools such as autoconf. If the ./configure script for your application allows a --prefix option, this is generally a good sign; you can normally test this with --help:

$ mkdir src
$ cd src
$ wget -q http://fooapp.example.com/fooapp-1.2.3.tar.gz
$ tar -xf fooapp-1.2.3.tar.gz
$ cd fooapp-1.2.3
$ pwd
/home/tom/src/fooapp-1.2.3
$ ./configure --help | grep -- --prefix
  --prefix=PREFIX    install architecture-independent files in PREFIX

Don’t do this if the security policy on your shared machine explicitly disallows compiling programs! However, it’s generally quite safe as you never need root privileges at any stage of the process.

Naturally, this is not a one-size-fits-all process; the build process will vary for different applications, but it’s a workable general approach to the task.

Installing

Configure the application or library with the usual call to ./configure, but use your home directory for the prefix:

$ ./configure --prefix=$HOME

If you want to include headers or link against libraries in your home directory, it may be appropriate to add definitions for CFLAGS and LDFLAGS to refer to those directories:

$ CFLAGS="-I$HOME/include" \
> LDFLAGS="-L$HOME/lib" \
> ./configure --prefix=$HOME

Some configure scripts instead allow you to specify the path to particular libraries. Again, you can generally check this with --help.

$ ./configure --prefix=$HOME --with-foolib=$HOME/lib

You should then be able to install the application with the usual make and make install, needing root privileges for neither:

$ make
$ make install

If successful, this process will insert files into directories like $HOME/bin and $HOME/lib. You can then try to call the application by its full path:

$ $HOME/bin/fooapp -v
fooapp v1.2.3

Environment setup

To make this work smoothly, it’s best to add to a couple of environment variables, probably in your .bashrc file, so that you can use the home-built application transparently.

First of all, if you linked the application against libraries also in your home directory, it will be necessary to add the library directory to LD_LIBRARY_PATH, so that the correct libraries are found and loaded at runtime:

$ /home/tom/bin/fooapp -v
/home/tom/bin/fooapp: error while loading shared libraries: libfoo.so: cannot open shared...
Could not load library foolib
$ export LD_LIBRARY_PATH=$HOME/lib
$ /home/tom/bin/fooapp -v
fooapp v1.2.3

An obvious one is adding the $HOME/bin directory to your $PATH so that you can call the application without typing its path:

$ fooapp -v
-bash: fooapp: command not found
$ export PATH="$HOME/bin:$PATH"
$ fooapp -v
fooapp v1.2.3

Similarly, defining MANPATH so that calls to man will read the manual for your build of the application first is worthwhile. You may find that $MANPATH is empty by default, so you will need to append other manual locations to it. An easy way to do this is by appending the output of the manpath utility:

$ man -k fooapp
$ manpath
/usr/local/man:/usr/local/share/man:/usr/share/man
$ export MANPATH="$HOME/share/man:$(manpath)"
$ man -k fooapp
fooapp (1) - Fooapp, the programmer's foo apper

This done, you should be able to use your private build of the software comfortably, and all without never needing to reach for root.

Caveats

This tends to work best for userspace tools like editors or other interactive command-line apps; it even works for shells. However this is not a typical use case for most applications which expect to be packaged or compiled into /usr/local, so there are no guarantees it will work exactly as expected. I have found that Vim and Tmux work very well like this, even with Tmux linked against a home-compiled instance of libevent, on which it depends.

In particular, if any part of the install process requires root privileges, such as making a setuid binary, then things are likely not to work as expected.

Sync tmux panes

Posted on by

If you have a Tmux window divided into panes, you can use the synchronize-panes window option to send each pane the same keyboard input simultaneously:

Synchronize panes demo

You can do this by switching to the appropriate window, typing your Tmux prefix (commonly Ctrl-B or Ctrl-A) and then a colon to bring up a Tmux command line, and typing:

:setw synchronize-panes

You can optionally add on or off to specify which state you want; otherwise the option is simply toggled. This option is specific to one window, so it won’t change the way your other sessions or windows operate. When you’re done, toggle it off again by repeating the command.

This is an easy way to run interactive commands on multiple machines, perhaps to compare their speed or output, or if they have a similar setup a quick and dirty way to perform the same administrative tasks in parallel. It’s generally better practice to use Capistrano or Puppet for the latter.

Bash history expansion

Posted on by

Setting the Bash option histexpand allows some convenient typing shortcuts using Bash history expansion. The option can be set with either of these:

$ set -H
$ set -o histexpand

It’s likely that this option is already set for all interactive shells, as it’s on by default. The manual, man bash, describes these features as follows:

-H  Enable ! style history substitution. This option is on
    by default when the shell is interactive.

You may have come across this before, perhaps to your annoyance, in the following error message that comes up whenever ! is used in a double-quoted string, or without being escaped with a backslash:

$ echo "Hi, this is Tom!"
bash: !": event not found

If you don’t want the feature and thereby make ! into a normal character, it can be disabled with either of these:

$ set +H
$ set +o histexpand

History expansion is actually a very old feature of shells, having been available in csh before Bash usage became common.

This article is a good followup to Better Bash history, which among other things explains how to include dates and times in history output, as these examples do.

Basic history expansion

Perhaps the best known and most useful of these expansions is using !! to refer to the previous command. This allows repeating commands quickly, perhaps to monitor the progress of a long process, such as disk space being freed while deleting a large file:

$ rm big_file &
[1] 23608
$ du -sh .
3.9G    .
$ !!
du -sh .
3.3G    .

It can also be useful to specify the full filesystem path to programs that aren’t in your $PATH:

$ hdparm
-bash: hdparm: command not found
$ /sbin/!!
/sbin/hdparm

In each case, note that the command itself is printed as expanded, and then run to print the output on the following line.

History by absolute index

However, !! is actually a specific example of a more general form of history expansion. For example, you can supply the history item number of a specific command to repeat it, after looking it up with history:

$ history | grep expand
 3951  2012-08-16 15:58:53  set -o histexpand
$ !3951
set -o histexpand

You needn’t enter the !3951 on a line by itself; it can be included as any part of the command, for example to add a prefix like sudo:

$ sudo !3850

If you include the escape string \! as part of your Bash prompt, you can include the current command number in the prompt before the command, making repeating commands by index a lot easier as long as they’re still visible on the screen.

History by relative index

It’s also possible to refer to commands relative to the current command. To subtitute the second-to-last command, we can type !-2. For example, to check whether truncating a file with sed worked correctly:

$ wc -l bigfile.txt
267 bigfile.txt
$ printf '%s\n' '11,$d' w | ed -s bigfile.txt
$ !-2
wc -l bigfile.txt
10 bigfile.txt

This works further back into history, with !-3, !-4, and so on.

Expanding for historical arguments

In each of the above cases, we’re substituting for the whole command line. There are also ways to get specific tokens, or words, from the command if we want that. To get the first argument of a particular command in the history, use the !^ token:

$ touch a.txt b.txt c.txt
$ ls !^
ls a.txt
a.txt

To get the last argument, add !$:

$ touch a.txt b.txt c.txt
$ ls !$
ls c.txt
c.txt

To get all arguments (but not the command itself), use !*:

$ touch a.txt b.txt c.txt
$ ls !*
ls a.txt b.txt c.txt
a.txt  b.txt  c.txt

This last one is particularly handy when performing several operations on a group of files; we could run du and wc over them to get their size and character count, and then perhaps decide to delete them based on the output:

$ du a.txt b.txt c.txt
4164    a.txt
5184    b.txt
8356    c.txt
$ wc !*
wc a.txt b.txt c.txt
16689    94038  4250112 a.txt
20749   117100  5294592 b.txt
33190   188557  8539136 c.txt
70628   399695 18083840 total
$ rm !*
rm a.txt b.txt c.txt

These work not just for the preceding command in history, but also absolute and relative command numbers:

$ history 3
 3989  2012-08-16 16:30:59  wc -l b.txt
 3990  2012-08-16 16:31:05  du -sh c.txt
 3991  2012-08-16 16:31:12  history 3
$ echo !3989^
echo -l
-l
$ echo !3990$
echo c.txt
c.txt
$ echo !-1*
echo c.txt
c.txt

More generally, you can use the syntax !n:w to refer to any specific argument in a history item by number. In this case, the first word, usually a command or builtin, is word 0:

$ history | grep bash
 4073  2012-08-16 20:24:53  man bash
$ !4073:0
man
What manual page do you want?
$ !4073:1
bash

You can even select ranges of words by separating their indices with a hyphen:

$ history | grep apt-get
 3663  2012-08-15 17:01:30  sudo apt-get install gnome
$ !3663:0-1 purge !3663:3
sudo apt-get purge gnome

You can include ^ and $ as start and endpoints for these ranges, too. 3* is a shorthand for 3-$, meaning “all arguments from the third to the last.”

Expanding history by string

You can also refer to a previous command in the history that starts with a specific string with the syntax !string:

$ !echo
echo c.txt
c.txt
$ !history
history 3
 4011  2012-08-16 16:38:28  rm a.txt b.txt c.txt
 4012  2012-08-16 16:42:48  echo c.txt
 4013  2012-08-16 16:42:51  history 3

If you want to match any part of the command line, not just the start, you can use !?string?:

$ !?bash?
man bash

Be careful when using these, if you use them at all. By default it will run the most recent command matching the string immediately, with no prompting, so it might be a problem if it doesn’t match the command you expect.

Checking history expansions before running

If you’re paranoid about this, Bash allows you to audit the command as expanded before you enter it, with the histverify option:

$ shopt -s histverify
$ !rm
$ rm a.txt b.txt c.txt

This option works for any history expansion, and may be a good choice for more cautious administrators. It’s a good thing to add to one’s .bashrc if so.

If you don’t need this set all the time, but you do have reservations at some point about running a history command, you can arrange to print the command without running it by adding a :p suffix:

$ !rm:p
rm important-file

In this instance, the command was expanded, but thankfully not actually run.

Substituting strings in history expansions

To get really in-depth, you can also perform substitutions on arbitrary commands from the history with !!:gs/pattern/replacement/. This is getting pretty baroque even for Bash, but it’s possible you may find it useful at some point:

$ !!:gs/txt/mp3/
rm a.mp3 b.mp3 c.mp3

If you only want to replace the first occurrence, you can omit the g:

$ !!:s/txt/mp3/
rm a.mp3 b.txt c.txt

Stripping leading directories or trailing files

If you want to chop a filename off a long argument to work with the directory, you can do this by adding an :h suffix, kind of like a dirname call in Perl:

$ du -sh /home/tom/work/doc.txt
$ cd !$:h
cd /home/tom/work

To do the opposite, like a basename call in Perl, use :t:

$ ls /home/tom/work/doc.txt
$ document=!$:t
document=doc.txt

Stripping extensions or base names

A bit more esoteric, but still possibly useful; to strip a file’s extension, use :r:

$ vi /home/tom/work/doc.txt
$ stripext=!$:r
stripext=/home/tom/work/doc

To do the opposite, to get only the extension, use :e:

$ vi /home/tom/work/doc.txt
$ extonly=!$:e
extonly=.txt

Quoting history

If you’re performing substitution not to execute a command or fragment but to use it as a string, it’s likely you’ll want to quote it. For example, if you’ve just found through experiment and trial and error an ideal ffmpeg command line to accomplish some task, you might want to save it for later use by writing it to a script:

$ ffmpeg -f alsa -ac 2 -i hw:0,0 -f x11grab -r 30 -s 1600x900 \
> -i :0.0+1600,0 -acodec pcm_s16le -vcodec libx264 -preset ultrafast \
> -crf 0 -threads 0 "$(date +%Y%m%d%H%M%S)".mkv

To make sure all the escaping is done correctly, you can write the command into the file with the :q modifier:

$ echo '#!/usr/bin/env bash' >ffmpeg.sh
$ echo !ffmpeg:q >>ffmpeg.sh

In this case, this will prevent Bash from executing the command expansion "$(date ... )", instead writing it literally to the file as desired. If you build a lot of complex commands interactively that you later write to scripts once completed, this feature is really helpful and saves a lot of cutting and pasting.

Thanks to commenter Mihai Maruseac for pointing out a bug in the examples.

Avoiding the arrow keys

Posted on by

For shell users, moving to the arrow keys on the keyboard is something of an antipattern, moving away from the home row so central to touch typists. It therefore helps to find ways to avoid using the arrow keys in order to maintain your flow.

Bash

The arrow keys in Bash are used to move back and forth on the current command line (left and right), and up and down through the command history (up and down). This leads to the old shell user’s maxim:

“Those that don’t remember history are doomed to press Up repeatedly.”

There are alternatives to both functions. To move left or right by one character on the command line without deleting characters already placed, we can use Ctrl-B and Ctrl-F.

However, to make things a bit faster, there are four other key combinations to move back and forth on a line that are worth learning too:

  • Alt-B — Move back a word
  • Alt-F — Move forward a word
  • Ctrl-A — Move to the start of the line
  • Ctrl-E — Move to the end of the line

Similarly, to move up and down through history, we can use Ctrl-P and Ctrl-N respectively. Don’t forget that rather than keep tapping one of these, you can search backward or forward through history with Ctrl-R and Ctrl-S.

Whoops, I think I just taught you some Emacs.

Vim

To avoid the arrow keys in normal mode in Vim, use h, j, k, and l instead. This can take a little getting used to, but the benefits in terms of comfort for your hands and speed is easily worth it; using the arrow keys is one of the Vim anti-patterns.

If you’re asking “How do I avoid the arrow keys to move in insert mode?”, the answer is that you shouldn’t be moving in insert mode at all; it’s inefficient. When in insert mode you should be inserting text, and maybe using backspace or delete every now and then. To move through text, switching back to normal mode is vastly more efficient.

Moving in command mode is similar. If you need to move around on the command line, in most cases you should pull up the command window so that you can edit the command in normal mode, and then just tap Enter to run it.

In general, when you start thinking about moving through any kind of text, you should reflexively hit Esc or Ctrl-[ to go into normal mode, in order to take advantage of a whole keyboard’s worth of navigation shortcuts.

Vim annoyances

Posted on by

Like any highly interactive application, Vim has a few annoyances even for experienced users. Most of these consist of the editor doing something unexpected, particularly with motions or the behavior of regular expressions and registers. This is often due to vi and Vim being such venerable editors; their defaults sometimes make less sense decades after their creation.

Fortunately, Vim being the configurable editor that it is, many of the more common annoyances can be easily worked around. Bear in mind that not all of these are necessarily problems; if you actually prefer the way something behaves, you should stick with it.

Cursor jumps around while joining lines

If you want to keep the cursor in place when you join lines with J, you can do this, dropping a mark before the operation to which you return afterwards:

nnoremap J mzJ`z

Jumping lands on top or bottom of screen

If you’d like to center the window automatically around the cursor after jumping to a location with motions like n (next search pattern occurrence) or } (end of next paragraph), you can arrange that by remapping to add a zz after the motion:

nnoremap n nzz
nnoremap } }zz

If you don’t need the jump to land you in the exact middle of the screen, but just don’t want it to land right at the edge, you could also use scrolloff:

set scrolloff=10

Note that this also starts scrolling the window with single-line movements like j and k at this boundary, too, which you may not want.

Skipping lines when wrap is set

By default, the j and k keys don’t move by row; they move by line. This means that if you have the wrap option set, you might skip across several rows to reach the same point in an adjacent line.

This can be frustrating if you prefer the more intuitive behavior of moving up to the character immediately above the current one. If you don’t like this behavior, you can fix it by mapping j to gj, and k to gk, which moves by rows rather than lines:

nnoremap j gj
nnoremap k gk

If you think you might need the default behavior at some point, you might want to include reverse mappings so you can move linewise with gj and gk:

nnoremap gj j
nnoremap gk k

Wrapping breaks words

By default, setting wrap displays lines by breaking words if necessary. You can force it to preserve words instead by only breaking at certain characters:

set linebreak

You can define the characters at which Vim should be allowed to break lines with the breakat option. By default this includes spaces and tabs.

Backup files are a nuisance

If you’re developing with a version control system, you might find the in-place backups Vim keeps for saved files with the ~ suffix more annoying than useful. You can turn them off completely with nobackup:

set nobackup

Alternatively, you can set a single directory for them to keep them out of the way with backupdir:

set backupdir=~/.vim/backup

Swap files are a nuisance

Swap files can be similarly annoying, and unnecessary on systems with a lot of memory. If you don’t need them, you can turn them off completely:

set noswapfile

If you do find the swap files useful but want to prevent them cluttering your current directory, you can set a directory for them with directory:

set directory=~/.vim/swap

Accidentally hitting unwanted keys in normal mode

Some of the keys in normal mode bring up functions that aren’t terribly useful to a lot of people, and tend to accidentally get hit when Caps Lock is on, or when aiming for another key. Common nuisance keys are:

  • F1 for help; :help is generally more useful for the experienced user
  • Q to start ex mode; annoying when you intended to start recording a macro with q
  • K to bring up a man page for the word under the cursor; annoying when you’re not writing in C or shell script, and there isn’t a sensible choice of keywordprg for your chosen language

The simplest way to deal with these is to remap them to <nop> so that they don’t do anything:

nnoremap <F1> <nop>
nnoremap Q <nop>
nnoremap K <nop>

Startup message is irritating

If you don’t like the startup message that appears when you open Vim without a filename, you can remove it by adding the I flag to shortmess:

set shortmess+=I

Can’t backspace past start of operation

If you’re in insert mode, by default you can’t use backspace to delete text past the start of the insert operation; that is, you can’t backspace over where you first pressed insert. This is old vi behavior, and if you don’t like it, you can make backspace work everywhere instead:

set backspace=indent,eol,start

Flashing screen is annoying

If you don’t require the error feedback, you can turn off the flashing screen for the “visual bell”:

set visualbell t_vb=

Don’t know what mode I’m in

If you lose track of the current mode, you can get a convenient --INSERT-- indicator at the bottom of the screen with:

set showmode

If you’re having this problem a lot, however, you might want to take stock of how much time you’re spending in insert mode when you’re not actively typing; it’s good Vim practice to stay out of it otherwise, even when you stop to think.

Keep making typos of common commands

If you’re fat-fingering :wq and similar commands a lot due to a sticky shift key, take a look at the Vim command typos fix. Also, don’t forget about ZZ and ZQ as quicker alternatives to :wq and :q!.

Don’t want case sensitive patterns

If you don’t care for case-sensitive searches and substitutions, you can turn it off completely:

set ignorecase

You might find the slightly cleverer behavior of smartcase is better, though. This option only applies case sensitivity if at least one of the letters in the pattern is uppercase; otherwise, case is ignored.

set smartcase

Doesn’t replace all occurrences

The default behavior for the substitute operation, :s/pattern/replacement/, is to replace only the first occurrence of the pattern. To make it replace all occurrences on the appropriate lines, you add the g suffix. If you find that you never need to substitute for only the first occurrence of a pattern in a line, you can add that flag to the patterns by default:

set gdefault

If you do happen to need to replace only the first occurrence one day, you can get the default behavior back by adding the g suffix to your pattern; its meaning is reversed by this option.

Can’t move into blank space in visual block mode

If you need to define a block in visual block mode with bounds outside the actual text (that is, past the end of lines), you can allow this with:

set virtualedit=block

This will let you move around the screen freely while in visual block mode to define your selections. As an example, this can make selecting the contents of the last column in a formatted table much easier.

Filename completion on command line is confusing

If you’re used to the behavior of shell autocomplete functions for completing filenames, you can emulate it in Vim’s command mode:

set wildmode=longest,list

With this set, the first Tab press (or whatever your wildchar is set to) will expand a filename or command in command mode to the longest common string it can, and a second press will display a list of all possible completions above the command line.

Yanking lines is inconsistent

D deletes from the cursor to the end of the line; C changes from the cursor to the end of the line. For some reason, however, Y yanks the entire line, both before and after the cursor. If this inconsistency bothers you, you can fix it with a remapping:

nnoremap Y y$

New splits appear in unintuitive places

If your language of choice is read from left to right, you may find it annoying that by default Vim places new vertical splits to the left of the current pane, and horizontal splits above the current pane. You can fix both:

set splitbelow
set splitright

Caps Lock sends Vim crazy

Caps Lock in normal mode makes Vim go haywire. This isn’t really Vim’s fault; Caps Lock is generally a pretty useless key. It’s often useful for Vim users to remap it, so that it sends the same signal as a more useful key; Control and Escape are common choices. You might even find this reduces strain on your hands.

Several of these fixes were inspired by Steve Losh’s .vimrc file. Thanks also to commenters Alpha Chen and Rob Hoelz for suggestions.

Vim misconceptions

Posted on by

Vim isn’t the best tool for every task, and there’s no reason you shouldn’t stick to your GUI IDE if you know it like the back of your hand and are highly productive in it. The very basic best practices for text editing in general apply just as well to more familiar editing interfaces as they do to Vim, so nobody should be telling you that Vim is right for everyone and everything and that you’re wrong not to use it.

However, because Vim and vi-like editors in general have a lot of trouble shaking off the connotations of their serverside, terminal-only, mouseless past, there are a few persistent objections to even trying Vim that seem to keep cropping up. If you’re someone curious about Vim but you heard it was useless for any of the following reasons, or if you’re an experienced user looking to convince a hesitant neophyte to try Vim, the following list might clear a few things up.

Vim takes too long to learn

If you want analogues to all of the features in your IDE, that would likely take some time, just as it would in any other new editor. However, if all you need to start is to be able to enter text, move around, and load and save files, you just need to know:

  • i to enter insert mode, Esc to leave it
  • Arrow keys to move around in both modes
  • :e <filename> to load a document
  • :w <filename> to save a document
  • :q to quit, :q! to ignore unsaved changes

To do pretty much everything Windows Notepad would let you do, on top of that you’d only need to learn:

  • dd to cut a line
  • yy to copy a line
  • p to paste a line
  • /<pattern> to search for text
  • n to go to the next result
  • :s/pattern/replacement to replace text

From that point, you only get faster as you learn how to do new things. So saying that Vim takes weeks to learn is a bit disingenuous when the essentials can easily be mastered with a few minutes’ practice.

Granted, the arrow keys are a bit of an anti-pattern, but you can worry about that later.

Vim has no GUI

Vim has a GUI version called Gvim for both Windows and Unix. For Mac OS X, the MacVim port is preferred. For experienced users the GUI’s only real advantage over the terminal version is a wider color palette, but it has a toolbar and other GUI elements which some new users may find useful.

Vim doesn’t have spell checking

Vim allows spell checking with system dictionaries, using :set spell and :set spelllang. Misspelt and unknown words are highlighted appropriately.

Vim doesn’t have syntax highlighting

Vim has support for syntax highlighting that can be turned on with :syntax enable, for a very wide variety of programming languages, markup languages, and configuration file syntaxes.

Vim only allows eight colours

This is a limitation of terminal emulators rather than of Vim itself, but most modern GUI terminal emulators allow 256 colours anyway with a little extra setup. The GUI version, Gvim, has full color support with the familiar rrggbb color definitions.

Vim doesn’t have class/function folding

Vim does in fact have support for folding, based on both code structure and manually defined boundaries. See :help folding for details.

Vim doesn’t have autocompletion

Vim allows basic completion using words already in the current buffer, and also more advanced omnicompletion using language dictionaries for constants, variables, classes, and functions.

Vim doesn’t have a file browser

Vim has had the Netrw plugin bundled for some time, which provides a pretty capable filesystem browser.

Vim can’t do network editing

Again, the bundled Netrw plugin handles this. Editing files over FTP and SCP links is pretty transparent. You can open a file on a remote server by entering the following, which will prompt for your username and password:

:e ftp://ftp.example.com/index.html

When you’re done editing, you just enter :w to save the file, and it’s automatically uploaded for you. You can record your FTP credentials in a .netrc file to save having to type in usernames and passwords all the time. URIs with scp:// work the same way; with a good public key infrastructure set up, you can use Vim quite freely as a network-transparent editor.

Vim doesn’t have tabs or windows

The model Vim uses for tabs and windows is rather different from most GUI editors, but both are supported and have been for some time.

Vim has too many modes

It has three major modes: normal mode, insert mode, and command mode. There’s a fourth that’s Vim-specific, called visual mode, for selecting text.

Most editors are modal in at least some respect; when you bring up a dialog box, or enter a prefix key to another command, you’re effectively changing modes. The only real difference is that context shifts in Vim are at first less obvious; the screen doesn’t look too different between normal, insert, and command mode.

The showmode option helps to distinguish between insert and normal mode, a common frustration for beginners. This gets easier when you get into the habit of staying out of insert mode when not actually entering text.

Vim doesn’t work with the mouse

Vim works fine with the mouse, in both Gvim and xterm-like terminal emulators, if you really want it. You can change the position of the cursor, scroll through the document, and select text as normal. Setting the below generally does the trick:

:set mouse=a

However, even a little experience in Vim may show you that you don’t need the mouse as much as you think. Careful use of the keyboard allows much more speed and precision, and it’s quite easy to run a complex editing session without even moving from the keyboard’s “home row”, let alone all the way over to the mouse.

Vim doesn’t do Unicode

Vim supports Unicode encodings with the encoding option. It’s likely you’ll only need to put the below in your .vimrc file and then never really think about encoding in your editor again:

:set encoding=utf-8

Vim isn’t being developed or maintained

The original author of Vim, and its current maintainer and release manager, is Bram Moolenaar. At the time of writing, he is working for Google, and is paid to spend some of his time developing Vim. The development mailing list for Vim is very active, and patches are submitted and applied to the publically accessible Mercurial repository on a regular basis. Vim is not a dead project.

Vim is closed source

Vim isn’t proprietary or closed source, and never has been. It uses its own GPL-compatible license called the Vim license.

The original vi used to be proprietary because Bill Joy based the code on the classic UNIX text editor ed, but its source code has now been released under a BSD-style license.

Vim eval feature

Posted on by

Using your .vimrc file on many machines with different versions and feature sets for Vim is generally not too much of a problem if you apply a little care in making a gracefully degrading .vimrc. In most cases, using the output of vim --version and checking the help files will tell you what features to check in order to determine which parts of your .vimrc configuration to load, and which to ignore.

There’s one particular feature that’s less obvious, though, and that’s eval. Vim’s help describes it like this in :help +eval:

N  *+eval*      expression evaluation |eval.txt|

The eval.txt document, in turn, describes the syntax for various features fundamental to Vimscript, including variables, functions, lists, and dictionaries.

All this makes eval perhaps the most central feature of Vim. Without it, Vim doesn’t have much of its programmable nature, and remains not much more than classic vi. If your particular build of Vim doesn’t include it, then Vimscript essentially does not work as a programming language, and elementary calls like function and let will throw errors:

E319: Sorry, the command is not available in this version: function
E319: Sorry, the command is not available in this version: let

If you’re getting this kind of error, you’re probably using a very stripped-down version of Vim that doesn’t include eval. If you just want to prevent the error by ignoring a block of code if the feature is unavailable, you can do this with has():

if has("eval")
    ...
endif

Vim will still be perfectly functional as a vi-style editor without +eval, but if you’re going to need any Vimscript at all, you should recompile Vim with a --with-features value for the ./configure line that does include it, such as normal, big, or huge, or install a more fully-featured packaged version. For example, on Debian-like systems, the vim-tiny package that is included in the netinst system does not include eval, but the vim and vim-runtime packages do.

Inspecting Vim’s source, in particular the ex_docmd.c file, gives some indication of how fundamental this feature is, applying the C function ex_ni which simply prints the “not available” error shown above to a large number of control structures and statements if the FEAT_EVAL constant is not defined:

#ifndef FEAT_EVAL
# define ex_scriptnames     ex_ni
# define ex_finish      ex_ni
# define ex_echo        ex_ni
# define ex_echohl      ex_ni
# define ex_execute     ex_ni
# define ex_call        ex_ni
# define ex_if          ex_ni
# define ex_endif       ex_ni
# define ex_else        ex_ni
# define ex_while       ex_ni
# define ex_continue        ex_ni
# define ex_break       ex_ni
# define ex_endwhile        ex_ni
# define ex_throw       ex_ni
# define ex_try         ex_ni
# define ex_catch       ex_ni
# define ex_finally     ex_ni
# define ex_endtry      ex_ni
# define ex_endfunction     ex_ni
# define ex_let         ex_ni
# define ex_unlet       ex_ni
# define ex_lockvar     ex_ni
# define ex_unlockvar       ex_ni
# define ex_function        ex_ni
# define ex_delfunction     ex_ni
# define ex_return      ex_ni
# define ex_oldfiles        ex_ni
#endif

Bash prompts

Posted on by

You can tell a lot about a shell user by looking at their prompt. Most shell users will use whatever the system’s default prompt is for their entire career. Under many GNU/Linux distributions, this prompt includes the username, the hostname, and the current working directory, along with a $ sigil for regular users, and a # for root.

tom@sanctum:~$

This format works well for most users, and it’s so common that it’s often used in tutorials to represent the user’s prompt in lieu of the single-character standard $ and #. It shows up as the basis for many of the custom prompt setups you’ll find online.

Some users may have made the modest steps of perhaps adding some brackets around the prompt, or adding the time or date:

[10:11][tom@sanctum:~]$

Still others make the prompt into a grand multi-line report of their shell’s state, or indeed their entire system, full of sound and fury and signifying way too much:

-(23:38:57)-(4 users)-(0.00, 0.02, 0.05)-
-(Linux sanctum 3.2.0-3-amd64 x86_64 GNU/Linux)-
[tom@sanctum:/dev/pts/2:/home/tom]{2}!?[MAIL]$

Then there are the BSD users and other minimalist contrarians who can’t abide anything more than a single character cluttering their screens:

$

Getting your prompt to work right isn’t simply a cosmetic thing, however. If done right and in a style consistent with how you work, it can provide valuable feedback on how the system is reacting to what you’re doing with it, and also provide you with visual cues that could save you a lot of confusion — or even prevent your making costly mistakes.

I’ll be working in Bash. Many of these principles work well for Zsh as well, but Zsh users might want to read Steve Losh’s article on his prompt setup first to get an idea of the syntactical differences.

Why is this important?

One of the primary differences between terminals and windowed applications is that there tends to be much less emphasis on showing metadata about the running program and its environment. For the most part, to get information about what’s going on, you need to request it with calls like pwd, whoami, readlink, env, jobs, and echo $?.

The habit of keeping metadata out of the terminal like this unless requested may appeal to a sense of minimalism, but like a lot of things in the command line world, it’s partly the olden days holding us back. When mainframes and other dinosaurs roamed the earth, terminal screens were small, and a long or complex prompt amounted to a waste of space and cycles (or even paper). In our futuristic 64-bit world, where tiny CRTs are a thing of the past and we think nothing of throwing gigabytes of RAM at a single process, this is much less of a problem.

Customising the prompt doesn’t have many “best practices” to it like a lot of things in the shell; there isn’t really a right way to do it. Some users will insist it’s valuable to have the path to the terminal device before every prompt, which I think is crazy as I almost never need that information, and when I do, I can get it with a call to tty. Others will suggest that including the working directory makes the length of the prompt too variable and distracting, whereas I would struggle to work without it. It’s worth the effort to design a prompt that reflects your working style, and includes the information that’s consistently important to you on the system.

So, don’t feel like you’re wasting time setting up your prompt to get it just right, and don’t be so readily impressed by people with gargantuan prompts in illegible rainbow colors that they copied off some no-name wiki. If you use Bash a lot, you’ll be staring at your prompt for several hours a day. You may as well spend an hour or two to make it useful, or at least easier on the eyes.

The basics

The primary Bash prompt is stored in the variable PS1. Its main function is to signal to the user that the shell is done with its latest foreground task, and is ready to accept input. PS1 takes the form of a string that can contain escape sequences for certain common prompt elements like usernames and hostnames, and which is evaluated at printing time for escape sequences, variables, and command or function calls within it.

You can inspect your current prompt definition with echo. If you’re using GNU/Linux, it will probably look similar to this:

tom@sanctum:~$ echo $PS1
\u@\h:\w\$

PS1 works like any other Bash variable; you can assign it a new value, append to it, and apply substitutions to it. To add the text bash to the start of my prompt, I could do this:

tom@sanctum:~$ PS1=bash-"$PS1"
bash-tom@sanctum:~$

There are other prompts — PS2 is used for “continued line” prompts, such as when you start writing a for loop and continue another part of it on a new line, or terminate your previous line with a backslash to denote the next line continues on from it. By default this prompt is often a right angle bracket >:

$ for f in a b c; do
>   do echo "$f"
> done

There are a couple more prompt strings which are much more rarely seen. PS3 is used by the select structure in shell scripts; PS4 is used while tracing output with set -x. Ramesh Natarajan breaks this down very well in his article on the Bash prompt. Personally, I only ever modify PS1, because I see the other prompts so rarely that the defaults work fine for me.

As PS1 is a property of interactive shells, it makes the most sense to put its definitions in $HOME/.bashrc on GNU/Linux or BSD systems. It’s likely there’s already code in there doing just that. For Mac OS X, you’ll likely need to put it into $HOME/.bash_profile instead.

Escapes

Bash will perform some substitutions for you in the first pass of its evaluation of your prompt string, replacing escape sequences with appropriate elements of information that are often useful in prompts. Below are the escape sequences as taken from the Bash man page:

\a     an ASCII bell character (07)
\d     the date in "Weekday Month Date" format (e.g., "Tue May 26")
\D{format}
       the format is passed to strftime(3) and the result is inserted into
       the prompt string; an empty format results in a locale-specific time
       representation. The braces are required
\e     an ASCII escape character (033)
\h     the hostname up to the first `.'
\H     the hostname
\j     the number of jobs currently managed by the shell
\l     the basename of the shell's terminal device name
\n     newline
\r     carriage return
\s     the name of the shell, the basename of $0 (the portion following
       the final slash)
\t     the current time in 24-hour HH:MM:SS format
\T     the current time in 12-hour HH:MM:SS format
\@     the current time in 12-hour am/pm format
\A     the current time in 24-hour HH:MM format
\u     the username of the current user
\v     the version of bash (e.g., 2.00)
\V     the release of bash, version + patch level (e.g., 2.00.0)
\w     the current working directory, with $HOME abbreviated with a tilde
       (uses the value of the PROMPT_DIRTRIM variable)
\W     the basename of the current working directory, with $HOME
       abbreviated with a tilde
\!     the history number of this command
\#     the command number of this command
\$     if the effective UID is 0, a #, otherwise a $
\nnn   the character corresponding to the octal number nnn
\\     a backslash
\[     begin a sequence of non-printing characters, which could be used to
       embed a terminal control sequence into the prompt
\]     end a sequence of non-printing characters

As an example, the default PS1 definition on many GNU/Linux systems uses four of these codes:

\u@\h:\w\$

The \! escape sequence, which expands to the history item number of the current command, is worth a look in particular. Including this in your prompt allows you to quickly refer to other commands you may have entered in your current session by history number with a ! prefix, without having to actually consult history:

$ PS1="(\!)\$"
(6705)$ echo "test"
test
(6706)$ !6705
echo "test"
test

Keep in mind that you can refer to history commands relatively rather than by absolute number. The previous command can be referenced by using !! or !-1, and the one before that by !-2, and so on, provided that you have left the histexpand option on.

Also of particular interest are the delimiters \[ and \]. You can use these to delimit sections of “non-printing characters”, typically things like terminal control characters to do things like moving the cursor around, starting to print in another color, or changing properties of the terminal emulator such as the title of the window. Placing these non-printing characters within these delimiters prevents the terminal from assuming an actual character has been printed, and hence correctly manages things like the user deleting characters on the command line.

Colors

There are two schools of thought on the use of color in prompts, very much a matter of opinion:

  • Focus should be on the output of commands, not on the prompt, and using color in the prompt makes it distracting, particularly when output from some programs may also be in color.
  • The prompt standing out from actual program output is a good thing, and judicious use of color conveys useful information rather than being purely decorative.

If you agree more with the first opinion than the second, you can probably just skip to the next section of this article.

Printing text in color in terminals is done with escape sequences, prefixed with the \e character as above. Within one escape sequence, the foreground, background, and any styles for the characters that follow can be set.

For example, to include only your username in red text as your prompt, you might use the following PS1 definition:

PS1='\[\e[31m\]\u\[\e[0m\]'

Broken down, the opening sequence works as follows:

  • \[ — Open a series of non-printing characters, in this case, an escape sequence.
  • \e — An ASCII escape character, used to confer special meaning to the characters that follow it, normally to control the terminal rather than to enter any characters.
  • [31m — Defines a display attribute, corresponding to foreground red, for the text to follow; opened with a [ character and terminated with an m.
  • \] — Close the series of non-printing characters.

The terminating sequence is very similar, except it uses the 0 display attribute to reset the text that follows back to the terminal defaults.

The valid display attributes for 16-color terminals are:

  • Style
    • 0 — Reset all attributes
    • 1 — Bright
    • 2 — Dim
    • 4 — Underscore
    • 5 — Blink
    • 7 — Reverse
    • 8 — Hidden
  • Foreground
    • 30 — Black
    • 31 — Red
    • 32 — Green
    • 33 — Yellow
    • 34 — Blue
    • 35 — Magenta
    • 36 — Cyan
    • 37 — White
  • Background
    • 40 — Black
    • 41 — Red
    • 42 — Green
    • 43 — Yellow
    • 44 — Blue
    • 45 — Magenta
    • 46 — Cyan
    • 47 — White

Note that blink may not work on a lot of terminals, often by design, as it tends to be unduly distracting. It’ll work on a linux console, though. Also keep in mind that for a lot of terminal emulators, by default “bright” colors are printed in bold.

More than one of these display attributes can be applied in one hit by separating them with semicolons. The below will give you underlined white text on a blue background, after first resetting any existing attributes:

\[\e[0;4;37;44m\]

It’s helpful to think of these awkward syntaxes in terms of opening and closing, rather like HTML tags, so that you open before and then reset after all the attributes of a section of styled text. This usually amounts to printing \[\e[0m\] after each such passage.

These sequences are annoying to read and to type, so it helps to put commonly used styles into variables:

color_red='\[\e[31m\]'
color_blue='\[\e[34m\]'
color_reset='\[\e[0m\]'

PS1=${color_red}'\u'${color_reset}'@'${color_blue}'\h'${color_reset}

A more robust method is to use the tput program to print the codes for you, and put the results into variables:

color_red=$(tput setaf 1)
color_blue=$(tput setaf 4)

This may be preferable if you use some esoteric terminal types, as it queries the terminfo or termcap databases to generate the appropriate escapes for your terminal.

The set of eight colors — or sixteen, depending on how you look at it — can be extended to 256 colors in most modern terminal emulators with a little extra setup. The escape sequences are not very different, but their range is vastly increased to accommodate the larger palette.

A useful application for prompt color can be a quick visual indication of the privileges available to the current user. The following code changes the prompt to green text for a normal user, red for root, and yellow for any other user attained via sudo:

color_root=$(tput setaf 1)
color_user=$(tput setaf 2)
color_sudo=$(tput setaf 3)
color_reset=$(tput sgr0)

if (( EUID == 0 )); then
    PS1="\\[$color_root\\]$PS1\\[$color_reset\\]"
elif [[ $SUDO_USER ]]; then
    PS1="\\[$color_sudo\\]$PS1\\[$color_reset\\]"
else
    PS1="\\[$color_user\\]$PS1\\[$color_reset\\]"
fi

Otherwise, you could simply use color to make different regions of your prompt more or less visually prominent.

Variables

Provided the promptvars option is enabled, you can reference Bash variables like $? to get the return value of the previous command in your prompt string. Some people find this valuable to show error codes in the prompt when the previous command given fails:

$ shopt -s promptvars
$ PS1='$? \$'
1 $ echo "test"
test
0 $ fail
-bash: fail: command not found
127 $

The usual environment variables like USER and HOME will work too, though it’s preferable to use the escape sequences above for those particular variables. Note that it’s important to quote the variable references as well, so that they’re evaluated at runtime, and not during the assignment itself.

Commands

If the information you want in your prompt isn’t covered by any of the escape codes, you can include the output of commands in your prompt as well, using command substitution with the syntax $(command):

$ PS1='[$(uptime) ]\$ '
[ 01:21:59 up 7 days, 13:04,  7 users, load average: 0.30, 0.40, 0.36 ]$

This requires the promptvars option to be set, the same way variables in the prompt do. Again, note that you need to use single quotes so that the command is run when the prompt is being formed, and not immediately as part of the assignment.

This can include pipes to format the data with tools like awk or cut, if you only need a certain part of the output:

$ PS1='[$(uptime | cut -d: -f5) ]\$ '
[ 0.36, 0.34, 0.34 ]$

For clarity, during prompt setup in .bashrc, it makes sense to use functions instead for reasonably complex commands:

prompt_load() {
    uptime | cut -d: -f5
}
PS1='[$(prompt_load) ]'$PS1

Note that as a normal part of command substitution, trailing newlines are stripped from the output of the command, so here the output of uptime appears on a single line.

A common usage of this pattern is showing metadata about the current directory, particularly if it happens to be a version control repository or working copy; you can use this syntax with functions to show the type of version control system, the current branch, and whether any changes require committing. Working with Git, Mercurial, and Subversion most often, I include the relevant logic as part of my prompt function.

When appended to my PS1 string, $(prompt vcs) gives me prompts that look like the following when I’m in directories running under the appropriate VCS. The exclamation marks denote that there are uncommitted changes in the repositories.

[tom@conan:~/.dotfiles](git:master!)$
[tom@conan:~/Build/tmux](svn:trunk)$
[tom@conan:~/Build/vim](hg:default!)$

In general, where this really shines is adding pieces to your prompt conditionally, to make them collapsible. Certain sections of your prompt therefore only show up if they’re relevant. This snippet, for example, prints the number of jobs running in the background of the current interactive shell in curly brackets, but only if the count is non-zero:

prompt_jobs() {
    local jobc
    while read -r _; do
        ((jobc++))
    done < <(jobs -p)
    if ((jobc > 0)); then
        printf '{%d}' "$jobc"
    fi
}

It’s important to make sure that none of what your prompt does takes too long to run; an unresponsive prompt can make your terminal sessions feel very clunky.

Note that you can also arrange to run a set of commands before the prompt is evaluated and printed, using the PROMPT_COMMAND. This tends to be a good place to put commands that don’t actually print anything, but that do need to be run before or after each command, such as operations working with history:

PROMPT_COMMAND='history -a'

Switching

If you have a very elaborate or perhaps even computationally expensive prompt, it may occasionally be necessary to turn it off to revert to a simpler one. I like to handle this by using functions, one of which sets up my usual prompt and is run by default, and another which changes the prompt to the minimal $ or # character so often used in terminal demonstrations. Something like the below works well:

prompt_on() {
    PS1="$color_prompt"'\u@\h:\w'"$(prompt_jobs)"'\$'"${color_reset}"' '
}
prompt_off() {
    PS1='\$'
}
prompt_on

You can then switch your prompt whenever you need to by typing prompt_on and prompt_off.

This can also be very useful if you want to copy text from your terminal into documentation, or into an IM or email message; it removes your distracting prompt from the text, where it would otherwise almost certainly differ from that of the user following your instructions. This is also occasionally helpful if your prompt does not work on a particular machine, or the machine is suffering a very high load average that means your prompt is too slow to load.

Further reading

Predefined prompt strings are all over the web, but the above will hopefully enable you to dissect what they’re actually doing more easily and design your own. To take a look at some examples, the relevant page on the Arch Wiki is a great start. Ramesh Natarajan over at The Geek Stuff has a great article with some examples as well, with the curious theme of making your prompt as well-equipped as Angelina Jolie.

Finally, please feel free to share your prompt setup in the comments (whether you’re a Bash user or not). It would be particularly welcome if you explain why certain sections of your prompt are so useful to you for your particular work.

Actually using ed

Posted on by

The classic ed editor is a really good example of a sparse, minimal, standard Unix tool that does one thing, and does it well. Because there are so many good screen-oriented editors for Unix, there’s seldom very much call for using ed, unless you’re working on very old or very limited hardware that won’t run anything else.

However, if part of the reason you use vi is because you think it will always be there (it may not be), then you should learn ed too. If your terminal is broken and neither vi nor nano will work, or you break it some other way, your choices may well be between cat and ed. Or even the grand heresy of sed -i…

Not a friendly editor

Even more than its uppity grandchild ex/vi, ed has developed a reputation as terse and intimidating to newcomers. When you type ed at the command line, nothing happens, and the only error message presented by default is ?. If you’re reading this, it’s likely your first and only experience with ed went something like this:

$ ed
help
?
h
Invalid command suffix
?
?
^C
?
exit
?
quit
?
^Z
$ killall ed
$ vi

So, ed is not a terribly intuitive editor. However, it’s not nearly as hard to learn as it might seem, especially if you’re a veteran vi user and thereby comfortable with the ex command set. With a little practice, you can actually get rather quick with it; there’s an elegance to its almost brutal simplicity.

It’s also very interesting to learn how ed works and how to use it, not just because it might very well be useful for you one day, but because it occupies an important position in the heritage of the sed stream editor, the ex line editor, the vi visual editor, the grep tool, and many other contexts.

Why is ed so terse?

When ed was developed, the usual method of accessing a Unix system was via a teletype device, on which it wouldn’t have been possible to use a screen-oriented editor like vi. Similarly, modems were slow, and memory was precious; using abbreviated commands and terse error messages made a lot of sense, because the user would otherwise be wasting a lot of time waiting for the terminal to react to commands, and didn’t have a whole lot of memory to throw around for anything besides the buffer of text itself.

Of course, this is almost a non-issue for most Unix-like systems nowadays, so one of the first things we’ll do is make ed a little bit less terse and more user-friendly.

Error messages

Start ed up the usual way:

$ ed

We’ll start by deliberately doing something wrong. Type b and press Enter:

b
?

There’s that tremendously unhelpful ? again. But if you press h, you can see what went wrong:

h
Unknown command

Of course, since it’s the future now, we can spare the terminal cycles to have ed print the error message for us every time. You can set this up by pressing H:

H
b
?
Unknown command

That’s a bit more useful, and should make things easier.

Quitting

You can quit ed with q. Go ahead and do that. If you had unsaved changes in a buffer, you could type Q to quit unconditionally. Repeating yourself works too:

q
?
Warning: buffer modified
q

Command prompt

Let’s invoke ed again, but this time we’ll use the -p option to specify a command prompt:

$ ed -p\*
*

We’ll use that from now on, which will make things clearer both for interpreting this tutorial and for remembering whether we’re in command mode, or entering text. It might even be a good idea to make it into a function:

$ ed() { command ed -p\* "$@" ; }

Basic text input

We’ll start by adding a couple of lines of text to the new buffer. When you start ed with no filename, it starts an empty buffer for you, much like vi does.

Because there are no lines at all at present, press a to start adding some at the editor’s current position:

*a
Start typing a few lines.
Anything you want, really.
Just go right ahead.
When you're done, just type a period by itself.
.

That’s added four new lines to the buffer, and left the editor on line 4. You can tell that’s the case because typing p for print, just by itself, prints the fourth line:

*p
When you're done, just type a period by itself.

A little more useful is n (or pn in some versions), which will show both the line number and the contents of the line:

*n
4       When you're done, just type a period by itself.

So just like in ex, the current line is the default for most commands. You can make this explicit by referring to the current line as .:

*.n
4       When you're done, just type a period by itself.

You can move to a line just by typing its number, which will also print it as a side effect:

*3
Just go right ahead.
*.n
3       Just go right ahead.

Pressing a like you did before will start inserting lines after the current line:

*a
Entering another line.
.
*n
4       Entering another line.

Pressing i will allow you to insert lines before the current line:

*i
A line before the current line.
.
*n
4       A line before the current line.

You can replace a line with c:

*c
I decided I like this line better.
.
*n
4       I decided I like this line better.

You can delete lines with d:

*6d

And join two or more lines together with j:

*1,2j

You can prepend an actual line number to any of these commands to move to that line before running the command on it:

*1c
Here's a replacement line.
.
*1n
1       Here's a replacement line.

For most of these commands, the last line to be changed will become the new current line.

Ranges

You can select the entire buffer with 1,$ or , for short (% works too):

*,p
Here's a replacement line.
Just go right ahead.
I decided I liked this line better.
Entering another line.

Or a limited range of specific lines:

*2,3p
Just go right ahead.
I decided I liked this line better.

These ranges can include a reference to the current line with .:

*2
Just go right ahead.
*.,4p
Just go right ahead.
I decided I liked this line better.
Entering another line.

They can also include relative line numbers, prefixed with + or -:

*2
*-1,+1p
Here's a replacement line.
Just go right ahead.
I decided I liked this line better.

You can drop a mark on a line with k followed by a lowercase letter such as a, and you’re then able to refer to it in ranges as 'a:

*3ka
*'ap
I decided I liked this line better.

Moving and copying

Move a line or range of lines to after a target line with m:

*1,2m$
*,p
I decided I liked this line better.
Entering another line.
Here's a replacement line.
Just go right ahead.

Copy lines to after a target line with t:

*2t4
*,p
I decided I liked this line better.
Entering another line.
Here's a replacement line.
Just go right ahead.
Entering another line.

Regular expressions

You can select lines based on classic regular expressions with the g operator. To print all lines matching the regular expression /re/:

*g/re/p
Here's a replacement line.

(Hmm, where have I seen that command before?)

You can invert the match to work with lines that don’t match the expression with v:

*v/re/p
I decided I liked this line better.
Entering another line.
Just go right ahead.
Entering another line.

Just like numbered line ranges, ranges selected with regular expressions can have other operations applied to them. To move every line containing the expression /re/ to the bottom of the file:

*g/re/m$
*,p
I decided I liked this line better.
Entering another line.
Just go right ahead.
Entering another line.
Here's a replacement line.

Searching

You can move to the next line after the current one matching a regular expression with /. Again, this will print the line’s contents as a side effect.

*/like
I decided I like this line better.

You can search backward with ?:

*?Here
Here's a replacement line.

Substituting

You can substitute for the first occurrence per line of an expression within a range of lines with the s command:

*1s/i/j
I decjded I like this line better.

You can substitute for all the matches on each line by adding the /g suffix:

*1s/i/j/g
*p
I decjded I ljke thjs ljne better.

Reading and writing

You can write the current buffer to a file with w, which will also print the total number of bytes written:

*w ed.txt
129

Having done this once, you can omit the filename for the rest of the session:

*w
129

Like most ed commands, w can be prefixed with a range to write only a subset of lines to the file. This would write lines 1 to 4 to the file ed.txt:

*1,4w ed.txt
102

You can use W to append to a file, rather than replace it. This would write lines 3 to 5 to the end of the file ed.txt:

*3,5W
71

You can read in the contents of another file after the current line (or any other line) with r. Again, this will print the number of bytes read.

*r /etc/hosts
205

The output of a command can be included by prefixing it with !:

*r !ps -e
5571

If you just want to load the contents of a file or the output of a command into the buffer, replacing its contents, use e, or E if you’ve got a modified buffer and don’t mind replacing it:

*E ed.txt
173

If you don’t like seeing the byte counts each time, you can start ed with the -s option for “quiet mode”.

Familiar syntax

Almost all of the above command sets will actually be familiar to vi users who know a little about ex, or Vimscript in Vim. It will also be familiar to those who have used sed at least occasionally.

Once you get good with ed, it’s possible you’ll find yourself using it now and then to make quick edits to files, even on systems where your favourite screen-based editor will load promptly. You could even try using ex.

Update 2016: Recent Hacker News discussion has reminded me of my long-standing mistake in asserting that ed(1) is included in every Unix and Unix-like system’s base installation. This is not even close to true–many others exclude it–and the claim has been removed, which I should have done years ago.

Thanks to A. Pedro Cunha for a couple of fixes to show correct typical output.

Terminal colour tolerance

Posted on by

Using 256 colour terminals with applications like Vim and tmux is pretty much a no-brainer; it’s been well-supported on most GUI terminal emulators for ages, including Windows emulators like PuTTY, and gives you a much wider spectrum of colour with which to work and to apply useful features like syntax highlighting and contextual colours for shell prompts.

The problem is that not all terminals are created equal. Occasionally, you’ll find yourself needing to use a console, perhaps with the linux type, when your X server doesn’t start or you’re using a KVM. Following the principles of graceful degradation, it’s a good idea to arrange your terminal configuration and any other applications that normally take advantage of the wider 256 colour spectrum to start up normally with no errors, and to use sensible values instead as supported by the terminal.

Bash

You can check the number of colours supported by your $TERM setting using the tput command:

$ tput colors
256

On systems using termcap, the syntax is slightly different:

$ tput Co
256

This provides you with an accessible way to set up graceful degradation for any colours that you use in your .bashrc file or other startup scripts. By way of example, in my .bashrc file I include the following stanza that adjusts the colour of my prompt depending on whether I have root privileges:

if ((EUID == 0)); then
    PS1=${color_root}${PS1}${color_undo}
else
    PS1=${color_user}${PS1}${color_undo}
fi

The variables contain the terminal escape codes used to start painting text in appropriate colours for the terminal type. I set these earlier on in the file by checking the output of tput colors:

colors=$(tput colors)

# Terminal supports 256 colours
if ((colors >= 256)); then
    color_root='\[\e[38;5;9m\]'
    color_user='\[\e[38;5;10m\]'
    color_undo='\[\e[0m\]'

# Terminal supports only eight colours
elif ((colors >= 8)); then
    color_root='\[\e[1;31m\]'
    color_user='\[\e[1;32m\]'
    color_undo='\[\e[0m\]'

# Terminal may not support colour at all
else
    color_root=
    color_user=
    color_undo=
fi

The above conditional restricts the colour space to conform to what the terminal explicitly specifies is supported. If no colours at all are supported, the variables are empty, and the prompt is simply printed with no colour at all.

Vim

A familiar structure from a lot of Vim colorscheme definition files is:

if has("gui_running") || &t_Co >= 256
    ...
endif

This has the effect of checking that either gVim is running, or that the terminal is capable of printing 256 colours. You can therefore use this and similar structures to delineate different sections of the file to apply based on the number of colours supported by the terminal.

if has("gui_running") || &t_Co >= 256
    ...
elseif &t_Co >= 88
    ...
elseif &t_Co >= 8
    ...
endif

It’s not really a good idea to define this sort of stuff in your .vimrc file; colour information should go into your colour scheme file.

Tmux

Things are a little tricker in tmux. Conditionals based on shell output are supported in recent versions. If you have a particular line of your configuration that will only work on terminals that support a certain number of colours, you can prefix that line with an #if-shell call:

if-shell 'test $(tput colors) -ge 256' 'set-option -g default-terminal "screen-256color"'

The above will only set the default terminal to screen-256color for new sessions if the test call returns a result greater than or equal to 256. Note that you need to wrap both the shell test and the configuration to run in quotes for this to work.

You can preface every applicable line with this test, or if you’d prefer to only run one test for efficiency reasons, you could arrange to load a separate local configuration file only if a single test passes:

if-shell 'test "$(tput colors)" -ge 256' 'source-file ~/.tmux.256.conf'

Unfortunately, there’s a race condition in tmux that means that the first shell can be established before the default-terminal configuration item is actually applied, meaning that the default $TERM of screen is used for the first window, but subsequent windows are correctly created with the screen-256color setting for $TERM. One possible workaround for this is to preserve the old terminal’s name in an environment variable:

containing_term=$TERM
if-shell 'test "$(tput colors)" -ge 256' 'set-option -g default-terminal "screen-256color"'

You can then check this variable’s value in your .bashrc file to see if it’s a terminal known to support 256 colours, and if it is, force the screen-256color terminal type for the shell:

case $containing_term in
    *256color) 
        TERM=screen-256color
        unset containing_term
        ;;
esac

This problem may well be fixed in future versions of tmux, but it still seems to be an issue even in the developing 1.7 version.

All of the above goes a long way to making your personal set of dotfiles more robust, so that when you need to bust them out on some older machine or less capable terminal, they’re more likely to work properly.