Tag Archives: vi

Shell from vi

Posted on by

A good sign of a philosophically sound interactive Unix tool is the facilities it offers for interacting with the filesystem and the shell: specifically, how easily can you run file operations and/or shell commands with reference to data within the tool? The more straightforward this is, the more likely the tool will fit neatly into a terminal-driven Unix workflow.

If all else fails, you could always suspend the task with Ctrl+Z to drop to a shell, but it’s helpful if the tool shows more deference to the shell than that; it means you can use and (even more importantly) write tools to manipulate the data in the program in whatever languages you choose, rather than being forced to use any kind of heretical internal scripting language, or worse, an over-engineered API.

vi is a good example of a tool that interacts openly and easily with the Unix shell, allowing you to pass open buffers as streams of text transparently to classic filter and text processing tools. In the case of Vim, it’s particularly useful to get to know these, because in many cases they allow you to avoid painful Vimscript, and to do things your way, without having to learn an ad-hoc language or to rely on plugins. This was touched on briefly in the “Editing” article of the Unix as IDE series.

Choosing your shell

By default, vi will use the value of your SHELL environment variable as the shell in which your commands will be run. In most cases, this is probably what you want, but it might pay to check before you start:

:set shell?

If you’re using Bash, and this prints /bin/bash, you’re good to go, and you’ll be able to use Bash-specific features or builtins such as [[ comfortably in your command lines if you wish.

Running commands

You can run a shell command from vi with the ! ex command. This is inherited from the same behaviour in ed. A good example would be to read a manual page in the same terminal window without exiting or suspending vi:

:!man grep

Or to build your project:

:!make

You’ll find that exclamation point prefix ! shows up in the context of running external commands pretty consistently in vi.

You will probably need to press Enter afterwards to return to vi. This is to allow you to read any output remaining on your screen.

Of course, that’s not the only way to do it; you may prefer to drop to a forked shell with :sh, or suspend vi with ^Z to get back to the original shell, resuming it later with fg.

You can refer to the current buffer’s filename in the command with %, but be aware that this may cause escaping problems for files with special characters in their names:

:!gcc % -o foo

If you want a literal %, you will need to escape it with a backslash:

:!grep \% .vimrc

The same applies for the # character, for the alternate buffer.

:!gcc # -o bar
:!grep \# .vimrc

And for the ! character, which expands to the previous command:

:!echo !
:!echo \!

You can try to work around special characters for these expansions by single-quoting them:

:!gcc '%' -o foo
:!gcc '#' -o bar

But that’s still imperfect for files with apostrophes in their names. In Vim (but not vi) you can do this:

:exe "!gcc " . shellescape(expand("%")) . " -o foo"

The Vim help for this is at :help :!.

Reading the output of commands into a buffer

Also inherited from ed is reading the output of commands into a buffer, which is done by giving a command starting with ! as the argument to :r:

:r !grep vim .vimrc

This will insert the output of the command after the current line position in the buffer; it works in the same way as reading in a file directly.

You can add a line number prefix to :r to place the output after that line number:

:5r !grep vim .vimrc

To put the output at the very start of the file, a line number of 0 works:

:0r !grep vim .vimrc

And for the very end of the file, you’d use $:

:$r !grep vim .vimrc

Note that redirections work fine, too, if you want to prevent stderr from being written to your buffer in the case of errors:

:$r !grep vim .vimrc 2>>vim_errorlog

Writing buffer text into a command

To run a command with standard input coming from text in your buffer, but without deleting it or writing the output back into your buffer, you can provide a ! command as an argument to :w. Again, this behaviour is inherited from ed.

By default, the whole buffer is written to the command; you might initially expect that only the current line would be written, but this makes sense if you consider the usual behaviour of w when writing directly to a file.

Given a file with a first column full of numbers:

304 Donald Trump
227 Hillary Clinton
3   Colin Powell
1   Spotted Eagle
1   Ron Paul
1   John Kasich
1   Bernie Sanders

We could calculate and view (but not save) the sum of the first column with awk(1), to see the expected value of 538 printed to the terminal:

:w !awk '{sum+=$1}END{print sum}'

We could limit the operation to the faithless electoral votes by specifying a line range:

:3,$w !awk '{sum+=$1}END{print sum}'

You can also give a range of just ., if you only want to write out the current line.

In Vim, if you’re using visual mode, pressing : while you have some text selected will automatically add the '<,'> range marks for you, and you can write out the rest of the command:

:'<,'>w !grep Bernie

Note that this writes every line of your selection to the command, not merely the characters you have selected. It’s more intuitive to use visual line mode (Shift+V) if you take this approach.

Filtering text

If you want to replace text in your buffer by filtering it through a command, you can do this by providing a range to the ! command:

:1,2!tr '[:lower:]' '[:upper:]'

This example would capitalise the letters in the first two lines of the buffer, passing them as input to the command and replacing them with the command’s output.

304 DONALD TRUMP
227 HILLARY CLINTON
3 Colin Powell
1 Spotted Eagle
1 Ron Paul
1 John Kasich
1 Bernie Sanders

Note that the number of lines passed as input need not match the number of lines of output. The length of the buffer can change. Note also that by default any stderr is included; you may want to redirect that away.

You can specify the entire file for such a filter with %:

:%!tr '[:lower:]' '[:upper:]'

As before, the current line must be explicitly specified with . if you want to use only that as input, otherwise you’ll just be running the command with no buffer interaction at all, per the first heading of this article:

:.!tr '[:lower:]' '[:upper:]'

You can also use ! as a motion rather than an ex command on a range of lines, by pressing ! in normal mode and then a motion (w, 3w, }, etc) to select all the lines you want to pass through the filter. Doubling it (!!) filters the current line, in a similar way to the yy and dd shortcuts, and you can provide a numeric prefix (e.g. 3!!) to specify a number of lines from the current line.

This is an example of a general approach that will work with any POSIX-compliant version of vi. In Vim, you have the gU command available to coerce text to uppercase, but this is not available in vanilla vi; the best you have is the tilde command ~ to toggle the case of the character under the cursor. tr(1), however, is specified by POSIX–including the locale-aware transformation–so you are much more likely to find it works on any modern Unix system.

If you end up needing such a command during editing a lot, you could make a generic command for your private bindir, say named upp for uppercase, that forces all of its standard input to uppercase:

#!/bin/sh
tr '[:lower:]' '[:upper:]'

Once saved somewhere in $PATH and made executable, this would allow you simply to write the following to apply the filter to the entire buffer:

:%!upp

The main takeaway from this is that the scripts you use with your editor don’t have to be in shell. You might prefer Awk:

#!/usr/bin/awk -f
{ print toupper($0) }

Or Perl:

#!/usr/bin/env perl
print uc while <>;

Or Python, or Ruby, or Rust, or …

ed supports this Incidentally, this “filtering” feature is where vi‘s heritage from ed ends as far as external commands are concerned. In POSIX ed, there isn’t a way to filter a subset of the buffer text through a command in one hit. It’s not too hard to emulate it with a temporary file, though, using all the syntax learned above:

*1,2w !upp > tmp
*1,2d
*0r tmp
*!rm tmp

However, there is a way to filter a whole file in one hit:

*e !upp < %

Actually using ed

Posted on by

The classic ed editor is a really good example of a sparse, minimal, standard Unix tool that does one thing, and does it well. Because there are so many good screen-oriented editors for Unix, there’s seldom very much call for using ed, unless you’re working on very old or very limited hardware that won’t run anything else.

However, if part of the reason you use vi is because you think it will always be there (it may not be), then you should learn ed too. If your terminal is broken and neither vi nor nano will work, or you break it some other way, your choices may well be between cat and ed. Or even the grand heresy of sed -i…

Not a friendly editor

Even more than its uppity grandchild ex/vi, ed has developed a reputation as terse and intimidating to newcomers. When you type ed at the command line, nothing happens, and the only error message presented by default is ?. If you’re reading this, it’s likely your first and only experience with ed went something like this:

$ ed
help
?
h
Invalid command suffix
?
?
^C
?
exit
?
quit
?
^Z
$ killall ed
$ vi

So, ed is not a terribly intuitive editor. However, it’s not nearly as hard to learn as it might seem, especially if you’re a veteran vi user and thereby comfortable with the ex command set. With a little practice, you can actually get rather quick with it; there’s an elegance to its almost brutal simplicity.

It’s also very interesting to learn how ed works and how to use it, not just because it might very well be useful for you one day, but because it occupies an important position in the heritage of the sed stream editor, the ex line editor, the vi visual editor, the grep tool, and many other contexts.

Why is ed so terse?

When ed was developed, the usual method of accessing a Unix system was via a teletype device, on which it wouldn’t have been possible to use a screen-oriented editor like vi. Similarly, modems were slow, and memory was precious; using abbreviated commands and terse error messages made a lot of sense, because the user would otherwise be wasting a lot of time waiting for the terminal to react to commands, and didn’t have a whole lot of memory to throw around for anything besides the buffer of text itself.

Of course, this is almost a non-issue for most Unix-like systems nowadays, so one of the first things we’ll do is make ed a little bit less terse and more user-friendly.

Error messages

Start ed up the usual way:

$ ed

We’ll start by deliberately doing something wrong. Type b and press Enter:

b
?

There’s that tremendously unhelpful ? again. But if you press h, you can see what went wrong:

h
Unknown command

Of course, since it’s the future now, we can spare the terminal cycles to have ed print the error message for us every time. You can set this up by pressing H:

H
b
?
Unknown command

That’s a bit more useful, and should make things easier.

Quitting

You can quit ed with q. Go ahead and do that. If you had unsaved changes in a buffer, you could type Q to quit unconditionally. Repeating yourself works too:

q
?
Warning: buffer modified
q

Command prompt

Let’s invoke ed again, but this time we’ll use the -p option to specify a command prompt:

$ ed -p\*
*

We’ll use that from now on, which will make things clearer both for interpreting this tutorial and for remembering whether we’re in command mode, or entering text. It might even be a good idea to make it into a function:

$ ed() { command ed -p\* "$@" ; }

Basic text input

We’ll start by adding a couple of lines of text to the new buffer. When you start ed with no filename, it starts an empty buffer for you, much like vi does.

Because there are no lines at all at present, press a to start adding some at the editor’s current position:

*a
Start typing a few lines.
Anything you want, really.
Just go right ahead.
When you're done, just type a period by itself.
.

That’s added four new lines to the buffer, and left the editor on line 4. You can tell that’s the case because typing p for print, just by itself, prints the fourth line:

*p
When you're done, just type a period by itself.

A little more useful is n (or pn in some versions), which will show both the line number and the contents of the line:

*n
4       When you're done, just type a period by itself.

So just like in ex, the current line is the default for most commands. You can make this explicit by referring to the current line as .:

*.n
4       When you're done, just type a period by itself.

You can move to a line just by typing its number, which will also print it as a side effect:

*3
Just go right ahead.
*.n
3       Just go right ahead.

Pressing a like you did before will start inserting lines after the current line:

*a
Entering another line.
.
*n
4       Entering another line.

Pressing i will allow you to insert lines before the current line:

*i
A line before the current line.
.
*n
4       A line before the current line.

You can replace a line with c:

*c
I decided I like this line better.
.
*n
4       I decided I like this line better.

You can delete lines with d:

*6d

And join two or more lines together with j:

*1,2j

You can prepend an actual line number to any of these commands to move to that line before running the command on it:

*1c
Here's a replacement line.
.
*1n
1       Here's a replacement line.

For most of these commands, the last line to be changed will become the new current line.

Ranges

You can select the entire buffer with 1,$ or , for short (% works too):

*,p
Here's a replacement line.
Just go right ahead.
I decided I liked this line better.
Entering another line.

Or a limited range of specific lines:

*2,3p
Just go right ahead.
I decided I liked this line better.

These ranges can include a reference to the current line with .:

*2
Just go right ahead.
*.,4p
Just go right ahead.
I decided I liked this line better.
Entering another line.

They can also include relative line numbers, prefixed with + or -:

*2
*-1,+1p
Here's a replacement line.
Just go right ahead.
I decided I liked this line better.

You can drop a mark on a line with k followed by a lowercase letter such as a, and you’re then able to refer to it in ranges as 'a:

*3ka
*'ap
I decided I liked this line better.

Moving and copying

Move a line or range of lines to after a target line with m:

*1,2m$
*,p
I decided I liked this line better.
Entering another line.
Here's a replacement line.
Just go right ahead.

Copy lines to after a target line with t:

*2t4
*,p
I decided I liked this line better.
Entering another line.
Here's a replacement line.
Just go right ahead.
Entering another line.

Regular expressions

You can select lines based on classic regular expressions with the g operator. To print all lines matching the regular expression /re/:

*g/re/p
Here's a replacement line.

(Hmm, where have I seen that command before?)

You can invert the match to work with lines that don’t match the expression with v:

*v/re/p
I decided I liked this line better.
Entering another line.
Just go right ahead.
Entering another line.

Just like numbered line ranges, ranges selected with regular expressions can have other operations applied to them. To move every line containing the expression /re/ to the bottom of the file:

*g/re/m$
*,p
I decided I liked this line better.
Entering another line.
Just go right ahead.
Entering another line.
Here's a replacement line.

Searching

You can move to the next line after the current one matching a regular expression with /. Again, this will print the line’s contents as a side effect.

*/like
I decided I like this line better.

You can search backward with ?:

*?Here
Here's a replacement line.

Substituting

You can substitute for the first occurrence per line of an expression within a range of lines with the s command:

*1s/i/j
I decjded I like this line better.

You can substitute for all the matches on each line by adding the /g suffix:

*1s/i/j/g
*p
I decjded I ljke thjs ljne better.

Reading and writing

You can write the current buffer to a file with w, which will also print the total number of bytes written:

*w ed.txt
129

Having done this once, you can omit the filename for the rest of the session:

*w
129

Like most ed commands, w can be prefixed with a range to write only a subset of lines to the file. This would write lines 1 to 4 to the file ed.txt:

*1,4w ed.txt
102

You can use W to append to a file, rather than replace it. This would write lines 3 to 5 to the end of the file ed.txt:

*3,5W
71

You can read in the contents of another file after the current line (or any other line) with r. Again, this will print the number of bytes read.

*r /etc/hosts
205

The output of a command can be included by prefixing it with !:

*r !ps -e
5571

If you just want to load the contents of a file or the output of a command into the buffer, replacing its contents, use e, or E if you’ve got a modified buffer and don’t mind replacing it:

*E ed.txt
173

If you don’t like seeing the byte counts each time, you can start ed with the -s option for “quiet mode”.

Familiar syntax

Almost all of the above command sets will actually be familiar to vi users who know a little about ex, or Vimscript in Vim. It will also be familiar to those who have used sed at least occasionally.

Once you get good with ed, it’s possible you’ll find yourself using it now and then to make quick edits to files, even on systems where your favourite screen-based editor will load promptly. You could even try using ex.

Update 2016: Recent Hacker News discussion has reminded me of my long-standing mistake in asserting that ed(1) is included in every Unix and Unix-like system’s base installation. This is not even close to true–many others exclude it–and the claim has been removed, which I should have done years ago.

Thanks to A. Pedro Cunha for a couple of fixes to show correct typical output.

Bash command existence

Posted on by

If you set environment variables like your EDITOR in your .bashrc file that refer to commands that you expect to be available on the system, it’s prudent to check that an appropriate command actually exists before making the setting.

A common way of approaching this is using which, in a syntax similar to the below, which sets EDITOR to the first executable instance of vi found in PATH, or blank if not found:

EDITOR=$(which vi)

Because the behaviour of which can be unexpected, it’s better practice to use one of Bash’s builtins to do this, either command, type, or hash. I prefer using hash, which searches PATH for a command of the given name, and loads it into Bash’s command hash table if found. An implementation like the below works well:

if hash vi 2>/dev/null; then
    export EDITOR=vi
fi

This ignores any error output from the hash call by redirecting it to the null device, and only defines the value for EDITOR if a matching command is found.

You can compact this syntax into one line:

hash vi 2>/dev/null && export EDITOR=vi

Thanks to commenter Yu-Jie Lin for pointing out the above abbreviation.

Vi mode in tmux

Posted on by

Updated May 2017 to show the new syntax for the copy-mode keys.

tmux offers a set of vi-like bindings for navigating a buffer in a window. These allow you to not only navigate through the buffer beyond what your screen is currently showing, but also to search all the output generated thus far, and to select and copy text that can be pasted in any other window in the tmux session.

You can enable this as a default setting in .tmux.conf with the following:

set-window-option -g mode-keys vi

You can confirm this is working by pressing Ctrl+B and then : in a tmux session to bring up the command line, and typing:

list-keys -T copy-mode-vi

In version 2.3 and below, the syntax is different:

list-keys -t vi-copy

This will bring up a complete list of the vi-like functionality available to you in this mode.

With this done, within a tmux instance, pressing Ctrl+B and then [ will enter copy mode, allowing you to copy text or view the history of the buffer, including searching with / and ?. Most of the basic vi movements work, including screenwise vertical movement commands like Ctrl+F and Ctrl+B. You can leave this mode just by pressing Enter, or you can start a selection by pressing Space on a character, moving to another, and then pressing Enter. If you have text copied like this you can paste it into any tmux window in that session by pressing Ctrl+B and then ].

If you don’t mind artifically introducing a few Vim-only features to the vi mode, you can set things up so that v starts a selection and y finishes it in the same way that Space and Enter do, more like Vim:

bind-key -T copy-mode-vi 'v' send -X begin-selection
bind-key -T copy-mode-vi 'y' send -X copy-selection-and-cancel

In version 2.3 and below, the syntax is very different:

bind-key -t vi-copy 'v' begin-selection
bind-key -t vi-copy 'y' copy-selection

Even if you’re using a terminal emulator in a GUI environment, if you get good at using this you’ll find it’s much faster and more precise than clicking and dragging to select text to copy and paste it. It keeps you from having to move to the mouse, and additionally eliminates the problem of copying whole lines split into more than one panes.

Note that all of the above assumes that your prefix is the default of Ctrl+B, which many tmux users, myself included, seem to prefer to change to the old GNU Screen prefix key of Ctrl+A.

Vi mode in Bash

Posted on by

Because the Bash shell uses GNU Readline, you’re able to set options in .bashrc and .inputrc to extensively customise how you enter your command lines, including specifying whether you want the default Emacs-like keybindings (e.g. Ctrl+W to erase a word), or bindings using a modal interface familiar to vi users.

To use vi mode in Bash and any other tool that uses GNU Readline, such as the MySQL command line, you need only put this into your .inputrc file, then log out and in again:

set editing-mode vi

If you only want to use this mode in Bash, an alternative is to use the following in your .bashrc, again with a login/logout or resourcing of the file:

set -o vi

You can check this has applied correctly with the following, which will spit out a list of the currently available bindings for a set of fixed possible actions for text:

$ bind -P

The other possible value for editing-mode is emacs. Both options simply change settings in the output of bind -P above.

This done, you should find that you open a command line in insert mode, but when you press Escape or Ctrl+[, you will enter an emulation of Vi’s normal mode. Enter will run the command in its present state while in either mode. The bindings of most use in this mode are the classic keys for moving to positions on a line:

  • ^ — Move to start of line
  • $ — Move to end of line
  • b — Move back a word
  • w — Move forward a word
  • e — Move to the end of the next word

Deleting, yanking, and pasting all work too. Also note that k and j in normal mode allow you to iterate through your history in the same way that Ctrl+P and Ctrl+N do in Emacs mode. Searching with ? and / doesn’t work by default, but the Ctrl+R and Ctrl+S bindings still seem to work.

If you are reasonably used to working with the modeless Emacs for most of your shell work but do occasionally find yourself in a situation where being able to edit a long command line in vi would be handy, you may prefer to press Ctrl+X Ctrl+E to bring up the command line in your $EDITOR instead, affording you the complete power of Vim to edit rather than the somewhat sparse vi emulation provided by Readline. If you want to edit a command you just submitted, the fc (fix command) Bash builtin works too.

The shell is a very different environment for editing text, being inherently interactive and fixed to one line rather than a static multi-line buffer, which leads some otherwise diehard vi users such as myself to prefer the Emacs mode for editing commands. For one thing, vi mode in Bash trips on the vi anti-pattern of putting you in insert mode by default, and at the start of every command line. But if the basic vi commands are etched indelibly into your memory and have become automatic, you may appreciate being able to edit your command line using these keys instead. It’s certainly worth a try at any rate, and I do still occasionally see set -o vi in the .bashrc files of a few of the pros on GitHub.

Arabesque passed 100,000 visits today, having existed little over a month, with over 60,000 unique visitors. Thanks very much to everyone who reads and comments on the articles.

Using more of ex

Posted on by

The original vi is a visual extension of an editor called ex, itself an evolution of the editors em and ed. The single-keystroke commands used in the visual mode of the ex editor were mappings to something that could be done with the command mode of ex. It’s therefore not quite right to say that vi evolved from ex; they’re the same program, differing only by invocation name and default mode.

Vim follows the same lineage, and so for most of the basic functions you can do in visual mode there exist analogues in the ex command language, sometimes enabling you to work with text in a more precisely defined way that can be difficult using visual mode.

There are certain standard ex commands that any Vim user will know, to edit a file, save a file, quit, and to perform substitutions. But the actual command set is vast, and there are a few tips using the ex command set that turn out to be very useful, particularly where filtering and transforming text is involved.

Using ranges intelligently

Most of the familiar linewise commands in Vim operate on ranges of lines. Such an operation familiar to most Vim users will be the global substitution:

:% s/text/replacement/

The % is shorthand for the entire file. By default, if you leave the % out, the substitute operation operates on the current line:

:s/text/replacement/

To make this explicit, you could define the range with ., which is a shorthand for the current line:

:. s/text/replacement/

You can use line numbers to define on which line you would like the replacement to occur:

:20 s/text/replacement/

This also allows comma-separated definitions, defining the start and end of a range, which can be absolute, as with the following that makes substitutions only between lines 20 and 30:

:20,30 s/text/replacement/

Or relative to the current line, where the following makes substitutions from two lines above to two lines below the current line:

:-2,+2 s/text/replacement/

It’s also worth noting that the last line in the file can be referred to with $:

:20,$ s/text/replacement/

Finally, if you have any marks defined in your text, you can refer to them in ranges by prefixing them with apostrophes. This will make substitutions from the line marked with a to the one marked b:

:'a,'b s/text/replacement/

This also works with Vim’s automatic marks, such as the ones created for you when you select some text in visual select mode. When you have some text selected and you press : to start a command, you might notice that Vim automatically inserts the range definition for you, as below:

:'<,'> s/text/replacement/

Other linewise operations

Substitution is far from the only linewise operation available in ex mode. You can use d to delete a range of lines:

:20,30 d

You can use y to yank them into the default register for pasting later:

:20,30 y

You can use m to move them to a specified line, and t to copy them:

:20,30 m 40
:20,30 t 40

And finally, you can simply print all matching lines with p:

:20,30 p

Text filters with g and v

The g and v commands can be used to define ranges based on the results of a regular expression search. Say you wanted to move all lines matching the regular expression /vim/ to the top of the file. This is easily done with:

:g/vim/ m 0

Conversely, you can retrieve a range of lines that don’t match a regular expression with v:

:v/vim/ m 0

These become especially useful because you can combine searched ranges with fixed ones, to only search a specified range of lines. In this example, only lines from 10 to 20 matching the regular expression /vim/ will be moved:

:10,20 g/vim/ m 0

Normal-mode operations

As a final tip, if you define a range of lines, you can also use the normal command to run a series of normal-mode keystroke commands on them. For example, if you wanted to add a semicolon to every line matching /vim/:

:g/vim/ norm A;

You can even extend this to run a macro held in a register over an appropriate range of lines, which I think is one of the most elegant examples of composing ex tools in clever ways:

:g/vim/ norm @a