Category Archives: Bash

Safely editing as root

Posted on by

Certain files on a UNIX-like system, such as /etc/passwd and /etc/sudoers, are integral for managing login and authentication, and it’s thus necessary to be very careful while editing them using sudo not to accidentally leave them in a corrupted state, or to allow others to edit them at the same time as you. In the worst case scenario it’s possible to lock yourself out of a system or out of root privileges in doing this, and things can only be fixed via physical access to the server or someone who knows the actual root password, which you may not necessarily know as a sudo user.

You should therefore never edit /etc/passwd, /etc/group, or /etc/sudoers by simply invoking them in your editor of choice. A set of simple utilities exist to help you make these edits safely.

vipw and vigr

If you want to safely edit the /etc/passwd file, for which you’ll need to have root privileges, you should use the vipw tool. It doesn’t require an argument.

# vipw

This will load a temporary copy of the file into your $EDITOR, and allow you to make changes. If all is well after you save and quit, you’ll see a message like:

You have modified /etc/passwd.
You may need to modify /etc/shadow for consistency.
Please use the command 'vipw -s' to do so.

If you’ve made changes which might require changing something in the /etc/shadow file, you should follow these instructions too.

The command to edit groups, vigr, works in much the same way:

# vigr

visudo

The analogous tool for editing the /etc/sudoers file is visudo. This file not only does the necessary lock and file corruption checking as vipw does, it also does some basic checking of the syntax of the file after you save it.

# visudo

If the changes you make to this file work correctly, you’ll simply be returned to your prompt. However, if you’ve made some sort of edit that means sudo won’t be able to correctly parse the file, you’ll get warned and prompted for an appropriate action:

visudo: >>> /etc/sudoers: syntax error near line 28 <<<
visudo: >>> /etc/sudoers: syntax error near line 29 <<<
visudo: >>> /etc/sudoers: syntax error near line 29 <<<
What now?

If you press ? here and then Enter, you’ll get a list of the actions you can take:

Options are:
(e)dit sudoers file again
e(x)it without saving changes to sudoers file
(Q)uit and save changes to sudoers file (DANGER!)

You’ll probably want the first one, to edit your changes again and make them work properly, but you may want to hose them and start again via the second option. You should only choose the third if you absolutely know what you’re doing.

sudoedit

In general, you can edit root-owned files using sudoedit, or sudo -e, which will operate on temporary copies of the file and overwrite the original if changes are detected:

$ sudo -e /etc/network/interfaces

This has the added bonus of preserving all of your environment variables for the editing session, which may not be the case when invoking an editor and file via sudo. This turns out to be handy for newer versions of sudo which do not preserve the user’s $HOME directory by default, meaning that configuration files for your editor, such as .vimrc, might not be read.

Bash job control

Posted on by

Oftentimes you may wish to start a process on the Bash shell without having to wait for it to actually complete, but still be notified when it does. Similarly, it may be helpful to temporarily stop a task while it’s running without actually quitting it, so that you can do other things with the terminal. For these kinds of tasks, Bash’s built-in job control is very useful.

Backgrounding processes

If you have a process that you expect to take a long time, such as a long cp or scp operation, you can start it in the background of your current shell by adding an ampersand to it as a suffix:

$ cp -r /mnt/bigdir /home &
[1] 2305

This will start the copy operation as a child process of your bash instance, but will return you to the prompt to enter any other commands you might want to run while that’s going.

The output from this command shown above gives both the job number of 1, and the process ID of the new task, 2305. You can view the list of jobs for the current shell with the builtin jobs:

$ jobs
[1]+  Running  cp -r /mnt/bigdir /home &

If the job finishes or otherwise terminates while it’s backgrounded, you should see a message in the terminal the next time you update it with a newline:

[1]+  Done  cp -r /mnt/bigdir /home &

Foregrounding processes

If you want to return a job in the background to the foreground, you can type fg:

$ fg
cp -r /mnt/bigdir /home &

If you have more than one job backgrounded, you should specify the particular job to bring to the foreground with a parameter to fg:

$ fg %1

In this case, for shorthand, you can optionally omit fg and it will work just the same:

$ %1

Suspending processes

To temporarily suspend a process, you can press Ctrl+Z:

$ cp -r /mnt/bigdir /home
^Z
[1]+  Stopped  cp -r /mnt/bigdir /home

You can then continue it in the foreground or background with fg %1 or bg %1 respectively, as above.

This is particularly useful while in a text editor; instead of quitting the editor to get back to a shell, or dropping into a subshell from it, you can suspend it temporarily and return to it with fg once you’re ready.

Dealing with output

While a job is running in the background, it may still print its standard output and standard error streams to your terminal. You can head this off by redirecting both streams to /dev/null for verbose commands:

$ cp -rv /mnt/bigdir /home &>/dev/null

However, if the output of the task is actually of interest to you, this may be a case where you should fire up another terminal emulator, perhaps in GNU Screen or tmux, rather than using simple job control.

Suspending SSH sessions

As a special case, you can suspend an SSH session using an SSH escape sequence. Type a newline followed by a ~ character, and finally press Ctrl+Z to background your SSH session and return to the terminal from which you invoked it.

tom@conan:~$ ssh crom
tom@crom:~$ ~^Z [suspend ssh]
[1]+  Stopped  ssh crom
tom@conan:~$

You can then resume it as you would any job by typing fg:

tom@conan:~$ fg %1
ssh crom
tom@crom:~$

Command line editing

Posted on by

By default, the Bash shell uses GNU Readline, which provides a set of Emacs-friendly key bindings that are a pretty workable way to edit long and complicated commands. If you learn a little about the chords available to you in editing Bash commands through this library, and combine that with a little old-school command-line magic, you can work at high speed with decent accuracy quite easily. Alternatively, if you don’t like the Emacs-style keybindings and would prefer to stick to a vi-friendly model, you can use set -o vi to edit your command line directly with vi keybindings.

However, if you’re building a particularly complex string of commands involving a lot of pipes, escapes, and redirections, it often turns out to be handy to actually load them into your favourite editor, to give you full facility to edit them in any way you wish. Bash provides a method for this in the form of its Ctrl+X, Ctrl+E binding.

To use it, you can type anything at the command prompt (including nothing at all) and press Ctrl+X, Ctrl+E to bring up your EDITOR, be it Vim, Emacs, or Nano, with the contents of the command line there to edit. As soon as you save and quit, the command will be run as stated, and will be entered into the command history as if you typed it out on the shell directly.

There’s also a handy built-in Bash shortcut, fc (short for “fix command”) to open the previous command in your editor, allow you to edit it, and then run it automatically when you quit. This is particularly useful if you’ve made a small mistake in a complex line of shell code.

If this happens to bring up the wrong editor, perhaps because your choice doesn’t match that of the system administrator, you can set your personal preference of editor like so:

$ export EDITOR=/usr/bin/vim

You can confirm this is working by checking your environment variables:

$ env

One downside of this method is that without special setup within your editor, you lose some of the benefits of things like tab completion. Fortunately it only takes a little creative mapping to make this work in Vim, taking advantage of the Ctrl+X, Ctrl+F file completion that’s already built in. You could even bind that straight to the Tab key if you don’t otherwise use it.

:inoremap <Tab> <C-X><C-F>

High-speed Bash

Posted on by

One of my favourite technical presentations I’ve read online has been Hal Pomeranz’s Unix Command-Line Kung Fu, a catalogue of shortcuts and efficient methods of doing very clever things with the Bash shell. None of these are grand arcane secrets, but they’re things that are often forgotten in the course of daily admin work, when you find yourself typing something you needn’t, or pressing up repeatedly to find something you wrote for which you could simply search your command history.

I highly recommend reading the whole thing, as I think even the most experienced shell users will find there are useful tidbits in there that would make their lives easier and their time with the shell more productive, beyond simpler things like tab completion. Here, I’ll recap two of the things I thought were the most simple and useful items in the presentation for general shell usage, and see if I can add a little value to them with reference to the Bash manual.

History with Ctrl+R

For many shell users, finding a command in history means either pressing the up arrow key repeatedly, or perhaps piping a history call through grep. It turns out there’s a much nicer way to do this, using Bash’s built-in history searching functionality; if you press Ctrl+R and start typing a search pattern, the most recent command matching that pattern will automatically be inserted on your current line, at which point you can adapt it as you need, or simply press Enter to run it again. You can keep pressing Ctrl+R to move further back in your history to the next-most recent match. On my shell, if I search through my history for git, I can pull up what I typed for a previous commit:

(reverse-i-search)`git': git commit -am "Pulled up-to-date colors."

This functionality isn’t actually exclusive to Bash; you can establish a history search function in quite a few tools that use GNU Readline, including the MySQL client command line.

You can search forward through history in the same way with Ctrl+S, but it’s likely you’ll have to fix up a couple of terminal annoyances first.

Additionally, if like me you’re a Vim user and you don’t really like having to reach for the arrow keys, or if you’re on a terminal where those keys are broken for whatever reason, you can browse back and forth within your command history with Ctrl+P (previous) and Ctrl+N (next). These are just a few of the Emacs-style shortcuts that GNU Readline provides; check here for a more complete list.

Repeating commands with !!

The last command you ran in Bash can be abbreviated on the next line with two exclamation marks:

$ echo "Testing."
Testing.
$ !!
Testing.

You can use this to simply repeat a command over and over again, although for that you really should be using watch, but more interestingly it turns out this is very handy for building complex pipes in stages. Suppose you were building a pipeline to digest some data generated from a program like netstat, perhaps to determine the top 10 IP addresses that are holding open the most connections to a server. You might be able to build a pipeline like this:

# netstat -ant
# !! | awk '{print $5}'
# !! | sort
# !! | uniq -c
# !! | sort -rn
# !! | sed 10q

Similarly, you can repeat the last argument from the previous command line using !$, which is useful if you’re doing a set of operations on one file, such as checking it out via RCS, editing it, and checking it back in:

$ co -l file.txt
$ vim !$
$ ci -u !$

Or if you happen to want to work on a set of arguments, you can repeat all of the arguments from the previous command using !*:

$ touch a.txt b.txt c.txt
$ rm !*

When you remember to user these three together, they can save you a lot of typing, and will really increase your accuracy because you won’t be at risk of mistyping any of the commands or arguments. Naturally, however, it pays to be careful what you’re running through rm!

Useless use of cat

Posted on by

If you’re reasonably confident with using pipes and redirection to build command-line strings in Bash and similar shells, at some point somebody is going to tell you off for a “useless use of cat“. This means that somewhere in your script or pipeline, you’ve used the cat command unnecessarily.

Here’s a simple example of such a use of cat; here, I’m running an Apache log through a pipe to awk to get a list of all the IP addresses that have accessed the server.

# cat /var/log/apache2/access.log | awk '{print $1}'

That works just fine, but I don’t actually need the cat instance there. Instead, I can supply a file argument directly to awk, and the result will be the same, with one less process spawned as a result:

# awk '{print $1}' /var/log/apache2/access.log

Most of the standard UNIX command-line tools designed for filtering text actually work like this; like me, you probably already knew that, but fell into the habit of using cat anyway.

For tools that don’t take a filename instead of standard input, such as mail, you can explicitly specify that a file’s contents should be used as the standard input stream. Consider this line, where I’m mailing a copy of my Apache configuration to myself:

# cat /etc/apache2/apache2.conf | mail tom@sanctum.geek.nz

That works fine, but we can compact it using < as a redirection symbol for the same result, and a command line a whole five characters shorter:

# mail tom@sanctum.geek.nz </etc/apache2/apache2.conf

If you overuse cat, then these two methods will probably enable you to fix that with a little effort. Incidentally, the second method can also be extended to provide in-place arguments for tools that read from files, by adding parentheses. Here, for example, I’m comparing the difference in output when I run grep with two different flags on the same file, using diff:

# diff -u <(grep -E '(a|b)' file.txt) <(grep -F '(a|b)' file.txt)