Default grep options

Posted on by
Reply

When you’re searching a set of version-controlled files for a string with grep, particularly if it’s a recursive search, it can get very annoying to be presented with swathes of results from the internals of the hidden version control directories like .svn or .git, or include metadata you’re unlikely to have wanted in files like .gitmodules.

GNU grep uses an environment variable named GREP_OPTIONS to define a set of options that are always applied to every call to grep. This comes in handy when exported in your .bashrc file to set a “standard” grep environment for your interactive shell. Here’s an example of a definition of GREP_OPTIONS that excludes a lot of patterns which you’d very rarely if ever want to search with grep:

GREP_OPTIONS=
for PATTERN in .cvs .git .hg .svn; do
    GREP_OPTIONS="$GREP_OPTIONS --exclude-dir=$PATTERN"
done
export GREP_OPTIONS

Note that --exclude-dir is a relatively recent addition to GNU grep‘s options, but it should only be missing on very legacy Linux machines by now. If you want to keep your .bashrc file compatible, you could apply a little extra hackery to make sure the option is available before you set it up to be used:

GREP_OPTIONS=
if grep --help | grep -- --exclude-dir &>/dev/null; then
    for PATTERN in .cvs .git .hg .svn; do
        GREP_OPTIONS="$GREP_OPTIONS --exclude-dir=$PATTERN"
    done
fi
export GREP_OPTIONS

Similarly, you can ignore single files with --exclude. There’s also --exclude-from=FILE if your list of excluded patterns starts getting too long.

Other useful options available in GNU grep that you might wish to add to this environment variable include:

  • --color — On appropriate terminal types, highlight the pattern matches in output, among other color changes that make results more readable
  • -s — Suppresses error messages about files not existing or being unreadable; helps if you find this behaviour more annoying than useful.
  • -E, -F, or -P — Pick a favourite “mode” for grep; devotees of PCRE may find adding -P for grep‘s experimental PCRE support makes grep behave in a much more pleasing way, even though it’s described in the manual as being experimental and incomplete

If you don’t want to use GREP_OPTIONS, you could instead simply set up an alias:

alias grep='grep --exclude-dir=.git'

You may actually prefer this method as it’s essentially functionally equivalent, but if you do it this way, when you want to call grep without your standard set of options, you only have to prepend a backslash to its call:

$ \grep pattern file

Of course, you could solve a lot of these problems simply by using ack … but that’s another post.

Safe MySQL updates

Posted on by
1

If you use the MySQL command line client a lot, you could consider adding the following to your .bashrc:

alias mysql='mysql --safe-updates'

Log out, then in again, or source your .bashrc file. This will quietly add in the --safe-updates parameter to any call you make for the mysql binary, which will prevent you from performing UPDATE or DELETE operations on any table if you neither specify a LIMIT condition nor a WHERE condition based on an indexed field. This stops you from accidentally deleting or updating every row in a table with queries like DELETE FROM `table` or UPDATE `table` SET `field` = 'value'.

If you really do want to delete a table’s entire contents, TRUNCATE `table` is more efficient anyway, and something you’re considerably less likely to fat-finger.

You can check this is loaded by typing alias with no arguments:

$ alias
alias mysql='mysql --safe-updates'

This very seldom gets in the way of actual operations you intend to run, and at the same time it prevents you from running potentially disastrous queries over your whole table. The MySQL manual lists it as for beginners, but regardless of your experience level in production environments this kind of safety valve can be a lifesaver, particularly if you’re not at your most lucid.

Like any Bash alias, if you want to call mysql directly without going through the alias, perhaps because you really do want to update a field in every single row of a table, then just prepend a backslash:

$ \mysql

Funnily enough, a valid alias of this option is --i-am-a-dummy. Pick whichever you think is more appropriate.

Vim as Debian default

Posted on by
Reply

The default text editor in installations of Debian and its derivatives is Nano, largely because it’s a simple and small editor. If you’re a Vim user, you might find it a little jarring to be presented with a modeless editor when you run commands like visudo or vipw.

Debian’s alternatives system makes this reasonably easy to adjust. If you have Vim installed, it should be available as one of the possible implementations of the “editor” alternative. You can check this with the update-alternatives command:

# update-alternatives --list editor
/bin/ed
/usr/bin/nano
/usr/bin/vim.basic

This shows that Vim is available as an alternative with /usr/bin/vim.basic, so you can update the symlink structure that defines the default editor like so:

# update-alternatives --set editor /usr/bin/vim.basic
... using /usr/bin/vim.basic to provide /usr/bin/editor (editor) in manual mode.

Now if you fire up visudo, vipw, or sudo -e you should find that Vim is fired up instead of the editor you didn’t want.

On my own workstation I have the latest Vim compiled from Mercurial and installed into /usr/local via checkinstall, so I had to add this to the alternatives system before I could use it:

# update-alternatives --install /usr/bin/editor editor /usr/local/bin/vim 200 \
    --slave /usr/share/man/man1/editor.1.gz editor.1.gz /usr/local/share/man/man1/vim.1.gz
# update-alternatives --set editor /usr/local/bin/vim
... using /usr/bin/vim.basic to provide /usr/bin/editor (editor) in manual mode.

Other relevant alternatives include the vi implementation for your system, which of course may not necessarily be Vim; some operating systems install the smaller and more vi-faithful nvi.

Tmux environment variables

Posted on by
1

The user configuration file for the tmux terminal multiplexer, .tmux.conf, supports defining and using environment variables in the configuration, with the same syntax as most shell script languages:

TERM=screen-256color
set-option -g default-terminal $TERM

This can be useful for any case in which it may be desirable to customise the shell environment when inside tmux, beyond setting variables like default-terminal. However, if you repeat yourself in places in your configuration file, it can also be handy to use them as named constants. An example could be establishing colour schemes:

TMUX_COLOUR_BORDER="colour237"
TMUX_COLOUR_ACTIVE="colour231"
TMUX_COLOUR_INACTIVE="colour16"

set-window-option -g window-status-activity-bg $TMUX_COLOUR_BORDER
set-window-option -g window-status-activity-fg $TMUX_COLOUR_ACTIVE
set-window-option -g window-status-current-format "#[fg=$TMUX_COLOUR_ACTIVE]#I:#W#F"
set-window-option -g window-status-format "#[fg=$TMUX_COLOUR_INACTIVE]#I:#W#F"

The explicit commands to work with environment variables in .tmux.conf are update-environment, set-environment, and show-environment, and are featured in the manual.

Packaging built software

Posted on by
1

The Debian package repository is enormous, and the Sid distribution is in most cases reasonably up-to-date, but it’s still sometimes desirable to build an application and install it into /usr/local when a packaged implementation either isn’t available or is too out of date, or if you’re involved in the development of a project and want to try out a fresh build that hasn’t been packaged yet.

The usual cycle of configuring, compiling, and installing many open-source applications for Unix-like systems applies here:

$ ./configure
$ make
# make install

The above is normally the approach taken to install code compiled on the machine, rather than through packages. One problem with this approach is that it doesn’t allow many of the advantages that a system running purely on packages does; dpkg -l will no longer give you a complete overview of all the system’s software, and to remove the software and its configuration files you may have to manually delete it rather than using apt-get purge.

Fortunately, there exists a tool called checkinstall to allow having the best of both worlds. After installing this tool via apt-get checkinstall, you’re able to build a package for your locally built software according to the rules defined in its Makefile, and install that the same way as any other package.

Instead of typing make install, type checkinstall, and you will be prompted for details about the package, including its description, which is then built and installed. In this example, I’m compiling Vim from source, which works very well. I’ve also successfully installed tools like pam_ssh_agent_auth and tmux this way.

With this done, the package’s files are installed in /usr/local, and it appears in the list of installed packages along with my explanation of its contents:

$ ./configure
$ make
$ sudo -s
# checkinstall
...
*****************************************
**** Debian package creation selected ***
*****************************************

This package will be built according to these values:

0 -  Maintainer: [ Tom Ryder <tom@sanctum.geek.nz> ]
1 -  Summary: [ Custom build of latest Vim 7.3 ]
2 -  Name:    [ vim ]
3 -  Version: [ 2:7.4 ]
4 -  Release: [ 1 ]
5 -  License: [ GPL ]
6 -  Group:   [ checkinstall ]
7 -  Architecture: [ amd64 ]
8 -  Source location: [ vim ]
9 -  Alternate source location: [  ]
10 - Requires: [  ]
11 - Provides: [ vim ]
12 - Conflicts: [  ]
13 - Replaces: [  ]

Note that I’m assigning it a version number greater than the Debian repository’s vim package, which is a simple way to prevent it being replaced. You can also do this via the /etc/apt/preferences file to prevent replacement of all packages from the checkinstall group.

With this done, my custom build of Vim now shows in the package list, and its files are correctly installed in /usr/local:

# dpkg -l | grep vim
ii  vim  2:7.4-1 Custom build of latest Vim 7.3
# dpkg -S vim
vim: /usr/local/share/vim/vim73/bugreport.vim
vim: /usr/local/share/vim/vim73/plugin
vim: /usr/local/share/vim/vim73/ftplugin/postscr.vim
vim: /usr/local/share/man/it.UTF-8/man1/vim.1.gz
...

The .deb package built by checkinstall is also present in my build directory for me to keep for later, or for installation on another compatible server:

$ ls *.deb
vim_7.4-1_amd64.deb

It’s worth noting that checkinstall is not a Debian-specific tool; it works for other packaging systems like RPM and Slackware, too.

Faster Vim search/replace

Posted on by
4

Entering search patterns and replacement strings in Vim can sometimes be a pain, particularly if the search or replacement text is already available in a register or under the cursor. There are a few ways to make working with search and replace in Vim quicker and less cumbersome for some common operations, and thereby save a bit of error-prone typing.

Insert contents of a register

As in command mode or insert mode, you can insert the contents of any register by pressing Ctrl+R and then the register key. For example, to directly insert the contents of register a, you can type Ctrl+R and then a while typing a search pattern or replacement string.

Reference contents of a register

Similarly, if you want to use the contents of a register in a pattern or replacement but don’t want to directly insert it, you can instead reference the contents of register a with \=@a:

:s/search/\=@a/

Both of the above tips work for both the search and replacement patterns, and for special registers like " (default unnamed register) and / (last searched text) as well as the alphabetical ones.

Insert word under cursor

If you happen to be hovering over a word that you want to use as as a search or replacement string, as a special case of the above you can do so by typing Ctrl+R and then Ctrl+W for a normal word, or Ctrl+R and then Ctrl+A for a space-delimited word.

Empty search string

You can use the previous search as a search pattern or replacement string by including it from the special / register, in the same way as any other register above, by inserting it with Ctrl+R and then /, or representing it with \=@/. There’s a convenient shorthand for this however in just using an empty search string:

:s//replacement/

This will replace all occurences of the previous search string with replacement. It turns out to be a particularly convenient shorthand when searching for words by pressing * or #.

Typo correction in Bash

Posted on by
2

If you make a typo in a long command line in Bash, instead of typing it all out again, you can either recall it from your history, or use caret notation to repeat the command with an appropriate substitution. This looks like the following:

# sudo apache2ctl restrat
Action 'restrat' failed.
The Apache error log may have more information.
# ^strat^start
sudo apache2ctl restart

The string after the first caret is the text to match, and the one after the second string is the text with which it should be replaced. This provides a convenient method of not only quickly correcting typos, but to change small parts of the command line in general quickly:

$ touch filea.txt
$ ^filea^fileb
touch fileb.txt
$ ^fileb^filec
touch filec.txt

For the particular case of correcting small errors in long paths for cd calls, it’s helpful to use the cdspell option for Bash, which I discuss in my earlier article on smarter directory navigation.

Bash command existence

Posted on by
6

If you set environment variables like your $EDITOR in your .bashrc file that refer to commands that you expect to be available on the system, it’s prudent to check that an appropriate command actually exists before making the setting.

A common way of approaching this is using which, in a syntax similar to the below, which sets the $EDITOR to the first executable instance of vi found in $PATH, or blank if not found:

EDITOR=`which vi`

Because the behaviour of which can be unexpected, it’s better practice to use one of Bash’s builtins to do this, either command, type, or hash. I prefer using hash, which searches $PATH for a command of the given name, and loads it into Bash’s command hash table if found. An implementation like the below works well:

hash vi &>/dev/null
if [ $? -eq 0 ]; then
    export EDITOR=vi
fi

This ignores any output from the hash call by redirecting it to the null device, and checks its return value in the special $? variable, only defining the value for $EDITOR if a matching command is found.

You can compact this syntax into one line:

hash vi &>/dev/null && export EDITOR=vi

Thanks to commenter Yu-Jie Lin for pointing out the above abbreviation.

Reloading tmux config

Posted on by
2

If you have made changes to your tmux configuration file in the ~/.tmux.conf file, it shouldn’t be necessary to start the server up again from scratch with kill-server. Instead, you can prompt the current tmux session to reload the configuration with the source-file command.

This can be done either from within tmux, by pressing Ctrl+B and then : to bring up a command prompt, and typing:

source-file ~/.tmux.conf

Or simply from a shell:

$ tmux source-file ~/.tmux.conf

This should apply your changes to the running tmux server without affecting the sessions or windows within them.

Learning /bin and /usr/bin

Posted on by
3

When you have some spare time, something instructive to do that can help fill gaps in your Unix knowledge and to get a better idea of the programs installed on your system and what they can do is a simple whatis call, run over all the executable files in your /bin and /usr/bin directories. This will give you a one-line summary of the file’s function if available from man pages.

tom@conan:/bin$ whatis *
bash (1) - GNU Bourne-Again SHell
bunzip2 (1) - a block-sorting file compressor, v1.0.4
busybox (1) - The Swiss Army Knife of Embedded Linux
bzcat (1) - decompresses files to stdout
...
tom@conan:/usr/bin$ whatis *
[ (1)                - check file types and compare values
2to3 (1)             - Python2 to Python3 converter
2to3-2.7 (1)         - Python2 to Python3 converter
411toppm (1)         - convert Sony Mavica .411 image to ppm
...

It also works on many of the files in other directories, such as /etc:

tom@conan:/etc$ whatis *
acpi (1)             - Shows battery status and other ACPI information
adduser.conf (5)     - configuration file for adduser(8) and addgroup(8) .
adjtime (3)          - correct the time to synchronize the system clock
aliases (5)          - Postfix local alias database format
...

Because packages often install more than one binary and you’re only in the habit of using one or two of them, this process can tell you about programs on your system that you may have missed, particularly standard tools that solve common problems. As an example, I first learned about watch this way, having used a clunky solution with for loops with sleep calls to do the same thing many times before.