Tag Archives: functions

Custom commands

Posted on by

As users grow more familiar with the feature set available to them on UNIX-like operating systems, and grow more comfortable using the command line, they will find more often that they develop their own routines for solving problems using their preferred tools, often repeatedly solving the same problem in the same way. You can usually tell if you’ve entered this stage if one or more of the below applies:

  • You repeatedly search the web for the same long commands to copy-paste.
  • You type a particular long command so often it’s gone into muscle memory, and you type it without thinking.
  • You have a text file somewhere with a list of useful commands to solve some frequently recurring problem or task, and you copy-paste from it a lot.
  • You’re keeping large amounts of history so you can search back through commands you ran weeks or months ago with ^R, to find the last time an instance of a problem came up, and getting angry when you realize it’s fallen away off the end of your history file.
  • You’ve found that you prefer to run a tool like ls(1) more often with a non-default flag than without it; -l is a common example.

You can definitely accomplish a lot of work quickly with shoving the output of some monolithic program through a terse one-liner to get the information you want, or by developing muscle memory for your chosen toolbox and oft-repeated commands, but if you want to apply more discipline and automation to managing these sorts of tasks, it may be useful for you to explore more rigorously defining your own commands for use during your shell sessions, or for automation purposes.

This is consistent with the original idea of the Unix shell as a programming environment; the tools provided by the base system are intentionally very general, not prescribing how they’re used, an approach which allows the user to build and customize their own command set as appropriate for their system’s needs, even on a per-user basis.

What this all means is that you need not treat the tools available to you as holy writ. To leverage the Unix philosophy’s real power, you should consider customizing and extending the command set in ways that are useful to you, refining them as you go, and sharing those extensions and tweaks if they may be useful to others. We’ll discuss here a few methods for implementing custom commands, and where and how to apply them.

Aliases

The first step users take toward customizing the behaviour of their shell tools is often to define shell aliases in their shell’s startup file, usually specifically for interactive sessions; for Bash, this is usually ~/.bashrc.

Some aliases are so common that they’re included as commented-out suggestions in the default ~/.bashrc file for new users. For example, on Debian systems, the following alias is defined by default if the dircolors(1) tool is available for coloring ls(1) output by filetype:

alias ls='ls --color=auto'

With this defined at startup, invoking ls, with or without other arguments, will expand to run ls --color=auto, including any given arguments on the end as well.

In the same block of that file, but commented out, are suggestions for other aliases to enable coloured output for GNU versions of the dir and grep tools:

#alias dir='dir --color=auto'
#alias vdir='vdir --color=auto'

#alias grep='grep --color=auto'
#alias fgrep='fgrep --color=auto'
#alias egrep='egrep --color=auto'

Further down still, there are some suggestions for different methods of invoking ls:

#alias ll='ls -l'
#alias la='ls -A'
#alias l='ls -CF'

Commenting these out would make ll, la, and l work as commands during an interactive session, with the appropriate options added to the call.

You can check the aliases defined in your current shell session by typing alias with no arguments:

$ alias
alias ls='ls --color=auto'

Aliases are convenient ways to add options to commands, and are very common features of ~/.bashrc files shared on the web. They also work in POSIX-conforming shells besides Bash. However, for general use, they aren’t very sophisticated. For one thing, you can’t process arguments with them: 

# An attempt to write an alias that searches for a given pattern in a fixed
# file; doesn't work because aliases don't expand parameters
alias grepvim='grep "$1" ~/.vimrc'

They also don’t work for defining new commands within scripts for certain shells:

#!/bin/bash
alias ll='ls -l'
ll

When saved in a file as test, made executable, and run, this script fails:

./test: line 3: ll: command not found

So, once you understand how aliases work so you can read them when others define them in startup files, my suggestion is there’s no point writing any. Aside from some very niche evaluation tricks, they have no functional advantages over shell functions and scripts.

Functions

A more flexible method for defining custom commands for an interactive shell (or within a script) is to use a shell function. We could declare our ll function in a Bash startup file as a function instead of an alias like so:

# Shortcut to call ls(1) with the -l flag
ll() {
    command ls -l "$@"
}

Note the use of the command builtin here to specify that the ll function should invoke the program named ls, and not any function named ls. This is particularly important when writing a function wrapper around a command, to stop an infinite loop where the function calls itself indefinitely:

# Always add -q to invocations of gdb(1)
gdb() {
    command gdb -q "$@"
}

In both examples, note also the use of the "$@" expansion, to add to the final command line any arguments given to the function. We wrap it in double quotes to stop spaces and other shell metacharacters in the arguments causing problems. This means that the ll command will work correctly if you were to pass it further options and/or one or more directories as arguments:

$ ll -a
$ ll ~/.config

Shell functions declared in this way are specified by POSIX for Bourne-style shells, so they should work in your shell of choice, including Bash, dash, Korn shell, and Zsh. They can also be used within scripts, allowing you to abstract away multiple instances of similar commands to improve the clarity of your script, in much the same way the basics of functions work in general-purpose programming languages.

Functions are a good and portable way to approach adding features to your interactive shell; written carefully, they even allow you to port features you might like from other shells into your shell of choice. I’m fond of taking commands I like from Korn shell or Zsh and implementing them in Bash or POSIX shell functions, such as Zsh’s vared or its two-argument cd features.

If you end up writing a lot of shell functions, you should consider putting them into separate configuration subfiles to keep your shell’s primary startup file from becoming unmanageably large.

Examples from the author

You can take a look at some of the shell functions I have defined here that are useful to me in general shell usage; a lot of these amount to implementing convenience features that I wish my shell had, especially for quick directory navigation, or adding options to commands:

Other examples

Variables in shell functions

You can manipulate variables within shell functions, too:

# Print the filename of a path, stripping off its leading path and
# extension
fn() {
    name=$1
    name=${name##*/}
    name=${name%.*}
    printf '%s\n' "$name"
}

This works fine, but the catch is that after the function is done, the value for name will still be defined in the shell, and will overwrite whatever was in there previously:

$ printf '%s\n' "$name"
foobar
$ fn /home/you/Task_List.doc
Task_List
$ printf '%s\n' "$name"
Task_List

This may be desirable if you actually want the function to change some aspect of your current shell session, such as managing variables or changing the working directory. If you don’t want that, you will probably want to find some means of avoiding name collisions in your variables.

If your function is only for use with a shell that provides the local (Bash) or typeset (Ksh) features, you can declare the variable as local to the function to remove its global scope, to prevent this happening:

# Bash-like
fn() {
    local name
    name=$1
    name=${name##*/}
    name=${name%.*}
    printf '%s\n' "$name"
}

# Ksh-like
# Note different syntax for first line
function fn {
    typeset name
    name=$1
    name=${name##*/}
    name=${name%.*}
    printf '%s\n' "$name"
}

If you’re using a shell that lacks these features, or you want to aim for POSIX compatibility, things are a little trickier, since local function variables aren’t specified by the standard. One option is to use a subshell, so that the variables are only defined for the duration of the function:

# POSIX; note we're using plain parentheses rather than curly brackets, for
# a subshell
fn() (
    name=$1
    name=${name##*/}
    name=${name%.*}
    printf '%s\n' "$name"
)

# POSIX; alternative approach using command substitution:
fn() {
    printf '%s\n' "$(
        name=$1
        name=${name##*/}
        name=${name%.*}
        printf %s "$name"
    )"
}

This subshell method also allows you to change directory with cd within a function without changing the working directory of the user’s interactive shell, or to change shell options with set or Bash options with shopt only temporarily for the purposes of the function.

Another method to deal with variables is to manipulate the positional parameters directly ($1, $2 … ) with set, since they are local to the function call too:

# POSIX; using positional parameters
fn() {
    set -- "${1##*/}"
    set -- "${1%.*}"
    printf '%s\n' "$1"
}

These methods work well, and can sometimes even be combined, but they’re awkward to write, and harder to read than the modern shell versions. If you only need your functions to work with your modern shell, I recommend just using local or typeset. The Bash Guide on Greg’s Wiki has a very thorough breakdown of functions in Bash, if you want to read about this and other aspects of functions in more detail.

Keeping functions for later

As you get comfortable with defining and using functions during an interactive session, you might define them in ad-hoc ways on the command line for calling in a loop or some other similar circumstance, just to solve a task in that moment.

As an example, I recently made an ad-hoc function called monit to run a set of commands for its hostname argument that together established different types of monitoring system checks, using an existing script called nmfs:

$ monit() { nmfs "$1" Ping Y ; nmfs "$1" HTTP Y ; nmfs "$1" SNMP Y ; }
$ for host in webhost{1..10} ; do
> monit "$host"
> done

After that task was done, I realized I was likely to use the monit command interactively again, so I decided to keep it. Shell functions only last as long as the current shell, so if you want to make them permanent, you need to store their definitions somewhere in your startup files. If you’re using Bash, and you’re content to just add things to the end of your ~/.bashrc file, you could just do something like this:

$ declare -f monit >> ~/.bashrc

That would append the existing definition of monit in parseable form to your ~/.bashrc file, and the monit function would then be loaded and available to you for future interactive sessions. Later on, I ended up converting monit into a shell script, as its use wasn’t limited to just an interactive shell.

If you want a more robust approach to keeping functions like this for Bash permanently, I wrote a tool called Bashkeep, which allows you to quickly store functions and variables defined in your current shell into separate and appropriately-named files, including viewing and managing the list of names conveniently:

$ keep monit
$ keep
monit
$ ls ~/.bashkeep.d
monit.bash
$ keep -d monit

Scripts

Shell functions are a great way to portably customize behaviour you want for your interactive shell, but if a task isn’t specific only to an interactive shell context, you should instead consider putting it into its own script whether written in shell or not, to be invoked somewhere from your PATH. This makes the script useable in contexts besides an interactive shell with your personal configuration loaded, for example from within another script, by another user, or by an X11 session called by something like dmenu.

Even if your set of commands is only a few lines long, if you need to call it often–especially with reference to other scripts and in varying contexts– making it into a generally-available shell script has many advantages.

/usr/local/bin

Users making their own scripts often start by putting them in /usr/local/bin and making them executable with sudo chmod +x, since many Unix systems include this directory in the system PATH. If you want a script to be generally available to all users on a system, this is a reasonable approach. However, if the script is just something for your own personal use, or if you don’t have the permissions necessary to write to this system path, it may be preferable to have your own directory for logical binaries, including scripts.

Private bindir

Unix-like users who do this seem to vary in where they choose to put their private logical binaries directory. I’ve seen each of the below used or recommended:

  • ~/bin
  • ~/.bin
  • ~/.local/bin
  • ~/Scripts

I personally favour ~/.local/bin, but you can put your scripts wherever they best fit into your HOME directory layout. You may want to choose something that fits in well with the XDG standard, or whatever existing standard or system your distribution chooses for filesystem layout in $HOME.

In order to make this work, you will want to customize your login shell startup to include the directory in your PATH environment variable. It’s better to put this into ~/.profile or whichever file your shell runs on login, so that it’s only run once. That should be all that’s necessary, as PATH is typically exported as an environment variable for all the shell’s child processes. A line like this at the end of one of those scripts works well to extend the system PATH for our login shell:

PATH=$HOME/.local/bin:$PATH

Note that we specifically put our new path at the front of the PATH variable’s value, so that it’s the first directory searched for programs. This allows you to implement or install your own versions of programs with the same name as those in the system; this is useful, for example, if you like to experiment with building software in $HOME.

If you’re using a systemd-based GNU/Linux, and particularly if you’re using a display manager like GDM rather than a TTY login and startx for your X11 environment, you may find it more robust to instead set this variable with the appropriate systemd configuration file. Another option you may prefer on systems using PAM is to set it with pam_env(8).

After logging in, we first verify the directory is in place in the PATH variable:

$ printf '%s\n' "$PATH"
/home/tom/.local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games

We can test this is working correctly by placing a test script into the directory, including the #!/bin/sh shebang, and making it executable by the current user with chmod(1):

$ cat >~/.local/bin/test-private-bindir
#!/bin/sh
printf 'Working!\n'
^D
$ chmod u+x ~./local/bin/test-private-bindir
$ test-private-bindir
Working!

Examples from the author

I publish the more generic scripts I keep in ~/.local/bin, which I keep up-to-date on my personal systems in version control using Git, along with my configuration files. Many of the scripts are very short, and are intended mostly as building blocks for other scripts in the same directory. A few examples:

  • gscr(1df): Run a set of commands on a Git repository to minimize its size.
  • fgscr(1df): Find all Git repositories in a directory tree and run gscr(1df) over them.
  • hurl(1df): Extract URLs from links in an HTML document.
  • maybe(1df): Exit with success or failure with a given probability.
  • rfcr(1df): Download and read a given Request for Comments document.
  • tot(1df): Add up a list of numbers.

For such scripts, I try to write them as much as possible to use tools specified by POSIX, so that there’s a decent chance of them working on whatever Unix-like system I need them to.

On systems I use or manage, I might specify commands to do things relevant specifically to that system, such as:

  • Filter out uninteresting lines in an Apache HTTPD logfile with awk.
  • Check whether mail has been delivered to system users in /var/mail.
  • Upgrade the Adobe Flash player in a private Firefox instance.

The tasks you need to solve both generally and specifically will almost certainly be different; this is where you can get creative with your automation and abstraction.

X windows scripts

An additional advantage worth mentioning of using scripts rather than shell functions where possible is that they can be called from environments besides shells, such as in X11 or by other scripts. You can combine this method with X11-based utilities such as dmenu(1), libnotify’s notify-send(1), or ImageMagick’s import(1) to implement custom interactive behaviour for your X windows session, without having to write your own X11-interfacing code.

Other languages

Of course, you’re not limited to just shell scripts with this system; it might suit you to write a script completely in a language like awk(1), or even sed(1). If portability isn’t a concern for the particular script, you should use your favourite scripting language. Notably, don’t fall into the trap of implementing a script in shell for no reason …

#!/bin/sh
awk 'NF>2 && /foobar/ {print $1}' "$@"

… when you can instead write the whole script in the main language used, and save a fork(2) syscall and a layer of quoting:

#!/usr/bin/awk -f
NF>2 && /foobar/ {print $1}

Versioning and sharing

Finally, if you end up writing more than a couple of useful shell functions and scripts, you should consider versioning them with Git or a similar version control system. This also eases implementing your shell setup and scripts on other systems, and sharing them with others via publishing on GitHub. You might even go so far as to write a Makefile to install them, or manual pages for quick reference as documentation … if you’re just a little bit crazy …

Advanced Vim macros

Posted on by

Vim’s massive command set in both normal and command mode makes the concept of a macro especially powerful. Most Vim users, if they ever use macros at all, will perhaps quickly record an edit to one line, starting with qq and finishing with q, executing with @q, and otherwise never give the concept much thought.

For slightly more advanced Vim users, macros may be more frequently useful, but they perhaps still believe that macros are a niche Vim function unto themselves, and not really open to the deft manipulation of text that pretty much everything else in Vim is.

As is typical in Vim, the rabbit hole of functionality goes much deeper than most users will ever plumb.

Vanilla Vim macros

Suppose I had a plaintext table loaded into a Vim buffer containing the name, subject matter expertise, birth year, and nationality of a few well-known programmers:

Stallman  Richard GNU 1953  USA
Wall  Larry   Perl  1954  USA
Moolenar  Bram  Vim 1961  Netherlands
Tridgell  Andrew  Samba  1967  Australia
Matsumoto  Yukihiro  Ruby  1965  Japan
Ritchie  Dennis  C  1941  USA
Thompson  Ken  Unix  1943  USA

Actually, that’s kind of untidy. Let’s start by formatting it a bit, by running it through Unix’s column tool.

:%!column -t

Stallman   Richard   GNU    1953  USA
Wall       Larry     Perl   1954  USA
Moolenar   Bram      Vim    1961  Netherlands
Tridgell   Andrew    Samba  1967  Australia
Matsumoto  Yukihiro  Ruby   1965  Japan
Ritchie    Dennis    C      1941  USA
Thompson   Ken       Unix   1943  USA

May as well sort it by surname too:

:%!sort -k1

Matsumoto  Yukihiro  Ruby   1965  Japan
Moolenar   Bram      Vim    1961  Netherlands
Ritchie    Dennis    C      1941  USA
Stallman   Richard   GNU    1953  USA
Thompson   Ken       Unix   1943  USA
Tridgell   Andrew    Samba  1967  Australia
Wall       Larry     Perl   1954  USA

That’s better.

Now, suppose we’ve got the task of replacing the fourth column of this table with the approximate age of the person, which we can get naturally enough by sutracting their birth year from the current year. This is a little awkward to do in pure ex, so we’ll record a macro for doing it on one line.

Experimenting a bit, we find that the following works well:

03wdei^R=2012-^R"^M^[0j

Broken down, this does the following:

  • 0 — Move to the start of the line
  • 3w — Skip three words, in this case to the fourth column
  • de — Delete to the end of the word
  • i — Enter insert mode
  • ^R= — Insert the contents of the special = register, which accepts an expression to evaluate
  • 2012-^R"^M — Enter the expression 2012-(birth year) and press Enter (literal ^M), which completes the operation, and inserts the result
  • ^[ — Leave insert mode
  • 0 — Return to the start of the line
  • j — Move down a line

So we record it into a macro a (to stand for “age”, no other reason) as we fix up the first line:

qa03wdei^R=2012-^R"^M^[0jq

Matsumoto  Yukihiro  Ruby   47  Japan
Moolenar   Bram      Vim    1961  Netherlands
Ritchie    Dennis    C      1941  USA
Stallman   Richard   GNU    1953  USA
Thompson   Ken       Unix   1943  USA
Tridgell   Andrew    Samba  1967  Australia
Wall       Larry     Perl   1954  USA

This now means that for each line, we can run the macro by just tapping @a a few times:

Matsumoto  Yukihiro  Ruby   47  Japan
Moolenar   Bram      Vim    51  Netherlands
Ritchie    Dennis    C      71  USA
Stallman   Richard   GNU    59  USA
Thompson   Ken       Unix   69  USA
Tridgell   Andrew    Samba  45  Australia
Wall       Larry     Perl   58  USA

That’s all pretty stock-standard macro stuff that you’ll likely have learned in other tutorials. The only thing that’s slightly voodoo (and certainly not vi-compatible) is the arithmetic done with the special = register. You can read about that in :help @=, if you’re curious.

Repeating Vim macros

As a first very simple hint, if you’re running a macro several times, don’t forget that you can prepend a count to it; in our case, 6@a would have fixed up all the remaining lines. To take advantage of this, it’s good practice to compose your macros so that they make sense when run multiple times; in the example above, note that the end of the macro is moving down onto the next line, ready to run the macro again if appropriate.

Similarly, if you’ve already run a macro once, you can run the same one again by just tapping the @ key twice, @@. This repeats the last run macro. Again, you can prepend a count to this, @a5@@.

The true nature of Vim macros

The registers that hold macros are the same registers that you probably use more frequently for operations like deleting, yanking, and pasting. Running a macro is simply instructing Vim to take the contents of a register and execute them as keystrokes, as if you were actually typing them. Thinking about macros this way has a couple of interesting consequences.

First of all, it means you needn’t restrict yourself to the single-keystroke vi commands for Vim when you compose macros. You can include ex commands as well, for example to run a substitution during a macro:

qb:s/foo/bar/g^Mq
@b

Secondly, and more interestingly, all of the operations that you can apply to registers in general work with what we normally call macros, with the old standards, delete, yank, and paste. You can test this with the example macro demonstrated above, by typing "ap, which will dump the raw text straight into the buffer. Also like other registers, it’ll show up in the output of the :registers command.

All this means is that like everything else in Vim, macros are just text, and all of Vim’s clever tools can be used to work with them.

Editing Vim macros in a buffer

If you’ve used macros even a little bit you’ll be very familiar with how frustrating they are to record. For some reason, as soon as you press qa to start recording a macro into register a, your normal fluency with Vim commands goes out the window and you keep making mistakes that will make the macro unsuitable to apply.

So, don’t compose macros on the fly. Just stop doing it. I recommend that if a macro is more than three keystrokes, you should compose it directly in a scratch Vim buffer so that you can test it iteratively.

Here’s an example using the macro above; suppose I realise partway through writing it that I made a mistake in typing 2011 instead of 2012. I finish recording the rest of the macro anyway, and dump the broken keystrokes into a new scratch buffer:

:enew
"ap

This gives me the contents of the macro in plain text in the buffer:

qa03wdei^R=2011-^R"^M^[0jq

So now all I have to do is change that bad year to 2012, and then yank the whole thing back into register a:

^"ay$

Now I can test it directly with @a on the appropriate file, and if it’s still wrong, I just jump back to my scratch buffer and keep fixing it up until it works.

One potential snag here is that you have to enter keystrokes like Ctrl+R as literal characters, but once you know you can enter any keystroke in Vim literally in insert or command mode by prefixing it with Ctrl+V, that isn’t really a problem. So to enter a literal Ctrl+R in insert mode, you type Ctrl+V, then Ctrl+R.

Running a Vim macro on a set of lines

It’s occasionally handy to be able to run a macro that you’ve got ready on a specific subset of lines of the file, or perhaps just for every line. Fortunately, there’s a way to do this, too.

Using the :normal command, you’re able to run macros from the ex command line:

:normal @a

All it takes is prefixing this with any sort of range definition to allow you to run a macro on any set of lines that you’re able to define.

Run the macro on each line of the whole buffer:

:% normal @a

Between lines 10 and 20:

:10,20 normal @a

On the lines in the current visual selection:

:'<,'> normal @a

On the lines containing the pattern vim:

:g/vim/ normal @a

When you get confident using this, :norm is a nice abbreviation to use.

Moving a Vim macro into a function

For really useful macros of your own devising, it’s occasionally handy to put it into a function for use in scripts or keystroke mappings. Here again the :normal command comes in handy.

Suppose I wanted to keep the age calculation macro defined above for later use on spreadsheets of this kind. I’d start by dumping it into a new buffer:

:enew
"ap

The macro appears as raw text:

03wdei^R=2012-^R"^M^[0j

I prefix it with a :normal call, and wrap a function definition around it:

function! CalculateAge()
    normal 03wdei^R=2012-^R"^M^[0j 
endfunction

Then all I need to do is include that in a file that gets loaded during Vim’s startup, possibly just .vimrc. I can call it directly from ex:

:call CalculateAge()

But given that I wanted it to be a quick-access macro, maybe it’s better to bind it to \a, or whatever your chosen <leader> character is:

nnoremap <leader>a :call CalculateAge()<CR>

Saving a Vim macro

If you want to have a macro always available to you, that is, always loaded into the appropriate register at startup, that can be done in your .vimrc file with a call to let to fill the register with the literal characters required:

let @a='03wdei^R=2012-^R"^M^[0j'

Appending extra keystrokes to a Vim macro

If you just want to tack extra keystrokes onto an existing macro and don’t care to edit it in a Vim buffer, you can do that by recording into it with its capital letter equivalent. So, if you wanted to add more keystrokes into the register b, start recording with qB, and finish with the usual q.

Recursive Vim macros

If you’re crazy enough to need this, and I never have, there’s an excellent Vim Tip for it. But personally, I think if you need recursion in your text processing then it’s time to bust out a real programming language and not Vimscript to solve your problem.

If the issue for which you think you need recursion is running a macro on every line of a buffer with an arbitrary number of lines, then you don’t need recursion; just record a one-line version of the macro and call it with :% normal @a to run it on every line.

Vim macro gotchas

Here are a few gotchas which will save you some frustration if you know about them ahead of time:

  • When you have a hammer, everything starts to look like a nail. Don’t try to solve problems with macros when there are better solutions in the ex command set, or available in external programs. See the Vim koans page for a couple of examples.
  • You need to insert keystrokes like Enter as literal Ctrl+M, so that they look like ^M. The convenience abbreviations in mappings, like <CR>, simply don’t work. You’re likely to find yourself chording Ctrl+V a lot.

  • Macros tend to stop, sometimes for apparently no reason and with no warning messages, as soon as they hit some sort of error. One particularly annoying instance of this is when you’re performing several substitutions in a macro, because if it doesn’t find any instances of a pattern it will throw an error and stop executing. You can prevent this by adding the e flag to the substitution call:

    :s/foo/bar/e

Thanks to user bairui for suggesting a safer alternative for yanking macro lines from buffers, which I’ve changed. He’s written a whole blog post replying to this one. It’s a good read if you’re interested in going in to even more depth about when to use macros in Vim.