Advanced Vim macros

Vim’s massive command set in both normal and command mode makes the concept of a macro especially powerful. Most Vim users, if they ever use macros at all, will perhaps quickly record an edit to one line, starting with qq and finishing with q, executing with @q, and otherwise never give the concept much thought.

For slightly more advanced Vim users, macros may be more frequently useful, but they perhaps still believe that macros are a niche Vim function unto themselves, and not really open to the deft manipulation of text that pretty much everything else in Vim is.

As is typical in Vim, the rabbit hole of functionality goes much deeper than most users will ever plumb.

Vanilla Vim macros

Suppose I had a plaintext table loaded into a Vim buffer containing the name, subject matter expertise, birth year, and nationality of a few well-known programmers:

Stallman  Richard GNU 1953  USA
Wall  Larry   Perl  1954  USA
Moolenar  Bram  Vim 1961  Netherlands
Tridgell  Andrew  Samba  1967  Australia
Matsumoto  Yukihiro  Ruby  1965  Japan
Ritchie  Dennis  C  1941  USA
Thompson  Ken  Unix  1943  USA

Actually, that’s kind of untidy. Let’s start by formatting it a bit, by running it through Unix’s column tool.

:%!column -t

Stallman   Richard   GNU    1953  USA
Wall       Larry     Perl   1954  USA
Moolenar   Bram      Vim    1961  Netherlands
Tridgell   Andrew    Samba  1967  Australia
Matsumoto  Yukihiro  Ruby   1965  Japan
Ritchie    Dennis    C      1941  USA
Thompson   Ken       Unix   1943  USA

May as well sort it by surname too:

:%!sort -k1

Matsumoto  Yukihiro  Ruby   1965  Japan
Moolenar   Bram      Vim    1961  Netherlands
Ritchie    Dennis    C      1941  USA
Stallman   Richard   GNU    1953  USA
Thompson   Ken       Unix   1943  USA
Tridgell   Andrew    Samba  1967  Australia
Wall       Larry     Perl   1954  USA

That’s better.

Now, suppose we’ve got the task of replacing the fourth column of this table with the approximate age of the person, which we can get naturally enough by sutracting their birth year from the current year. This is a little awkward to do in pure ex, so we’ll record a macro for doing it on one line.

Experimenting a bit, we find that the following works well:

03wdei^R=2012-^R"^M^[0j

Broken down, this does the following:

  • 0 — Move to the start of the line
  • 3w — Skip three words, in this case to the fourth column
  • de — Delete to the end of the word
  • i — Enter insert mode
  • ^R= — Insert the contents of the special = register, which accepts an expression to evaluate
  • 2012-^R"^M — Enter the expression 2012-(birth year) and press Enter (literal ^M), which completes the operation, and inserts the result
  • ^[ — Leave insert mode
  • 0 — Return to the start of the line
  • j — Move down a line

So we record it into a macro a (to stand for “age”, no other reason) as we fix up the first line:

qa03wdei^R=2012-^R"^M^[0jq

Matsumoto  Yukihiro  Ruby   47  Japan
Moolenar   Bram      Vim    1961  Netherlands
Ritchie    Dennis    C      1941  USA
Stallman   Richard   GNU    1953  USA
Thompson   Ken       Unix   1943  USA
Tridgell   Andrew    Samba  1967  Australia
Wall       Larry     Perl   1954  USA

This now means that for each line, we can run the macro by just tapping @a a few times:

Matsumoto  Yukihiro  Ruby   47  Japan
Moolenar   Bram      Vim    51  Netherlands
Ritchie    Dennis    C      71  USA
Stallman   Richard   GNU    59  USA
Thompson   Ken       Unix   69  USA
Tridgell   Andrew    Samba  45  Australia
Wall       Larry     Perl   58  USA

That’s all pretty stock-standard macro stuff that you’ll likely have learned in other tutorials. The only thing that’s slightly voodoo (and certainly not vi-compatible) is the arithmetic done with the special = register. You can read about that in :help @=, if you’re curious.

Repeating Vim macros

As a first very simple hint, if you’re running a macro several times, don’t forget that you can prepend a count to it; in our case, 6@a would have fixed up all the remaining lines. To take advantage of this, it’s good practice to compose your macros so that they make sense when run multiple times; in the example above, note that the end of the macro is moving down onto the next line, ready to run the macro again if appropriate.

Similarly, if you’ve already run a macro once, you can run the same one again by just tapping the @ key twice, @@. This repeats the last run macro. Again, you can prepend a count to this, @a5@@.

The true nature of Vim macros

The registers that hold macros are the same registers that you probably use more frequently for operations like deleting, yanking, and pasting. Running a macro is simply instructing Vim to take the contents of a register and execute them as keystrokes, as if you were actually typing them. Thinking about macros this way has a couple of interesting consequences.

First of all, it means you needn’t restrict yourself to the single-keystroke vi commands for Vim when you compose macros. You can include ex commands as well, for example to run a substitution during a macro:

qb:s/foo/bar/g^Mq
@b

Secondly, and more interestingly, all of the operations that you can apply to registers in general work with what we normally call macros, with the old standards, delete, yank, and paste. You can test this with the example macro demonstrated above, by typing "ap, which will dump the raw text straight into the buffer. Also like other registers, it’ll show up in the output of the :registers command.

All this means is that like everything else in Vim, macros are just text, and all of Vim’s clever tools can be used to work with them.

Editing Vim macros in a buffer

If you’ve used macros even a little bit you’ll be very familiar with how frustrating they are to record. For some reason, as soon as you press qa to start recording a macro into register a, your normal fluency with Vim commands goes out the window and you keep making mistakes that will make the macro unsuitable to apply.

So, don’t compose macros on the fly. Just stop doing it. I recommend that if a macro is more than three keystrokes, you should compose it directly in a scratch Vim buffer so that you can test it iteratively.

Here’s an example using the macro above; suppose I realise partway through writing it that I made a mistake in typing 2011 instead of 2012. I finish recording the rest of the macro anyway, and dump the broken keystrokes into a new scratch buffer:

:enew
"ap

This gives me the contents of the macro in plain text in the buffer:

qa03wdei^R=2011-^R"^M^[0jq

So now all I have to do is change that bad year to 2012, and then yank the whole thing back into register a:

^"ay$

Now I can test it directly with @a on the appropriate file, and if it’s still wrong, I just jump back to my scratch buffer and keep fixing it up until it works.

One potential snag here is that you have to enter keystrokes like Ctrl+R as literal characters, but once you know you can enter any keystroke in Vim literally in insert or command mode by prefixing it with Ctrl+V, that isn’t really a problem. So to enter a literal Ctrl+R in insert mode, you type Ctrl+V, then Ctrl+R.

Running a Vim macro on a set of lines

It’s occasionally handy to be able to run a macro that you’ve got ready on a specific subset of lines of the file, or perhaps just for every line. Fortunately, there’s a way to do this, too.

Using the :normal command, you’re able to run macros from the ex command line:

:normal @a

All it takes is prefixing this with any sort of range definition to allow you to run a macro on any set of lines that you’re able to define.

Run the macro on each line of the whole buffer:

:% normal @a

Between lines 10 and 20:

:10,20 normal @a

On the lines in the current visual selection:

:'<,'> normal @a

On the lines containing the pattern vim:

:g/vim/ normal @a

When you get confident using this, :norm is a nice abbreviation to use.

Moving a Vim macro into a function

For really useful macros of your own devising, it’s occasionally handy to put it into a function for use in scripts or keystroke mappings. Here again the :normal command comes in handy.

Suppose I wanted to keep the age calculation macro defined above for later use on spreadsheets of this kind. I’d start by dumping it into a new buffer:

:enew
"ap

The macro appears as raw text:

03wdei^R=2012-^R"^M^[0j

I prefix it with a :normal call, and wrap a function definition around it:

function! CalculateAge()
    normal 03wdei^R=2012-^R"^M^[0j 
endfunction

Then all I need to do is include that in a file that gets loaded during Vim’s startup, possibly just .vimrc. I can call it directly from ex:

:call CalculateAge()

But given that I wanted it to be a quick-access macro, maybe it’s better to bind it to \a, or whatever your chosen <leader> character is:

nnoremap <leader>a :call CalculateAge()<CR>

Saving a Vim macro

If you want to have a macro always available to you, that is, always loaded into the appropriate register at startup, that can be done in your .vimrc file with a call to let to fill the register with the literal characters required:

let @a='03wdei^R=2012-^R"^M^[0j'

Appending extra keystrokes to a Vim macro

If you just want to tack extra keystrokes onto an existing macro and don’t care to edit it in a Vim buffer, you can do that by recording into it with its capital letter equivalent. So, if you wanted to add more keystrokes into the register b, start recording with qB, and finish with the usual q.

Recursive Vim macros

If you’re crazy enough to need this, and I never have, there’s an excellent Vim Tip for it. But personally, I think if you need recursion in your text processing then it’s time to bust out a real programming language and not Vimscript to solve your problem.

If the issue for which you think you need recursion is running a macro on every line of a buffer with an arbitrary number of lines, then you don’t need recursion; just record a one-line version of the macro and call it with :% normal @a to run it on every line.

Vim macro gotchas

Here are a few gotchas which will save you some frustration if you know about them ahead of time:

  • When you have a hammer, everything starts to look like a nail. Don’t try to solve problems with macros when there are better solutions in the ex command set, or available in external programs. See the Vim koans page for a couple of examples.
  • You need to insert keystrokes like Enter as literal Ctrl+M, so that they look like ^M. The convenience abbreviations in mappings, like <CR>, simply don’t work. You’re likely to find yourself chording Ctrl+V a lot.
  • Macros tend to stop, sometimes for apparently no reason and with no warning messages, as soon as they hit some sort of error. One particularly annoying instance of this is when you’re performing several substitutions in a macro, because if it doesn’t find any instances of a pattern it will throw an error and stop executing. You can prevent this by adding the e flag to the substitution call:

    :s/foo/bar/e
    

Thanks to user bairui for suggesting a safer alternative for yanking macro lines from buffers, which I’ve changed. He’s written a whole blog post replying to this one. It’s a good read if you’re interested in going in to even more depth about when to use macros in Vim.

42 thoughts on “Advanced Vim macros

  1. This is impressive, but also rather insane. :) The moment you start working with code like “normal 03wdei^R=2012-^R”^M^[0j” you are using the wrong tool, IMO. The Unix toolbox has easier ways of solving this problem.

  2. Thanks! Very useful. Didn’t know about @= and "ap that’s really makes life much easier! :))

    @Mark Wilden: IMHO the idea of the post was to show how to write complex macros in an easy way. :))

  3. This is a really great post. I’m pretty new to Vim and it has an answer for everything…so much to learn, so little time.

    Thanks for taking the time to show me macros in Vim.

    • That’s the most thorough critique any of my posts have received so far! Thank you.

      First of all, I have changed the article to instead use your advice about a safer method of yanking text back into a macro. It’s something I’ve somehow never run into but I can definitely see potential problems for people who make the Enter key actually do something in normal mode aside from move down a line. I always liked "ayy then @a etc because it was easy to type, and the side-effect of moving down a line was I guess actually kind of useful for me.

      I like your Vimscript implementation of my macro. Very elegant. But the point was more for me to come up with some macro of sufficient length that I could demonstrate how a long macro could be manipulated. There’s almost certainly yet another way to do this with an external call to Awk, for example.

      I code exclusively on Unix, and so solve many of these problems with tools like awk and sort where I can, simply because for me they’re always available, independent of Vim version, and usually with a cleaner syntax. You’re right this doesn’t work for Windows users, but admittedly I just don’t know anything about Windows anymore and usually don’t think about it. Most of the rest of this blog is about the Unix-like environment.

      Granted, if this were an article about use of external commands vs Vim builtins then I would probably be a bit more circumspect in using them.

      You don’t go into any detail about where recursive macros are more useful than unwieldy, but I’ll give you the benefit of the doubt! I don’t feel comfortable talking about them as useful, as I learned them long ago and have never once had a good reason to use them. I suspect that most people who want to use a recursive macro simply want to run a macro on every line of a subset of lines, which I explain how to do.

      At any rate, I’ve linked to your critique in the article and I appreciate your thoughts.

  4. Very good artical. But i don’t know in cmmand ” %!column -t”, what does % used for? Whats the difference between%!column -tand!column -t`. Thanks.

    • Any prefix to a ! command specifies a range of lines in the buffer that should be used for the standard input of that command, and replaced with the standard output. In this case, the % is a shorthand 1,$, or the entire file.

  5. years ago i was stunned when a friend of mine extracted a table (tr/td) full of matrimonial data into a comma separated list using vim. I was even more stunned when he did it on 576 such files in the same macro. It saved me at least 15 days to convert a static site, one matrimonial query – create a HTML page, to a dynamic LAMP site.

  6. Pingback: Random nonsense #3 « Recursed

  7. It’s impressive, thanks for sharing. I’m kinda noob with Vim and I knew it was powerful but now I’ll be a Vim ninja! Thanks again.

    Greetings from Mexico City D:

    • You’re very welcome! I wasn’t expecting to get quite so many “international” visitors, it’s been great to see how this material is useful outside the Anglosphere as well.

  8. Hello, first thing – I’m a total ViM noob. that being said, I think I can improve your macro:

    03wdei^R=2012-^R"^M^[0j – since you already have 0 at the begining, couldn’t you remove it at the end? so it would look like this:

    03wdei^R=2012-^R"^M^[j

    I know it’s a tiny thing, but being a noob and looking around the web I want to see if I’m doing this right :)

    • Yes, the second 0 isn’t strictly necessary. However the macro as a whole is probably suboptimal; the point is it’s a reasonably awkward set of steps to run that would be better to automate with Vim.

  9. Pingback: Lazy Reading for 2012/06/10 – DragonFly BSD Digest

  10. Hello,

    Where do i run this ..? in command line mode ..? I have tried this macro., but i got “Not an editor command” : 03wdei^R=2012-^R”^M^[0j

    Thanks for sharing your experiments.

    –sg

  11. I’ve been a vi (now vim) user for decades now and almost never use macros, and I think the reason why is because most things that can be accomplished with them can also be done with ex commands (usually substitute). For example the following can handle your “replace the birth year with age” step. Since you can easily undo/modify/redo these commands I find it easier to get them right than macros.

    :% s/\d\{4}/\=2012-submatch(0)/
    

    notes

    • % on every line
    • s substitute
    • \d\{4} four consecutive digits
    • \=2012-submatch(0) expression subtract matched value from 2012
    • Yes, one of the grand ironies of advancing in Vim is that by the time you have really thorough knowledge of how macros work, you’ve also developed a proper understanding of Vimscript, so you end up using macros less.

      I often feel that macros are kind of a “brute-force” solution for quickly getting through an ad-hoc editing task that may very well be better done with external calls or Vimscript, as so many commenters here and on Reddit have suggested alternatives. But when it’s 1am and you’re tired and this text file needs to be formatted a certain way in the next few minutes, and you know how to do it almost automatically with normal mode commands and don’t want to look up the Vimscript to do it “properly”, macros are a godsend.

  12. Pingback: 2012w25 « blog.padowi.se

  13. Do you know of a way to record searches (with the forward-slash key)?

    For instance, I have a MySQL schema file that I want to change into a PostgreSQL schema file. MySQL uses AUTO_INCREMENT on columns (e.g. for IDs). I have a macro that creates a PostgreSQL “CREATE SEQUENCE” for this, but I have to break up the use of the macro and finding the next occurrence of an ID column and finding its table name.

    Some code may help. I want to change this:

    CREATE TABLE "foobar"(
      "id" int(11) NOT NULL AUTO_INCREMENT,
      ... any number of lines of table construction statements ...
    );
    CREATE TABLE "bazblat"(
      "id" int(11) NOT NULL AUTO_INCREMENT,
      ... any number of lines of table construction statements ...
    );
    

    … into this:

    CREATE TABLE "foobar"(
      "id" integer NOT NULL,
      ... any number of lines of table construction statements ...
    );
    CREATE SEQUENCE "foobar_id_seq"
      START WITH 1
      INCREMENT BY 1
      NO MINVALUE
      NO MAXVALUE
      CACHE 1;
    CREATE TABLE "bazblat"(
      "id" integer NOT NULL,
      ... any number of lines of table construction statements ...
    );
    CREATE SEQUENCE "bazblat_id_seq"
      START WITH 1
      INCREMENT BY 1
      NO MINVALUE
      NO MAXVALUE
      CACHE 1;
    

    As you may be able to see, I would need to search for each create table statement (/CREATE TABLE), then yank the table name (f”yf”), then find the end of the create table statement (/;), then start the create sequence statement using the yanked word in my buffer. I would then repeat this macro through the end of the file. However, Vim doesn’t allow the recording of the searches (/CREATE TABLE) as far as I can figure it.

  14. Hey there just wanted to give you a quick heads up.

    The words in your article seem to be running off the screen in Internet explorer. I’m not sure if this is a format issue or something to do with browser compatibility but I thought I’d post to let you know. The layout look great though! Hope you get the problem resolved soon. Thanks

  15. Pingback: My #noVIMember Link List « Binary Delusions

  16. Pingback: Compilado de enlace « programacion@droope

  17. Hey thanks, I’ve used Vim for a very long time but it is always great to learn more. Your advanced register usage article is great too. Thanks a lot for posting.

Leave a Reply

Your email address will not be published. Required fields are marked *

You can use Markdown if you want.

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>