Packaging built software

The Debian package repository is enormous, and the Sid distribution is in most cases reasonably up-to-date, but it’s still sometimes desirable to build an application and install it into /usr/local when a packaged implementation either isn’t available or is too out of date, or if you’re involved in the development of a project and want to try out a fresh build that hasn’t been packaged yet.

The usual cycle of configuring, compiling, and installing many open-source applications for Unix-like systems applies here:

$ ./configure
$ make
# make install

The above is normally the approach taken to install code compiled on the machine, rather than through packages. One problem with this approach is that it doesn’t allow many of the advantages that a system running purely on packages does; dpkg -l will no longer give you a complete overview of all the system’s software, and to remove the software and its configuration files you may have to manually delete it rather than using apt-get purge.

Fortunately, there exists a tool called checkinstall to allow having the best of both worlds. After installing this tool via apt-get install checkinstall, you’re able to build a package for your locally built software according to the rules defined in its Makefile, and install that the same way as any other package.

Instead of typing make install, type checkinstall, and you will be prompted for details about the package, including its description, which is then built and installed. In this example, I’m compiling Vim from source, which works very well. I’ve also successfully installed tools like pam_ssh_agent_auth and tmux this way.

With this done, the package’s files are installed in /usr/local, and it appears in the list of installed packages along with my explanation of its contents:

$ ./configure
$ make
$ sudo -s
# checkinstall
...
*****************************************
**** Debian package creation selected ***
*****************************************

This package will be built according to these values:

0 -  Maintainer: [ Tom Ryder <tom@sanctum.geek.nz> ]
1 -  Summary: [ Custom build of latest Vim 7.3 ]
2 -  Name:    [ vim ]
3 -  Version: [ 2:7.4 ]
4 -  Release: [ 1 ]
5 -  License: [ GPL ]
6 -  Group:   [ checkinstall ]
7 -  Architecture: [ amd64 ]
8 -  Source location: [ vim ]
9 -  Alternate source location: [  ]
10 - Requires: [  ]
11 - Provides: [ vim ]
12 - Conflicts: [  ]
13 - Replaces: [  ]

Note that I’m assigning it a version number greater than the Debian repository’s vim package, which is a simple way to prevent it being replaced. You can also do this via the /etc/apt/preferences file to prevent replacement of all packages from the checkinstall group.

With this done, my custom build of Vim now shows in the package list, and its files are correctly installed in /usr/local:

# dpkg -l | grep vim
ii  vim  2:7.4-1 Custom build of latest Vim 7.3
# dpkg -S vim
vim: /usr/local/share/vim/vim73/bugreport.vim
vim: /usr/local/share/vim/vim73/plugin
vim: /usr/local/share/vim/vim73/ftplugin/postscr.vim
vim: /usr/local/share/man/it.UTF-8/man1/vim.1.gz
...

The .deb package built by checkinstall is also present in my build directory for me to keep for later, or for installation on another compatible server:

$ ls *.deb
vim_7.4-1_amd64.deb

It’s worth noting that checkinstall is not a Debian-specific tool; it works for other packaging systems like RPM and Slackware, too.

Unix as IDE: Building

Because compiling projects can be such a complicated and repetitive process, a good IDE provides a means to abstract, simplify, and even automate software builds. Unix and its descendents accomplish this process with a Makefile, a prescribed recipe in a standard format for generating executable files from source and object files, taking account of changes to only rebuild what’s necessary to prevent costly recompilation.

One interesting thing to note about make is that while it’s generally used for compiled software build automation and has many shortcuts to that effect, it can actually effectively be used for any situation in which it’s required to generate one set of files from another. One possible use is to generate web-friendly optimised graphics from source files for deployment for a website; another use is for generating static HTML pages from code, rather than generating pages on the fly. It’s on the basis of this more flexible understanding of software “building” that modern takes on the tool like Ruby’s rake have become popular, automating the general tasks for producing and installing code and files of all kinds.

Anatomy of a Makefile

The general pattern of a Makefile is a list of variables and a list of targets, and the sources and/or objects used to provide them. Targets may not necessarily be linked binaries; they could also constitute actions to perform using the generated files, such as install to instate built files into the system, and clean to remove built files from the source tree.

It’s this flexibility of targets that enables make to automate any sort of task relevant to assembling a production build of software; not just the typical parsing, preprocessing, compiling proper and linking steps performed by the compiler, but also running tests (make test), compiling documentation source files into one or more appropriate formats, or automating deployment of code into production systems, for example, uploading to a website via a git push or similar content-tracking method.

An example Makefile for a simple software project might look something like the below:

all: example

example: main.o example.o library.o
    gcc main.o example.o library.o -o example

main.o: main.c
    gcc -c main.c -o main.o

example.o: example.c
    gcc -c example.c -o example.o

library.o: library.c
    gcc -c library.c -o library.o

clean:
    rm *.o example

install: example
    cp example /usr/bin

The above isn’t the most optimal Makefile possible for this project, but it provides a means to build and install a linked binary simply by typing make. Each target definition contains a list of the dependencies required for the command that follows; this means that the definitions can appear in any order, and the call to make will call the relevant commands in the appropriate order.

Much of the above is needlessly verbose or repetitive; for example, if an object file is built directly from a single C file of the same name, then we don’t need to include the target at all, and make will sort things out for us. Similarly, it would make sense to put some of the more repeated calls into variables so that we would not have to change them individually if our choice of compiler or flags changed. A more concise version might look like the following:

CC = gcc
OBJECTS = main.o example.o library.o
BINARY = example

all: example

example: $(OBJECTS)
    $(CC) $(OBJECTS) -o $(BINARY)

clean:
    rm -f $(BINARY) $(OBJECTS)

install: example
    cp $(BINARY) /usr/bin

More general uses of make

In the interests of automation, however, it’s instructive to think of this a bit more generally than just code compilation and linking. An example could be for a simple web project involving deploying PHP to a live webserver. This is not normally a task people associate with the use of make, but the principles are the same; with the source in place and ready to go, we have certain targets to meet for the build.

PHP files don’t require compilation, of course, but web assets often do. An example that will be familiar to web developers is the generation of scaled and optimised raster images from vector source files, for deployment to the web. You keep and version your original source file, and when it comes time to deploy, you generate a web-friendly version of it.

Let’s assume for this particular project that there’s a set of four icons used throughout the site, sized to 64 by 64 pixels. We have the source files to hand in SVG vector format, safely tucked away in version control, and now need to generate the smaller bitmaps for the site, ready for deployment. We could therefore define a target icons, set the dependencies, and type out the commands to perform. This is where command line tools in Unix really begin to shine in use with Makefile syntax:

icons: create.png read.png update.png delete.png

create.png: create.svg
    convert create.svg create.raw.png && \
    pngcrush create.raw.png create.png

read.png: read.svg
    convert read.svg read.raw.png && \
    pngcrush read.raw.png read.png

update.png: update.svg
    convert update.svg update.raw.png && \
    pngcrush update.raw.png update.png

delete.png: delete.svg
    convert delete.svg delete.raw.png && \
    pngcrush delete.raw.png delete.png

With the above done, typing make icons will go through each of the source icons files in a Bash loop, convert them from SVG to PNG using ImageMagick’s convert, and optimise them with pngcrush, to produce images ready for upload.

A similar approach can be used for generating help files in various forms, for example, generating HTML files from Markdown source:

docs: README.html credits.html

README.html: README.md
    markdown README.md > README.html

credits.html: credits.md
    markdown credits.md > credits.html

And perhaps finally deploying a website with git push web, but only after the icons are rasterized and the documents converted:

deploy: icons docs
    git push web

For a more compact and abstract formula for turning a file of one suffix into another, you can use the .SUFFIXES pragma to define these using special symbols. The code for converting icons could look like this; in this case, $< refers to the source file, $* to the filename with no extension, and $@ to the target.

icons: create.png read.png update.png delete.png

.SUFFIXES: .svg .png

.svg.png:
    convert $< $*.raw.png && \
    pngcrush $*.raw.png $@

Tools for building a Makefile

A variety of tools exist in the GNU Autotools toolchain for the construction of configure scripts and make files for larger software projects at a higher level, in particular autoconf and automake. The use of these tools allows generating configure scripts and make files covering very large source bases, reducing the necessity of building otherwise extensive makefiles manually, and automating steps taken to ensure the source remains compatible and compilable on a variety of operating systems.

Covering this complex process would be a series of posts in its own right, and is out of scope of this survey.

Thanks to user samwyse for the .SUFFIXES suggestion in the comments.

This entry is part 5 of 7 in the series Unix as IDE.