Category Archives: Terminal

RSS with Newsbeuter

Posted on by
18

The recent announcement that Google Reader will no longer be available from July 1st has prompted many of its current users to look for alternative RSS reader applications. Despite the panic, there are plenty of other web-based and GUI options, but text user interface enthusiasts (and Arabesque readers) may find Newsbeuter worth a look in particular.

Newsbeuter reading an article

Newsbeuter refers to itself as “the Mutt of RSS readers”, alluding to its keystroke-driven ncurses(3) interface, plaintext configuration with many options, and extensive feature set. If you like the idea of using a client-side RSS reader in a terminal, then this may be ideal for you.

Having a client-side reader is particularly valuable if you follow feeds which aren’t available on the public internet, or if you would prefer to keep your subscriptions relatively private. While Google Reader’s search is very good, it’s also handy to have a local cache of feed items to search, which is a feature of Newsbeuter.

Installing Newsbeuter

Newsbeuter can be downloaded and built from source, or there are packages available in most Linux distributions. On Debian-derived systems, it’s available in the newsbeuter package:

# apt-get install newsbeuter

Newsbeuter will throw an error if you try to start it with no feeds defined. We’ll be fixing that shortly.

Exporting Google Reader feeds

If you’re using Google Reader, you should start by exporting your feeds in OPML format format using Google Takeout. You can do this by going to Reader Settings -> Import/Export -> Download your data through Takeout:

Export Google Reader feeds

This leads you to the Google Takeout page, and offers you a download of all of your Google Reader data, which you can retrieve by clicking Create Archive. The downloaded zip file will contain (within a couple of directories) a file called subscriptions.xml. This is the OPML file containing the URLs and categorizations of all the feeds to which you were subscribed. Save that somewhere accessible on the Linux or BSD machine on which you intend to run Newsbeuter.

Importing feeds into Newsbeuter

Once you have your subscriptions.xml file ready for import, you can import the data straight into Newsbeuter using the -i option:

$ newsbeuter -i subscriptions.xml
Import of subscriptions finished.

With this done, you should be able to start Newsbeuter with no options, and its main interface will start with the URLs to all your feeds:

Newsbeuter with imported URLs

You’ll note that none of these have any items yet; this is because the defaults for Newsbeuter are to fetch the articles only on demand, not automatically. You can start this process by pressing R for Reload All, at which point the titles of your feeds will appear, along with a count of their unread items:

Newsbeuter with feed titles and counts

Some useful keystrokes

From here, the basics are pretty intuitive; you can move around with the cursor keys, and select feeds and items within them with Enter. You can press q to move up a screen, and to quit the program; Q will quit unconditionally from any screen.

You can move to next and previous feed items with Ctrl-j and Ctrl-k. A nice quick way to read everything is to cycle through unread items across all feeds with n. You can save the complete text of an article with s, and search for articles matching a string (not a regular expression) with /.

You can press o to open the feed’s URL in a browser; this works fine if you’re using an X server, but you can also configure this to be a command-line browser like lynx if you’d prefer with the browser option in the configuration file. If you’re using PuTTY and you’re going to be copy-pasting URLs from your terminal window, it helps to make sure you’ve configured it to easily select URLs on double-click.

A complete list of all the keystrokes is available by pressing ?.

Managing feeds

Adding, removing, and tagging feeds is all done with the urls file. This might be saved in either ~/.config/newsbeuter/urls, or ~/.newsbeuter/urls. Either way, you can edit it directly within the program using E, which will start your $EDITOR to manage the URLs. Add and remove feed URLs, save the file, quit, and you’re done; Newsbeuter will reload its defined feeds automatically once the editor is closed.

Tags

If you imported your feeds from Google Reader and you were using folders to keep your feeds organised, you may note that in your urls file in Newsbeuter the names of the folders are included in quotes at the end of each line:

http://www.debian-administration.org/atom.xml "Tech"
http://www.jerkcity.com/jc.rss "Comics"
http://www.kiwiblog.co.nz/feed "Politics"

These are tags, Newsbeuter’s way of organising feeds non-hierarchically. If you have such tags defined, you can limit your view of feeds to a particular tag by pressing t to show only those matching feeds. You can press Ctrl-t to back out of that view and show all feeds again.

Creating a new tag is done by editing the urls file as above. Add the tag in quotes after the appropriate feed URLs. Note that you can have more than one tag for each URL:

http://www.debian-administration.org/atom.xml "Tech" "Debian"

Configuration

The Newsbeuter configuration file might be in either ~/.config/newsbeuter/config or ~/.newsbeuter/config. The following options might be useful:

  • auto-reload yes — Check all feeds for new items on startup, and periodically thereafter.
  • reload-time 30 — Re-check all feeds automatically every 30 minutes.
  • notify-beep yes — Send a console beep every time new items are found. You will probably only want this if you are dealing sensibly with bells, for example with a visual bell system in GNU Screen or tmux, otherwise you may find an audible bell annoying.
  • confirm-exit yes — Prompt before quitting. Tapping q to get to the top screen is a little error-prone, and it’s easy to quit accidentally.

The colorscheme for the application can also be customized here, and the keybindings too. See the Newsbeuter documentation for a complete list of configuration options.

User agents

You may find that some feeds don’t return any information when you use Newsbeuter, probably because the user agent string it sends is not recognised as an RSS reader. The feed for the Abstruse Goose comic is an example, as is Toothpaste for Dinner.

The easiest way to work around this is to make Newsbeuter identify itself as a better-known RSS reader. I’ve found that pretending to be Liferea works:

user-agent "Liferea/1.4.14 (Linux; en_US.UTF8; http://liferea.sf.net/)"

With this done and Newsbeuter restarted, the feeds seem more willing to yield their items for reading.

Daniel Aleksandersen points out in the comments that this is probably because Newsbeuter used a suspicious user agent string until his patch for 2.6. If you are using Newsbeuter 2.6 or newer, then you may not need to do the above.

Special feeds

If you can’t directly retrieve your feed from a URL, but need to generate it programatically from a script or use a tool like curl to retrieve it, you can use special exec: URLs in the urls file to manage this. For example, to retrieve an RSS feed of my work’s network changelist, I do something like this:

"exec:ssh work curl http://changelog.worknet/rss.xml"

This retrieves the feed using curl(1) over ssh(1), and presents it as a normal feed in Newsbeuter. Note the quotes are required for any command that includes spaces.

Though I will miss Google Reader, I’ve found Newsbeuter a great replacement, and it fits very nicely as a permanent window in my tmux(1) session. Hopefully you’ll find it suits you too, and works well with your terminal-based workflow.

TERM strings

Posted on by
17

A certain piece of very misleading advice is often given online to users having problems with the way certain command-line applications are displaying in their terminals. This is to suggest that the user change the value of their TERM environment variable from within the shell, doing something like this:

$ TERM=xterm-256color

This misinformation sometimes extends to suggesting that users put the forced TERM change into their shell startup scripts. The reason this is such a bad idea is that it forces your shell to assume what your terminal is, and thereby disregards the initial terminal identity string sent by the emulator. This leads to a lot of confusion when one day you need to connect with a very different terminal emulator.

Accounting for differences

All terminal emulators are not created equal. Certainly, not all of them are xterm(1), although many other terminal emulators do a decent but not comprehensive job of copying it. The value of the TERM environment variable is used by the system running the shell to determine what the terminal connecting to it can and cannot do, what control codes to send to the program to use those features, and how the shell should understand the input of certain key codes, such as the Home and End keys. These things in particular are common causes of frustration for new users who turn out to be using a forced TERM string.

Instead, focus on these two guidelines for setting TERM:

  1. Avoid setting TERM from within the shell, especially in your startup scripts like .bashrc or .bash_profile. If that ever seems like the answer, then you are probably asking the wrong question! The terminal identification string should always be sent by the terminal emulator you are using; if you do need to change it, then change it in the settings for the emulator.

  2. Always use an appropriate TERM string that accurately describes what your choice of terminal emulator can and cannot display. Don’t make an rxvt(1) terminal identify itself as xterm; don’t make a linux console identify itself as vt100; and don’t make an xterm(1) compiled without 256 color support refer to itself as xterm-256color.

In particular, note that sometimes for compatibility reasons, the default terminal identification used by an emulator is given as something generic like xterm, when in fact a more accurate or comprehensive terminal identity file is more than likely available for your particular choice of terminal emulator with a little searching.

An example that surprises a lot of people is the availability of the putty terminal identity file, when the application defaults to presenting itself as an imperfect xterm(1) emulator.

Configuring your emulator’s string

Before you change your terminal string in its settings, check whether the default it uses is already the correct one, with one of these:

$ echo $TERM
$ tset -q

Most builds of rxvt(1), for example, should already use the correct TERM string by default, such as rxvt-unicode-256color for builds with 256 colors and Unicode support.

Where to configure which TERM string your terminal uses will vary depending on the application. For xterm(1), your .Xresources file should contain a definition like the below:

XTerm*termName: xterm-256color

For rxvt(1), the syntax is similar:

URxvt*termName: rxvt-unicode-256color

Other GTK and Qt emulators sometimes include the setting somewhere in their preferences. Look for mentions of xterm, a common fallback default.

For Windows PuTTY, it’s configurable under the ”’Connections > Data”’ section:

Setting the terminal string in PuTTY

More detail about configuring PuTTY for connecting to modern systems can be found in my article on configuring PuTTY.

Testing your TERM string

On Linux systems, an easy way to test the terminal capabilities (particularly effects like colors and reverse video) is using the msgcat(1) utility:

$ msgcat --color=test

This will output a large number of tests of various features to the terminal, so that you can check their appearance is what you expect.

Finding appropriate terminfo(5) definitions

On Linux systems, the capabilities and behavior of various terminal types is described using terminfo(5) files, usually installed as part of the ncurses package. These files are often installed in /lib/terminfo or /usr/share/terminfo, in subdirectories by first letter.

In order to use a particular TERM string, an appropriate file must exist in one of these directories. On Debian-derived systems, a large collection of terminal types can be installed to the system with the ncurses-term package.

For example, the following variants of the rxvt terminal emulator are all available:

$ cd /usr/share/terminfo/r
$ ls rxvt*
rxvt-16color  rxvt-256color  rxvt-88color  rxvt-color  rxvt-cygwin
rxvt-cygwin-native  rxvt+pcfkeys  rxvt-unicode-256color  rxvt-xpm

Private and custom terminfo(5) files

If you connect to a system that doesn’t have a terminfo(5) definition to match the TERM definition for your particular terminal, you might get a message similar to this on login:

setterm: rxvt-unicode-256color: unknown terminal type
tput: unknown terminal "rxvt-unicode-256color"
$

If you’re not able to install the appropriate terminal definition system-wide, one technique is to use a private .terminfo directory in your home directory containing the definitions you need:

$ cd ~/.terminfo
$ find
.
./x
./x/xterm-256color
./x/xterm
./r
./r/rxvt-256color
./r/rxvt-unicode-256color
./r/rxvt
./s
./s/screen
./s/screen-256color
./p
./p/putty-256color
./p/putty

You can copy this to your home directory on the servers you manage with a tool like scp:

$ scp -r .terminfo server:

TERM and multiplexers

Terminal multiplexers like screen(1) and tmux(1) are special cases, and they cause perhaps the most confusion to people when inaccurate TERM strings are used. The tmux FAQ even opens by saying that most of the display problems reported by people are due to incorrect TERM settings, and a good portion of the codebase in both multiplexers is dedicated to negotiating the differences between terminal capacities.

This is because they are “terminals within terminals”, and provide their own functionality only within the bounds of what the outer terminal can do. In addition to this, they have their own type for terminals within them; both of them use screen and its variants, such as screen-256color.

It’s therefore very important to check that both the outer and inner definitions for TERM are correct. In .screenrc it usually suffices to use a line like the following:

term screen

Or in .tmux.conf:

set-option -g default-terminal screen

If the outer terminals you use consistently have 256 color capabilities, you may choose to use the screen-256color variant instead.

If you follow all of these guidelines, your terminal experience will be much smoother, as your terminal and your system will understand each other that much better. You may find that this fixes a lot of struggles with interactive tools like vim(1), for one thing, because if the application is able to divine things like the available color space directly from terminal information files, it saves you from having to include nasty hacks on the t_Co variable in your .vimrc.

Terminal colour tolerance

Posted on by
4

Using 256 colour terminals with applications like Vim and tmux is pretty much a no-brainer; it’s been well-supported on most GUI terminal emulators for ages, including Windows emulators like PuTTY, and gives you a much wider spectrum of colour with which to work and to apply useful features like syntax highlighting and contextual colours for shell prompts.

The problem is that not all terminals are created equal. Occasionally, you’ll find yourself needing to use a console, perhaps with the linux type, when your X server doesn’t start or you’re using a KVM. Following the principles of graceful degradation, it’s a good idea to arrange your terminal configuration and any other applications that normally take advantage of the wider 256 colour spectrum to start up normally with no errors, and to use sensible values instead as supported by the terminal.

Bash

You can check the number of colours supported by your $TERM setting using the tput command:

$ tput colors
256

On BSD-derived systems, the syntax is slightly different:

$ tput Co
256

This provides you with an accessible way to set up graceful degradation for any colours that you use in your .bashrc file or other startup scripts. By way of example, in my .bashrc file I include the following stanza that adjusts the colour of my prompt depending on whether I have root privileges:

if [[ $EUID -eq 0 ]]; then
    PS1=${COLOR_ROOT}${PS1}${COLOR_UNDO}
else
    PS1=${COLOR_USER}${PS1}${COLOR_UNDO}
fi

The variables contain the terminal escape codes used to start painting text in appropriate colours for the terminal type. I set these earlier on in the file by checking the output of tput colors:

COLORS=$(tput colors)

# Terminal supports 256 colours
if [[ $COLORS -ge 256 ]]; then
    COLOR_ROOT='\[\e[38;5;9m\]'
    COLOR_USER='\[\e[38;5;10m\]'
    COLOR_UNDO='\[\e[0m\]'

# Terminal supports only eight colours
elif [[ $COLORS -ge 8 ]]; then
    COLOR_ROOT='\[\e[1;31m\]'
    COLOR_USER='\[\e[1;32m\]'
    COLOR_UNDO='\[\e[0m\]'

# Terminal may not support colour at all
else
    COLOR_ROOT=
    COLOR_USER=
    COLOR_UNDO=
fi

The above conditional restricts the colour space to conform to what the terminal explicitly specifies is supported. If no colours at all are supported, the variables are empty, and the prompt is simply printed with no colour at all.

Vim

A familiar structure from a lot of Vim colorscheme definition files is:

if has("gui_running") || &t_Co >= 256
    ...
endif

This has the effect of checking that either gVim is running, or that the terminal is capable of printing 256 colours. You can therefore use this and similar structures to delineate different sections of the file to apply based on the number of colours supported by the terminal.

if has("gui_running") || &t_Co >= 256
    ...
elseif &t_Co >= 88
    ...
elseif &t_Co >= 8
    ...
endif

It’s not really a good idea to define this sort of stuff in your .vimrc file; colour information should go into your colour scheme file.

Tmux

Things are a little tricker in tmux. Conditionals based on shell output are supported in recent versions. If you have a particular line of your configuration that will only work on terminals that support a certain number of colours, you can prefix that line with an #if-shell call:

if-shell 'test `tput colors` -ge 256' 'set-option -g default-terminal "screen-256color"'

The above will only set the default terminal to screen-256color for new sessions if the test call returns a result greater than or equal to 256. Note that you need to wrap both the shell test and the configuration to run in quotes for this to work.

You can preface every applicable line with this test, or if you’d prefer to only run one test for efficiency reasons, you could arrange to load a separate local configuration file only if a single test passes:

if-shell 'test `tput colors` -ge 256' 'source-file ~/.tmux.256.conf'

Unfortunately, there’s a race condition in tmux that means that the first shell can be established before the default-terminal configuration item is actually applied, meaning that the default $TERM of screen is used for the first window, but subsequent windows are correctly created with the screen-256color setting for $TERM. One possible workaround for this is to preserve the old terminal’s name in an environment variable:

CONTAINING_TERM=$TERM
if-shell 'test `tput colors` -ge 256' 'set-option -g default-terminal "screen-256color"'

You can then check this variable’s value in your .bashrc file to see if it’s a terminal known to support 256 colours, and if it is, force the screen-256color terminal type for the shell:

case "$CONTAINING_TERM" in
    *256color) 
        TERM=screen-256color
        unset CONTAINING_TERM
        ;;
esac

This problem may well be fixed in future versions of tmux, but it still seems to be an issue even in the developing 1.7 version.

All of the above goes a long way to making your personal set of dotfiles more robust, so that when you need to bust them out on some older machine or less capable terminal, they’re more likely to work properly.

256 colour terminals

Posted on by
9

Using 256 colours in terminals is well-supported in Linux distributions these days, and also in Windows terminal emulators like PuTTY. Using 256 colours is great for Vim colorschemes in particular, but also very useful for Tmux colouring or any other terminal application where a slightly wider colour space might be valuable. Be warned that once you get this going reliably, there’s no going back if you spend a lot of time in the terminal.

Xterm

To set this up for xterm or emulators that use xterm as the default value for $TERM, such as xfce4-terminal or gnome-terminal, it generally suffices to check the options for your terminal emulator to ensure that it will allow 256 colors, and then use the TERM string xterm-256color for it.

An earlier version of this post suggested changing the TERM definition in .bashrc, which is generally not a good idea, even if bounded with conditionals as my example was. You should always set the terminal string in the emulator itself if possible, if you do it at all.

Be aware that older systems may not have terminfo definitions for this terminal, but you can always copy them in using a private .terminfo directory if need be.

Tmux

To use 256 colours in Tmux, you should set the default terminal in .tmux.conf to be screen-256color:

set -g default-terminal "screen-256color"

This will allow you to use color definitions like colour231 in your status lines and other configurations. Again, this particular terminfo definition may not be present on older systems, so you should copy it into ~/.terminfo/s/screen-256color on those systems if you want to use it everywhere.

GNU Screen

Similarly, to use 256 colours in GNU Screen, add the following to your .screenrc:

term screen-256color

Vim

With the applicable options from the above set, you should not need to change anything in Vim to be able to use 256-color colorschemes. If you’re wanting to write or update your own 256-colour compatible scheme, it should either begin with set t_Co=256, or more elegantly, check the value of the corresponding option value is &t_Co is 256 before trying to use any of the extra colour set.

The Vim Tips Wiki contains a detailed reference of the colour codes for schemes in 256-color terminals.

Terminal annoyances

Posted on by
3

Largely for legacy support reasons, terminal emulators and consoles include a few features that were useful on older terminal systems, but that are not needed or even a hindrance today. Perhaps chief among these is the legacy of flow control characters, referred to in the stty manual as “XON/XOFF flow control”. With this enabled, as it is by default on many systems, pressing Ctrl+S in a terminal will prevent both input and output of characters until terminal writing is resumed again with Ctrl+Q. For people not aware of this feature this can be disconcerting, as the terminal apparently simply freezes and won’t come unstuck; if you’re on an SSH connection you may incorrectly think there’s a network problem.

While this may still have some limited applications, with slow serial terminals more or less a thing of the past it’s mostly just annoying, particularly when using applications that are very heavy on Ctrl chords, and especially when you use GNU Screen or some other terminal multiplexer with Ctrl+A as the prefix key that invites fat-fingering Ctrl+S. You can turn flow control off completely by including this stty call in your .bashrc:

stty -ixon

This also frees these keys up for other uses, such as being bound in Vim or Emacs, or for the forward incremental history search feature in Bash, which I find much more useful.

Similarly annoying is the beep emitted from the PC speaker under some circumstances when using the console, for example when pressing Tab to invoke Bash autocompletion when no further completions are available. I also prefer to turn this off, which can be done with:

setterm -bfreq 0

I don’t find myself missing either feature, though I’m told a few people still find flow control handy in some circumstances. There’s ongoing interest in disabling the feature by default in certain distributions.