Additional sshd ports

Occasionally you may find yourself using a network behind a firewall that doesn’t allow outgoing TCP connections with a destination port of 22, meaning you’re unable to connect to your OpenSSH server, perhaps to take advantage of a SOCKS proxy for encrypted and unfiltered web browsing.

Since these restricted networks almost always allow port 443 out, since it’s the destination port for outgoing HTTPS requests, an easy workaround is to have your OpenSSH server listen on port 443 if it isn’t already using the port.

This is sometimes given as a rationale for changing the sshd port completely, but you don’t need to do that; you can simply add another Port directive to sshd_config(5):

Port 22
Port 443

After restarting the OpenSSH server with this new line in place, you can verify that it’s listening with ss(8) or netstat(8)

# ss -lnp src :22
State      Recv-Q Send-Q    Local Address:Port      Peer Address:Port
LISTEN     0      128                  :::22                  :::*
users:(("sshd",3039,6))
LISTEN     0      128                   *:22                   *:*
users:(("sshd",3039,5))
# ss -lnp src :443
State      Recv-Q Send-Q    Local Address:Port      Peer Address:Port
LISTEN     0      128                  :::443                 :::*
users:(("sshd",3039,4))
LISTEN     0      128                   *:443                  *:*
users:(("sshd",3039,3))

You’ll then be able to connect to the server on port 443, the same way you would on port 22. If you intend this setup to be permanent, it would be a good idea to save the configuration in your ssh_config(5) file, or whichever SSH client you happen to use.

SSH, SOCKS, and cURL

Port forwarding using SSH tunnels is a convenient way to circumvent well-intentioned firewall rules, or to access resources on otherwise unaddressable networks, particularly those behind NAT (with addresses such as 192.168.0.1).

However, it has a shortcoming in that it only allows us to address a specific host and port on the remote end of the connection; if we forward a local port to machine A on the remote subnet, we can’t also reach machine B unless we forward another port. Fetching documents from a single server therefore works just fine, but browsing multiple resources over the endpoint is a hassle.

The proper way to do this, if possible, is to have a VPN connection into the appropriate network, whether via a virtual interface or a network route through an IPsec tunnel. In cases where this isn’t possible or practicable, we can use a SOCKS proxy set up via an SSH connection to delegate all kinds of network connections through a remote machine, using its exact network stack, provided our client application supports it.

Being command-line junkies, we’ll show how to set the tunnel up with ssh and to retrieve resources on it via curl, but of course graphical browsers are able to use SOCKS proxies as well.

As an added benefit, using this for browsing implicitly encrypts all of the traffic up to the remote endpoint of the SSH connection, including the addresses of the machines you’re contacting; it’s thus a useful way to protect unencrypted traffic from snoopers on your local network, or to circumvent firewall policies.

Establishing the tunnel

First of all we’ll make an SSH connection to the machine we’d like to act as a SOCKS proxy, which has access to the network services that we don’t. Perhaps it’s the only publically addressable machine in the network.

$ ssh -fN -D localhost:8001 remote.example.com

In this example, we’re backgrounding the connection immediately with -f, and explicitly saying we don’t intend to run a command or shell with -N. We’re only interested in establishing the tunnel.

Of course, if you do want a shell as well, you can leave these options out:

$ ssh -D localhost:8001 remote.example.com

If the tunnel setup fails, check that AllowTcpForwarding is set to yes in /etc/ssh/sshd_config on the remote machine:

AllowTcpForwarding yes

Note that in both cases we use localhost rather than 127.0.0.1, in order to establish both IPv4 and IPv6 sockets if appropriate.

We can then check that the tunnel is established with ss on GNU/Linux:

# ss dst :8001
State      Recv-Q Send-Q   Local Address:Port       Peer Address:Port
ESTAB      0      0            127.0.0.1:45666         127.0.0.1:8001
ESTAB      0      0            127.0.0.1:45656         127.0.0.1:8001
ESTAB      0      0            127.0.0.1:45654         127.0.0.1:8001

Requesting documents

Now that we have a SOCKS proxy running on the far end of the tunnel, we can use it to retrieve documents from some of the servers that are otherwise inaccessible. For example, when we were trying to run this from the client side, we found it wouldn’t work:

$ curl http://private.example/contacts.html
curl: (6) Couldn't resolve host 'private.example'

This is because the example subnet is on a remote and unroutable LAN. If its name comes from a private DNS server, we may not even be able to resolve its address, let alone retrieve the document.

We can fix both problems with our local SOCKS proxy, by pointing curl to it with its --proxy option:

$ curl --proxy socks5h://localhost:8001 http://private.example/contacts.html
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<html>
    <head>
        <title>Contacts</title>
...

Older versions of curl may need to use the --socks5-hostname option:

$ curl --socks5-hostname localhost:8001 http://private.example/contacts.html

This not only tunnels our HTTP request through to remote.example.com and returns any response, it does the DNS lookup on the other end too. This means we can not only retrieve documents from remote servers, we can resolve their hostnames too, even if our client side can’t contact the appropriate DNS server on its own. This is what the h suffix does in the socks5h:// URI syntax above.

We can configure graphical web browsers to use the SOCKS proxy in the same way, optionally including DNS resolution:

Browsers are not the only application that can use SOCKS proxies; many IM clients such as Pidgin and Bitlbee can use them too, for example.

Making things more permanent

If this all works for you and you’d like to set up the SOCKS proxy on the far end each time you connect, you can add it to your ssh_config file in $HOME/.ssh/config:

Host remote.example.com
    DynamicForward localhost:8001

With this done, you should only need to type the hostname of the machine to get a shell and to set up the dynamic forward in the background:

$ ssh remote.example.com

Restricting public keys

It may be the case that while you’re happy to allow a user or process to have public key authentication access to your server via the ~/.ssh/authorized_keys file, you don’t necessarily want to give them a full shell, or you may want to restrict them from doing things like SSH port forwarding or X11 forwarding.

One method that’s supposed to prevent users from accessing a shell is by defining their shell in /etc/passwd as /bin/false, which does indeed prevent them from logging in with the usual ssh or ssh command syntax. This isn’t a good approach because it still allows port forwarding and other SSH-enabled services.

If you want to restrict the use of logins with a public key, you can prepend option pairs to its line in the authorized_keys file. Some of the most useful options here include:

  • from="<hostname/ip>" — Prepending from="*.example.com" to the key line would only allow public-key authenticated login if the connection was coming from some host with a reverse DNS of example.com. You can also put IP addresses in here. This is particularly useful for setting up automated processes through keys with null passphrases.
  • command="<command>" — Means that once authenticated, the command specified is run, and the connection is closed. Again, this is useful in automated setups for running only a certain script on successful authentication, and nothing else.
  • no-agent-forwarding — Prevents the key user from forwarding authentication requests to an SSH agent on their client, using the -A or ForwardAgent option to ssh.
  • no-port-forwarding — Prevents the key user from forwarding ports using -L and -R.
  • no-X11-forwarding — Prevents the key user from forwarding X11 processes.
  • no-pty — Prevents the key user from being allocated a tty device at all.

So, for example, a public key that is only used to run a script called runscript on the server by the client runscript@client.example:

command="runscript",client="client.example",no-pty,no-agent-forwarding,no-port-forwarding ssh-rsa AAAAB2....19Q runscript@client.example

A public key for a user whom you were happy to allow to log in from anywhere with a full shell, but did not want to allow agent, port, or X11 forwarding:

no-agent-forwarding,no-port-forwarding,no-X11-forwarding ssh-rsa AAAAD3....19Q user@client.example

Use of these options goes a long way to making your public key authentication setup harder to exploit, and is very consistent with the principle of least privilege. To see a complete list of the options available, check out the man page for sshd.

SSH agents

Public key authentication has a lot of advantages for connecting to servers, particularly if it’s the only allowed means of authentication, reducing the chances of a brute force password attack to zero. However, it doesn’t solve the problem of having to type in a password or passphrase on each connection, unless you’re using a private key with no passphrase, which is quite risky if the private key is compromised.

Thankfully, there’s a nice supplement to a well-secured SSH key setup in the use of agents on trusted boxes to securely store decrypted keys per-session, per-user. Judicious use of an SSH agent program on a trusted machine allows you to connect to any server for which your public key is authorised by typing your passphrase to decrypt your private key only once.

SSH agent setup

The ssh-agent program is designed as a wrapper for a shell. If you have a private and public key setup ready, and you have remote machines for which your key is authorised, you can get an idea of how the agent works by typing:

$ ssh-agent bash

This will prompt you for your passphrase, and once entered, within the context of that subshell, you will be able to connect to authorised remote servers without typing in the passphrase again. Once loaded, you can examine the identities you have by using ssh-add -l to see the fingerprints, and ssh-add -L for the public keys:

$ ssh-agent bash
Enter passphrase for /home/user/.ssh/id_rsa:
Identity added: /home/user/.ssh/id_rsa (/home/user/.ssh/id_rsa)
$ ssh-add -l
2048 07:1e:7d:c4:8a:0e:bc:b0:74:40:71:49:7c:70:9c /home/user/.ssh/id_rsa (RSA)
$ ssh-add -L
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+WvWXmVPx6UYB/uf+HTh1Y5zEVOmSeFfj6IC0fwN
lELVoFco9qdM4cuh6E6UaDURezjLSiayKt237DFHMgK9Hp4QPgN3ZJ7f7mesH7EHRnpLcvt0Rl3k1I4
C6gConwmkPZj3ax/cr6DAI9v7Ggeo7YPdKYhntB4TCEZfXlfihF5Vh5A2Od8cCNqy5KFKsFaLoI8Gwr
+ZC0CoxIoW6t5t6C/ZNRK2ojVwRWvp3nxcZsOzSdZJu3jcNHGSr0fxpdythRrOjzdDHgCiBuH+7mGKa
tLewbchdj8AgdeCE410xDJkov+tQuGYXZQAOx+JzWgiDI0VzWZsaV2QuyEF4NyG/
/home/user/.ssh/id_rsa

You can set up your .bashrc file to automatically search for accessible SSH agents to use for the credentials for new connections, and to prompt you for a passphrase to open a new one if it need be. There are very workable instructions on GitHub for setting this up.

If you want to shut down the agent at any time, you can use ssh-agent -k.

$ ssh-agent -k
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 790 killed;

SSH agent forwarding

Where the configuration of the remote machine allows it, you can forward authentication requests made from the remote machine back to the agent on your workstation. This is handy for working with semi-trusted gateway machines that you trust to forward your authentication requests correctly, but on which you’d prefer not to put your private key.

This means that if you connect to a remote machine from your workstation running an SSH agent with the following, using the -A parameter:

user@workstation:~$ ssh -A remote.example.com

You can then connect to another machine from remote.example.com using your private key on workstation:

user@remote:~$ ssh another.example.com

SSH agent authentication via PAM

It’s also possible to use SSH agent authentication as a PAM method for general authentication, such as for sudo, using pam_ssh_agent_auth.

Uses for ~/.ssh/config

For system and network administrators or other users who frequently deal with sessions on multiple machines, SSH ends up being one of the most oft-used Unix tools. SSH usually works so well that until you use it for something slightly more complex than starting a terminal session on a remote machine, you tend to use it fairly automatically. However, the ~/.ssh/config file bears mentioning for a few ways it can make using the ssh a client a little easier.

Abbreviating hostnames

If you often have to SSH into a machine with a long host and/or network name, it can get irritating to type it every time. For example, consider the following command:

$ ssh web0911.colo.sta.solutionnetworkgroup.com

If you interact with the web0911 machine a lot, you could include a stanza like this in your ~/.ssh/config:

Host web0911
    HostName web0911.colo.sta.solutionnetworkgroup.com

This would allow you to just type the following for the same result:

$ ssh web0911

Of course, if you have root access on the system, you could also do this by adding the hostname to your /etc/hosts file, or by adding the domain to your /etc/resolv.conf to search it, but I prefer the above solution as it’s cleaner and doesn’t apply system-wide.

Fixing alternative ports

If any of the hosts with which you interact have SSH processes listening on alternative ports, it can be a pain to both remember the port number and to type it in every time:

$ ssh webserver.example.com -p 5331

You can affix this port permanently into your .ssh/config file instead:

Host webserver.example.com
    Port 5331

This will allow you to leave out the port definition when you call ssh on that host:

$ ssh webserver.example.com

Custom identity files

If you have a private/public key setup working between your client machine and the server, but for whatever reason you need to use a different key from your normal one, you’ll be using the -i flag to specify the key pair that should be used for the connection:

$ ssh -i ~/.ssh/id_dsa.mail srv1.mail.example.com
$ ssh -i ~/.ssh/id_dsa.mail srv2.mail.example.com

You can specify a fixed identity file in .ssh/config just for these hosts instead, using an asterisk to match everything in that domain:

Host *.mail.example.com
    IdentityFile ~/.ssh/id_dsa.mail

I need to do this for Mikrotik’s RouterOS connections, as my own private key structure is 2048-bit RSA which RouterOS doesn’t support, so I keep a DSA key as well just for that purpose.

Logging in as a different user

By default, if you omit a username, SSH assumes the username on the remote machine is the same as the local one, so for servers on which I’m called tom, I can just type:

tom@conan:$ ssh server.network

However, on some machines I might be known as a different username, and hence need to remember to connect with one of the following:

tom@conan:$ ssh -l tomryder server.anothernetwork
tom@conan:$ ssh tomryder@server.anothernetwork

If I always connect as the same user, it makes sense to put that into my .ssh/config instead, so I can leave it out of the command entirely:

Host server.anothernetwork
    User tomryder

SSH proxies

If you have an SSH server that’s only accessible to you via an SSH session on an intermediate machine, which is a very common situation when dealing with remote networks using private RFC1918 addresses through network address translation, you can automate that in .ssh/config too. Say you can’t reach the host nathost directly, but you can reach some other SSH server on the same private subnet that is publically accessible, publichost.example.com:

Host nathost
    ProxyCommand ssh -q -W %h:%p public.example.com

This will allow you to just type:

$ ssh nathost

More information

The above are the .ssh/config settings most useful to me, but there are plenty more available; check man ssh_config for a complete list.

SSH tunnels and escapes

Quite apart from replacing Telnet and other insecure protocols as the primary means of choice for contacting and administrating services, the OpenSSH implementation of the SSH protocol has developed into a general-purpose toolbox for all kinds of well-secured communication, whether using both simple challenge-response authentication in the form of user and password logins, or for more complex public key authentication.

SSH is useful in a general sense for tunnelling pretty much any kind of TCP traffic, and doing so securely and with appropriate authentication. This can be used both for ad-hoc purposes such as talking to a process on a remote host that’s only listening locally or within a secured network, or for bypassing restrictive firewall rules, to more stable implementations such as setting up a persistent SSH tunnel between two machines to ensure sensitive traffic that might otherwise be sent in cleartext is not only encrypted but authenticated. I’ll discuss a couple of simple examples here, in addition to talking about the SSH escape sequences, about which I don’t seem to have seen very much information online.

SSH tunnelling for port forwarding

Suppose you’re at work or on a client site and you need some information off a webserver on your network at home, perhaps a private wiki you run, or a bug tracker or version control repository. This being private information, and your HTTP daemon perhaps not the most secure in the world, the server only listens on its local address of 192.168.1.1, and HTTP traffic is not allowed through your firewall anyway. However, SSH traffic is, so all you need to do is set up a tunnel to port forward a local port on your client machine to a local port on the remote machine. Assuming your SSH-accessible firewall was listening on firewall.yourdomain.com, one possible syntax would be:

$ ssh user@firewall.yourdomain.com -L5080:192.168.1.1:80

If you then pointed your browser to localhost:5080, your traffic would be transparently tunnelled to your webserver by your firewall, and you could act more or less as if you were actually at home on your office network with the webserver happily trusting all of your requests. This will work as long as the SSH session is open, and there are means to background it instead if you prefer — see man ssh and look for the -f and -N options. As you can see by the use of the 192.168.1.1 address here, this also works through NAT.

This can work in reverse, too; if you need to be able to access a service on your local network that might be behind a restrictive firewall from a remote machine, a perhaps less typical but still useful case, you could set up a tunnel to listen for SSH connections on the network you’re on from your remote firewall:

$ ssh user@firewall.yourdomain.com -R5022:localhost:22 -f -N

As long as this TCP session stays active on the machine, you’ll be able to point an SSH client on your firewall to localhost on port 5022, and it will open an SSH session as normal:

$ ssh localhost -p 5022

I have used this as an ad-hoc VPN back into a remote site when the established VPN system was being replaced, and it worked very well. With appropriate settings for sshd, you can even allow other machines on that network to use the forward through the firewall, by allowing GatewayPorts and providing a bind_address to the SSH invocation. This is also in the manual.

SSH’s practicality and transparency in this regard has meant it’s quite typical for advanced or particularly cautious administrators to make the SSH daemon the only process on appropriate servers that listens on a network interface other than localhost, or as the only port left open on a private network firewall, since an available SSH service proffers full connectivity for any legitimate user with a basic knowledge of SSH tunnelling anyway. This has the added bonus of transparent encryption when working on any sort of insecure network. This would be a necessity, for example, if you needed to pass sensitive information to another network while on a public WiFi network at a café or library; it’s the same rationale for using HTTPS rather than HTTP wherever possible on public networks.

Escape sequences

If you use these often, however, you’ll probably find it’s a bit inconvenient to be working on a remote machine through an SSH session, and then have to start a new SSH session or restart your current one just to forward a local port to some resource that you discovered you need on the remote machine. Fortunately, the OpenSSH client provides a shortcut in the form of its escape sequence, ~C.

Typed on its own at a fresh Bash prompt in an ssh session, before any other character has been inserted or deleted, this will drop you to an ssh> prompt. You can type ? and press Enter here to get a list of the commands available:

$ ~C
ssh> ?
Commands:
    -L[bind_address:]port:host:hostport  Request local forward
    -R[bind_address:]port:host:hostport  Request remote forward
    -D[bind_address:]port                Request dynamic forward
    -KR[bind_address:]port               Cancel remote forward

The syntax for the -L and -R commands is the same as when used as a parameter for SSH. So to return to our earlier example, if you had an established SSH session to the firewall of your local network, to forward a port you could drop to the ssh> prompt and type -L5080:localhost:80 to get the same port forward rule working.

Shortcut for adding SSH keys

If you’ve dabbled with SSH much, for example by following the excellent suso.org tutorial a few years ago, you’ll know about adding keys to allow passwordless login (or, if you prefer, a passphrase) using public key authentication. Specifically, you copy the public key ~/.ssh/id_rsa.pub or ~/.ssh/id_dsa.pub off the machine from which you wish to connect into the /.ssh/authorized_keys file on the target machine. That will allow you to open an SSH session with the machine from the user account on the local machine to the one on the remote machine, without having to type in a password.

tom@conan:~$ scp ~/.ssh/id_rsa.pub crom:.ssh/conan.pubkey
tom@conan:~$ ssh crom
Password:
tom@crom:~$ cd .ssh
tom@crom:~$ cat .ssh/conan.pubkey >>~/.ssh/authorized_keys

However, there’s a nice shortcut that I didn’t know about when I first learned how to do this, which has since been added to that tutorial too — specifically, the ssh-copy-id tool, which is available in most modern OpenSSH distributions and combines this all into one less error-prone step. If you have it available to you, it’s definitely a much better way to add authorized keys onto a remote machine.

tom@conan:~$ ssh-copy-id crom

Incidentally, this isn’t just good for convenience or for automated processes; strong security policies for publically accessible servers might disallow logging in via passwords completely, as usernames and passwords can be guessed. It’s a lot harder to guess an entire SSH key, so forcing this login method will reduce your risk of script kiddies or automated attacks brute-forcing your OpenSSH server to zero. You can arrange this by setting ChallengeResponseAuthentication to no in your sshd_config, but if that’s a remote server, be careful not to lock yourself out!