How to rule remote shell sessions with tmux and mosh

If you’re like most ssh users when your connection breaks it’s bad news for you. Not only do you have to reconnect but your session gets destroyed and you have to make all the moves to restore the previous state. This doesn’t have to be that way. I’d like to say some words about two tools that solve these problems in the most elegant way possible.

tmux

tmux is a terminal multiplexer: it enables a number of terminals to be created, accessed, and controlled from a single screen. tmux may be detached from a screen and continue running in the background, then later reattached.

In the world of tmux there are windows and panes within windows. You can think of tmux windows as workspaces on the desktop that are aligned in a horizontal manner. It’s like having a number of virtual monitors next to each other each running different shell sessions. You can move across these windows as desired. With the use of panes you can split individual windows horizontally and/or vertically as desired, each pane housing a different session. This is pretty useful for tailing various log files in different panes and monitoring them at once.

You simply have to run the tmux command to create a new tmux session. Once a session exists upon reconnecting over ssh you have to invoke tmux attach to reconnect to your already existing session.

If you’re like me you may want to use tmux by default upon ssh’ing to servers. To make this happen you have to include export LC_TMUX_SESSION_NAME=yourusername into your ~/.bashrc and wrap scp on the client side and invoke tmux automatically on the server side. On a related note you can also take a look at my tmux.conf which I believe defines more intuitive shortcuts than the default configuration.

There are a number of alternatives to tmux that I’d like to list starting with the most powerful towards the least powerful. GNU Screen is yet another terminal multiplexer but its feature set, usability and configurability is rather limited compared to tmux. dtach is like a minimalistic tmux featuring one pane inside one window and it only provides a minimal set of options. Finally, with the use of the nohup command you can make your (typically long-running) script immune to hangups and hence it can survive ssh disconnects.

mosh

Remote terminal application that allows roaming, supports intermittent connectivity, and provides intelligent local echo and line editing of user keystrokes.

mosh is the other piece of the puzzle leading to the remote shell nirvana. After apt-getting mosh on the client and mosh-server on the server instead of invoking ssh yourserver.com invoke mosh yourserver.com. From this point on you don’t have to worry about reconnecting to ssh or having to wait for the server to echo back your characters anymore.

Ethernet cable labelling

Ever felt that you have too many cables in your home and don’t know which one leads where?  Welcome to my world!  Here’s my recipe to how to make the situation easier.

It all starts with a sticky tape.

ethernet-cable-labelling-1

Now comes the pen.

ethernet-cable-labelling-2

Let’s cut the tape with scissors.

ethernet-cable-labelling-3

At this point you should remove the protective foil from the back of the tape to expose the glue and shape a loop.

ethernet-cable-labelling-4

As a final move, pull the cable through the loop and push the tape against the cable to make it stick.

ethernet-cable-labelling-5

When doing all the above properly, you’ll get a of sight that makes the pants of any nerd wet.

Make your OpenWrt router automatically enable your home server and notify you

Here’s the situation: I have a Mini-ITX home server in my LAN that sometimes gets disabled when the power goes down. I could setup its BIOS to get automatically enabled when it gets power but considering the nature of power outages it may happen that the power goes out and on in rapid succession over a short amount of time which isn’t really good for the hardware.

I’d rather choose my server to be checked on a 5 minute basis and have it automatically woken up and I also wanna be notified by email on such an occasion so that I can SSH into it and uncrypt the crypto partition.

I’m about to carry out this task using my OpenWrt driven router. Everything is pretty straightforward, except that I can’t send emails through GMail SMTP using Openwrt because mini_sendmail lacks SASL support so I’ll just fetch a PHP script that’ll actually send the mail.

openwrt:/usr/bin/supervise-host

Be aware that the above wol utility is a home-cooked one.

openwrt:/etc/crontab/root

# m h  dom mon dow   command
0,5,10,15,20,25,30,35,40,45,50,55 * * * * supervise-host mybox http://mydomain.com/notifyme.php

At this ponit restart cron on OpenWrt.

mydomain.com:/var/www/notifyme.php

< ?php
mail('admin@mydomain.com', $_GET['host'] . ' has been rebooted', 'uncrypt me!');
?>

Enjoy!

Streamlined OpenVPN configuration for LANs

I have a reoccuring task of setting up OpenVPN for the LANs of small enterprises and adding / removing users.  Usually they have a dumb little TP-Link or D-Link router facing the public Internet, we bring a relatively powerful PC to their office and my job is to configure the PC as an OpenVPN gateway (among other things).  OpenVPN traffic gets forwarded to our PC through the dumb little router using port forwarding.  Well, this is not particularly challenging to me but I was looking for a way to automate this process as much as I can because managing clients can be cumbersome.

Let’s clarify a task at hand: An OpenVPN gateway has to be set up for a /24 LAN in order to provide access to all hosts on the LAN.  Privilege management will be implemented using PKI.  On top of that we’ll use tls-auth so the HMAC firewall will only answer if the received packet signature is valid, thus effectively making the OpenVPN service undetectable by any scanning techniques.

The LAN should reside on a class A private subnet (10.x.y.0/24) where x and y should be randomly choosen because it’ll minimize the probability of address collision with other subnets used with OpenVPN.

First of all, the PKI should not reside on the server on which the OpenVPN daemon runs for security reasons.  I store it on my home partition which is heavily encrypted and regularly backed up.  I create a directory under ~/openvpn for every OpenVPN installations where I store the server and client configuration files and the PKI.  Only the needed files will be transferred to the server or to the clients.

This post will describe the implementation of the above configuration and will provide a set of scripts to make the task very efficient.

1) Set up the ~/openvpn infrastructure

mkdir ~/openvpn
cd ~/openvpn

# User credentials will be temporarily published under the directory below for user download.  This should be a trusted host.
# It’s probably needless to say but I mention that $PUBLISH_URL should not under any circumstances be listable by the web server.
cat >config <<END
PUBLISH_PATH=yourhost:/var/www/pki
PUBLISH_URL=http://yourhost.com/pki
END

wget /wordpress/wp-content/uploads/openvpn-scripts.tar.bz2
tar xjf openvpn-scripts.tar.bz2 -C ~/bin
rm openvpn-scripts.tar.bz2

2) Set up the server directory

cd ~/openvpn
mkdir SERVERNAME
cd SERVERNAME

3) Set up the PKI

mkdir easy-rsa
cp -r /usr/share/doc/openvpn/examples/easy-rsa/2.0/* easy-rsa
cd easy-rsa
# Edit the all the KEY_* variables in ./vars so you won’t have to type them anymore.
. ./vars
./clean-all
./build-ca
./build-key-server server
./build-dh
cd ..
mkdir ccd

4) Create server configuration

openvpn –genkey –secret ta.key

cat >server.conf << END
mode server
local 10.X.Y.Z
tls-server
dev tun
proto udp
port 1194
client-config-dir ccd
ifconfig 10.8.0.1 10.8.0.2
push “route 10.X.Y.0 255.255.255.0″
push “route 10.8.0.0 255.255.255.0”
route 10.8.0.0 255.255.255.0
keepalive 10 120
ca ca.crt
cert server.crt
key server.key
dh dh1024.pem
tls-auth ta.key 0
log server.log
verb 3
END

# This will be used by the synchronization script to rsync the configuration to the server through SSH.
echo SERVERHOSTNAME > server.hostname

5) Create general client configuration

# This is the client configuration from which the all individual client configurations will be generated.
# Don’t touch “username” as it will be automatically replaced with the name of the relevant user during the generation process.

cat >client.conf << END
dev tun
proto udp
nobind
remote OPENVPN-GATEWAY-HOST 1194
client
ca server.crt
tls-auth server-ta.key 1
cert username.crt
key username.key
verb 3
END

6) Add users

openvpn-add-user username1
openvpn-add-user username2

# The configuration will be automatically transferred to the server.

7) Publish client credentials

openvpn-publish-user-credentials username1
openvpn-publish-user-credentials username2

# Which outputs something like this:
# User credentials are accessible from http://yourhost.com/pki/servername-username1-65378842373270.zip
# User credentials are accessible from http://yourhost.com/pki/servername-username2-10200344763221.zip
# …

# These URLs are meant to be mailed to the relevant users and removed eventually.

8) Unpublish client credentials

openvpn-unpublish-user-credentials username1
openvpn-unpublish-user-credentials username2

# Which removes the relevant files from the server.

9) Revoke client credentials

openvpn-revoke-user-credentials username

# The configuration will be automatically transferred to the server.

Setting up HTTPS with Apache using a CAcert certificate

openssl req -new -nodes -out yourdomain.com.csr -keyout yourdomain.com.pem
# Type your domain name to the Common Name field.

# Log in to CAcert, go to Server Certificates > New, select “Sign by class 3 root certificate”, hand the CSR and get the CRT which you should place to your server as /etc/apache2/ssl/yourdomain.com.crt

cat << END > /etc/apache2/sites-available/yourdomain.com
NameVirtualHost *:443
<VirtualHost *:443>
DocumentRoot /var/www
Options FollowSymLinks
SSLEngine on
SSLCertificateFile /etc/apache2/ssl/yourdomain.com.crt
SSLCertificateKeyFile /etc/apache2/ssl/yourdomain.com.pem
</VirtualHost>
END

a2ensite yourdomain.com
a2enmod ssl
apache2ctl restart

 

Installing OpenWrt Kamikaze on the ASUS WL500GPV2

I’m a big fan of OpenWrt for more than a year when I got to know it.  I daydreamed about a powerful router that is fully controllable and runs Linux.  Little did I know in those times about specialized Linux distributions running on routers. I can remember when I started to chat with my good friend, Dömi about this topic and he immediately mentioned OpenWrt. Fast forward one month and an ASUS WL500GPV2 was sitting on my desk running OpenWrt Kamikaze.

I’d like to mention that although I’ll talk about the V2, I wish I would have bought the V1 because that’s much more hackable. You can upgrade its memory and/or replace its Mini PCI WiFi card, but you can’t do that with the V2, unfortunately.

The reason I put this guide together is that I wanted a detailed configuration guide on the topic because it’s quite demanding to rebuild my mental model of how things work after some months. Be aware that that this HOWTO is not for the faint of heart. I assume you know what you’re doing.

1) Download the firmware

wget http://downloads.openwrt.org/snapshots/trunk/brcm-2.4/openwrt-brcm-2.4-squashfs.trx

2) Set up localhost to hang on the 192.168.1.x subnet

sudo ifconfig br0 down  # in case you use VirtualBox host networking
sudo brctl delbr br0  # also for VirtualBox
sudo ifconfig eth1 192.168.1.2

3) Flash the router firmware

# Boot the router into diagnostic mode by pressing the reset button right after restart. Wait until the power LED starts blinking.
atftp –trace –option “timeout 1” –option “mode octet” –put –local-file openwrt-brcm-2.4-squashfs.trx 192.168.1.1
# Wait at least half a minute otherwise the router won’t get flashed properly and boots into diagnostic mode.

4) Set up SSH public key authentification

# Disable and enable router and wait for the boot that takes about half minutes.
telnet 192.168.1.1
passwd
# At this point telnetd gets stopped and the dropbear sshd gets started.
exit
scp ~/.ssh/id_rsa.pub root@192.168.1.1:/etc/dropbear/authorized_keys
ssh root@192.168.1.1

5) Set up WAN connection

# Use this for cable modem connections:
uci set network.wan.proto=dhcp

# Use this for ADSL connections:
uci set network.wan.proto=pppoe
uci set network.wan.username=yourusername
uci set network.wan.password=yourpassword

# Let’s commit the changes finally:
uci commit network
ifup wan
reboot  # If the WAN connection is still down at this point.

6) Set up WiFi connection

uci set wireless.wl0.disabled=0
uci set wireless.@wifi-iface[0].ssid=yourssid
uci set wireless.@wifi-iface[0].encryption=psk2
uci set wireless.@wifi-iface[0].key=yourpassword
uci commit wireless
wifi

7) Install packages

opkg update
opkg install mc ndyndns etherwake ntpclient openvpn kmod-fs-ext2 fdisk e2fsprogs kmod-usb2 kmod-usb-storage nmap
# Here you can remove every luci related packages if you don’t need the web interface.

8) Restore configuration

# It’s pretty easy to restore configuration from a backup once you made a backup like this:
ssh your-router ‘tar czf – /etc /root’ > kamikaze-backup-`date +%Y-%m-%d_%H-%M-%S`.tar.gz

9) Supercharge storage space

# Plug in the pendrive.
mkfs.ext2 /dev/scsi/host0/bus0/target0/lun0/part1
mount /dev/scsi/host0/bus0/target0/lun0/part1 /mnt
cp -a /bin /etc /home /lib /root /sbin /usr /www /mnt
mkdir /mnt/dev /mnt/jffs /mnt/mnt /mnt/proc /mnt/rom /mnt/sys /mnt/tmp /mnt/var
umount /mnt

cat < /etc/config/bootfromexternalmedia
config bootfromexternalmedia
    option target   '/mnt'
    option device   '/dev/scsi/host0/bus0/target0/lun0/part1'
    option modules  'usbcore ehci-hcd scsi_mod sd_mod usb-storage jbd ext2'
    option enabled  '1'
END
cat < /sbin/init.new
#!/bin/sh
. /etc/functions.sh
config_load "bootfromexternalmedia"
local section=\$CONFIG_SECTION
config_get      "target"   "\$section" "target"
config_get      "device"   "\$section" "device"
config_get      "gpiomask" "\$section" "gpiomask"
config_get      "modules"  "\$section" "modules"
config_get_bool "enabled"  "\$section" "enabled" '1'
[ "\$enabled" -gt 0 ] && {
    [ -n "\$gpiomask" ] && {
        echo "\$gpiomask" > /proc/diag/gpiomask
    }
    for module in \$modules; do {
        insmod \$module
    }; done
    sleep 5s
    mount -o rw "\$device" \$target
    [ -x \$target/sbin/init ] && {
        . /bin/firstboot
        pivot \$target \$target
    }
}
exec /bin/busybox init
END

chmod a+x /sbin/init.new
ln -f -s /sbin/init.new /sbin/init
reboot