Running OpenVZ with IPv4 and IPv6 on Digital Ocean

Posted 28/06/2015 19:38

With containers all the rage at the moment (LXC/LXD, Docker, Rocket etc), I thought it would be interesting to see if it was possible to get a mature container implementation (OpenVZ) running on the cloud provider Digital Ocean.

I have been running OpenVZ with CentOS 5 & 6 in production for over 5 years now and I have found it to be rock solid and has a simple set of management tools.

First Steps

Firstly you need to sign up for a CentOS 6 Droplet in one of the many data centers that Digital Ocean provide.

I chose the following settings:
  • Hostname: testopenvz
  • Size: $5/mo
  • Region: London 1
  • Image: CentOS 6.5 x64
  • Settings: IPv6

Once the Droplet is up and running, SSH into it and enable a SWAP file.

Then ensure your Droplet is all up to date:

yum upgrade -y

Install some useful tools:

yum install wget nano -y

OpenVZ Installation

Now the time has come to install OpenVZ kernel and management utilities.

wget -P /etc/yum.repos.d/ http://ftp.openvz.org/openvz.repo
rpm --import http://ftp.openvz.org/RPM-GPG-Key-OpenVZ
yum install vzkernel vzctl vzquota ploop -y

Enable OpenVZ IPv4 and IPv6 network settings in /etc/sysctl.conf by adding the following lines:

# On Hardware Node we generally need
# packet forwarding enabled and proxy arp disabled
net.ipv4.ip_forward = 1
net.ipv6.conf.default.forwarding = 1
net.ipv6.conf.all.forwarding = 1
net.ipv4.conf.default.proxy_arp = 0
net.ipv6.conf.all.proxy_ndp = 1

# Enables source route verification
net.ipv4.conf.all.rp_filter = 1

# Enables the magic-sysrq key
kernel.sysrq = 1

# We do not want all our interfaces to send redirects
net.ipv4.conf.default.send_redirects = 1
net.ipv4.conf.all.send_redirects = 0

Modify /etc/sysconfig/network and add these lines:

NETWORKING_IPV6=yes
IPV6FORWARDING=yes

Because we will be running NAT on the Droplet firewall we need to ensure that OpenVZ doesn't disable connection tracking by modifying /etc/modprobe.d/openvz.conf as follows:

options nf_conntrack ip_conntrack_disable_ve0=0

Setting up KEXEC

Digital Ocean annoyingly do not allow you to boot your Droplet with a custom kernel. As OpenVZ requires this, the only way to boot the custom kernel is to use the kexec utility, which allows you to boot a standard kernel and then replace it with a custom kernel.

Install kexec:

yum install -y kexec-tools

Then create the following init file /etc/init.d/kexecvz which will start the OpenVZ kernel on boot:

#!/bin/sh
#
# kexecvz
#
# chkconfig: 2345 90 60
### BEGIN INIT INFO
# Provides:          localkexec
# Required-Start:
# Required-Stop:
# Should-Start:
# Default-Start:     S
# Default-Stop:
# X-Interactive:     true
# Short-Description: kexec
### END INIT INFO

case "$1" in
  start|"")
        if grep -q kexeced /proc/cmdline; then
                exit 0
        fi
        /sbin/kexec --load `ls -1t /boot/vmlinuz-*stab* | head -n 1` --initrd=`ls -1t /boot/initramfs-*stab* | head -n 1` --command-line="`cat /proc/cmdline` kexeced"
        /sbin/kexec -e
        ;;
  restart|reload|force-reload)
        echo "Error: argument '$1' not supported" >&2
        exit 3
        ;;
  stop)
        # No-op
        ;;
  *)
        echo "Usage: $0 [start|stop]" >&2
        exit 3
        ;;
esac

Run the following commands to start it on boot:

chmod +x /etc/init.d/kexecvz
chkconfig kexecvz on

Configure OpenVZ Default Container Settings

Modify the /etc/vz/vz.conf file and add the following lines:

VE_LAYOUT=simfs
DEF_OSTEMPLATE="centos-6-x86_64"

Checking OpenVZ is running

Now reboot your Droplet and when it comes back check it is running the OpenVZ kernel.

uname -a

The output should have the word "stab" in the version which indicates the OpenVZ kernel is running:

Linux testopenvz 2.6.32-042stab108.5

Enable NAT on the Droplet firewall

As Digital Ocean only allows a single IPv4 address on their Dropets we need to use NAT in order to allow the containers to access the Internet.

/sbin/iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
service iptables save

Create First Container

Now we can create our first container with Container ID 1000 and hostname testcontainer1.localdomain:

vzctl create 1000 --hostname testcontainer1.localdomain

We will add a private IPv4 address to each of our containers so they can communicate with each other and use NAT to access the IPv4 Internet. For this example I am using the 172.16.1.0/24 subnet.

vzctl set 1000 --ipadd 172.16.1.2 --save

Now we add an IPv6 address to your container from the range of 16 IPv6 addresses that Digital Ocean provide with each Droplet. To find your range go into your Digital Ocean control panel and go to Settings > Networking.

vzctl set 1000 --ipadd 2a03:xxxx:1:d0::424:6002 --save

Copy your SSH public key from your Droplet into the container so you can SSH into your container.

mkdir /vz/private/1000/root/.ssh/
chmod og= /vz/private/1000/root/.ssh/
cp /root/.ssh/authorized_keys /vz/private/1000/root/.ssh/

Now you can start the container and then login to it:

vzctl start 1000
vzctl enter 1000

Test Container Connectivity

Now you are logged into your container, you should test your IPv4 and IPv6 network connectivity.

ping google.com
ping6 google.com

If both succeed, then you're good to go, so exit our of your container back to your Droplet.

exit

Add NAT SSH Port Forward

In order to reach your container from the Internet you can either use the IPv6 address that you assigned to your container, or if you don't have IPv6 connectivity then you need to setup a NAT port forward for SSH from your Droplet's public IP to your container.

iptables -t nat -A PREROUTING -p tcp -i eth0 --dport 8882 -j DNAT --to-destination 172.16.1.2:22
service iptables save

You should now be able to SSH to your Droplet's public IP using port 8882.

Final Thoughts

You should now be all setup to add additional containers and IP addresses as needed. Remember to setup a firewall to protect your Droplet and your containers though!.

Setting up Stunnel for secure communication on CentOS 5 and 6

Posted 15/03/2015 12:03

Sometimes you need to secure communication for an internet service that does not support TLS functionality. For example, I needed to perform secure file synchronization over the Internet using rsync, but it does not support TLS. I didn't want to use SSH tunneling as that requires additional security lockdown to prevent the remote user from running shell commands.

To solve this problem the tool Stunnel provides an encrypted TCP tunnel back to your un-encrypted service.

So the flow of communication would be as follows:

rsync client -> stunnel on client machine -> encrypted TLS link -> stunnel on server machine -> rsync server

My requirements for this project are as follows:

  • Support CentOS 5 and 6 clients (server would be running on CentOS 6)
  • Use at least TLSv1, but support TLSv1.1 and TLSv1.2 if client OS supports it
  • Disable SSLv3
  • Disable RC4 cipher

Installing Stunnel

The version of Stunnel that comes with CentOS 5 and 6 does not support TLSv1.1 or TLSv1.2. So straight off the bat I needed to build the latest Stunnel into an RPM. The SPEC file to do this is here. It works with both CentOS 5 and CentOS 6, although CentOS 5's OpenSSL library does not support higher than TLSv1.

To install Stunnel:

yum install stunnel

Configure a Certificate Authority

Using the EasyRSA tool from the OpenVPN project you can create your own Certificate Authority (CA).

Download the latest from EasyRSA Releases and unpack into /root.

cd /root
wget https://github.com/OpenVPN/easy-rsa/releases/download/v3.0.0-rc2/EasyRSA-3.0.0-rc2.tgz
tar zxvf EasyRSA-3.0.0-rc2.tgz

Now create your new CA.

cd EasyRSA-3.0.0-rc2
./easyrsa init-pki
./easyrsa build-ca

Configure Stunnel Server

First lets create a certificate for this server. You will need to enter the password you set for your CA key in the previous step.

./easyrsa build-server-full {your server hostname} nopass

Now copy the CA cert, server key and server cert into /etc/stunnel and secure the permissions on the private key file.

cp pki/ca.crt /etc/stunnel/
cp pki/private/{your server hostname}.key /etc/stunnel/
cp pki/issued/{your server hostname}.crt /etc/stunnel/
chmod 600 /etc/stunnel/*.key

Now create /etc/stunnel/stunnel.conf file containing:

client = no
foreground = yes
syslog = yes
debug = 6

[rsyncd]
accept = :::1873
connect = 127.0.0.1:873
cert = /etc/stunnel/stunnel.pem
CAfile = /etc/stunnel/ca.crt
cert = /etc/stunnel/{your server hostname}.crt
key = /etc/stunnel/{your server hostname}.key
ciphers = ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!3DES:!MD5:!PS
sslVersion = all
options = NO_SSLv2
options = NO_SSLv3
options = CIPHER_SERVER_PREFERENCE

The config file specifies a list of ciphers that are currently deemed secure using Mozilla's SSL Config Generator. Although at time of reading this list may have changed.

You should now be able to start stunnel in the foreground for testing by running:

stunnel

Use your favorite process manager (upstart, supervisord etc to run the program in the background).

Configure Stunnel Client

On your client machine install stunnel and copy the ca.crt file into /etc/stunnel, then create the stunnel.conf file:
client = yes
foreground = yes

[rsyncd]
accept = :::873
connect = {your server hostname}:1873
CAfile = /etc/stunnel/ca.crt
sslVersion = all
options = NO_SSLv2
options = NO_SSLv3
verify = 2

You should now be able to start stunnel in the foreground for testing by running:

stunnel

To test this is working try connecting rsync to 127.0.0.1 and it should copy the files from the remote server.

IPv6 Privacy Extensions in Fedora 20

Posted 17/02/2014 19:41

Previously I blogged about Enabling IPv6 Privacy Extensions in Fedora 18. Unfortunately in Fedora 20, the Network Manager has a bug in it that means that the setting is not used.

Thankfully there has been an issue logged already and a fixed Network Manager can be installed from the testing repo, heres how:

sudo yum update --enablerepo=updates-testing NetworkManager

Now restart, and when you run ifconfig, you should see an additional randomly generated IPv6 address.

Fedora 19 Gnome 3 OpenVPN default route bug workaround

Posted 27/10/2013 14:10

In Fedora 19 and Gnome 3 there is a rather annoying bug when using OpenVPN, the 'Use this connection only for resources on its network' tick box does not remained ticked, and causes the default route to be updated to point through the OpenVPN tunnel.

In some situations (mine) I do not want the default route to go down the OpenVPN tunnel, and so this was a problem.

Luckily there is a simple workaround until it gets fixed, open the relevant file for your VPN connection, for example /etc/NetworkManager/system-connections/Work.

Then find the section:

[ipv4]
method=auto

And add the line:

never-default=true

Hey presto, the tick box is now ticked!

Using Fail2Ban to block bruteforce Wordpress login attacks

Posted 26/10/2013 23:45

A friend of mine hosts a lot of Wordpress sites and we regularly see a lot of brute force attempts from many different IP addresses repeatedly tring to login to the admin section of the site at wp-login.php

E.g.

mysite.com mysite.com 198.27.74.223 - - [26/Oct/2013:17:42:16 +0000] "POST /wp-login.php HTTP/1.0" 200 3747 "-" "-"
mysite.com mysite.com 208.84.146.60 - - [26/Oct/2013:17:42:16 +0000] "POST /wp-login.php HTTP/1.0" 200 3747 "-" "-"

This affects the load on the web server and we start to get load alerts from the ISP that provides the server.

To try and counter this I have setup the Fail2Ban tool on the server, which automatically blocks any IP that tries to login too many times repeatedly.

Configuring Fail2Ban

First install fail2ban, for CentOS it is available in the EPEL repository.

yum install fail2ban
Next, copy the jail.conf to jail.local for editing:
cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local
Now edit /etc/fail2ban/jail.local and add the following lines:
[apache-wp-login]

enabled = true
action   = iptables[name=wplogin, port=http, protocol=tcp]
           sendmail-whois[name=wplogin, dest=root, sender=fail2ban@example.com]
filter  = apache-wp-login
logpath = /var/log/httpd/access_log
maxretry = 5

This file tells fail2ban what to do when the filter apache-wp-login matches, it sets up an action using iptables that blocks port http (80) and sends an alert E-mail.

The maxretry line tells fail2ban to only allow 5 repeat login attempts before blocking. The default time window is 5 minutes, which is specified at the top of that file.

Next define a filter to match wp-login.php requests by creating the file /etc/fail2ban/filter.d/apache-wp-login.conf

[Definition]

# Option:  failregex
# Notes.:  Regexp to catch Apache dictionary attacks on Wordpress wp-login
# Values:  TEXT
#
failregex = [\w\.\-]+ [\w\.\-]+ .*] "POST /wp-login.php

The regex above matches a custom log format I use that shows both the requested HTTP host header and the match vhost server name fields.

Now start the service and ensure it starts on boot:

service fail2ban start
chkconfig fail2ban on

Now your server will block repeat requests for wp-login.php and E-mail you when a block occurs.

October 2013 RPM Updates

Posted 20/10/2013 21:02

Hot on the heals of the CentOS 5.10, PHP 5.4.21 and PHP 5.5.5 releases, my RPMs have now been updated as follows:

  • CentOS 5 32 bit - PHP 5.4.21
  • CentOS 5 32 bit - Apache HTTPD 2.2.25
  • CentOs 5 32 bit - Freeswitch 1.2.14
  • CentOS 6 32 bit - PHP 5.4.21
  • CentOS 6 32 bit - PHP 5.5.5

All of these RPMs are available by following these instructions.

Installing Sublime Text on Fedora 19

Posted 06/10/2013 13:44

Sublime Text is a very good, but lightweight text editor for Windows, Mac and Linux.

Unfortunately the developers do not provide packages for Fedora (or any Linux distribution) and because it is non-free it is not in the Fedora official repositories.

This is how to install it and create a launcher for Fedora 19 using Gnome3.

Getting the tarball

First download the tarball from the Sublime Text site, then unpack it into /opt/sublime.

E.g.
cd /opt
wget http://c758482.r82.cf2.rackcdn.com/Sublime%20Text%202.0.2%20x64.tar.bz2
tar jxvf Sublime\ Text\ 2.0.2\ x64.tar.bz2
mv Sublime\ Text\ 2 sublime

Creating a Launcher for Gnome 3

In order to integrate Sublime Text with the Gnome 3 Desktop Environment, create the following file:

/usr/share/applications/sublime.desktop

[Desktop Entry]
Name=Sublime
Comment=Application for editing text files
Exec=/opt/sublime/sublime_text %U
Icon=/opt/sublime/Icon/128x128/sublime_text.png
Terminal=false
Type=Application
Categories=Programming;

I have found that you need to log out of Gnome and re-login before the new desktop entry is detected, but you will then be able to see the icon in the Applications list and be able to assign it as a default application for text files.

Enabling IPv6 Privacy Extensions on Fedora 18

Posted 06/04/2013 11:20

When using IPv6 on client computers (i.e not servers) it is common to use automatic address configuration (know as SLAAC). This means you do not have to statically assign every device with an IP address.

Unfortunately the default way that many IPv6 stacks operate is to use your network card's MAC address as the basis of your global IPv6 address. This has the (sometimes) undesirable effect of giving your machine an automatically configured static IPv6 address.

This can be a privacy issue, as all IPv6 compatible sites you visit will be able to use your IPv6 address as a type of unique identifier (a bit like a cookie) to identify you as a visitor across their site, and potentially any other sites they operate.

There is mode that can be enabled on Fedora 18 to use a random IPv6 address to avoid this issue, which is known as RFC3041.

To enable this on your network card in Fedora 18, modify the following file:

/etc/sysconfig/network-scripts/ifcfg-p1p1

Add the line:

IPV6_PRIVACY=rfc3041

Then disable your network interface and re-enable. You can test whether your IPv6 address is random by visiting ipv6.nl.

Problems with CentOS 5.9, Postfix and MySQL

Posted 26/02/2013 23:09

The latest version of CentOS, 5.9, has updated their Postfix (an SMTP mail server) package to require mysql because it now supports reading user and domain lists from a MySQL database.

Unfortunately this has caused issues with anyone using the Oracle or MariaDB MySQL distributions.

Update 2013-02-27

The folks over at RedHat have fixed the bug I logged within 1 day!

There is a new postfix RPM available that removes the dependency of "mysql", and instead only requires "libmysqlclient.so.15", which is satisfied by either MySQL-shared-compat or MariaDB-compat. Open Source for the win.

Its all about Requires

Software for any RedHat based operating system is packaged using an RPM (RedHat Package Manager) file. These files describe what features the software requires and provides to the system, along with what files it contains.

The postfix package in CentOS 5.9 requires the "mysql" package.

The "mysql" package in CentOS 5.9 provides the MySQL 5.0 client utility for accessing MySQL servers.

The problems arise when trying to use either Oracle's MySQL or MariaDB packages.

The latest version of Oracle's MySQL 5.5 series is 5.5.30. None of its packages now claim to provide "mysql", which means when installing Postfix these packages will not be installed. Instead the vanilla mysql client package will be installed.

Unfortunately when you try to install the MySQL-client package from Oracle, it will conflict, because both the "mysql" and "MySQL-client" packages try to manage the same files, e.g. /usr/bin/mysql.

For the MariaDB packages, the current stable version is 5.5.29. The MariaDB-server package claims to provide "mysql", so when installing postfix, yum will try and install MariaDB-server.

This works fine, but seeing as most servers should have some sort of MTA installed (for sending cron emails for example) and only database servers need MySQL-server install, it seems silly to go installing MariaDB-server on every server.

This means we are now in a rather awkward situation:

  • Oracle's MySQL client cannot be installed on a server running postfix
  • MariaDB's MySQL server must be installed on a server running postfix

A twist in the tale

It turns out that postfix doesn't actually require "mysql" at all. It requires the MySQL libraries so that it can communicate with a MySQL server, it doesn't need the command line client utility.

It seems the people who package postfix for CentOS 6 (and Fedora) realize this, and their packages do not require "mysql", but instead require only "mysql-libs".

The effect of this small change is to allow either MariaDB or Oracle's packages to be installed cleanly, as yum will pull in MariaDB-compat or MySQL-shared-compat respectively.

I do not know why the CentOS 5.9 package of postfix has this difference in requirements.

A solution, of sorts

A short term solution is to grab the postfix SRPM from a CentOS mirror, edit the RPM SPEC file to disable the requirement for mysql and increment its build version. Then rebuild it and put that in your own YUM repository. This will allow you to install the updated postfix without any MySQL dependencies.

A proper solution

As it stands currently Oracle's MySQL-client package is not installable on CentOS/RHEL 5.9 running postfix. These packages are specifically marketed as compatible with this OS, so for them to be incompatible is a big problem.

The upstream resolutions as I see them are:

  • Oracle and MariaDB to modify their client RHEL/CentOS packages to provide "mysql" - This will ensure compatibility with any other packages that require "mysql"
  • CentOS/RHEL update their postfix package to require "mysql-libs" rather than "mysql"

Installing Steam on Fedora 18

Posted 19/02/2013 20:34

The good folks over at Valve have recently launched their Steam client for Linux and ported some of their games to run on Linux.

Unfortunately the officially supported client only runs on Ubuntu, however Fedora People have kindly re-packaged it into an RPM suitable for running on Fedora 18.

Installing Steam

To install steam on Fedora run the following commands:

su
cd /etc/yum.repos.d/
wget http://spot.fedorapeople.org/steam/steam.repo
yum install steam

Installing 32bit textures

You can then start Steam using the normal application starter. However to run games there is also a need to install a texture package that is not installed automatically. Please note that even if you are running 64bit Fedora you should install the 32bit package as the games are compiled to use the 32bit textures.

yum install libtxc_dxtn.i686

That's all there is to it. Enjoy Linux gaming!