When using IPv6 on client computers (i.e not servers) it is common to use automatic address configuration (know as SLAAC). This means you do not have to statically assign every device with an IP address.

Unfortunately the default way that many IPv6 stacks operate is to use your network card's MAC address as the basis of your global IPv6 address. This has the (sometimes) undesirable effect of giving your machine an automatically configured static IPv6 address.

This can be a privacy issue, as all IPv6 compatible sites you visit will be able to use your IPv6 address as a type of unique identifier (a bit like a cookie) to identify you as a visitor across their site, and potentially any other sites they operate.

There is mode that can be enabled on Fedora 18 to use a random IPv6 address to avoid this issue, which is known as RFC3041.

To enable this on your network card in Fedora 18, modify the following file:

/etc/sysconfig/network-scripts/ifcfg-p1p1

Add the line:

IPV6_PRIVACY=rfc3041

Then disable your network interface and re-enable. You can test whether your IPv6 address is random by visiting ipv6.nl.

The latest version of CentOS, 5.9, has updated their Postfix (an SMTP mail server) package to require mysql because it now supports reading user and domain lists from a MySQL database.

Unfortunately this has caused issues with anyone using the Oracle or MariaDB MySQL distributions.

Update 2013-02-27

The folks over at RedHat have fixed the bug I logged within 1 day!

There is a new postfix RPM available that removes the dependency of "mysql", and instead only requires "libmysqlclient.so.15", which is satisfied by either MySQL-shared-compat or MariaDB-compat. Open Source for the win.

Its all about Requires

Software for any RedHat based operating system is packaged using an RPM (RedHat Package Manager) file. These files describe what features the software requires and provides to the system, along with what files it contains.

The postfix package in CentOS 5.9 requires the "mysql" package.

The "mysql" package in CentOS 5.9 provides the MySQL 5.0 client utility for accessing MySQL servers.

The problems arise when trying to use either Oracle's MySQL or MariaDB packages.

The latest version of Oracle's MySQL 5.5 series is 5.5.30. None of its packages now claim to provide "mysql", which means when installing Postfix these packages will not be installed. Instead the vanilla mysql client package will be installed.

Unfortunately when you try to install the MySQL-client package from Oracle, it will conflict, because both the "mysql" and "MySQL-client" packages try to manage the same files, e.g. /usr/bin/mysql.

For the MariaDB packages, the current stable version is 5.5.29. The MariaDB-server package claims to provide "mysql", so when installing postfix, yum will try and install MariaDB-server.

This works fine, but seeing as most servers should have some sort of MTA installed (for sending cron emails for example) and only database servers need MySQL-server install, it seems silly to go installing MariaDB-server on every server.

This means we are now in a rather awkward situation:

• Oracle's MySQL client cannot be installed on a server running postfix
• MariaDB's MySQL server must be installed on a server running postfix

A twist in the tale

It turns out that postfix doesn't actually require "mysql" at all. It requires the MySQL libraries so that it can communicate with a MySQL server, it doesn't need the command line client utility.

It seems the people who package postfix for CentOS 6 (and Fedora) realize this, and their packages do not require "mysql", but instead require only "mysql-libs".

The effect of this small change is to allow either MariaDB or Oracle's packages to be installed cleanly, as yum will pull in MariaDB-compat or MySQL-shared-compat respectively.

I do not know why the CentOS 5.9 package of postfix has this difference in requirements.

A solution, of sorts

A short term solution is to grab the postfix SRPM from a CentOS mirror, edit the RPM SPEC file to disable the requirement for mysql and increment its build version. Then rebuild it and put that in your own YUM repository. This will allow you to install the updated postfix without any MySQL dependencies.

A proper solution

As it stands currently Oracle's MySQL-client package is not installable on CentOS/RHEL 5.9 running postfix. These packages are specifically marketed as compatible with this OS, so for them to be incompatible is a big problem.

The upstream resolutions as I see them are:

• Oracle and MariaDB to modify their client RHEL/CentOS packages to provide "mysql" - This will ensure compatibility with any other packages that require "mysql"
• CentOS/RHEL update their postfix package to require "mysql-libs" rather than "mysql"

The good folks over at Valve have recently launched their Steam client for Linux and ported some of their games to run on Linux.

Unfortunately the officially supported client only runs on Ubuntu, however Fedora People have kindly re-packaged it into an RPM suitable for running on Fedora 18.

Installing Steam

To install steam on Fedora run the following commands:

su
cd /etc/yum.repos.d/
wget http://spot.fedorapeople.org/steam/steam.repo
yum install steam

Installing 32bit textures

You can then start Steam using the normal application starter. However to run games there is also a need to install a texture package that is not installed automatically. Please note that even if you are running 64bit Fedora you should install the 32bit package as the games are compiled to use the 32bit textures.

yum install libtxc_dxtn.i686

That's all there is to it. Enjoy Linux gaming!

Background Info

For the last few weeks I have been encountering a strange problem with making IP WHOIS queries against the RIPE database, which covers all European IPs.

I first encountered the problem during a routine server upgrade and reboot. Suddenly some of our software that we run on these servers started producing errors saying that WHOIS lookups could not be performed.

After some investigation it transpired that it was only IP WHOIS lookups against the RIPE database that were failing. What is more, it was only happening on a couple of our servers, even though they all sat behind the same shared firewall and are Source NATed to the same public IP.

As time went by I upgraded more servers, and each time the newly upgraded server also started exhibiting the behaivour. Naturally my first thought was that something in the kernel upgrades that had warranted the reboot were to blame.

I began to downgrade some of the servers to their previous kernel versions, but this did not fix the issue. Stranger still some of the servers running the new kernels started working again, but intermittently!

Break out TCPDUMP

To try and understand what was going on I started running tcpdump on the firewall server to try and see the difference between a working server and a non-working server.

The results of a working server looked like this:

13:14:24.291132 IP x.x.x.x.40474 > 193.0.6.135.43: S 1723346221:1723346221(0) 
win 5840 mss 1460,sackOK,timestamp 3097608306 0,nop,wscale 4

The results of a non-working server looked like this:

12:58:26.886531 IP x.x.x.x.47159 > 193.0.6.135.43: S 9443771:9443771(0) 
win 5840 mss 1460,sackOK,timestamp 2068177 0,nop,wscale 4

Initially, the the packets looked the same with nothing obviously wrong.

The only thing that was different was that the timestamp of the newly rebooted non-working server packet was much lower than the server that had been running for months and was able to perform WHOIS lookups fine.

Surely this is perfectly acceptable, even behind NAT, because TCP connections use packet sequence numbers, not timestamps to order packets? If this wasn't the case, surely NAT would break things all the time?

As it turns out (after much searching) there is an extension to TCP called PAWS (Protect Against Wrapped Sequences) that is designed to prevent older packets from the same connection interfering with current TCP communication when using high bandwidth and high latency links.

Unfortunately it seems that the RIPE network has PAWS enabled, and it seems when making WHOIS requests from multiple servers behind the same public IP causes packets to be dropped because they have the conflicting combinations of timestamp and sequence numbers.

The Resolution

The resolution to this problem turned out to be very simple, disable TCP timestamps in our outgoing packets.

sysctl net.ipv4.tcp_timestamps=0

This means that PAWS cannot operate, and then immediately all the servers were able to perform WHOIS lookups with no problems.

Interestingly, enabling PAWS on your network can potentially introduce a DOS attack vector, by the attacker forging a packet to set a host's timestamp artificially high, and preventing future genuine communication.

Recently I have been working on a project using software called Freeswitch, which is an excellent open source SIP server.

The project required the use of a text-to-speech (TTS) speech engine called Cepstral.

However Cepstral's product suffers with concurrency problems when used with many concurrent phone calls. Additionally there is about a 1 second delay before TTS audio actually starts to play, which can be off-putting for the callers.

To overcome these issues I have implemented a caching mechanism using Freeswitch's built in integration with the LUA scripting language.

Our system tends to 'say' the same things over and over again, so by caching the TTS output to a wav file this allowed Freeswitch to just play back the sound file, rather than generate the same audio over and over again.

The TTS Cache Script

The script below should be installed into the scripts directory in Freeswitch, commonly /opt/freeswitch/scripts/tts_cache.lua.

-- This script generates a wav file of the sentence passed in to it.
-- It uses the Cepstral swift command to perform text-to-speech conversion.
-- If the wav file already exists for this sentence, then it is not
generated.
api         = freeswitch.API();
msg         = argv[1];
msgMd5      = api:execute( "md5", msg );
filename    = '/var/lib/tts_cache/' .. msgMd5 .. '.wav';
cmd         = '/opt/swift/bin/swift';

-- Set a channel variable so that we know which file to play back.
session:setVariable( 'tts_file', filename );

-- Check whether the file already exists.
file, errMsg = io.open( filename, "r" )
if not file then
api:execute( 'system', cmd .. ' -o "' .. filename .. '" "' .. msg .. '"' );
end

Using The TTS Cache Script

To use the script, first create a directory to save the cache files in and ensure Freeswitch can write to it:

mkdir /var/lib/tts_cache
chown freeswitch /var/lib/tts_cache

Next, create a phrase macro to allow you use it within a dial plan or IVR setting, commonly this goes in /opt/freeswitch/conf/lang/en/ivr/tts_cache.xml:

<include>
  <!--Provides a phase to speak custom text-->
  <macro name="tts_cache">
    <input pattern="(.*)">
      <match>
        <action function="execute" data="lua(tts_cache.lua '$1')"/>
        <action function="play-file" data="${tts_file}" />
      </match>
    </input>
  </macro>
</include>

Finally, in your dial plan you can use this script as so:

<action name="phrase" data="tts_cache,Hello World" />

Now the first time the phrase "Hello World" is requested, it is passed into the Cepstral swift command, which generates a wav file, and then when ever the same phrase "Hello World" is requested in the future, Freeswitch will just playback the wav file, which is much quicker.

I have recently been investigating backup solutions at work. I have been trying out rdiff-backup, which is a tool that allows you to remotely synchronise files and maintain snapshots of previous versions.

I have written up my notes in the Wiki.

According to Urban Dictionary, a Dog Pile is:

A group of people jumping on on person and creating a tower of people while crushing the people on bottom.

This term is used in the area of cache invalidation too. It means that when a cached object expires, it causes a rush to refresh the object from source, which can sometimes lead to the source being overwhelmed with requests.

NginX has protection against this when using its reverse proxy cache feature. More in the wiki.

Apologies for the lack of updates recently. I have been really busy with a new product at work.

Anyway, today's post is a quick article on converting M4A files to MP3 files for usage on ageing portable players (like mine).

More in the wiki.

Apache web server traditionally writes to local log files in /var/log/httpd.

At work we have been looking into PCI compliance, and it requires that log files are stored centrally so that if a server gets compromised and the local log files are modified, there is still an authoritative copy on the central log server.

Syslog is the standard Linux and UNIX way for transmitting log entries to a central server.

Problem is Apache only supports logging to syslog for it's error log, and not it's access log.

Thankfully the problem is relatively easy to fix with a short Perl script.

Apache Log Config

I am doing this on a RedHat server, so the config file locations will be specific to that.

Create a new file called /etc/httpd/conf.d/syslog.conf:

LogLevel warn
LogFormat "%v %V %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" vcombined
CustomLog "|/usr/bin/httpd_syslog" vcombined
ErrorLog syslog:local2

Next create a Perl script /usr/bin/httpd_syslog to accept piped logs from Apache and send them to the local syslog service:

#!/usr/bin/perl
use strict;
use Sys::Syslog qw( :DEFAULT setlogsock );
setlogsock('unix');
openlog('httpd', 'cons,pid','local1');

#Read from STDIN and log to syslog
while (my $log = <STDIN>) {
        syslog('notice', $log);
}
closelog();

You can now reload Apache and both error logs and access logs should start flowing into /var/log/messages.

Syslog Config

Next up, we need to split out the error logs and access logs into their own files again, rather than have them appearing in the main /var/log/messages file.

To do this we need to modify /etc/syslog.conf:

Comment out the line:

*.info;mail.none;authpriv.none;cron.none  -/var/log/messages

And replace it with:

*.info;mail.none;authpriv.none;cron.none;local1.none;local2.none  -/var/log/messages
local1.*  -/var/log/httpd_access_log
local2.*  -/var/log/httpd_error_log

This tells the local syslog service not to log entries from local1 and local2 facilities (which we are using for Apache) into /var/log/messages, and instead log them into separate log files.

Central Syslog Server

To instruct the local syslog server to send all entries to a central server, add the following line to /etc/syslog.conf:

*.* @IP_ADDRESS_OF_LOG_SERVER

Now restart syslog:

service syslog restart

Log Rotation

Finally, ensure that the new log files get rotated when other syslog files are rotated by modifying /etc/logrotate.d/syslog, modify the top line so it looks like this:

/var/log/messages /var/log/secure /var/log/maillog 
/var/log/spooler /var/log/boot.log /var/log/cron 
/var/log/httpd_access_log /var/log/httpd_error_log {

Today we started using Git. We are using Github to host our repositories for a private project. Previously we have been an SVN shop and have built various deployment systems around it.

I have encountered the following issues, and will separate them into two sections; those relating to Git as a tool, and those relating to Github as a service.

Github Issues

User management and access control

As a Systems Administrator and Developer I spend my time both writing applications and maintaining services for other developers. One of these services is version control. In the past we used SVN and setup a master password list containing users for each developer. It wasn't perfect, but because it was hosted internally, the only way to access it from outside was using a VPN that authenticated against the master Active Directory server.

Git as a tool seems to provide good user management and access control by utilising SSH, where each user can have either a separate system account, or login to a shared account using an SSH key.

In the event that an employee left and needed their account revoked, simply removing that user or their key would suffice. In addition because it would be hosted internally, any access would be done through the VPN, and presumably their AD account would also be shut down.

However with Github, which encourages 'social coding', the access control and user management is severely limited. Firstly, each user has their own Github account that is in no way related to the company's project account. They are then invited to join a company repository as a 'collaborator'. Unfortunately this is done on a per-repository basis, meaning that removal of a user must be done manually for each.

No prizes for guessing for who is going to get lumbered with that task!

Archive Support

Github have disabled archive support in Git, meaning that you cannot export a tagged release for packaging. Instead you have to clone or pull a repo, and then package it. This is inefficient as it also pulls down all commit changes, which are not necessary when packaging software.

Git Issues

The first issue relates to the previous issue. This is extracting code from a repo for packaging without also downloading all the commit changes.

There is no equivalent of svn export, and whilst archive appears to be what I need, Github does not support this. I don't know if this is a problem with Git or Github, either way it is annoying.

The other issue is complexity. Granted I am new to all this distributed version control stuff, so will stick with it. But when I started using SVN it just clicked and immediately made my life easier. I'm not sure when I will be needing Git's advanced features, but currently Git itself doesn't seem to offer any benefits to my work flow (although Github's code review is useful).

Conclusion

So all in all, not too bad of a start with Git. Although at the moment I am not using the full power of Git (i.e. forking/merging) as I have absolutely no need for it and would prefer to spend my time coding than messing with version control.

I am currently just pushing back to the master repository at Github, and although this isn't very 'cool', it suits my needs well (as SVN did).

My message to those considering switching to Git would be: If you are having trouble with conflicting commits or are looking to use a hosted service like Github that gives you added extras (like code reviews) then go for it.

If on the other hand you are happy with SVN and its limitations, then stick with it, and if you are looking for the added benefits of a hosted service, try something like Unfuddle.