At Wormly we’re ardent believers in the maxim that performance is a feature.  Google’s SPDY protocol is an experimental extension to HTTP with the goal of reducing the latency inherent in the HTTP protocol.  Or, rather more simply: “Get those pages delivered to the user faster!”

From our perspective the biggest benefits come from stream multiplexing and header compression.

Multiplexed streams allow the browser to simultaneously request a bunch of resources over a single TCP connection – particularly handy because it avoids the latency penalty of setting up multiple TLS sessions for HTTPS.

Header compression (both request and response headers) helps further reduce the amount of data that must traverse the pipes of the internet.  These headers are probably bigger than you think – take an example request to wormly.com:

That’s 348 bytes just in the request header. And most users’ upstream bandwidth is rather more limited than your web servers’.

SPDY is currently supported by Firefox and Chrome – on Windows, Linux, Mac and Android platforms.  As it happens more than 70% of our users are on Chrome, so the benefits of adopting SPDY will be seen widely.

Enter mod_spdy

Google has released a very simple to use Apache extension to add SPDY support to your web server. We use Apache to serve www.wormly.com and we were also using mod_php to serve the parts of our web application which are written in PHP.

This actually presents a problem with mod_spdy, because its’ implementation is threaded, and PHP is not guaranteed to be thread safe.  The recommended approach is to replace mod_php with a FastCGI interface to serve any PHP requests from a dedicated pool of PHP processes – leaving the Apache processes free of any trace of the PHP interpreter.

We actually intended to switch to this sort of setup some time ago – and to switch from Apache to Nginx in the process.  However a still-unresolved limitation in Nginx whereby all FastCGI requests are buffered – meaning PHP cannot ensure that data is immediately flushed to the client – has so far prevented us from doing so.

Happily the availability of mod_spdy pushed us to find a solution; and ultimately we found it in mod_fastcgi, which does support unbuffered connections to the upstream PHP server.

Switching to FastCGI also allowed us to switch Apache to the Worker MPM rather than the Prefork MPM – and the benefits of these two changes significantly reduced the volatility of response times through our web application.  So we had a win even before deploying SPDY!

After this, installing mod_spdy is as simple as:

Restart Apache, and you’re done.

Note that Google’s included package management handles automatic updates of the mod_spdy package.

It should be noted that mod_fastcgi is not commonly included in CentOS / Red Hat Enterprise linux distributions and their derivatives (including Amazon Linux).  You can either compile it from source or use a 3rd party repository which does include it: for example the RPMForge, DAG or City-Fan.

One worrying trend we’ve noticed in recent months is the increasing likelihood that our customers’ spam filters catch our monthly invoices, either sending them to the oft-ignored spam folder, or rejecting them outright.

Needless to say this is concerning because our customers either won’t know their credit card is being charged (if they’re on our auto-bill system) or simply won’t know that payment is due; risking suspension of their account.

Intuitively it makes sense that spam filters would attach a high spam score to invoices & payment requests, as these sorts of documents very often feature in spam and phishing attempts.

So, what to do about it?

Our experiments revealed that nearly every major spam filtering system is substantially less likely to classify an email as spam if it originates from a well known, reputable mail service such as GMail, Yahoo Mail, Hotmail, etc.  The identity of the originator is determined by IP address rather than the unreliable From: header.

Bear in mind that your web server either has no reputation value at all or – worse – has an IP address that was previously leased to less scrupulous operators. As availability of IP addresses becomes tighter you can certainly expect that the IPs attached to your shiny new server have been used by numerous websites & servers before reaching you.

We’ve experienced this situation a number of times – as we maintain a large number of servers in physically disparate locations (hence on different networks) and need to ensure email alerts can be delivered from all of them.

Google to the rescue!

At this point a possible solution becomes clear – route important email via a well known and reputable service to improve its chances of successful delivery.

We’ve been trialling this by utilizing the SMTP relay service provided by Google Apps Premier Edition – which powers all email destined for the wormly.com domain. It’s dramatically improved the situation for us thus far, and provides the additional benefit of archiving all web server outbound email within GMail.

To assist if you’d like to try something like this, I’ve posted a howto for configuring Postfix to relay via GMail’s SMTP service.

Using Google’s SMTP service to relay your outbound mail is a handy way to be able to send mail from Amazon EC2 instances, or other machines running IP addresses considered to be of dubious quality in the spam fighting world.

It’s also quite valuable in that your outbound emails will appear in your GMail sent messages folder and hence appear inline in conversations when your users reply.

Setup Instructions

These instructions are developed for Centos 5.2 – but no doubt they can be applied to other distributions with minimal modification.

Ensure that you have Postfix:
yum install postfix -y

We need to create the client keys for the TLS connection to Google’s SMTP service – and to do that we first need to become our own certificate authority (CA):

/etc/pki/tls/misc/CA -newca

Follow the prompts and make intelligent responses.

Now, create the client keys/certs (again with intelligent responses, and ensuring you use the same common name and country code):

cd /etc/postfix
openssl genrsa -out postfixclient.key 1024
openssl req -new -key postfixclient.key -out postfixclient.csr
openssl ca -out ./postfixclient.pem -infiles postfixclient.csr


Now you can configure Postfix accordingly. Add these to the top of /etc/postfix/main.cf
relayhost = [smtp.gmail.com]:587
smtp_connection_cache_destinations = smtp.gmail.com
relay_destination_concurrency_limit = 1
default_destination_concurrency_limit = 5
smtp_sasl_auth_enable=yes
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_use_tls = yes
smtp_sasl_security_options = noanonymous
smtp_sasl_tls_security_options = noanonymous
smtp_tls_note_starttls_offer = yes
tls_random_source = dev:/dev/urandom
smtp_tls_scert_verifydepth = 5
smtp_tls_key_file=/etc/postfix/postfixclient.key
smtp_tls_cert_file=/etc/postfix/postfixclient.pem
smtp_tls_enforce_peername = no
smtpd_tls_req_ccert =no
smtpd_tls_ask_ccert = yes
soft_bounce = yes


And store your password in /etc/postfix/sasl_passwd
gmail-smtp.l.google.com username@yourdomain.com:password
smtp.gmail.com username@yourdomain.com:password


(Note that if you’re using a regular gmail account instead of Google Apps For Your Domain, you would use username@gmail.com above)

Get Postfix to parse the password file:
postmap /etc/postfix/sasl_passwd

Optionally configure Postfix to run on a different port (so as not to clash with your regular SMTP relay):

Open /etc/postfix/master.cf and change the first line to:
10025 inet n - n - - smtpd


Now start Postfix!
service postfix start

And add it to your startup scripts:
ln -s /etc/init.d/postfix /etc/rc3.d/S96postfix

Skype, an essential communication tools for millions of individuals and businesses worldwide has been unable to authenticate users during the past 14 hours, rendering the service unusable.

14 hours – and counting. One can scarcely imagine the magnitude of the technical failure that causes such a lengthy outage.

Although Skype offers paid-for, business critical services including inbound geographic number routing and outbound PSTN dialling, they have long – and wisely – avoided any commitment to deliver emergency call services. And you can understand their reluctance to start now.

This event also highlights the challenge of keeping customers informed; a typical Skype user almost nevers dials www.skype.com into their browser, so how to get the word out about the outage and status updates?

Luckily (or not), many major media outlets are covering the issue more than adequately.

Fingers crossed for Skype’s engineers that they can effect a resolution soon.

Don’t be shy – your customers really want to know just how reliable your service is. So go ahead and brag about it with our Public Uptime Reports.

When enabled, you can place one of our funky uptime badges on your site showing uptime from the previous 24 hour, 7, or 30 day period. You can also link through to a detailed uptime report where visitors can examine your uptime history on a yearly, monthly, or daily basis.

Take a look at this example – and click to see the full report:

It’s a great way to show your customers that uptime is important to you – Could this transparency be your edge over the competition?

Most sysadmins share a slightly compulsive tendency to need proof that things are working properly. This is particularly evident if said sysadmin is setting up a corporate mail server – it has to work first time, no excuses.

One of the more painful parts of that process has been testing the newly configured SMTP server before delegating the domain’s MX to it. This involves speaking SMTP through your telnet client, or re-configuring your mail client to speak directly with the new server, then un-re-configuring it.

Neither is a difficult task, of course, but it’s a solid 30 seconds (minimum) that I’d rather not spend.

So I whipped up a handy little SMTP test tool, which does the SMTP-speak for you, and renders the whole conversation in your browser via an IFRAME.

Neat, handy, and of course free. Try it out.