SMS Delivery Outage

For a number of short periods between  Feb 10 17:05 and  Feb 11 09:36 (GMT+11) our New Jersey node (66.246.75.38) failed to deliver SMS and Phone call alerts for some customers.

This was caused by the node’s inability to resolve the DNS record needed to connect with both our primary and backup SMS gateways.

This in turn was caused by failures of the New Jersey data center’s DNS resolvers.

That, however, should not have been a problem because the standard operating environment (SOE) for all Wormly nodes includes a private DNS cache & resolver in order to prevent exactly this sort of problem.

However this particular data center provider uses a DHCP based network configuration process, which caused /etc/resolv.conf to be updated by their DHCP server, thus reverting DNS resolution back to their servers.

We have ensured that this cannot occur again by setting the immutable attribute on /etc/resolv.conf – something which is now part of our SOE.

Needless to say we apologize to the customers this has inconvenienced – and should mention that of course no charges were billed for the failed deliveries.

We’re also pleased to report that our internal monitoring alerted us to this situation, so even in the absence of contact from a couple of helpful customers we would have been able to identify and correct this problem in short order.

Thanks for your support and understanding!

Filed under: Announcements,Meta — Jules @ 10:44 - February 11, 2009 :: Comments Off on SMS Delivery Outage

Are Your Servers’ Clocks Accurate?

Despite the prevalence of NTP, many sysadmins do not keep their servers running on the correct time.  This is unfortunate, as it can make troubleshooting via log files much more difficult.

To celebrate 2008 finishing up one second longer than most years, Wormly now reports if a servers’ clock is running slow or fast via the Health Monitoring tab.  e.g.:

So if you notice your servers running with an inaccurate clock, it might pay to put something like the following into cron:

/usr/bin/rdate -s time-nw.nist.gov

Note that this feature is currently only available for Linux servers. We hope to make it available for Windows servers in future.

Feature Deployed @ 2000-01-02 00:30 GMT

Filed under: Announcements,Features — Jules @ 10:42 - January 2, 2009 :: Comments Off on Are Your Servers’ Clocks Accurate?

Adwords Broad Matching: When a visit is not a visit

Don’t want to pay for a search phrase containing the word “visit”?  No problem, just add it to the Adwords campaign negative keyword list.

Then Google will happily sell you clicks on the keyword “visits” instead.

Google is more than happy to offer “broad” keyword matching to increase the inventory of searches available for click & sale, but when it comes specifying keywords you don’t want to pay for, Google insists you be very specific. And able to deliver very poor spelling.

Peek Behind The Keyword Curtain

Just discovering what search phrases Google is selling you can be challenging, and is undoubtedly out of reach for the vast majority of Adwords buyers. Neither Google’s Adwords reporting, nor Analytics product – even with “ad tracking” switched on – will reveal the actual search phrases that were sold to you.

The only reporting Google offers is a carbon copy of the keywords you chose, which Google by default broadly matches against what users actually type into the search engine in order to display your ad and sell you the resultant click.

Instead, you need to scrape the referrer headers from your web server logs to determine what search terms you have been paying for. Which, of course, we do.

The Broad Match

One of the phrases we buy advertisements on is “website monitoring” – certainly a reasonable fit for our target audience. Imagine, then, our surprise at discovering that the following happy searcher was delivered to our website. For a modest fee, of course.

The usefulness of Google’s broad matching abilities have been discussed at length.  We know that the vast majority of advertisers have little idea of what keywords they are ultimately paying for, given the on-by-default nature of “broad matching”.

Even putting this substantial issue aside it is reasonable to expect that one can explicitly choose not to pay for irrelevant searches; for visitors with zero value.

Instead we find that our campaign negative keyword list grows daily, with such delights as “survailance”, “surveilance” and an ever increasing number of spelling variations. Each time because we’ve paid Google to deliver a worthless visit to our website.

Filed under: Google Adwords,Marketing,Sales Process — Jules @ 11:32 - December 17, 2008 :: Comments Off on Adwords Broad Matching: When a visit is not a visit

Phone Call Alerts Now Available

Owing either to high traffic events or server administrators going on holiday without a contingency plan, our users are likely to see lots of downtime throughout the festive season.

To help out, we’ve brought back the plain old telephone system.

As of today, phone call alerts are available in both our server health and uptime monitoring systems.  For example, you can configure phone alerts if free disk space falls below 5% for more than 30 minutes – or if CPU load stays at 100% for a bit too long.

Calls are charged at a flat rate of $0.40 per call.

Naturally you can also configure a phone call if your site goes down altogether.  A useful escalation schedule might be: Email when the downtime first occurs, send an SMS after 10 minutes, and make the phone call after 30 minutes.

We reckon that a phone call is still the best way to wake up your normally over-caffeinated sysadmin at 4am when The Bad Stuff happens. That little SMS *bleep* noise from their phone doesn’t always do the trick.  And knowing that it’s all automatic is even better.

You can learn more about phone call alerts on this page.

Feature Deployed @ 2008-12-01 09:00 GMT

Filed under: Announcements,Features — Jules @ 12:59 - December 3, 2008 :: Comments Off on Phone Call Alerts Now Available

Health Monitoring Alerts Now Available

We’re very pleased to announce the immediate availability of our server health monitoring alert system; a feature which has been at the top of our most-requested list for some time now.

A simple screenshot explains it nicely:
setup server health alerts

Naturally you can also read about the feature in more detail.

For existing users, simply click on your hosts’ Health Monitoring tab and follow the instructions to start using this new feature.

Feature Deployed @ 2008-11-14 09:00 GMT

Filed under: Announcements,Features — Jules @ 08:59 - November 17, 2008 :: Comments Off on Health Monitoring Alerts Now Available

Are spam filters damaging your cash flow?

One worrying trend we’ve noticed in recent months is the increasing likelihood that our customers’ spam filters catch our monthly invoices, either sending them to the oft-ignored spam folder, or rejecting them outright.

Needless to say this is concerning because our customers either won’t know their credit card is being charged (if they’re on our auto-bill system) or simply won’t know that payment is due; risking suspension of their account.

Intuitively it makes sense that spam filters would attach a high spam score to invoices & payment requests, as these sorts of documents very often feature in spam and phishing attempts.

So, what to do about it?

Our experiments revealed that nearly every major spam filtering system is substantially less likely to classify an email as spam if it originates from a well known, reputable mail service such as GMail, Yahoo Mail, Hotmail, etc.  The identity of the originator is determined by IP address rather than the unreliable From: header.

Bear in mind that your web server either has no reputation value at all or – worse – has an IP address that was previously leased to less scrupulous operators. As availability of IP addresses becomes tighter you can certainly expect that the IPs attached to your shiny new server have been used by numerous websites & servers before reaching you.

We’ve experienced this situation a number of times – as we maintain a large number of servers in physically disparate locations (hence on different networks) and need to ensure email alerts can be delivered from all of them.

Google to the rescue!

At this point a possible solution becomes clear – route important email via a well known and reputable service to improve its chances of successful delivery.

We’ve been trialling this by utilizing the SMTP relay service provided by Google Apps Premier Edition – which powers all email destined for the wormly.com domain. It’s dramatically improved the situation for us thus far, and provides the additional benefit of archiving all web server outbound email within GMail.

To assist if you’d like to try something like this, I’ve posted a howto for configuring Postfix to relay via GMail’s SMTP service.

Filed under: Servers,Web 2.0,Web Services — Jules @ 15:56 - November 11, 2008 :: Comments Off on Are spam filters damaging your cash flow?

Relay mail via Google SMTP with Postfix

Using Google’s SMTP service to relay your outbound mail is a handy way to be able to send mail from Amazon EC2 instances, or other machines running IP addresses considered to be of dubious quality in the spam fighting world.

It’s also quite valuable in that your outbound emails will appear in your GMail sent messages folder and hence appear inline in conversations when your users reply.

Setup Instructions

These instructions are developed for Centos 5.2 – but no doubt they can be applied to other distributions with minimal modification.

Ensure that you have Postfix:
yum install postfix -y

We need to create the client keys for the TLS connection to Google’s SMTP service – and to do that we first need to become our own certificate authority (CA):

/etc/pki/tls/misc/CA -newca

Follow the prompts and make intelligent responses.

Now, create the client keys/certs (again with intelligent responses, and ensuring you use the same common name and country code):

cd /etc/postfix
openssl genrsa -out postfixclient.key 1024
openssl req -new -key postfixclient.key -out postfixclient.csr
openssl ca -out ./postfixclient.pem -infiles postfixclient.csr

Now you can configure Postfix accordingly. Add these to the top of /etc/postfix/main.cf
relayhost = [smtp.gmail.com]:587
smtp_connection_cache_destinations = smtp.gmail.com
relay_destination_concurrency_limit = 1
default_destination_concurrency_limit = 5
smtp_sasl_auth_enable=yes
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_use_tls = yes
smtp_sasl_security_options = noanonymous
smtp_sasl_tls_security_options = noanonymous
smtp_tls_note_starttls_offer = yes
tls_random_source = dev:/dev/urandom
smtp_tls_scert_verifydepth = 5
smtp_tls_key_file=/etc/postfix/postfixclient.key
smtp_tls_cert_file=/etc/postfix/postfixclient.pem
smtp_tls_enforce_peername = no
smtpd_tls_req_ccert =no
smtpd_tls_ask_ccert = yes
soft_bounce = yes

And store your password in /etc/postfix/sasl_passwd
gmail-smtp.l.google.com username@yourdomain.com:password
smtp.gmail.com username@yourdomain.com:password

(Note that if you’re using a regular gmail account instead of Google Apps For Your Domain, you would use username@gmail.com above)

Get Postfix to parse the password file:
postmap /etc/postfix/sasl_passwd

Optionally configure Postfix to run on a different port (so as not to clash with your regular SMTP relay):

Open /etc/postfix/master.cf and change the first line to:
10025 inet n - n - - smtpd

Now start Postfix!
service postfix start

And add it to your startup scripts:
ln -s /etc/init.d/postfix /etc/rc3.d/S96postfix

Filed under: Servers — Jules @ 17:06 - November 5, 2008 :: Read comments »

New Features & Announcements – Don’t Miss Out!

It’s been of some concern here at Wormly HQ that loads of great new features are going unnoticed by you all – and our lack of a coherent announcement strategy was certainly to blame for this.

So starting today, you will be able to stay abreast of all new Wormly features and announcements by following the notification link shown within the Wormly console to this blog. Simple as that. Naturally you could add this feed to your reader as well.

And to celebrate we’ve just rolled out a new feature, keep reading the next post to learn about it!

Filed under: Announcements,Features — Jules @ 12:23 - August 26, 2008 :: Comments Off on New Features & Announcements – Don’t Miss Out!

Date and Time Now Shown in Alert Message Bodies

I’m pleased to report we’ve rolled out another much-requested feature – the time-stamping of all alert messages.

Times are expressed in the timezone you’ve specified – the globally set zone or a host specific-one if available.

They’re shown in short form for the SMS & Short Email channels, e.g:
26/Aug 11:15

And long form for email and IM channels, e.g:
Tue, 26 Aug 2008 11:15:53 +1000

Feature Deployed @ 2008-08-26 02:00 GMT

Filed under: Announcements,Features — Jules @ 12:16 - :: Comments Off on Date and Time Now Shown in Alert Message Bodies

AdWords: Google Continues to Take Out The Trash

Courtesy of the Google AdWords Help Center:

Even if you have a high quality ad, if advertisers below you are not bidding very much, your actual CPC may not be high enough to qualify your ad to appear in a top position.

With this new formula, instead of considering your actual CPC, we’ll consider your maximum CPC bid, which you control. This means that your ad’s eligibility to be promoted is no longer dependent on the bids of advertisers below you.

This announcement indicates that Google remains keen to eliminate the long-tail approach to gaining cheap traffic, widely used by so-called (albeit misnamed) arbitrageurs in the SEM industry.

Until this change is implemented, advertisers can continue to place low quality ads in the top positions by bidding on keywords with little competition – generally lengthy, specific phrases.

Like most changes they make, this should be a win for Google and their users – higher prices for their ad inventory and / or better quality ads presented to the users.

Filed under: Google Adwords,Marketing — Jules @ 09:09 - August 21, 2007 :: Comments Off on AdWords: Google Continues to Take Out The Trash
« Previous PageNext Page »