We have an interesting challenge here at Buyometric. The number of emails we send out on a daily basis is increasing rapidly, but the timeframe we have to deliver them is remaining constant. Imagine increasing the quantity of breakfast cereal that you pour into your bowl by 5% each morning. One of these days, you’re going to end up in a mess.
Our goal is to deliver an individually tailored email to every one of our users by 10am at the very latest. The email contains some deals that are only available for one day, so the earlier you receive the deals the longer you have to ponder a purchase before time runs out – or the deal sells out. Our classification and email creation processes is completed at around 8am, which gives us a 2 hour window to deliver the emails.
That’s no problem when your sending 10 emails, or 100, or 1,000 or 10,000. But what happens when you are sending hundreds of thousands of emails?
We’ve been finding out. Recently our 10am deadline has been slipping. Some days our mail server would crash altogether, and on others it would grind to a halt after chewing through all the available memory. Sometimes emails can’t be delivered and our server needs to keep trying. On these days, the extra load of repeated delivery attempts would cause something of a nuclear reaction and kill our sending processes dead.
The problem peaked about two weeks ago, where the last emails to be delivered were being received at around 2pm. This was totally unacceptable for us – something had to be done.
The fine folks at Catn, who provide and maintain our infrastructure, worked with us to come up with a solution. First we tried switching to a memory based mail server, but that didn’t cut it, so we ended up moving the server to a machine with a faster disc and much higher memory allocation. We’ve also started compressing our emails prior to dispatch to reduce the overall file size and eliminate any unnecessary overhead.
The result are looking good. We’re back within our target window, with email currently being fully dispatched shortly after 9am. That enables us to roughly double in size before we need to look at a further upgrade. At this rate we’d better book that in right around Christmas time…
You may have noticed a few occasions during the past month when you didn’t receive your daily offers email, or that your email came through late in the day. I’d like to put your mind at rest – “it’s not you, it’s me”!
We suffered a few growing pains during October as the number of people using our service continued to expand. From day one, we always wanted to be in full control of our technology architecture. Our emails are the most important element of our business, and so we wanted to send every single one ourselves, from our own servers. It might surprise you to hear that within the daily deals space we are pretty much unique in this respect. Most companies use a third party to deliver their emails – such as SendGrid, Amazon or Mailchimp. They do this so that they don’t have to worry about scaling their mail servers, sender reputation and creating sophisticated tracking systems to report open and click through rates. That’s all very well, but we made the decision that we didn’t want to surrender control of one of the most critical aspects of our business to somebody else.
We are a technology company, and so we took on the challenge of building our own server configuration that would cope with the level of emails we planned to send each day. We wanted to be in full control of our sender reputation with the major email providers – choosing to live or die by the quality of the emails we send our users. And we wanted to build the sophisticated tracking systems ourselves, so we could feed them straight back in to our recommendation engine and behavioural profiles of users.
And so that’s what we’ve done.
All was going swimmingly until mid October, when our current set up started to groan under the strain. Reports of missing emails started to roll in and successful delivery rates began to fall. For two weeks the technical team battled with the servers and our huge database – optimising queries, replacing queue systems and boosting memory. We overhauled the entire system to produce a lean, mean, emailing machine.
And the hard work has paid off. Delivery is back around the 100% mark. Our open rates are once again industry leading and our users are receiving the deals they are most interested in. To top things of, in the last week we’ve achieved Priority Sender status with Yahoo mail which we’re very pleased about.
Of course if you do notice anything strange with your emails, shout at us and we’ll look in to it.