On scaling from 40 visitors a day to 6 every second

When we launched the Bomb Sight website at the end of November, none of the project team had imagined quite how much interest there would be in the project.

We were pleased when we saw some people starting to tweet about it, and saw the number of visitors rise to 40 a day this time last week.

When I spotted at lunchtime on Wednesday that we were seeing an increasing level of traffic from Twitter, I sent a tongue-in-cheek email to the team saying I thought it might be starting to go viral. At that point we were seeing about one visitor a minute, and by the end of the day we were seeing about one every ten seconds, and had served 2,235 visitors over the course of the day. The web site was starting to feel a bit slow at times, but we think it was serving most visitors who tried to access the site.

A tweet from @qikipedia to their 357,000 followers suggests we were struggling by Thursday morning, but we kept serving visitors at about the same level of demand as we were on Wednesday... until 4pm on Thursday when we saw the first media article about the project.

That's when we really realised that the server couldn't keep up with the demand of 8,000 visitors an hour - and you would likely have been seeing errors quite often - so Patrick and I started to look at the server to see what we could do to help it scale with the resources we had available. I think we managed to squeeze a bit more capacity out of it, so some more people were able to view the site if they were lucky enough to get through. We definitely weren't able to make it available to everyone who was trying to access the site though.

Thankfully we spotted a tweet that evening from CloudFlare, a company that specialises in delivering content faster and more reliably by sitting between the users accessing the site and the website itself. They shelter your server from intense levels of traffic by keeping a copy of your pages on their own servers and serving that out to users on your behalf.

Thursday evening was spent trying to get this set up and working, and by 1am we were seeing about 30 users a minute again. It wasn't quite the 5 minute setup they advertise - as we waited for DNS changeovers, tried to get to grips with the configuration needed, competed with everyone else to access our own pages, and tried to keep our server online enough for CloudFlare to cache copies of the pages - but by the end of the night I was feeling a bit more comfortable that we'd be able to keep the website running a bit better for the next day. The front page of the site (the main map) was being cached by CloudFlare, and that was the most important part to keep working.

We had already served 18,459 visitors on Thursday, but that still wasn't the busiest day we'd see. By far.

By Friday morning, Patrick noticed that CloudFlare had started serving out a 404 Not Found error message instead of the homepage, which we battled with for a while, but managed to get rid of in the end. At 11am we were serving about 40 visitors a minute, until the BBC published their article around noon and we hit 380 visitors a minute, or 6 a second. And probably quite a few error pages too, unfortunately, as we worked to get rid of the remaining ones.

Friday saw around 185,000 visitors coming to the site, which just would not have been possible if we didn't have the support of CloudFlare to serve the vast majority of that traffic for us.

Up to this point on Tuesday, we have already had somewhere between 300,000 - 500,000 visitors* to the Bomb Sight website, and are currently serving about 900 visitors an hour. According to CloudFlare, these visitors have viewed upwards of 1.8 million pages between them.

We have shifted around 3 terabytes of data, of which CloudFlare has served about 2.7TB for us. 1.2TB of that were in the first 24 hours, and at the peak we were seeing a throughput of about 150GB an hour. That's a lot of data, especially when it was wasn't planned for in advance.

The moral of the story - if you think there's even the slightest possibility your project may go viral, plan in advance to add in a service such as CloudFlare to take some of the traffic away from your servers and give you some breathing space. Especially considering it's free to use their service!

As we posted on Thursday, we're sorry if you've had troubles accessing the site so far, but we hope that you were eventually able to access the site, and will be able to explore it further now that we're better equipped.

We are noticing from the statistics that people are starting to come back and explore the site more - with almost 25% of visitors today being people who have viewed the site more than once - which is a good sign that the data is of value and interest to people who want to learn more about the history London. We're also starting to see more people finding the site from Google searches about the Blitz, and about particular parts of London.

* it's difficult to accurately measure the number of visitors to the site, partly because some users will have seen error messages, others may not have been counted if they viewed the site embedded in a media article, and different services give statistics calculated in different ways. The visitor statistics in this post are all from Google Analytics while the data statistics are from CloudFlare.

This post is cross-posted from the Bomb Sight project blog

Freelance Drupal services coming in 2009

From the start of 2009 I am going to be self-employed, and while I will still be working much of my time on the projects I've been working on for the past three years at OpenBand/M.C. Dean, I would also like to start taking on some small Drupal-based projects to go side-by-side with that.

Some of the services I am planning to offer in the New Year include:

  • Drupal site creation
    If you are looking to get a site set up for your small business or organisation, and you like the power that Drupal can give to your site, then I can set a site up for you to meet your needs. I'm particularly interested in creating sites based around geographic information or related to the tourism industry, but I will happily consider any project.
  • Drupal site setup support
    If you need advice on how best to achieve your requirements with existing Drupal modules, I can help point you in the right direction and get you started with your Drupal site setup. This can either be on a remote basis, or on a face-to-face basis in the London area if needed.
  • Drupal module development
    If there isn't already a module in the Drupal community to do what you need, I can help you by building a module to meet your requirements.

If you're interested in taking advantage of these freelance Drupal services for your project, please contact me to discuss your needs.

dankarran.com goes Drupal

According to my drupal.org user profile, it's been three years now since I joined the Drupal community, and in that time I've been involved with building a large Drupal-based collaborative platform at work, helped friends create Drupal sites and developed a number of small Drupal sites in a freelance capacity, as well as contributing a number of modules back to the community, but had not managed to get around to actually migrating my own website to Drupal. Until today, that is.

Today marks the soft launch of my new Drupal website, an initial version that has most of the content from the old site but still needs some user interface improvements and the like.

The blog is up and running, now also with RSS feeds for each of the main categories and also each of the tags. The photos are all in there, though I'm missing any gallery functionality at the moment, so they're going to be a little difficult to find for now.

Watch out for more over the coming weeks and months as I find time to make improvements, expand what's here and tie in with other services on the web like Twitter, Flickr and delicious.

Holiday cottages on Drupal

Manx farm holidays in Smeale, Isle of ManLast night saw the launch of the new Smeale Farm Cottages website (first launched back in 2000), helping market a new holiday cottage that they recently opened.

The new site features an improved availability calendar that makes it both easier to maintain and easier for visitors to understand as well as the ability for the customer to edit any of the pages themselves whenever they want to change any of the information.

Also featured is a location map thanks to data from the OpenStreetMap project.

As you might expect, the website is built using the Drupal open source content management system with the addition of the image module, a customised version of the zen theme and a heavily customised availability module to drive the calendars.

I'm hoping to release this availability calendar module back to the community as a new module in the near future.

How the GeoCMSs compare

At the State of the Map conference it was great to be able to meet up with two guys who also have interests in creating geographically able content management systems (GeoCMS), Andrew Turner who created the GeoPress plugin for MovableType and WordPress and Henri Bergius who is one of the founders of the Midgard CMS.

Before their talk on GeoClue we had a good opportunity to sit down and talk through some of the current functionality of the different systems, see where they differ, and try to agree on some common base functionality that we felt should be present across the different platforms.

The features included things like ability to save a location (obviously), how many locations could be used to reference each post, the presence of maps and which providers they used, the ability to post location information through the blogging API, the inclusion of Microformats (hCard), syndication formats (GeoRSS, KML, etc.), OpenSearch capabilities, reverse geocoding of coordinates to give place information, posting by email, and a couple of others.

When I get a chance I'm going to build up a table over on the Geospatial Content Management System Wikipedia page to compare the systems we talked about (WordPress, Midgard and Drupal) but also others such as Joomla, TikiWiki and Plone. Any input on those would be much appreciated as I haven't done much with them to date.

Update: I didn't notice that Henri had already blogged a little about this, and after the conference went off and added maps to Midgard using Mapstraction... cool!

Track OpenStreetMap diaries through RSS

A year ago I was really interested in seeing the community aspect of the OpenStreetMap website improve, and the recent update to Rails brought a lot of the functionality that I was looking for, with space to be tweaked and improved upon. Since I started to learn Ruby on Rails recently, what better way was there to help, but build on the functionality that others have put in already.

I outlined some ideas on the wiki and decided that some of my first priorities would be adding RSS to the diaries, making them easier to post to, improving the user profiles and also the messaging interface. I've added a few small changes over the past few days, but today saw the release of the biggest one so far: an RSS feed for all diary entries, so you can now subscribe to updates of everyone mapping on OpenStreetMap. Over time I'm also hoping to add other feeds for individual diaries, for your friends and also for those mapping nearby.

If you haven't used OpenStreetMap's diary feature, now is the time to give it a try and let people know what you're working on mapping at the moment. Right now, you can post to it by viewing your own diary (through your account page) and clicking on 'new post', but I'll be looking to make it easier for users to post as well.

Update: looks like it needs a little tweaking still, but the basics are there.

Design for geodan.org

Design for geodan.orgGeodan.org has a new theme, for the most part at least. I got the basics done on Monday night having begun with the basic bluemarine theme that Drupal ships with and has enabled by default. I had an idea in my head of what I wanted to acheive with the theme and it didn't take me too long to get it into a state that I was relatively happy with.

It's best viewed in Firefox or Safari at the moment as I can't get the opacity levels to have any effect in Internet Explorer. It's just no good if you can't see the earth through the body of the page (thanks go to NASA for that beautiful Blue Marble image of the earth).

Drupal geospatial testbed

It's been long enough that I've been working with Drupal and not had a site of my own that runs off it. I'd love to migrate dankarran.com to Drupal but it would take some time to migrate. So for now I will leave this site as it is and create a new site aimed at demonstrating Drupal as a Geospatial Content Management System.

geodan.org will be a testbed for geospatial functionality within Drupal. I'll be using existing modules such as the location module which gives a basis for handling geographic information about users and nodes in a site, gmap module which allows for Google Maps to be easily created, as well as a number of others. In addition to these I'll be adding the KML module I've been working on as part of my day job, and also the WFS Server module which still has a little way to go before it's ready.

Right now it's just a shell, but over time it will grow. And as it seems somebody over at the OGC recently picked up on my last post about Drupal as a WFS I should try and make that sooner rather than later...

My first paid Drupal site

Over the past week I've been working on building, theming and tweaking a new Drupal site for a friend. It's my first paid (as in, with chocolate) Drupal-based website outside of the ones we maintain at work, and it was great to be able to put into practice some of the tips and tricks I've picked up over the past eight months or so.

I'm quite pleased with the amount of work that Drupal removes from creating a website (steep learning curve aside), and how flexible it is out of the box, without even beginning to look into the many modules that you can plug and play with the framework. I'd say the majority of the time spent working on the site has been walking my friend through, showing her how to add new content, link it all together and all that good stuff.

The site went live today but isn't quite polished, so I'll leave the link off for now. At least until I've had a chance to make all those last minute tweaks that inevitably don't show up until you leave the nice comfortable environment of Safari and Firefox and see the thing torn to pieces in Internet Explorer.


Subscribe to RSS - Web