Work

Adding more to your maps with marine charts

I haven't posted a lot over the past year about what I've been working on, but I've spent a lot of time building a web application that tracks ships around the world using positions relayed by satellite. I'll post more on the project soon, but for now I just want to share one of the ways that we're improving the usefulness of the maps as part of this project.

At their heart, the maps in the application are based on OpenStreetMap data and satellite imagery from MapBox, but when your clients are mostly looking at their vessels in seas and oceans around the world, traditional web mapping data doesn't really give enough context. Marine charts can fill in those big blue gaps and aid with using the map.

Map of the Isle of Man with Marine Charts overlaid
Map of the Isle of Man with Marine Charts overlaid
Map data © OpenStreetMap & MapBox, Marine charts © Crown Copyright, 2013. All Rights reserved. Licence no. EMS-EK001. Not to be used for navigation.

MarineFIND - with whom I've been working with to deliver this project - provide marine charts in various formats, including as a tile map service or web map service for clients who are looking to add charts to their own applications. The charts are made available either individually in their original form, or joined together to form an almost seamless detailed chart of the world. The combined chart layer cuts the landmass out, leaving them transparent, allowing the layer to be used on top of satellite imagery or other maps.


Map label overlay

Satellite labelling solution

Combining two raster datasets - the merged charts and the OpenStreetMap maps - into a single map causes some issues when it comes to labelling. Especially around coastal areas, labels from OpenStreetMap may be obscured when overlaying the marine charts.

This can be overcome to some extent by showing the charts as semi-opaque (see left), but for the satellite data we found another way of doing it (see right). Starting with an unlabelled satellite imagery layer from MapBox (which can be coloured to your liking), we then add the marine charts, and finally a separate MapBox Streets layer showing just line data and labels, meaning that any overlapping labels will be shown on top of the charts as expected.

If you want to find out more about using charts for your own applications, please feel free to get in touch, or contact MarineFIND for information about licensing.

On scaling from 40 visitors a day to 6 every second

When we launched the Bomb Sight website at the end of November, none of the project team had imagined quite how much interest there would be in the project.

We were pleased when we saw some people starting to tweet about it, and saw the number of visitors rise to 40 a day this time last week.

When I spotted at lunchtime on Wednesday that we were seeing an increasing level of traffic from Twitter, I sent a tongue-in-cheek email to the team saying I thought it might be starting to go viral. At that point we were seeing about one visitor a minute, and by the end of the day we were seeing about one every ten seconds, and had served 2,235 visitors over the course of the day. The web site was starting to feel a bit slow at times, but we think it was serving most visitors who tried to access the site.

A tweet from @qikipedia to their 357,000 followers suggests we were struggling by Thursday morning, but we kept serving visitors at about the same level of demand as we were on Wednesday... until 4pm on Thursday when we saw the first media article about the project.

That's when we really realised that the server couldn't keep up with the demand of 8,000 visitors an hour - and you would likely have been seeing errors quite often - so Patrick and I started to look at the server to see what we could do to help it scale with the resources we had available. I think we managed to squeeze a bit more capacity out of it, so some more people were able to view the site if they were lucky enough to get through. We definitely weren't able to make it available to everyone who was trying to access the site though.

Thankfully we spotted a tweet that evening from CloudFlare, a company that specialises in delivering content faster and more reliably by sitting between the users accessing the site and the website itself. They shelter your server from intense levels of traffic by keeping a copy of your pages on their own servers and serving that out to users on your behalf.

Thursday evening was spent trying to get this set up and working, and by 1am we were seeing about 30 users a minute again. It wasn't quite the 5 minute setup they advertise - as we waited for DNS changeovers, tried to get to grips with the configuration needed, competed with everyone else to access our own pages, and tried to keep our server online enough for CloudFlare to cache copies of the pages - but by the end of the night I was feeling a bit more comfortable that we'd be able to keep the website running a bit better for the next day. The front page of the site (the main map) was being cached by CloudFlare, and that was the most important part to keep working.

We had already served 18,459 visitors on Thursday, but that still wasn't the busiest day we'd see. By far.

By Friday morning, Patrick noticed that CloudFlare had started serving out a 404 Not Found error message instead of the homepage, which we battled with for a while, but managed to get rid of in the end. At 11am we were serving about 40 visitors a minute, until the BBC published their article around noon and we hit 380 visitors a minute, or 6 a second. And probably quite a few error pages too, unfortunately, as we worked to get rid of the remaining ones.

Friday saw around 185,000 visitors coming to the site, which just would not have been possible if we didn't have the support of CloudFlare to serve the vast majority of that traffic for us.

Up to this point on Tuesday, we have already had somewhere between 300,000 - 500,000 visitors* to the Bomb Sight website, and are currently serving about 900 visitors an hour. According to CloudFlare, these visitors have viewed upwards of 1.8 million pages between them.

We have shifted around 3 terabytes of data, of which CloudFlare has served about 2.7TB for us. 1.2TB of that were in the first 24 hours, and at the peak we were seeing a throughput of about 150GB an hour. That's a lot of data, especially when it was wasn't planned for in advance.

The moral of the story - if you think there's even the slightest possibility your project may go viral, plan in advance to add in a service such as CloudFlare to take some of the traffic away from your servers and give you some breathing space. Especially considering it's free to use their service!

As we posted on Thursday, we're sorry if you've had troubles accessing the site so far, but we hope that you were eventually able to access the site, and will be able to explore it further now that we're better equipped.

We are noticing from the statistics that people are starting to come back and explore the site more - with almost 25% of visitors today being people who have viewed the site more than once - which is a good sign that the data is of value and interest to people who want to learn more about the history London. We're also starting to see more people finding the site from Google searches about the Blitz, and about particular parts of London.

* it's difficult to accurately measure the number of visitors to the site, partly because some users will have seen error messages, others may not have been counted if they viewed the site embedded in a media article, and different services give statistics calculated in different ways. The visitor statistics in this post are all from Google Analytics while the data statistics are from CloudFlare.

This post is cross-posted from the Bomb Sight project blog

Bomb Sight: explore London during the Blitz

For the past few months I've been part of a team working on a project that's helping make available information about where bombs fell in London during the period of World War II known as The Blitz.

Secret until 1972Records and maps from the WWII Bomb Census survey 1940-1945 have been available at the National Archives in Kew since they were declassified in 1972. The public has been able to access the original records there if they knew what they were looking for, and can get to Kew to visit the archives (something that's well worth doing, if you get a chance) but the maps hadn't been available online... until today.

Bomb SightThe Bomb Sight website launched today to make this information more accessible to the public. Covering the period between 7th October 1940 to 6th June 1941, you can find out where bombs fell during the Blitz and also access photos and stories that relate to that area during the war.

The project was funded as part of the JISC Content Programme to digitise a subset of the maps and make them available online - in both raster and vector forms - and provide a mobile augmented reality interface to explore where bombs fell in the area around you.

The main bomb damage map is the place to start exploring. Zoom out to get an idea of how badly hit each and every part of London was, or zoom in to an area (e.g. bombs falling near Tower Bridge) to see more detail.

You may want to use the layers menu to switch from the modern map to the original maps used to record bomb locations.

You can click through to see a little more information about each bomb (e.g. these two east and west of Tower Bridge) and see related information about the area.

You can also explore statistics for all areas of London down to a borough (e.g. Southwark) and ward level (e.g. Riverside, Southwark).

Keep an eye out for the Android app I'm working on for the project, which will be coming soon. In the mean time, you can already explore an area using your phone, simply by visiting bombsight.org in your browser.

If you're interested in the work we've been doing with the project, follow us on Twitter @BombSightUK, and please help spread the word to others who may be interested in exploring London's past. You can also read some more information about the work over the past year in the project blog.

Tag Street: QR codes for estate agents

QR codeQR codes are becoming a common sight in the UK, used for all sorts of things, from innovative marketing campaigns to museum guides. The codes can be a great way for people to get more information about a topic right there and then.

In my day job at Inteeka we have been looking at ways of using QR codes in the estate agent industry, where it's important for potential customers to be able to get at property information with ease.

By scanning a QR code, potential tenants or buyers can be taken directly to the estate agent's mobile site, giving detailed information about that specific property; they can find out pricing information, photos of the interior and any other important information, as well as letting the customer register their interest while they are still thinking about it.

Tag Street board and stickerTo let estate agents try out QR codes for their own portfolio of properties, Inteeka is launching a service called Tag Street, allowing agents to order QR code stickers and signage boards they can attach to their existing property marketing signs. Both boards and stickers will be free (with the exception of postage) and estate agents will only pay if their customers are making use of the boards to view property information.

If you're interested, you can find out more over on the Tag Street site. If you have any feedback, please tweet us @tag_st or get in touch.

Drupal developer with an interest in all things geo

After two years of working from home, I've decided it's time to make a move back into an office and look for some contract-based or perhaps permanent work in London.

I have four years of experience using, developing and helping guide the development of Drupal projects as well as a background and interest in all things geographic, from maps to open data (as you've probably seen from the topics I cover in my blog). With these skills I am looking to find some work as a Drupal developer for an organisation based in London, ideally integrating my geographic interest. Alternatively, I'm open to other opportunities that I may be suitable for.

If you have, or know of, any positions coming up from January onwards, I'd love to hear from you to discuss the details. You can find out more information about me in my CV (pdf) or on my LinkedIn profile.

OpenBand Labs has a new look

OpenBand has today unveiled its new OpenBand Labs website to help improve the information around the work we've been doing over recent years as well as hopefully invite some discussion.

OpenBand Labs

We hope to make some improvements to the new site over the coming days as we continue to add some more information and blog about some of our experiences here at DrupalCon. We'll also be adding the slides and some writeups about the XMPP talk Darren gave yesterday and the distributed enterprise talk Ben gave today.

We had a great presentation yesterday with Darren Ferguson talking about the XMPP Framework, and today with Ben Lavender talking getting a chance to demonstrate our collaboration platform.

Freelance Drupal services coming in 2009

From the start of 2009 I am going to be self-employed, and while I will still be working much of my time on the projects I've been working on for the past three years at OpenBand/M.C. Dean, I would also like to start taking on some small Drupal-based projects to go side-by-side with that.

Some of the services I am planning to offer in the New Year include:

  • Drupal site creation
    If you are looking to get a site set up for your small business or organisation, and you like the power that Drupal can give to your site, then I can set a site up for you to meet your needs. I'm particularly interested in creating sites based around geographic information or related to the tourism industry, but I will happily consider any project.
  • Drupal site setup support
    If you need advice on how best to achieve your requirements with existing Drupal modules, I can help point you in the right direction and get you started with your Drupal site setup. This can either be on a remote basis, or on a face-to-face basis in the London area if needed.
  • Drupal module development
    If there isn't already a module in the Drupal community to do what you need, I can help you by building a module to meet your requirements.

If you're interested in taking advantage of these freelance Drupal services for your project, please contact me to discuss your needs.

OpenBand sponsoring DrupalCon DC 2009

OpenBand, an M.C. Dean CompanyMy employer, OpenBand, is going to be a Gold sponsor of DrupalCon DC in March 2009 and a number of our team members will be attending the conference.

We have a few presentations to give, and will be keen to see many of the other sessions that are going to be given.

In the Powering collaboration in a distributed enterprise session we'll be giving an overview of the work that we do, the collaboration platform we've been building (largely on Drupal) for our customers over the past three years or so, and some of the modules that we've contributed back to the Drupal community during that time.

Miglius Alaburda will be presenting a session titled Introducing a new File Framework about a new and powerful way of handling files in Drupal.

Alex Karshakevich will be giving a talk about Data Visualization in a Social Networking Context, taking advantage of some of the social networking capabilities of Drupal.

Darren Ferguson will be talking about Drupal with XMPP Integration and all the functionality that he has built up around the XMPP framework, allowing users of a Drupal site to use instant messaging capabilities.

It's also great to see some of our associates presenting about work which we've sponsored, like Audrey Foo's talk about how to Search private content between multiple drupal sites with Distributed search (and the related Add powerful features and improve performance of your drupal site with Apachesolr Search), Justin Miller's talk titled Beyond The Web: Drupal Meets The Desktop (And Mobile), as well as others in areas we're involved in like OpenID and RDF.

With all the sessions that have been proposed by attendees, this is shaping up to be a great conference!

Updated: Added Darren's XMPP talk

DrupalCon Barcelona 2007 this week

DrupalCon Barcelona 2007Tomorrow I'll be traveling down to Barcelona with my colleagues for this year's DrupalCon.

Much like last year's OSCMS Summit (which basically turned into a DrupalCon) and DrupalCon Brussels it will be a great chance to meet up with other Drupal developers and users, see what others are using the platform for, join in discussions about its future and hopefully promote some of the pieces that we've been developing at work or have sponsored.

I'll hopefully be doing a lightning talk on the use of Drupal as a GeoCMS - if there are enough people interested - perhaps demonstrating some Google Earth integration through the KML module, GeoRSS integration, or even WFS integration (if I can get it working before then).

If you're going to DrupalCon and are interested in the geo aspect of Drupal (or geo in general), let me know or catch me there - I'll be the one with the 'geodaniel' name badge.

Geo brings people to Drupal

I recently discovered the iCommunity.tv localised video news site (via High Earth Orbit) and was especially interested as it was being run on Drupal.

This was one of the first places out on the internet that I'd seen Drupal's KML module being put to good use, for example to be able to view in Google Earth all of the citizen journalist videos from YouTube that are tagged as being from the city you're interested in (e.g. Stuttgart video news and its associated KML feed of Stuttgart video news). Grass roots journalism is only useful to readers if the content is filtered to your needs, if it's in the area you want to know about and it's the type of news that you're looking for. The iCommunity.tv site does that by tagging videos by location (both coordinates and the name of the nearest city) and by topic (politics, community life, arts, etc.) and lets you filter by a combination of those*.

I heard back from Chris Haller of iCommunity.tv after posting a comment on his announcement of the site. He told me that he was previously a Mambo/Joomla user, attracted to Drupal both by its flexibility and by its geo-capabilities. It's great to see that happening, and someone told me exactly the same thing yesterday too. It's also great that organisations like the Open Source Geospatial Foundation are using Drupal as their platform of choice.

Its good to see Drupal becoming more and more of a GeoCMS and people taking the tools and finding ways to apply them in useful and practical ways.

* KML feeds for multiple tags do not currently work in the KML module, as I've just discovered

Pages

Subscribe to RSS - Work