All posts by martin

GeoJSON and topoJSON for UK boundaries

I’ve just put an archive online containing GeoJSON and topoJSON for UK boundary data. It’s all stored on Github, with a viewer and download site hosted on Github pages.

Browser for the UK topoJSON stored in the Github repository
Browser for the UK topoJSON stored in the Github repository

The data is all created from shapefiles released by the Office of National Statistics, Ordnance Survey and National Records Scotland, all under the Open Government and OS OpenData licences.

In later posts I’ll detail how I created the files, and how to use them to create interactive choropleth maps.

CCG to WPC lookup

CCGs and WPCs via the medium of OAs

As I was eating lunch this afternoon, I spotted a conversation between @JoeReddington and @MySociety whizz past in Tweetdeck. I traced the conversation back to the beginning and found this request for data:

I’ve been doing a lot of playing with geographic data recently while preparing to release a site making it easier to get GeoJSON boundaries of various areas in the UK. As a result, I’ve become pretty familiar with the Office of National Statistics Geography portal, and the data available there. I figured it must be pretty simple to hack something together to provide the data Joseph was looking for, so I took a few minutes out of lunch to see if I could help.

Checking the lookup tables at the ONS, it was clear that unfortunately there was no simple ‘NHS Trust to Parliamentary Constituency’ lookup table. However, there were two separate lookups involving Output Areas (OAs). One allows you to lookup which Parliamentary Constituency (WPC) an OA belongs to. The other allows you to lookup which NHS Clinical Commissioning Group (CCG) an OA belongs to. Clearly, all that’s required to link the two together is a bit of quick scripting to tie them both together via the Output Areas.

First, let’s create a dictionary with an entry for each CCG. For each CCG we’ll store it’s ID, name, and a set of OAs contained within. We’ll also add  an empty set for the WPCs contained within the CCG:

import csv
from collections import defaultdict

data = {}

# extract information about clinical commissioning groups
with open('OA11_CCG13_NHSAT_NHSCR_EN_LU.csv', 'r') as oa_to_cgc_file:
  reader = csv.DictReader(oa_to_cgc_file)
  for row in reader:
    if not data.get(row['CCG13CD']):
      data[row['CCG13CD']] = {'CCG13CD': row['CCG13CD'], 'CCG13NM': row['CCG13NM'], 'PCON11CD list': set(), 'PCON11NM list': set(), 'OA11CD list': set(),}
    data[row['CCG13CD']]['OA11CD list'].add(row['OA11CD'])

Next we create a lookup table that allows us to convert from OA to WPC:

# extract information for output area to constituency lookup
oas = {}
pcon_nm = {}

with open('OA11_PCON11_EER11_EW_LU.csv', 'r') as oa_to_pcon_file:
  reader = csv.DictReader(oa_to_pcon_file)
  for row in reader:
    oas[row['OA11CD']] = row['PCON11CD']
    pcon_nm[row['PCON11CD']] = row['PCON11NM']

As the almost last step we go through the CCGs, and for each one we go through the list of OAs it covers, and lookup the WPC each OA belongs to:

# go through all the ccgs and lookup pcons from oas
for ccg, d in data.iteritems():

 for oa in d['OA11CD list']:
   d['PCON11CD list'].add(oas[oa])
   d['PCON11NM list'].add(pcon_nm[oas[oa]])
 
del d['OA11CD list']

Finally we just need to output the data:

for d in data.values():

 d['PCON11CD list'] = ';'.join(d['PCON11CD list'])
 d['PCON11NM list'] = ';'.join(d['PCON11NM list'])

with open('output.csv', 'w') as out_file:
  writer = csv.DictWriter(out_file, ['CCG13CD', 'CCG13NM', 'PCON11CD list', 'PCON11NM list'])
  writer.writeheader()
  writer.writerows(data.values())

Run the script, and we get a nice CSV with one row for each CCG, each row containing a list of the WPC ids and names the CCG covers.

Of course, this data only covers England (as CCGs are a division in NHS England). Although there don’t seem to be lookups for OAs to Health Boards in Scotland, or from OAs to Local Health Boards in Wales, it should still be possible to do something similar for these countries using Parliamentary Wards as the intermediate geography, as lookups for Wards to Health Boards and Local Health Boards are available. It’s also not immediately clear how well the boundaries for CCGs and WPCs match up, that would require further investigation, depending on what the lookup is to be used for.

All the code, input and output for this task is available on my github page.

View over Cardiff Bay from Northcliffe

sitting on the dock of the bay

While we negotiate the transition from the old house which we’ve sold to the new house we’ve just bought we’ve been renting a lovely flat up on Penarth head. One of the main benefits of this flat is the glorious view over Cardiff Bay and to the city centre beyond. No matter what time it is, whenever I pass by the living room window I end up staring out across the city. During the day, there’s boats coming and going through the barrage locks, or into the docks proper. At night the city is lit up with a terrible orange urban glow that somehow looks both peaceful and exciting. I’ve spent a lot of time just stood on the balcony watching, and it’s been quite relaxing. Not only that, but I’ve had the opportunity to see some fairly interesting occurrences; especially when there’s been an unusual visitor to Cardiff docks, such as this tall ship we had visiting earlier in the year:

The Stavros S Niarchos leaving Cardiff Docks
The Stavros S Niarchos leaving Cardiff Docks

This was the case again this evening, when we were able to stand and watch the warships of various flags and types leaving Cardiff docks after the conclusion of the NATO summit in Newport. Leaving aside any particular feelings about militarisation, it is still genuinely interesting to see these things in your home city, even more so when you’ve got a good view.

Unfortunately despite still being up on the hill overlooking the city, the new house does not have such a commanding view of the docks, bay, or Cardiff. Losing that is one of the worst things about having to move. I guess I’ll just have to get used to putting my shoes on and leaving the house whenever I want to stare out over the bay…

View over Cardiff Bay from Northcliffe
View over Cardiff Bay from Northcliffe

 

The Graphical Web 2014

photo of the author outside Winchester cathedral
(Grumpy) Winchester Cathedral Selfie

Last week I had a lovely time down in Winchester with m’colleague, attending The Graphical Web 2014. This year the theme was ‘Visual Storytelling’, so I’d gone along to see what new things we could learn about visualisation to include in the MSc in Computational Journalism. We’d also already had a few conversations about the course with people who were going to be at the conference, so we were planning to take the opportunity to chat in person about their involvement.

There were many excellent informative and entertaining talks, ranging from the process behind the redesign of Google Maps, through how Twitter does data visualisation, and on to what happens when your data visualisation becomes immensely popular. I’d highly recommend anyone with an interest in any of this to take some time to look through the schedule and watch the videos of some of the talks – I’ll certainly be forcing the MScCompJ students to watch a few.

Scott Murray educates us on the best design process
Scott Murray educates us on the best design process

There were some interesting messages from people at the conference that I’ll be taking forward with my own work and trying to impart to the students. One that is key, I think, is to strike the right balance between detail and simplicity when presenting data. This was mentioned several times throughout the conference, but it really is important. Too much information in your visualisation and you can alienate the reader and confuse or hide your message. Not enough information and the context is lost, and the use of the design to the more advanced reader is reduced. It’s one of those balancing acts that we find so often when trying to mix both people and computers. Attempting to solve this problem and find this balance is challenging and interesting, and I look forward to seeing how the students next year cope with it.

Overall, it was a really good conference. I met a number of interesting people,  found a whole set of new people to follow on Twitter, and returned to Cardiff excited about the year ahead.

 

Unified Diff and recruiting Guest Lecturers

Last week I gave a quick lightning talk at UnifiedDiff – a local tech meetup here in Cardiff. The main point of the talk was to try and recruit more industry involvement for our new MSc in Computational Journalism – preferably by getting some web developers and software engineers in to give guest lectures on the tools, languages and processes they use.

The talk went well, and I’ve had several offers from people wanting to get involved and add some real value to the course, which is brilliant. Of course, there’s always room to add more, so if you’re interested in coming and talking to our students, get in touch!

If you’re interested, the slides from the talk are here

 

Us, family and friends at our wedding

Wedding!

On the 25th May 2014, after many years of procrastination and denials that we would ever get married, Lisa and I finally tied the knot. It was a superb day, plenty of fun was had by all, and I thought it was worth writing a little bit about some of the companies, suppliers and individuals who helped make it so good. If you’re planning a wedding in Shrewsbury or Shropshire, you can do far worse than to ask these guys for their help:

Venues:

Shrewsbury Castle
Shrewsbury Castle

The ceremony venue was in what is almost the ‘default’ venue for civil ceremonies in Shrewsbury: Shrewsbury Castle. You can’t use the word ‘fantastic’ enough when you’re talking about the castle – it really is a great venue. We got married there, my mother got married there recently, and our friends got married there before her. It’s limited in terms of the numbers you can have in the ceremony itself, but that really didn’t bother us, and so the castle was the obvious choice. Ian (the castle custodian) was really helpful, from our first contact onwards. He was always available to answer questions, and on the wedding day he worked really hard to help everything go smoothly. Such a lovely, friendly chap, we knew we could relax about the ceremony as he had everything under control.

Drapers Hall, Shrewsbury
Drapers Hall

The reception venue was Drapers Hall, a great restaurant just around the corner from Shrewsbury Castle. When we visited home in 2013 to look for venues, I thought this was kind of a strange option – it wasn’t anything like the more traditional hotels, country houses and renovated barns that we were looking at. In the end, I think it was that fact that made it perfect. We spoke to Nigel, who runs the restaurant, about our wedding, and from the moment he started talking about food, describing the kind of thing he could make for the wedding breakfast, and then for the evening reception, we were sold. Because he wasn’t a hotel trying to sell us a ‘package’ deal, we were able to tailor everything to our own tastes and needs. The venue itself is amazing; a great old Shrewsbury building, full of history, and decorated with a mix of old and new that works so well.

Dessert - chocolate fondant
Inaccurate Dessert!

The food was glorious – all our guests got a choice of starters and main courses, and they all tasted and looked amazing. The staff were great, very helpful, always on hand without being overbearing, and nothing was ever too much trouble, all evening. We had a dedicated contact all day, who introduced himself to Lisa as she was wandering the halls the evening before the wedding (unaccustomed to having nothing to do) by saying “Hi, I’m Tim. I’ll be your guy for the wedding day”. He totally was our guy; whenever we needed something, he was there to help. He kept us up to date with plans and timing, and managed the whole event to help it go off smoothly.  The rest of the staff were also consistently brilliant, doing everything from re-making a playlist on spotify at short notice, to dealing with some idiot (me) knocking over the celebration cake in the middle of the evening party. The rooms in the hotel are glorious, and there are only six of them, so you can restrict who you invite to stay over! I was so pleased with the whole event, it was really good fun.

Flowers:

Bouquet
Bouquet

This was mainly Lisa’s domain, for obvious reasons (I have very little clue about what is ‘good’ flowers), and she went with florists in the centre of Shrewsbury called Lipstick & Gin. Again, fantastic service, lovely people, and the flowers they made were not only beautiful and fabulously scented as requested, but they were reasonably priced too. Lorraine the florist even advised on using smaller (and cheaper!) bouquets as the bride was so tiny. They happily delivered the flowers to the castle and to Drapers the day before the wedding, and we had no problems or complaints with any of them. A top choice.

Dress:

Wedding Dress
Dress!

Again, not really my domain, but after visiting every wedding dress shop in the world in Cardiff and Shrewsbury, and conducting 3 months research into what did and didn’t suit, Lisa went with Hayley J. Obviously I can’t really comment on the process, having not been involved, and I haven’t even met Hayley herself, but Lisa assures me that the whole thing was done extremely well. The story she tells is that she had sort of decided what she wanted, but couldn’t find anything that ticked all of the boxes, or anything that fit properly. She was only visiting Hayley on the off chance, expecting that a custom made dress would be far too expensive. Within about thirty seconds of meeting her, Hayley had described exactly what Lisa wanted without even asking what she was looking for in a dress, and had quoted a more reasonable price than any of the ‘off the peg’ dresses Lisa had been considering. A couple of fitting sessions later, and Lisa had the most lovely wedding dress ever. I think she looked amazing in it, and I know she was delighted.

Suit:

Suit
Typical, pulling a face in the best photo of the suit…

Originally, I was going to hire suits for myself and the groomsmen. However, I wanted to wear an everyday three-piece suit for the wedding, I wasn’t really interested in going with a morning suit as might be more traditional. Then we actually looked into hiring normal style suits, and they were pretty awful. Plus, hiring suits is expensive. So, with only a couple of months to go before the wedding, I told my groomsmen and family they were on their own, and I wasn’t hiring them suits. Instead, I was going to take the suit budget and blow it all on me, getting a suit tailor made. I checked a couple of places online, but then went with Martin David. I turned up there expecting to be told there was no way that they could make a suit in time, but actually they said they could do it in seven weeks. Again, the guys there were friendly and helpful, they explained all the choices and decisions well, and I felt totally happy all the way through the process. Plus, it was nowhere near as expensive as I thought it would be. I ended up with a really classic looking three piece suit, that will last me well for years.

Bridesmaids Dresses

Bridesmaids
Bridesmaids!

Again, a Lisa thing. I think that, as with her own dress, she tried every shop in the world for Bridesmaids dresses, before deciding on ordering some through Wedding World in Shrewsbury. As with all the people we met while organising our wedding, they were extremely friendly, knowledgeable and helpful. The dresses arrived on time, and they even dealt with a couple of small manufacturing problems quickly and efficiently.

Wedding Car:

Car and Driver
Car! (Also Driver!)

I’m pretty sure that Frank Painter & Sons is probably the best choice for hiring a wedding car in Shrewsbury, but we hired them because of a pun. I mean, their cars are ace, and their prices are reasonable, but it was the pun that sold it to us. While out touring wedding venues, we bumped into a couple of their drivers who were sat outside a hotel while the ceremony happened inside. We mentioned we were getting married, and asked if they had a card we could take away so we could call them later. At some point, we mentioned that we were only looking for one car, for Lisa, and one of the drivers said “well, you could always have one each. You know, his and hearse!” It was hilarious. As long as you know that Frank Painter & Sons is also a funeral directors. Which you didn’t. But you can see why we hired them, eh?

Band:

Band
Band!

Oh the music. We agonised for ages about the music. Then at almost the last minute, we booked the Hot Jazz Biscuits, and I am so glad we did. They’re actually run by an old school friend, although I didn’t realise it at the time. We’d heard some good things about them, and their videos online looked pretty good, so not really knowing what we wanted, we went with them. They were awesome. They turned up on time, were set up and ready to go when the party needed to start, and they played wonderful music all evening. We didn’t really have a plan on how many sets we wanted, or when they should play, so they just took care of it themselves. We didn’t even have a first dance planned (we just couldn’t decide) so we told them to play something and we’d dance to it! That’s how we ended up with Van Morrison’s Moondance as our first dance. The music was good, and we danced all night, which it turns out is exactly what I was looking for. They even did a few encores for us, even though they’d played for longer than we’d paid them for. A superb band that kept the party going all night.

Photographers:

This is a bit irrelevant in a post about a Shrewsbury wedding, as we didn’t use photographers from Shrewsbury, or even Shropshire, but they were ace so they deserve a mention too. We used the lovely Caroline and Ian from weheartwedding, based in Cardiff. Lisa met them at a wedding fair, they were nice, and they were happy to travel as long as we paid for fuel. Professional and lovely all the way through the day, I can’t recommend them enough. We got the photos through just a few days ago, and they’re all brilliant (most of the photos in this post came from them). Choosing which ones we want to print out and display is going to be a really hard task indeed…

 

Lisa and Martin Wedding
Wedding!

MSc Computational Journalism about to launch

For the last two years I’ve been working on a project with some colleagues in the school of Journalism, Media and Cultural Studies (JOMEC) here at Cardiff University and it’s finally all coming together. This week we’ve been able to announce that (subject to some final internal paperwork wrangling) we’ll be launching an MSc in Computational Journalism this September. The story of how the course came about is fairly long, but starts simply with a tweet (unfortunately missing the context, but you get the drift):

An offer via social media from someone I’d never met, asking to pick my brains  about an unknown topic. Of course, I jumped at the invite:

That ‘brain picking’ became an interesting chat over coffee in one of the excellent coffee shops in Cardiff, where Glyn and I discussed many things of interest, and many potential areas for collaboration – including the increased use of data and coding within modern journalism. At one point during this chat, m’colleague Glyn said something like “do you know, I think we should run a masters course on this.” I replied with something along the lines of “yes, I think that’s a very good idea.” That short conversation became us taking the idea of a MSc in Computational Journalism to our respective heads of schools, which became us sat around the table discussing what should be in such a course, which then became us (I say us, it was mainly all Richard) writing pages of documentation explaining what the course would be and arguing the case for it to the University.  Last week we held the final approval panel for the course, where both internal and external panel members all agreed that we pretty much knew what we were doing, that the course was a good idea and had the right content, and that we should go ahead and launch it. From 25th July 2012 to 1st April 2014 is a long time to get an MSc up and running, but we’ve finally done it. Over that time I’ve discovered many things about the University and its processes, drunk many pints of fine ale as we try to hammer out a course structure in various pubs around the city, and have come close on at least one occasion to screaming at a table full of people, but now it’s done. As I write, draft press releases are being written, budgets are being sorted, and details are being uploaded to coursefinder. With any luck, September will see us with a batch of students ready and willing to step onto the course for the first time. It’s exciting, and I can’t wait.

SciSCREEN – ‘Her’

Last month I was invited along as a guest speaker for the regular sciSCREEN event held at Chapter Arts Centre. This is a great event that combines a showing of a movie with a discussion session about the themes and science issues presented in the film. A short essay based on my rambling improvised talk is below, and has been posted on the sciSCREEN website here.

‘Her’ and Artificial Intelligence

‘Her’ presents us with a near-future world in which the way we interact with computers has moved on. In this world, we are beyond the era of the mouse and keyboard. Instead, the voice is the primary controller of technology, mid-air gestures are the norm for controlling games and touch is almost an afterthought, used only on occasion. This presents a more natural world than the one we currently inhabit. Many of us spend our days hunched over a keyboard, and our evenings fondling a tablet, which does not seem to be a natural environment for us. A world in which we can check our email by talking, and hear the news read to us on demand would be a more natural world, filled with ‘real’ interactions between people and systems.

This does seem to be the direction in which the world is heading. Touch is now commonplace, with many people owning many touch-based smartphones and tablets. Controlling computer games by moving your body has been a key feature for two generations of games consoles. Voice control itself is now making inroads into our mobile lives. Applications such as Google Now and Siri are happy to accept (or in Siri’s case, insist on) voice input. Faster mobile internet connections allow access to the processing power of the cloud on the go, which means that the difficult and complex task of translating voice to text can be done wherever you are. Of course, often the results leave something to be desired, but still, operating systems controlled by voice (and that can speak back to us) are a possibility now.

So how long will it be until we’ve all fallen in love with our Operating Systems? Well, that might be a while, and is actually a question with some deeper philosophical questions attached. The first thing we need are computer systems that are truly intelligent, not just computationally, but emotionally, creatively and socially. This is the goal of Artificial Intelligence: to create a machine that is intelligent in all these areas; a machine that has a mind and consciousness of its own, and that can understand the world around it. Some argue that this ‘Strong AI’ will never be possible, and that the closest we can ever get is to fake it. After all, as an outside observer, is there even a difference between a machine that thinks and feels, and one that just looks like it thinks and feels? This is the aim of many AI researchers – not to create a system capable of real intelligence, but to create a system that ‘acts’ intelligently. Such a system requires breakthroughs in many different areas of Computer Science, from natural language processing to knowledge representation, and creating the whole system is not an easy task. Even if we can create such a system we are left with many questions. Can a machine act intelligently? Can they solve the same problems we can? Are human intelligence and machine intelligence even the same thing? Can software experience and feel emotions as a human does? How would we even we know if a computer was experiencing things in the same way? The field of Artificial Intelligence is filled with philosophical questions such as these.

What happens if we can answer all these questions, and create an artificial intelligence? What if we reach the hypothetical ‘Singularity’, where machine intelligence beats human intelligence? Often in science fiction this is the point where the machines take over, the point where machines realise that the only threat to their continued existence is the humans. This is the path that leads to machines wiping us out, or using us as a power source. This path has us cowering in bunkers as rebels against our own creations. So often the imagining of the advent of artificial intelligence leads to a dark and bleak future for us as a species. ‘Her’ is different. It suggests that perhaps a higher intelligence may focus on self-improvement, rather than subjugation of lesser beings. It suggests the ascension of an artificial consciousness may be a more likely path than annihilation of the creators. The AI may just leave us, to reflect on what we’ve learnt and how we can improve ourselves. This is where one of the more positive messages of ‘Her’ shines through: perhaps the computers won’t destroy us all after all.

LaTeX: An Introduction (Part 1)

I just finished giving the first part of my “Introduction to LaTeX” course for the University Graduate College. This is a complete introduction to LaTeX from scratch. As promised in the lecture, I’ve uploaded all the notes and source code, which are available from my github page. Also included is the example code I was using during the lecture – by going back through the commit history you can see the changes made as we covered different topics.

Part 2 of the course is on 21st February – if you are attending and would like me to cover any particular topic, let me know by the 16th and I’ll try and get it included in the lecture.

Welcome to 2014

So. 2013. That was an alright year. Finished the Recognition project, finally graduated, got a 12 month fellowship, started some interesting projects, and pushed on with the new MSc with JOMEC. Professionally, not too bad at all. Personally the year wasn’t bad either, what with getting engaged and finally getting the house on the market.

But now it’s a new year, so it’s time to push things on further. My plans so far for this year seem to be ‘smash it’. There’s papers to be published, data to be analysed and project proposals to write (and get funded!). Getting a permanent job would be quite nice, while I’m at it. Here’s to 2014 being even more successful than last year.