TweetCues Twitter Experiment

Posted on by
Reply

As I mentioned in the last post, our Twitter Experiment (called TweetCues) is now live. Just this morning I updated it to version 2.0 to begin collection of data for a journal paper, assuming of course that our conference paper on the initial findings gets accepted somewhere.

TweetCues Homepage

TweetCues Homepage

This has been an interesting project that has taken quite a long while to come to fruition. Stuart had the idea for the experiment way back at the beginning of the Recognition project – in fact the initial project document is dated from March 2011. We thought it would be a good quick easy win for the start of the project, and would get us a publication very quickly – how wrong we were.

The reasons for it taking so long to get off the ground are many. Firstly, there’s the event we don’t talk about from last April. We were also distracted by a couple of other conference papers that needed writing, and then a new idea for the next experiment that became quite distracting. There was also the small matter of closing the SocialNets project and getting the Recognition project off the ground in the first year deliverables. However, in the background this project has always been ticking along.

I actually started implementing the experiment in August and once I got going it was pretty quick. I used Django for the site, so development was quick and easy, and my git logs show a commit from the beginning of September with the message “basically got everything working“. It didn’t go live then, and was actually a very different experiment. There was a lot of concern over comparing tweets and showing the content – we wanted to remove the actual content of the tweet from the decision making as much as possible, so I was doing a lot of tweet analysis using AlchemyAPI to try and match tweets so they were as close as possible in terms of topics or keywords. This rarely worked. Between August and December I spent time bug fixing and attempting to improve the tweet matching, but it wasn’t really happening.

Then in December we find a git commit message: “removed all the old code to start to create new version post meeting with Marc Buehner“. We decided at some point that as we’re scientists that don’t generally do people-based experiments it would be good to get some feedback from someone that does, so in December 2011 we met with Marc Buehner from the School of Psychology for a meeting that produced a lot of useful ideas for strengthening the experiment, but that meant it would be easier to rewrite from scratch than to hack around at the edges of the application. It was at this meeting that we came up with the ultimate solution for removing the content of the tweet from the decision making process: don’t show the content. How it took us that long to come up with this solution I will never know.

By January the experiment was “pretty much totally working” according to my git commits, so the re-implementation did not take too long. Working out deployment onto the school web server took a little while, but eventually it was done, we went live, and collected our initial data. The findings are pretty interesting, (which will be covered in another post) but the analysis of this data revealed that we could actually do with collecting a little more information about the user interaction with the experiment – hence v2.0 going live this morning.

Update

Posted on by
Reply

So, I’ve been kind of ignoring this blog, haven’t I?

Ah well, never mind. I have a good excuse as I’ve been very busy working on something else, which I’ll explain in a post sometime soon. For now though I’ve made a decision to use this blog more and also to include more regular small updates on work things as well as the other stuff I’ve been doing outside of work.

So, what work stuff have I been doing lately?

Twitter Experiment

We recently launched our twitter based experiment here. Uptake has been very good, the initial results are quite interesting and we’re currently prepping a short paper for SocialCom which I personally think has a pretty good chance of being accepted.  I’ll post about some of the initial findings soon.

Our analysis has revealed a couple more pieces of data that we need to collect, so I’ve got a couple of modifications to make to the site then we can hopefully start spreading the link a bit further to get a few hundred more respondents for a nice journal article.

Foursquare Experiment

The initial plans for this have been scaled back and refined, but this should be online soon. I’ll post more in the next couple of weeks as I finish off development on this and get it ready for deployment.

CUROP Summer Projects

Walter and I were successful in applying for some CUROP funding for a summer project looking at retweet flow in Twitter. We’ve had a lot of interest in the project, so much so that we’ve managed to convince a couple of lovely people to part with some more cash so we can take on another student for an additional summer project, this one based on Twitter ranking. I’ll post about both projects in detail once we’ve made our decision on which students to take on for the summer.

Mobisoc Website

Over the course of about a day Matthew Williams and I recently hacked together a website for the MobiSoc researchers here at CU, which is now live and will hopefully start to flourish as more people get involved. It’s not much at the moment, but it could be good…

 

As well as those projects there’s been discussions with some friendly scientists from Cambridge about getting our hands on some personality data, the ever present EU deliverables that have needed sorting, and the logistics for the next project partner meeting in Athens in June. It’s been a busy couple of months.

Truckers of Husk + Gallops + Kutosis Review

Posted on by
Reply

Last Saturday I went along to The Globe in Roath, accompanied by my excellent musical companion Ms. Jones (and joined later on by our token northener Mark) to watch a number of bands that we’d seen before, but in a new setting. This is a review of that what happened there.

Let’s start with the venue, The Globe. The Globe is not the worst venue in Cardiff (Bogiez/Barfly, The SU Great Hall). It’s tucked out of the way in the corner of Roath, so anyone going to a gig there is *meant* to be there, which is nice. It’s an old cinema and has a lovely balcony, meaning a great view of the bands, and most importantly the sound in the place is not terrible. The only bad thing about the venue this week was that they’d sold out of tolerable ale, which is no great problem. We arrived about twenty minutes after doors were due to open, and unsurprisingly the doors were not open yet. For some reason The Globe exists in a timezone that is about 45 minutes behind the rest of Cardiff, so don’t ever bother getting there on time, it’s just not going to happen. Rubbish timekeeping and crappy beer selection aside, it’s a decent venue in which to watch a band and on this night there were three on offer.

KUTOSIS

I’m fairly sure I’ve seen these guys a number of times before, at least once or twice at the SWN festival, maybe somewhere else as well. They’re a decent band rocking a mid-late 90′s vibe that marks them out as guys that grew up with grunge and late 90′s indie and decided the best thing to do when they started a band was to emulate that. Unfortunately you sometimes feel that they forgot to really add anything to that era, and haven’t quite managed to push things on. Don’t get me wrong, it’s not a totally bad thing – they play well, they put effort into what they do, and their songs have a catchy uptempo feel. I enjoyed watching them, they were having fun and playing well, with some good riffs, tight playing and a good overall feel to the music. There are some interesting things going on with their songs, but it just feels like they need to push things a little bit further out of a safe zone and do something a bit more interesting. That all being said, they’re well worth a watch, especially if you’re like me and you grew up with grunge and late 90′s indie and enjoy that kind of thing.

This track of theirs has a real White Stripes feel:

This is a more recent track that seems in parts to show more development, it’s got more going on:

Overall they were enjoyable, played well, and I’ll give them 4 out of some.

Gallops

Again, I’ve seen Gallops a number of times before, mainly at SWN. On paper they’re not my thing, focusing fairly heavily on electronic sounds, but in reality I really love them. They’re an awesome live act and every time I see them they have something new to show. The main driving force behind them is the thundering and perfect drums; their drummer is an absolute animal, arrogant and agressive and with every right to sit behind the drumkit with the knowledge that he’s almost certainly the most talented guy with a set of drumsticks in the room. He almost imbalances the band, as with his drumming there’s not a lot the other band members can do to match his talent, which is probably one of their only weak points. The only other glaring weak point would be the guy with the mac. I’ve said it before and I’ll say it again, a laptop is not an instrument. If your job could be replaced with someone pressing play on a tape deck you’re superfluous, get the fuck off the stage. That aside, this band have grown massively in the two/three years I’ve been watching them, and I’m looking forward to their album in the next couple of months to see where they’re going next.

Once again these guys were great live. They clearly spend a lot of time practising getting it right, and it pays off. I’ll give them six out of some.

Truckers of Husk

Ah, the main event. A band that I first saw at the 2nd SWN Festival, in whatever Y Fuwch Goch was before it was that (and then failed to be that and became the Moon Bar). A band that have swiftly become one of my favourite live acts in Cardiff, a band who I love going to see. Their post-rock math vibes are great, and they are another band (like Gallops) who continue to grow and push stuff forward. Their first album, (released last year) is on permanent rotation in my Spotify playlist, and I still can’t get enough of it. In case you didn’t get it, I love this band.

Today’s gig was exceptional, the band presenting themselves along with the 1924 documentary ‘The Great White Silence’, the fairly harrowing documentary about Scott’s ill fated voyage to the South Pole. Dressed all in white the band played their songs along with the film, fitting the tracks to the projection behind them, timing things beautifully to coincide with the action. I’m fairly certain they’d put the effort in to trim songs here and there to make them fit, cutting some noise here, adding some there and it all paid off. They presented themselves on stage as a band having fun and doing well doing it, but you could tell that they recognised this was (one of?) their biggest gig to date and had put the effort in to make it something special. The songs flew by and by the time the movie drew to a close, with our intrepid arctic explorers dead in the arctic wasteland and the band themselves moving the drumkit down into the crowd for the last flour covered intense burst, we were all thoroughly sated. Plus the saxophonist kept his shirt on.

This track of theirs is off the latest album and is really good:

I think this is my favourite:

I give them 8 out of some.

All in all, a thoroughly excellent night of entertainment. Top marks.

100 Movies in the Cinema in 1 Year

Posted on by
Reply

After reading this blog post at the end of January I decided that watching at least 100 movies in the cinema in a year was the kind of challenge I could get behind. Last year’s challenge to get fit and healthy was fairly successful and seemed like much harder work than going to the cinema a few times could ever be. After all, 100 movies a year is less than 2 a week, so it must be easy, right?

Unfortunately I started this project late, so at the time of reading the article and deciding to go ahead with it I was four weeks into the year and had only seen three films at the cinema. This is not the kind of progress needed for this challenge! So, I got myself a cineworld unlimited card and hit the cinema hard, aiming to get through February and be caught up to where I should be by the end of the month.

Of course, being a massive data geek I have been recording everything, making a note of every movie I have seen this year so far, which means I’m amassing a fairly large amount of data on my movie watching habits, which can mean only one thing. Crappy Excel Graphs! I’ll be posting some throughout the year to mark how the challenge is going, and the first lot are here.

The first shows the number of movies seen in total for each day of the year against the number of movies I would need to see in order to hit the target of 100 by December 31st. As you can see, the late start in January did me no favours, but by the end of February I’ve almost caught up to be where I need to be.

Movies Seen against Target Number

The second crappy excel graph shows the movie viewing rate (the number of movies seen divided by the number of weeks elapsed) against the target rate, along with the actual number of movies watched that week. It’s clear to see that January was a wash out, but that February was excellent and helped to bring the target rate down below 2 movies a week.

Movie viewing rate against target, presented with actual number of films seen in a week

So, that’s where we’re at. It’s March and I’ve seen 16 films at the cinema, at an average cost of £3.18 per movie. I’m well on the way to 100 and I’ll keep you updated with more crappy excel graphs and rubbish averages along the way…

Django + Last.FM Authentication

Posted on by
Reply

Following a conversation with Jon last week, I’ve been having thoughts about playing around with Last.FM data again, having not looked at that API for a couple of years. We had discussed the possibility of using the Last.FM web services as part of a project at a hackathon that some of the undergraduates are running in a couple of weeks time, and since I’ve moved from primarily developing using Java to mainly using Python since I last did anything with Last.FM I thought it would be useful to develop some basic python examples using the Last.FM API.

Yesterday I made a basic skeleton Django site that uses Last.FM for user authentication and includes a basic API wrapper for making queries. The code is all up on github for those that are interested.

matplotlib + mac os x 10.7 (Lion)

Posted on by
Reply

Today I spent a good long chunk of my day shouting at my computer and trying to install matplotlib, a very handy python library. I finally solved it, so figured it was worth posting about here, in case anyone else has the same problem.

I recently updated my system python to version 2.7, so I’ve had to go through reinstalling various libraries that I use on a semi-regular basis. Matplotlib is one of these that up until now I’ve had no call for. However, my experiment project that I’m working on needs some pretty graphs, so I needed to get matplotlib installed.

I tried to install using pip, I tried to install using easy_install, and I tried to install by downloading the source and compiling it. None of these methods worked and I was often getting compiler errors like this:

llvm-g++-4.2: error trying to exec '/usr/bin/../llvm-gcc-4.2/bin/powerpc-apple-darwin11-llvm-g++-4.2': execvp: No such file or directory

Which is very annoying. So I spent some time googling, and it seemed that the reason for these errors is that the compiler was trying to build a universal version of some library – a version that would work on both 32 and 64 bit intel macs and older powerpc based macs. However, Apple have now removed all traces of powerpc support from osX and Xcode, so the compiler was failing. There is a way to add ppc support back to Lion, but either I was doing it wrong or those instructions are slightly out of date, because it didn’t solve my problem.  Now, depending on how I’d tried to fix the problem, it would fail with a lot of errors in compiling “ft2font.cpp”. Other times it would be accompanied with “i686-apple-darwin10-gcc-4.0.1: Internal error”. After uninstalling and reinstalling Xcode for the third time I realised I was going about this all wrong. I was trying to fix the problem, rather than look for why the problem was occuring. Why would my laptop try and build a universal library with powerpc support if it knew it couldn’t compile or run ppc software?

The answer lay with my version of python. Running

file $(which python)

revealed that my version of python was also universal:


/usr/local/bin/python: Mach-O universal binary with 3 architectures
/usr/local/bin/python (for architecture i386): Mach-O executable i386
/usr/local/bin/python (for architecture ppc): Mach-O executable ppc
/usr/local/bin/python (for architecture x86_64): Mach-O 64-bit executable x86_64

with support for ppc, i386 and x86_64 architectures. Because of this the compiler was trying to compile all the supporting libraries as universal as well.

I replaced python with the latest version (2.7.2), using the installer built only for i386/x86_64, and the problem was solved. Matplotlib built from source fine, and I’m pretty sure (though I didn’t test it) that I could have installed using pip or easy_install too.

A few hours spent swearing at the computer and an important lesson learned. Worth it.

Introduction to Django talk

Posted on by
Reply

Last week I was asked by Stuart Allen to give a talk to the students in his first year “Problem Solving with Python” course on the topic of django, the python web framework. As I’m not lecturing at the moment I like to do these odd talks or cover the odd lecture just to keep up the skill set for when I start doing it again properly, so I agreed. Plus, I think django is ace and it’s always nice to try and brainwash students into using educate students about your favourite technology :-)

The talk was actually really well timed as the students are currently working on a coursework in another module (“Web Applications”) where they are required to develop a blog application using MySQL and PHP. This meant I was able to show off just how simple it would be to do it in django instead - so I gave a quick introductory talk then did some live coding and created a pretty functional blog application in about twenty five minutes. I think it annoyed quite a few of the students to see the difference in complexity between hand coding everything in PHP and having django take care of most of the boilerplate code. It may even have been a bit evil of me, but hey, that’s what I’m here for.

The slides for the talk are attached to this post (Introduction to Django Talk), although they’re of the “you need to be there to get the details” style of slide design, so they may not be that useful. Thanks to Mike Cantelon for his django keynote theme, and the lovely flickr users who uploaded photos with a CC license so I could reuse them.

BoxUK “For the Social Good” Hackday

Posted on by
1

Yesterday I attended the “For the Social Good” hackday, organised by BoxUK and held at the Students Union at Cardiff University. As you may have gathered from the title, it was a hackday themed around creating apps that had some benefit to society. The event ran from 10am to 10pm, so it was a fairly short hack event compared to some, which had a big influence on what could be done during the time available. In total, given the time needed for introductions and getting started in the morning, then for presentations and judging at the end of the day there was only actually about nine and a half hours of coding time, so it was quite a tough day. I went along with Matt Williams and Mark Greenwood as our team and over the course of the day we managed to put together a fairly functional web app.

We took what we learnt from hosting and participating in the Foursquare hackathon and used that to guide what we did at this hackday. At the last event we had been slow to get going, took too long to form a team and decide upon an idea. We had aimed too high to begin with, and then also let feature creep affect what we wanted to build. Finally, our teams were far too big, meaning it was hard to divide labour up so that everyone was participating and had something to do. We finished with a working app at the end of the event, but we were determined not to make the same mistakes this time so we could get a more successful outcome. (Not that FourCrawl wasn’t a successful outcome of course, but more on that in the next couple of weeks.)

Going in to this hackday we solved the team size problem naturally, as there were only three of us able to attend which made forming a small team compulsory! To try and get a good start on the day of the event we met together on the Wednesday before to have a think about ideas and a bit of a brainstorming session. While we had a lot of ideas, most of them were too complex for a short day long hack event, so unfortunately we didn’t really come up with anything. We did have a vague idea of doing something related to cold weather though, to make whatever ‘it’ was topical, given that we’re heading into winter now. Fortunately at least one of us (not me) had obviously been thinking about it further after the meeting and on Saturday evening Matt sent round a message on G+ that he’d found a good data set we could use for an app that would be both easy to code within the time limit, and would hit the brief for the hackday pretty well. He’d found a dataset on data.gov.uk related to road accidents that included information on weather and road surface conditions that we could use to show people where to avoid in icy weather, or to give them an idea of previous incidents in their area. With an idea starting to form we went into Sunday in a pretty good state, and ready to start coding very quickly.

The day started well, everyone arrived pretty much on time, which made for a quick start. Our app idea solidified easily, and early on we realised that while we had historical data to display it would also be nice to have some current updates, so we added a social element to the idea, by scraping twitter for a certain hashtag so that people could report problems with icy roads in their area, along the lines of the ‘#uksnow’ hashtag that has been used frequently the last couple of winters. We were able to jump in quickly to start getting things built. Matt was in charge of getting the data into a database and building an api to query to retrieve it, using django to get a backend up and running quickly. I was in charge of building a front end map to display the data on, primarily working with HTML and javascript (and the Google Maps API obviously) and then Mark was in charge of getting the twitter scraping and postcode geo-locating working using python (so it could be slotted into the django backend), accessing the twitter API through tweepy.

Things went really well with the implementation, I spent a while cannibalising elements from previous sites to get a frontend up and running quickly (and also thank you bootstrap!) and by lunch we were pretty close to starting to consolidate things together. We set a target of starting to deploy by 6, so that by the time dinner was served Matt was starting to battle his server and its stubborn refusal to allow easy deployment of django apps. Things went so smoothly in fact that we were all finished and ready with almost an hour to spare, meaning we were the only team able to slope off for a self-congratulatory pint before the presentations and judging. We headed to the Taf (a place I hadn’t been in at least five years) for a swift one, then went back to the hackday for the final presentations.

Our presentation was first, and Matt demoed the app well while I played around on the laptop next to him. After us the rest of the teams went, and there were some really good apps shown off. My particular favourite was the ‘occupy where‘ app which allowed you to search for an ‘occupy’ camp or demo near a location, then presented information pulled in from multiple sources about that particular ‘occupy’ movement. A nice idea, well executed. Following the presentations the judges spent some time talking, then announced the winners. The surprise of the day was that they’d decided our app was the best and that we’d won! I was pretty shocked, but it was a very nice surprise. A group of first years from COMSC were runners up, and then a third year COMSC student (Tom Hancocks) won the best individual developer prize, so COMSC actually walked away with all the prizes! After a quick tear down of the venue it was time to head to the Taf for some celebratory drinks, a satisfying end to a really great day.

UPDATE 25/11/11:

There some excellent other write ups by various other people involved in the event below. I’ll add to this list as and when I find any more:

http://www.boxuk.com/news/box-uks-public-hacking-event-a-resounding-success

http://johngreenaway.co.uk/hack-day-for-the-social-good

http://marvelley.com/2011/11/24/for-the-social-good-box-uk-hackday-1-recap/

http://whitehandkerchief.co.uk/blog/?p=53

http://www.cs.cf.ac.uk/newsandevents/hackday.html

http://www.cardiff.ac.uk/news/articles/diwrnod-hacio-7933.html

http://www.boxuk.com/blog/the-box-uk-hack-day-a-review 

Foursquare Hackathon Post Mortem

Posted on by
Reply

Note: I wrote this post just after the hackathon, but then forgot to actually publish it! 

There’s a fair amount I learnt from taking part in and helping to organise the foursquare hackathon here at the university that probably deserves its own post. I’ll split it into two parts, the taking part and the organisation.

Organising a hack event

There were a couple of things I learnt from helping to organise the event that perhaps weren’t clear to begin with:

  1. Expect dropouts. A lot of them. We ended up with about 50% of the people that said they would attend actually attending. Which is fine because we hadn’t planned on providing catering etc. anyway, but if we had, we may have overspent massively.
  2. Get someone in to run the day that doesn’t actually want to code. All three of us organising the event really wanted to get involved and create stuff, so there was no-one left to organise the other essential things like food. Too often people would talk about ordering food, or going out for dinner and then get sucked back into programming and it would get forgotten about. By the end of the weekend we’d all eaten really badly, often very late at night. Having someone there to do things like order pizza or make coffee runs would probably have helped things go a lot smoother.
  3. If you’re going to be communicating with the world at large using twitter or something similar, make sure you communicate with each other within the organising team about posting updates. Often two people would try and reply to a query at the same time, which made us look a bit silly. Not a major issue, but it might help give a more professional feel, event if you’re total amateurs.

Participating in a hack event

We also learnt a lot about participating in a hack event. My top tips for a successful project would be:

  1. Do your research. If you can, go into the event with an idea already, just in case no-one else has any. It’ll help you get started quicker.
  2. Keep it small. Don’t over reach. 24/48 hours is not a long time to code something, and by making it too complicated you’ll be disappointed with the outcome. Simple is best. Add extra features later if you have time.
  3. Small, agile teams. These projects have to be small because of the time constraints, so if teams get too big there won’t be enough for everyone to do. People will end up feeling useless or left out, which is never good. I would say a maximum of 4 people per team.
  4. If you want to learn something new, a hack event can be a great place to be forced to up your skills in a particular area very quickly. It can also be a great place to learn new skills and tools from others. However, this may lead to the end result not being ideal. If you can live with that, great, you’ll walk away happy. I personally upped my javascript knowledge over the weekend from approximately 0 to something approaching a passable working knowledge, which was great.
  5. Know your tools. This is pretty much common sense, but if you’re not after learning something new and you just want a successful app/outcome, pick and use the tools you know really well. We went with django on the back end because myself and Matt know it pretty well, and we were able to get that part of the site running really really quickly. Had we gone with something else it may not have worked so well.
  6. Get coding. Screw the design, you can worry about that later. Start coding early, and code fast.
So thats the things I learnt from the weekend. Hopefully we’ll be able to use this knowledge again in the future, both organising and participating in future events.

Colourful Foursquare category icons

Posted on by
5

UPDATE 31/01/2011 – new version of the downloader script is here

A couple of projects I’m working on at the moment need the ability to add foursquare venues to a map. This is pretty straightforward, but what is also needed is the ability to distinguish between the categories of the venues. To do this, I’m adding custom markers to a google map at a venue location, with the marker having an icon representing the venue category.

Foursquare already have lots of category icons ready made, and they supply the urls in the /venues/category endpoint, which is handy.  It seems from official responses on the forum that they don’t mind you downloading these icons for use in your apps, and they also supply the icons in several sizes, so hacking together a script to download all the available icons is pretty easy. The following python code will do just that – grab the latest category hierarchy from Foursquare, process it to extract all the category id’s and icon locations, then download each icon in all available sizes:

import urllib2
import urllib
import json

# supply oauth token as parameter
params = {
    'oauth_token' : 'VALID_FOURSQUARE_OAUTH_TOKEN'
}

api_base_url = 'https://api.foursquare.com/v2'

icons = {}

# function to extract category info from json and
# process any sub-categories
def process_category( category_json ):
    icons[category_json['id']] = category_json['icon']

    if 'categories' in category_json:
        [ process_category( c ) for c in category_json['categories'] ]

if __name__ == "__main__":

    url = api_base_url + '/venues/categories?' + urllib.urlencode( params )

    # try and retrieve the json
    try:
        response = urllib2.urlopen( url )
    except urllib2.HTTPError, e:
        raise e
    except urllib2.URLError, e:
        raise e

    # read the json
    raw_data = response.read( )
    py_data = json.loads( raw_data )

    categories_json = py_data['response']

    # extract all the categories
    for category in categories_json['categories']:
        process_category( category )

    # download all the icons
    for foursq_id, icon in icons.iteritems( ):
        sizes = [ '_32', '_64', '_256', '']
        for size in sizes:
            try:
                url = icon.replace( '.png', '%s.png' % size )
                u = urllib2.urlopen( url )
                file_name = '%s%s.png' % ( foursq_id, size )
                localFile = open( file_name, 'w' )
                localFile.write( u.read( ) )
                localFile.close( )
            except Exception:
                pass

After running this script, we have a local copy of all the category icons to use in our application downloaded. In my case, I need to be able to differentiate between venues within the categories, so it would be nice to have different coloured icons for my different classes of venue. Luckily this is also pretty simple. You’ll need ImageMagick installed, but if it is, the following code will go through all the greyscale icons downloaded with the previous script and convert them to whichever colours you like:

import os

#
# colour name, and imagemagick colour name
colours = {
    'red' : 'red',
    'crimson' : 'crimson',
    'pink' : 'DeepPink',
    'purple' : 'purple',
    'indigo' : 'indigo',
    'blue' : 'blue',
    'navy' : 'navy',
    'turquoise' : 'turquoise1',
    'teal' : 'teal',
    'lime' : 'lime',
    'yellow' : 'yellow',
    'orange' : 'orange',
}

#
# list all the files in the current directory
current_dir = os.getcwd( )
files = os.listdir( current_dir )

#
# run through the file list
for fname in files:
    # for all the png's
    if '.png' in fname:
        # get the file name
        stripped_name = fname[ :fname.rfind( '.' ) ]
        # convert to a new colour and save
        for colour, magickcolour in colours.iteritems( ):
            colour_name = '%s_%s.png' % ( stripped_name, colour)
            os.system( 'convert %s +level-colors %s, %s'
                        % ( fname, magickcolour, colour_name ) )

The colour names come from this list of imagemagick colours, you can obviously pick whichever colours you want. If you run this script in the directory with the downloaded Foursquare category icons, you’ll end up with a multitude of differently sized, differently coloured icons:

And that’s that. Later I’ll post about how to use these as custom markers in a google map.

UPDATE 3/10/11:

As this example stores icons by category id, it actually duplicates some of the icons, which is un-necessary. This version (Foursquare Category Icons downloader) stores the icons by icon name, so saving the downloading and processing of around 100 duplicate icons. It also splits the different sizes into different directories for better file management.

UPDATE 5/10/11:

There’s a change in the category json coming which will break this code. Use “v=20111002″ as an extra parameter in the code that calls the Foursquare API and it’ll keep working.

Foursquare Hackathon Cardiff – Day 2

Posted on by
Reply

Wow. I thought Saturday was busy, but I was wrong. Saturday was a cake walk compared to the crazy nightmare that was Sunday. The challenge of trying to get something (anything) working before the app submission deadline was incredible. Coding was fast and intense, and required a lot of good communication and fast thinking. Unfortunately we hit a few roadblocks that slowed us down (2 hours debugging problems with django’s static file serving really didn’t help, neither did jQuery mobile being an absolute pain in the rear end), but by the end of the day we had a nice looking site with some actual working functionality. Nowhere near the level of functionality that we’d hoped for, but it’s a start. All three of us that were working on the app towards the end are aiming to carry on the development, so hopefully I’ll have another project to post about up here as the months go on.

A number of people didn’t make it to the Sunday, which is a shame, but the core teams of both projects were in early and working hard, and by the end of the day both had been declared ‘working’ enough to submit. Maybe they’ll do ok against the competition, but we’ll see. A couple more hours on our pub crawl app could really have helped as there’s some key functionality missing, but there just wasn’t time to get it working. Overall though, the event was a success. We managed to pull in people from outside our own hyper-local school development team, we all managed to learn some new things and teach some things to others, and I think most people genuinely had fun. It’s definitely an experience I’m looking forward to repeating, both as a hackday organiser and as just a participant. BOX UK have a ‘For the Social Good‘ hackday coming up in November that I’ve already registered for, and I’ll be keeping my eyes open for others in the future, and for opportunities to organise more.

I think I learnt a few things about organising and participating in these sorts of events, so I’ll be putting up a post in the next few days to share the knowledge.

Foursquare Hackathon Cardiff – Day 1

Posted on by
Reply

I was fully intending to try and live-blog the progress of the foursquare hackathon today, even going as far as to install a live blog plugin, but things have been going very quickly and we’ve been hitting the coding hard so I just haven’t had time!

We’ve managed to get a decent crowd in, a roughly 50:50 split of people we already knew from the School and other people from elsewhere. At the peak today we had fifteen people in, but at this stage we’re down to ten – but those who have left are coming back tomorrow! There’s an interesting mix of skills and knowledge levels and a number of people have come who are just eager to see how other people do things and to learn more about foursquare or coding in general. Luckily everyone is talkative and keen to get on and create something cool.

We started this morning at about 10am and managed to come up with a few ideas in a brief brainstorming session, eventually splitting ourselves into groups working on two ideas. The first idea is being driven forward by Vlad and is quite related to the work we’re doing on Recognition – creating a system to pull in information on a venue or location and create a ‘digest’ of what’s relevant about the place. This could come in handy later on for our research. The second idea is a more ‘fun’ idea – building a system for creating, managing and participating in pub crawls. There’s a couple of parts to this: a management website that allows users to search foursquare venues and add them to a crawl, a backend that manages the storage and retrieval of users and crawls and an application front end that allows you to participate in the crawl itself. At this stage early on day one we’ve got most of the backend built, and the management web app is coming along well. Tomorrow we’re going to start the android app.

Some pictures of the day so far are below:

Foursquare Hackathon

Posted on by
Reply

Along with Matthew Williams and Chris Gwilliams I’m helping to organise the Cardiff edition of the Foursquare global hackathon this weekend. I’m hoping it’ll be a pretty good event. Signup‘s are looking pretty positive and I’m hopeful that we’ll get some decent apps being created.

The organising is actually quite fun, but it is remarkable how much there is to do and remember. Thankfully the School of Computer Science & Informatics are being very supportive and allowing us to host it there, which makes things easier. I’ll be posting here throughout the event (and afterwards).

Django + Foursquare + OAuth = webapp?

Posted on by
Reply

Moving on from my fun with getting django + twitter + oauth to play nicely, it’s now time to start writing a web app that uses foursquare as the authentication/login method. As part of this, I need a basic django framework for doing authentication with foursquare. This may be useful to someone else at some point (including me when I forget in three months time how I did things), so I’ve written up the details and added the code to github.

Firstly, you need to register an application with Foursquare to get your Client ID and Client Secret. You can enter any name and website you want, but the callback url needs to be our local development machine (assuming that’s where you’re doing the development), and to be the callback url that we’re going to specify in a minute:

Form to register an application

Form to register an application with Foursquare

The more observant will notice that the callback url is currently http://127.0.0.1/foursq_auth/callback/, but that the django web server actually runs on port 8000, so the url really needs to be http://127.0.0.1:8000/foursq_auth/callback/. Unfortunately the form validation doesn’t allow you to use a ‘:’ in your callback url. However, once the consumer is registered you can edit the callback url, and the form for that doesn’t validate the url (or does but doesn’t moan about the colon). Eventually, you are looking for the consumer to be registered with a callback url like this:

Once that is registered, we can get on with writing the code. The basic premise is pretty simple, we need just five urls in our django app, five views to go with them, and two basic html pages, one for login, and one to show we are logged in. Assuming you have a django project started (for reference the one I’m using here is called ‘Dj4sq’) we can start by creating our foursquare app:

django-admin.py startapp foursq_auth

We then want to edit urls.py for our main django project to allow the new app to handle its own views:

urlpatterns = patterns('',
    (r'^foursq_auth/', include('Dj4sq.foursq_auth.urls')),
)

and create and edit a file urls.py in the foursq_auth app with the five views we want:

    urlpatterns = patterns('foursq_auth.views',
    # main page redirects to start or login
    url(r'^$', view=main, name='main'),
    # receive OAuth token from 4sq
    url(r'^callback/$', view=callback, name='oauth_return'),
    # logout from the app
    url(r'^logout/$', view=unauth, name='oauth_unauth'),
    # authenticate with 4sq using OAuth
    url(r'^auth/$', view=auth, name='oauth_auth'),
    # main page once logged in
    url( r'^done/$', view=done, name='oauth_done' ),
)

We then need our two basic html pages login.html and done.html:

<html>
    <head>
        <title>Foursquare OAuth Example</title>
    </head>
    <body>
    <div id="login">
        <p>Sign in with Foursquare to begin.</p>
        <p><a href="auth/">Login</a></p>
    </div>
    </body>
</html>
<html>
    <head>
        <title>Foursquare OAuth Example</title>
    </head>
    <body>
        <p>Hi {{ name }}, you've successfully logged in! </p>
	<a href="../logout/">Logout</a>
    </body>
</html>

All that’s left after that is to write the views to tie it all together. We need a few variables defined for our code to work:

CLIENT_ID = 'YOUR_CLIENT_ID'
CLIENT_SECRET = 'YOUR_CLIENT_SECRET'

request_token_url = 'https://foursquare.com/oauth2/authenticate'
access_token_url = 'https://foursquare.com/oauth2/access_token'
redirect_url = 'http://127.0.0.1:8000/foursq_auth/callback'

We can then write the views. The first one, ‘main’ is super easy, as it only needs to return our login page:

def main( request ):
    return render_to_response( 'foursq_auth/login.html' )

The second, ‘auth’, is the first step in our authentication process, requesting an authorisation code from foursquare and redirecting the user to the page to authorise our app:

def auth( request ):
    # build the url to request
    params = {'client_id' : CLIENT_ID,
            'response_type' : 'code',
            'redirect_uri' : redirect_url }
    data = urllib.urlencode( params )
    # redirect the user to the url to confirm access for the app
    return HttpResponseRedirect('%s?%s' % (request_token_url, data))

If the user accepts our app, they’ll be re-directed to our callback url, and an authorisation code will be passed back as one of the parameters of the url, so the next view, ‘callback’ needs to deal with this. It then needs to post a request to foursquare with the authorisation code in order to receive the access_token for the user:

    # get the code returned from foursquare
    code = request.GET.get('code')

    # build the url to request the access_token
    params = { 'client_id' : CLIENT_ID,
               'client_secret' : CLIENT_SECRET,
               'grant_type' : 'authorization_code',
               'redirect_uri' : redirect_url,
               'code' : code}
    data = urllib.urlencode( params )
    req = urllib2.Request( access_token_url, data )

    # request the access_token
    response = urllib2.urlopen( req )
    access_token = json.loads( response.read( ) )
    access_token = access_token['access_token']

    # store the access_token for later use
    request.session['access_token'] = access_token

    # redirect the user to show we're done
    return HttpResponseRedirect(reverse( 'oauth_done' ) )

If we’ve got the access token, we’re all set. From now on we can make any calls to the Foursquare API that require authorisation, as long as we supply this token as a parameter to the request named ‘oauth_token’. To prove that we’re logged in, we’ll re-direct the user at the end of the callback to a ‘done’ page, which will display some user details:

def done( request ):
    # get the access_token
    access_token = request.session.get('access_token')

    # request user details from foursquare
    params = { 'oauth_token' : access_token }
    data = urllib.urlencode( params )
    url = 'https://api.foursquare.com/v2/users/self'
    full_url = url + '?' + data
    print full_url
    response = urllib2.urlopen( full_url )
    response = response.read( )
    user = json.loads( response )['response']['user']
    name = user['firstName']

    # show the page with the user's name to show they've logged in
    return render_to_response('foursq_auth/done.html', {'name':name})

And with that, we’re all done. We can fire up the django server with:

python manage.py runserver

open our browser to 127.0.0.1:8000/foursq_auth/ and go through the login process:

Login page for our app

Approving the app at Foursquare

Successfully logged in

And that’s that – A Django web app doing OAuth authentication with Foursquare. The full code is up on github if you want to take a closer look.