Last week I was asked by Stuart Allen to give a talk to the students in his first year “Problem Solving with Python” course on the topic of django, the python web framework. As I’m not lecturing at the moment I like to do these odd talks or cover the odd lecture just to keep up the skill set for when I start doing it again properly, so I agreed. Plus, I think django is ace and it’s always nice to try and brainwash students into using educate students about your favourite technology 😃
The talk was actually really well timed as the students are currently working on a coursework in another module (“Web Applications”) where they are required to develop a blog application using MySQL and PHP. This meant I was able to show off just how simple it would be to do it in django instead - so I gave a quick introductory talk then did some live coding and created a pretty functional blog application in about twenty five minutes. I think it annoyed quite a few of the students to see the difference in complexity between hand coding everything in PHP and having django take care of most of the boilerplate code. It may even have been a bit evil of me, but hey, that’s what I’m here for.
The slides for the talk are attached to this post (Introduction to Django Talk), although they’re of the “you need to be there to get the details” style of slide design, so they may not be that useful. Thanks to Mike Cantelon for his django keynote theme, and the lovely flickr users who uploaded photos with a CC license so I could reuse them.
Yesterday I attended the “For the Social Good” hackday, organised by BoxUK and held at the Students Union at Cardiff University. As you may have gathered from the title, it was a hackday themed around creating apps that had some benefit to society. The event ran from 10am to 10pm, so it was a fairly short hack event compared to some, which had a big influence on what could be done during the time available. In total, given the time needed for introductions and getting started in the morning, then for presentations and judging at the end of the day there was only actually about nine and a half hours of coding time, so it was quite a tough day. I went along with Matt Williams and Mark Greenwood as our team and over the course of the day we managed to put together a fairly functional web app.
We took what we learnt from hosting and participating in the Foursquare hackathon and used that to guide what we did at this hackday. At the last event we had been slow to get going, took too long to form a team and decide upon an idea. We had aimed too high to begin with, and then also let feature creep affect what we wanted to build. Finally, our teams were far too big, meaning it was hard to divide labour up so that everyone was participating and had something to do. We finished with a working app at the end of the event, but we were determined not to make the same mistakes this time so we could get a more successful outcome. (Not that FourCrawl wasn’t a successful outcome of course, but more on that in the next couple of weeks.)
Going in to this hackday we solved the team size problem naturally, as there were only three of us able to attend which made forming a small team compulsory! To try and get a good start on the day of the event we met together on the Wednesday before to have a think about ideas and a bit of a brainstorming session. While we had a lot of ideas, most of them were too complex for a short day long hack event, so unfortunately we didn’t really come up with anything. We did have a vague idea of doing something related to cold weather though, to make whatever ‘it’ was topical, given that we’re heading into winter now. Fortunately at least one of us (not me) had obviously been thinking about it further after the meeting and on Saturday evening Matt sent round a message on G+ that he’d found a good data set we could use for an app that would be both easy to code within the time limit, and would hit the brief for the hackday pretty well. He’d found a dataset on data.gov.uk related to road accidents that included information on weather and road surface conditions that we could use to show people where to avoid in icy weather, or to give them an idea of previous incidents in their area. With an idea starting to form we went into Sunday in a pretty good state, and ready to start coding very quickly.
The day started well, everyone arrived pretty much on time, which made for a quick start. Our app idea solidified easily, and early on we realised that while we had historical data to display it would also be nice to have some current updates, so we added a social element to the idea, by scraping twitter for a certain hashtag so that people could report problems with icy roads in their area, along the lines of the ‘#uksnow’ hashtag that has been used frequently the last couple of winters. We were able to jump in quickly to start getting things built. Matt was in charge of getting the data into a database and building an api to query to retrieve it, using django to get a backend up and running quickly. I was in charge of building a front end map to display the data on, primarily working with HTML and javascript (and the Google Maps API obviously) and then Mark was in charge of getting the twitter scraping and postcode geo-locating working using python (so it could be slotted into the django backend), accessing the twitter API through tweepy.
Things went really well with the implementation, I spent a while cannibalising elements from previous sites to get a frontend up and running quickly (and also thank you bootstrap!) and by lunch we were pretty close to starting to consolidate things together. We set a target of starting to deploy by 6, so that by the time dinner was served Matt was starting to battle his server and its stubborn refusal to allow easy deployment of django apps. Things went so smoothly in fact that we were all finished and ready with almost an hour to spare, meaning we were the only team able to slope off for a self-congratulatory pint before the presentations and judging. We headed to the Taf (a place I hadn’t been in at least five years) for a swift one, then went back to the hackday for the final presentations.
Our presentation was first, and Matt demoed the app well while I played around on the laptop next to him. After us the rest of the teams went, and there were some really good apps shown off. My particular favourite was the ‘occupy where’ app which allowed you to search for an ‘occupy’ camp or demo near a location, then presented information pulled in from multiple sources about that particular ‘occupy’ movement. A nice idea, well executed. Following the presentations the judges spent some time talking, then announced the winners. The surprise of the day was that they’d decided our app was the best and that we’d won! I was pretty shocked, but it was a very nice surprise. A group of first years from COMSC were runners up, and then a third year COMSC student (Tom Hancocks) won the best individual developer prize, so COMSC actually walked away with all the prizes! After a quick tear down of the venue it was time to head to the Taf for some celebratory drinks, a satisfying end to a really great day.
UPDATE 25/11/11:
There some excellent other write ups by various other people involved in the event below. I’ll add to this list as and when I find any more:
**Note: I wrote this post just after the hackathon, but then forgot to actually publish it! **
There’s a fair amount I learnt from taking part in and helping to organise the foursquare hackathon here at the university that probably deserves its own post. I’ll split it into two parts, the taking part and the organisation.
Organising a hack event
There were a couple of things I learnt from helping to organise the event that perhaps weren’t clear to begin with:
Expect dropouts. A lot of them. We ended up with about 50% of the people that said they would attend actually attending. Which is fine because we hadn’t planned on providing catering etc. anyway, but if we had, we may have overspent massively.
Get someone in to run the day that doesn’t actually want to code. All three of us organising the event really wanted to get involved and create stuff, so there was no-one left to organise the other essential things like food. Too often people would talk about ordering food, or going out for dinner and then get sucked back into programming and it would get forgotten about. By the end of the weekend we’d all eaten really badly, often very late at night. Having someone there to do things like order pizza or make coffee runs would probably have helped things go a lot smoother.
If you’re going to be communicating with the world at large using twitter or something similar, make sure you communicate with each other within the organising team about posting updates. Often two people would try and reply to a query at the same time, which made us look a bit silly. Not a major issue, but it might help give a more professional feel, event if you’re total amateurs.
Participating in a hack event
We also learnt a lot about participating in a hack event. My top tips for a successful project would be:
Do your research. If you can, go into the event with an idea already, just in case no-one else has any. It’ll help you get started quicker.
Keep it small. Don’t over reach. 24/48 hours is not a long time to code something, and by making it too complicated you’ll be disappointed with the outcome. Simple is best. Add extra features later if you have time.
Small, agile teams. These projects have to be small because of the time constraints, so if teams get too big there won’t be enough for everyone to do. People will end up feeling useless or left out, which is never good. I would say a maximum of 4 people per team.
If you want to learn something new, a hack event can be a great place to be forced to up your skills in a particular area very quickly. It can also be a great place to learn new skills and tools from others. However, this may lead to the end result not being ideal. If you can live with that, great, you’ll walk away happy. I personally upped my javascript knowledge over the weekend from approximately 0 to something approaching a passable working knowledge, which was great.
Know your tools. This is pretty much common sense, but if you’re not after learning something new and you just want a successful app/outcome, pick and use the tools you know really well. We went with django on the back end because myself and Matt know it pretty well, and we were able to get that part of the site running really really quickly. Had we gone with something else it may not have worked so well.
Get coding. Screw the design, you can worry about that later. Start coding early, and code fast.
So thats the things I learnt from the weekend. Hopefully we’ll be able to use this knowledge again in the future, both organising and participating in future events.
UPDATE [01 August 2014]: Even more breaking! More recent version of the script here
UPDATE 31/01/2011 - new version of the downloader script is here
A couple of projects I’m working on at the moment need the ability to add foursquare venues to a map. This is pretty straightforward, but what is also needed is the ability to distinguish between the categories of the venues. To do this, I’m adding custom markers to a google map at a venue location, with the marker having an icon representing the venue category.
Foursquare already have lots of category icons ready made, and they supply the urls in the /venues/category endpoint, which is handy. It seems from official responses on the forum that they don’t mind you downloading these icons for use in your apps, and they also supply the icons in several sizes, so hacking together a script to download all the available icons is pretty easy. The following python code will do just that - grab the latest category hierarchy from Foursquare, process it to extract all the category id’s and icon locations, then download each icon in all available sizes:
import urllib2 import urllib import json
# supply oauth token as parameter params ={ 'oauth_token':'VALID_FOURSQUARE_OAUTH_TOKEN' }
api_base_url ='https://api.foursquare.com/v2'
icons ={}
# function to extract category info from json and # process any sub-categories defprocess_category( category_json ): icons[category_json['id']]= category_json['icon']
if'categories'in category_json: [ process_category( c )for c in category_json['categories']]
# extract all the categories for category in categories_json['categories']: process_category( category )
# download all the icons for foursq_id, icon in icons.iteritems(): sizes =['_32','_64','_256',''] for size in sizes: try: url = icon.replace('.png','%s.png'% size ) u = urllib2.urlopen( url ) file_name ='%s%s.png'%( foursq_id, size ) localFile =open( file_name,'w') localFile.write( u.read()) localFile.close() except Exception: pass
After running this script, we have a local copy of all the category icons to use in our application downloaded. In my case, I need to be able to differentiate between venues within the categories, so it would be nice to have different coloured icons for my different classes of venue. Luckily this is also pretty simple. You’ll need ImageMagick installed, but if it is, the following code will go through all the greyscale icons downloaded with the previous script and convert them to whichever colours you like:
# # list all the files in the current directory current_dir = os.getcwd() files = os.listdir( current_dir )
# # run through the file list for fname in files: # for all the png's if'.png'in fname: # get the file name stripped_name = fname[:fname.rfind('.')] # convert to a new colour and save for colour, magickcolour in colours.iteritems(): colour_name ='%s_%s.png'%( stripped_name, colour) os.system('convert %s +level-colors %s, %s' %( fname, magickcolour, colour_name ))
The colour names come from this list of imagemagick colours, you can obviously pick whichever colours you want. If you run this script in the directory with the downloaded Foursquare category icons, you’ll end up with a multitude of differently sized, differently coloured icons.
And that’s that. Later I’ll post about how to use these as custom markers in a google map.
UPDATE 3/10/11:
As this example stores icons by category id, it actually duplicates some of the icons, which is un-necessary. This version stores the icons by icon name, so saving the downloading and processing of around 100 duplicate icons. It also splits the different sizes into different directories for better file management.
UPDATE 5/10/11:
There’s a change in the category json coming which will break this code. Use “v=20111002” as an extra parameter in the code that calls the Foursquare API and it’ll keep working.
Wow. I thought Saturday was busy, but I was wrong. Saturday was a cake walk compared to the crazy nightmare that was Sunday. The challenge of trying to get something (anything) working before the app submission deadline was incredible. Coding was fast and intense, and required a lot of good communication and fast thinking. Unfortunately we hit a few roadblocks that slowed us down (2 hours debugging problems with django’s static file serving really didn’t help, neither did jQuery mobile being an absolute pain in the rear end), but by the end of the day we had a nice looking site with some actual working functionality. Nowhere near the level of functionality that we’d hoped for, but it’s a start. All three of us that were working on the app towards the end are aiming to carry on the development, so hopefully I’ll have another project to post about up here as the months go on.
A number of people didn’t make it to the Sunday, which is a shame, but the core teams of both projects were in early and working hard, and by the end of the day both had been declared ‘working’ enough to submit. Maybe they’ll do ok against the competition, but we’ll see. A couple more hours on our pub crawl app could really have helped as there’s some key functionality missing, but there just wasn’t time to get it working. Overall though, the event was a success. We managed to pull in people from outside our own hyper-local school development team, we all managed to learn some new things and teach some things to others, and I think most people genuinely had fun. It’s definitely an experience I’m looking forward to repeating, both as a hackday organiser and as just a participant. BOX UK have a ‘For the Social Good’ hackday coming up in November that I’ve already registered for, and I’ll be keeping my eyes open for others in the future, and for opportunities to organise more.
I think I learnt a few things about organising and participating in these sorts of events, so I’ll be putting up a post in the next few days to share the knowledge.
I was fully intending to try and live-blog the progress of the foursquare hackathon today, even going as far as to install a live blog plugin, but things have been going very quickly and we’ve been hitting the coding hard so I just haven’t had time!
We’ve managed to get a decent crowd in, a roughly 50:50 split of people we already knew from the School and other people from elsewhere. At the peak today we had fifteen people in, but at this stage we’re down to ten - but those who have left are coming back tomorrow! There’s an interesting mix of skills and knowledge levels and a number of people have come who are just eager to see how other people do things and to learn more about foursquare or coding in general. Luckily everyone is talkative and keen to get on and create something cool.
We started this morning at about 10am and managed to come up with a few ideas in a brief brainstorming session, eventually splitting ourselves into groups working on two ideas. The first idea is being driven forward by Vlad and is quite related to the work we’re doing on Recognition - creating a system to pull in information on a venue or location and create a ‘digest’ of what’s relevant about the place. This could come in handy later on for our research. The second idea is a more ‘fun’ idea - building a system for creating, managing and participating in pub crawls. There’s a couple of parts to this: a management website that allows users to search foursquare venues and add them to a crawl, a backend that manages the storage and retrieval of users and crawls and an application front end that allows you to participate in the crawl itself. At this stage early on day one we’ve got most of the backend built, and the management web app is coming along well. Tomorrow we’re going to start the android app.
Along with Matthew Williams and Chris Gwilliams I’m helping to organise the Cardiff edition of the Foursquare global hackathon this weekend. I’m hoping it’ll be a pretty good event. Signup’s are looking pretty positive and I’m hopeful that we’ll get some decent apps being created.
The organising is actually quite fun, but it is remarkable how much there is to do and remember. Thankfully the School of Computer Science & Informatics are being very supportive and allowing us to host it there, which makes things easier. I’ll be posting here throughout the event (and afterwards).
Moving on from my fun with getting django + twitter + oauth to play nicely, it’s now time to start writing a web app that uses foursquare as the authentication/login method. As part of this, I need a basic django framework for doing authentication with foursquare. This may be useful to someone else at some point (including me when I forget in three months time how I did things), so I’ve written up the details and added the code to github.
Firstly, you need to register an application with Foursquare to get your Client ID and Client Secret. You can enter any name and website you want, but the callback url needs to be our local development machine (assuming that’s where you’re doing the development), and to be the callback url that we’re going to specify in a minute:
The more observant will notice that the callback url is currently http://127.0.0.1/foursq_auth/callback/, but that the django web server actually runs on port 8000, so the url really needs to be http://127.0.0.1:8000/foursq_auth/callback/. Unfortunately the form validation doesn’t allow you to use a ‘:’ in your callback url. However, once the consumer is registered you can edit the callback url, and the form for that doesn’t validate the url (or does but doesn’t moan about the colon). Eventually, you are looking for the consumer to be registered with a callback url like this:
Once that is registered, we can get on with writing the code. The basic premise is pretty simple, we need just five urls in our django app, five views to go with them, and two basic html pages, one for login, and one to show we are logged in. Assuming you have a django project started (for reference the one I’m using here is called ‘Dj4sq’) we can start by creating our foursquare app:
django-admin.py startapp foursq_auth
We then want to edit urls.py for our main django project to allow the new app to handle its own views:
and create and edit a file urls.py in the foursq_auth app with the five views we want:
urlpatterns = patterns('foursq_auth.views', # main page redirects to start or login url(r'^$', view=main, name='main'), # receive OAuth token from 4sq url(r'^callback/$', view=callback, name='oauth_return'), # logout from the app url(r'^logout/$', view=unauth, name='oauth_unauth'), # authenticate with 4sq using OAuth url(r'^auth/$', view=auth, name='oauth_auth'), # main page once logged in url(r'^done/$', view=done, name='oauth_done'), )
We then need our two basic html pages login.html and done.html:
The second, ‘auth’, is the first step in our authentication process, requesting an authorisation code from foursquare and redirecting the user to the page to authorise our app:
defauth( request ): # build the url to request params ={'client_id': CLIENT_ID, 'response_type':'code', 'redirect_uri': redirect_url } data = urllib.urlencode( params ) # redirect the user to the url to confirm access for the app return HttpResponseRedirect('%s?%s'%(request_token_url, data))
If the user accepts our app, they’ll be re-directed to our callback url, and an authorisation code will be passed back as one of the parameters of the url, so the next view, ‘callback’ needs to deal with this. It then needs to post a request to foursquare with the authorisation code in order to receive the access_token for the user:
# get the code returned from foursquare code = request.GET.get('code')
# build the url to request the access_token params ={'client_id': CLIENT_ID, 'client_secret': CLIENT_SECRET, 'grant_type':'authorization_code', 'redirect_uri': redirect_url, 'code': code} data = urllib.urlencode( params ) req = urllib2.Request( access_token_url, data )
# store the access_token for later use request.session['access_token']= access_token
# redirect the user to show we're done return HttpResponseRedirect(reverse('oauth_done'))
If we’ve got the access token, we’re all set. From now on we can make any calls to the Foursquare API that require authorisation, as long as we supply this token as a parameter to the request named ‘oauth_token’. To prove that we’re logged in, we’ll re-direct the user at the end of the callback to a ‘done’ page, which will display some user details:
defdone( request ): # get the access_token access_token = request.session.get('access_token')
# request user details from foursquare params ={'oauth_token': access_token } data = urllib.urlencode( params ) url ='https://api.foursquare.com/v2/users/self' full_url = url +'?'+ data print full_url response = urllib2.urlopen( full_url ) response = response.read() user = json.loads( response )['response']['user'] name = user['firstName']
# show the page with the user's name to show they've logged in return render_to_response('foursq_auth/done.html',{'name':name})
And with that, we’re all done. We can fire up the django server with:
python manage.py runserver
open our browser to 127.0.0.1:8000/foursq_auth/ and go through the login process:
And that’s that - A Django web app doing OAuth authentication with Foursquare. The full code is up on github if you want to take a closer look.
It’s the last full day at the festival and we wake up even later than the day before. It’s so late we decide to skip breakfast, and as it’s time for lunch we’ll head out for curry. A quick curry and rice and we’re up and running for the day once more. The shows today don’t start until later on, so we have a bit of a walk around, grab a coffee and cake, then once more split, as Lisa decides to go back to the flat briefly while I head to pub. I sit out of the rain doing the crossword, have a pint and await Lisa’s return. She arrives, we finish the crossword, and head off to the show. The first show of the day is Ruby Wax in her one woman play (that’s got two women in it): Losing It. It’s a fantastic and honest play about mental illness, really well done, with some laughs and a lot of truth.
Following the show we take a walk over to near Arthur’s seat, then back towards town for the last show of our festival. On the way we find a really ace pub that I think will be my local when I make it to Edinburgh full time. We have a quick pint, then go to the Queens Hall for Henry Rollins. The man is amazing, he gets up on stage and just talks for the evening, and everything he says is amusing or thoughtful and brilliant. He’s basically just telling us what he’s done for the last year, with some old road stories thrown in for good measure, but he’s such a great speaker its one of the highlights of the week. I think most academics I’ve seen talk could learn hugely from the passion and thought he puts into telling stories.
Post show we head back to the pub for a couple of pints, then grab a chinese takeaway and head back to the flat. With that, we’re done. Wasted, destroyed. Great week.
Shows: 2
Pints: A few
Chinese Food: some
Aging rock stars: one
Wednesday we wake up later than intended, so miss the planned start to the day at the Shakespeare breakfast. In all, it’s a very slow start to the day, the pace of the previous few days is beginning to get to us. We head off for a walk and a bagel to wake up, and I get butter all over my face from a very lovely, but very buttery and bacon filled bagel. After soaking up a bit of sunshine in the cafe we head down to town for our first show, an adaptation of Secret Window, Secret Garden. The venue is at C soco, which is the massive hole in the ground in the Old Town where the fire was a few years back, the one that took out the original Gilded Balloon and the School of Informatics’ AI department. The venue itself is in one of the buildings left standing on the site, which may well have been part of the University from looking at it, and the specific venue we’re in is right at the top. I estimate we climbed about 4000ft to reach it, but I may be wrong.
The play itself is nicely done, they’ve cut the story down to the minimum of necessary elements to cover everything in the plot while still bringing the play in at an hour, and the acting is pretty solid, if not stellar. The lead character is by far the best actor of the lot, doing a very good job at playing the role of the author losing his mind. They stick to the plot of the book too, rather than the film, which is pleasing. The guy playing John Shooter troubles me a little, and it’s not until afterwards that I realise it’s because he looks, sounds, and acts very much like someone at COMSC.
We’ve got a gap in the afternoon, so we head down to one of the ‘attractions’ of Edinburgh we haven’t visited yet, the Botanic Gardens. It’s a fair enough walk, mostly downhill. We cause a problem in the cafe at the east gate by attempting to buy things with a twenty pound note, the horror. They have nowhere near enough change, but there’s a solution to the problem if they let me off the odd 35p and I just pay them £5. They agree, so I effectively steal 35p of goods off a registered charity. In your face, plants.
It’s a nice place for a wander round, as lovely gardens tend to be. We decide to put up the cash to go into the hothouses, being as we’ve walked all the way down there. Surprisingly, they’re hot, and full of plants. I can’t get the Jurassic Park theme out of my head as I walk around, the whole place is reminiscent of the science labs in the sequels that have been taken over by plants, there’s a nice bit of ageing decay to them. Very few dinosaurs though, luckily.
After the botanic gardens we head to a pub down the road for a pretty decent dinner and a couple of pints, then make long trek back up the hill and into town. Lisa wants to head back to the flat, but I can’t be bothered, so I head down to the Brewdog bar on Cowgate to wait for her. I’m quite a fan of Brewdog beer, and it’s a nice bar (if a bit small), serving some very lovely booze. While I’m stood at the bar reading the paper it begins to absolutely piss it down with rain, so by the time Lisa gets back into town she is soaked through once more. As ever, the theory ‘go to the pub, not home’ has paid off.
We head up to the Assembly Hall to meet Lisa’s colleague and his wife for a pint before Sarah Millican. We have a nice chat for half an hour, then get into the longest queue yet for a show. We end up with seats up on the balcony, but it’s a fine venue that seems to hold a lot of people without feeling massive, so the view is pretty good and you still feel pretty close to the stage. Sarah Millican is good, but the show seems like a collection of one liners and jokes rather than a coherent whole. Despite that it’s hilariously funny, and a good time is had by both of us. I steal two badges at the end of the show, making it both a registered charity and a comedian that I’ve effectively cheated in one day.
Following the show we head down to the Conference Centre for an Amnesty gig. I thought the queue for the Millican show was big, but the queue at the EICC was huge, snaking all the way round to the back of the building and out into the street. I have a minor altercation with some old ladies in the queue who are queue jumping, and refuse to move when I inform them of the fact. Another chap also tries to let them know, but they ignore him too. So we talk loudly about the older generation just thinking they can do what they want, and how old people have no manners anymore and don’t understand about how being British means we treat each other decently and are polite and so on. I’m half convinced we quite embarrassed the one old lady, but her friend was made of sterner stuff. The old ladies have annoyed me massively, but it’s wearing off by the time we get into the venue, until I discover that bottles of beer at the bar are £3.80 each, which almost makes things even worse. Luckily the show is laugh balls funny. It’s basically ten minutes sets from a whole mix of comedians that we wouldn’t have otherwise seen, with a large number of laughs coming from making the signers at the side of the stage make inappropriate gestures. It’s a great gig, with performances from Mark Watson, Ed Byrne, Russell Kane, Jenny Eclair, David O’Doherty, Holly Walsh and Roisin Conaty. Once more we both enjoy it, laugh too much, and leave happy. We head home satisfied for the evening.
Shows: 3
Pints: God knows
Queues: Massive
Old ladies: Three (irritating and rude)
