A Cautionary Tale of Open Data

Disclaimer – some of the ideas within this blog post are not mine. I had lunch today with a good friend and colleague and he said some stuff which made real sense. I thought I would write them up and share, I’m saying this with a disclaimer as it provides me with 2 things – covering my butt if he finds out I’ve discussed his ideas and wants to be credited (which he will be) and me being able to pass them off as someone else’s ideas if you disagree. A win-win situation, if you’re into such phrases. Edit: Jim Morton (aka @premierkissov) is the man behind the chat behind the blog.

So, a cautionary tale….

Neil* likes to dabble with creating websites, he’s not bad at it, but he’s certainly no genius. His local council, Southfordsley District Council* have just started publishing some open data. It’s not a bad site, with some interesting excel spreadsheets, the odd RSS feed and a few shape files. During a flash of inspiration, Neil decides he is going to build a website using some of this open data (because that’s what all the cool geeks are talking about) to provide a useful service to the residents of Southfordsley District Council. How inspiring and fabulous, I hear you say. What a lovely chap with a heart of gold, devoting his time and skills to do something for me, my hero.

So Neil wants to build an application that pulls together the information about the council’s leisure facilities, a map of their locations, usage statistics and a schedule of classes and swim sessions. That way, the lucky residents of Southfordsley can, with a flick of a wrist and a click of their mouse, decide whether or not they want to go swimming today at 1pm because the trend shows this is a quiet time of day at their nearest pool and it’s before the school lessons take place. Hopefully the changing room floor will be relatively pubic hair free, heaven!

So Neil sets about building this modern technological marvel. The swimming pool usage statistics are up on the website in Excel format so that is nice and easy. So are the locations of all the swimming pools. The opening times, however, are not on the site, so Neil flexes his scraper muscles and sets about collecting this data with some funky snake related programming called Python (is that related to ASP?). The swimming sessions are also not available as open data, so Neil scrapes all of this as well. This guy really knows how to party!

Lovely. The site is pulled together with a bit of code here, and a bit of code there, and before you can say “Public Data Corporation” the new “Swimming in Southfordsley” is ready for launch. Word spreads quickly about this new site and very soon he is receiving well over 10 hits per day, yes, you heard me, 10 hits! He is a local hero, there is even a little bit about him in the local paper wearing speedos, a swimming hat and holding his laptop.

But oh, those salad days didn’t last for long. Within a couple of months things had changed and poor Neil was no longer “resident of the month”. The District Council website didn’t take into account data scrapers and they merrily changed their table data willy nilly, without a second thought for “Swimming in Southfordsley”. So very quickly Neil’s website became out of date and that’s where the real trouble started. John* turned up thinking he was going to Swim Fit, but ended up in Water Babies, 25 year old Marilyn* wanted to do bums, tums and thighs and found herself with the 50+ men only session (they didn’t mind too much), and when Steven* turned up, the pool was shut. The town was in uproar, and with his head hanging in shame, Neil shut down his website, never to utter the words “open data” again.

*All names have been changed to protect identities.

So, what is the moral of this story? Open data is not straightforward and it will only be done correctly if it is done in a sustainable manner. Data needs to be taken from source, not rehashed into a spreadsheet that is out of date before you have even published it. Consideration needs to be given to what people might do with the data, it needs to be easy for them to do something with it, and we need to ensure the data stream is durable and reliable. Would you use a website if you knew it was displaying incorrect information? No, you would go somewhere else.

I don’t have the answers, but I do know that open data won’t survive if councils believe throwing a few spreadsheets at us is the way forward.

We also need to know what people are doing with the data. Someone has already mentioned in a blog post somewhere (sorry, I’ve forgotten where now) about creating a GetTheApp site to post when you have done something with data. A great idea that could show what can be done.

The work Tim Davies is doing with his http://opendatacookbook.net/ site is fabulous and a much-needed resource. But councils, please don’t make the fruits of his labour rotten because the data is a bit off.


Forehead….meet palm

Here are my thoughts after my first ever experience of an unconference – UK GovCamp 2011 at Microsoft yesterday…

Why did I want to go? Well, having chatted to quite a few campers on Twitter it seemed a great opportunity to get to meet a few of them in person. I’m a naturally enthusiastic person with a real passion to make a difference, and this seemed like a place where I might fit in. I was right, what a lovely bunch of people. All there for slightly different reasons (as can be seen by the fabulous Wordle of our introductions) but with the same drive and willingness to participate.

The first session I attended was OK, a few good discussions around why getting stuff done in Local Government is so difficult – too many meetings, no-one willing to make a decision, risk aversion, IT “getting in the way”, la-di-la-di-la. Nothing new learned, but I did manage to put my point across that IT departments don’t usually say No for the fun of it, there’s *usually* a good reason behind it. Go to them asking “I need to achieve this…” not “I want this piece of software…”.

The next session was around hyperlocalism with Will Perrin and a few other really interesting people I didn’t get chance to note down who they were. Talk About Local train up people in communities to run their own hyperlocal sites, and these are quite often the older generation. Nick Booth of podnosh made a great comment that 10 years ago you would never have imagined the people running websites would be women in their sixties, but that is now becoming more common. With the work Nick does with his Social Media Surgeries, the “older” generation are no longer being left behind, and are in fact taking the lead in using technology to help with their community work.

Lunch, *nom*nom*nom*. A bit of scheming with Carl Haggerty and Louise K *scheme*scheme*.

After lunch I helped run a session around “Doing Something with Data” and “Making Data Work for You” with Will Perrin (I think he might think I’m stalking him, I’m not, honest guv!) and quite a few others who are really heavily involved in the open data world. I shan’t try and name them all as….well….to be quite honest, my memory is shite and I just can’t remember (note to self: take more notes next time). Anyway, having chatted around the problems of open data – although publishing it can be quite tricky, it’s the getting people to do something with it that’s the hard bit, I had my forehead meet palm moment…

I’d only ever really only focussed on the geek use of open data – fabulously pretty visualisations, super-duper clever mash-ups to create a new perspective. But one of the largest groups of people that could benefit from open data are activists that need data in order to get their point across. From a simple community saying our pothole hasn’t been fixed, it’s been 6 weeks and according to your data it should have been done 4 weeks ago, to a major campaign using data to locate a rare newt colony in order to stop the development of a new motorway.  In the same way as we are getting “Data Journalists” popping up all over the place pointing, we need to enable the growth of “Data Activists”. But we need to do a number of things in order to really turn this open data concept into an actual movement:

  1. Give Tim Davies all the help you can give him with creating his open data cookbook, which is an absolutely fabulous concept that will give people ideas for what they can do with data and show them how to do it
  2. Use the same principles of the social media surgery, and get out there to train community groups to use data – Data Activism
  3. Come up with a set of a dozen or so datasets that every council should be “encouraged” to publish – not just spending data, let’s have some interesting stuff. And to publish it in the same way, so that someone can create something very easily
  4. Help the geeks meet the community activists so they can work together and fulfil their own personal goals, and do great stuff with data

Having spent a considerable amount of time and effort promoting the benefits of open data, I really don’t want to see it fade away, or be crushed by the Public Data Corporation. So I’ve been so busy trying to get data out there, I didn’t even think that approaching it from the opposite side would give me the answers. Keep Open Data Alive!!!

The next session I went along to to really learn something new, and I did. Nicky Getgood ran a session on archiving, and there were some interesting discussions around the issues of digital archiving, we saw some great old footage of a cow running around Birmingham and Andy Mabbett made a great point about the accessibility of old material and the lack of it.. I plan on doing a lot more research on this so I can take it back to work and get properly involved with the work being done with Heritage and Culture.

The final session was around the use of Communities of Practice and we eventually got on to talk about #twitternar (a great new word from Carl Haggerty) and helping LouLouK take her #lgovsm sessions to the next level. Keep your eyes peeled for interesting stuff happening with that over the next few weeks.

So, to sign off, I would like to mention a few people (because it’s good to make sure people know that they do mean something to me):

  • Karen Ramsay-Smith for the great company and the knowledge that there is someone I can rant with at work
  • Si Whitehouse for being lovely and still praising my Chocolat presentation!
  • Andy Mabbett for entertaining me, questioning my musical taste and being an all-round good guy
  • Nick Booth for smiling politely at me when I droned on about Muse, for inspiring me, and for making me feel like an old friend with hugs
  • Dan Slee – I don’t think I need to say any more, I think he would be on everyone’s list
  • Louise Kidney for great posts, honest chats and someone who needs to get more confidence and know what a good person they are
  • Will Perrin for being another inspiration
  • Dr Sue Black for being such an awesome woman that I’m ashamed to say I have never heard of until yesterday. Glad I had the guts to go up and introduce myself
  • Carl Haggerty for #twitternar and his lovely Devonshire accent

Have a medal…

It seems a little odd that in 2011 I am taking part in an “Advancing Women in the Workplace” programme. Well, you only had to be a fly on the wall at our launch event on Mon 17th Jan to see why. The first part of the launch was for our leadership team, the 18 delegates and their line managers. Considering this is a programme for women, there was a surprisingly high number of men attending the event, which if you do the maths, you can figure out there is a rather low % of women in more senior management.

Now, this kind of programme is going to sit uncomfortably with some people (including me to some extent), as it will be seen as a form of positive discrimination. So I want to ponder some thoughts as to why it is much harder for women to advance in the workplace, and does it actually have anything to do with gender? These are my initial thoughts, and I would love to hear any comments people have. I’m sure they will change over the next 12 months while I do this programme, but I’d like to see how what I think now compares with what I think when the programme is over.

  1. Self-promotion – men seem to be a lot better than women at promoting themselves. “Look, I’ve done the washing up”, “I emptied the dishwasher”, “I did the ironing”. My standard response: “Have a f**king medal”. I don’t spend my time pointing out the things I have achieved, I just do them. Does that carry over into the workplace? Do successful people, regardless of gender, make more of a fuss about when they have done things well?
  2. Jobs for the boys – is the old school mentality still there? I think until we can break the cycle of having the overwhelming majority of male senior managers, this may always be the case. I don’t know enough about this yet, so if anyone can enlighten me on this, please do.
  3. Part-timers – this is where I have my strongest feelings on why women aren’t advancing as well as men. There is still a wide-spread belief that in order to do a senior management job, you have to work an excessive number of hours, not even 37 will do.  This doesn’t fit well with that elusive “work-life” balance, and as the job of childcare, or care of relatives, is usually the responsibility of the woman, we are inherently disadvantaged. I am extremely passionate about the fact that the most important indicator of a good manager/director/exec is not how many hours they put in, but what they achieve in that time. Until we can change the culture of working long hours, we’re screwed. Our examples of women in senior management show that – our assistant chief exec is a woman, but it is her husband that took responsibility for the childcare. It’s great that she has got to the position she’s in, but she didn’t do it with the work-life balance I want in my life. One of our service heads went part time to look after his children when they were growing up. He openly admits that he believes this stunted his career. The inability to work part-time is the biggest killer for advancing a career, regardless of whether you are a man or a woman. As a higher percentage of women need to work part-time, the impact is greater for women, but the problem is not caused by gender.
  4. Big Society & NHS Reform – following on from point 3, the changes to the NHS, the Big Society idea, all of these are going to put added pressure for people to use some of their time to look after their relatives or people in their local community. I might get my kids into school and achieve a nice senior position, but then (please don’t let this happen) one of my parents might get dementia. The NHS aren’t going to be in a position to provide the support I need in order to carry on in my job and care for my parents. So the pressure to work part-time will increase, and again, this will fall to woman.

So, those are my first thoughts. No doubt they will change, and I hope they do otherwise I will have learned nothing.