Making data make sense


This one is a bit of a long read, not my usual style – it’s actually the tidied-up speaker notes from the last few talks I’ve given about data and open data. The intention was to publish it as an essay. Then I remembered how much I hate writing essays, and so here it is as a blog post instead. It’s a bit about open data, a bit about operational intelligence, and a lot about cultural challenges in the public sector. I’ve spoken using this same material, or subsets of it, at about five different events in the last year and a half. Sometimes it’s called ‘making data make sense’ and sometimes the title is ‘making data useful’. Either way – it’s about making sure that the work we do with our information and evidence actually has a point.

Useful data

The subtitle for the most recent talk was, “What do we know?” There are a couple of ways I interpret this.

  1. The first is, how well do we know our business, our performance, our efficiency and effectiveness? How well do we track our budgets and our people and our other resources using the data we hold? How much do we know about what we even know, what data we have available right now to support our decisions?
  2. The second emphasis is on our skills base, the things we know how to do or the things we know we want to do that we think are technically achievable, and the way we enable them to happen (or indeed the way we prevent them from happening, sometimes without even meaning to). What do we know about what is already happening in the organisation, the people who are doing things we’d like to see more of, making best use of new tools and networks?

“Bad data leads to bad decisions.”

I heard it said, a few weeks ago, that “bad data leads to bad decisions”. I don’t agree. I can see why this is a tidy soundbite, it sounds like it makes perfect sense, but in fact that just isn’t how decisions actually work. If I expand this phrase and think about each step along the road, starting from bad data, what I get is this:

  1. Bad data leads to bad information.
  2. Bad information leads to badly-informed decisions
  3. Badly informed decisions might be bad decisions. Maybe.

Yes, some decisions are bad. But the data has way less to do with that than you’d think. We don’t make decisions based solely on data. We’re not machines, we’re human beings; messy and complex creatures with the most amazing capacity for self-deceit. We totally believe that we are rational beings who examine all the evidence before making hard choices, but the truth is that we will have narrowed down the options quite brutally based on ideology, emotion, anecdote, memory and context before we ever get to making a choice based on the numbers. We prioritise these other things – aware or not – to support a decision that makes sense to us given the context; and we prioritise the softer side because as humans we are influenced far more by stories than by datasets. We relate better to anecdotes and case studies because these are about tangible people, not abstract units of citizenhood, and people matter to us.

That doesn’t mean that data has no place in decision making. It’s just that, with all these other things going on, when we feed data into a decision …

  • it needs to make sense,
  • it needs to be useful – fulfilling a purpose/answering a question
  • and it needs to be relevant.

I’ve worked in local government, at Devon County Council, for about fifteen years now, and my role has morphed and flexed according the needs of my team, my line manager and my lead officer. One thing that has stayed constant is that it has always been about data. At the moment my role takes in data sharing, and open data; it covers how we use data in decision making, and it covers how we use data to communicate. Apart from my council job, I’m also the cofounder of ODI Devon – which is more open data, lots more playing and lots of tinkering to see what we can do with data. Sometimes it’s hard to see where the lines are drawn as any work I do for ODI Devon benefits the council in some way, and people I meet in my council role are often connected to the ODI Devon and ODI HQ networks. The topics are the same, and the goals overlap significantly. The real differences are in the structures I work with.

“with my other hat on …”

In local government meetings I hear people talk about wearing different hats. It sounds like some kind of quick change vaudeville act, or the impressionists on family prime time TV when I was a child. But in real life it’s not as simple, or as convenient, as changing hats. I have multiple roles and responsibilities, but I’m always the same person with the same ethos, the same skill set, and the same network. I cannot be someone else, believe something different, discount my capabilities or ignore sections of my networked knowledge just because I am in a meeting with a different hat on. So I don’t change hats. I wear one hat, all the time, and my hat and I get out a lot and stand in different places and see what the view is like from there.

From the time I spent on an Accelerator Launchpad programme, and the subsequent startup development I’ve done, the view is about purpose but also about fun, play, discovery, experimentation. The purpose is something that MUST exist but crucially, is constantly questioned – why are we doing this? Why is this the product we want to make? Who wants it? Who will buy it? What problem will it solve for the people we want to reach? On the other hand, in the public sector, the viewpoint is heavy on compliance, governance, delivery, process. The purpose is not questioned. We are here to deliver services, that’s what we are for, these are services that people need and we are here to provide them, enough said.

This raises an important learning point both for public sector people who have been outside the walls, and for those who have to manage them when they come back inside; to the traditional cultures and hierarchies in our organisations, if you’re looking at tackling open data or data management, if you want to innovate and iterate and be more agile … some of the most useful people in your organisation might look like they’re trying to have fun instead of do work. We need to spread the word that it’s possible to be doing both at the same time – and everybody wins.

The whole open data topic is still a bit scary and new and no-one’s quite sure how to do it. The popular metaphor of the moment is that it’s like teenage sex; everyone thinks they should be doing it, everyone says they’re doing it a lot, but nobody is really sure what they’re supposed to be doing or how often. And there is a huge amount of hype around it too.

IMG_0501You’ve probably seen the Gartner Hype Cycle before; here it is co-opted to show the trend around open data. We’ve seen the rising expectations – it’s going to change the way we govern, it’s going to change the way we spend in the public sector, it’s going to create efficiencies, it’s going to put billions of pounds into the economy … it’s going to fix all the problems, all the trees will bloom again and beer will be a penny a pint and so on; if we get this right everything will be wonderful. I think we reached peak hype a while ago, where we realised that it’s not that simple. This shiny new idea can’t fix everything and we still have a lot of hurdles to navigate and actually it’s starting to look a lot like hard work – and worse, work that we aren’t totally equipped to master.

When that happens, expectations slide back down until we reach the trough of disillusionment, and I believe that’s where a lot of councils are now … we can see we haven’t got it right and we haven’t got all the equipment or skills, and we’re not sure what’s next. And because we are under-resourced and a bit fearful we find good reasons not to do this, or to do too much, and one of those reasons is always “nobody really wants this” or “does anybody use it? is there a business case?’

The best treatment of this I have seen yet is this short series of ‘data knot’ poems by Jeni Tennison, Technical Director of the Open Data Institute. Here’s a sample;

OWNER: I do not know what data you want

So I do not know what to give you.

USER: I do not know what data you have

So I do not know what to ask you for.

The whole blog post is here, and I recommend you take a look. It’s spot on.

The good news is, we are starting to move into a more pragmatic place, and we can see what the plateau of productivity might look like. We are beginning to decide what we want to do with open data and how we’re going to manage it. It’s a good place to be in, because it’s making us ask the right questions at last.

IMG_0602Devon County Council is one of the Cabinet Office Open Data Exemplar group of councils; obviously this makes us proud, but we are also aware that it’s a specific set of attributes that gained us a place there. All the councils in the group are taking different approaches here; for instance some have found the resources to invest in really wonderful systems, bought in or built internally – really really well put together datastores and open data portals like the Hampshire Hub, Leeds Data Mill or the Trafford Innovation Lab. These places can clearly show that they have got their data in order, in the right place and the right system, and can get it out or share it in all the right ways.

On the other side you have Devon; we haven’t put significant resources into any one project or product but we are constantly trying out small things to see if they’re useful and whether they can be adopted on a bigger scale somewhere.  The attributes that got us a place in that group are our attitude to learning and to change, our willingness to get out of the building and talk to people, and our willingness to try new things quickly and cheaply to see if they will be useful or not. So we don’t have all the answers, or some kind of winning formula, but we do have a fairly comprehensive list of mistakes we made and things we learned.

IMG_0646

This graphic shows our open data journey, from early work to sort out a public-facing Facts & Figures web page, through to the Open Data Champion status. It shows how we started with some small scale hack events, supported by Plymouth University; we got some expert training from the Open Data Institute for a group of twenty staff; we experimented with publishing datasets to a Devon County Council GitHub repository; we started an Open Data Forum. While all this was happening I was spending half of my time in London on the Accelerator programme, and setting up the ODI Node when I finished that.  Bits and pieces and small projects under the radar, rather than big corporate programmes and contracts.

But even small projects under the radar need a business case, even if there’s no cash involved. There has to be a purpose. With the new tools we learned, we started to treat these as agile user stories. A user story simply but clearly identifies a user with a need, a purpose and an outcome.

IMG_0509

Looking at the problems we thought we wanted to solve, framed as user stories, was very helpful. It helped us to establish a number of potential users and purposes, based on internal assumptions, that we could test by then talking to people from those user groups to see if we had it roughly right. But it was when we turned that focus on ourselves, as local government officers, that we saw something even more useful. We realised that inside the public sector we focus on transparency as a measure, and that’s really not helpful.

IMG_0505

What is our motivation when we look at trying new things? Far too often it seems like it’s “stay out of trouble, look busy, do the bare minimum”. Business-as-usual takes priority because people are depending on us and we can’t afford to break the process because of a wild idea – the stakes are too high. So it turns publishing open data into a compliance thing, something we ought to do but not as important as helping actual people. We’re all very busy, and resources are stretched, and everybody agrees that of course open data is the right thing to do and transparency is a jolly lovely thing, but … lovely things don’t score highly on an options appraisal or an impact assessment.

Puppies and sunshine

puppy

What’s the business case for transparency? –  it’s just one of those things, something you should just do just because. Why wouldn’t you? If you can share you should. Being against transparency is like saying that you hate puppies and sunshine.  Of course some people do hate puppies and sunshine, and indeed transparency, but it’s not a thing you want to go around proclaiming. People would think you had a personality disorder, or some dark and terrible secret, or both. So, sadly, transparency for its own sake is not a very effective motivator in the public sector. The important thing though, is that if it’s done right transparency is an outcome – a natural output of a system that manages its data well for the benefit of all concerned; a behaviour that people display because they feel secure and know they have their data under control and in good order.

But if transparency on its own is not a driver, what is? I have two candidates. Community support is important because we need to share what we know if we want other people to take on the services we can’t afford to keep. We have resources, and information is one of them, that will help communities make decisions about funding, location and service arrangements. If we want to devolve and divest this is a thing we need to do more and better. But it’s the second one, intelligence, that I want to concentrate on.

Organisational intelligence is critical, both the most selfish and the most effective motivator. We keep failing to notice that the first customer for our data is us. If our data is clean and structured and current enough to be published, that makes it oh so much easier for us and our colleagues and our partners to find it and use it too. Not to mention making it easier to answer questions through FOI, queries from MPs, drawing up impact assessments, putting together options appraisals, creating area and neighbourhood plans, performance and process reporting … I could go on and on and on but you probably get the idea. Ask any data analyst, any management information person, what proportion of their day is spent finding data, chasing data, cleaning data and structuring it to be used for a specific report or calculation. If they reply with anything less than 80% I want to know their secret.

Sadly, even when we understand the importance of intelligence, we still have a lot of work to do before we get things just right. Remember when I mentioned compliance? Well, the public sector knows how to do compliance, oh yes, and we’ve had decades to perfect our response. As a rule, open data and management information based on a compliance mindset looks like this:

IMG_0508

… said no one ever …

 

This, the phrase above, this is what we still think people want to know about the council’s activity. We have had so many years of reporting against arbitrary targets and strategic plans, while we try to do the actual work we know needs doing, that when we hear ‘performance’ we think KPIs and BVPIs and we reach for the spreadsheets. We might just keep putting out the same things we always have, with no one looking at them or using them, but it’s what we know, it’s what we have data for, and it looks like transparency from a certain angle if you sort of close your eyes a bit and put your head on one side.

Where this becomes a real problem, is when we start to spend a lot of money or apply a lot of resource to presenting unhelpful or irrelevant data in a pleasingly-arranged display, usually a dashboard. Now, I like dashboards as much as the next man – probably more, in fact – and dashboards are a fantastic tool for transparency and intelligence if they are done right. If you have your data in a state and a system where it can be shared, then building a visually effective presentation layer is much easier; and councils that have mastered this step tend to have strong performance management; genuine self awareness of how the organisation is doing, how it is perceived, and where the future challenges are.  And it has to be said that there are some amazing dashboards out there, more and more of them being both useful and beautiful to look at, which is not a trivial thing to achieve.

Cargo cults

But what happens is we look at those and say: “ooh, look at Council X; they’re really well managed, and they’ve got a great dashboard that shows everything that’s going on, so if we have a dashboard like that we’ll be OK as well.”. It’s a cargo cult. We believe that if we build something that looks like intelligence, we will attract intelligence. It’s magical thinking, basically. The most dangerous bit of that is the belief that we can skip the hard work, take a huge short cut past the self-awareness bit, and simply collect our data into one huge bucket and feed it into some mystical piece of software that will make it meaningful. All the data goes in, all the answers come out.

IMG_0506

What happens if we skip the questioning and the purpose and the organisational self awareness, and just take all the data we have and load up the dashboard generator? We get all the right buzzwords, some pretty infographics, and probably some confused and resentful viewers. Maybe even a few complaints about wasting taxpayers’ money on window dressing and corporate spin.

The thing about dashboards, the really really important thing that’s made clear in the work that we try to skip, is that it is only the presentation layer. It is not the system, and it is not the complete solution. Building a dashboard properly means asking a load of questions, some of which will mean questioning assumptions about our purpose and our understanding of what people need from us. It’s an uncomfortable process, but necessary. Any public facing dashboard that hasn’t asked those questions is a pointless exercise.

IMG_0507

Luckily, the questions that you should be asking are simple, common sense ones like:

  • Is this topic one that people care about?
  • What questions do people ask about it?
  • Will this data answer the question?
  • How do we show an answer in the clearest way?

If the data can’t answer an actual person’s actual question, then it’s not useful. If the data you have fed into your dashboard generator is not useful, then the dashboard is pointless. It is not providing intelligence. Real intelligence means knowing what you have and what’s going on, within some subjective framework of relevance to you or your audience. You need to manage your data into a state where you can trust it, or where you can share it with the right level of caution. There are a lot of data enthusiasts out there who will totally forgive bad data, patchy data, if you are up front about it; they will willingly help you improve your data – but they can’t help you with no data at all, and outright spin and sparkly irrelevant infographics will damage your relationship with them very badly.

You need to get out more

IMG_0599

As a sector, we need to ask the questions to establish what matters to people, and what would be useful. But how do you know what matters and what’s useful? You need to get out more. You can’t do this from behind your desk. If you want to know what matters and what’s useful to the people you want to communicate with, you have to talk to them. Our first test audience was the Members sitting on scrutiny committees, who have been a fantastic source of honest feedback and suggestions for improvement. We also talk to our local hack societies and developer conferences, who tell us that what they want most in the whole world is to know what data we actually have, please, can you just release some sort of list and then we’ll tell you what would be helpful to us? As you can see, we didn’t have to go far to get some opinions that trashed our assumptions about what people wanted.

And it’s not just getting out of your chair, or out of the room, although those help as a start. Be bold. Get out of the building. Get right out of the building and find new networks. You can’t do this if you only talk to the people you normally talk to who work where you work. Get right out of the building and talk to the general public. Join some meetups or community projects. Make friends. Ask your dad, your nan, your teenage children. Try to explain to your friends (who don’t work in local government) what you want to build and why. Trust me, this works. It’s brilliant. You will hear lots of people telling you that they want stuff that you think isn’t what they should want. This might make you cranky or confused, but in the long run it will be good for you and you will get better results because of it.

So when I said I didn’t have the answers or the winning formulae, that’s half the story. I have some tips. They’re based, obviously, on all the things I’ve done wrong or wasted time pursuing; all the assumptions I’ve had shattered and had to re-educate myself around. Here they are:

  1. Get out and make friends
  2. Find out what matters to people
  3. Test your assumptions – be prepared to be wrong
  4. Question your purpose
  5. Try small things – fail fast, and fail cheap
  6. Transparency is a behaviour, not a measure
  7. Dashboards are not magical intelligence generators
Advertisements

About jargonaut

Unashamed geek lost in policy land. Frequently required to believe three impossible things before breakfast, and implement them by tea time.
This entry was posted in The Geek Wonk Interface. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s