The 3 Commitments of a Data-Informed Organization: Part 2

Find Objective Ways To Tell if Your Efforts Are Working

May 28, 2024

[Short on time? Read the TLDR version at the end.]

At last we come to the data!

Of course, the data is still not the point. It’s only purpose is to tell whether your efforts are working and to guide your response if not.

I ended the last issue with a focus on the concrete experience of those the results are intended to benefit. That’s not an accident. In fact, it is critical to identifying the best data for assessing results.

In his book Trying Hard Is Not Good Enough Mark Friedman asserts that all performance measures boil down to three questions:

How much did we do? How well did we do it? Is anybody better off?

Most local government reporting is heavy on the first two and light on the last.

That’s not surprising - the first two questions fall squarely in the organization’s comfort zone. They focus on what you do and measure things that you have some control over.

Those things are important, of course. A police department that doesn’t know its area coverage or a sanitation department that can’t tell you how much they pick up where probably have other issues as well.

But your customers do not care.

Metrics Are A Leadership Issue

The kinds of metrics you use to measure performance are not just a technical issue. They are a leadership concern as well.

First, because being accountable to your community means knowing whether and how your services or initiatives meet their expectations.

Second, because metrics that convey impact on customer experience are ideal for communicating impact to those customers. That in turn helps you advocate for the resources needed to deliver it (as every local government in North Carolina is trying to do right now).

Sometimes it’s pretty easy.

For residential trash pickup, say, it’s probably enough to tell people how often you pick up, how often mistakes happen, and how quickly you respond to them.

Other examples might include the typical wait for a rapid rehousing program, recidivism rates for domestic violence offenders attending an intervention program, or fire department response times.

But often it’s much more challenging.

For example, it can be hard to measure impact in a way that makes sense to customers. Take stormwater infrastructure. Done well, nothing happens. Done badly, nothing still may happen, at least for a while. You need to get creative, perhaps by using actual volume and infrastructure data to create scenarios of likely outcomes at different levels of investment.

When Results Are Not All In Your Control

More often than not, though, the challenge is that individual programs are only one of many contributors to the overall result.

That’s a major problem in public safety. Communities think of police performance as measured by crime levels, but enforcement is just one of many factors and far from the controlling one. Similar issues arise around homelessness, housing affordability, economic development, and public health, among others.

The temptation is to tell a story that puts the agency in the best light (inevitably changing it to adapt to changing trends). That may work in the short term, but erodes both organizational effectiveness and public trust in the long term.

That’s why it’s important to think through both proximate and ultimate goals. Lower crime is an ultimate goal, while policing is one of several efforts that seek to affect it.

Communicate that.

Measure both, but communicate them in a way that reflects their actual relationship. Your measurements of community well-being don’t align with your org chart. So don’t present them that way.

For example, while the police are primarily responsible for compiling and reporting crime statistics, it may be better to present those statistics somewhere other than on police department web pages. An independent area for reporting community-level metrics like crime allows you to tie community-level results to all the programs that influence them.

A Final Note On The Word ‘Objective’

We tend to think that data is about numbers and that translating things to numbers is what makes them objective.

But numbers (and other data) are always embedded within narratives. Narratives guide what we measure and how we interpret what those measures mean.

While objectivity may be elusive, our best shot at it is to focus on the concrete outcomes that community members actually see and experience. With those as anchors, the stories we tell around them are likewise anchored. They then help you both to be more effective and to support a more constructive discussion within the community (i.e., politics).

Now you have identified clear results, good measures, and an effective way to communicate them. But all that effort is naught if you don’t actually use them. That is the topic of the next issue.

Links & Thoughts

Data-informed community engagement. I watched a National League of Cities webinar last week on Using Data-Informed Community Engagement for Safer Cities. The framework, pioneered by the Newark Public Safety Collaborative, aims to address the fact that public safety is more than a law enforcement issue by engaging multiple stakeholders in collaborative problem-solving based on data.
AI! AI! AI! For a deep dive into the wild, shifting landscape of AI and the companies behind it, I recommend the 2024 MAD (Machine Learning, AI & Data) Landscape. Way too much detail for most people (including me), but I found it interesting to skim, especially the 24 themes they’re thinking about in 2024. HT to Vicki Boykis.

tldr

This issue examines the second of three commitments of a data-informed organization, to find objective ways to tell if your efforts are working. There are three important considerations:

The best metrics directly reflect the experience of your customers. It’s much easier to measure what the organization does than it is to measure outcomes, so leadership here is key. This focus on outcome metrics keeps you more accountable to your community and makes it easier to justify the resources needed.
When outcomes are not all in the organization’s control, it is important to clearly maintain a distinction between proximate outcomes and ultimate ones. Measure both, but communicate them in a way that reflects their actual relationship. This has important practical consequences.
More generally, be aware that data is always embedded within narratives. Carefully anchoring your organization’s narratives in outcomes and their measures both enhances organizational effectiveness and supports more constructive political discussion.

To learn more about what I do and how we can work together visit DeepWeave.com.