The 3 Commitments of a Data-Informed Organization: Part 3

Take Action and Change Course If Your Efforts Are Not Working

Jun 04, 2024

[Short on time? Read the TLDR version at the end.]

Evidence-based programs are all the rage lately. It makes sense: why would we focus on programs with no evidence in their favor?

But there is an issue lurking here.

“Evidence-based” is too often is taken to mean something generally proven to work. Just implement this in your jurisdiction and … voilà … success!

Good luck with that!

A recent paper by Megan Stevenson underscores this for interventions in the criminal legal space. Her conclusion is stark: most “have little lasting effect when evaluated with gold standard methods.”

The issue is that social systems have ‘stabilizing forces” that drive people back to where they started from. Earl Mu’min Simms, a community advocate and leader in San Francisco recently gave them a punchier name: “immune systems.”1

Does that mean that we should give up on evidence-based programs?

Absolutely not.

Stevenson herself emphasizes that the lesson is that we must stop thinking about social systems as if they are machines amenable to simple mechanical fixes. An evidence-based program is one that has been shown to work somewhere. It can be a good place to start, but will need to be adapted to the very different particular circumstances of your where in ways that may not be immediately obvious.

That is the point of the third commitment of a data-informed organization: to take action and change course if your efforts are not yet working. In other words, to learn.

Using data to design solutions is a process, one that starts by assuming that the first iteration will likely not work, at least not as well as you hope. A second iteration will be needed, or a third, … or a tenth.

Easy to say, but it usually requires changes in how you do things.

Thwarting Learning in Grant-Making

Take a concrete example: grants given out to nonprofits and community organizations.

You are probably familiar with how it works. Organizations submit proposals, some number are selected through an evaluation process, then contracts are prepared, with a fairly detailed budget and a scope of work that is probably copy/pasted from the proposal.

The effort kicks off, but things don’t work as expected. The organization realizes that funds need to be invested somewhat differently than planned.

Too bad; the contract itself denies them the ability to put that learning into practice. That’s not only possible, it’s common, as I’ve come to realize in working with organizations that received ARPA funds through the City of Asheville.

Fortunately, the City took a different approach, one that explicitly created room for learning. Between the original proposal and the final contract, the City worked with each organization to take both the statement of work and the budget up a level, focusing on naming outcomes and broad categories of spending rather than trying to specify every detail.

The result is that, in a number of cases, organizations were able to pivot to address significantly changed circumstances and critical learning during implementation.

In other words, the City is holding them accountable to the goals rather than to the small print.

I’ll be writing more later about Asheville’s approach to its ARPA funding and some of the interesting stories that have resulted, but here I want to underscore the point that commitment #3 meant making a significant change to the normal course of business. Without that change, the ability of the City and its grantees to “take action and change course” would have been hamstrung.

Learning on the Inside

Internally-driven initiatives may not be constrained by legal contracts, but they also require significant changes in how things are done. Cultural constraints replace legal ones.

Cultural barriers can arise at any point, but frankly the single biggest reason organizations fail to keep commitment #3 is that they fail to do commitments #1 and #2. The usual order that I’ve observed is:

Decide what we’re going to do
Come up with some metrics
Do it

Iteration often happens, but it tends to be painful and haphazard. It’s painful because the initiative is presented as an answer rather than a first step toward an answer, so people tend to be vocally disappointed. And it’s haphazard because not designing for iteration means you’re scrambling rather than approaching the next iteration systematically.

That last is key. Systematic learning is not something that “just happens,” it’s a process that needs to be designed, planned, and supported by the organization.

I’ll discuss that in more depth in a couple weeks. Next week I want to talk briefly about the difference between measuring success for operations versus for strategic initiatives.

Links & Thoughts

Behavior change is contextual. Here’s a specific example of the ways that you have to be careful about “evidence-based” practice recommendations. Elina Halonen reviewed a recent much-touted paper on behavioral interventions and concluded that it “oozed confirmation bias.” This LinkedIn post offers a nice summary and link to the full review.
Having to be in all the meetings. A non-profit executive whose service covers multiple social issues, from housing to domestic violence to health, recently commented that the only way to discover all the relevant funding opportunities was to participate in all the monthly collaborations across all the focus areas. This parallels what I’ve observed even inside a single organization: the only way to know everything relevant to you is to be in all the meetings. Intra-organizationally and community-wide, finding ways to share information across siloes without meetings is one of the most impactful things you can do.

tldr

This issue examines the last of three commitments of a data-informed organization, to take action and change course if your efforts aren’t working as expected.

Implementing evidence-based programs can seem like a short-cut to having to build learning into your own efforts. Not so, unfortunately. They can be a good place to start, but will need to be adapted to your particular circumstances in ways that may not be immediately obvious. Learning has to be a part of it.
Existing processes and culture are typically not designed to support iteration and learning. They often thwart them or make them harder than needed, and may even actively prevent them.
Systematic learning and effective execution of the three commitments is not something that “just happens,” it’s a process that needs to be designed, planned, and supported by the organization.

To learn more about what I do and how we can work together visit DeepWeave.com.

Earl Mu’min Simms is a community leader in San Francisco who is helping address the intersection between homelessness and criminal justice. I was delighted to get to meet him at the Safety + Justice Challenge annual convening in Houston last month. I love the analogy of an immune system in understanding how systems tend to bounce back after any change, even if no one intentionally drives them back.