Systems vs Systems, Part 2
Two Other Legs and Putting Them Together
[Short on time? Scroll down to read the TLDR version.]
Last week I began exploring a three-leg framework for balancing efforts toward building a data-informed organization, starting with leadership and accountability. This week I’ll look at the other two and how they interconnect.
Leg #2: Data Management & Governance
When leaders get excited about managing performance, spreadsheets happen. This is usually followed either by nothing or by a mess.
Nothing is when the burst of enthusiasm was just that: a burst, not an real commitment. Otherwise you get the mess.
This leg helps avoid both by managing data availability, quality, and risk.
Data availability is about ensuring that the data exists and is accessible.
That requires designing and maintaining a process. It needn’t be complicated: someone tasked with preparing the data, a reminder for them to do it, a way to transfer responsibility when they leave, and a means to get the data to users is enough. Automate this if you can, of course, but the key is establish a process from the start.
Data quality means ensuring the data is complete and accurate enough for the decisions it supports. Missing and inconsistent values wreak havoc with your ability to draw conclusions.
For internal data, the key is to manage the point of input through process or with simple techniques like input validation. Once bad data is in your systems, it’s very hard to fix.
For external data you must be knowledgeable about the data’s source, including the assumptions made in aggregating it or any quality issues it has. A reliable dataset will provide that information. You may not be able to control the quality, but you can account for it in your decisions.
Data risk is about managing all the ways that collecting or using data might compromise security or privacy or harm the organization or those the data describes.
Here it’s vital to ask basic questions. Are there security implications or sensitive information that might impact individuals or disparately impact certain groups? Personally identifying information (PII) like social security numbers is obvious, but simply including legal names can inadvertently expose transgender folks who choose not to share the gender assigned them at birth.
In addition, every dataset has built-in bias which can lead to inequity and other harms. What we collect or don’t, who is included or not, where we do or don’t collect it all play into what we can “know” from the data.
Be sure to engage with people the data represents so they can raise issues that you may not see.
How do you get started with this leg? My advice: don’t start with data. Start with a key resource or strategy decision in service of a high-priority outcome. Let that decision drive what data you consider.1 Then work through the three points above.
The key is to ask the questions. The most egregious data problems don’t arise because people aren’t smart enough, but because smart people don’t know to ask.
Leg #3: Capacity-Building and Support
Supporting people and building capacity is about training, tools, and time (you’ve probably noticed that I like threes).
First, don’t assume people know how to do it. They probably don’t and even those who do can use refreshers. That means training.
Training should take multiple forms to support the many ways people learn. Classes are a good foundation for shared vocabulary, values, and systems. Newsletters and videos are also great, as are opportunities for staff to teach each other by sharing successes and challenges. Finally, a consulting model can build capacity by letting people collaborate with experts or colleagues with more experience.
Second, you need to invest in tools beyond spreadsheets.2 That includes technical tools for data management, visualization, and reporting plus staff to support them. No less important are conceptual tools like checklists and step-by-step processes. I’ve built several based on the results-based accountability (RBA) framework, but the specific framework and tools are less important than having them.
Finally, if you want staff to collect, report, and use data in their work, make sure they have the time they need to do it. Data-informed organizations front-load work to avoid (expensive) future problems, but you can’t both do that and make everything reactively urgent.
In getting started, your first focus should be on change management: train people on anything you want them to do differently and communicate incessantly about why. I also recommend picking a framework that you can use to build shared language and processes around. Simpler is better, but consistency is all.
Connecting It All
Each leg is just a type of effort. What makes them a system for contending with the system you want to change is how they connect.
The first leg, leadership, is also the most important connector. Its most important task is to make all the other work matter. Data that isn’t used and valued won’t be kept up, won’t be high quality, and may constitute a hidden risk waiting to blow up in your face. Leadership is also the backstop of authority needed to enforce processes and to convene stakeholders.
Training and support are also critical for connecting things, first as a core tool for change management, but also to both impart new skills and unlock latent capacity. Data governance, for example, isn’t something you can just delegate to a few geeks in the basement. Front-line staff and processes are your best guards against quality and risk issues, and that means all staff need to be aware.
Finally, the best way to learn to be an effective data-informed organization is to make the process of learning itself a data-informed endeavor. Create a process or, even better, tie into an existing process (annual budgets, anyone?) in which you name the outcomes you seek, find ways to measure progress, and regularly evaluate and adjust as you go.
Apologies that this article is longer than my 800-word target maximum, but I really wanted to get this dose of theory done and move on to some practical examples. Next week: rethinking performance management for grants.
Further Reading
The article above is part of a larger series. Here it is so far:
Links & Thoughts
Leading with Illiteracy. The Royal Statistical Society sent letters to the main political parties in the UK calling for data literacy training for political leaders based on a survey that found barely half could handle simple probability questions. My guess is that the issue is not confined to the UK Parliament. HT: Semafor Flagship newsletter.
Forest versus Trees. Really interesting article on impact measurement, including some great historical context. Worth the read for multiple reasons, but this stuck out for me: “It is getting too common for impact measurement to focus on the granularity of individual projects, companies or initiatives with the expectation that measurement at these levels will somehow add up to overall impact across much broader fields such as sectors, industries and even systems.”
tldr
Last week I discussed the first leg of the framework, leadership and accountability.
Leg #2, data management & governance, is about managing data availability, quality, and risk with a focus on process and engaging the people who are impacted.
Leg #3, capacity-building and support, is about giving people the training they need to take on a new way of doing things, as well as providing appropriate tools and the time needed to do more work up front.
None of the legs can work on their own; only connecting them creates a system that can contend with the system you want to change.
Leadership’s job is to make all the other work matter and to provide the authority for other to drive the change.
Training and shared tools and frameworks are core tools for change management, as well as for deepening the ways staff support each other in the new culture.
Data management and governance is largely about creating processes that effectively embed data practices in the way the organization works, including making becoming a data-informed organization itself a data-informed process.
I help organizations think about how to use data to improve results for themselves and their communities. To learn more about what I do and how we can work together visit DeepWeave.com.
If you want to share thoughts on anything I’ve said here or have ideas about further questions or topics you’d like me to explore, please feel free to reply to the newsletter email or contact me here.
Actually, this is my advice even if you’re not just getting started. The decision is the whole point, so that’s where you should start.
I know, I’m hating on spreadsheets a lot. I actually love spreadsheets, but when it comes to data management and governance, they tend to be a bit of a disaster without super-strong processes around them.