Goodharts Law
It’s not just about the metrics
The use of Metrics should broaden your understanding, not make your decisions.
People and organizations adapt their behavior to meet and exceed measures.
Metrics simplify complex systems, potentially overlooking critical factors that can't be easily quantified.
Goodhart's Law is named after British economist Charles Goodhart, who first formulated it in the 1970s. Contrary to popular belief, he didn't actually say, "When a measure becomes a target, it ceases to be a good measure." Instead, his original statement was, "Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes." However, the original wording can seem like it is not written in English, so let's break it down.
What Goodhart meant is this: when people or organizations set specific metrics or indicators as goals to be achieved, those metrics can lose their effectiveness as accurate measures of performance. Which is the opposite of what we are commonly told. This happens because individuals and organizations will adapt their behavior to optimize for that metric. This can distort the original relationship and render the metric less useful for its intended purpose. This distortion is amplified when the metric is used as an individual or team's performance measure and/or has financial incentives attached.
Goodhart's Law has significant implications in various fields, including camps, and it serves as a cautionary reminder of the potential pitfalls of relying too heavily on single metrics as indicators for evaluating and achieving desired outcomes. (Which might seem confusing after reading this. But both can be true similtaniously)
The most frequent example of this I see at camps is incident reporting. Camps often celebrate the staff who have the fewest incident reports, and as a result, people simply stop reporting incidents to get praise. This, of course, poses challenges for us as directors when parents are blindsided by their campers' accounts of what happened, and we remain oblivious. (Fewer incident reports doesn’t mean fewer incidents)
So, what's the solution?
We need to understand that while metrics and Key Performance Indicators (KPIs) are vital data points, relying on them exclusively can lead to a shallow understanding of complex systems. All complex systems have multiple interrelated variables. This is certainly true of camper behavior, staff behavior, and our program operations. Any metric that attempts to record something as varied as behavior must be doing so in an extremely simplified way.
In addition, metrics, by their nature, only draw our attention to problems that can be quantified. This means that only those elements that are easily measurable receive attention.
By focusing solely on the number of incident reports, camps are overlooking the critical nuances and subtleties of camper and staff behavior. Important but less quantifiable factors like camp culture or staff morale are often overlooked, even though they play significant roles in incidents. So, when we ask why there are so many reports and how we can reduce them, we should really be asking: Are there enough staff? Are they well enough trained? Are the kids scheduled for age-appropriate activities? Is it because there is no AC in the cabin? These types of questions are much harder to gather information on and almost impossible to quantify. So, we resort to counting the reports, even though we know that counting the reports themselves leads to an incomplete understanding of the solution. We do it because it's a clean and easy way to capture information on a spreadsheet, and, better yet, it's an easy number to show that a downward trend has been created.
The problem is compounded when your metric, the number of reports, is presented in isolation, divorced from the broader context of what causes behavior at camp. Without context, it's hard to show how your data interlinks and allows people to focus on metrics that confirm their existing beliefs or hypotheses (confirmation bias). People see what they want to see.
So how do we escape this trap?
We find ways of collecting qualitative feedback, staff insights, and camper and parent perspectives to use along side our metric. This qualitative data provides a more holistic perspective that allows us to use our data to tell a more compelling and emotional story. (Its Data plus emotional resonance that changes minds)
We should create an environment where learning from both successes and failures is encouraged. Celebrate the initiative a staff took to reduce behaviors and understand why it may have failed. Celebrate the effort, not just the results, so that others can learn. When behaviors don't improve or there is a sudden uptick, focus on understanding why and use the insights to improve processes rather than assigning blame. [Hint: It’s the heat]
Continuously monitor for unintended consequences that may arise from metric-driven decision-making and be prepared to adapt and address issues promptly.
Finally, assess the relevance and effectiveness of what you are tracking. As circumstances change, adjust the metrics accordingly to ensure they align with the camp's evolving goals.
TL;DR (Too Long; Didn't Read)
Goodhart's Law warns that when you turn a metric into a target, it loses its value as an indicator. Relying solely on metrics simplifies complex systems, overlooking crucial nuances. To avoid this, incorporate qualitative insights, promote continuous learning, and regularly evaluate the relevance of your metrics in achieving your goals.
