This is a nice list to read through. The concept quoted, Goodhart’s law, came up in a project review today, though I didn’t know it was named. It feels nearly unavoidable, but at least recognition helps.
Thanks! I had hoped the original article might provide approaches for balancing the positives and negatives of metrics, but didn't find any. The "Building Less Flawed Metrics: Dodging Goodhart and Campbell’s Laws" paper you linked is exactly what I was looking for. :)
Keep your eye on the prize, and the prize isn't measured by metrics. Any prize in the real world is something complex and complicated, hard to describe and prone to squishiness.
Metrics are things that are "easy" to measure and compare across systems. Setting goals in terms of metrics is appealing and even necessary because it makes things measurable and comparable in the real world, but you need to have some cross-check where you can come back and verify that you are moving closer to your real world goal.
---
A real world example: Consumer products are often distinguished by appearance, and surface quality is an important facet of appearance. You purchase a surface quality tester and improve the measurements produced by the surface tester to the point where you beat a competitor's surface measurements. Then you have a customer clinic and have real potential customers evaluate the two products again. Customers prefer the competitor product because you were not actually measuring attractiveness or appearance of quality, you were measuring something that the surface tester measures.
If the impact of your work can be losslessly reduced to something that can be directly and cheaply measured, you're probably solving an easy problem or a theoretical one.
"How to Game the College Rankings -
Northeastern University executed one of the most dramatic turnarounds in higher education. Its recipe for success? A single-minded focus on just one list."
Nothing says it will not work overall unless you have real examples that it STOPs becoming useful?
Every fucking person goes on about the story of Bill Atkinson and the -2000 Lines Of Code.
He was not fired for not competing the KPI, nor did anyone expect him to be.
In fact he 'beat' the managers in the story and they stopped asking him for submitting lines of code because they are not stupid.
No one talks about if it was a good measure for other staff, which at the time(early 80's) might have been good at a personal level or at a corporation level.
People can see gaming and people can see when it's not fair. Just because you have a friend of a friend who read a story about gaming doesn't mean this 'it ceases to be a good measure' is true.
> Goodhart's Law: When a measure becomes a target, it ceases to be a good measure.
This makes me think of the first song I posted to YouTube. I couldn't help but curiously flip through the viewer statistics. Automatically I began wondering how I could prevent the 50% drop-off at X seconds, and what type of person was sticking around beyond the drop-off. And before I looked much into it I realized this would be a great way to suck all the life out of something that's supposed to be fun.
This is one of the many reasons I don't run analytics on my website (in addition to the more obvious ones, such as privacy). In my case, I still get useful "second-order" information from the frequency of people writing emails to me about the content, and it feels a lot nicer to read that someone found something I wrote useful than to try to micro-optimize for what people are willing to click on.
We changed the submitted title to that of the page. The submitted title was Hacker Laws: “When a measure becomes a target, it ceases to be a good measure.”, which of course is Goodhart's Law. Some of the comments in this thread are referencing that one.
It's commendable that the author counteracts the tendency towards cynicism with the occasional rebuttal, although to fully embrace the spirit of kindness, I would maybe skip the Dilbert reference.
Remember: If everyone really were as stupid and/or evil as some of these maxims postulate, it'd be highly unlikely that you just happen to be the exception. Also: management, design, law, marketing, journalism, social sciences, and politics are all disciplines just as challenging as programming or engineering.
And if you can't think of any reason why one of those fields should be difficult, you're just as likely to better at it as that field's practitioners being better at your job.
Would anyone care about discussing some 'rule' with barely greater explanatory power than flipping a coin? People aren't going to adjust their thought patterns, in practice, for such an improvement.
DRY is dangerous if applied to pedantically; sometimes similar code should be allowed to exist to make overall architecture simpler, or allow similar components to evolve (possibly Divergantly) over time.
The Pragmatic Programmer also warns against overzealous deduplication. Stuff that are coincidentally the same should be left untouched. Only remove duplication of knowledge.
28 comments
[ 3.0 ms ] story [ 70.7 ms ] threadMetrics are things that are "easy" to measure and compare across systems. Setting goals in terms of metrics is appealing and even necessary because it makes things measurable and comparable in the real world, but you need to have some cross-check where you can come back and verify that you are moving closer to your real world goal.
---
A real world example: Consumer products are often distinguished by appearance, and surface quality is an important facet of appearance. You purchase a surface quality tester and improve the measurements produced by the surface tester to the point where you beat a competitor's surface measurements. Then you have a customer clinic and have real potential customers evaluate the two products again. Customers prefer the competitor product because you were not actually measuring attractiveness or appearance of quality, you were measuring something that the surface tester measures.
https://www.bostonmagazine.com/news/2014/08/26/how-northeast...
"How to Game the College Rankings - Northeastern University executed one of the most dramatic turnarounds in higher education. Its recipe for success? A single-minded focus on just one list."
When a measure becomes a target it will be gamed.
Nothing says it will not work overall unless you have real examples that it STOPs becoming useful?
Every fucking person goes on about the story of Bill Atkinson and the -2000 Lines Of Code.
He was not fired for not competing the KPI, nor did anyone expect him to be.
In fact he 'beat' the managers in the story and they stopped asking him for submitting lines of code because they are not stupid.
No one talks about if it was a good measure for other staff, which at the time(early 80's) might have been good at a personal level or at a corporation level.
People can see gaming and people can see when it's not fair. Just because you have a friend of a friend who read a story about gaming doesn't mean this 'it ceases to be a good measure' is true.
This makes me think of the first song I posted to YouTube. I couldn't help but curiously flip through the viewer statistics. Automatically I began wondering how I could prevent the 50% drop-off at X seconds, and what type of person was sticking around beyond the drop-off. And before I looked much into it I realized this would be a great way to suck all the life out of something that's supposed to be fun.
McIntye's Law: Under the right circumstances, anything I tell you could be wrong.
I preface much of my advice to young developers with this.
Under the right circumstances, anything I tell you could be wrong. Including this.
Remember: If everyone really were as stupid and/or evil as some of these maxims postulate, it'd be highly unlikely that you just happen to be the exception. Also: management, design, law, marketing, journalism, social sciences, and politics are all disciplines just as challenging as programming or engineering.
And if you can't think of any reason why one of those fields should be difficult, you're just as likely to better at it as that field's practitioners being better at your job.
The Second Law of Consulting: The customer's list of constraints is incomplete, because fish don't think about water.
Otherwise it is a not a statement that can be objectively "factual" and is as such not a proposition.
Everything else is culture and can thus be discarded.