Measuring outcomes is hard, especially in comparison to outputs which are easier to quantify but are different than outcomes. True measures of outcomes usually can only be tracked over a long period of time, whereas outputs will give instant feedback.
Look at what has been happening in the schools with standardize testing. Proponents think they are measuring outcomes (effectiveness in preparing children to be skilled & productive members of society) with standardized tests. They are actually measuring outputs (test scores) as a proxy for the outcomes.
The result is the system being measured will adapt. Schools "teach to the test" which improves the outputs but we don't know the effect on the desired outcomes. It may result in children being less prepared.
This is a well known effect in companies that measure individual performance in terms of outputs (widgets manufactured, # of upsells, etc). The employees will deliver the required metrics but it often comes at a cost to quality and customer satisfaction which is a negative outcome to the company.
Which is the main lesson in quantifying outcomes: Measuring the wrong thing can be worse than not measuring at all.
"Measuring the wrong thing can be worse than not measuring at all."
I'm calling bullshit on that one.
Yes, measuring the wrong thing can incentivize poor outcomes, but movement in the wrong direction is visible and can be corrected, whereas lethargic inaction just lazily drifts into a morass.
You have to measure outcome. Period. We can have great discussions around what kinds of measurements to have and how to apply them, but money should not go out the door without some indication that something is happening as a result of it -- even the wrong thing. In fact, big visible indicators of the wrong thing happening as a result of measurements is a terrific result. If we're arguing over measurement, the game is already lost.
Another way of stating this is that let's say I pick a measurement to implement. It's a terrible measurement, and it causes much pain and suffering. You come to me and say "This is a terrible measurement and causes much pain and suffering!" To which I respond "How do you know this?"
So you show me a study -- another form of measurement! Then we have a conversation about including parts of your study in my measurement, scrapping my measurement entirely, and so forth.
We cannot have meaningful conversations about management without measuring things. Once we start measuring things, the world is wide open about which things to measure. But if you convince me that the act of measuring itself is broken, we have no further basis for any conversation around performance at all. Management has no function. There is no quantifiable return for any possible amount of money I might invest. Very bad.
We don't need to wait around to pick the perfect measurement or do nothing at all. We need to pick measurements -- any measurements -- and rapidly iterate towards finding what works best for us.
If the testing system is broken, and I have no doubt that it is, it's broken based on bureaucrats creating a large, brittle, hard-to-change architecture of testing, instead of just trying to continuously evolve what works best. It has failed because we put the same idiots in charge of school testing that were creating the mess the tests were supposed to fix in the first place.
No, he shows you an anecdote or two. Which isn't worth very much, but it's more than what you had before. It's not like the original measurement had any studies validating it either. But it looks scientific: it has numbers!
You know what measures outcomes exceptionally well? The free market. Turns out, if you suck at what you do, or a project is mismanaged, it fails. When there isn't a constant source of revenue via taxes, people suddenly start to care whether or not money is being spent wisely. And wouldn't you know it, they come up with all sorts of sophisticated ways to determine whether or not they're likely to fail and adjust accordingly.
That kind of money is tiddlywinks for city's and states, the real ongoing scam is what is called the 'capital expenditure budgets' of these orgnizations. Massive sums of money, that tend to go no-bid, or fixed bid contracts with the authorities buddies. Services to constituents get cut, but the capital exp. budget never gets cut, it is always increased.
£1 billion (and counting) to transport people along a pretty short route in a city that has excellent bus and taxi services is a wonder of our time - there are only half a million or so people in Edinburgh so that's £2000 per person for a service most of us will never use....
Golly that sounds terrible. San Francisco must be an unbearable cesspit. Oh wait, that's right, it's my favorite city in the US, has a better than decent public transport system, and given a choice many people will pay extra to live there.
I've heard some amazing stories of mismanagement and excessively liberal policies (e.g. Homeless people being paid stipends by the city) but there are plenty of well managed cities I would move out of in favor of San Francisco. (My favorite post in the linked thread was a comparison of SF to Saudi Arabia.)
Maybe JWZ is right when he says the city of SF is clueless about business. It certainly seems to be clueless about the nighttime entertainment business, which is where his rants and his increasingly bad temper seem to come from.
Do people understand that SF's supposed dysfunction is a major contributor to its charm? If it is supposedly "worst run" is that something to aspire to? A lot of people mistake "despite" for "because of" when it comes to stuff like this.
13 comments
[ 2.8 ms ] story [ 38.8 ms ] threadLook at what has been happening in the schools with standardize testing. Proponents think they are measuring outcomes (effectiveness in preparing children to be skilled & productive members of society) with standardized tests. They are actually measuring outputs (test scores) as a proxy for the outcomes.
The result is the system being measured will adapt. Schools "teach to the test" which improves the outputs but we don't know the effect on the desired outcomes. It may result in children being less prepared.
This is a well known effect in companies that measure individual performance in terms of outputs (widgets manufactured, # of upsells, etc). The employees will deliver the required metrics but it often comes at a cost to quality and customer satisfaction which is a negative outcome to the company.
Which is the main lesson in quantifying outcomes: Measuring the wrong thing can be worse than not measuring at all.
I'm calling bullshit on that one.
Yes, measuring the wrong thing can incentivize poor outcomes, but movement in the wrong direction is visible and can be corrected, whereas lethargic inaction just lazily drifts into a morass.
You have to measure outcome. Period. We can have great discussions around what kinds of measurements to have and how to apply them, but money should not go out the door without some indication that something is happening as a result of it -- even the wrong thing. In fact, big visible indicators of the wrong thing happening as a result of measurements is a terrific result. If we're arguing over measurement, the game is already lost.
Another way of stating this is that let's say I pick a measurement to implement. It's a terrible measurement, and it causes much pain and suffering. You come to me and say "This is a terrible measurement and causes much pain and suffering!" To which I respond "How do you know this?"
So you show me a study -- another form of measurement! Then we have a conversation about including parts of your study in my measurement, scrapping my measurement entirely, and so forth.
We cannot have meaningful conversations about management without measuring things. Once we start measuring things, the world is wide open about which things to measure. But if you convince me that the act of measuring itself is broken, we have no further basis for any conversation around performance at all. Management has no function. There is no quantifiable return for any possible amount of money I might invest. Very bad.
We don't need to wait around to pick the perfect measurement or do nothing at all. We need to pick measurements -- any measurements -- and rapidly iterate towards finding what works best for us.
If the testing system is broken, and I have no doubt that it is, it's broken based on bureaucrats creating a large, brittle, hard-to-change architecture of testing, instead of just trying to continuously evolve what works best. It has failed because we put the same idiots in charge of school testing that were creating the mess the tests were supposed to fix in the first place.
No, he shows you an anecdote or two. Which isn't worth very much, but it's more than what you had before. It's not like the original measurement had any studies validating it either. But it looks scientific: it has numbers!
I guess that's better than £1 billion on a not-very-good idea that fails for stupid reasons:
http://en.wikipedia.org/wiki/Edinburgh_Trams
£1 billion (and counting) to transport people along a pretty short route in a city that has excellent bus and taxi services is a wonder of our time - there are only half a million or so people in Edinburgh so that's £2000 per person for a service most of us will never use....
http://www.sfweekly.com/2009-12-16/news/the-worst-run-big-ci...
I've heard some amazing stories of mismanagement and excessively liberal policies (e.g. Homeless people being paid stipends by the city) but there are plenty of well managed cities I would move out of in favor of San Francisco. (My favorite post in the linked thread was a comparison of SF to Saudi Arabia.)