If you skipped straight to the HN comments, the article is worth reading. The author makes some thoughtful suggestions and points out some common pitfalls.
Although recently, I've noticed another reason to tread lightly with metrics: Metrics have become a highly-charged topic among developers who consume a lot of Twitter, Reddit, HN and other developer chatter. The cynical response to any metrics tracking program is to claim that they will be gamed, or abused, or misused by management. Once your team has adopted this cynical mindset, they're unlikely to see metrics any other way, so they get to work gaming and abusing the system.
In my experience, getting as much team buy-in as possible is the key to making metrics a success. It's healthy to invite the teams to an open discussion about what metrics they think should be tracked. Give them an opportunity to openly discuss objections, and make a valid effort to acknowledge and work with their suggestions as you develop your metrics program. Ideally, you would find one or two metrics that benefit the team members directly rather than the company or management.
It's important that developers can see the metrics as an open target to help them excel within the company, rather than a punitive measure or something that management can use as a substitute for real engagement. When team metrics decline, use that opportunity to open a discussion with the team. If you let metrics become a punitive measure or a guide for who gets promoted or fired, they will absolutely be gamed to death.
I'd like to see evidence that cynicism about metrics leads to abuse and gaming rather than just leaving. That perspective seems cynical in itself.
In my experience metrics make no difference on success, the dynamics of the team and the workflow have been set up for success or failure before you started measuring it. Metrics should never be a target, once a measure becomes a target it ceases to be a good measure.
Measures should be used to give you awareness of time and quantity. Quality as efficiency is inaccurate, it ignores the properties of the work created. Being aware of time and quantity gives managers of large empires the ability to summarize the work of many individuals into a few measures. This kills the point of individually excelling and creates the grounds for union largess. In a few years everybody realizes that the work is not created equal and you create an incentive for real quality minded people to started ranking everybody and allowing room for clearing the bottom 10%.
The same ideas in the article are pushed across youtube, across HN and the internet culture at large, in time with each other. It's disappointing to see so many smart people walk in mental lock step with one another. It devalues the individual contribution and shows there is little left for the majority of these content creators to say that hasn't been said elsewhere.
Saying that the team dynamics have set up for success or failure on the get go sheds light only on teams that are on the positive side of that statement. What if my team is on the negative side? Should I just accept the loss and continue to a new project?
I agree that the dynamics of a dev team are extremely important. But how can one drive to get better without understanding their current status?
In my personal experience, there should be an open conversation between the team on deciding what to measure and what does the team want to improve on even if the team already has good dynamics. My moral as a developer was always up when my team got props for something, because I was a part of that success.
I think the focus needs to be on the team as a whole, instead of getting dragged into a comparisons game. The developers should be aware that this isn't an individuals race.
If you work in a large organization then you will still be measured even if you're team doesn't take the measurements on their own. Good metrics can only benefit you since you can control the conversation with your own data instead of data from a corporate figure that only looks at you as another row in their spreadsheet.
It is a race of individuals when you see people on HN posting about making hundreds of thousands for doing the job of multiple people. Sure not everybody wants to do that. What is the point in hobbling those that do by forcing them to fit a measure?
The individual seeing with metrics is not the action taking mind. The person who sees through metrics is a heart hidden behind a corporate body. The heart cannot imagine greater heights, it can only regulate flow. Blood pressure spikes get eliminated for "normality".
This lack of thought and focus on flow eliminates morals and devalues changing forms and structures in favour of a mindless gigantism. Corporations don't change individuals, they lump responsibility for failure upon me's and you's. The corporation's goal is to grow the flow at human cost. I among many would prefer not become one of Howard's humanoids (Network 1976) and would instead prefer to go out on the stage of individual achievement, if at all. We are all vulnerable to the snakes that encircle and poison hearts. Ideas go in one side of the heart and come out the other side dead.
I don't think so. Basically, with metrics you are saying that you need to reduce a multidimensional value down to a single dimension. In addition, metrics are open to gaming. Take for example cycle time:
>Cycle Time: If you can only have 1 metric, make it cycle time. Cycle time measures the amount of time from work started to work delivered; in other words, the amount of time from first commit to production release. Cycle time is important because it is a proxy for how optimized your team is.
If cycle time is being optimized for, then eventually, someone on the team will discover that if they make an "experimental prototype" on their laptop before checking it in, their cycle time goes down a lot.
There really is no substitute for an involved manager, however that does not scale, and so you go to metrics. But, I would argue, you have already damaged the culture.
You’d never take a team wide metric like cycle time (which is a function of the whole company) and use it to evaluate a single person. There shouldn’t be any benefit to an individual to have a low cycle time.
> You’d never take a team wide metric like cycle time (which is a function of the whole company) and use it to evaluate a single person
In theory, yes. But in practice, I bet there are a lot of people here who have stories of a metric that was supposed to only be used team wide, but was used for evaluating an individual.
Honestly I don't think you can avoid getting measured, even if the person measuring only does it in their head. If you feel you're doing a good job than it's best for you to measure that on your own and show proof for that instead of your boss relating on their instincts...
Also if you feel you can do better than it's probably best that you understand why in order to improve that, so your boss won't have the need to let you go.
Everyone's looking at the one negative side but there are tons of positives that apply to a large percentage of the worforce
When leaders start micromanaging and overreaching, there's really no more autonomy and the responsibilities for outcomes are taken away from the individual. Notice: This can happen at the management-level as well!
Similar with metrics, it becomes a diversion to attend to waste (extra efforts to produce quality data), while removing autonomy and opportunities.
Those metrics that do work, are owned by people themselves, and are used as tools, not as a whip, carrot or other games of diversions. As tools, the metrics can be changed or removed, as seen necessary.
Visible metrics that tie to company, product or infrastructure health are highly useful if they are relatively objective and not subject to gaming. But tying them to individual performance is always the wrong thing in knowledge work; not only is it dehumanizing but it also fails to account for the subjectivity in any creative profession where people with the same job title work on radically different things.
As a manager there are some individual metrics that could serve as a “smell” to investigate but you shouldn’t publish those metrics or ever use them in communications or as reasoning for any decision.
I think metrics are very useful but only when used properly. A lack of metrics means we can't always see what's going on and goals may be unclear; an overreliance on metrics is gameable and too impersonal.
Laos, as the article indicates to some extent, what you measure becomes what matters. As a manager, the metrics you create will (to an extent) help to define what kind of team you run and build.
What is this "culture" thing in the article? It's undefined although apparently there's a danger of it being bad. Things can "impact" it. It can be grown and it can be damaged. I'm left thinking it may be a god or it may even be a skin rash.
What are they selling? Oh, project management software.
A try: the set of circumstances that cause your current employees to be motivated and productive.
If you currently have employees that have a lot of freedom, responsibility and little oversight and who thrive on that, then introducing more formal metrics can make them feel their work is becoming less fun and they become less motivated. But other employees like clear structure and formal methods and they'd welcome such things.
Usually the company has one style of management from the start, and people who like working under that style self-selected over time. That gives you your "culture", and changing it means you may make your company's management style less of a good fit for its employees.
I've been stuck working on a problem for the last two weeks, I know this by the lingering feeling of dread, and anyone can tell by the empty squares in my commit heat-map. How can metrics help me make progress, except to tell me to try something new: Read papers, study similar solutions, reach out to other that can help. Nothings has worked so far, but it does not deter me.
I did learn a curious fact about the productivity of the most prolific writers, musicians, comedians and other creative professions. Count the number of words written, notes composed and new jokes, then divide those by a reasonable number of working hours, let's say 5 days a week with 8 hours of work over the productive period in their life.
Unfortunately, I can't find the exact numbers anymore, but it was staggeringly low, something in the vein of 1 word written per hour, 1 note composed per day, and one new joke per week.
Sure there are things to be learned from data, just not always what fits the agenda of the managerial class.
> Unfortunately, I can't find the exact numbers anymore, but it was staggeringly low, something in the vein of 1 word written per hour, 1 note composed per day, and one new joke per week.
This is so ridiculous as to be offensive. Bring the downvotes, I don't care.
Write down what you do. Regularly talk to a colleague about your thoughts, it helps to get them clearer.
And some things just need weeks of reflection, research, trying things out. Rome wasn't built in a day. When you'll get there you'll have produced something of actual value.
I find the best way to deal with long running, difficult problems is to (1) document what you are doing and (2) prototype as fast and often as possible. That way you have a measurable record of a systematic attempt to tackle the problem. If the team metrics don’t capture that, you have an easy argument to make that they should and a practical way to do so (commits to a readme or wiki or prototype branch are measurable).
It also helps your teammates and managers identify when you are spinning your wheels and need help. As a tech lead/architect, I find prototypes and documentation about a hard problem extremely helpful for identifying when I’ve made a bad technical decision and selected an approach that simply isn’t efficient or workable. Otherwise, it’s hard to distinguish between that and when the individual dev is simply not up to the job.
It’s also much easier to solicit feedback when you’re stuck if you can offer a record of what you have tried.
If you use metrics to evaluate performance, expect EVERYTHING to be gamed around them.
My friend at Facebook says that everything is based on metrics. So people will ask others to give them a “thanks” in the Performance system for Mundane things like going to a meeting. Numbers of reviews that are commented on are also a metric, so people will +1 a review.
This happens everywhere, not just Facebook. Whatever metrics you employ will incentivize your employees to target those. If you value LOC then those will be targeted. If you value number of checkins, then those will be targeted. And that’s how it should be because that’s what you as a manager are defining to your employees is what is important and valued.
So if you do choose metrics then be aware that it will change the culture.
That's the lovely thing about working in HFT: can make the PnL the metric, as it's much easier to connect work to PnL impact, and then everybody is working towards increasing an actually useful metric.
Sounds like a lot of people in this thread are confusing metrics with measures. A metric is just that. Its just a number.
It means nothing except giving you historical context over time.
A measure is when you expect a specific quota or compare a metric against another.
Goodharts law, "a metric that becomes a measure ceases to be a good metric".
In other words, once its a measure, its gamed, and useless.
So how do you prevent this? Let people tell their own story. Have metrics, no quotas. Dont compare teams. Dont compare individuals.
Have metrics, lots of them. Preferably enough thats its impossible to game all of them. Only let the full picture tell the story, never get too focused on one specific metric.
This applies to everything. Too focused on ARR? Guess what, you will make terrible deals at a discounted rate, have high churn, and bad customer support. Too focused on churn? You will hold on to customers you should let go. You have to always look at the bigger picture.
28 comments
[ 2.6 ms ] story [ 86.9 ms ] threadAlthough recently, I've noticed another reason to tread lightly with metrics: Metrics have become a highly-charged topic among developers who consume a lot of Twitter, Reddit, HN and other developer chatter. The cynical response to any metrics tracking program is to claim that they will be gamed, or abused, or misused by management. Once your team has adopted this cynical mindset, they're unlikely to see metrics any other way, so they get to work gaming and abusing the system.
In my experience, getting as much team buy-in as possible is the key to making metrics a success. It's healthy to invite the teams to an open discussion about what metrics they think should be tracked. Give them an opportunity to openly discuss objections, and make a valid effort to acknowledge and work with their suggestions as you develop your metrics program. Ideally, you would find one or two metrics that benefit the team members directly rather than the company or management.
It's important that developers can see the metrics as an open target to help them excel within the company, rather than a punitive measure or something that management can use as a substitute for real engagement. When team metrics decline, use that opportunity to open a discussion with the team. If you let metrics become a punitive measure or a guide for who gets promoted or fired, they will absolutely be gamed to death.
In my experience metrics make no difference on success, the dynamics of the team and the workflow have been set up for success or failure before you started measuring it. Metrics should never be a target, once a measure becomes a target it ceases to be a good measure.
Measures should be used to give you awareness of time and quantity. Quality as efficiency is inaccurate, it ignores the properties of the work created. Being aware of time and quantity gives managers of large empires the ability to summarize the work of many individuals into a few measures. This kills the point of individually excelling and creates the grounds for union largess. In a few years everybody realizes that the work is not created equal and you create an incentive for real quality minded people to started ranking everybody and allowing room for clearing the bottom 10%.
The same ideas in the article are pushed across youtube, across HN and the internet culture at large, in time with each other. It's disappointing to see so many smart people walk in mental lock step with one another. It devalues the individual contribution and shows there is little left for the majority of these content creators to say that hasn't been said elsewhere.
I agree that the dynamics of a dev team are extremely important. But how can one drive to get better without understanding their current status?
In my personal experience, there should be an open conversation between the team on deciding what to measure and what does the team want to improve on even if the team already has good dynamics. My moral as a developer was always up when my team got props for something, because I was a part of that success.
I think the focus needs to be on the team as a whole, instead of getting dragged into a comparisons game. The developers should be aware that this isn't an individuals race.
If you work in a large organization then you will still be measured even if you're team doesn't take the measurements on their own. Good metrics can only benefit you since you can control the conversation with your own data instead of data from a corporate figure that only looks at you as another row in their spreadsheet.
The individual seeing with metrics is not the action taking mind. The person who sees through metrics is a heart hidden behind a corporate body. The heart cannot imagine greater heights, it can only regulate flow. Blood pressure spikes get eliminated for "normality".
This lack of thought and focus on flow eliminates morals and devalues changing forms and structures in favour of a mindless gigantism. Corporations don't change individuals, they lump responsibility for failure upon me's and you's. The corporation's goal is to grow the flow at human cost. I among many would prefer not become one of Howard's humanoids (Network 1976) and would instead prefer to go out on the stage of individual achievement, if at all. We are all vulnerable to the snakes that encircle and poison hearts. Ideas go in one side of the heart and come out the other side dead.
>Cycle Time: If you can only have 1 metric, make it cycle time. Cycle time measures the amount of time from work started to work delivered; in other words, the amount of time from first commit to production release. Cycle time is important because it is a proxy for how optimized your team is.
If cycle time is being optimized for, then eventually, someone on the team will discover that if they make an "experimental prototype" on their laptop before checking it in, their cycle time goes down a lot.
There really is no substitute for an involved manager, however that does not scale, and so you go to metrics. But, I would argue, you have already damaged the culture.
In theory, yes. But in practice, I bet there are a lot of people here who have stories of a metric that was supposed to only be used team wide, but was used for evaluating an individual.
Similar with metrics, it becomes a diversion to attend to waste (extra efforts to produce quality data), while removing autonomy and opportunities.
Those metrics that do work, are owned by people themselves, and are used as tools, not as a whip, carrot or other games of diversions. As tools, the metrics can be changed or removed, as seen necessary.
As a manager there are some individual metrics that could serve as a “smell” to investigate but you shouldn’t publish those metrics or ever use them in communications or as reasoning for any decision.
Laos, as the article indicates to some extent, what you measure becomes what matters. As a manager, the metrics you create will (to an extent) help to define what kind of team you run and build.
What are they selling? Oh, project management software.
If you currently have employees that have a lot of freedom, responsibility and little oversight and who thrive on that, then introducing more formal metrics can make them feel their work is becoming less fun and they become less motivated. But other employees like clear structure and formal methods and they'd welcome such things.
Usually the company has one style of management from the start, and people who like working under that style self-selected over time. That gives you your "culture", and changing it means you may make your company's management style less of a good fit for its employees.
I did learn a curious fact about the productivity of the most prolific writers, musicians, comedians and other creative professions. Count the number of words written, notes composed and new jokes, then divide those by a reasonable number of working hours, let's say 5 days a week with 8 hours of work over the productive period in their life.
Unfortunately, I can't find the exact numbers anymore, but it was staggeringly low, something in the vein of 1 word written per hour, 1 note composed per day, and one new joke per week.
Sure there are things to be learned from data, just not always what fits the agenda of the managerial class.
This is so ridiculous as to be offensive. Bring the downvotes, I don't care.
However that’s in terms of a finished product which typically has a lot removed or edited over time.
And some things just need weeks of reflection, research, trying things out. Rome wasn't built in a day. When you'll get there you'll have produced something of actual value.
It also helps your teammates and managers identify when you are spinning your wheels and need help. As a tech lead/architect, I find prototypes and documentation about a hard problem extremely helpful for identifying when I’ve made a bad technical decision and selected an approach that simply isn’t efficient or workable. Otherwise, it’s hard to distinguish between that and when the individual dev is simply not up to the job.
It’s also much easier to solicit feedback when you’re stuck if you can offer a record of what you have tried.
My friend at Facebook says that everything is based on metrics. So people will ask others to give them a “thanks” in the Performance system for Mundane things like going to a meeting. Numbers of reviews that are commented on are also a metric, so people will +1 a review.
This happens everywhere, not just Facebook. Whatever metrics you employ will incentivize your employees to target those. If you value LOC then those will be targeted. If you value number of checkins, then those will be targeted. And that’s how it should be because that’s what you as a manager are defining to your employees is what is important and valued.
So if you do choose metrics then be aware that it will change the culture.
A measure is when you expect a specific quota or compare a metric against another.
Goodharts law, "a metric that becomes a measure ceases to be a good metric".
In other words, once its a measure, its gamed, and useless. So how do you prevent this? Let people tell their own story. Have metrics, no quotas. Dont compare teams. Dont compare individuals. Have metrics, lots of them. Preferably enough thats its impossible to game all of them. Only let the full picture tell the story, never get too focused on one specific metric.
This applies to everything. Too focused on ARR? Guess what, you will make terrible deals at a discounted rate, have high churn, and bad customer support. Too focused on churn? You will hold on to customers you should let go. You have to always look at the bigger picture.
Only metrics, no measures.