- that complexity metrics are highly correlated with LOC.
- people can produce & maintain software at about 10 LOC/h. This seems to be a constant of the mind.
It means you're better off doing 10 lines of python than 10 lines of ASM as the 10 lines of python will do a lot more.
Both facts are discussed at length in "Making Software" (Oram, Wilson)
Of course, that developer almost certainly introduced the most bugs, as well. So, maybe? :)
Though, agreed you are almost certainly better pulling back up and asking more product faced questions. The most prolific author, as an example, produced more sentences read by someone. Not just more sentences.
While I recognize the prevalence of Goodhart's law among smart thinkers, it fails to meet my "important truth" threshold because it's not hard to find counterexamples:
* In baseball, batting average and OPS are metrics that are richly rewarded. Players who successfully strive to improve their average/OPS will help their team win more, and they'll get richer for it.
* In finance, ROI is rewarded, and investors whose ROI is higher will generally be more successful
* In sales, "conversions" is a useful metric that salespeople strive toward and are validly rewarded for improving
* In software, if a metric like "story points" (as decided by group via planning poker) could be generated consistently for every issue worked on, there's no apparent reason that developers or teams would be worse off if developers used "story points completed" as a goal to strive toward
I suspect the prevalence of folks citing Goodhart's law is because it is usually true. But this may just be a consequence of the frequent disparity between "metrics available" and "desired underlying phenomenon." There's no fundamental reason that they can't be the same.
I think a better version is "you get what you measure".
Metrics fail when what they measure isn't directly related to what you want. I'd much rather have n conversions than n000 LoC, if all I get to pick is n.
(I suspect this is especially an issue when you find a metric that retroactively correlates reasonably well with something desired. For example, you find that closing a ticket in less than 20 minutes correlates with the issue getting an incomplete solution. Then you make it into a metric, at which point you're bitten by the fact the correlation no longer works: people just leave the ticket open in a tab for a few hours, ignoring it but letting the clock run, before they close it. It's not enough for a metric to historically correlate with good things, "gaming" the metric has to cause the good things as well. Obligatory relevant XKCD (the helpful comments one): https://xkcd.com/810/)
If you reward a team for increasing story points completed, then you will very quickly find that every single story has the maximum possible estimate. In the longer term you'd get stories split into more smaller stories far more often than is reasonable.
It's an incredibly obvious example of a measure that's useful only so long as there's no incentive to game it.
I saw that happen in a team whose OKR was stupidly tied to story points. They went from 20 points per sprint to 70 overnight, by splitting stories as most as possible. It was hilarious when they reached their quarter OKR in less than a month.
It wasn’t that long ago that OPS was not rewarded, and batting average was king. It took a lot of energy to change that tide and reward players for focusing on other metrics.
My point here is that Goodhart’s law still held, but the fallacy is when people assume that metrics are static.
I was just writing a post that said the same exact thing with about 12 times as many words. Good thing I read your comment before I wasted another 20 minutes writing the other half of my response
Please stick with me here, probably going to be a super long post but I'm sharing my thoughts on the whole topic end-to-end with the goal of addressing the root problem here,
I agree that such a metric exists which is the same as the desired outcome. Such a metric does, in fact, exist. The metric IS the desired outcome. The problem comes when you try to write a generic, reusable formula that describes the desired outcome and attempt to implement a system that teams can use to apply the metric to their work. It ends up being effectively impossibly complex to create the formula, and even if you could create it, it would take longer it than it would to write the software that it's supposed to measure.
What we're left with is 'the best that we can do', and that's what we have been doing. We absolutely COULD make more accurate(yet still very imperfect) metrics that more closely model the actual goals of a project by simply including more factors(bugs, readability of code, difficulty of feature, size of feature/LOC, number of modules/classes that the code spans or communicates with in some way, how easy-to-read and easy-to-modify the existing code that needs to be altered is, etc times 1,000,000). But adding more factors just makes the metric exponentially more complex and difficult to implement. I'm sure people have tried using way more fine-grained metrics than what we have now - but nobody knows about them because nobody uses them anymore because they weren't worth the time and effort.
So we're kind of left with what we have, which in my opinion as a software engineer is definitely not good enough. Which is pretty bad, because by 'good enough' I just mean better than doing nothing at all.
The only problem here that can be solved, and therefore that is worth solving, is why these metrics are still commonly enforced despite being counterproductive. Obviously managers managers-of-managers feel a need to have some nice, concrete, objective numbers that they can look at to see how well developers and teams are doing. The people pushing these metrics are often the ones who have no other way of understanding what exactly developers are doing, either because they've never written software before or because they are too far removed from the level of the guys actually writing code to pay attention to what they're actually doing beyond just looking at a single number.
It makes sense for managers who don't know what engineers are doing to seek out a different way to measure their performance and progress. The solution isn't do nothing, and it's not story points or any of that BS. But what are the alternatives?
The only one I can think of is, don't put a manager in a position to manage individual developers if he/she is unable to understand what those individuals are even doing. Having an experienced senior engineer with the right skillset for management directly in charge of every team is amazingly valuable in my opinion. It's also vital that those higher up on the ladder are able to trust these manager-engineers, who are basically acting as an abstraction layer between the regular engineers and higher management.
A lot of teams already have this in the form of a senior engineer or software architect, my team has one and it works out amazingly. We all feel that doing a good job will be recognized and rewarded, problems are easily identified and addressed, we get things done very quickly, everyone gets along, and there's not a lot of pressure.
This is just my own single anecdotal experience and it has worked out very well but I'm aware it may not always be that way, so I'm interested in hearing others thoughts/experiences/solutions
Over the last year, I have averaged 8.95 LOC* per hour as a full time* developer.
Lines of Java + TypeScript, my two most used languages, purposefully excluding configs, html/css, etc.
Assuming 8 hours of work per day; this is less than the time I spend at work, but probably more than I spend coding, due to meetings/email/overhead.
246 workdays, which is 30 days off, some of which are public holidays and the rest are vacation.
I'm a relatively junior dev.
I think that people who claim to write many lines of code per hour just don't count small amounts of time lost and don't realize it adds up or feel it should be exempt from counting. It's like the calories in broken cookies, or the ten minutes it takes to get out the door when you only spent a few seconds grabbing a few things.
One thing that I think can give you perspective is if you experience a job where you account for your time like lawyers do, in 6 or 15 minute increments. It's surprisingly hard to get your billable time over 50%, assuming you're honest about it.
I've seen some of the worst developers commit more LOCs than others (churn in badly written code, or "cut-and-paste hell"). I've also seen developers sandbagging the metric (as well as artificially inflating code coverage measurements) by checking in fluff code; a null check on every pointer or reference parameter can get you 5-10 lines of actual live and executable code that is practically guaranteed bug-free and politically correct, even for functions you haven't really started implementing yet. Saw this being used in a USB stack once, it had been worked on by serial contractors and was a real eye-opener (when we got it, it was 5K LOC or so; we whacked it down to about 1100 LOC by removing a bunch of badly-motivated layers; removing the junk sped it up quit a bit, too).
My position is that if you're evaluating people by comparing metrics, you're probably a terrible manager.
But if you measure it to determine performance, you are not paying developers to create solutions, you are paying them to create LOC. That could have quite a negative influence.
If we could channel energy from creating performance metrics into a form of reporting that continually narrows down the problems that need solving, productivity would be served better.
Otherwise we could probably use keystrokes per second just as well.
I actually think it's important for LOC to be considered as a key performance metric, because it makes it much easier for me to outsource my work to a team of these delightful little fellows:
It’s really a complicated topic - is it a new project vs. maintenance, etc. In new project my metric was around 500 LOC of C++ code (production ready) per day. In maintenance, it could be a line a day (if you’re lucky).
Well, when comparing languages, sure. 10 lines of python gets a lot more done than 10 lines of assembly, but as a metric of how good your code is or the developer is, it’s not so great. Usually 9 lines of python is better than 10 lines of python (assuming lines means code points or statements or whatever rather than physical lines, otherwise 10 lines could be better if it’s visually laid out clearer)
Does anybody know of a study documenting that measuring and reporting software code quality metrics actually improve the project/product team's ability to deliver?
So far I've only found studies documenting that measuring code quality improve the team's ability to improve the measurement figures. That doesn't count. More or less any measurement will be gamed if anyone think they are used for something. But has anybody measured if these measurements actually are helpfull?
This is a terrific question, and a topic I've thought about quite a bit while we've built our product (context: as the OP, I've toiled on "how to prove metrics are signal-bearing" for a couple years now).
The problem we always end up at is "what measurable output can we rely upon to tell the story that a project's fortunes are improving"? To date, its been challenging to find a compelling answer. This may be why I'm unaware of any study that addresses the question.
One possible workaround would be to take experts in the field, and try to translate their qualitative interpretation into input that drives a metric. e.g., if Andrew Clark from React could list three directories that harbor disproportionate tech debt in their project, we could calibrate our Open Repo Directory Browser to identify the factors that were unique to that directory. This could then help managers pick out when and where its appropriate to make payments to reduce their tech debt.
Unfortunately, the prominent open source developers aren't answering my emails at this point. And even if they did, it's still a long road to translate qualitative expert opinion into provable long-term benefits. But I think the environment for making that leap is as ripe as ever. We just need to keep working toward figuring out the most desirable+measurable outputs.
> The longest has taken two weeks and is still ongoing.
Even worse, the typical braindead “solution” to this problem is to insist that everything be broken up into chunks of a few hours or less each before work can begin. The end result is “if it can’t be done in a couple of hours, it isn’t worth doing”.
I have a solution to these metric gotchas. Don't install Jira at all and then you won't be tempted by any of them. Better yet, don't hire a manager at all. Instead get one or two senior devs and include them in some business discussions and incentives.
Whenever I see a manager reaching for a "code metric", my immediate thought is that they're so far removed from the actual engineering of the product that they don't feel like they have any feedback so they want to reach for something to feel like they do.
gasp include senior devs in a conversation about development? Blasphemy. At so many places it's like there's some unwritten law that you can't include developers in any sort of business conversation, you need to hire a project manager as a go-between in order to badly translate things between business and development so neither side actually knows what's going on.
Why is this so true. I usually assume that when a large percentage of people do something that seems dumb, it's usually because there is some reason I don't understand and not because they all coincidentally decided to be idiots in the same way. Occasionally there is a common pitfall that does cause them all to be dumb in the same way, and more often there's a good reason. In this case though, it's so common but I can't see how there could possibly be a reason for this. Maybe tech companies just tend to have really small meeting rooms with only a few chairs in them.
I think the problem is this: if you talk to any good software developer (or scientist for that matter), they're always going to speak in probabilities and hedge everything. And they're absolutely correct to do this, because it's the reality.
The problem is, the business people (generally) have no interest in this. They're interested in certainties. They aren't being irrational either -- they need to go out and sell, raise money, etc., and to do that they need to be able to say "we're shipping on december 5th" and "absolutely we can do that for you". Having the developers say "well, uh, maybe we can do that in three months but we don't really know" makes their lives 10x harder even if it's true.
So this is where project managers come into the picture. They don't promise the most efficient use of human resources. What they offer is a predictable use of human resources. In reality this drags everyone down into a very gloomy mediocrity and leads to ineffectual organizations. But to the business person, it seems great because they gain the illusion of control.
In that light, our current predicament makes perfect sense: project managers are the shamanic priests doing a rain dance, and developers are the weather. The weather stays unpredictable, but if it happens to rain the shaman can take the credit, and if the drought remains, it's because the ritual wasn't done correctly, you see. The illusion of control is the product they're selling. What the managers are buying is someone that says "yes, we can have it rain on the 19th" instead of talking to someone that says "we can predict the weather about a day out, maybe"
(God I hope no project manager I work with ends up reading this, heh).
Stupid question: can't the businessman do its job? I mean, it's not like the market is constant everywhere, they must be dealing with probabilities too.
Considering that the metrics above are still considered viable at the dawn of 2020 proves how much opportunity still remains for managers to improve the measurement -> incentives -> long-term outcomes achieved by their team. To their credit, the instinct that leads managers to utilize these metrics is sound. They know that the best businesses make their best decisions using data. But so much depends upon whether the data is any good. When you place trust in a metric like commit count, you're liable to end off worse than if you'd used no data at all (i.e., by pitting the team against one another).
Our opinion is that a less bad metric would be one that harnessed the signal lurking deep within lines of code, while rinsing away its noise. The metric would consider "issues resolved," but only after normalizing for implementation complexity, so developers that work in front-end systems don't take the lion's share of credit after resolving a myriad of tiny tickets.
>> The metric would consider "issues resolved," but only after normalizing for implementation complexity, so developers that work in front-end systems don't take the lion's share of credit after resolving a myriad of tiny tickets.
Only if you are not being rewarded for solving your own bugs. Otherwise, I've seen this go horribly awry.
I was at a Big 5 consulting company where one project was big on "how well you managed issues and brought them to resolution". That meant that good developers who prioritized code coverage and mostly bug-free code did not have good issue resolution case studies to cite during promotion cycles. After a year of good developers getting dinged on this, some good developers banded together to purposefully leave known bugs in the release (especially bugs that could be blamed on vague specs -- so instead of trying and resolve issues during dev cycles, they were propagated to production). Then, the developer would swoop in and do lots of "investigation", produce a fancy write-up and email (carefully cc'ing senior management) and throw the business analyst under the bus.
Senior Management naturally was impressed and some of the developers got promoted for their "leadership" and management of "critical bugs."
I worked at a healthcare company where the chief architect was considered a genius by upper management because he would dive in and "save" production; in reality he was the cause of all the bugs that broke it but he did not allow anyone to fix the problems.
Is this operating under the assumption that there is no code review?
With code review, lines of code and commit count become much harder to game. If you try to submit changes with unnecessary whitespace, unnecessary code constructs, or incomplete code, your reviewer will see it and tell you to fix it. Ultimately, trying to increase your lines of code or commit count will come back to haunt you as you spend much more time addressing comments from code review.
It's still super-easy to game. Just prioritize all the low-hanging fruit. Got some tickets for increasing the font size or replacing the copy in the privacy policy? Why not do those instead of working on the important but hard to debug/estimate problem of some database transactions being slow, but only sometimes?
At a previous job of mine, they implemented a "personal improvement plan" for employees who were below management's "desired productivity". There was no warning, no heads up that they would be measuring our GitHub accounts, and it had absolutely no context of what each person had been working on. These reviews came after a 360 peer review, and absolutely blindsided the team. Several of my coworkers left early for the day and didn't come in for the rest of the week. I got calls all day from various people asking what happened in my 1:1, what was said, etc...
I wish I could show management how negatively and immediately that move affected the culture & mood of engineering at that company. It was an overnight change, and going forward it basically killed all trust in management. I stayed around for a little bit longer, but I should have quit the second they started those measurements.
It absolutely murdered the confidence of our junior devs. Of the senior engineers who were put on it, none of them really had any idea what they were doing wrong or how to improve, as no feedback was given about any of that, just that they weren't enough. Some of our seniors had 15+ years of experience in software and were industry experts, but it didn't matter: They hadn't been hitting the number of commits and pull requests and tickets per week that management insisted on.
When management was confronted about it, they admitted that code metrics aren't a good measurement of productivity, but then insisted that they "needed something".
It turns out, they only used code metrics as a bludgeon to get rid of the employees they wanted gone, and since then that's all I've been able to see code metrics as. They're a battering weapon used to give HR a performance reason to justify firing you.
I'm sorry this experience left you scared of metrics but it doesn't mean they are bad, just that people need to know how to use them.
I've had this exact conversation before and the extremism of "metrics are always weapons" was a contributing factor for me to leave a, say, less metrics-inclined workplace. Just saying there has to be balance.
I've been a professional software developer for twelve years, and while I'm trying to keep an open mind, I have yet to see any useful metrics for what we do. Like, useful AT ALL. I can think of zero numerical metrics that can accurately tell you what's going on with a team. Number of commits, burndown charts, velocity, issues closed/created, whatever, they're all simultaneously easily game-able and also create bad incentives.
And by the way this isn't sour grapes, metrics have almost always made me look good. One job I left, I had so many more commits than the next person that it took them two years after I left to catch up to me. And yet I'm absolutely certain I was not the most useful person on that team. The most useful person was the guy that was writing the extremely sophisticated modelling kernel, and if you only looked at the metrics he probably looked borderline useless, but in actual terms he was by far the most valuable and important person on our team.
What scares programmers is, everyone on our team knew that, and any sort of manager with programming experience probably knows that, but you know that those stats are going to eventually float up to someone who doesn't know that the numbers are bogus. (After all, they wouldn't be tracking these numbers if they knew how bad they are). So decisions do get made from these numbers, even when people promise they wont.
I'm completely with you on that. As for an example where some metrics might be useful: some form of complexity measurement of the code helps a new developer on the team discover where to poke to find out issues, as well as which parts are critical.
While far from perfect, it gives you a starting point that's possibly better than "randomly open files in project".
Now, this comes with interesting points:
- the metrics in this case are in the hands of the professional in the field
- the metrics are there to serve the person itself, without affecting others
Metrics on commits, line of code, tickets closed are useless from my point of view.
If you ask in a developer team who is weak at the moment (assuming there is no room for improvement is stupid) gives better results. A decent programmer is able to spot fake ones and is able to notice performance difference over time (not perfectly, but definitely better than metrics).
Not to mention, I would be surprised if none of the team would start complaining if someone is doing no work and still getting salary.
You can spend an entire day teaching your team something important about the project you are working on, have no commits about it.
Or you can use half the day walking around thinking how to solve a bug.
Not a single line is written, but that's how some problems are solved.
I started counting in my working hours even when sometimes I wake up in the middle of the night and my mind solves programming problems for 1 hour (let's say). I have a decent employee, being at my desk typing is not what I'm paid for, I'm paid for solving problems with software engineering to the best of my knowledge.
I’ve been a professional programmer for a while and worked at a variety of companies. When the teams I’ve been on decided to implement tons of metrics, it’s usually because we had tons of code quality problems. This was usually because we had inexperienced or inattentive programmers wreaking havoc.
I currently work at a company where all the devs are pretty good and the code is decent. We don’t really need much in the way of metrics. If anything, we have more metrics than we should. I trust my current team to do the right thing more than I trust a linter.
I’m a manager of ~25 developers (who report through a few leads). I also come from a dev background. IMO metrics are dangerous, but can also be useful. Often I find basic code metrics useful as smell tests, but would never use them alone, wouldn’t set them as goals, and wouldn’t use them alone to rank performance in a team.
If someone has a very low number of commits relative to others on their team, I’ll investigate deeper - review a few Pull Requests, check if they’re working on something outside source control etc. sometimes I find problems, other times not.
I would think this is a common approach. I use the same approach. Scales up to hundreds of developers.
I do generally try to enforce the goal that on any working day there should be at least one commit from the developer. If I notice a developer is consistently missing this metric it usually indicates trouble in the project they are working on or that the developer is just not delivering enough.
Ugh the "velocity" one really hits home. A couple of jobs ago one of our project managers was constantly showing burn down charts (which, of course, never burned down); along with our "velocity". The funny thing is, because we were doing t-shirt-size/fibonacci, our velocity numbers varied so wildly that it was essentially a random number generator. Then we were required fit all our stories within a "budget" of our "average" velocity.. What are you supposed to do with the average of a random number between 0 and 200? Apparently a lot, because these insane numbers that had no actual tie to reality were used to determine how much we were allowed to "commit" to. Never mind that some months we'd finish 50 story points and some months 250 (because, again, the numbers were a fiction); we were always trying to fit things into 90 story points.
Luckily all this useless theater only took up an entire day every two weeks. If you assume everyone on the team was making about $50 an hour, and it was about 6-8 people in attendance to these things, then it came out that this very dumb "estimation" process took 6 people * $50/hr * 6 hours... so about 1800 dollars. (Not including the project managers salary, which given that he drove around a very fancy luxury car, was probably a lot more than the rest of us).
And people wonder why I don't take scrum seriously.
To be fair, that's very poorly implemented. You should have reference stories for what is a 1 in complexity and the highest value should be low, like 13, which forces you to break things down and forces the PM to write better specs.
We did all of these things. But we had this ridiculous Fibonacci thing where you would end up with a conversation like this:
dev: well, this looks like it's somewhere between a 5 and an 8, so let's call it 6.
pm: no 6 is not allowed, it must be either a 5 or an 8.
dev: ok, lets call it an 8.
Ok so that's a very plausible conversation, right? I'm sure we've all had it. But just that rule alone made the estimate on this story 20% off, just to satisfy a rule. I'm perfectly fine with that, if we're only talking about that one story. If we're both happy that this might go 20% faster or 20% slower than predicted, no big deal. But if you're going to include it in a larger rollup, you've included a very large error term with your number that you need to carry along with the rest of the calculations. Did you ever take a physics course in college? (A real one -- not one of the baby ones they gave to business and liberal art majors). If they were good, they probably made you deal with carrying an error term in your math. You probably even hated it because it was a pain in the ass to deal with, but it's also kind of important. If you're adding together 50 numbers that are all off by 20%-50% and chucking away the error term, your summary and conclusion are absolute garbage if you do not properly keep the error term included. Math is unforgiving like that. Presenting that absolute garbage as "data" is, in my opinion, professional incompetence. It just is. I guess the softening factor is that so many people do it that it's not an individual incompetence, it's a lack of competence in the profession itself essentially.
Well yeah, you don't have to use fib numbers. Not really sure why that's popular anyway.
Personally I prefer being on a small team with excellent devs that don't require this kind of stuff. I built products that made many millions of dollars a year with just me, a PM, and one other dev with none of this stuff.
In order to add value to a dev team as a manager, one must have an intimate understanding of the work developers do. There isn't any value in abstracting away from that work when evaluating the contribution of team members.
"Context, action, outcome" is a sufficiently standardized formalism for evaluating performance (assuming outcomes are measured in terms of business value). Accurately reporting context, action, outcome requires understanding both what devs do, why they do it, and how it ultimately produces business value. There's no shortcut. If your management can't meaningfully do that because they are non technical or managing too many people, or whatever, then they are actively harming your team and should be removed with prejudice.
There is no reasonable way to conclude that devs can be effectively measured with garbage stats. Even suggesting it seriously indicates incompetence or laziness.
Is the measure of an architect the number of lines in a blueprint? Is the measure of a craftsman the number of finished goods produced? It seems to me that different projects in the same domain should use different attributes to evaluate success. Why do we attempt to use uniform measures across-the-board? Is it lazy management or does delving deep into project-specific metrics make it hard for large enterprises to compare employees in similar functions? Or something else?
Why not let developers justify their work and point out its positive impact themselves? It would require some overhead, but they could highlight the appropriate metric if it can be quantified and also justify more qualitative improvements.
There’s a tendency for “metrics” to evolve into a process where the police give you speeding tickets based on the number of times you opened the passenger-side windows of your car. Oh, and if you get a flat tire it must mean that you should lose your license.
Also, beware of pretty graph summaries that are presented without a healthy list of caveats and follow-ups and further explanations. Beware if no one with management power is asking the follow-up questions that matter.
Author concludes (before promoting his product) "The metric would consider "issues resolved," but only after normalizing for implementation complexity", so story points then?
The 'best' software productivity metric I've ever seen is one I created and tested on a large code base. It only measures coding, but it's better than the other metrics for that. It is based on information theory and credits code removed equally to code added. In the end, I reported the results to the devs and then stopped doing it because there was too much opportunity for abuse by my superiors. Once they got wind of it, they wanted to know everyone's scores and I could see where it was heading.
The metric: It was based on a very powerful (and slow) compression algorithm such that it could measure the change in code. Since it used compression, whitespace, boiler plate, etc had essentially no impact. It was carefully tested by comparing the results against human assessments.
The only clear issue was that it did over-credit mass changes (renaming a root package in a java project which caused all files to be changed). This could probably have been handled by adjusting the scores.
After looking at other metrics (bugs reported, comments on bug reports, code reviews completed, comments in code reviews, etc) it was clear that some people made _huge_ contributions through activities that were ignored by the algorithm, and some people who had great software productivity scores contributed little in these other activities. They were putting all their time into their own code.
FWIW, In the same exercise, I came up with another good metric to identify problematic code (rather than problematic people). That metric used machine learning to find code that needed to be refactored. But that's a story for another time.
The only telltale metric of software work that I know is the ratio of time spent making progress to time spent not being able to make progress because of past mistakes made and shortcuts taken.
It's reasonable to consider that the reason that software developers are saddled with such shallow process and tooling metrics is that they themselves don't have any better alternatives to offer.
If we roll over and accept the lowest possible maturity organization and process methodologies like Scrum, then we ultimately deserve what we get.
We have a roll to play, and it ultimately requires stepping up and seeing our jobs as more than mere tool-wielding. Or said otherwise, in the rush to call ourselves "engineers" we've stopped being engineers and stopped even pursuing an understanding of the body of knowledge of engineering (which leads to better grasp of the subject area that comprises process and measurement).
Velocity is a standard we're held to because we didn't step up to provide better standards. And we didn't do that because we didn't believe it was our responsibility to learn about the rest of software development. And in that vacuum, people who know even less than us are left to dictate pseudo science to people who should already be in command of the science.
We have a role to play if we wish to play it. It's not obligatory, but there are real costs to pay when we don't.
There's no requirement or expectation to go above and beyond in an effort to catch up on all the knowledge and understanding we've shed from our wheelhouse. It's totally fine and understandable if we don't. But we also have to accept that nature abhors a vacuum.
67 comments
[ 3.0 ms ] story [ 129 ms ] threadBoth facts are discussed at length in "Making Software" (Oram, Wilson)
>Thus, the "most valuable developer" as measured by LoC tends to be whoever is adding the most CSS, whitespace, and third-party libraries.
It's still as terrible as ever to positively correlate developer productivity/quality/skill with LOC.
Though, agreed you are almost certainly better pulling back up and asking more product faced questions. The most prolific author, as an example, produced more sentences read by someone. Not just more sentences.
* In baseball, batting average and OPS are metrics that are richly rewarded. Players who successfully strive to improve their average/OPS will help their team win more, and they'll get richer for it.
* In finance, ROI is rewarded, and investors whose ROI is higher will generally be more successful
* In sales, "conversions" is a useful metric that salespeople strive toward and are validly rewarded for improving
* In software, if a metric like "story points" (as decided by group via planning poker) could be generated consistently for every issue worked on, there's no apparent reason that developers or teams would be worse off if developers used "story points completed" as a goal to strive toward
I suspect the prevalence of folks citing Goodhart's law is because it is usually true. But this may just be a consequence of the frequent disparity between "metrics available" and "desired underlying phenomenon." There's no fundamental reason that they can't be the same.
Metrics fail when what they measure isn't directly related to what you want. I'd much rather have n conversions than n000 LoC, if all I get to pick is n.
(I suspect this is especially an issue when you find a metric that retroactively correlates reasonably well with something desired. For example, you find that closing a ticket in less than 20 minutes correlates with the issue getting an incomplete solution. Then you make it into a metric, at which point you're bitten by the fact the correlation no longer works: people just leave the ticket open in a tab for a few hours, ignoring it but letting the clock run, before they close it. It's not enough for a metric to historically correlate with good things, "gaming" the metric has to cause the good things as well. Obligatory relevant XKCD (the helpful comments one): https://xkcd.com/810/)
It's an incredibly obvious example of a measure that's useful only so long as there's no incentive to game it.
My point here is that Goodhart’s law still held, but the fallacy is when people assume that metrics are static.
* In baseball, batting average and OPS require you to compete against other teams.
* In finance, ROI is tested in the marketplace.
* In sales, conversions require you to convince a skeptical customer.
The software metrics are all things we have fairly direct control over.
Maybe that suggests an amendment to Godhart's law:
> When a measure that is controlled by the measuree becomes a target, it ceases to be a good measure.
I agree that such a metric exists which is the same as the desired outcome. Such a metric does, in fact, exist. The metric IS the desired outcome. The problem comes when you try to write a generic, reusable formula that describes the desired outcome and attempt to implement a system that teams can use to apply the metric to their work. It ends up being effectively impossibly complex to create the formula, and even if you could create it, it would take longer it than it would to write the software that it's supposed to measure.
What we're left with is 'the best that we can do', and that's what we have been doing. We absolutely COULD make more accurate(yet still very imperfect) metrics that more closely model the actual goals of a project by simply including more factors(bugs, readability of code, difficulty of feature, size of feature/LOC, number of modules/classes that the code spans or communicates with in some way, how easy-to-read and easy-to-modify the existing code that needs to be altered is, etc times 1,000,000). But adding more factors just makes the metric exponentially more complex and difficult to implement. I'm sure people have tried using way more fine-grained metrics than what we have now - but nobody knows about them because nobody uses them anymore because they weren't worth the time and effort.
So we're kind of left with what we have, which in my opinion as a software engineer is definitely not good enough. Which is pretty bad, because by 'good enough' I just mean better than doing nothing at all.
The only problem here that can be solved, and therefore that is worth solving, is why these metrics are still commonly enforced despite being counterproductive. Obviously managers managers-of-managers feel a need to have some nice, concrete, objective numbers that they can look at to see how well developers and teams are doing. The people pushing these metrics are often the ones who have no other way of understanding what exactly developers are doing, either because they've never written software before or because they are too far removed from the level of the guys actually writing code to pay attention to what they're actually doing beyond just looking at a single number.
It makes sense for managers who don't know what engineers are doing to seek out a different way to measure their performance and progress. The solution isn't do nothing, and it's not story points or any of that BS. But what are the alternatives?
The only one I can think of is, don't put a manager in a position to manage individual developers if he/she is unable to understand what those individuals are even doing. Having an experienced senior engineer with the right skillset for management directly in charge of every team is amazingly valuable in my opinion. It's also vital that those higher up on the ladder are able to trust these manager-engineers, who are basically acting as an abstraction layer between the regular engineers and higher management.
A lot of teams already have this in the form of a senior engineer or software architect, my team has one and it works out amazingly. We all feel that doing a good job will be recognized and rewarded, problems are easily identified and addressed, we get things done very quickly, everyone gets along, and there's not a lot of pressure.
This is just my own single anecdotal experience and it has worked out very well but I'm aware it may not always be that way, so I'm interested in hearing others thoughts/experiences/solutions
Managers are engineers of towering technical competence.
That is shocking and horrifying if that's the norm for software development.
Lines of Java + TypeScript, my two most used languages, purposefully excluding configs, html/css, etc. Assuming 8 hours of work per day; this is less than the time I spend at work, but probably more than I spend coding, due to meetings/email/overhead. 246 workdays, which is 30 days off, some of which are public holidays and the rest are vacation. I'm a relatively junior dev.
One thing that I think can give you perspective is if you experience a job where you account for your time like lawyers do, in 6 or 15 minute increments. It's surprisingly hard to get your billable time over 50%, assuming you're honest about it.
My position is that if you're evaluating people by comparing metrics, you're probably a terrible manager.
If we could channel energy from creating performance metrics into a form of reporting that continually narrows down the problems that need solving, productivity would be served better.
Otherwise we could probably use keystrokes per second just as well.
https://en.wikipedia.org/wiki/Drinking_bird
So far I've only found studies documenting that measuring code quality improve the team's ability to improve the measurement figures. That doesn't count. More or less any measurement will be gamed if anyone think they are used for something. But has anybody measured if these measurements actually are helpfull?
The problem we always end up at is "what measurable output can we rely upon to tell the story that a project's fortunes are improving"? To date, its been challenging to find a compelling answer. This may be why I'm unaware of any study that addresses the question.
One possible workaround would be to take experts in the field, and try to translate their qualitative interpretation into input that drives a metric. e.g., if Andrew Clark from React could list three directories that harbor disproportionate tech debt in their project, we could calibrate our Open Repo Directory Browser to identify the factors that were unique to that directory. This could then help managers pick out when and where its appropriate to make payments to reduce their tech debt.
Unfortunately, the prominent open source developers aren't answering my emails at this point. And even if they did, it's still a long road to translate qualitative expert opinion into provable long-term benefits. But I think the environment for making that leap is as ripe as ever. We just need to keep working toward figuring out the most desirable+measurable outputs.
https://blog.fefe.de
Even worse, the typical braindead “solution” to this problem is to insist that everything be broken up into chunks of a few hours or less each before work can begin. The end result is “if it can’t be done in a couple of hours, it isn’t worth doing”.
The metrics are all an effort to abstract developer output into something a manager with no/minimal domain experience can understand.
It feels very similar to the measurement/quantification problems distorting academic output.
Whenever I see a manager reaching for a "code metric", my immediate thought is that they're so far removed from the actual engineering of the product that they don't feel like they have any feedback so they want to reach for something to feel like they do.
The problem is, the business people (generally) have no interest in this. They're interested in certainties. They aren't being irrational either -- they need to go out and sell, raise money, etc., and to do that they need to be able to say "we're shipping on december 5th" and "absolutely we can do that for you". Having the developers say "well, uh, maybe we can do that in three months but we don't really know" makes their lives 10x harder even if it's true.
So this is where project managers come into the picture. They don't promise the most efficient use of human resources. What they offer is a predictable use of human resources. In reality this drags everyone down into a very gloomy mediocrity and leads to ineffectual organizations. But to the business person, it seems great because they gain the illusion of control.
In that light, our current predicament makes perfect sense: project managers are the shamanic priests doing a rain dance, and developers are the weather. The weather stays unpredictable, but if it happens to rain the shaman can take the credit, and if the drought remains, it's because the ritual wasn't done correctly, you see. The illusion of control is the product they're selling. What the managers are buying is someone that says "yes, we can have it rain on the 19th" instead of talking to someone that says "we can predict the weather about a day out, maybe"
(God I hope no project manager I work with ends up reading this, heh).
Considering that the metrics above are still considered viable at the dawn of 2020 proves how much opportunity still remains for managers to improve the measurement -> incentives -> long-term outcomes achieved by their team. To their credit, the instinct that leads managers to utilize these metrics is sound. They know that the best businesses make their best decisions using data. But so much depends upon whether the data is any good. When you place trust in a metric like commit count, you're liable to end off worse than if you'd used no data at all (i.e., by pitting the team against one another).
Our opinion is that a less bad metric would be one that harnessed the signal lurking deep within lines of code, while rinsing away its noise. The metric would consider "issues resolved," but only after normalizing for implementation complexity, so developers that work in front-end systems don't take the lion's share of credit after resolving a myriad of tiny tickets.
Only if you are not being rewarded for solving your own bugs. Otherwise, I've seen this go horribly awry.
I was at a Big 5 consulting company where one project was big on "how well you managed issues and brought them to resolution". That meant that good developers who prioritized code coverage and mostly bug-free code did not have good issue resolution case studies to cite during promotion cycles. After a year of good developers getting dinged on this, some good developers banded together to purposefully leave known bugs in the release (especially bugs that could be blamed on vague specs -- so instead of trying and resolve issues during dev cycles, they were propagated to production). Then, the developer would swoop in and do lots of "investigation", produce a fancy write-up and email (carefully cc'ing senior management) and throw the business analyst under the bus.
Senior Management naturally was impressed and some of the developers got promoted for their "leadership" and management of "critical bugs."
Bad Incentives --> Bad Outcomes
I worked at a healthcare company where the chief architect was considered a genius by upper management because he would dive in and "save" production; in reality he was the cause of all the bugs that broke it but he did not allow anyone to fix the problems.
I completely don't get this weird iframe for the entire article thing.
With code review, lines of code and commit count become much harder to game. If you try to submit changes with unnecessary whitespace, unnecessary code constructs, or incomplete code, your reviewer will see it and tell you to fix it. Ultimately, trying to increase your lines of code or commit count will come back to haunt you as you spend much more time addressing comments from code review.
I wish I could show management how negatively and immediately that move affected the culture & mood of engineering at that company. It was an overnight change, and going forward it basically killed all trust in management. I stayed around for a little bit longer, but I should have quit the second they started those measurements.
It absolutely murdered the confidence of our junior devs. Of the senior engineers who were put on it, none of them really had any idea what they were doing wrong or how to improve, as no feedback was given about any of that, just that they weren't enough. Some of our seniors had 15+ years of experience in software and were industry experts, but it didn't matter: They hadn't been hitting the number of commits and pull requests and tickets per week that management insisted on.
When management was confronted about it, they admitted that code metrics aren't a good measurement of productivity, but then insisted that they "needed something".
It turns out, they only used code metrics as a bludgeon to get rid of the employees they wanted gone, and since then that's all I've been able to see code metrics as. They're a battering weapon used to give HR a performance reason to justify firing you.
I've had this exact conversation before and the extremism of "metrics are always weapons" was a contributing factor for me to leave a, say, less metrics-inclined workplace. Just saying there has to be balance.
And by the way this isn't sour grapes, metrics have almost always made me look good. One job I left, I had so many more commits than the next person that it took them two years after I left to catch up to me. And yet I'm absolutely certain I was not the most useful person on that team. The most useful person was the guy that was writing the extremely sophisticated modelling kernel, and if you only looked at the metrics he probably looked borderline useless, but in actual terms he was by far the most valuable and important person on our team.
What scares programmers is, everyone on our team knew that, and any sort of manager with programming experience probably knows that, but you know that those stats are going to eventually float up to someone who doesn't know that the numbers are bogus. (After all, they wouldn't be tracking these numbers if they knew how bad they are). So decisions do get made from these numbers, even when people promise they wont.
Now, this comes with interesting points: - the metrics in this case are in the hands of the professional in the field - the metrics are there to serve the person itself, without affecting others
Metrics on commits, line of code, tickets closed are useless from my point of view. If you ask in a developer team who is weak at the moment (assuming there is no room for improvement is stupid) gives better results. A decent programmer is able to spot fake ones and is able to notice performance difference over time (not perfectly, but definitely better than metrics).
Not to mention, I would be surprised if none of the team would start complaining if someone is doing no work and still getting salary.
You can spend an entire day teaching your team something important about the project you are working on, have no commits about it. Or you can use half the day walking around thinking how to solve a bug. Not a single line is written, but that's how some problems are solved.
I started counting in my working hours even when sometimes I wake up in the middle of the night and my mind solves programming problems for 1 hour (let's say). I have a decent employee, being at my desk typing is not what I'm paid for, I'm paid for solving problems with software engineering to the best of my knowledge.
I currently work at a company where all the devs are pretty good and the code is decent. We don’t really need much in the way of metrics. If anything, we have more metrics than we should. I trust my current team to do the right thing more than I trust a linter.
If someone has a very low number of commits relative to others on their team, I’ll investigate deeper - review a few Pull Requests, check if they’re working on something outside source control etc. sometimes I find problems, other times not.
I’m curious what people think of this approach?
I do generally try to enforce the goal that on any working day there should be at least one commit from the developer. If I notice a developer is consistently missing this metric it usually indicates trouble in the project they are working on or that the developer is just not delivering enough.
Luckily all this useless theater only took up an entire day every two weeks. If you assume everyone on the team was making about $50 an hour, and it was about 6-8 people in attendance to these things, then it came out that this very dumb "estimation" process took 6 people * $50/hr * 6 hours... so about 1800 dollars. (Not including the project managers salary, which given that he drove around a very fancy luxury car, was probably a lot more than the rest of us).
And people wonder why I don't take scrum seriously.
dev: well, this looks like it's somewhere between a 5 and an 8, so let's call it 6.
pm: no 6 is not allowed, it must be either a 5 or an 8.
dev: ok, lets call it an 8.
Ok so that's a very plausible conversation, right? I'm sure we've all had it. But just that rule alone made the estimate on this story 20% off, just to satisfy a rule. I'm perfectly fine with that, if we're only talking about that one story. If we're both happy that this might go 20% faster or 20% slower than predicted, no big deal. But if you're going to include it in a larger rollup, you've included a very large error term with your number that you need to carry along with the rest of the calculations. Did you ever take a physics course in college? (A real one -- not one of the baby ones they gave to business and liberal art majors). If they were good, they probably made you deal with carrying an error term in your math. You probably even hated it because it was a pain in the ass to deal with, but it's also kind of important. If you're adding together 50 numbers that are all off by 20%-50% and chucking away the error term, your summary and conclusion are absolute garbage if you do not properly keep the error term included. Math is unforgiving like that. Presenting that absolute garbage as "data" is, in my opinion, professional incompetence. It just is. I guess the softening factor is that so many people do it that it's not an individual incompetence, it's a lack of competence in the profession itself essentially.
Personally I prefer being on a small team with excellent devs that don't require this kind of stuff. I built products that made many millions of dollars a year with just me, a PM, and one other dev with none of this stuff.
"Context, action, outcome" is a sufficiently standardized formalism for evaluating performance (assuming outcomes are measured in terms of business value). Accurately reporting context, action, outcome requires understanding both what devs do, why they do it, and how it ultimately produces business value. There's no shortcut. If your management can't meaningfully do that because they are non technical or managing too many people, or whatever, then they are actively harming your team and should be removed with prejudice.
There is no reasonable way to conclude that devs can be effectively measured with garbage stats. Even suggesting it seriously indicates incompetence or laziness.
Also, beware of pretty graph summaries that are presented without a healthy list of caveats and follow-ups and further explanations. Beware if no one with management power is asking the follow-up questions that matter.
The metric: It was based on a very powerful (and slow) compression algorithm such that it could measure the change in code. Since it used compression, whitespace, boiler plate, etc had essentially no impact. It was carefully tested by comparing the results against human assessments.
The only clear issue was that it did over-credit mass changes (renaming a root package in a java project which caused all files to be changed). This could probably have been handled by adjusting the scores.
After looking at other metrics (bugs reported, comments on bug reports, code reviews completed, comments in code reviews, etc) it was clear that some people made _huge_ contributions through activities that were ignored by the algorithm, and some people who had great software productivity scores contributed little in these other activities. They were putting all their time into their own code.
FWIW, In the same exercise, I came up with another good metric to identify problematic code (rather than problematic people). That metric used machine learning to find code that needed to be refactored. But that's a story for another time.
It's reasonable to consider that the reason that software developers are saddled with such shallow process and tooling metrics is that they themselves don't have any better alternatives to offer.
If we roll over and accept the lowest possible maturity organization and process methodologies like Scrum, then we ultimately deserve what we get.
We have a roll to play, and it ultimately requires stepping up and seeing our jobs as more than mere tool-wielding. Or said otherwise, in the rush to call ourselves "engineers" we've stopped being engineers and stopped even pursuing an understanding of the body of knowledge of engineering (which leads to better grasp of the subject area that comprises process and measurement).
Velocity is a standard we're held to because we didn't step up to provide better standards. And we didn't do that because we didn't believe it was our responsibility to learn about the rest of software development. And in that vacuum, people who know even less than us are left to dictate pseudo science to people who should already be in command of the science.
We have a role to play if we wish to play it. It's not obligatory, but there are real costs to pay when we don't.
There's no requirement or expectation to go above and beyond in an effort to catch up on all the knowledge and understanding we've shed from our wheelhouse. It's totally fine and understandable if we don't. But we also have to accept that nature abhors a vacuum.