HN Explorer

Ask HN: Developer performance metrics?

6 points by canterburry ↗ HN
Does anyone know of, or think, a system for measuring useful developer performance metrics?

I was thinking of something which is purely data driven, connects to build, source control, code quality metrics reports etc and builds a picture of each developer on the team based on their contributions, breaks, fixes, # of languages worked on etc?

Do you think it's even possible to build a good picture of a developer's performance based on data driven metrics?

18 comments

[ 3.1 ms ] story [ 57.7 ms ] thread
allanmacgregor
Going through the same process myself, short answer there is no easy metric that you can add that really quantifies a developer.

I would start measuring projects or teams first, getting a solid set of metrics for each and once you have that you can start tracking down individual developers.

Even so tracking developer productivity can be hard, a good chunk of our work happens in our head. Who is more valuable the developer that wrote thousands of lines of code or the one that changed one but fixed a critical bug in production?

I highly recommended this book http://shop.oreilly.com/product/0636920020134.do

ctoestreich
Moreover, "bugs" can linger around in code for a long time and you can't really track that metric. Code changing over time in a repository could be to fix bugs or refactor/rewrite etc. I think the original posters question was more around whether there are existing resources to track these things to which the answer I feel is no. Combining blogs posts, stack exchange score, speaking topics and other real world things can give you some semblance of a developers engagement with the development community but doesn't translate one-to-one on their actual productivity levels on a team. Although these things usually indicate a greater potential for success.
ramtatatam
I'm facing the same problem - need to measure performance for each member of my team and indeed no easy solution here.

Currently I'm measuring:

- How many tickets have been closed by each developer (we use mantis)

- What is the (average, average of 5 best, average of 5 worst) time for solving tickets

- For each average from above I obtain number of lines changed and also calculate average (mantis connected with git)

So for given period of time I know:

- how many tickets were closed by each dev

- in average - best time, worst time and usual time for solving tickets

- in average - amount of lines of code for worst time tickets, best time tickets and usual time tickets

I then look if lines representing average score are closer to lower or upper averages.

However all above is just to provide some orientation, my team is still small enough so I know each developer myself and also I'm involved in solving most issues so I base on that knowledge.

joshschreuder
Sometimes I spend all day on one ticket to find a one line solution and sometimes I can close one in 10 minutes with a 20 line solution.

I get that you want averages but even between senior and junior devs it seems like this would be massively weighted towards devs who don't pay attention or bother to look for a 'right' solution and instead go for the quickest

Is tickets closed a good metric?

ramtatatam
As mentioned I use this as a support indicator since I work close with my team and I know them quite well (so I'm usualy aware of such situations you highlighted). This indicator was to help me to identify potential problems rather than to make decisions about rises :-)
gus_massa
I'm not sure about tickets, but I grade a lot of math tests. In some cases, there are a lot of easy to grade tests (almost empty, straightforward solutions, easy to read fonts, ...) and a few very difficult to understand cases. Some of the graders "specializes" in difficult to grade papers, they read very few, but the contribution is very important.

Not all bugs/tickets are created equal. Just counting them would incentivize to cherrypick the easy ones.

ramtatatam
That's why number of tickets is only part of the story. The time and lines of code add additional insights. Anyways this was never meant to be "precise" indicator - the goal here was to have some sort of analysis that is easy and fast to come up with.
sheepmullet
> need to measure performance for each member of my team and indeed no easy solution here.

Why do you need a hard number? If you work on a <10 person team then you should already have a good idea of how everyone is performing.

I worked at one place where our bonuses were based on tickets closed and it lead to so many problems. We would get support/qa to create multiple bugs for a single issue and we then would do different fixes for each one of them even though they all really stemmed from the same core issue. A lot of bugs came in over night so a couple of us would log into the bug tracker from home and assign ourselves all the easy bugs. Devs, when they got half way through a bug and realised it was going to be more difficult than they thought would just put it back, or leave it half finished. Etc. Be very careful what you wish for :)

ramtatatam
And that's precisely why number of tickets is just part of the story :-) Not sure if you have taken your time to read entire post. And indeed - I know my team and as pointed this simple indicator helps me find out if there may be problems like burnouts or if tasks rotation is right but what would you suggest to use to all those folks who are building new team?

And just little comment on your first suggestion - you need to have some data supporting your story when you go to see the financial exec and negotiate rises. Your subjective opinion may mean a lot to you, but in many cases may be not enough when you want to do money talk.

sheepmullet
> And that's precisely why number of tickets is just part of the story

Hopefully one you are wise enough to keep to yourself :). I've seen so many problems even with the use of relatively sane metrics.

> what would you suggest to use to all those folks who are building new team?

Management, not metrics. At best you can use metrics as a rule of thumb to give you early indicators of possible issues.

For a new team though you need to work together with them for a while to learn about their performance and capabilities. There is no other way. Perhaps one guy in your team is thoughtful and methodical and takes a week per issue but can solve complex issues, and another girl in your team can fly through half a dozen simpler issues in a week. That's actually the kind of diversity you want to encourage and it's something you can easily destroy if you consider #tickets closed an important metric.

You should never mention metrics as a measure of performance with either your subordinates or your superiors. Your subordinates will work to the metrics rather than do what needs doing. And your superiors will start relying on the metrics instead of on your judgment.

For example if you collect and use the #tickets solved per month as a metric then if you need to downsize your manager will say lets get rid of the people solving less than x tickets per month. Oh wait that includes some of your best people! Now you have an uphill battle, especially if you have used the #tickets closed as a reason to promote someone in your team.

Edit: Basically the potential gains are minimal and the potential issues are numerous.

sheepmullet
> you need to have some data supporting your story when you go to see the financial exec and negotiate rises.

Dodgy metrics probably won't convince your exec either.

If you are consistently building your team members capabilities then convincing the exec is easy.

For example you can say Ben has solved complex cases A, B, and C in the last six months, and the work is clearly at a senior level but his pay doesn't reflect it. Lets fix it.

Or: John has taken on supervision and mentoring tasks for Sam, Ben, and Steve, while still performing at a high level. He has mentioned to me in our one on ones that he wants to take on more leadership tasks, and we have project Y coming up. I think we should put him in charge of the new project.

Etc etc. You don't need silly metrics.

ramtatatam
I think you missed the last sentence of my post.
weee_username
When I'm measuring developer performance I look at one stat: Do they make reasonable deadlines without generating a ton of bugs?

Why is this? Well, it's because the amount of tickets, code, or even time spent "logged in" doesn't matter. The only thing that matters to a business is if they can deliver on time so the product can keep moving forward and others aren't waiting for them.

ramtatatam
So you assume your devs are senior enough to come up with achievable deadlines even when they are under pressure. Such methodology will work in favor to senior people or people who can negotiate better time frames to deliver their bits.
canterburry
My challenges are mainly around retention and grooming the team. We work with lots of offshore resources who change frequently and whom we don't talk to directly always. However, when changes happen we want to retain the best and know who to cut. I'd also like to know who I can pull from where when a new project comes up.

I was hoping to gather metrics in the following dimensions:

1. Versatile (can work on multiple stacks and languages) 2. Team Player (watches out for the team and does what needs doing regardless if assigned to them) 3. Detail oriented and delivers quality code (doesn't break the build, good test coverage, adheres to formatting guidelines) 4. Productive 5. Communication skills (can and does communicate appropriately)

For many of these, we could gather proxy measures, i.e.

1. Versatile (in the last year, how many different file types did this person touch or work on. 2. Team Player (did they fix a build broken by someone else, how often do they fix their own broken builds)

etc etc

sheepmullet
> 1. Versatile (in the last year, how many different file types did this person touch or work on.

These kind of metrics never provide accurate information. You might as well just pick your "good" devs from a hat.

For example, I'm the main code reviewer for our front end solution but I haven't written a single line of code for it. Am I less versatile than a dev who does basic work across the entire project?

canterburry
I agree. Leads and other management roles are more difficult to peg but usually this is known simply by the person's role. A good dev lead could be judged by the number of code review comments or % code reviewed. Since a lead is also responsible for the team, their performance could be judged by overall team measures such as % test coverage of the overall code base, amount of time a ticket sits around etc. The team's metrics may become a way to judge the lead?

But I agree, it's all imperfect. I also don't think these metrics would be the only criteria by which someone is judged.