Solid review of the dev hiring process, the pitfalls and even reference to Seth Godin. Joel Spolsky put some time and effort into a short list of best practices but watching how someone works is a great way to gauge their potential contribution.
Of course if you're like myself, you may be a little nervous about coding with a "reviewer" at your back. The best way to get past uneasisness is pair programming if you can afford to dedicate a developer for a few hours (most startups can't, but they have to if they want to grow in a rational way).
This seemed like horrible advice to me. The gist of the article is, "... put them into your team for a day and watch them work. After the day is over get your team together and let them tell you if you should hire this person or not."
I won't even bother listing all the possible shortcomings - it's a great water cooler idea but it won't fly in the real world.
* Person comes in, doesn't work out, and then goes to work for a competitor and can now (a) talk about your processes, (b) tell them who the key people are to target
* Person comes in and tells technical recruiters who your top talent is
* Person comes in, explains (good/bad) reasons that this code sucks and whoever wrote it was wrong, and makes everyone feel like a fool for working there
* Developers A, B, and C love the guy yet manager hates him (or vice versa)
Interesting idea in practice but I seriously doubt any company does this instead of the traditional techniques he mentions. The "traditional" techniques have evolved over 100 years of working with and interviewing engineers and software folks - there's a reason those techniques are still used in 2009.
You may not like the article, but your "shortcomings" are all wrong.
Person comes in, doesn't work out, and then goes to work for a competitor and can now (a) talk about your processes, (b) tell them who the key people are to target
If they aren't good enough for your team and your competitor ends up hiring them, what does that say about your competitor? Besides, I highly doubt that the candidate can find out anything remotely interesting about your team in just a day.
Person comes in and tells technical recruiters who your top talent is
Again, highly doubtful that a candidate will be be able to evaluate your top talent and even if they do, so what? All the good devs have contacts with recruiters anyways and if you aren't making it attractive enough for them to stay, then they will leave regardless of some noob interviewee.
Person comes in, explains (good/bad) reasons that this code sucks and whoever wrote it was wrong, and makes everyone feel like a fool for working there
If you can't take a little criticism, then you really need to get out of the game.
Developers A, B, and C love the guy yet manager hates him (or vice versa)
Easy, fire the manager. I've had this except in reverse and our manager heeded our advice. Do you really want to work for a manager who doesn't listen to his developers?
I agree that worrying about leaking info to competitors is silly, but...
> If they aren't good enough for your team and your competitor ends up hiring them, what does that say about your competitor?
There are a number of reasons you may say no to a candidate who is highly qualified. They may not be a cultural fit for your specific organization, or you may have more than one qualified candidate but only one position to fill. So just because you wind up rejecting them, doesn't mean they suck.
No, coding on the whiteboard or on paper, or even the 5 minute exercise on the laptop is not really coding.
It doesn't matter whether or not it's "really coding". All that matters is how effective it is in evaluating your candidate.
I have interviewed over 2,500 devloper candidates and every single one has had to code with pencil and paper, in a room alone for 15 to 30 minutes. This has always been, by far, the most effective thing I could have done.
I never cared what they actually wrote. I never once found out if it would even compile or run. But I never cared. The only purpose of the coding problem was the discussion afterward. This told me volumes.
Given a person, a discussion, a problem, and 1 to 3 pages of written code, I could ask many questions focused on a single issue and take it any direction I wanted. And learn what I needed to know about that person...
How did they attack the problem? What did they feel comfortable using? How did the deal with the situation? What kind of attitude did they have? How much did they enjoy dealing with the problem and discussing it? How well did they defend their choices? How willing were they to take criticism? How willing were they to stand up for what they believed in?
These are the things I want to know now. Not 3 months after they start working. Programming with pencil and paper and discussing afterward is the best way I've ever found to find these things out.
I'd love to hear more about your methodology. Could you give an example of a problem that you ask the candidates to solve? What types of questions do you ask them when they are done?
Here's one of my favorite examples that can work for almost any language.
Remember, before any of this happens, I have already helped the candidate get comfortable and have made the purpose of the exercise clear: to assess where they're at and how/where they might fit in. It is not a test. Just an exercise to help us both. I offer them a soda or coffee, a little privacy, and this little problem...
You have an array. Call it "a" or whatever you want. It has a bunch of elements, numeric, alphanumeric, or whatever. You decide. Sort it. Without using a second array or a pre-existing function or routine. While I'm explaining this I'm sketching it out with my own pencil and paper. I suggest that they sketch out what they want to do themselves and then write some code (in the language being evaluated) or just pseudo code for general purposes. Be prepared to discuss whatever you want to present. Don't go nuts, just a few pages and 15 to 30 minutes. And have fun.
When I return, I have them explain how they approached it. (Here's what my code will do...) Then we go through the code line by line. At this point, it's incredibly easy for me to ask questions, such as...
Why did you name that variable that name?
Why did you use a for loop?
How else could you have done iteration?
How would you do it with 2 loops?
How would you do it with 1 loop?
Which variables are global? Which are local? Why?
Why did you reuse the variable "i" in the inner loop? (Oops)
How can you make it faster?
How could you make it clearer?
How would you change it if you knew the probability of the original order?
How would you refactor this?
How would you extend this to do...?
Which code would you put in a library for reuse?
You kinda get the picture. No 2 interviews are the same. Imagine the programmers you already know having this discussion with you and how much you'd learn about them.
There are no right or wrong answers, just learning. Which is what you want.
This simple test eliminates the 90% of applicants who are not suited for this work and 100% of the posers. You can tell right away who they are.
OTOH, good programmers shine on this. It's actually fun to hang out and talk about this stuff with them. I have even had candidates email me later with revised code based on our discussion. Those are the motivated ones. Big points for that.
If you know how an insertion sort (for example) works, then you can pretty much answer all your questions fairly easily. Knowing algos is good, but does not in itself make you competent in every respect.
Having said that, I am not trying to be critical. I completely agree that having a discussion like this is extremely valuable and can let you find out a lot about a person and even eliminate most of the chaff (although I have met a couple of people who I wouldn't necessarily hire, but who could have done well in a discussion like that).
After your discussion, you might even decide you really like a person, but does your team like them? How about this, can you tell if a person can stick to a deadline? A longer discussion is necessary before this becomes apparent.
You may like what you're doing, but there's no way that you can prove that you didn't eliminate your best candidates, and I'd be highly suspicious of your veracity if you told me that you've never made a bad hire in 2500 interviews. Anyone who says that they can figure out everything that they needed to know about a person by asking them to do a one-dimensional task only nominally related to the job at hand (i.e. "here's a piece of paper. draw me a diagram of digging a ditch"), is probably missing some important information.
I understand what you're saying about not caring about the coding result, and I think you're right -- too many geeks focus too much on the brain-teaser part, and not enough on the human interaction part. But even then, it's an artificial environment, and people are nervous and stiff. Newer candidates are unpracticed. Candidates who have had a few interviews are more likely to succeed. Some small number people can bullshit their way through any interview. The best you can say about an interview process is that it might give you a chance to know if you'll absolutely hate working with someone.
You bring up many good points. I'm just sharing what I have done that has worked. And it has worked very well.
My approach doesn't prove anything. It's just a tool. Like any tool, it has to be handled properly. I can immediately tell who is comfortable and who is not. I adjust for that. And this tool is never the last step, just an early and important one.
I don't learn everything about the candidate, but I learn a lot. And no one can bullshit their way through this; they have to know how to do it. Posers shouldn't waste my time; I will identify them with this every time.
We've come up with a new idea to try out in the next round of hires (I might blog about it at some point).
Basically we think up a tiny but worthwhile project related to the area were hiring in. i.e. a supporting app to go with our current software, or a feature extension (without going too deep into our source code :D) etc.
Then we pick the most likely candidates (maybe interview them) and hold a week long coding sprint. We plan to pay the candidate expenses and a fair salary (something like £6-7 and hour).
We'll have about 10 candidates maximum and pick 2 or 3 of our current development team to work with them. Spend one day of the week making sure they are roughly on the same page in their work practices then just let them come up with a solution and build it.
And actual final product doesnt matter too much: but it's a quickfire way to see them working as a team and figure out who would fit into our process.
Plus some of our current developers get to work with them and will be able to advise who they would find best to work with.
It will either work really well or fail miserably: but it's a nice idea nontheless!
imo, the greatest improvement in this process is made when both sides (interviewer and interviewee) realize it's a two way street - a mutually beneficial relationship.
Both sides, pick some problems that NONE of you know the answer to and work through them together. If the interviewer wants to do a group interview, let the person interviewing bring some friends along, too.
I'm really tired of all the people who think they have the line on the best way to do something, particularly when that something involves interacting with human beings. We're all hill climbing here. Nobody even knows if we're on the right mountain.
He has some points, but misses others. Ability to code on a whiteboard is a valuable skill for a collaborative lead developer. For a junior guy getting his direction from elsewhere, not so much. Some people's work persona comes out right away. Other people take months. One great developer who worked for me seemed like a total failure till he came out of his shell.
Human beings are too complex for one-size-fits-all evaluation. This applies to the interviewer as much as the interviewee. Will your "best way" be best for me? Probably not. Can I learn from works best for you? Absolutely!
I agree there is no silver bullet. You may not agree with the article, but it id generate some discussion and got people to share their thoughts (including you :)). This way we can all learn from each others experiences and discover methods that can work for us.
How many of us spent the first day of our programming jobs actually programming?
I don't see how they would expect to see somebody in their element with a brand new code base, new development environment etc.
And if the remedy to that is to give them coding problems in a contained environment in which they don't need to have a git account/dev server login/etc., then what makes this different from a normal coding interview?
18 comments
[ 4.7 ms ] story [ 53.7 ms ] threadOf course if you're like myself, you may be a little nervous about coding with a "reviewer" at your back. The best way to get past uneasisness is pair programming if you can afford to dedicate a developer for a few hours (most startups can't, but they have to if they want to grow in a rational way).
I won't even bother listing all the possible shortcomings - it's a great water cooler idea but it won't fly in the real world.
* Person comes in, doesn't work out, and then goes to work for a competitor and can now (a) talk about your processes, (b) tell them who the key people are to target
* Person comes in and tells technical recruiters who your top talent is
* Person comes in, explains (good/bad) reasons that this code sucks and whoever wrote it was wrong, and makes everyone feel like a fool for working there
* Developers A, B, and C love the guy yet manager hates him (or vice versa)
Interesting idea in practice but I seriously doubt any company does this instead of the traditional techniques he mentions. The "traditional" techniques have evolved over 100 years of working with and interviewing engineers and software folks - there's a reason those techniques are still used in 2009.
Person comes in, doesn't work out, and then goes to work for a competitor and can now (a) talk about your processes, (b) tell them who the key people are to target
If they aren't good enough for your team and your competitor ends up hiring them, what does that say about your competitor? Besides, I highly doubt that the candidate can find out anything remotely interesting about your team in just a day.
Person comes in and tells technical recruiters who your top talent is
Again, highly doubtful that a candidate will be be able to evaluate your top talent and even if they do, so what? All the good devs have contacts with recruiters anyways and if you aren't making it attractive enough for them to stay, then they will leave regardless of some noob interviewee.
Person comes in, explains (good/bad) reasons that this code sucks and whoever wrote it was wrong, and makes everyone feel like a fool for working there
If you can't take a little criticism, then you really need to get out of the game.
Developers A, B, and C love the guy yet manager hates him (or vice versa)
Easy, fire the manager. I've had this except in reverse and our manager heeded our advice. Do you really want to work for a manager who doesn't listen to his developers?
> If they aren't good enough for your team and your competitor ends up hiring them, what does that say about your competitor?
There are a number of reasons you may say no to a candidate who is highly qualified. They may not be a cultural fit for your specific organization, or you may have more than one qualified candidate but only one position to fill. So just because you wind up rejecting them, doesn't mean they suck.
It doesn't matter whether or not it's "really coding". All that matters is how effective it is in evaluating your candidate.
I have interviewed over 2,500 devloper candidates and every single one has had to code with pencil and paper, in a room alone for 15 to 30 minutes. This has always been, by far, the most effective thing I could have done.
I never cared what they actually wrote. I never once found out if it would even compile or run. But I never cared. The only purpose of the coding problem was the discussion afterward. This told me volumes.
Given a person, a discussion, a problem, and 1 to 3 pages of written code, I could ask many questions focused on a single issue and take it any direction I wanted. And learn what I needed to know about that person...
How did they attack the problem? What did they feel comfortable using? How did the deal with the situation? What kind of attitude did they have? How much did they enjoy dealing with the problem and discussing it? How well did they defend their choices? How willing were they to take criticism? How willing were they to stand up for what they believed in?
These are the things I want to know now. Not 3 months after they start working. Programming with pencil and paper and discussing afterward is the best way I've ever found to find these things out.
Remember, before any of this happens, I have already helped the candidate get comfortable and have made the purpose of the exercise clear: to assess where they're at and how/where they might fit in. It is not a test. Just an exercise to help us both. I offer them a soda or coffee, a little privacy, and this little problem...
You have an array. Call it "a" or whatever you want. It has a bunch of elements, numeric, alphanumeric, or whatever. You decide. Sort it. Without using a second array or a pre-existing function or routine. While I'm explaining this I'm sketching it out with my own pencil and paper. I suggest that they sketch out what they want to do themselves and then write some code (in the language being evaluated) or just pseudo code for general purposes. Be prepared to discuss whatever you want to present. Don't go nuts, just a few pages and 15 to 30 minutes. And have fun.
When I return, I have them explain how they approached it. (Here's what my code will do...) Then we go through the code line by line. At this point, it's incredibly easy for me to ask questions, such as...
You kinda get the picture. No 2 interviews are the same. Imagine the programmers you already know having this discussion with you and how much you'd learn about them.There are no right or wrong answers, just learning. Which is what you want.
This simple test eliminates the 90% of applicants who are not suited for this work and 100% of the posers. You can tell right away who they are.
OTOH, good programmers shine on this. It's actually fun to hang out and talk about this stuff with them. I have even had candidates email me later with revised code based on our discussion. Those are the motivated ones. Big points for that.
Did you find that success on this exercise tracked well with success on the job for those that you ended up hiring?
Yes. If someone didn't work out in the long run, it was rarely because of their programming skill, but for some other reason.
Having said that, I am not trying to be critical. I completely agree that having a discussion like this is extremely valuable and can let you find out a lot about a person and even eliminate most of the chaff (although I have met a couple of people who I wouldn't necessarily hire, but who could have done well in a discussion like that).
After your discussion, you might even decide you really like a person, but does your team like them? How about this, can you tell if a person can stick to a deadline? A longer discussion is necessary before this becomes apparent.
I understand what you're saying about not caring about the coding result, and I think you're right -- too many geeks focus too much on the brain-teaser part, and not enough on the human interaction part. But even then, it's an artificial environment, and people are nervous and stiff. Newer candidates are unpracticed. Candidates who have had a few interviews are more likely to succeed. Some small number people can bullshit their way through any interview. The best you can say about an interview process is that it might give you a chance to know if you'll absolutely hate working with someone.
My approach doesn't prove anything. It's just a tool. Like any tool, it has to be handled properly. I can immediately tell who is comfortable and who is not. I adjust for that. And this tool is never the last step, just an early and important one.
I don't learn everything about the candidate, but I learn a lot. And no one can bullshit their way through this; they have to know how to do it. Posers shouldn't waste my time; I will identify them with this every time.
Basically we think up a tiny but worthwhile project related to the area were hiring in. i.e. a supporting app to go with our current software, or a feature extension (without going too deep into our source code :D) etc.
Then we pick the most likely candidates (maybe interview them) and hold a week long coding sprint. We plan to pay the candidate expenses and a fair salary (something like £6-7 and hour).
We'll have about 10 candidates maximum and pick 2 or 3 of our current development team to work with them. Spend one day of the week making sure they are roughly on the same page in their work practices then just let them come up with a solution and build it.
And actual final product doesnt matter too much: but it's a quickfire way to see them working as a team and figure out who would fit into our process.
Plus some of our current developers get to work with them and will be able to advise who they would find best to work with.
It will either work really well or fail miserably: but it's a nice idea nontheless!
Both sides, pick some problems that NONE of you know the answer to and work through them together. If the interviewer wants to do a group interview, let the person interviewing bring some friends along, too.
He has some points, but misses others. Ability to code on a whiteboard is a valuable skill for a collaborative lead developer. For a junior guy getting his direction from elsewhere, not so much. Some people's work persona comes out right away. Other people take months. One great developer who worked for me seemed like a total failure till he came out of his shell.
Human beings are too complex for one-size-fits-all evaluation. This applies to the interviewer as much as the interviewee. Will your "best way" be best for me? Probably not. Can I learn from works best for you? Absolutely!
A little humility goes a long way.
And if the remedy to that is to give them coding problems in a contained environment in which they don't need to have a git account/dev server login/etc., then what makes this different from a normal coding interview?