This article, and the one it links to for support, are self-refuting as far as I'm concerned. They make me think "be sure to have a coding test" if it keeps people who think like this away (specifically, people who think that they shouldn't have to try very hard and that their experience entitles them to preferential treatment).
Hey, that sounds like a good idea all round! It will also let experienced programmers know that they shouldn't bother applying to your shop (where their experience is not going to be recognized and valued).
They'd be right about that. Experience in and of itself counts for nothing. There are a great many programmers who have years of experience programming badly. Nor have I ever met a great programmer who cared a fig about how many years of experience they or anyone else had.
Good luck figuring out from a programming test whether someone is actually going to add value (rather than just be good at solving tests). Experience delivering quality work on time is a reasonable predictor of future success.
Those two look mostly the same on paper. I've looked at the linked-in profiles for people that I've worked with before. There are ones who absolutely sucked and were a net negative for the team, but who "helped deliver feature X" or "drove completion of project Y", but in fact, they probably actually delayed or at least lowered the quality bar of those very things they tout.
Absolutely. I've interviewed many "senior" developers who couldn't complete even very basic programming tasks. They could talk your ear off about their "experience", but ask them to actually do something and they were completely lost.
The author uses unfortunate wording. His claim is that requiring full applications as a part of the interview process is unnecessary. He does not claim coding itself should not be a part of the process.
He makes clear he's talking about full, to spec applications in the introduction, but he calls these "coding tests." His conclusion makes it clear he does not consider "coding tests" to be the fizzbuzz sort of questions:
For those who administer coding tests, please reconsider what you’re doing. For junior developers, they’re fabulous tools. But for anyone who has a body of code samples all ready to go, ask for one. Ask a few technical questions that require code in an interview.
Why do you feel insulted if someone asks you to show for an hour that you can do what you intend to do for the next couple years? I mean - it's not like you can't do that, right?
Depends. When I interview a candidate I can say if he is a good programmer or not after talking with him for 30 minutes.
May be because I am a programmer.
And I think I am bad interviewer if I cannot do that.
Do we really TEST Lawyers, Surgeons, Cooks in an interview?
Lawyers have to learn for many years and pass exams. Then they have a record of what they actually did, along with results in public records. Same exams situation for surgeons. Cooks usually are hired based on previous jobs (the really good ones are "tested" by entering competitions / getting additional training). Simple / begginer cooks are sometimes tested on the interview.
Those occupations are nothing like being a programmer.
The very premise of an interview is this: I see you're allegedly qualified from your résumé, but I'm skeptical. Prove you are who you say you are. This can be phrased much more tactfully, and it's great if the interviewer does so, but at some deep level every interviewee has to decide if they're comfortable with this dynamic or not. I think a technical question can raise the exact same hackles as requesting code.
If someone isn't comfortable with a complete stranger trying to validate whether they are the person described in the résumé, I suggest that they work hard on their networking so that they are always in a situation where the prospective employer is convinced in advance that they are a good hire.
That might work if the guy interviewing you have seen it, but I just really want to reply to can "can you write a function to traverse a tree" with "yes, on page 3 of the source code to the compiler I implemented in clojure".
I had much the same experience. I was interviewing with Steve Rosenberg at KL Group, and he asked me a question about Java coding. "Here," I said, pulling out a sheaf of papers, "is the source code for a small Scheme interpreter I wrote in Java..."
He waved me off, that was good enough for him. The rest of the interview was then about cultural fit and ability to ship software on time without drama.
(And no, what I wrote is not in the same league as Clojure, much less the same ballpark. But it was good enough to demonstrate some basic understanding of Java and familiarity with undergraduate-level Computer Science.)
Of course, this should also go the other way. As an interviewee, your task is both to prove yourself and to make the interviewers prove that their company is somewhere you want to work. So, ask probing questions about their culture, their code, their processes, their structure, the foundations of their business. And if they don't give good answers, or get upset about the questions, run away.
Absolutely. And I would say the same thing to interviewers: Part of the dynamic is of a candidate being skeptical that the company and the job are as good as they sound, and asking you to prove it. If that rankles an interviewer, seeking candidates through networking where the candidate arrives with a certain realistic view of the prospects is also a good thing.
It's surprising how many previously-good-looking candidates cannot write, say, a simple function to walk a tree. Now, part of the problem is certainly the fact that it's harder to write code with a pen and paper or a blackboard, and you have to account for that. It can still give valuable insight.
We always ask our candidates to write code in our interviews, but is there a reason why many companies seem to do this with pen & paper, or a blackboard?
We sit them in front of a computer with various editors/IDEs and ask them to code a function/class or two there on the machine.
Isn't this better for the candidate (ie less stressful, interviews are stressful enough already) and more realistic, as they can use editor/IDE features and the compiler?
Exactly, I wouldn't mind a future employee to ask me for some code samples so as to gauge the quality of our work... After all, I've seen some code from some companies before where I'd just run away if it was my first day at work there...
It's pointless to downvote the OP as his attitude is pretty common in my experience. There are many programmers who react with disgust if you ask them anything that might be construed as beneath them... embrace that attitude, because it can help to end the interview early and prevent wasting everyone's time.
This is the truth. I've had candidates walk out of interviews when they learn that they'd be asked to write pseudo code showing some basic algorithmic competence.
Actually, I make a point of asking to see a sample of production code anywhere I interview, and have done for a while. Nowhere has ever refused, although most places have asked to wait until the following interview if not warned in advance (which is best if possible).
This has been a very enlightening practice. To date, almost 100% of the places that showed me code put me off enough that I wouldn't have accepted any offer from them. The way they wrote their code told me more about the deep-rooted problems they had than anything anyone could ever say or not say in an interview.
It was fairly obvious to all concerned that those places and I weren't a good fit, though, and most didn't make me an offer in the end anyway. But if feedback was at all honest, asking to see their code has nothing to do with that, and in fact it was more often taken as a positive sign that I was genuinely interested, which at that stage I was.
My other favourite thing to do is turn questions around. If the people interviewing me at a smaller company are supposedly the senior technical people and they're assessing my abilities during a discussion about a code snippet (mine or theirs) in the interview, then I'm going to assess theirs as well: I will use a certain level of terminology, or mention related concepts, or allude to alternative design possibilities, and watch for their reactions and where the discussion goes. In other words, I would be assessing technical interviewers just the same way that I assess a candidate from the other side of the table, and for much the same reasons.
By the way, you should never feel insulted just because someone asks you to write code at interview, no matter what your level. I used to get some mild irritation from that, but having interviewed supposedly very experienced candidates, I have found that even those with great-looking CVs can be clueless no-hires in practice. The point of the basic coding test isn't to make the really good people stand out, it's to make the really bad people stand out. Of course, if you rapidly produce a decent answer to the simple coding problem, continuing with more simple coding problems starts to say more about the interviewer/company policy than anything else, and you might reasonably question whether they are wasting your time at that point.
But those companies was properly not those he was replying to.
Sure if you want to get into Frogcreek or Google then you need to spend months preparing (which is why I wont, since it is in the end just another job), but honestly many companies would be happy hiring somebody who cared enough to write a blogpost.
I always ask coding questions and I always ask a question or two that are extremely difficult. The goal is not to get the correct answers (although that is a nice bonus), but instead to see how the candidate thinks a problem through. A candidate's thought process is more important than their mad hacking skillz.
"how a candidate thinks" sounds like a very subjective criterion.
I wish there were more data behind people's interview practices. Do we know that people that think in way X are better employees than those that think in way Y? Why has HR mostly resisted empirical analysis?
Yes. It is very subjective. Which is a shame. Empirical methods have revolutionized so much of the world, yet this piece of it remains resistant to positive change.
My biggest pet peeve with these types of tests is this: there are a lot of companies out there, and I’m sending out resumes to each one that I can find.
That's your first mistake.
Why should a company take you seriously if you don't take them seriously? Every resume you send out should be custom written and carefully crafted to address their requirements, not yours. Do you do any research into the companies you're applying to? You should.
Rather than sending out 50 stock resumes, you'd probably be better off picking the top half dozen or so opportunities and do everything you can to stand out. This includes a custom cover letter, follow up emails, thank you notes, etc., whatever it takes to make the connection.
I simply do not have the time to write fifteen code samples a day, just because you want to evaluate me against your coding test. Period.
There's your second mistake. Misdirected attitude. Again, you should be focused on their needs, not yours. If you don't have time to do 15 poorly, then just do one or two very, very well. No employer or recruiter gives a damn that you've chosen a shotgun approach.
Give of yourself without expectations of something in return. You may be surprised at the "unexpected" dividends you'll reap later.
FWIW: I have conducted over 2500 technical interviews and every single one of them has had to code, regardless of experience or background. What good is a programmer who doesn't want to code at the one point when it would provide the most benefit to everyone involved? In my experience, the best programmers were the most eager to give it a shot.
I'm in agreement with everything except that it's not a matter of focusing on the company's needs. If you're sending resumes to 15 companies a day you aren't even focusing on your own needs.
Figure out who you want to work for and then figure out how to communicate with someone there who is going to want to hire you.
I agree that sending out resumes as fast as you can email is a bad idea, and I agree with the attitude you proscribe, but I think you're disagreeing with something the author never said.
He did not say "don't require coding as a part of the interview process." He did say "don't require fully functional, complete applications as a part of the interview process." I've never heard of anyone asking for that, but the author claims he has.
You should send out letters with the goal of maximizing the opportunity you get a job offer. Thus it might be very valid to spam a lot of companies with a generic offer.
This doesn't apply if you want to work of a few select companies, such as Frogcreek or Google, but most companies have so similar that they aren't worth customizing for.
From my experience, I've seen a huge difference in the quality of developers between companies that conducted strenuous coding tests and those that didn't, although I'm not sure if the relationship is causal. It certainly does help weed out candidates like the author, who is clearly not passionate about the companies he's applying to.
I once took a class in Industrial and Organizational Psychology. The professor moonlighted as a consultant for corporations. His advice was for companies to give real-life problems in interviews, as it's the most effective indicator of future job performance. The equivalent in our industry would be coding tests.
2,500 technical interviews? That would mean that you've been interviewing someone every day for 6 years straight. Assuming 3 interviews per day, that would be 2 years straight including Saturdays and Sundays.
I don't doubt you but having been a manager myself for 7+ years, I real have trouble believing that "every single one of them has had to code". I've done 500+ interviews myself and most have focused on coding questions but not all. I personally think that focusing on code 100% is really a big mistake in interviewing.
Of course, if you mean by "interviews", interviews done by a team where some someone on the team focused on code, then I agree with you. I would often find myself focusing on character questions or past experience while other members of my team focused on the coding questions.
How deep have you taken interview coding? Have you found that pseudo coding at a whiteboard is as effective as giving a candidate a test assignment on a box in the office?
I've also wondered about the effectiveness of reviewing candidate-submitted code samples vs some type of coding during the interview. Any thoughts there?
Have you found any other strong correlations between finding the best programmers in interviews other than the eagerness to tackle a programming problem?
I'm about to start searching to fill two positions now - I've struck out in my personal network, mainly because at this point I know people who are more senior and the positions I have are on the junior side, both in pay and responsibilities.
> But asking a programmer to write a complete application for you is like asking an electrician to build a small electrical grid for you before they work on your house. Can you imagine if every plumber had to prove that they could snake pipes before they could work on someone’s plumbing?
I wonder if he realizes that we require electricians and plumbers to be licensed?
If you were interviewing to be a juggler, would you find it odd if they asked you to juggle during the interview?
The reason why plumbers and electricians aren't asked to perform on a small scale is that they are required to be licensed. Until developers are licensed (and I don't see that happening any time soon), small-scale tasks are the best way to learn about a candidate.
[Apologies to whoever I stole the juggling analogy from (I think it was on Reddit).]
I've been on both sides of this situation. I don't think there is a clear answer.
The fundamental problem is that interviews are a horrible way to select people. There is little correlation between doing well in an interview and being capable at your job. So in theory the more you make the interview be like the job, the more accurate an assessment it is.
For instance I have never seen an interview method produce better results when looking for a front end HTML person than giving them a picture of the page you want, giving them cutouts of all of the pieces they need, and asking them to write a web page that looks like that, will scale reasonably as you resize the browser, and with reasonably clean HTML. Seriously, I've seen the results that this test produces, and invariably it gives you someone that can actually produce web pages. Occasionally it lets someone with poor work ethic slip through, but that seems to be better than the alternative. Particularly if the interviewer does not have the skillset you are looking to hire.
And then there is the time issue. I have seen interviewing for a new person suck up top talent for weeks on end. You need more resources, and you are losing proven, trained resources to do something they probably don't like! Yet unless you can have someone with the skills you're hiring in the room, the interview process can't weed out people who don't have the right skills. And as Joel's fizzbuzz test demonstrates, most candidates don't have the skills you need.
How bad is it? Well Joel claims a significant portion of candidates can't write a program to count numbers, but with every multiple of 3 replaced by "fizz", every multiple of 5 by "buzz", and every multiple of 15 replaced by "fizzbuzz". I believe it. I have found that over 2/3 of candidates when given a double-loop to compute the intersection of 2 arrays can't figure out how to make it more efficient. And good luck if you ask people to do something unusual like a depth-first traversal of a tree!
But there is a limit. It is easy to produce a simple application that looks reasonable, but which takes a lot longer to put together than you think it will. If you can't, knowing how it is supposed to go, personally do the exercise in an hour or two on a freshly set up machine, the exercise is too hard. Personally I far prefer having a test with some simpler questions rather than a full blown mini-application.
But the opposite extreme of depending on interviews to filter out bad people is worse. Much worse.
If I'm really excited about a position, I'll write my own code sample before I go to an interview. I'll typically write either a simple igoogle module or possibly a simple feature on my own website, proxying some of their calls through my own web server.
I find it does a tremendous amount of good on both ends. It lets me explore their website and look at their apis, I get a better understanding of their tech stack, and by looking at their code (at least their client javascript code), I brush up on areas that they've tackled.
On the company's end, we can dive into some tangible code that I've written and they can code review my work and ask technical questions based on that. It also shows them I'm serious and will hit the ground running.
I much prefer talking about relevant code that I've just written to a trip to the whiteboard to answer an esoteric question or hard brain teaser. It's more like a real work environment and the questions, at least I think, are more authentic to the task: why did you implement that algorithm this way instead of another? what are the code's limitations or where can it be improved? how would it scale?
55 comments
[ 2.2 ms ] story [ 102 ms ] threadWhat counts is the value you can add now.
He makes clear he's talking about full, to spec applications in the introduction, but he calls these "coding tests." His conclusion makes it clear he does not consider "coding tests" to be the fizzbuzz sort of questions:
For those who administer coding tests, please reconsider what you’re doing. For junior developers, they’re fabulous tools. But for anyone who has a body of code samples all ready to go, ask for one. Ask a few technical questions that require code in an interview.
And I think I am bad interviewer if I cannot do that.
Do we really TEST Lawyers, Surgeons, Cooks in an interview?
Those occupations are nothing like being a programmer.
If someone isn't comfortable with a complete stranger trying to validate whether they are the person described in the résumé, I suggest that they work hard on their networking so that they are always in a situation where the prospective employer is convinced in advance that they are a good hire.
He waved me off, that was good enough for him. The rest of the interview was then about cultural fit and ability to ship software on time without drama.
(And no, what I wrote is not in the same league as Clojure, much less the same ballpark. But it was good enough to demonstrate some basic understanding of Java and familiarity with undergraduate-level Computer Science.)
We sit them in front of a computer with various editors/IDEs and ask them to code a function/class or two there on the machine.
Isn't this better for the candidate (ie less stressful, interviews are stressful enough already) and more realistic, as they can use editor/IDE features and the compiler?
It was bizarre.
This has been a very enlightening practice. To date, almost 100% of the places that showed me code put me off enough that I wouldn't have accepted any offer from them. The way they wrote their code told me more about the deep-rooted problems they had than anything anyone could ever say or not say in an interview.
It was fairly obvious to all concerned that those places and I weren't a good fit, though, and most didn't make me an offer in the end anyway. But if feedback was at all honest, asking to see their code has nothing to do with that, and in fact it was more often taken as a positive sign that I was genuinely interested, which at that stage I was.
My other favourite thing to do is turn questions around. If the people interviewing me at a smaller company are supposedly the senior technical people and they're assessing my abilities during a discussion about a code snippet (mine or theirs) in the interview, then I'm going to assess theirs as well: I will use a certain level of terminology, or mention related concepts, or allude to alternative design possibilities, and watch for their reactions and where the discussion goes. In other words, I would be assessing technical interviewers just the same way that I assess a candidate from the other side of the table, and for much the same reasons.
By the way, you should never feel insulted just because someone asks you to write code at interview, no matter what your level. I used to get some mild irritation from that, but having interviewed supposedly very experienced candidates, I have found that even those with great-looking CVs can be clueless no-hires in practice. The point of the basic coding test isn't to make the really good people stand out, it's to make the really bad people stand out. Of course, if you rapidly produce a decent answer to the simple coding problem, continuing with more simple coding problems starts to say more about the interviewer/company policy than anything else, and you might reasonably question whether they are wasting your time at that point.
Sure if you want to get into Frogcreek or Google then you need to spend months preparing (which is why I wont, since it is in the end just another job), but honestly many companies would be happy hiring somebody who cared enough to write a blogpost.
I wish there were more data behind people's interview practices. Do we know that people that think in way X are better employees than those that think in way Y? Why has HR mostly resisted empirical analysis?
Of course it is. What part of the interview process isn't subjective?
That's your first mistake.
Why should a company take you seriously if you don't take them seriously? Every resume you send out should be custom written and carefully crafted to address their requirements, not yours. Do you do any research into the companies you're applying to? You should.
Rather than sending out 50 stock resumes, you'd probably be better off picking the top half dozen or so opportunities and do everything you can to stand out. This includes a custom cover letter, follow up emails, thank you notes, etc., whatever it takes to make the connection.
I simply do not have the time to write fifteen code samples a day, just because you want to evaluate me against your coding test. Period.
There's your second mistake. Misdirected attitude. Again, you should be focused on their needs, not yours. If you don't have time to do 15 poorly, then just do one or two very, very well. No employer or recruiter gives a damn that you've chosen a shotgun approach.
Give of yourself without expectations of something in return. You may be surprised at the "unexpected" dividends you'll reap later.
FWIW: I have conducted over 2500 technical interviews and every single one of them has had to code, regardless of experience or background. What good is a programmer who doesn't want to code at the one point when it would provide the most benefit to everyone involved? In my experience, the best programmers were the most eager to give it a shot.
Figure out who you want to work for and then figure out how to communicate with someone there who is going to want to hire you.
He did not say "don't require coding as a part of the interview process." He did say "don't require fully functional, complete applications as a part of the interview process." I've never heard of anyone asking for that, but the author claims he has.
This doesn't apply if you want to work of a few select companies, such as Frogcreek or Google, but most companies have so similar that they aren't worth customizing for.
I once took a class in Industrial and Organizational Psychology. The professor moonlighted as a consultant for corporations. His advice was for companies to give real-life problems in interviews, as it's the most effective indicator of future job performance. The equivalent in our industry would be coding tests.
I don't doubt you but having been a manager myself for 7+ years, I real have trouble believing that "every single one of them has had to code". I've done 500+ interviews myself and most have focused on coding questions but not all. I personally think that focusing on code 100% is really a big mistake in interviewing.
Of course, if you mean by "interviews", interviews done by a team where some someone on the team focused on code, then I agree with you. I would often find myself focusing on character questions or past experience while other members of my team focused on the coding questions.
I've also wondered about the effectiveness of reviewing candidate-submitted code samples vs some type of coding during the interview. Any thoughts there?
Have you found any other strong correlations between finding the best programmers in interviews other than the eagerness to tackle a programming problem?
I'm about to start searching to fill two positions now - I've struck out in my personal network, mainly because at this point I know people who are more senior and the positions I have are on the junior side, both in pay and responsibilities.
http://news.ycombinator.com/item?id=89669
http://news.ycombinator.com/item?id=834513
http://news.ycombinator.com/item?id=835092
Hope these help.
I wonder if he realizes that we require electricians and plumbers to be licensed?
The reason why plumbers and electricians aren't asked to perform on a small scale is that they are required to be licensed. Until developers are licensed (and I don't see that happening any time soon), small-scale tasks are the best way to learn about a candidate.
[Apologies to whoever I stole the juggling analogy from (I think it was on Reddit).]
The fundamental problem is that interviews are a horrible way to select people. There is little correlation between doing well in an interview and being capable at your job. So in theory the more you make the interview be like the job, the more accurate an assessment it is.
For instance I have never seen an interview method produce better results when looking for a front end HTML person than giving them a picture of the page you want, giving them cutouts of all of the pieces they need, and asking them to write a web page that looks like that, will scale reasonably as you resize the browser, and with reasonably clean HTML. Seriously, I've seen the results that this test produces, and invariably it gives you someone that can actually produce web pages. Occasionally it lets someone with poor work ethic slip through, but that seems to be better than the alternative. Particularly if the interviewer does not have the skillset you are looking to hire.
And then there is the time issue. I have seen interviewing for a new person suck up top talent for weeks on end. You need more resources, and you are losing proven, trained resources to do something they probably don't like! Yet unless you can have someone with the skills you're hiring in the room, the interview process can't weed out people who don't have the right skills. And as Joel's fizzbuzz test demonstrates, most candidates don't have the skills you need.
How bad is it? Well Joel claims a significant portion of candidates can't write a program to count numbers, but with every multiple of 3 replaced by "fizz", every multiple of 5 by "buzz", and every multiple of 15 replaced by "fizzbuzz". I believe it. I have found that over 2/3 of candidates when given a double-loop to compute the intersection of 2 arrays can't figure out how to make it more efficient. And good luck if you ask people to do something unusual like a depth-first traversal of a tree!
But there is a limit. It is easy to produce a simple application that looks reasonable, but which takes a lot longer to put together than you think it will. If you can't, knowing how it is supposed to go, personally do the exercise in an hour or two on a freshly set up machine, the exercise is too hard. Personally I far prefer having a test with some simpler questions rather than a full blown mini-application.
But the opposite extreme of depending on interviews to filter out bad people is worse. Much worse.
I find it does a tremendous amount of good on both ends. It lets me explore their website and look at their apis, I get a better understanding of their tech stack, and by looking at their code (at least their client javascript code), I brush up on areas that they've tackled.
On the company's end, we can dive into some tangible code that I've written and they can code review my work and ask technical questions based on that. It also shows them I'm serious and will hit the ground running.
I much prefer talking about relevant code that I've just written to a trip to the whiteboard to answer an esoteric question or hard brain teaser. It's more like a real work environment and the questions, at least I think, are more authentic to the task: why did you implement that algorithm this way instead of another? what are the code's limitations or where can it be improved? how would it scale?