Ask HN: Am I being too hard on the developers I interview?
Yet, the results of this are disappointing. About 10% of candidates will be able to produce a solution in 10 minutes or less, another 30% of candidates will eventually produce something close to a solution in 30 minutes (though often with errors), and 60% of candidates will simply be unable to produce anything close to a solution. These are not junior or graduate developers, but senior, experienced programmers asking for £50k or £60k.
I try to make the candidates as comfortable as possible while doing this, but even so, the inability of 90% of the people I interview to develop a solution to a simple question is astonishing.
Am I being too hard on the candidates in asking them to do this?
-------------------------------------------------------------------
This is, verbatim, the problem sheet I show them:
Pascal’s Triangle
Each element in Pascal’s Triangle is found by adding the two numbers to the right and left of it on the row above it; where these don’t exist, they are assumed to be zero.
1
1 1
1 2 1
1 3 3 1
1 4 6 4 1
1 5 10 10 5 1
(This is correctly centered in the actual document.)
Write a function that takes a single integer n as a parameter and returns the nth row of Pascal’s Triangle.
For example:
pascal(1) should return [1]
pascal(4) should return [1, 3, 3, 1]
76 comments
[ 3.2 ms ] story [ 138 ms ] threadIn general though it's better to ask people problems related to things they'll actually be working on. E.g., have them analyze a bug you recently found in your own code.
I think if you are going to give this to people in an interview you should a) make sure you've told them to expect puzzles and b) give them a few questions, some as a warm up with plenty of time to settle in.
Personally, I have a really good track record of finding and hiring talent. So much so that, friends in other companies will ask me to find them someone. Anyways, one of the most powerful things I do to find talent is ask them either send me some code ahead of time or better yet bring a laptop with a working project they have done, on it. I then ask one simple question, show me the piece of code in this project that you wrote and are most proud of. Someone that is BS'ing can't BS through that. It shows real world application, and people are generally most proud of things that they had to overcome adversity on, they are happy to talk about how they solved the problem. People try to make interviewing hard, but with that single question you will know everything you need to know about their technical skills. Personality skills is another matter, but I just take them to lunch and hang out with them to figure out if they are a fit personality wise.
It's hard to find good people.
I'm not surprised by your 90% no-solve stat. Candidates I interview who put Java/C# on their resumes can barely "write a function that returns true if the first letter of a string parameter is capitalized". At least half flat out can't do it.
I actually do ask them to write the code - I'd expect most programmers to know the basic syntax of the language the claim to be proficient in, but I've seen Ruby developers who are unable to correctly use Array#each, which would be a red flag.
Pascal's Triangle is as much a graphical presentation as a mathematical one.
Let's assume for a second that your candidate does not already know what Pascal's Triangle is.
What exactly do you mean by the element above and to the right?
In the third row second column you have the number two despite there clearly being no element on the second row to the "right".
You and I know that you don't actually use the element "above and to the right" but rather, the "element directly above".
You set your candidate up for failure then you are surprised that a large number of them fail?
If what you seek is to determine how the candidate solves problems including getting you to clarify your question then fine, but I don't get the impression that is what you are after.
You present the question as a simple problem to be solved while you sit and watch. It is simple if you know ahead of time how to represent Pascal's Triangle and what the rules "really are" from a programming perspective. If you don't you are likely screwed.
I wager a guess, that the 10% that could solve your problem already knew how.
Are you really looking to hire people who happen to know the answer to your slightly tricky question?
You might as well ask them how many toilets there are in Boston then come back and complain about the 90% who got it wrong.
I would say you should pass on the 60% who can't solve it, unless they take the initiative and send you a solution after the interview (I've done this, when I could solve the problem but not in time). That might prevent you from passing on an otherwise good candidate who happened to choke once. The 30% in the middle you can attribute to interview stress, if other indications are that they know how to write code.
The truth is, though, that a large percentage of "professional programmers" have never built anything from scratch and have been wearing training wheels the whole time. 90 percent of what professional programmers do is maintenance at ~10 LoC per day, never leaving the comfort of the IDE or building a system from the ground, and so there are a lot who've never been in a green field or had to solve a hard programming problem. You don't want to hire such people to build new stuff. You're not being "too hard" on them by filtering them out.
In fact, I'm surprised that your fail rate is only 60%.
because I went through an interview process not too long ago, and they asked me to take a test that was actually very simple stuff, fibonacci sequence, dealing with strings, etc.
And for the life of me I just couldn't do it on paper, if I was still in college I might have done it, but now that I'm a professional developer, I DON'T code on paper anymore, it feels weird and awkward. Maybe do a test, and let a few take the test on the computer, doesn't have to be in an IDE, just let them hack away at notepad, or something.
I know most people believe developers should be those guys that can code their way out of any problem with just matchsticks or something, but really, would you ever encounter a situation where you need a developer to actually write code on paper?
And before anyone says I'm just a bad programmer, I am usually the go to guy at my workplace, the guy that usually solves hard issues, and the more competent I become at actually useful stuff, and actual day to day programming challenges, the more I feel far from things like your test.
Alternatively, you could also use a real world problem, I think there is nothing that can get a guy to prove they are a good developer than making subtle bugs in some code and asking the guy to fix them.
1) They've never solved problems like this with code before.
2) They fail to grasp how such problem solving skills can judge their "real world" experience.
3) Someone across the street/city/village will pay them more than you are paying and have more softball interview questions because they are more desperate to bring in any programming talent they can find.
While reasons 1 and 2 are good reasons to pass on candidates, reason 3 is unfortunately a bigger challenge to solve.
It may be unpopular to say this, but I would take a hard look at the problems you need your developers to solve and come up with a more real world scenario to use for interviews and see if you get better results.
The point I'm getting at, is that like any science experiment you need to control for the variables. Just like an economics experiment might control for wealth, or education, you have to control for people who get nervous during interviews, people who have a blind spot for this particular problem, etc.
An easy addition to your methodology would be to have three different problems and let them choose. I would see how that affects their performance.
In my company I give potential hires a test for them to complete at home, before even interviewing them. The average time to complete the test is 6 hours, but we've never had anyone outright refuse to attempt it, and it gives us a giant steaming pile of data. So if my experience is any guide (and it might not be) I would think you could make your test more intensive, and potentially end up with more and better candidates.
Took me about 15 minutes to do. Doesn't seem like an unreasonable question to ask.
- (NSArray)pascal:(int)n { if (n == 1) return @[@(1)];
NSArray array = [self pascal:n-1]; NSMutableArray* mArray = [@[] mutableCopy]; for(int i = 0; i < array.count+1; i++) { int r = i - 1; if (r >= 0 && i < array.count) [mArray addObject:@([array[r] integerValue] + [array[i] integerValue])]; else if (i < array.count) [mArray addObject:@(0 + [array[i] integerValue])]; else [mArray addObject:@(0 + [array[i-1] integerValue])]; }
return [mArray copy]; }
pascal_r = (n) -> lines = [[1]] for i in [1...n] last = lines[i-1] lines[i] = [] for j in [0..i] lines[i][j] = (last[j-1] or 0) + (last[j] or 0) lines[n-1]
console.log pascal_r(1) console.log pascal_r(4)
Edit: Bloody HN formatting got the better of me again.
One was that every technical question after the third one was usually a waste. Anyone who hit it out of the park on the first three, hit almost all of the subsequent ones out of the park. No one who struggled through the first three would suddenly start doing well on the subsequent questions.
Another thing I noticed is people fell into a Gaussian distribution. Some were on the left end of it, but most of those were weeded out by the recruiters, or by resumes or finally by phone screens. Just like a bell curve, most of the people were in the middle. Of the dozens I interviewed in person, only three were on the right end of the bell curve. Which is before you even get to soft skills.
Your results speak for themselves. You have found an exercise that helps mark the top 10% from the 11-40% from the 41-100%.
You might want to read some blog posts like Jeff Atwood's "Why can't programmers program?"
The story reminds me of a blog post from a while ago [1] which concluded with an update that after raising his rates by $5 the first candidate to apply passed with flying colors.
[1] http://yakovfain.com/2012/10/11/the-degradation-of-java-deve....
That gibberish. Unless you know what "Pascal’s Triangle" is prior to seeing this you have no shot of even understanding the exercise.
You haven't even said how you're meant to iterate through the numbers. If this is meant to be an explanation of a "Pascal’s Triangle" it fails at that.
You aren't testing programming, you're testing obscure maths knowledge and the ability to decrypt your question/explanation.
I bet a lot of the other posters had to google this just to answer it in their impressively quick times.
I've never needed this in my programming career up until now. Have you?
My thought process: I take a number, add the right and left on the same row, then put the result on the upper row. Or, add the left and right and current and put it on the row above. After a while I guess you mean any position is the sum of the numbers to the 1 position upper left and upper right.
If you look at the graphic that always comes with this question, there's only one way you could possibly interpret it. If you somehow misinterpret the text, you would know immediately when you test your mental model against the graphic.
To be fair, the screwed up formatting in the OP makes it a little harder to understand; in an interview it would be laid out correctly.
Would you still think all of those with the diagram?
"...adding the two numbers to the right and left of it on the row above it..."
Doesn't make a whole lot of sense. What is the "it"?
row 1: is obviously easy. row 2: I think you are adding 0 to 1 and 1 to 0 to get 1 and 1 row 3: You are adding 0 + 1 for the first item, 1 + 1 for the second, and 1 + 0 for the third. row 4: You are adding 0 + 1 for the first number, 1 + 2 for the second...but the directions say "to the left AND right of it" which makes me think that the answer should be 4, not 3. 1 + 2 + 1. Really, you are only adding the number to the left of "it" (which still isn't clear). And actually, for the first half of the line you are adding the number to the left and for the second half of the line you are adding the number to the right.
What you want:
row 1: 0 + 1 row 2: 0 + 1, 1 + 0 row 3: 0 + 1, 1 + 1, 1 + 0 row 4: 0 + 1, 1 + 2, 2 + 1, 1 + 0 row 5: 0 + 1, 1 + 3, 3 + 3, 3 + 1, 1 + 0 row 6: 0 + 1, 1 + 4, 4 + 6, 6 + 4, 1 + 0
So I think your directions are probably confusing people.
Of course, you still probably aren't going to increase your percentages much, but you could be clearer in what the problem actually is.
No, just take the obvious.
> Each element in Pascal’s Triangle is found by adding the two numbers to the right and left of it on the row above it; where these don’t exist, they are assumed to be zero.
Subject of the sentence: 'Each element'
So, "adding two numbers to the right and left of it on the row above it" means "adding the two numbers to the above right and above left of an element."
> So I think your directions are probably confusing people
Not my directions!
Still, this thread should be useful to OP. There are people who get the problem, and there are people who don't get the problem, and smartness isn't the deciding factor.
(I'm stupid, and not a programmer, but I understood what the problem was and what the desired result would be. Other people are smarter than I am, and are programmers, but didn't seem to grok the question.)
And, for the record, I'm a programmer, think I'm reasonably intelligent, and still not entirely sure I get the question based on the description. Language is tough sometimes.
P(i,j) = the number at i-th row and j-th column, could be defined as:
P(i, j) = P(i-1, j-1) + P(i-1, j+1) - for the indented pyramid diagram
or
P(i, j) = P(i-1, j-1) + P(i-1, j) - for the non indented triangle (the way it would appear in a basic 2D data structure, eg: int P[][]).
Also, you will have to check if P(i-1,j-1) exists.
No, it's a fairly straight forward and simple algorithm.
The only ambiguity is due to the fact that it references relative spatial positions...but this question always provides a graphic that makes everything completely clear:
If someone couldn't understand the question with that description and graphic, they are either having a "brain fart"/interview jitters or have some deficiency in their reading comprehension, spatial reasoning, etc that they need to work on. The former is perfectly understandable and will happen no matter what you do in the interview; the latter tells you that the interviewee needs work in some area which is useful information.> You aren't testing programming, you're testing obscure maths knowledge
Not at all, to answer this question requires absolutely no previous knowledge. All you need to answer is in the question.
You can pretty much just translate the question directly into code:
Took me maybe a couple minutes to translate the question into a fairly straightforward recursive function and type it out. From there it would be easy to improve in a lot of ways: caching, making it iterative, etc...all of which are relatively obvious and easy.I doubt there are any interviewers who wouldn't be basically satisfied with the most brain dead implementation. That demonstrates basic competence. The question has ample room to demonstrate a variety of skills beyond that. I'd expect the better interviewees (even with little math skill/background knowledge) to at least be able to make some observations about the properties of the triangle. I'd also expect the better ones to be able to talk about different approaches and their benefits/drawbacks.
For candidates with better math skills or background knowledge, they might know offhand/be able to write a better algorithm for computing the nth row. If the interviewer is expecting that as a baseline, sure I agree that's a bit unreasonable...but I don't think that ever happens.
I think this is a perfectly fine interview question with the caveats that it's very well known (and hence there will be some with a solution memorized) and one shouldn't rely on it too much.
It helps to have a good interviewer who will be able to explain the purpose of the question, and guide the interviewee if they get stuck. This is holds true for any interview question though.
For example, does the triangle already exist or are we generating it/calculating it? Are we to iterate over the table? And if so then how? What data format is it in?
Let's also discuss this line:
> Each element in Pascal’s Triangle is found by adding the two numbers to the right and left of it on the row above it; where these don’t exist, they are assumed to be zero.
The way it is worded is just horrible. Try this one:
"At the tip of Pascal's Triangle is the number 1, which makes up the zeroth row. The first row (1 & 1) contains two 1's, both formed by adding the two numbers above them to the left and the right, in this case 1 and 0 (all numbers outside the Triangle are 0's). Do the same to create the 2nd row: 0+1=1; 1+1=2; 1+0=1. And the third: 0+1=1; 1+2=3; 2+1=3; 1+0=1. In this way, the rows of the triangle go on infinitly. A number in the triangle can also be found by nCr (n Choose r) where n is the number of the row and r is the element in that row. For example, in row 3, 1 is the zeroth element, 3 is element number 1, the next three is the 2nd element, and the last 1 is the 3rd element. "
With this diagram: http://www.mathsisfun.com/images/pascals-triangle-1.gif
I've studied programming sure; anyone who's studied programming should have no problem with this. In terms of this particular problem, not really. I've come across it before, and understood it immediately then as well.
I think you are fundamentally misunderstanding the process/reason behind such a question.
> For example, does the triangle already exist or are we generating it/calculating it? Are we to iterate over the table? And if so then how? What data format is it in?
These aren't reasonable/good questions. That is, only a person who hasn't paid attention, misunderstood the question, etc would ask these. They represent a flaw in the reader, not the problem as it was phrased.
I think the issue here is you just don't understand the process and reasoning behind this question. You seem to think it's some sort of "gotcha" when that's not the case at all. There's absolutely no reason you can't ask questions like you pose. Everyone sometimes has issues with the way things are worded...and being able to ask for clarification is important.
I don't see how your explanation improves anything; it doesn't add any information it's just more explicit and convoluted. Many (I'd say most) programmers can solve the problem easily the first time they encounter it with no background knowledge whatsoever. I did it, as have many others. There will be many people who need a little help, that's fine...this is not a test to weed them out. It's to weed out the people who don't ask for a little help, or just can't do it at all.
Regarding the above problem, in my opinion it has everything you need for a good understanding. The only thing it's missing would be some limits for n.
If you have trouble understanding this task, it means you have trouble understanding basic algorithmic problems as most of them are defined in a similar manner.
To also, answer jlangenauer's question: this would be a perfect interview task for a junior. If you find many candidates that fail this kind of problems, you should seriously consider a way of better screening your candidates. (phone interview, online tasks, etc).
I'm not yet a great programmer, but no doubt I'm becoming one
Wanna hire me, for free :)
Location: SF,CA
----------------------------------------------------------------------------------------------------------------------------
my site: codefella.herokuapp.com
Did you say you're in MATLAB?
I am toying with doing a circuit simulator. I did a general purpose numeric integrator for mass-springs. Circuits are about the same. As I think about it, Laplace is not relevant because I want nonlinear circuit elements. I am curious to try solving circuits with Laplace, though. This is all for fun.
God says... masses union Togo continently depraved some enough_said parity largely ignoble disgusted I'm_done Courage STRICT
-----
If you do eigenvalues/vectors, it requires factoring polynomials. Factoring polynomials is a black art, I'm pretty sure. Scientific numeric computing is not for amateurs. MATLAB used a fortran LINPACK for a long time, just because that shit's hard.
I don't really care for interview questions that are unlikely to show up in the real world. This feels like sort of a "gotcha" question. Perhaps you will get the best of the best in your 10%. But I suspect you are missing a lot of candidates who are very good at real-world programming.
You seem to be paying average - average+10k (for all, not just senior) developer salaries, yet you specify senior programmer. At this level, you're only going to be attracting mid-level talent. There could a problem with your recruitment targeting, which could mean you're not getting ideal people in the door to begin with.
At the mid-level, you should expect a high failure rate on simple tasks. You're dealing with people who wound up on the job market after getting a bunch of experience. This means that a lot of them were fired, came from untenable/poorly managed companies which went under/downsized, or were working under such unbearable conditions that they quit. Employed devs trade up - unless your pay packet, benefits or feel-good factor (eg if you're an NGO, charity, working on 'real' problems, etc) give your company an edge, you're not dragging people away from jobs where they're encouraged to develop their skills, work under good conditions and understand software engineering processes. Job postings regularly ask for ridiculous and irrelevent things, so a lot of the employed devs who read your job ad and apply will not meet the criteria you specify. I know it's a huge generalisation, but the best devs are usually only on the market a couple of times in their lives, and they're quickly snapped up by companies at a premium and kept happy with various incentives. Tl;dr: you should be prepared for most candidates that you interview to not be appropriate for the job you've posted.
Thirdly, there's a problem with your, ahem, problem. It's not a great problem, imo because there's a fair amount of nuance to it. I'd want to shine by giving a good solution - vaguely remembering that a value in Pascal's triangle can be derived from combinations or permutations or something similar I'd spend some time trying to figure out if there's some algorithm "n!/(n+k)! - no, that's not it...", then I'd try to do it in-place calculating only half of the triangle, and then probably wind up using two rows of half length to calculate the numbers there. I'd be trying to impress you by coming up with a solution with more useful properties (memory boundedness, complexity, speed) than "returns the row of pascal's triangle". You're probably not even hiring people for their comp sci experience.
Maybe you just wanted to vent, or to advertise here - that's ok, but keep in mind that you're going to have to quickly reject unsuitable candidates and quickly pick up suitable ones, that it's just a numbers game, and that this is why most companies that offer mid-range compensation are "always hiring". The problem isn't too hard as such, it's perhaps a little undirected. Also, some people just interview terribly.
Dunno, just some food for thought.