Ask HN: Are my technical interview questions too hard?
Our funnel was 150 applications -> 75 potentially viable applicants -> 20 interviews -> 8 finalists.
I thought all 8 finalists would breeze through my technical interview but, in reality, 6 of them completely failed it. I'm trying to determine if I'm just horrible at sniffing out bad programmers or if my questions were too hard, bad, or both.
The format was a 1 hour in-person interview using a pre-setup JSFiddle to perform the following tasks. (I made it clear that asking me questions, using Google, using libraries like jQuery, or whatever else you'd do during a real-world scenario was perfectly fine). In the comments below I added details about why I asked certain questions and what I was looking for.
1) Create a 25x25 red circle using HTML/CSS
2) Place a matching green circle next to it.
3) Now make 10 balls in the HTML instead of 2.
4) Using CSS, make their colors alternate between red and green.
5) Empty the HTML pane and re-create the balls using JavaScript.
6) Now make it a 10x10 grid of balls.
7) Switch the direction of the stripes (from vertical to horizontal usually).
8) After a delay of 1 second, animate all of the balls' color to black using jQuery's animate function.
9) Now, instead of them all animating at once, animate them one by one.
10) Read the documentation for async.js' eachLimit function and use it to animate the balls to black 3 at a time and then once they're all finished, animate them all to blue.
11) Write your own eachLimit function that matches the signature of async.js' and use it to do the same task.
Most candidates were struggling by task 9. Only the 2 we hired were able to complete all 11 tasks.
54 comments
[ 3.8 ms ] story [ 134 ms ] thread1-7 - basic HTML/CSS/JS proficiency (and ability to effectively Google for help if they didn't know about CSS' nth-child or whatnot)
8 - get them familiar with the jQuery animate function (which I wanted to use later as a visual proxy for any async function) -- I was more than happy to link them to the jQuery docs or explain the parameters here if they weren't familiar with it.
9 - testing understanding of async functions and how to use callbacks. Ideally they used recursion but I would have accepted a calculated timing-based solution using setTimeouts as well.
10 - wanted to see how they'd go about learning something new given the documentation -- async.js' docs don't have great examples included so I was hoping to see them search for them on Stack Overflow if they didn't quite understand the callback formats.
11 - algorithm design and recursion. But mostly I thought people would get stuck here and I wanted to see how they dealt with that and what they did to get themselves un-stuck.
Your funnel also looks pretty normal. I wouldn't be surprised if all 8 fail the test for front-end, welcome to hiring in the real world.
My suggestion would be offload some of those basic questions to an automated at home code test. It'll filter out candidates who can't FizzBuzz, so you're left with a more reasonable 1 hour technical interview with less pointless candidates.
The timing though was a little ridiculous. You could, and perhaps SHOULD, easily spend an hour on 10 and an hour on 11.
IME, this is the biggest problem with the current state of the interview process - what I call Hollywood hacker testing. Unrealistic timing expectations based on what’s seen in Hollywood movies.
The take home test is worse for this! At least this wasn’t part of your process...you don’t want have a hiring process where the only person who can put in 60 hours wins.
Be realistic, if you really care about improving the hiring process and not being another dumping ground on Glassdoor for interviewees you’ve angered.
As for the questions themselves, I'd say that they are too finely graduated on the low end, and not finely graduated enough on the high end. This means that you ended up in a situation where you can't tell if one of your candidates is better than the other(s), while you can tell a lot about the relative skill levels of the people you didn't want to hire. You want more discrimination at the high end so that you know which of your candidates you would like to hire, and which you must hire.
The other thing that you could do is just be willing to hire junior engineers who need training, alongside the senior engineers who can train them. Anyone who got question 8 correct, for instance, but not the rest, could have been a good choice for this role. They would likely have learned a lot over the next year or two, and then you'd have a lot of capable engineers.
e.g. i imagine the largest cost is going to be onboarding the people that you've decided to hire, not the stages in the preceeding application/scheduleinterview/interview/techinterview/offer steps, so however you fiddle with the funnel it's probably not going to make the process 2x as cost efficient, but there is some risk that the funnel is perhaps selecting for the wrong people .
One of the things I struggle with is what you mentioned above -- what's the opportunity cost of losing people who have a higher ceiling / learning velocity but less immediately applicable experience.
I don't know how to measure that before hiring someone though. Is there a good way to assess that?
I probabbly couldn't do many of those (well I could some) off the top of my head, but if I can google, yeah I can do that. I know enough to get most of those done.
Heck I'll try them on my own sometime.
You should look at a people deeper. Listen to them more carefully. Try to understand what they are interested in. You don't need their resume nor tests for that. If you feel one is interested by programming, hire him immediatly and try to work with him. If you were wrong, assume your responsability.
If you can't do that, send me your candidates. I'll tell you who are the goods.
A lot of people are "interested in programming" that doesn't make them a good hire, or that they'll ever do the job. I think the interviewer wants folks who can do the job, now.
A lot of people say they are interested in programming but very few are. Programming is difficult. More and more people do that for the social position. They should do something they are good at. Most of them who persevere in computers become managers.
If a recruiter isn't able to evaluate someone by talking, he shouldn't recruit people.
I'm also not sure what you mean by "ask for money to pass a test".
As much as I'd like to hire every person interested in coding, unfortunately that would be detrimental to our core mission of building a product. And in the medium-run it'd be bad for everyone anyway because we'd just go out of business.
I definitely learned I can't evaluate people's technical ability only by talking... which worries me. Because prior to this I expected that I would have been able to predict who would be able to code and who wouldn't. Some of my "favorite" candidates before the technical interview wouldn't have been able to complete FizzBuzz.
If you can't evaluate if you can trust someone (for something), how can your business be successful? Relationships are everything in trade.
Embrace the risk and pick someone. The one you trust the more.
It ought to give you a bit of self reflection if you realize that your solution to the problem you faced has been pretty sub par.
Ask specific, probing questions to assess knowledge in your ideal areas, in some combination with an open ended problem solving question.
For this position we need people who can actually write code so testing coding ability seemed relevant. Hiring someone who can talk about code but not write a for loop or if statement wouldn't have been a good hire.
Make the test 5 tasks long.
- make a 10x10 grid of circles, center it in the page horizontally and vertically
- alternate the colors of the circle
- have them switch direction
- animate, etc.
The reason I did it that way was to give several easy "warm up" questions at the beginning to get them comfortable.
I like your idea of combining several of the first steps together though -- I may try that and see if that's more or less stressful for the candidate.
It might not be the questions but the rest of the process.
I figure I can teach someone the Node-specific stuff if they have a solid foundation. Similarly, I think I can teach almost anyone HTML/CSS so I wasn't too concerned about those parts either; they were mostly just necessary as setup for the later questions.
The things some people tend not to ever be able to fully grasp are recursion and async programming which is why I wanted to make sure to include those.
If this is really a JS role, a Node developer could potentially be an excellent fit with a little CSS and jQuery refresher but would fail the interview miserably.
Discussing how to approach the more difficult questions was a big part of the interview and the thought process behind how they approached it was more important to me than coming up with the "right" answer.
We didn't have any exclusively-Node developers in our finalists (I would have loved that though; Node is my jam) but for those that had less JS-specific experience we gave them a couple of weeks to brush up on HTML/CSS and pointed them towards refreshing their memory on relevant JS topics (even going as far as buying a few of them the JavaScript the Good Parts book to read before the technical interview).
- You said it's a 1 hour interview. It's really more like a 30 minute interview, because the interviewee will likely spend at least half of that time (probably more) thinking about how you're judging their behavior rather than focusing on the tasks. A good interviewer can prevent this sort of thing from spiraling out of control, but it's never eliminated completely.
- #2-#4 are redundant. If the interviewee can create a red circle, they can create more than one of them with a different color. Interviewees will underestimate their time management, so they'll take longer on these trivial tasks. They might even overanalyze what you're trying to do here, thinking that there's more to what you're looking for and spend longer on something trivial
- Questions like #3, #5, #9 are the kinds of questions interviewees will overanalyze. They might wonder what you mean, or whether there's more to what you're looking for. They can ask questions, sure, but you expect X and you won't tell them that you expect X, and they'll spend stupid amounts of time wondering what your X is. It's not an exaggeration either. Sometimes interviewers implicitly expect certain things without making them explicit. And you might only find out later (maybe even never), when they tell you that they wanted something that you definitely could've provided but the interviewer never specified. Or worse, you might spend additional time doing X and then they actually wanted Y. Interviewees will try to think about and tease out these things, which takes away substantial time from solving the actual problem
- Get rid of delay of 1 second for #8, this just slows down interviewees and doesn't give you as an interviewer any special knowledge about the interviewee
- #9 and #10 are basically the same question, get rid of one of them
- Use common sense. You probably expected that most of those 20 people should breeze through that interview. If they get something wrong or get confused about something, don't necessarily filter them out -- Think about the weaknesses in your own evaluation before doing that. Interviews are guidelines not hard rules, don't make the process kafkaesque
- I'm glad you're asking this sort of thing, because people should absolutely care a lot about false negatives. The 4 that made it might be the best ones, they might be mediocre candidates that were just good at your particular brand of interviewing. You want the best people on the team, not the people that are best at passing some weird interview that has little to do with their job
i'm not a frontend dev , but having an 11 step technical question in only 1 hour seems a bit excessive. i'm more used to having a single technical question in one hour - with a few facets and some asymptotics, with an optional harder extension question for the less common person who can get a good solution to the first problem rapidly.
it kind of depends on what you want for the role --- do you have a bunch of boxes in the org chart where you want people who are intermediate-ish and can hit the ground running cranking out new features? or do you want some people with really good potential and are prepared to invest a bit in training them?
instead of focusing on the technical interview stage of the pipeline, another option would be to try adjusting the earlier stages. What evidence do you have that the filtering from 150 applicants down to 8 finalists is not throwing away a bunch of potentially good hires? e.g. What would happen if you tried to run all 150 applicants through the technical interview first?
The technical skills are important, for sure, but we'd rather have someone less experienced who was motivated to learn because this was their dream job than someone with tons of experience that didn't really care about having _this_ job.
That's a good point about the timing; I did allow people to go over the time limit if they seemed like they were still making progress and needed it. Prior to the interviews I thought steps 1-8 (which were setting up for 9-11) would take about 15 minutes and we'd have the remaining 45 for the meat of the interview.
This gives two main advantages. The first is you have plenty of time for a conversation. Even assuming ideal conditions it takes between 2 and 3 times longer to walk someone through a solution as you implement it than just to incept an idea. Second is the interview platform is all ready stressful and deeply unnatural so building in some buffer for that necessary context switch helps you extract more signal.
That all being said, if you beat a 2% rate on your total funnel you're doing pretty well. Most people just aren't going to be a good fit; that's just the reality of this industry with huge choice for the candidate (you almost certainly will not be anyone's first pick; the numbers are against you), near limitless specializations, and no real qualification body.
10 is filtering out people who have anxiety about doing things in an interview that they don't yet know how to do -- if your work environment doesn't require learning something brand new while a manager looks over your shoulder, you may be better off having it as a homework assignment.
11 is fine but the people who froze on 9 and 10 because they didn't understand 9, don't like math, had their brain freeze up on them at the "I don't know what async.js is!!!" step, or something else don't get to it.
In addition, while I would be able to do this in an hour, the fact that there are 11 points may be pre-emptively stressing some people (some people get anxious at long lists of tasks, time-limited, that don't entirely fit in their working memory), and things like reading through documentation can take a bunch more time if someone is more thorough or less efficient at searching (prefers tutorials to documentation, has trouble googling "async.js" because of e.g. blog posts on ajax hogging up their search results page), if someone's eyes are tired (at this point they have been coding at least half an hour), and so on.
My suggestions would be, present the questions 1 by 1 not 11 at once, verbally in addition to having the current question in writing. At the looking up docs steps, have the link to the docs ready, or otherwise give a heads-up to the interviewees before the interview ("you may want to read up on async.js as preparation"). Otherwise, give the same exercise as homework.
After questions 1-8, if the candidate is satisfactory, say "great job! I only have a few more questions", thank them for their patience, generally validate their good answers. Make them feel like you're asking for their help vs. just evaluating them.
The questions flow pretty naturally from one to the next since it's mostly building on the previous step each time.
And yeah, I tried to make them feel comfortable and less like this was a test and more like a collaborative coding exercise. Wanted to try to make it as low stress as possible but I know that's hard given the interview environment.
Re 9 - I was expecting people to use callbacks to do it rather than computing timeouts (but a couple people did try to compute the timeouts and that was a fine solution as well -- asking people about why they chose what they chose led to helpful discussions). Something like this is what I would have done:
Mostly confusion between calling a function and passing the result as a parameter and passing a reference to that function. But also struggling with the recursive nature of calling a function from inside of itself.
I should also say it wasn't the syntax or JS/jQuery-specific aspects of it that were the problem. Most couldn't explain in English/pseudocode what their strategy for solving the problem would be.
If it's an entry-level role and you don't have good signals for the steps prior to the interview, maybe consider a pre-interview assignment that includes some asynchronous concepts, and administer it much earlier in the pipeline. That's a slightly lesser investment but should be as good a signal.
If it's a senior role, then your questions are way too easy, but also you should go on Linkedin and ask individual people with good experience to apply, that's what everyone else is doing.
Edited to add: Well, I saw in your other answers that you are filtering candidates based on "really wanting" the job very early in the process... I work in a region that is not the hottest job market wise and even here no one could afford to do this. Software devs are just too in demand... you're going to get pretty mercenary candidates unless you go with personal referrals. If the non-technical aspects of the candidates are more important, which with your ordering of the hiring pipeline it sounds like they are, then you need to consider hiring fewer people and growing less fast, pr hiring for fit first and training the recursion skills later.
it's not that the questions are too hard, it's that the whole challenge is bogus. if you have to ask these questions, make it a take-home challenge and discuss the implementation with people when they come to the in-person interview. being able to explain the hows and whys of already-written code is much more valuable than being able to write code under pressure, and much closer to what someone would actually be doing at work.
even better would be to figure out what exactly you're looking for in candidates and design the interviews to reflect that. right now it seems like you're looking for people who are good at coding under pressure... is that a normal part of the job? it very well might be, but i doubt it. think more about what you really want from people on a day-to-day basis.
4 - Somewhat advanced IMHO, and how often do you need this?
5/6/7 - Tedious.
8 - Okay, showing some knowledge of JavaScript. 9 - Showing the ability to write a control structure.
10/11 - Showing the ability to interact with the DOM and modify someone else's code.
Why would you give this code test after the interview? Why not give it to the potentially viable applicants first?
What kind of salary are you looking at? Anyone over 2-3 years of experience should be able to answer most/all of these questions.
If we're going to spend a ton of time onboarding and training someone we want them to be happy with what they're working on so they're more likely to stick with us longer term.
It was surprising how many naked resumes we received with no cover letter and no indication that the candidate even knew what company they were applying to.
Good people do not need to apply anywhere to get well-paid jobs. You want to poach people, find people from open-source contributions or your network recommendations.
*test results in a foreign, artificial, high-stress environment, using an IDE no one uses for normal work.