"All existing interview techniques are bad because they're just about proving the interviewer is smarter than the interviewee. Now let me tell you why my interview technique is so much better and I am so much smarter than those people."
Substitute kinder and more considerate for that second "smarter". I've had successful interviews where I felt like garbage and unsuccessful ones that left me feeling good about the experience. Both kindness and consideration override a waterboarding about the decisions behind variable names.
Eh, maybe bragging is too strong of a word, but it did feel rather hypocritical that he propped up the straw man that interviewing was all about making the interviewer feel smarter than the interviewee. Then proceeded to tell an anecdote about how he was smarter/better than all the other people interviewing a particular candidate.
I didn't get that from his writing. Yes, you can look at it that way but it's perhaps cynical. I thought he was genuinely trying to say here is a better way.
You should read it again. He's talking about kindness and giving people space to come out their shell (amongst other things). That's not him bragging, that's him helping the interview go well (if it can).
100% honest AND 100% correct. Many of the best developers I've worked with underestimated their abilities, and many of the worst overestimated.
Experience can be similar too. Many of the best see themselves "merely" as an important contributor to a team effort and remember the mistakes they made, etc. Many of the worst honestly believe they did a great job, led the effort, etc.
Interviewing programmers is easy. Get them to talk about something they've done that they were proud of and let them explain how they did it into the deep details.
Good programmers really enjoy talking about the details of the technology and how they solved challenging problems. You'll be able to see them geek out a little bit and see that they really enjoyed getting to do it.
Outside of baseline, core competence, that is the litmus test for hiring good programmers because it shows level of interest. Level of interest leads to continuous learning, which is essential to this field. Unless you have some ego-driven reason why you want to believe that YOUR stuff is too complicated for them to learn of course...but that's usually more of a reflection on you than them.
I realized this about 5 years ago and I've not made a single bad hire with the approach since. 100% success rate at this time.
A good resume is a trick. A trick designed to get an interviewer to ask about something on it. Because every item is something that you can talk about with passion and detail for as long as you want or need.
>Good programmers really enjoy talking about the details of the technology and how they solved challenging problems. You'll be able to see them geek out a little bit and see that they really enjoyed getting to do it.
I've had this at several companies. I think I've always gotten past this litmus test in one of the interviews, be it on the phone, via Skype, or in person. The problem, for me personally, is the inevitable followup interview consisting of "go to the board and scribble out a function that does X", and then of course "after you're done, I'll add important details and I expect you to be able to modify the function to accommodate as if you were using a text editor".
I admit it, I fail at the "get in front of the board" problems. I'm downright terrible at them. I have no problem with the conversational interviews. I'm not bad at presenting tech to my peers at meetings and book clubs, but in an interview environment I'll freeze up and, at best, come up with a very inefficient implementation. I hate it and it has probably cost me opportunities at places where the rest of the interview felt like it went great.
Yes I know what the Big O is, I know what the Big O of the optimal implementation would be. I can talk all about hashes, final, static, or whatever you think you'll trip me up on. But when I'm up there writing code, every little mistake psyches me out more and more, especially I can't just hit "backspace" to fix it.
At my previous job, after the phone interview they sent me a programming project to implement in a day or two. After submitting it, they had me go to an interview and talk about the solution with a group of engineers, followed by a short conversational interview. That project was my first time writing ruby and I got the position and have great recommendations since leaving the place.
Makes me wish that recommendations held more sway, or that more companies would do more of this "here's a task, send us the solution when you're done with it" type of thing. Dishonest candidates would easily be exposed when they're forced to discuss the solution in depth.
Of course, upon walking out of the building, all the answers magically pop into my head...
I've tried this in the past when hiring for a startup in two different ways: one with having the entire interview based around simply discussing past projects and gauging the level of interest; the other combining this type of discussion with a coding problems or two.
The result: the correlation between passion and competence is not 1.
Initially I wanted to give them the benefit of the doubt and hire primarily on talk alone, however I found that some people are passionate and interested yet can't code very well. After having to fire folks because of it, I decided to actually make them write code during the interview. Sure enough, I discovered a few candidates who could talk the talk, but when faced with an actual problem* couldn't actually solve it.
* When I give interview problems I like to actually give problems that I have faced in my job.
And admittedly, that's true. I suppose I shouldn't have let that come off as a purist "only ask this" situation. You need some technical problem that they can solve based on the skills you need and the skills you can judge. For those types of problems, I tend to focus on watching HOW they get to a solution and why rather than pure syntax.
I shouldn't have represented my original comment as the only thing necessary. Getting a mainframe guy to geek out on me for an Android position is probably not going to be all that beneficial.
Being able to talk about past projects is not passion, it is simple professionalism. How can you work in a field for long without being able to talk intelligently about it.
I also don't find testing for actual coding skills to be a problem as most candidates don't make it to the interview if they don't have some sort of portfolio of code samples.
A portfolio of code samples is not always available. Juniors often don't have code samples; neither do people who have only really worked on proprietary projects. I don't like to discount potential good developers just because they spend their free time doing things other than coding.
> spend their free time doing things other than coding
This is a valid point. A good work/life balance is critical. But to counter it, don't you want to look at candidates who _want_ to get hired, so they've spent a little extra time writing something for potential employers to look at?
A developer who gives up at "I don't have code samples" isn't someone I would want to hire.
If they didn't want to get hired, they wouldn't have applied for the job. Assuming that "I don't have code samples" is equivalent to "I don't care" is a silly assumption, and one that I've found is completely false in reality. Some of the developers I've hired have years of experience but no code samples, and after hiring them I discovered I had made the right choice. They just have other priorities on the side (children are often a big one).
Industrial psychology has been over this ground many times.
Prove they are generally smart, do a work sample test representing the work they'd actually be doing, for a kicker test their integrity. You'll have > 50% wins.
> Good programmers really enjoy talking about the details of the technology and how they solved challenging problems.
I don't really enjoy talking about anything and usually do so as a necessity. I guess I don't fit in with the extrovert engineers people are looking for. I also contend that not being able to talk well about how you did something doesn't mean that you aren't good at it. Talking about doing something you are good at is a different skill. Being be able to communicate is important sure. Is it the number one thing you're looking for? And candidates have to be happy about the ordeal too?
Actually, effective and reliable communication is the number one thing most all companies are looking for in engineers. No one wants a developer that is resistant to daily peer cooperation. A developer that can't clearly explain their decisions to their stakeholders is just about useless. A line item about communication is almost always present on a dev job listing's requirements, much more common than any stack/language flavor of the month.
Code is easy, being able to communicate is not. We engineers love to loathe non-tech people who talk too much while producing nothing tangible. Even more though, effective engineers loathe their peers that don't talk enough.
Isn't there something you've done in your career that you take enough pride in that you'd enjoy explaining it to people who would appreciate it (not including that time you gamed an interview)?
I don't enjoy doing anything with other people, but it's a necessity given that we need jobs to survive. I could be happy just hacking on something in a basement and never talking to a soul if I could. I can wear a smile if you need me to, but I definitely wont mean it. Sorry.
what about new grad? most of them only have school projects/assignments that involve the fundamentals, so I can see how the "coding interview" is a decent method here
A new grad should certainly be able to talk about their many school projects. Took an OS class and wrote a micro-kernel? I'd LOVE to hear about it! You enjoyed algorithms class? Let's talk about that!
New grads are different. For that you're usually hiring more for an entry level position with the expectation that there will be a lot of on the job learning.
In my experiences this is generally not true, bad programmers cannot talk the talk.
I've found the problem is sometimes missing good candidates because that particular one found you intimidating in some way. You can try to mitigate this some by having multiple interviewers to try to find someone on your team they can relate to. But really even this tends to work in your favor, because if they don't click with anyone on your team they probably won't do well even if they are good.
I'm defining bad as someone who knows how to write code but has been doing it "wrong" for so long and they're not really willing to unlearn their bad practices.
I think the comic sums up pretty well how stupid and broken large parts of our industry are--99% of the time we're fighting dumb things like bad CSS or sudden design changes or something else.
It really makes me wonder if our tools are broken, if we're all trying to solve the wrong problems, or if we simply are hiring too many people for whatever the project is.
For me, nothing else comes close to evaluating talent.
I prefer making my candidate comfortable and writing something simple. 15 minutes, privacy, a drink, and pencil & paper. I hate white boards for this.
I come back & see what they've done. It doesn't have to be perfect. It doesn't even have to be right. I am less concerned about their code than our discussion about their code...
How did you approach the problem?
Why did you name your variables like this?
What is that supposed to be doing?
What happens if...?
What would you change if I added <x> to the problem?
How else could you have done this part?
Why did you choose your way?
15 minutes of coding followed by 15 minutes of Q & A about their code tells me alot. Not everything, but alot. And no programmer, no matter how nervous or introverted, should have a problem with this. As a matter of fact, most of the best programmers I've interviewed have embraced and enjoyed this.
Fizzbuzz is a great problem for this. I'm still amazed by how many "experienced" programmers can't do this.
I'd love to work for a retriever or pointer. Herders or guard dogs would probably be okay, too. Terriers would be nightmare bosses.
I can only assume that I have never noticed these putative canine job postings because I have been looking (or smelling) in the wrong place. At the interview, should I attempt to alter my personal odor, or leave it as is? Will I be asked to sketch out an implementation of "disc fetch" on the whiteboard, or will I be given an actual flying disc and a long hallway? Should I remove experience working for cats from my resume, or leave it in?
He/she didn't say he/she wouldn't code during an interview, just only made a commentary about his/her reaction if expected to do so with pen and paper.
I think I might walk out too, because it's clear that power will be exerted on any candidates who choose to subject themselves to it - it says a lot about the type of work environment that is fostered.
I've had to code for interviews where I was given a project to complete in my own time, and then talk about it later if it was a good solution. I aced that process and got the job.
I bombed the whiteboard/paper tech interview when I was first invited to interview at my current job (lots of ex co-workers here), and was declined as a candidate.
Months later I was invited again, this time just to start, skipping the interview. I'm finishing up my last week here and have nothing but praise and recommendations from my higher-ups, yet I would have never had a shot if people I worked with in the past didn't work here and reach out to me.
Actually at one of the places I recently interviewed, I got past the phone screens, cultural interviews, and even a google hangouts/screen-share session where I coded live while talking about the solution -- but I still had a 'get on the whiteboard and write some twist on academic sort functions' and of course froze up more and more with each time I used the eraser. I'm confident I would have done better with a computer.
I don't code on paper and whiteboards. Most of us aren't trained to, and state-dependent learning is a real thing. You cannot expect most people to perform as well in a drastically different environment from how they typically do their job.
Bad bad bad idea. There is nothing more off putting than to be asked to dance like a monkey and write code with someone over your shoulder.
Ask for references, read his open source code and talk to him over lunch about some of the stuff he's worked on. Someone who doesn't know how to develop software will be easy to spot.
1. No one's looking over his shoulder (re-read GP).
2. If you think coding = dancing like a monkey, maybe you should do something else.
3. I've done everything you suggest, but the interview test is much more effective. You can bullshit through your suggestions (references are a joke), but not mine.
I never know how to approach this question, because often the answer is, "the problem is so simple and general to the point that all variable names are either academic (solved computer science problem with technical name for each concept involved) or hopelessly generic ("counter").
When I code things that actually do things, I just name things what they should be called. I think functional programming makes this glib prescription much more tenable, as many of the things you write are abstract of the implementation, so you never find yourself naming something "screen_refresh_counter_for_process_x."
Maybe so, but if that's the case, isn't the entire process a case of unspecified, morphing requirements?
I'll flatter myself and assume you liked what you saw in what I wrote. But if that's the case, why code at all? Why not just sit down and have a short conversation?
I feel like there's something about a rigorous, test-of-endurance interview process that feels professional. "We can't afford to hire anyone but the best."
So experienced hackers with good instincts end up ignoring them in favor of a gauntlet they know is not representative of actual work.
> I'm still amazed by how many "experienced" programmers can't do this.
... can't do this in your incredibly artificial, unrealistic context.
I hate live coding in an interview, especially on paper or whiteboard. I know you say you don't care about syntax and little stuff, but that's not what happens in my brain. I suddenly forget the most basic things. "Wait, do I use semicolons at the end of lines in Ruby? Aaaah, I've been using Ruby for 5 years. How can I not know this. I'm going to bomb this interview. I should just leave now."
I know it sounds ridiculous to you, but something very similar actually happened to me. Give me a block of time and a sample problem to work on at a computer, fine. Ask me to refactor code that already exists, whatevs. I'll happily write and discuss code. Just don't give me that blank piece of paper. My brain goes crazy.
And the funny part is that I've been told one of my best qualities as a developer is my calm, thoughtful performance under pressure. When somebody accidentally drops that table in production or both load balancers go down, I'm your guy. I'll get it fixed quickly, without a lot of panic and fussing about. Just don't give me that piece of paper in an interview.
I can't explain it, but it's true, for me. Maybe it's just that I deal with real crises often enough to have developed a mental muscle for it, but I interview so rarely that I don't have the patterns yet. Either way, I think you're missing out on great candidates with this approach.
Sure, fizzbuzz is trivially easy – under normal circumstances. But some people find these types of exercises irrationally difficult in the context of an interview, especially when coupled with the artificiality of writing it out by hand on paper.
> For a professional programmer, anything less is just an excuse or an attitude problem.
You're entitled to your opinion. I disagree. I think you're projecting your personal experience of the world onto other people, which is your prerogative as the interviewer. However, speaking from personal experience, I know some people who would otherwise make excellent teammates, top contributors, etc., will fail this test, and many who pass this test will make terrible employees.
As I said elsewhere, you will get many false positives and some false negatives. Thus, I think it's a bad test and a waste of time.
... can't do this in your incredibly artificial, unrealistic context
Oh, then you're a non-hire (rightfully so). If you can't adjust to such a simple context, there's no way I can be sure you can handle an occasional tight deadline, urgent and critical failure in production, rewrite of stinky and hairy piece of code left by a less competent colleague, external system that mystically fails for no reason and writes no logs whatsoever, etc, ad infinium.
Programming profession is hard and full of rough edges, unfortunately.
If you can't adjust to such a simple context, there's no way I can be sure you can handle an occasional tight deadline, urgent and critical failure in production, rewrite of stinky and hairy piece of code left by a less competent colleague, external system that mystically fails for no reason and writes no logs whatsoever, etc, ad infinium.
How does your team manage to accomplish all these tight deadlines while having to use paper and whiteboards to rewrite stinky code? Do your production servers run A4 or legal? What's your cardstock uptime? I've had trouble switching to glossy for all but the most specialized of tasks, I'd be really curious to hear about your paper stack.
I think it is really pretty simple: in an unfamiliar environment and especially using totally unrealistic tools (paper or whiteboard), your "muscle memory" loses its memory and your thought process is totally derailed by thinking about the pen and whiteboard. In a familiar environment, the mechanical process of typing is subconscious and your brain can concentrate on the problem, and not get tied up thinking about how to write in straight lines on a whiteboard.
So it may not be perfect but I like approaching interviews in a similar way to the article. The questions leading up to and including #7 will let you know whether they can code or not so I'm not sure your #8 is really necessary.
No one I generally know writes good or relevant code in 15 minute increments. In my own experience, I generally get warmed up by the 3rd or 4th interview, and it's an acting class as I regurgitate stuff.
"What was your approach to the problem?" = I pretend to think for a few seconds, then answer. You're basically just figuring out if I've encountered your almost exact interview previously and I've got the canned answer ready.
Currently we have a timed programming exercise we let people do at home, then review and ask questions like this over email as a filter, it seems to work fairly well.
In person, I just like to ask people about their work experience, especially painful failures and learning experiences they've encountered.
Things like variable names are meaningless, imho. Every company tends to have an arbitrary set of rules they like to follow, unless someone is using things like 1a a1 11a1 a1a1 altogether where I can't read it, I've gotten over caring.
I kind of agree with you, point 7 is unrealistic, What problem could you give to the interviewee that could take a couple of days and that he/she will resolve it willingly and honestly. Yes it sounds nice but in practice I don't think so, totally agree with points 1-6 though
I got fizzbuzzed in an interview recently. I knew exactly what to do, came out with it in a simple way in python.
Then "how about without the modulo operator". The solution is so obvious - using the modulo operator, I had a mental block. It was fine once I got nudged in the right direction, but under normal circumstances I wouldn't have needed the nudge.
> Or write a function that takes an integer and returns that integer's index on the Fibonacci sequence. That last example: they wanted me to do this with pen and paper over the phone.
I fail to see what is the difficulty on that, looks like very straightforward and easy question to do, even over the phone.
I think the focus was on the fact that it is arbitrary. The person you are interviewing may not even know nor want to care what a Fibonacci sequence is. He is saying the goal is to understand the candidate and the way they think, not to trivia them about random stuff ( since it will put them down if they don't know it )
I've seen a lot of writeups about interviewing here on hacker news. Things I liked in this particular one:
- Admission that arbitrary coding challenges that are unrealistic to the applied job are ridiculous
- Recognition that you need to understand the engineer and approach things from their point of view, not from your own selfish view as the interviewer
- Urging that being polite and humble is important. Even if you don't like the candidate, there is no need to attack or diminish them.
One question I have: What is wrong with copying and pasting solutions to issues that have been solved before by others? I tend to do this all the time myself ( albeit I often end up rewriting the solutions to tailor them more to the problem )
This is the biggest thing I struggle with - people confuse interview anxiety on my part with lack of knowledge or experience. You immediately look like a junior engineer if your mind keeps going blank on technical questions (that you know the answer to) or if you downplay your own accomplishments. Interviewing feels like a crap shoot at times, because it's entirely dependent on me being able to force myself to be extroverted and confident.
However, I have a lot of respect for companies that do interviewing well (testing skill and experience in a less contrived and awkward manner), and the ones that have made the process as painless as possible tend to be well-run companies.
You can learn a lot from the process with each company, especially at an on-site. Dick-swinging contests definitely happen, which to me means that the company values rock star coders over programmers who are skilled, pleasant to work with and good at communication (and moves to the "won't accept offer" pile).
Because you don't have to write code on a whiteboard with someone staring at you in your daily work. You're free to sit alone, make a stupid off-by-one error, misspell a var without feeling like you'll be fired for it.
Technical discussions at work aren't so high-stakes as ones in an interview. If you do badly in an interview, you've wasted an entire day for nothing. If you fail to speak in a technical discussion at work, best case no one notices, worst case they think "Hm. Looks like Brian didn't get much sleep last night". Just the knowledge of these stakes can affect your performance.
I have pretty bad social anxiety, which is never a problem in my day-to-day work, where I know and trust the (awesome) people I work with. I'm more stressed having to meet a dozen people over a very brief period of time than I am fixing a failing production server, but I do perfectly fine at the interview as long as my anxiety isn't bad that day (by the 2nd or 3rd company I've spoken with, I start to get more comfortable with it).
I'm interviewing right now and all except one have lead to offers, so no, I'm actually good at my job. I'm just a better engineer than I am an interviewee.
One way which may help with the interview anxiety is just to interview more. Even if you're not really looking right now, why not interview at a few places just to work on your chops? You could also role-play interviews with friends, family and co-workers. It may not remove the anxiety, but exposing yourself to the process a little more could really help.
I've had some issues with the whiteboard in the past. After I blew one interview (partially based on my whiteboard performance), I spent a lot of time coding solutions with a pen and paper, working on my penmanship (so that I could read my own writing) and just getting used to solving problems away from a screen. It really helped me get over a lot of my issues with that particular situation.
Thanks, I totally agree with your advice. I actually did buy a whiteboard a while back, and drilled a lot of algorithm problems, and it's helped a lot.
I got excited when I saw the cover of Thinking Fast and Slow. But he failed to mention what I found to be one of the most insightful parts of that book: interviewing.
He kinda touched on it at the end with #1: "Know what you're looking for".
Kahneman is more explicit in Thinking Fast and Slow: come up with a simple rubric of 5 or 6 traits you're looking for in your new hire and ask questions that allow you to score each candidate on a scale of 1 to 5 for each. When you're done, choose the candidate with the highest score.
Maybe the technical ability score gets double-weighted for developers. But it seems like a lot of the interview process (and discussion of it) fixates on this. Well this and, of course, vague impressions of the candidate's personality. In my experience, this misses other key things that make up a competent programmer and worthwhile employee.
In the spirit of Thinking Fast and Slow, here's a question. As an interviewer, what are the 5 most important traits or qualities you look for in a candidate?
I work at what can only be defined as a megacorporation. The interview process is a machine, and more often than not interviewers are not personally invested in the process as much as they could be - the candidate is destined to be on someone else's team.
This is of course a double edged sword - on the one hand you only have a generic sense of what you are looking for and on the other you have no vendetta for hiring someone who is sub-par to fill your urgent needs.
In every challenging software development job, you'll be solving problems you've never encountered before. That's the nature of software. Once you understand the problem, you move on to something else.
So, why is it such a stretch to ask people to solve Fibonacci? I don't understand this mentality. Reminds me of "why should I learn calculus? I'll never use it in real life!" from college. You're supposed to be a professional software developer, and should be able to handle an unfamiliar problem.
If you're a professional musician auditioning to play Mozart, you wouldn't complain that your audition piece is Beethoven.
> In every challenging software development job, you'll be solving problems you've never encountered before.
The problem is, the vast majority of software development jobs are not this. That was the point of the comic in the post. There is a whole lot of maintenance, API gluing, and spec work that needs doing. Even at the "elite" companies like Google.
The current trend in hiring seems to be at least partially born of Google's ill-advised[1] strategy of hiring generic software engineers that could theoretically be slotted into any role at the company. Hire for a position. If that position involves challenging, novel problems then throw things like that at the candidates in the interview. If it doesn't, don't.
[1] After they reached a certain point as a company
Interesting post. This author lives in Santa Barbara. Hiring in SB is vastly different than it is in SF. Due to the small pool of companies and talent, everyone knows everyone. So there's not much mystery to it.
The single best thing I've done when interviewing is ask people to bring a laptop with some of their code on it. I don't care the language o the framework. I don't care if it was school, side project, or even their current job.
It puts people on their home turf, where they are most confident and comfortable. They can show me what they are proud of. I can ask lots of questions about it and see their thinking without having to be a total dick asking for an esoteric algorithm on a white board. There's also a lot to be said for how well someone know show to use their tools. Are they navigating proficiently, or are they poking around the IDE like they just started out.
This sounds like a softball interview, but you would be amazed the number of people it weeds out. You probably need something more rigorous beyond that, but don't start with the hard problems when an easy filter can really give you a lot of insight.
(Note that generally you can't share your current job's code, but people are fairly comfortable opening their own laptop and poking around in the code. I never ask for people to send me that kind of code).
Let's talk about BRAIN FREEZE. And I mean the total, complete, 300-baud mush when some question --for whatever reason-- kills your ability to think.
I've had it happen twice, and man it ain't fun. The second was like an out-of-body experience. I forgot the question, I forgot my train of thought, I knew I wasn't responding, I knew that I knew that the interview knew that I was just sitting there, saying "ummmmmmmmmm" for about 147 seconds now. Yikes! Yet in day to day, these would have been easy questions. And TBH I didn't even want to job at that point.
It's critical that you as an interviewee learn to monitor and CONTROL your physiological response to stress. And I've learned that as an interviewer, I try to pay close attention to physical stress responses from the candidate -- flushed cheeks, signs of sweat, sudden lock-up -- when I see these signs, I back off and let them come back to normal first.
I find these whiteboard problems a breath of fresh air. Coming from an academic background, I've seen the downsides with the alternatives.
* Talking about something you've done? You're selecting for marketing, not programming.
* Letters of recommendation? You're selecting for ability to network, not programming.
* Academic qualifications? You're selecting for persistence, not programming.
* Offline tests? OK right now because they're a minority, but I guarantee if they became the standard, you'd get an astounding amount of cheating.
I'm not a marketer and I really resent how in academia to have any chance you have to "sell sell sell". I appreciate anyone who gives me a chance to just objectively demonstrate my abilities, whether I do well or whether I do poorly.
It's still marketing. "Something you've done" could be the 'is-negative' repo that was briefly on the frontpage today [1], and the candidate could launch into a heart-rending speech on the virtues of modular programming. It doesn't say anything about whether they can actually program. Or "something you've done" could be an incredibly sophisticated engine and the candidate isn't good at marketing and comes off worse than the first guy, despite being a top-rate programmer.
You're looking for an edge case to break a hard rule.
In an actual interview, when they are talking about something they've done and explaining it, the interviewer will be able to ask more questions and hopefully (as an interviewer) have enough of a technical background to separate the BS from the skills.
If two chemists are talking to each other about chemistry, one's not going to be able to easily just make stuff up that the other blindly accepts.
Well, I guess you might be right with the chemist example, if it's an aquihire type of situation. With something like Google though, I have a feeling these kind of talks wouldn't scale well (in general any time you say the word "hopefully", that's a sign you might be relying too much on luck). Maybe whiteboards for entry level positions, and a combination of whiteboards and fireside chats for senior positions?
>I'm not a marketer and I really resent how in academia to have any chance you have to "sell sell sell".
You have to do this everywhere. There are limited resources. You are competing for them with others. You need to convince a gatekeeper that you deserve those resources. In academia, you are competing for grant money, lab space, etc. on the basis that your work is of more import than that of someone else. In enterprise software sales, you are attempting to convince a customer that your product is a better solution to their business problem than of some competitor's. In an interview, you need to demonstrate that you can accomplish a function better than one of your peers who is also vying for the job, and probably has an almost identical skills profile to you.
Old fart here. My favorite interview question is this:
Tell me about a project, any project large or small, that was 100% you. You thought of it, you coded it, you wrote the tests, you documented it. The only other requirement is that the project has to have at least 10 users who installed, used, and continue to use the project without any help from you.
Lots of people don't have an answer. Some you need to coax a little, because a small project is fine. For example one that I did close to 30 years ago was move/copy utilities that used regexp. You could do
move =.fortran =.f
and it did what you wanted (first = is .* and second is $1 and they could repeat).
Anyone with basic C skills could have done this (or perl, I think someone else did a perl version and that won). So people would think a project like that doesn't count, but it does. Small is fine, you just had to do the whole thing and people needed to be able to use it without asking for help.
This covers a lot of ground and my experience is if you get lucky enough to find someone who has done projects like this, and there is at least some overlap with what we are doing, yeah, we got lucky.
I've noticed that there is a great number of interviewers that do not take the initiative to just be nice. I suppose they just forget that the interview process is a two-way street.
One thing I find annoying is that interviewers expect you to think out loud "continuously". If I stop talking for 30s or 1 minute, I am in trouble. That is not applicable to everyone. I myself feel more confident to think in silent (may be 5-10 minutes at least) before I can tell interviewers my whole train of thought. Speak out loud what I'm thinking could easily interrupt it!!! Everyone is different!
I feel like Notch because I know I will never create something as popular as my first comic. I put so much more creativity and effort into writing this post but only a few hundred people read it.
Interviewing is actually very easy. Every job I post gets 100s of applicants - so finding them is not a problem. I've got a piece of software that lets me A/B these candidates resumes & profiles side-by-side so I rank-sort them quickly. And all "interviews" are small-boxed projects that I ACTUALLY PAY FOR THE WORK.
See, most interviews are along the lines of:
Q: Are you smart and good looking?
A: Yes!
This is why I don't really intervew all. When I take the "best" ranked applicants I give them actual work; small slices from our actual code base. I provide them a complete, functional dev environment with one restrciton: cannot commit to git. I can see their work directly, I work with them directly and get real-time data on what it's like to work with them, how they handle a project, how they communicate blockages, how they figure out our existing systems.
In four to eight hours of work I know who is the right fit for our team. My cost to find and filter is very very low and rather than waste 2-4 hours per interview for 10 candidates I pay for 16-24 hours of work for 2 or 3 candidates. The hard costs are the same, it consumes less of my (or my engineers) time and the candidate fit is very good.
See, most interviews are along the lines of: Q: Are you smart and good looking? A: Yes!
In my current company we hire a lot of developers through contracting companies. These people often have very difficult-to-read resumes - mostly because they are trying to get through filters that work based on searching for keywords, but they're rarely well-written documents even within this constraint. Fine, we're not looking to hire a writer.
But a lot of these have "Communication Skills: 9/10" on them. Whether this is added by the candidate or the contracting company, I don't know.
I really liked the cartoon. Clever, and insightful - we go through very difficult and interesting problems that are often unrelated to the job.
I disagree a bit with the center pane, though (with the dog typing loops and the caption "I have no idea what I'm doing"). It is, in my opinion, very impressive to be able to find the largest sum of all square matrices in an NxN matrix at a whiteboard in 45 minutes. It's impressive enough (especially not knowing the exact question in advance) to be a positive indicator of the ability to do what is in pane 3, especially considering how many variables and logical basket weaving is often involved in moving some text a few pixels to the right. As long as you are willing to tolerate a very high false negative rate, this could be a good way to go.
Unfortunately, it does mean that most developers looking for jobs will need to intensely study data structures and algorithms, and walk around with them loaded in "exam ready" memory" in a way that few people would need to simply to do their jobs. Personally, I'd say most people would retain this mental state anywhere from 1-3 weeks after a college algorithms course, provided they were a good student who understood the material. And it also means that lots of developers who could absolutely do this if they were willing to drop everything to study 50 or so hours and train at a white board will not be hired because they were more interested in just doing their jobs (which, ironically, is what they'd be doing at the new job). So we are going through a very inefficient re-taking of exams every time we interview. So I'm not surprised that some people have decided to stop interviewing, or just give up on finding a job that requires this sort of interview.
It's up to a company to decide if they want to give up on hiring these developers, or to tolerate this kind of false negative rate. However I have close to zero interest in hearing about how hard it is to hire good developers if this is their approach, since it suggests to me that the aren't going to make any effort to find a way to avoid these false negatives.
102 comments
[ 2.4 ms ] story [ 175 ms ] threadYou should read it again. He's talking about kindness and giving people space to come out their shell (amongst other things). That's not him bragging, that's him helping the interview go well (if it can).
Shrug. I thought it was a good article.
On desktop found this app to adjust contrast per-website:
https://chrome.google.com/webstore/detail/high-contrast/djcf...
Experience can be similar too. Many of the best see themselves "merely" as an important contributor to a team effort and remember the mistakes they made, etc. Many of the worst honestly believe they did a great job, led the effort, etc.
That would be the Dunning-Kruger effect in action https://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect
Good programmers really enjoy talking about the details of the technology and how they solved challenging problems. You'll be able to see them geek out a little bit and see that they really enjoyed getting to do it.
Outside of baseline, core competence, that is the litmus test for hiring good programmers because it shows level of interest. Level of interest leads to continuous learning, which is essential to this field. Unless you have some ego-driven reason why you want to believe that YOUR stuff is too complicated for them to learn of course...but that's usually more of a reflection on you than them.
I realized this about 5 years ago and I've not made a single bad hire with the approach since. 100% success rate at this time.
I've had this at several companies. I think I've always gotten past this litmus test in one of the interviews, be it on the phone, via Skype, or in person. The problem, for me personally, is the inevitable followup interview consisting of "go to the board and scribble out a function that does X", and then of course "after you're done, I'll add important details and I expect you to be able to modify the function to accommodate as if you were using a text editor".
I admit it, I fail at the "get in front of the board" problems. I'm downright terrible at them. I have no problem with the conversational interviews. I'm not bad at presenting tech to my peers at meetings and book clubs, but in an interview environment I'll freeze up and, at best, come up with a very inefficient implementation. I hate it and it has probably cost me opportunities at places where the rest of the interview felt like it went great.
Yes I know what the Big O is, I know what the Big O of the optimal implementation would be. I can talk all about hashes, final, static, or whatever you think you'll trip me up on. But when I'm up there writing code, every little mistake psyches me out more and more, especially I can't just hit "backspace" to fix it.
At my previous job, after the phone interview they sent me a programming project to implement in a day or two. After submitting it, they had me go to an interview and talk about the solution with a group of engineers, followed by a short conversational interview. That project was my first time writing ruby and I got the position and have great recommendations since leaving the place.
Makes me wish that recommendations held more sway, or that more companies would do more of this "here's a task, send us the solution when you're done with it" type of thing. Dishonest candidates would easily be exposed when they're forced to discuss the solution in depth.
Of course, upon walking out of the building, all the answers magically pop into my head...
The result: the correlation between passion and competence is not 1.
Initially I wanted to give them the benefit of the doubt and hire primarily on talk alone, however I found that some people are passionate and interested yet can't code very well. After having to fire folks because of it, I decided to actually make them write code during the interview. Sure enough, I discovered a few candidates who could talk the talk, but when faced with an actual problem* couldn't actually solve it.
* When I give interview problems I like to actually give problems that I have faced in my job.
I shouldn't have represented my original comment as the only thing necessary. Getting a mainframe guy to geek out on me for an Android position is probably not going to be all that beneficial.
I also don't find testing for actual coding skills to be a problem as most candidates don't make it to the interview if they don't have some sort of portfolio of code samples.
This is a valid point. A good work/life balance is critical. But to counter it, don't you want to look at candidates who _want_ to get hired, so they've spent a little extra time writing something for potential employers to look at?
A developer who gives up at "I don't have code samples" isn't someone I would want to hire.
Would you hire a graphic artist, a writer, a painter, etc. without previewing their work first.
Prove they are generally smart, do a work sample test representing the work they'd actually be doing, for a kicker test their integrity. You'll have > 50% wins.
http://mavweb.mnsu.edu/howard/Schmidt%20and%20Hunter%201998%...
I don't really enjoy talking about anything and usually do so as a necessity. I guess I don't fit in with the extrovert engineers people are looking for. I also contend that not being able to talk well about how you did something doesn't mean that you aren't good at it. Talking about doing something you are good at is a different skill. Being be able to communicate is important sure. Is it the number one thing you're looking for? And candidates have to be happy about the ordeal too?
Code is easy, being able to communicate is not. We engineers love to loathe non-tech people who talk too much while producing nothing tangible. Even more though, effective engineers loathe their peers that don't talk enough.
Bad programmers also enjoy talking about that. They can make it sound like it was an awesome project.
You can't tell until you actually look at their code and realize it's all a spaghetti mess.
I've found the problem is sometimes missing good candidates because that particular one found you intimidating in some way. You can try to mitigate this some by having multiple interviewers to try to find someone on your team they can relate to. But really even this tends to work in your favor, because if they don't click with anyone on your team they probably won't do well even if they are good.
It really makes me wonder if our tools are broken, if we're all trying to solve the wrong problems, or if we simply are hiring too many people for whatever the project is.
For me, nothing else comes close to evaluating talent.
I prefer making my candidate comfortable and writing something simple. 15 minutes, privacy, a drink, and pencil & paper. I hate white boards for this.
I come back & see what they've done. It doesn't have to be perfect. It doesn't even have to be right. I am less concerned about their code than our discussion about their code...
15 minutes of coding followed by 15 minutes of Q & A about their code tells me alot. Not everything, but alot. And no programmer, no matter how nervous or introverted, should have a problem with this. As a matter of fact, most of the best programmers I've interviewed have embraced and enjoyed this.Fizzbuzz is a great problem for this. I'm still amazed by how many "experienced" programmers can't do this.
It bears essentially no relation to my work day-to-day.
If you won't code for me in an interview, why would I possibly be interested in you?
Thanks for saving me so much time.
Good. Go work for them.
My "prima donna" filter is working nicely.
I can only assume that I have never noticed these putative canine job postings because I have been looking (or smelling) in the wrong place. At the interview, should I attempt to alter my personal odor, or leave it as is? Will I be asked to sketch out an implementation of "disc fetch" on the whiteboard, or will I be given an actual flying disc and a long hallway? Should I remove experience working for cats from my resume, or leave it in?
I think I might walk out too, because it's clear that power will be exerted on any candidates who choose to subject themselves to it - it says a lot about the type of work environment that is fostered.
I bombed the whiteboard/paper tech interview when I was first invited to interview at my current job (lots of ex co-workers here), and was declined as a candidate.
Months later I was invited again, this time just to start, skipping the interview. I'm finishing up my last week here and have nothing but praise and recommendations from my higher-ups, yet I would have never had a shot if people I worked with in the past didn't work here and reach out to me.
Actually at one of the places I recently interviewed, I got past the phone screens, cultural interviews, and even a google hangouts/screen-share session where I coded live while talking about the solution -- but I still had a 'get on the whiteboard and write some twist on academic sort functions' and of course froze up more and more with each time I used the eraser. I'm confident I would have done better with a computer.
I don't code on paper and whiteboards. Most of us aren't trained to, and state-dependent learning is a real thing. You cannot expect most people to perform as well in a drastically different environment from how they typically do their job.
Ask for references, read his open source code and talk to him over lunch about some of the stuff he's worked on. Someone who doesn't know how to develop software will be easy to spot.
2. If you think coding = dancing like a monkey, maybe you should do something else.
3. I've done everything you suggest, but the interview test is much more effective. You can bullshit through your suggestions (references are a joke), but not mine.
(2500 technical interviews)
I never know how to approach this question, because often the answer is, "the problem is so simple and general to the point that all variable names are either academic (solved computer science problem with technical name for each concept involved) or hopelessly generic ("counter").
When I code things that actually do things, I just name things what they should be called. I think functional programming makes this glib prescription much more tenable, as many of the things you write are abstract of the implementation, so you never find yourself naming something "screen_refresh_counter_for_process_x."
I don't give a shit about what you wrote on the page, I care what you have to say about it.
Your comment tells me more about you than you imagine.
I'll flatter myself and assume you liked what you saw in what I wrote. But if that's the case, why code at all? Why not just sit down and have a short conversation?
I feel like there's something about a rigorous, test-of-endurance interview process that feels professional. "We can't afford to hire anyone but the best."
So experienced hackers with good instincts end up ignoring them in favor of a gauntlet they know is not representative of actual work.
... can't do this in your incredibly artificial, unrealistic context.
I hate live coding in an interview, especially on paper or whiteboard. I know you say you don't care about syntax and little stuff, but that's not what happens in my brain. I suddenly forget the most basic things. "Wait, do I use semicolons at the end of lines in Ruby? Aaaah, I've been using Ruby for 5 years. How can I not know this. I'm going to bomb this interview. I should just leave now."
I know it sounds ridiculous to you, but something very similar actually happened to me. Give me a block of time and a sample problem to work on at a computer, fine. Ask me to refactor code that already exists, whatevs. I'll happily write and discuss code. Just don't give me that blank piece of paper. My brain goes crazy.
And the funny part is that I've been told one of my best qualities as a developer is my calm, thoughtful performance under pressure. When somebody accidentally drops that table in production or both load balancers go down, I'm your guy. I'll get it fixed quickly, without a lot of panic and fussing about. Just don't give me that piece of paper in an interview.
I can't explain it, but it's true, for me. Maybe it's just that I deal with real crises often enough to have developed a mental muscle for it, but I interview so rarely that I don't have the patterns yet. Either way, I think you're missing out on great candidates with this approach.
It doesn't have to compile or even be right. But you should at least be able to pseudo code your way through it, explain what you did, and discuss it.
For a professional programmer, anything less is just an excuse or an attitude problem.
> For a professional programmer, anything less is just an excuse or an attitude problem.
You're entitled to your opinion. I disagree. I think you're projecting your personal experience of the world onto other people, which is your prerogative as the interviewer. However, speaking from personal experience, I know some people who would otherwise make excellent teammates, top contributors, etc., will fail this test, and many who pass this test will make terrible employees.
As I said elsewhere, you will get many false positives and some false negatives. Thus, I think it's a bad test and a waste of time.
Programming profession is hard and full of rough edges, unfortunately.
Citation needed.
That's the part I don't get. It's so arbitrary and artificial. All you can be sure of is that I can solve fizzbuzz in 15 minutes.
> Programming profession is hard and full of rough edges, unfortunately.
I agree completely, but I think paper programming exercises produce false positives AND false negatives.
The flip side equivalent: many modern writers cannot use computers to be creative: http://mashable.com/2014/02/15/modern-writers-technology/
Here is a list of (mostly manual) typewriters that authors use: http://site.xavier.edu/polt/typewriters/typers.html
"What was your approach to the problem?" = I pretend to think for a few seconds, then answer. You're basically just figuring out if I've encountered your almost exact interview previously and I've got the canned answer ready.
Currently we have a timed programming exercise we let people do at home, then review and ask questions like this over email as a filter, it seems to work fairly well.
In person, I just like to ask people about their work experience, especially painful failures and learning experiences they've encountered.
Things like variable names are meaningless, imho. Every company tends to have an arbitrary set of rules they like to follow, unless someone is using things like 1a a1 11a1 a1a1 altogether where I can't read it, I've gotten over caring.
(I think a longer work sample test on the candidates own time is probably still better, but any approximation is better than nothing)
Then "how about without the modulo operator". The solution is so obvious - using the modulo operator, I had a mental block. It was fine once I got nudged in the right direction, but under normal circumstances I wouldn't have needed the nudge.
I fail to see what is the difficulty on that, looks like very straightforward and easy question to do, even over the phone.
- Admission that arbitrary coding challenges that are unrealistic to the applied job are ridiculous
- Recognition that you need to understand the engineer and approach things from their point of view, not from your own selfish view as the interviewer
- Urging that being polite and humble is important. Even if you don't like the candidate, there is no need to attack or diminish them.
One question I have: What is wrong with copying and pasting solutions to issues that have been solved before by others? I tend to do this all the time myself ( albeit I often end up rewriting the solutions to tailor them more to the problem )
Overall decent suggestions for interviewing.
However, I have a lot of respect for companies that do interviewing well (testing skill and experience in a less contrived and awkward manner), and the ones that have made the process as painless as possible tend to be well-run companies.
You can learn a lot from the process with each company, especially at an on-site. Dick-swinging contests definitely happen, which to me means that the company values rock star coders over programmers who are skilled, pleasant to work with and good at communication (and moves to the "won't accept offer" pile).
Sounds like a bad excuse to me.
If you get nervous on simple coding questions, how will you perform on your daily work?
Unless you're applying for a very junior position, chances are your work will involve plenty of technical discussions.
Technical discussions at work aren't so high-stakes as ones in an interview. If you do badly in an interview, you've wasted an entire day for nothing. If you fail to speak in a technical discussion at work, best case no one notices, worst case they think "Hm. Looks like Brian didn't get much sleep last night". Just the knowledge of these stakes can affect your performance.
I'm interviewing right now and all except one have lead to offers, so no, I'm actually good at my job. I'm just a better engineer than I am an interviewee.
I've had some issues with the whiteboard in the past. After I blew one interview (partially based on my whiteboard performance), I spent a lot of time coding solutions with a pen and paper, working on my penmanship (so that I could read my own writing) and just getting used to solving problems away from a screen. It really helped me get over a lot of my issues with that particular situation.
He kinda touched on it at the end with #1: "Know what you're looking for".
Kahneman is more explicit in Thinking Fast and Slow: come up with a simple rubric of 5 or 6 traits you're looking for in your new hire and ask questions that allow you to score each candidate on a scale of 1 to 5 for each. When you're done, choose the candidate with the highest score.
Maybe the technical ability score gets double-weighted for developers. But it seems like a lot of the interview process (and discussion of it) fixates on this. Well this and, of course, vague impressions of the candidate's personality. In my experience, this misses other key things that make up a competent programmer and worthwhile employee.
In the spirit of Thinking Fast and Slow, here's a question. As an interviewer, what are the 5 most important traits or qualities you look for in a candidate?
This is of course a double edged sword - on the one hand you only have a generic sense of what you are looking for and on the other you have no vendetta for hiring someone who is sub-par to fill your urgent needs.
So, why is it such a stretch to ask people to solve Fibonacci? I don't understand this mentality. Reminds me of "why should I learn calculus? I'll never use it in real life!" from college. You're supposed to be a professional software developer, and should be able to handle an unfamiliar problem.
If you're a professional musician auditioning to play Mozart, you wouldn't complain that your audition piece is Beethoven.
The problem is, the vast majority of software development jobs are not this. That was the point of the comic in the post. There is a whole lot of maintenance, API gluing, and spec work that needs doing. Even at the "elite" companies like Google.
The current trend in hiring seems to be at least partially born of Google's ill-advised[1] strategy of hiring generic software engineers that could theoretically be slotted into any role at the company. Hire for a position. If that position involves challenging, novel problems then throw things like that at the candidates in the interview. If it doesn't, don't.
[1] After they reached a certain point as a company
It puts people on their home turf, where they are most confident and comfortable. They can show me what they are proud of. I can ask lots of questions about it and see their thinking without having to be a total dick asking for an esoteric algorithm on a white board. There's also a lot to be said for how well someone know show to use their tools. Are they navigating proficiently, or are they poking around the IDE like they just started out.
This sounds like a softball interview, but you would be amazed the number of people it weeds out. You probably need something more rigorous beyond that, but don't start with the hard problems when an easy filter can really give you a lot of insight.
(Note that generally you can't share your current job's code, but people are fairly comfortable opening their own laptop and poking around in the code. I never ask for people to send me that kind of code).
I've had it happen twice, and man it ain't fun. The second was like an out-of-body experience. I forgot the question, I forgot my train of thought, I knew I wasn't responding, I knew that I knew that the interview knew that I was just sitting there, saying "ummmmmmmmmm" for about 147 seconds now. Yikes! Yet in day to day, these would have been easy questions. And TBH I didn't even want to job at that point.
It's critical that you as an interviewee learn to monitor and CONTROL your physiological response to stress. And I've learned that as an interviewer, I try to pay close attention to physical stress responses from the candidate -- flushed cheeks, signs of sweat, sudden lock-up -- when I see these signs, I back off and let them come back to normal first.
* Talking about something you've done? You're selecting for marketing, not programming.
* Letters of recommendation? You're selecting for ability to network, not programming.
* Academic qualifications? You're selecting for persistence, not programming.
* Offline tests? OK right now because they're a minority, but I guarantee if they became the standard, you'd get an astounding amount of cheating.
I'm not a marketer and I really resent how in academia to have any chance you have to "sell sell sell". I appreciate anyone who gives me a chance to just objectively demonstrate my abilities, whether I do well or whether I do poorly.
EXPLAINING something you've done.
[1] https://github.com/kevva/is-negative
In an actual interview, when they are talking about something they've done and explaining it, the interviewer will be able to ask more questions and hopefully (as an interviewer) have enough of a technical background to separate the BS from the skills.
If two chemists are talking to each other about chemistry, one's not going to be able to easily just make stuff up that the other blindly accepts.
You have to do this everywhere. There are limited resources. You are competing for them with others. You need to convince a gatekeeper that you deserve those resources. In academia, you are competing for grant money, lab space, etc. on the basis that your work is of more import than that of someone else. In enterprise software sales, you are attempting to convince a customer that your product is a better solution to their business problem than of some competitor's. In an interview, you need to demonstrate that you can accomplish a function better than one of your peers who is also vying for the job, and probably has an almost identical skills profile to you.
Tell me about a project, any project large or small, that was 100% you. You thought of it, you coded it, you wrote the tests, you documented it. The only other requirement is that the project has to have at least 10 users who installed, used, and continue to use the project without any help from you.
Lots of people don't have an answer. Some you need to coax a little, because a small project is fine. For example one that I did close to 30 years ago was move/copy utilities that used regexp. You could do
move =.fortran =.f
and it did what you wanted (first = is .* and second is $1 and they could repeat).
Anyone with basic C skills could have done this (or perl, I think someone else did a perl version and that won). So people would think a project like that doesn't count, but it does. Small is fine, you just had to do the whole thing and people needed to be able to use it without asking for help.
This covers a lot of ground and my experience is if you get lucky enough to find someone who has done projects like this, and there is at least some overlap with what we are doing, yeah, we got lucky.
http://kelukelu.me/interview/powerpose.html
Also, I'm embarrassed by how bad my art is. I was learning to use a brand new tablet.
See, most interviews are along the lines of: Q: Are you smart and good looking? A: Yes!
This is why I don't really intervew all. When I take the "best" ranked applicants I give them actual work; small slices from our actual code base. I provide them a complete, functional dev environment with one restrciton: cannot commit to git. I can see their work directly, I work with them directly and get real-time data on what it's like to work with them, how they handle a project, how they communicate blockages, how they figure out our existing systems.
In four to eight hours of work I know who is the right fit for our team. My cost to find and filter is very very low and rather than waste 2-4 hours per interview for 10 candidates I pay for 16-24 hours of work for 2 or 3 candidates. The hard costs are the same, it consumes less of my (or my engineers) time and the candidate fit is very good.
But a lot of these have "Communication Skills: 9/10" on them. Whether this is added by the candidate or the contracting company, I don't know.
I disagree a bit with the center pane, though (with the dog typing loops and the caption "I have no idea what I'm doing"). It is, in my opinion, very impressive to be able to find the largest sum of all square matrices in an NxN matrix at a whiteboard in 45 minutes. It's impressive enough (especially not knowing the exact question in advance) to be a positive indicator of the ability to do what is in pane 3, especially considering how many variables and logical basket weaving is often involved in moving some text a few pixels to the right. As long as you are willing to tolerate a very high false negative rate, this could be a good way to go.
Unfortunately, it does mean that most developers looking for jobs will need to intensely study data structures and algorithms, and walk around with them loaded in "exam ready" memory" in a way that few people would need to simply to do their jobs. Personally, I'd say most people would retain this mental state anywhere from 1-3 weeks after a college algorithms course, provided they were a good student who understood the material. And it also means that lots of developers who could absolutely do this if they were willing to drop everything to study 50 or so hours and train at a white board will not be hired because they were more interested in just doing their jobs (which, ironically, is what they'd be doing at the new job). So we are going through a very inefficient re-taking of exams every time we interview. So I'm not surprised that some people have decided to stop interviewing, or just give up on finding a job that requires this sort of interview.
It's up to a company to decide if they want to give up on hiring these developers, or to tolerate this kind of false negative rate. However I have close to zero interest in hearing about how hard it is to hire good developers if this is their approach, since it suggests to me that the aren't going to make any effort to find a way to avoid these false negatives.