One problem with this is that it presupposes that the interviewer is skilled enough to spot problems, ask the right probing questions, etc. Casey is definitely skilled enough. I can’t say that’s universal.
Another problem, which TFA hints at, is bias. System Design interviews are often terrible for this reason. If you present me with some scenario that I know is trivial to handle with a single server, I’m going to want to discuss that, but you are probably expecting me to talk about a message queue, a caching layer, etc. Both are valid depending on the situation, but if you’ve only ever known one type, you may dismiss others out of hand.
I can probably go into great detail if we're talking about a project I worked on in the past 3-6 months. But beyond that, the details of past work start to become fuzzier. So if I encounter an interviewer like this, I'd better hope that my most recent project fits the bill, because I'd probably stumble through it for earlier projects.
I dunno who Casey Muratori is but I’ve followed the same principle for years! It’s a much better experience for the candidate and gives better signal provided the questioner can actually conduct a good interview and ask probing questions.
> Asks questions with the goal of having the candidate teach him about his project.
I really like this. Asking candidates about their PhD thesis has not gone as well as I hoped, there's usually an increasing level of panic as they realise I read it before the interview. Asking about patches they've written to open source has the same effect.
Asking about something I can't verify changes the stress profile on it a lot. Going to change to this strategy. Thank you HN :)
Getting overconfident as an interviewer is exactly how you get conned. Everyone makes bad hires sometimes. No process is fool-proof, only fool-resistant.
Most people should not be hired if he gets the needed detail. Most people don't have significant open source experience. That means for most people the projects are things they can talk about at a high level, but this interview is digging into things that any court would consider trade secrets. Even if you don't have a NDA, you still have a moral, ethical (and likely legal but check your local laws) obligation to not discuss the details of someone else's code in that level.
I wish the HN title was "How to effectively conduct programming interviews", like it is in the article.
I find Casey Muratori completely insufferable and this title riled me up, but the content is actually pretty good. My perception of him is that he is a good engineer, but generally overconfident and unwilling to approach other peoples' viewpoints with an open mind. The current title played right into my biases.
(At the time of writing, the HN title is "Casey Muratori: I can always tell a good programmer in an interview")
I like this approach. The only problem (or at least one of the problems) is that many people do not have suitable personal projects, as in they might be 10 years old. If one of the goals is to filter out people that don't work on projects in their free time, then this works, but I know many good programmers who don't, or at least they don't anymore due to having kids and other responsibilities outside of work.
As this article touches on, I'm big into pair programming interviews. I was part of the pair programming step at a past company and we'd always use StringCalc [0]. It's starts out super easy and gets gently progressively harder. The goal isn't to finish but to just to see how people think. We would do pretty legit pairing where we'd help if they got stuck on anything or thought we had a better solution. This shows so much about how someone thinks, collaborates, and responds to feedback. Often within 10 mins I could tell how it was going to go. We always had to finish, though, just in case.
Of course it depends on what type of app you have and how your company works and so on and so forth.
That’s absolutely right, but there’s another issue with the LeetCode-style interview that hasn’t been getting much attention lately, including in this article. My company is hiring right now, and we’ve shifted all of our initial interviews to Zoom, where we include a brief coding task. However, it’s becoming more and more clear that many applicants are using LLMs to produce their code. It’s reached the point where it feels almost impossible to assess whether someone can actually code on their own in these remote settings. On the other hand, it’s much harder to lean on an LLM in a more open-ended, conversational interview without it feeling unnatural. That, I think, is one of the biggest flaws in the current remote coding interview setup.
> [Linear] mentioned that this approach is working very well for them, and they have achieved, at the time of writing, 96% retention.
Is this evidence that their hiring process is sound, and is it more a consequence of Linear being a rocketship? Perhaps if their retention number includes when a bad hire is let go, this is more believable that they are meeting their standards.
I’ve only worked at small startups, but usually “retention” means that no one has left for somewhere better.
Leetcode tests programming; interviewing about previous projects tests software engineering, or at the very least the person's understanding of it.
My problem with that is that when I get interviewed about a project, I will talk about the whole project, not just my personal contributions to it which I believe is the primary purpose of a job interview. Of course, knowing about the whole of the project is important too, but if I just have an overview and little to no contributions, I wouldn't be a valuable asset to the company I'm interviewing at.
(Not to dismiss my own contributions of course, I'm a competent engineer and can do anything - it's more a matter of what I'm enthusiastic and energised about. I wouldn't be energised by clicking around in the AWS console, but I know of it, for example).
> "Another benefit of LeetCode style interviews is that they are somehow standardized. Standardized processes help large organizations stay consistent."
I may be overlooking some really important concepts here, but there I bug. It looks like the need is not for a "good programmer" (the article should define what is a good programmer btw) but for a programmer who applies to the standards for whatever it means. It is a bit chilling. And afaik those leetcode interviews tend to fade away.
> "How they navigate unknown codebases."
That point seems very short-term sighted. For how long a codebase is considered as being "unknown" ?
Globally, "good" is not defined and the "scope" of the programmer's job isn't even touched, which I think changes the way you hire someone.
Anways, it was a nice read although I don't really know what to conclude. The pair programming concept is, for sure, the best I would like to experience in an interview.
Honestly I don't really care about a candidate's previous projects. It's fun to talk about if we have the time for sure, especially if it's immediately relevant to the role.
I want to get a glimpse into how the candidate is going to handle our tech stack, our code base, not one of their old code bases.
I still think this is very possible to learn without LeetCode (which I also disdain and never use).
Interviews have been research extensively. Yet every article I've seen gives me the strong impression that nobody is even aware it exists, much less has tried to look for it (or even better yet found it!). Everyone is "this is what I think works", but nobody gives me any reason to think their system works - they haven't verified it in anyway and so it might just be luck that they have hired good people.
Most people are reasonably good, so luck doesn't seem that unlikely - someone should draw resumes at random from their pile and make an offer to whoever wins - I'm curious how selecting for people who are lucky (or who God approves of if you want to go there) compares to your process. If you cannot show me data on why your process is better than that (or something else) I have to assume you don't really know.
PLEASE, when someone talks about how to interview can they at least put forth the effort to cite real research. If you cite someone else you can tell me you think it is invalid for whatever reason, but at least show me you care enough to read it before you tell me what I should do.
Personally I don't think this topic exciting enough to dig into (scientific papers tend to be hard to read, I want "an executive summary"). But when I interview someone I limit myself to the questions my HR department tells me to ask because they are scientifically validated to be useful.
>...instead, he follows a drill-down approach where he chooses a project the candidate has worked on, asks questions with the goal of having the candidate teach him about his project.
This approach is similar to what I have always said with open source contributions to credible projects since these days, Leetcode puzzles can be solved with AI tools.
They claim a way to interview is pair programming to solve a bug in a codebase unknown to the interviewee. They also claim to allow AI.
I get what they are aiming for, but I foresee trouble:
For one, this kind of interview is way harder to set up than simply asking the candidate to solve an algorithmic question (which is flawed but way simpler).
Also, it can be hard to fine tune so that it's not unfair to the candidate. Some bugs can only be solved after days of looking at the problem, so you have to iterate over this interview setup to find the right difficulty, unfairly ditching candidates in the process. And it becomes "a project"; a lot of companies cannot afford to spend much time on this.
Finally, if you're pairing how do you refrain from helping too much? You're not the one being interviewed, the candidate is! If you allow AI, how do you tweak the problem so that it's both self-contained and reasonably easy, while keeping it impervious to being one-shot by AI?
Of course, traditional interview techniques share some of these problems, but they are way easier to set up. That's what missing here, there's a cost/benefit analysis for interviewers too.
> Try to get as narrow as possible, even to implementation details.
I think this is a great way to tell if someone knows their craft but it could also select for people with 1) really good memory 2) really good bs. I have made lots of technical decisions that I stand by but I have a hard time remembering what I made for breakfast. I kind of have to have the code in front of me to remember those kinds of details.
43 comments
[ 4.0 ms ] story [ 46.0 ms ] threadAnother problem, which TFA hints at, is bias. System Design interviews are often terrible for this reason. If you present me with some scenario that I know is trivial to handle with a single server, I’m going to want to discuss that, but you are probably expecting me to talk about a message queue, a caching layer, etc. Both are valid depending on the situation, but if you’ve only ever known one type, you may dismiss others out of hand.
I'm curious, how can he be so sure? How does he know that he has never failed a competent programmer or that he has never passed a poor programmer?
Just because he hasn't seen it fail, it doesn't mean his process hasn't failed.
And junior developers who have personal or university projects it’s sometimes too shallow for this check described in this article.
> Asks questions with the goal of having the candidate teach him about his project.
I really like this. Asking candidates about their PhD thesis has not gone as well as I hoped, there's usually an increasing level of panic as they realise I read it before the interview. Asking about patches they've written to open source has the same effect.
Asking about something I can't verify changes the stress profile on it a lot. Going to change to this strategy. Thank you HN :)
I find Casey Muratori completely insufferable and this title riled me up, but the content is actually pretty good. My perception of him is that he is a good engineer, but generally overconfident and unwilling to approach other peoples' viewpoints with an open mind. The current title played right into my biases.
(At the time of writing, the HN title is "Casey Muratori: I can always tell a good programmer in an interview")
As this article touches on, I'm big into pair programming interviews. I was part of the pair programming step at a past company and we'd always use StringCalc [0]. It's starts out super easy and gets gently progressively harder. The goal isn't to finish but to just to see how people think. We would do pretty legit pairing where we'd help if they got stuck on anything or thought we had a better solution. This shows so much about how someone thinks, collaborates, and responds to feedback. Often within 10 mins I could tell how it was going to go. We always had to finish, though, just in case.
Of course it depends on what type of app you have and how your company works and so on and so forth.
[0] https://github.com/MokshankSoni/StringCalc_TDD
Is this evidence that their hiring process is sound, and is it more a consequence of Linear being a rocketship? Perhaps if their retention number includes when a bad hire is let go, this is more believable that they are meeting their standards.
I’ve only worked at small startups, but usually “retention” means that no one has left for somewhere better.
My problem with that is that when I get interviewed about a project, I will talk about the whole project, not just my personal contributions to it which I believe is the primary purpose of a job interview. Of course, knowing about the whole of the project is important too, but if I just have an overview and little to no contributions, I wouldn't be a valuable asset to the company I'm interviewing at.
(Not to dismiss my own contributions of course, I'm a competent engineer and can do anything - it's more a matter of what I'm enthusiastic and energised about. I wouldn't be energised by clicking around in the AWS console, but I know of it, for example).
I may be overlooking some really important concepts here, but there I bug. It looks like the need is not for a "good programmer" (the article should define what is a good programmer btw) but for a programmer who applies to the standards for whatever it means. It is a bit chilling. And afaik those leetcode interviews tend to fade away.
> "How they navigate unknown codebases." That point seems very short-term sighted. For how long a codebase is considered as being "unknown" ?
Globally, "good" is not defined and the "scope" of the programmer's job isn't even touched, which I think changes the way you hire someone.
Anways, it was a nice read although I don't really know what to conclude. The pair programming concept is, for sure, the best I would like to experience in an interview.
I want to get a glimpse into how the candidate is going to handle our tech stack, our code base, not one of their old code bases.
I still think this is very possible to learn without LeetCode (which I also disdain and never use).
Most people are reasonably good, so luck doesn't seem that unlikely - someone should draw resumes at random from their pile and make an offer to whoever wins - I'm curious how selecting for people who are lucky (or who God approves of if you want to go there) compares to your process. If you cannot show me data on why your process is better than that (or something else) I have to assume you don't really know.
PLEASE, when someone talks about how to interview can they at least put forth the effort to cite real research. If you cite someone else you can tell me you think it is invalid for whatever reason, but at least show me you care enough to read it before you tell me what I should do.
Personally I don't think this topic exciting enough to dig into (scientific papers tend to be hard to read, I want "an executive summary"). But when I interview someone I limit myself to the questions my HR department tells me to ask because they are scientifically validated to be useful.
This approach is similar to what I have always said with open source contributions to credible projects since these days, Leetcode puzzles can be solved with AI tools.
[0] https://news.ycombinator.com/item?id=45530672
I get what they are aiming for, but I foresee trouble:
For one, this kind of interview is way harder to set up than simply asking the candidate to solve an algorithmic question (which is flawed but way simpler).
Also, it can be hard to fine tune so that it's not unfair to the candidate. Some bugs can only be solved after days of looking at the problem, so you have to iterate over this interview setup to find the right difficulty, unfairly ditching candidates in the process. And it becomes "a project"; a lot of companies cannot afford to spend much time on this.
Finally, if you're pairing how do you refrain from helping too much? You're not the one being interviewed, the candidate is! If you allow AI, how do you tweak the problem so that it's both self-contained and reasonably easy, while keeping it impervious to being one-shot by AI?
Of course, traditional interview techniques share some of these problems, but they are way easier to set up. That's what missing here, there's a cost/benefit analysis for interviewers too.
This is a good point.
I will add a note in the post to highlight this.
I think this is a great way to tell if someone knows their craft but it could also select for people with 1) really good memory 2) really good bs. I have made lots of technical decisions that I stand by but I have a hard time remembering what I made for breakfast. I kind of have to have the code in front of me to remember those kinds of details.