I'm currently running the interview process for the company I work for and I allow AI on the coding tests for the same reason I allow googling/etc:
I want to simulate as close to the real environment you will be coding in if you come and work here.
My "rules" on AI/LLMs at work is that you can use whatever tools you want (much like SO before) but any code you commit, you are responsible for (also unchanged from before LLMs). I never want to hear "the LLM wrote that" when asked about how something works.
If you can explain, modify, build on top of the code that an LLM spits out then fine by me. We don't require LLM usage (no token quotas or silly things like that) nor do we disallow LLMs. At the end of the day they are all tools.
I've always found leet coding, whiteboarding, memorizing algorithms, etc to just be a silly form of hazing and a bad indicator of how well the employee will perform if you hire them. In the same way I still think my college was stupid for making us write, on paper, a C program with all the correct headers/syntax/etc for an exam or how I got a 90/100 on my final SQL exam because I didn't end my, otherwise perfect, queries with a semicolon.
Makes sense. It's like we were prohibited from using calculators in school when I was growing up. If you're testing for on-the-job performance then you should have on-the-job tools.
I'm curious what they will come up with. For smaller companies, leetcode style tests probably aren't the best, but for large companies that hire 10s of thousands of devs, leetcode style tests have a lot of good qualities. People who criticize leetcode usually prefer take home projects, pair programming with a current employee, "just a conversation" or something else, but these all have serious drawbacks at scale. Despite leetcode's flaws it has a lot of benefits:
* Objective and clear grading criteria.
* Unlimited supply of questions.
* Easily time bound to 30 min or 1 hr.
* Legal - i.e., no worries about disparate impact.
And still these engineers cannot build anything. If you were to ask if I would hire any of my Google colleagues as a startup founder, I would say never.
interviews should be about determining whether you can use the tools available to deliver the desired product, not some sort of purity test to determine whether you can build without tools. also I want to see you interact w AI because as of right now it's both an incredible and deeply flawed tool and your ability to recognize when it's about to walk you off a cliff is of increasing importance as we discover the limits of what it can do.
Zuckerberg has no clue about software development. He always wants young people, huge bullpen offices, moonshot projects like the Metaverse that fail.
Facebook open source software does not have great code quality. In some projects that always have been a huge mess they are now adding claude.md files to guide their beloved "AI". They did not add these files for humans before.
I think Facebook software is a lost case where it does not matter if you perform the weekly rewriting by LLM or by kLOC driven humans.
Meta interviews in the past have been straight regurgitation of leetcode. They want to measure your flow and typing speed (sarcasm).
So, the interview can now be 2 leetcode hards in 20 min. Earlier, it was typing solution code from rote memory. Now it is furious vibe-coding copy-pasta from one browser window to another.
More seriously, what will the new questions look like ? In the age of LLMs how does one measure ability objectively. Is it with extreme underspecification of the probem ?
This makes sense in principle, but then how do you do technical evaluation? I'm generally most interested in hearing the candidate think out loud about the problem, and explain their approach. LLMs do the "thinking", and also explain the approach.
This is born out of necessity. I have seen some of the coding interview "cheating" tools in the market today and it is ridiculous just how good they are at helping a candidate fake not just the answer but their entire thought process. Leetcode style interviewing is basically dead, at least if done remotely.
The only way to fight this from the employer side is to embrace these tools and change your evaluation criteria.
I do interviews for my company. We allow AI (and even encourage it). Use or lack of use of AI has zero correlation with candidate performance. In fact, I’d say there’s a slight negative trend against those who use AI.
We have candidates build a _very_ simple fullstack app. Think something like a TODO app, but only like 2 core functions. This is basic CRUD 101 stuff.
I’ve seen a boatload of candidates use AI only to flame out after the first round of prompting. They literally get stuck and can’t move forward.
The good candidates clearly know their fundamentals. They’re making intentional decisions, telling the AI scoped actions to take, and reviewing pretty much everything the AI generates. They reject code they don’t like. They explain why code is good or bad. They debug easily and quickly.
LLMs are very, very talented but context still matters. They can’t build proper understanding of the non-technical components. They can’t solve things in the simplest way possible. They can’t decide that to explain to the interviewer “if performance is a concern, I’d do X but it’s a time-bound interview so I’m going to Y. Happy to do X if you need”.
Interesting. I interviewed at Meta a decade ago and was put off by the process (inexperienced devs who memorized leetcode medium problems). But I still get daily requests from their recruiters to try again (I am a senior leader in ML engineering). I would need to spend a fair amount of time thinking about what it would be like to do an interview with LLMs.
But honestly, I'd rather spend that time on figuring out how to use LLMs to interview people better (for example I already had an LLM write a collaboritive web editor with built-in code runner, so I don't need to license coderpad). I could see updating my prompt to have the coding agent generate a text box for entering prompts during the interview. Either way, I still expect candidates to be able to explain what a hash table is in their own words.
14 comments
[ 5.4 ms ] story [ 34.5 ms ] threadI want to simulate as close to the real environment you will be coding in if you come and work here.
My "rules" on AI/LLMs at work is that you can use whatever tools you want (much like SO before) but any code you commit, you are responsible for (also unchanged from before LLMs). I never want to hear "the LLM wrote that" when asked about how something works.
If you can explain, modify, build on top of the code that an LLM spits out then fine by me. We don't require LLM usage (no token quotas or silly things like that) nor do we disallow LLMs. At the end of the day they are all tools.
I've always found leet coding, whiteboarding, memorizing algorithms, etc to just be a silly form of hazing and a bad indicator of how well the employee will perform if you hire them. In the same way I still think my college was stupid for making us write, on paper, a C program with all the correct headers/syntax/etc for an exam or how I got a 90/100 on my final SQL exam because I didn't end my, otherwise perfect, queries with a semicolon.
* Objective and clear grading criteria.
* Unlimited supply of questions.
* Easily time bound to 30 min or 1 hr.
* Legal - i.e., no worries about disparate impact.
* Programming language/framework agnostic.
Facebook open source software does not have great code quality. In some projects that always have been a huge mess they are now adding claude.md files to guide their beloved "AI". They did not add these files for humans before.
I think Facebook software is a lost case where it does not matter if you perform the weekly rewriting by LLM or by kLOC driven humans.
So, the interview can now be 2 leetcode hards in 20 min. Earlier, it was typing solution code from rote memory. Now it is furious vibe-coding copy-pasta from one browser window to another.
More seriously, what will the new questions look like ? In the age of LLMs how does one measure ability objectively. Is it with extreme underspecification of the probem ?
The only way to fight this from the employer side is to embrace these tools and change your evaluation criteria.
We have candidates build a _very_ simple fullstack app. Think something like a TODO app, but only like 2 core functions. This is basic CRUD 101 stuff.
I’ve seen a boatload of candidates use AI only to flame out after the first round of prompting. They literally get stuck and can’t move forward.
The good candidates clearly know their fundamentals. They’re making intentional decisions, telling the AI scoped actions to take, and reviewing pretty much everything the AI generates. They reject code they don’t like. They explain why code is good or bad. They debug easily and quickly.
LLMs are very, very talented but context still matters. They can’t build proper understanding of the non-technical components. They can’t solve things in the simplest way possible. They can’t decide that to explain to the interviewer “if performance is a concern, I’d do X but it’s a time-bound interview so I’m going to Y. Happy to do X if you need”.
But honestly, I'd rather spend that time on figuring out how to use LLMs to interview people better (for example I already had an LLM write a collaboritive web editor with built-in code runner, so I don't need to license coderpad). I could see updating my prompt to have the coding agent generate a text box for entering prompts during the interview. Either way, I still expect candidates to be able to explain what a hash table is in their own words.