This is very interesting, I've been using LLM to learn new things that way and it really worked. To some extent, learning with LLM is better than taking any course, even with a tutor, as I am getting something prepared for me, in terms of my experience, progress level, etc.
LLM is going to change schools and universities a lot, teachers, tutors will have to find themselves in the new reality, as they have a strong competitor with infinite resources and huge knowledge, patient and ready to work with every student in a distinct way, according to student's needs, level, intelligence, etc.
Instruction-based tutoring is dead from that perspective, why should I follow someone reciting a book or online tutorial, while there is a tool that can introduce me into subject in a better and more interesting way?
Sure, there are great teachers, who are inspiring people, who are able to present the topic in a great way, the point is, they are minority. Now, everyone can have a great tutor for a few dollars a month (or for free, if you don't need generating too much data quickly).
Very cool, I have personally been studying zk-cryptography with a similar approach, works really well with some caveats. Will save this article and try this version as well when the time comes!
Interesting article - but perhaps a bit light on details in some places, like:
> I generated a list of the most common interview tasks
How? I suppose they mean gathered, or searched for, not strictly generated?
Also a little light on details of the actual interview.
I'm also a little confused about the listing of "problems" - do they refer to some specific leet-code site's listing of problems?
It seems like half-way between naming an actual algorithm/problem and naming a concrete exercise.
As for:
> How is it that we do not use this "forgotten and forbidden" coding in our daily production code, even though all highly reusable, useful code is essentially an exploitation of the intersection between classical algorithmic thinking and real-world problems?
I'm not sure what to say - most of this stuff lives in library code and data structure implementations for any language in common use?
Indeed the one saving grace of leet code interview is arguably that it shows if the candidate can choose sane data structures (and algorithms) when implementing real-world code?
another POV is, it used to be cool to work for Google
it's been so uncool and harrowing for a while now, to deal with their leetcode BS. i mean obviously this guy is well meaning but didn't learn anything, other than for the paycheck and whatever desperate circumstances require that.
the LLM stuff being used to solve their interview process is an inflection point where it really steeply goes down to want to work for Google for any reason other than money
maybe this is why Google deepmind researchers keep leaving to start their own successful companies
Your "no compiler" rule on day 3 taught you more than the LLM did. The LLM made concepts click. But the binary search vanishing under interview stress proves that understanding something and being able to produce it under pressure are totally different skills. Nobody talks about this enough in the "just use ChatGPT to learn" discourse.
There is this famous quote from Bentley on asking programmers to write binary search
>I’ve assigned this problem [binary search] in courses at Bell Labs and IBM. Professional programmers had a couple of hours to convert the above description into a program in the language of their choice; a high-level pseudocode was fine. At the end of the specified time, almost all the programmers reported that they had correct code for the task. We would then take thirty minutes to examine their code, which the programmers did with test cases. In several classes and with over a hundred programmers, the results varied little: ninety percent of the programmers found bugs in their programs (and I wasn’t always convinced of the correctness of the code in which no bugs were found).
>I was amazed: given ample time, only about ten percent of professional programmers were able to get this small program right. But they aren’t the only ones to find this task difficult: in the history in Section 6.2.1 of his Sorting and Searching, Knuth points out that while the first binary search was published in 1946, the first published binary search without bugs did not appear until 1962.
The invariants are "tricky", not necessarily hard but also not trivial to where you can convert your intuitive understanding back into code "with your eyes closed". Especially since most implementations you write will only be "subtly flawed" rather than outright broken. Randomizing an array is also one of the algorithms in this class, conceptually easy but most implementations will be "almost right", not actually generating all permutations.
Note: I haven't done any tech interview in 6 years.
I'm kind of surprised they still do leetcode-style questions on remote interviews these days. I thought those types of interviews would be 100% gamed by now.
They've been gamed in the "study for the test" sense for years—a sort of human over-fitting—but managers did not mind. I've heard some insist that this is a feature, not a bug. (Depending on levels of cynicism, it's either testing for diligent workers who put effort into preparation, or selecting out non-conformists who aren't willing to put up with management bullshit.)
LLMs make it easier to cheat and give managers a push to develop new, AI-aware, assessment methods, but don't really change the underlying organizational dynamics that led to these tests in the first place.
I've seen that problem in an interview before, and I thought the solution I hit upon was pretty fun (if dumb).
class Solution:
def findMin(self, nums: List[int]) -> int:
class RotatedList():
def __init__(self, rotation):
self.rotation = rotation
def __getitem__(self, index):
return nums[(index + self.rotation) % len(nums)]
class RotatedListIsSorted():
def __getitem__(self, index) -> bool:
rotated = RotatedList(index)
print(index, [rotated[i] for i in range(len(nums))])
return rotated[0] < rotated[len(nums) // 2]
def __len__(self):
return len(nums)
rotation = bisect_left(RotatedListIsSorted(), True)
print('rotation =>', rotation)
return RotatedList(rotation)[0]
I think it is really interesting that you can define "list like" things in python using just two methods. This is kind of neat because sometimes you can redefine an entire problem as actually just the questions of finding the binary search of a list of solutions to that problem; here you are looking for the leftmost point that it becomes True. Anyway, I often bomb interviews by trying out something goofy like this, but I don't know, when it works, it is glorious!
I only did these types of interviews when applying for internships in uni, but I really don't think you would get away with using an inbuilt binary search (via bisect left) in a question that is basically "binary search with a quirk".
It's certainly not an "elitist initiation ordeal" to ask a potential crewmate to sketch a library function they've selected, what abstraction or ideas it implicitly contains, and what bearing those have on their overall approach. GP's use of `bisect_left` is instructive here:
In [2]: Solution().findMin([1, 0, 1, 1])
Out[2]: 4
However, requesting a bug-free implementation of `bsearch()` (to say nothing of the actual problem being solved) during a timed, in-person interview is less a structured rehearsal of an individual's unique capacities, and more of a jumping-in ritual proving only a willingness to die for their cybergang.
Recently had a coding interview in which I was allowed to search online but not use any AI. On the first google search, the interviewer realized that the first result is now AI generated and said I couldn’t use anything from there. So I had to just click on different links and piece together what I needed from inside the pages
I'm always interested in write-ups when folks try new attacks on self-study.
I will also admit that this part hurt my heart to read (vicarious embarrassment):
> the recruiter mentioned I needed to pay more attention to code debuggability (whatever this means - I assume that under the corpo-language, they mean that I wrote invalid code)
I completely understand why that line caused vicarious embarrassment.
Looking back, I realize my brain was(is) operating on a completely different definition of that word based on my daily constraints.
I plan to write more about this in Part Two, but at that point in time, I wasn't even aware of this alternative understanding of the term.
In telco, when a remote node crashes at a client's site, I often only have access to a heavily restricted subset of logs, and the debugging communication loop via email can take days to understand "what happens".
Because of that, I write defensive, strictly encapsulated code, and I think in terms of domain-specific states and objects that can be explicitly tracked from an external PoV.
Similarly, during game jams, "debuggable and maintainable" means to me that the code is modular enough that I can completely rip out and rewrite a core mechanic in the final 3 hours just because the game design suddenly changed.
My habit of writing code optimized for remote logs and sudden architectural shifts actually became my biggest enemy under the algorthimic interview (or 45-minute LeetCode) constraint.
It makes the core algorithmic state less clear and hides algorithmic mistakes under layers of defensive "if" statements (where I would normally drop a debug log).
I am simply used to not trusting the inputs, whereas in algorithmic problems, the whole point is to exploit constraints that you need to be absolutely sure about.
So the "if" statements that usually increase "debuggability" in telco or during game jams are the exact opposite of the "debuggability" term used in algorithmic thinking.
Thanks for naming this issue so clearly - it is a very valid reality check.
17 comments
[ 3.2 ms ] story [ 48.4 ms ] threadLLM is going to change schools and universities a lot, teachers, tutors will have to find themselves in the new reality, as they have a strong competitor with infinite resources and huge knowledge, patient and ready to work with every student in a distinct way, according to student's needs, level, intelligence, etc.
Instruction-based tutoring is dead from that perspective, why should I follow someone reciting a book or online tutorial, while there is a tool that can introduce me into subject in a better and more interesting way?
Sure, there are great teachers, who are inspiring people, who are able to present the topic in a great way, the point is, they are minority. Now, everyone can have a great tutor for a few dollars a month (or for free, if you don't need generating too much data quickly).
> I generated a list of the most common interview tasks
How? I suppose they mean gathered, or searched for, not strictly generated?
Also a little light on details of the actual interview.
I'm also a little confused about the listing of "problems" - do they refer to some specific leet-code site's listing of problems?
It seems like half-way between naming an actual algorithm/problem and naming a concrete exercise.
As for:
> How is it that we do not use this "forgotten and forbidden" coding in our daily production code, even though all highly reusable, useful code is essentially an exploitation of the intersection between classical algorithmic thinking and real-world problems?
I'm not sure what to say - most of this stuff lives in library code and data structure implementations for any language in common use?
Indeed the one saving grace of leet code interview is arguably that it shows if the candidate can choose sane data structures (and algorithms) when implementing real-world code?
it's been so uncool and harrowing for a while now, to deal with their leetcode BS. i mean obviously this guy is well meaning but didn't learn anything, other than for the paycheck and whatever desperate circumstances require that.
the LLM stuff being used to solve their interview process is an inflection point where it really steeply goes down to want to work for Google for any reason other than money
maybe this is why Google deepmind researchers keep leaving to start their own successful companies
>I’ve assigned this problem [binary search] in courses at Bell Labs and IBM. Professional programmers had a couple of hours to convert the above description into a program in the language of their choice; a high-level pseudocode was fine. At the end of the specified time, almost all the programmers reported that they had correct code for the task. We would then take thirty minutes to examine their code, which the programmers did with test cases. In several classes and with over a hundred programmers, the results varied little: ninety percent of the programmers found bugs in their programs (and I wasn’t always convinced of the correctness of the code in which no bugs were found).
>I was amazed: given ample time, only about ten percent of professional programmers were able to get this small program right. But they aren’t the only ones to find this task difficult: in the history in Section 6.2.1 of his Sorting and Searching, Knuth points out that while the first binary search was published in 1946, the first published binary search without bugs did not appear until 1962.
The invariants are "tricky", not necessarily hard but also not trivial to where you can convert your intuitive understanding back into code "with your eyes closed". Especially since most implementations you write will only be "subtly flawed" rather than outright broken. Randomizing an array is also one of the algorithms in this class, conceptually easy but most implementations will be "almost right", not actually generating all permutations.
It is also odd that this article appears here after someone complained about vibe coding killing the interest in algorithms.
This game is played often. People have valid complaints, then someone posts a "rebuttal" ("LLMs are not bad for $X---they are good for $X").
Anyway, he uses LLMs more in the search capability, which is less controversial than generative AI and vibe coding.
I'm kind of surprised they still do leetcode-style questions on remote interviews these days. I thought those types of interviews would be 100% gamed by now.
LLMs make it easier to cheat and give managers a push to develop new, AI-aware, assessment methods, but don't really change the underlying organizational dynamics that led to these tests in the first place.
I've seen that problem in an interview before, and I thought the solution I hit upon was pretty fun (if dumb).
I think it is really interesting that you can define "list like" things in python using just two methods. This is kind of neat because sometimes you can redefine an entire problem as actually just the questions of finding the binary search of a list of solutions to that problem; here you are looking for the leftmost point that it becomes True. Anyway, I often bomb interviews by trying out something goofy like this, but I don't know, when it works, it is glorious!Good luck on your second round!
I will also admit that this part hurt my heart to read (vicarious embarrassment):
> the recruiter mentioned I needed to pay more attention to code debuggability (whatever this means - I assume that under the corpo-language, they mean that I wrote invalid code)
In telco, when a remote node crashes at a client's site, I often only have access to a heavily restricted subset of logs, and the debugging communication loop via email can take days to understand "what happens". Because of that, I write defensive, strictly encapsulated code, and I think in terms of domain-specific states and objects that can be explicitly tracked from an external PoV.
Similarly, during game jams, "debuggable and maintainable" means to me that the code is modular enough that I can completely rip out and rewrite a core mechanic in the final 3 hours just because the game design suddenly changed.
My habit of writing code optimized for remote logs and sudden architectural shifts actually became my biggest enemy under the algorthimic interview (or 45-minute LeetCode) constraint. It makes the core algorithmic state less clear and hides algorithmic mistakes under layers of defensive "if" statements (where I would normally drop a debug log).
I am simply used to not trusting the inputs, whereas in algorithmic problems, the whole point is to exploit constraints that you need to be absolutely sure about.
So the "if" statements that usually increase "debuggability" in telco or during game jams are the exact opposite of the "debuggability" term used in algorithmic thinking.
Thanks for naming this issue so clearly - it is a very valid reality check.