Valuable where exactly. It's good that that's defined clearly in readme at least so that contributors don't waste time. And value of such contributions are defined by project itself and those responsible for it.
PS I can imagine rules like that to appear in some opensource projects after tha recent pandemic of the "awesome project" PRs.
Whenever you post an interview question online your interviewers life get a little bit harder. These questions get blacklisted for being associated with their company and they then have to synthesize a new question
On the other hand, some of the questions can be rather interesting to solve. I also notice there are just a few patterns/techniques that you can use to solve them so the questions themselves do not seem all that important.
* Graph operations (traversal, best path, minimum spanning set)
* Maps & Hashing.
* Heaps or trees. E.g. finding a median or top-k elements.
Every once in a while you'll get a more exotic problem that requires something like a bloom filter or a trie.
But that's just enough to get the solution. A good interviewer will also see if you can build this solution in a way that has effective abstractions. They'll also probe real-world concerns like data locality. For instance, one solution might be better in terms of big-O notation performance but might have worse access data pattern than other solutions.
Not a bad idea. The idea of 'fizzbuzz' was to just filter out the total unable to do it at all people. Not to filter for people who can do coding interviews. Honestly, that these questions are out there and people study for them make an interviewers life easier. 'Oh you saw this problem online great! explain why it works'. It cuts through a lot of gruff. Had it happen a few times with the particular problem I like to use. If they can read cracking the coding interview and understand the problems in it and explain them back you are half way there!
Don't problems have to be rotated eventually anyway? You'd think otherwise there would be secret silos of problems and those in the know will have much better results than anyone else.
I'm just going to use this list to steal a problem for my interviews.
It's hard to find a problem that is good at selecting candidates yet doable in 30-60 minutes and different from the problems other colleagues use in their interviews. Most problems are either too difficult or too easy.
The blacklist can only get so long. There is a bank of questions but they only really get blacklisted if a bunch of candidates show up and they’ve obviously memorized an answer to it. Or if it shows up in a high-profile place. Like, if your question is the first question in a new edition of “Cracking the Coding Interview” it will probably get dropped.
I think individual questions that got posted would be blacklisted ten years ago. But nowadays, there are so many people doing interviews and such a large bank of questions (public and otherwise) that the blacklist isn’t so big a deal. Because there are so many vetted questions to ask, you can pick one and not worry to hard candidates may have done drills using that specific question. And if they have, you can have a second question on standby.
As an interviewer, I’m happy to see questions posted. You get to see people share their experiences answering it and make the question better, rephrase things to be less ambiguous or misleading, maybe tweak some wording that trips up non-native speakers, or decide that people have a bad experience answering it and you should drop the question. As an interviewer, I also ask questions from the blacklist. I’m not gonna say something arrogant like “I can tell” if you knew the question ahead of time, but my job as interviewer is to get the best information about you that I can to the hiring committee, and balancing that with the goal of giving you a pleasant enough experience during the interview that you want to accept an offer. Trying to catch cheaters is not my top priority.
All of this being colored by the facts that the interview process itself is a bit traumatic and people running interviews often don’t run them well.
The way some companies work is that there's a set list of question archetypes, but interviewers come up with their own methods of presenting the problem. For example, a seam carving problem might get presented as finding a horizon in an image. I would often present the same problem as finding a pass through a mountain range given a height map of the terrain.
Also, it's possible for interviewers to detect candidates who study the problem beforehand. A couple questions at Dropbox were spread online for years, and it would become clear which candidates read up on those questions online. Some would propose, without prompting, solutions to older iterations of the interview question.
The last time I did this my experience was a little different. There was a big bank of questions that you could vote on like Reddit, and questions would get marked as “banned” or unmarked.
You’re free to ask the questions that you liked, but I found it helpful to use one of the questions from the bank that had a ton of commentary, and then polish it to my own interview style.
What's the point of hiding the interview questions? What does the hiring company care whether the applicant learned that years ago or crammed for it recently?
I've had to stop interviewees and if they've seen a question before - it's pretty obvious when they have.
As an interviewer you then have to scrap that question and ask a new question on the fly. This is bad for the candidate because there is now less time and it might be a question the interviewer is less comfortable asking.
Booking.com feels like a particularly Silicon Valley-savvy European company. They seemed to be an early adopter of HN: Who's Hiring? threads, for instance.
In my experience and also from what my friends and co-workers told me, it's a lot more about your network and your experience. If you have a good resume and people recommending you, you skip the technical interview part.
I interviewed for one American company once. Didn't like it despite the nice hotel. I also didn't get the job.
If you want a job in those companies, sure. But if you want a decent job in a decent software company, you don't have to go through all that show. There are companies that don't ask that sort of questions and instead they interview you in a sane manner (they ask about past experience, they let you talk about what you actually know... The whole interview feels like a conversation).
I disagree. I've been interviewing heavily the past 3 years and I've noticed that at least for the major US tech hubs (say, SFBA, Seattle, NYC, as well as LA, and from my understanding, London in the UK too) pretty much all the companies I would rate as "decent" or better to have a career as a SWE are gatekeeped by leetcode interviews. Difficulty and selectivity varies of course.
There are exceptions, but they are getting fewer and fewer. i.e. some companies that did not leetcode when I got my current job (5 years ago) are now leetcoding. Even companies in that famous "they don't whiteboard" list will leetcode you from my experience (I reckon its really a team by team basis thing).
You might be able to find a good company that will not leetcode you, but if you are trying to maximize your potential to get a good job at a good company, the best strategy is to grind leetcode, as opposed to hunting for a White Whale.
Maybe your definitions aren't the same as mine (which is perfectly fine!).
My own definition of a good company to work for as a SWE would be somewhere that:
- treats and respects tech and technologists as valuable first class citizens. Not merely a necessary evil.
- good compensation and perks.
- good working environment.
- great, talented colleagues.
- bonus: looks great on your resume and opens doors to potential future opportunities.
I would say most jobs at most well known tech companies (say, FAANG and others) would qualify...and most are gatekeeped by leetcode. I'm guessing you would not consider these to be good companies?
leetcode and puzzle-based interviews are indicative of thoughtless interview process that are lacking evidence as to their efficacy. Google used to focus on puzzles so everyone followed. Eventually google thought to look at the efficacy and realized it was poor, so google focused much less on the puzzles and it's taking a long time for folks to catch up. If an organization claims to respect evidence (as most large tech companies do) but have lackluster or nonexistent feedback loops around interviews, that hypocrisy is a red flag.
Take home tests like leetcode show a lack of respect and trust in the engineers, which I consider a bad work environment.
The high pay is usually because the organizations have tons of cash and are rarely indicative of interesting work (or the work is interesting but the product is discarded after completion). Most large, high paying, highly respected software companies have been stagnating for over a decade and sailing through on their monopolies. Then there are ethical considerations. At least that's how I justify avoiding that world. Maybe I'm totally out of touch with reality.
Either end of the hiring channel is extremely noisy, recruiters work by spamming and candidates get seen by spamming. In my anecdotal experience, maybe 1 in 10 of any given job listing or applicant is worth pursuing.
These questions are a less bad way of filtering people out than others. They're not the only filters used, and others have different tradeoffs. Sometimes if you're hitting these questions it's because the other filters have already excluded you.
That's true but then companies would have a problem where too many people still look qualified after going through rounds of interviews.
If a given company has a lot of people who want to work for it then it makes sense for that company to have people attempt very challenging programming exercises so that the org can filter the applicant pool down to the number of positions they have open.
These companies have the luxury of being able to have hiring processes that turn away perfectly good applicants that couldn't pass their tests (false negatives).
> it makes sense for that company to have people attempt very challenging programming exercises
Perhaps. I would also encourage companies to look past programming ability, and consider things like work ethic etc. I would argue that once you can do the job, reliably showing up and working hard every day has a lot of impact.
You are 100% correct. As a manager I would love to test for work ethic, honesty, being a team-player, etc. The problem is I can only infer those qualities based on signals that are inherently noisy. The only way you really know of those qualities is by working with them over a period of time. A solution to this is to hire through your network, especially via someone who has worked with the candidate. But that certainly limits your pool.
> too many people still look qualified after going through rounds of interviews
Maybe we're not as special as we think and not just a few mythical uberengineers are qualified to do the take-something-from-here-and-put-it-there coding that is 90% of the field...
> That's true but then companies would have a problem where too many people still look qualified after going through rounds of interviews.
Be ready for a surprise. Almost anyone who can spend 3-6 months on solving every single problem at LeetCode\HackerRank can crack these so called high caliber interviews. Unfortunately, a lot of us deny to take that route
The Google ones are wildly off—I work there as a SWE. Just review CLRS [1] until you hit minimum spanning trees, skipping starred sections like the one on matroids. Do a handful of LeetCode medium and hard questions on the way. A few weeks of study is sufficient if you've already seen this stuff in college.
Of course, I don't necessarily agree that LeetCode style questions should do the gatekeeping at Big N, but the procedure above has worked for me at three of the big companies. I'm sorry if you have dependents and need to work full-time while you study.
Google has a bank of vetted questions that can be asked during interviews. It's obviously against company policy to share this information, just like it is against internal policy to share any other private, internal information.
I would suggest putting a barrier to hiring that flavors younger candidates that have graduated from a CS program more recently as unethical.
The point I"m making here is: Is your interviewing bar set to hire individuals who are qualified and capable of performing the job you're hiring for? The leetcode interviews hire those who are good at leetcoding, not the ones that are experienced.
> To be frank, if you don't have the time or ability to review 1-2 classes worth of college material, then that raises some serious red flags.
Not nearly as much of a red flag as your stance is on this.
These questions have their own tricks and expectations. For example, in place shuffling. The way that question is worded implies you would know "oh yes fisher yates shuffle". Without looking it up, would you know what that algorithm entails? Would you know to study it? Would you even be guaranteed to cover it in your studies?
It's hardly a consideration about IQ when you have questions like detecting a loop in a linked list. (Theres a restriction that common question has.. but the solution of it was the product of an academic research paper in the 60s) Asking someone, where that's not fresh in their head is no longer an iq test but a trivia hazing ritual. Will they use that in their job? (No they shouldn't)
So back to your statement-
Reasons why they didn't have enough time:
1. They just got laid off and they're navigating the new life change and possible loss of income. It's really hard to manage the interviews, the recruiters, opportunities, and emotional challenges at that time. It's not a conducive time to interview well, go through all of these coding challenges, and review 1-2 classes worth of college material (which may or may not be relevant at the time).
2. They have a family or dependents (there are laws that protect against this)
3. Their current work place is expecting unrealistic hours and they're trying to get out. (Weekends+nights)
4. Their age, the longer you're in software engineering, the less that it becomes about "I submitted optimal code always" and the more it becomes about managing the project in the correct order and execution. (Are good practices being followed, how do we identify the nasty corner cases, etc) If you're a senior engineer or higher and they can answer about how to implement a red black tree from scratch.. that's a red flag. (Also, Age discrimination is illegal)
From what I've heard, Apple teams have more autonomy in how they conduct interviews for their prospective future team-mates, and (shock!) some actually will not leetcode you (correct me if wrong), or at least evaluate you very differently from say, Google or Facebook.
Once I studied a list like this and then got one of the questions in an interview. I did a good job with the theatrics of not knowing the fully correct answer and struggled through it with flashes of genius. I got the job, after a few years I couldn’t really hack it but I’m still reaping the benefits of having that name start my career.
Unpopular opinion: solving programming riddles makes you not only a better programmer but a better all-around problem solver and thinker.
Before I could solve your average programming riddle I scoffed at them. Now that I have some experience solving them I feel like it is truly a form of mind expanding knowledge and should be valued not only in hiring but in life in general.
While we are all biased toward seeking to validate our own choices I believe it also can be said that some choices are more effective at producing a given outcome than others.
Could you describe a time in your life when the knowledge gained from solving programming riddles impacted other parts of your life noticeably? I also very much enjoy solving these so-called "toy" problems but I can't say they've markedly improved my life aside from helping me get a job where I don't really use them. I've found that learning statistics and linear algebra provided me with more insight into decision making in my day-to-day than just straight algorithmic knowledge
I think the problem is the industry standards and expectations for candidates. The fact that OP links an archive of interview questions is indicative of the underlying problem.
Giving a candidate a problem they haven't seen before and observing their problem solving skills and mentality doesn't sound so bad on paper. However the process has been gamified and as a result the expectations for what a candidate should know and accomplish have increased. This ends up favoring younger candidates and/or fresh grads, since on average they can expend greater time and energy into studying for these types of interviews and their algorithms & data structure courses are still fresh in their minds. A more senior candidate or a person with more responsibilities outside of work (taking care of children, parents, etc) won't be able to invest as much time into studying these types of problems. The original purpose of the interview is also obfuscated - did the candidate genuinely show good problem solving skills or did they see the question before and merely memorized the solution? As a hiring manager I've seen multiple instances of the latter.
I think the utility of spending time on these problems varies from person to person. I personally find it more useful to delve into distributed systems, cryptopgraphy, networking, or operating systems. These topics have made me a much better engineer and improved my problem solving skills, more so than programming riddles which gave me diminishing returns over time.
71 comments
[ 2.9 ms ] story [ 138 ms ] threadThis comment in the README is pretty shitty - those sorts of fixes are still valuable.
* Searching algorithms
* Sorting algorithms
* Dynamic Programming (memoization)
* Graph operations (traversal, best path, minimum spanning set)
* Maps & Hashing.
* Heaps or trees. E.g. finding a median or top-k elements.
Every once in a while you'll get a more exotic problem that requires something like a bloom filter or a trie.
But that's just enough to get the solution. A good interviewer will also see if you can build this solution in a way that has effective abstractions. They'll also probe real-world concerns like data locality. For instance, one solution might be better in terms of big-O notation performance but might have worse access data pattern than other solutions.
All questions and their solutions should be exposed, so big tech can get to work on fixing their interview process?
Don't problems have to be rotated eventually anyway? You'd think otherwise there would be secret silos of problems and those in the know will have much better results than anyone else.
It's hard to find a problem that is good at selecting candidates yet doable in 30-60 minutes and different from the problems other colleagues use in their interviews. Most problems are either too difficult or too easy.
I am all supportive of people trying to game these big companies. Play stupid games, win stupid prizes.
I think individual questions that got posted would be blacklisted ten years ago. But nowadays, there are so many people doing interviews and such a large bank of questions (public and otherwise) that the blacklist isn’t so big a deal. Because there are so many vetted questions to ask, you can pick one and not worry to hard candidates may have done drills using that specific question. And if they have, you can have a second question on standby.
As an interviewer, I’m happy to see questions posted. You get to see people share their experiences answering it and make the question better, rephrase things to be less ambiguous or misleading, maybe tweak some wording that trips up non-native speakers, or decide that people have a bad experience answering it and you should drop the question. As an interviewer, I also ask questions from the blacklist. I’m not gonna say something arrogant like “I can tell” if you knew the question ahead of time, but my job as interviewer is to get the best information about you that I can to the hiring committee, and balancing that with the goal of giving you a pleasant enough experience during the interview that you want to accept an offer. Trying to catch cheaters is not my top priority.
All of this being colored by the facts that the interview process itself is a bit traumatic and people running interviews often don’t run them well.
Also, it's possible for interviewers to detect candidates who study the problem beforehand. A couple questions at Dropbox were spread online for years, and it would become clear which candidates read up on those questions online. Some would propose, without prompting, solutions to older iterations of the interview question.
You’re free to ask the questions that you liked, but I found it helpful to use one of the questions from the bank that had a ton of commentary, and then polish it to my own interview style.
As an interviewer you then have to scrap that question and ask a new question on the fly. This is bad for the candidate because there is now less time and it might be a question the interviewer is less comfortable asking.
e.g. Why is this the right solution? Why not another approach? What are the tradeoffs? Where does this solution fail?
Interviews should be a balance between testing problem solving, ability to learn, and knowledge that applies to the actual job they would be doing.
I interviewed with booking.com in Amsterdam a couple years ago and it was the same thing — same questions testing prep.
I interviewed for one American company once. Didn't like it despite the nice hotel. I also didn't get the job.
If you want a job in those companies, sure. But if you want a decent job in a decent software company, you don't have to go through all that show. There are companies that don't ask that sort of questions and instead they interview you in a sane manner (they ask about past experience, they let you talk about what you actually know... The whole interview feels like a conversation).
There are exceptions, but they are getting fewer and fewer. i.e. some companies that did not leetcode when I got my current job (5 years ago) are now leetcoding. Even companies in that famous "they don't whiteboard" list will leetcode you from my experience (I reckon its really a team by team basis thing).
You might be able to find a good company that will not leetcode you, but if you are trying to maximize your potential to get a good job at a good company, the best strategy is to grind leetcode, as opposed to hunting for a White Whale.
My own definition of a good company to work for as a SWE would be somewhere that:
- treats and respects tech and technologists as valuable first class citizens. Not merely a necessary evil.
- good compensation and perks.
- good working environment.
- great, talented colleagues.
- bonus: looks great on your resume and opens doors to potential future opportunities.
I would say most jobs at most well known tech companies (say, FAANG and others) would qualify...and most are gatekeeped by leetcode. I'm guessing you would not consider these to be good companies?
leetcode and puzzle-based interviews are indicative of thoughtless interview process that are lacking evidence as to their efficacy. Google used to focus on puzzles so everyone followed. Eventually google thought to look at the efficacy and realized it was poor, so google focused much less on the puzzles and it's taking a long time for folks to catch up. If an organization claims to respect evidence (as most large tech companies do) but have lackluster or nonexistent feedback loops around interviews, that hypocrisy is a red flag.
Take home tests like leetcode show a lack of respect and trust in the engineers, which I consider a bad work environment.
The high pay is usually because the organizations have tons of cash and are rarely indicative of interesting work (or the work is interesting but the product is discarded after completion). Most large, high paying, highly respected software companies have been stagnating for over a decade and sailing through on their monopolies. Then there are ethical considerations. At least that's how I justify avoiding that world. Maybe I'm totally out of touch with reality.
These questions are a less bad way of filtering people out than others. They're not the only filters used, and others have different tradeoffs. Sometimes if you're hitting these questions it's because the other filters have already excluded you.
If a given company has a lot of people who want to work for it then it makes sense for that company to have people attempt very challenging programming exercises so that the org can filter the applicant pool down to the number of positions they have open.
These companies have the luxury of being able to have hiring processes that turn away perfectly good applicants that couldn't pass their tests (false negatives).
Perhaps. I would also encourage companies to look past programming ability, and consider things like work ethic etc. I would argue that once you can do the job, reliably showing up and working hard every day has a lot of impact.
Maybe we're not as special as we think and not just a few mythical uberengineers are qualified to do the take-something-from-here-and-put-it-there coding that is 90% of the field...
Be ready for a surprise. Almost anyone who can spend 3-6 months on solving every single problem at LeetCode\HackerRank can crack these so called high caliber interviews. Unfortunately, a lot of us deny to take that route
Of course, I don't necessarily agree that LeetCode style questions should do the gatekeeping at Big N, but the procedure above has worked for me at three of the big companies. I'm sorry if you have dependents and need to work full-time while you study.
[1] https://en.wikipedia.org/wiki/Introduction_to_Algorithms
I would suggest putting a barrier to hiring that flavors younger candidates that have graduated from a CS program more recently as unethical.
The point I"m making here is: Is your interviewing bar set to hire individuals who are qualified and capable of performing the job you're hiring for? The leetcode interviews hire those who are good at leetcoding, not the ones that are experienced.
These questions test for some mixture of (1) IQ, (2) communication abilities, (3) willingness to put in the time, and (4) ability to learn.
Traditional interviews which discuss candidate experience test for only (2) and knowledge questions test for only (3) and (4).
To be frank, if you don't have the time or ability to review 1-2 classes worth of college material, then that raises some serious red flags.
Not nearly as much of a red flag as your stance is on this.
These questions have their own tricks and expectations. For example, in place shuffling. The way that question is worded implies you would know "oh yes fisher yates shuffle". Without looking it up, would you know what that algorithm entails? Would you know to study it? Would you even be guaranteed to cover it in your studies?
It's hardly a consideration about IQ when you have questions like detecting a loop in a linked list. (Theres a restriction that common question has.. but the solution of it was the product of an academic research paper in the 60s) Asking someone, where that's not fresh in their head is no longer an iq test but a trivia hazing ritual. Will they use that in their job? (No they shouldn't)
So back to your statement- Reasons why they didn't have enough time:
1. They just got laid off and they're navigating the new life change and possible loss of income. It's really hard to manage the interviews, the recruiters, opportunities, and emotional challenges at that time. It's not a conducive time to interview well, go through all of these coding challenges, and review 1-2 classes worth of college material (which may or may not be relevant at the time).
2. They have a family or dependents (there are laws that protect against this)
3. Their current work place is expecting unrealistic hours and they're trying to get out. (Weekends+nights)
4. Their age, the longer you're in software engineering, the less that it becomes about "I submitted optimal code always" and the more it becomes about managing the project in the correct order and execution. (Are good practices being followed, how do we identify the nasty corner cases, etc) If you're a senior engineer or higher and they can answer about how to implement a red black tree from scratch.. that's a red flag. (Also, Age discrimination is illegal)
I think Netflix is also similar?
Before I could solve your average programming riddle I scoffed at them. Now that I have some experience solving them I feel like it is truly a form of mind expanding knowledge and should be valued not only in hiring but in life in general.
That's not an unpopular opinion among people who devote energy to solving programming riddles.
The need to validate one's own life choices is a common cognitive bias, if nothing else.
Giving a candidate a problem they haven't seen before and observing their problem solving skills and mentality doesn't sound so bad on paper. However the process has been gamified and as a result the expectations for what a candidate should know and accomplish have increased. This ends up favoring younger candidates and/or fresh grads, since on average they can expend greater time and energy into studying for these types of interviews and their algorithms & data structure courses are still fresh in their minds. A more senior candidate or a person with more responsibilities outside of work (taking care of children, parents, etc) won't be able to invest as much time into studying these types of problems. The original purpose of the interview is also obfuscated - did the candidate genuinely show good problem solving skills or did they see the question before and merely memorized the solution? As a hiring manager I've seen multiple instances of the latter.
I think the utility of spending time on these problems varies from person to person. I personally find it more useful to delve into distributed systems, cryptopgraphy, networking, or operating systems. These topics have made me a much better engineer and improved my problem solving skills, more so than programming riddles which gave me diminishing returns over time.