I cannot load this page on mobile. When trying to scroll it moves back up. Firefox doesn't offer me to set up the "view mode" for it and Save as PDF is also broken. It's funny how, from memory, all the pages that were broken because of some fancy effect were about programmers.
Asking developers to program in front of strangers is primarily testing for ability to perform in front of strangers and developer skills secondarily (if at all.)
I never ask people to code in front of me. I ask them questions to get a feel for where their areas of expertise are, and to get to know them.
A good interview should be conducted more like pair-programming than performance art. The interviewer should be explaining things about the problem space and asking questions about the interviewee's code (especially when they're going astray).
My approach is not ad hoc. I have refined my questions and technique over a decade of conducting interviews and much more being on the other end of them.
Let me be clear; it's what works best for me. It may not work best for everyone. But I feel our industry doesn't even really put much thought into interviews right now. They just find some coding puzzles online and say "Here do these." That sucks. There's always something better you could be doing IMHO.
The job I actually landed had the technical part of the job interview just like that - a laptop, a set of requirements (fixing an existing application), internet access, and some time.
I did a lot better than I do on whiteboard style interviews, so I'm definitely biased to preferring this :) .
> Hello, my name is David. I would fail to write bubble sort on a whiteboard. I look code up on the internet all the time. I don't do riddles.
Does the interviewer explain what bubble sort is, or do they expect the candidate to know it? The first is a reasonable interview, the second one isn't.
I had a phone interview where I was asked how I would write quicksort. I responded: I wouldn't. Interviewer chuckled, said good answer and moved on. I got the job.
Or tell them you want them to write an implementation of bubble sort on the whiteboard and then give them a 1 minute allowance to use the Internet. Then you can test their Google skills too!
Bubble Sort is a common sorting algorithm. I would expect anyone w/ a CS degree to know sort of what it is.
But, for the most part; I can't imagine anyone ever having to write it on the job unless they are doing super low level stuff. All the 'current' languages provide sorting algorithms built in.
I feel like it probably functions more as a proxy of general algorithms knowledge. If you've forgotten even the most basic algorithms like bubble/insertion/selection/merge/quick sort then that indicates that you've been neglecting your algorithms knowledge since college.
It also kind of depends on the person. If you're interviewing an intern who's still in college, absolutely they should still remember bubble sort. If they don't know it off the top of their heads, they weren't paying attention in class. If you're interviewing someone who went to a boot camp then it would be expected that they wouldn't know a lot of the trivia like bubble sort that nobody really uses.
I guess I would expect just about anyone to know what a bubble sort is, but I wonder if that's bias from the classes I took. Every class on algorithms I ever took would start off with a bubble sort as an example of a sorting algorithm, then use it as exhibit A for figuring out the big O performance of an algorithm (the outcome of which was always, "...and that's why you should never use a bubble sort").
I assume that most people were taught most of the same material as I was, but in reality that may not be the case.
It's very hard to pin down what sorts of things "everyone" should be expected to know. I agree with the consensus that whiteboard coding and CS trivia is not a good way to run interviews, but I bet many/most of the people who complain the loudest about interviews have their own misconceptions about what constitutes common knowledge for programmers.
Any hiring process based on asking several current employees to take turns talking to someone for 30-60 minutes and make a hiring decision based on that is going to be deeply flawed. However, the only alternatives I've heard are so much more costly, lengthy, and elaborate that I just can't imagine many companies moving to a different process anytime soon.
This debate is also complicated because different people want very different things out of the process. Job seekers want processes that minimize false negatives (i.e., they don't want to fail the interview even though they'd make good employees), but employers want processes that minimize false positives (they don't want people who get through the hiring process that end up being bad employees that the company is now stuck with).
The current process probably doesn't do either of these things. But every more elaborate process I've heard people say they want seems to go to one extreme or the other- it either greatly favors the job seeker who hates false negatives, or greatly favors the employer who hates false positives. It's a hard problem with no good answers yet unfortunately. :(
> I guess I would expect just about anyone to know what a bubble sort is, but I wonder if that's bias from the classes I took.
It was covered in my first year CS courses... something like 9 years ago. I've not once needed to be able to re-implement it (or any sorting algorithm) in the intervening time finishing my degree and five years doing software dev as my day-job, much less re-implement it from memory. I would need to look it up to refresh myself on its implementation and characteristics.
> “Whiteboard” interviews are widely hated. They also discriminate against people who are already underrepresented in the field.
I don't particularly like whiteboard interviews. How can they be discrimination if the expectations are laid out and are such that anyone can reasonably practice to reach them.
I understand you can create a test that is deliberately discriminatory (e.g. voting tests).
You can create a test and realize later that people from certain backgrounds do worse (IQ tests apparently?)
But with whiteboarding you are taking knowledge people should have (lets set aside the idiocy of some of these problems). It's foundational stuff for the industry. So the problem might be with the nature of the test, but anyone can go grab a set of whiteboard markers and start working problems. In fact this is part of why I so dislike these interviews, they are setup to make it easy to game them. So how can that really be discriminatory?
underrepresented candidates have been acing our whiteboard interviews, and getting hired.
The whole meme is so stupid I want to cry.
"Asking field-relevant questions with objective answers is discriminatory, instead just have a chat. "
Mother of god.
Yes I understand the limitations and downsides of whiteboard questions, and in fact in future I'm thinking I won't do them on white boards anymore. But I'll still be asking people who claim to be programmers to do some basic programming before I hire them.
I think the point is if someone is under-represented and they merely do "OK" on the whiteboard, other latent biases might downgrade the candidate in the final analysis. I do think it's relevant to ask people to code in some form, but this article makes me rethink the things I'm going to look for in those interview questions. Also, perhaps more useful might be to give them a small problem before they come in to the interview (1-2h work at most) and then go over it with them as part of the interview. That way you evaluate better how they work on a team, explain their thought process and actually do the job they'll be hired to do.
>I think the point is if someone is under-represented and they merely do "OK" on the whiteboard, other latent biases might downgrade the candidate in the final analysis
But if you don't do a whiteboard interview, everyone is by default only OK. Biases rule the day. If you do a whiteboard interview, then at least they get a chance to do great and defy your biases.
Asking them to do real programming is good. Asking them to remember things from an irrelevant layer of abstraction and write code in a way that is very unlike the actual job is the problem.
The alternative is not no technical interviews, but it is important to recognize that different interview styles have different biases. Like a take home problem is biased against programmers with young kids. Whiteboarding is often a skill in itself, and as a result, you end up selecting for people that have the resources to study and practice whiteboarding.
I learned how to whiteboard through peers and career programs in college. There's a lot of people that don't have that guidance, or all those people to practice with. And there's a lot of people that will believe that flailing in their first whiteboard interview means they're unqualified to be a programmer, not realizing that it's a skill in itself.
Whiteboard questions also tend to be the nonsensical algorithms that you memorize for interviews and never use in practice, which again is influenced by one's peer group - you learn, study, and practice together. This also is why race and gender matter, even when you control for income and education.
The point isn't that these are insurmountable for someone from untraditional circumstances, but it is harder. And that means you let in weaker candidates with more advantaged upbringings in favor of stronger ones with less traditional ones. This is why google offers a seminar for all its interviewees to give them interview practice - so that people can start from a more equal playing field.
Obviously, a lot of the alternatives suffer from the same issues. But like all good programmers, I believe 1. this analysis and process is more important than the specific implementations, and 2. by understanding the issues and looking at the data, we can iterate to better solutions.
Had a 2 hour technical interview last month where there was a lot of white boarding, but the panel of interviewers understood when I said "Well I don't know the specifics of X but between Google and stack overflow I'll manage." - got the job by the way...
I disagree with this quote. There is value in memorization. Times tables are an insanely useful thing to memorize. Memorizing historical events and literature can make you into a great lateral thinker.
Wasn't this in reaction to the story that Homeland Security had asked the question to a programmer entering the US? I don't know how this was extrapolated to a broken job interview process.
I believe it's in reaction to the malpractice of hiring programmers according to their results on a purely theoretical exam at the whiteboard, and sometimes disregarding completely their precedent works, published projects, etc.
I'm sure there are other examples predating it, but the following I believe is the 1st account of a Google interview gone bad for someone who very likely didn't deserve to get kicked in the butt (Max Howell, author of Homebrew for MacOS). https://twitter.com/mxcl/status/608682016205344768
How do you know the answer you find is a good one?
Say you think you need a binary tree, so you can get Log(n) lookup, you look it up, implement it, it works, great. Not bad.
Then a lady comes along who actually understands the problem domain, and knows that cache locality will be a big win in this case, and she uses a sorted array with a binary search. Latency is cut in half and the company makes a brazillon dollars. Better.
1% might be about the share of developers who work on battery drivers, video codecs or similar stuff which would be perfomance or battery draining relevant.
And then along comes a 10x programmer who realizes that you can avoid the lookup entirely if you partition your data along this other axis. Latency goes negative and your company makes a few more brazillions off time travel. But the 10x programmer is a misogynistic asshole who can't balance a binary tree on a whiteboard. Now I have no idea which completely fictional character to hire.
Your argument is flawed and represents what's wrong with our industry's interviewing process. Respectfully, of course.
For most developers we don't need to even consider these trade-offs. It really depends what stack you're working with. Are you a front-end engineer? Then this doesn't even make sense because you wouldn't be putting that much into storage anyways that O() is going to matter.
I'm an iOS developer and quite frankly you shouldn't be doing that much business logic on the phone. We implemented an algorithm that had a O(n^4) runtime without optimizing it because the dataset was so small it wasn't worth the time to over optimize.
You're looking for an extremely rare case and discrediting 95-99% of the time we're programming. Sure we can do speed-ups but do you really have to filter every single developer for those criteria? You really don't. Guess what? Unicorns are built off the backs of medicore developers. The codebases once they start expanding are highly unoptimized. Slack was built from contractors. Twitch apparently is still having growing pains on their iOS app because the original developer wasn't very skilled. I was turned down from thumbtack even though I knew more about iOS than the iOS team lead that interviewed me because they valued CTCI more. The iOS team lead was taking advice from me about system design. What would you rather have? Someone that knows system design or someone that memorized how to serialize an n-ary tree? What is actually going to make you the bazillion dollars? A functioning app that can scale, or someone maybe helping you serialize a data structure (with a very known solution) on an iOS app which is completely useless?
It's simply an illusion that you need to have developers who have memorized useless problems in the case that they may one day possibly might see an issue they could optimize 1% better by introducing more complicated code.
This is a tough subject. I don't believe deep understanding of algorithms and data structures is necessary for the vast majority of engineering tasks. Most (almost all) engineers can get by understanding complexity notation and a matrix comparing time/space complexity for various algorithms and data stuctures, no memorization required. And even that is overkill, as most of time the "natural" one suggested by a problem will suffice. For example, "I need to map keys to value, so I'll use the Python dict class" will get the job done with acceptable performance almost every time.
Whiteboard interviewing based on these things isn't useful unless the candidate is being considered for a position in which knowing these things is fundamentally necessary (e.g. they'll be responsible for developing new algorithms or squeezing out the last 10% of optimization available).
On the other hand, substituting requirements for having public code profiles, and the assortment of take-home projects, in-office projects, pair programming to solve a problem the company is facing (especially if unpaid) is just as bad, for different reasons.
And certainly not asking technical questions is problematic.
I just don't know what the solution is. It's one of those things where I know a bad interview when I see it.
I throw real life problems I've run into (and fixed!) at people and ask them to give me things they would do to troubleshoot, and compare it to what I did. Always nice when somebody suggests something new.
I also ask 10,000 foot questions. "Say you're wanting to build a new app that behaves like Instagram. What's your hosting platform, your technology stack, and explain why?".
> I throw real life problems I've run into (and fixed!) at people and ask them to give me things they would do to troubleshoot
Yes. How does the person think and work? I've interviewed ops folks who are technical gurus whose walked through a problem with me and pulled up with "well, in that case the server is fine and it's not my problem". They can work somewhere else, I don't care how good they are at "making the server work".
I've also interviewed ops people who are all "well, if performance in production has degraded I want to sit down with the source control tools for your config management system and apps and work through the changesets with the devs and work out where we should start looking." I don't give a fuck if that gal needs to google some of the answers to the problems she finds, she's got what I want.
>I also ask 10,000 foot questions. "Say you're wanting to build a new app that behaves like Instagram. What's your hosting platform, your technology stack, and explain why?".
Please forgive the uncomfortable question but isn't that a tacit admission that many developers' role consists of simply gluing together libraries and frameworks built by better engineers who actually do know how fundamental algorithms, etc. work?
"simply" indicates you've never tried to stand up a fairly recent javascript stack :D
that being said, you're not wrong. Maybe 5% (maybe less) of the guys out there are smart enough to build libraries and frameworks. the next 45% of the guys out there are smart enough to see all the pieces and figure out how to fit them together.
the other 50% understand syntax and can write code, but aren't capable of any sort of bigger picture stuff.
and if my recent interviews are any indication, the split outside of silicon valley is more like 2/18/80.
Shoot, I'm not in that "let's build brilliant frameworks" crew. I don't think I am at least. Haven't really tried :D
Agreed. The Google style questions are silly. But an engineer really should be able to at least answer an EASY question. Stick with a question where the answer is "put it in a hash table". No trick questions. Just a baseline, does this person have a pulse, so nobody's time gets wasted.
Admittedly, I was without internet working from home for a few days, and I found it a very nice experience. Not being able to go to Hacker News while compiling managed to keep me focused and on the flow. When I had to look up stuff I found myself digging in /usr/share/doc to find the reference manuals... in a way it took longer than looking up the answer in Google, but it forced you to better understand the system you are using and build your solution from that understanding, sometimes considering third options that one would have missed when stack overflow already contains a bunch of snippets to copy paste.
Of course, I am idealizing the experience (eventually I really needed internet and ended up going to public libraries to use it)... but like everything, maybe we are getting too used to the instantaneous and forgetting that the alternatives are actually not so bad and sometimes empowering.
Bubble sort the first sort I learned 17 years ago the 3rd week in my first year in CS at college. 4th week we learned quick sort and a bunch of other sorts that are faster aka forget everything you learned about it.
Whiteboard coding questions are useful as a baseline test of preparedness.
You don't need to be a genius to solve them. You just need to spend some time practicing them and reading cracking the code interview(OR you need to be a recent college graduate who did well in their classes).
I agree that the super difficult Google style questions are silly, but if an engineer can't solve an easy-mid level difficulty whiteboard question (For example, I usually ask something like, "traverse/print a tree"), well... my response to them is why didn't you prepare for it?
You really should have known that you were going to be asked to do something like this, and you didn't even bother to prepare for the easy stuff.
Not sarcasm. If you don't know what fizz buzz means, it means that you didn't even bother to spend 5 minutes googling "programming interview questions".
Do you really want to hire someone who didn't bother to put in even the tiniest, most basic level of effort into preparing for your interview?
Its like going to an onsite interview and asking "so what does this company even do anyway? I didn't check before coming here today."
Why should I have to prepare for some arbitrary question, though? You could ask any number of basic "whiteboard" questions and I have to prepare for them all? So you are going to deny what could be the best candidate for someone who just did a bunch of studying prior to the interview? How does this normally turn out for you? Do they tend to last and be good employees?
But to answer your question more seriously, I would not hire someone who hasn't prepared at all.
Why? Because they don't care. Why would I want to hire someone who cares so little about the position that they aren't willing to put in the most basic, easiest level of effort?
If they don't care about preparing, then they probably don't care about other things, like the quality of their work.
To guys in SV: are managers, scrum masters, product owners etc also whiteboarded during interview??
(disclaimer) From my freelance experience in EU and the amount of "fun & satisfaction" from a the actual work is inversely proportional to the amount of drilling and grilling during interview.
this. maybe he would need to be reminded which sort it is, but he could do it. unless the white board interviewer expected perfect syntax in a particular language.
30 years in the business, just recently failed a screen coding test call because I blanked on find string pairs question. Just couldn't think for a minute.
During the normal day, look it up, slap head and go on.
yes the process is broken. why? because optimizing the process itself is a Hard problem
the naive solution would say, hey, this whiteboard stuff is a synthetic benchmark. just use a real one! (ex: solve a coding problem with access to the internet)
but for large companies (and this applies to school applications as well), it is not merely a process of evaluation, but of filtering.
there are more qualified candidates than there are positions. and thus whatever process is used to reduce this number will always be somewhat arbitrary
perhaps it would seem less "unfair" if it were random, but is it really?
Hello, my name is Tim. I'm a lead at Google with over 30 years coding experience and I need to look up how to get length of a python string.
<sarcasm>
The problem with Python is the arcane, arbitrary and nonsensical syntax. It sure as hell isn't anything as obvious as length(str) or string.length like any reasonable language.
And god help you if you want to do something complicated like define a function.
</sarcasm>
But seriously: I have no problems with the rest of the article, but that one just seems silly. How many times do you have to look something like that up before it sticks? For those of you not versed in the more esoteric secrets of Python, you would use the built-in function len, as in len(str).
I used Python for years, although I'm a little rusty now. I ran into that kind of problem all the time. Your first two examples are what came to mind first.
Just as many things I love about python, I hate about python.
Each time I get to the part where I need to modify the path which the import mechanism uses to look for modules, clean up the loaded modules to make sure I'm not reimporting an already loaded module by the same name, in order to import a module which is in the SAME DIRECTORY as - get this -__file__, I need another coffee.
As someone who has worked with a lot of languages over the years I can never remember which one is `length(x)`, `len(x)`, `x.length`, `x.len`, and so on.
For me the answer is "never", apparently. Then again I tend to struggle with basic syntax no matter what language I'm using or how frequently I use it.
I've come to the conclusion that the 3 algorithms on whiteboard interview is generally worthless for determining an engineer's on-the-job performance.
Instead, "Build X in 1-2 hours using your computer and resources available" was always a better proxy for productive engineers and largely removing both false positives and false negatives.
Its counter intuitive but whiteboarding is actually more inclusive IMO. Anyone who has 35 bucks can lock themselves in a room for 3 months with a copy of CTCI and a pad of paper and come out with an 80+k salary. You cant do that in any other industry.
If we judge candidates purely on work experience that just favors the incumbents who have already broken into the industry and presumably already quite comfortable in life.
That assumes the interview accurately measures something useful. The more I read about this area the more I think a 1-2h pre-interview practical coding assignment, then a 1 on 1 discussion during the interview is better. You then get to understand how they think, how they design solutions, and some sense of how they interact. In other words, how they'd do the job.
> a 1-2h pre-interview practical coding assignment
then you'll only select for people who have nothing better to do with their time than to donate it to your fun programming escapade. Experienced devs will say it's not worth their time and wont apply.
No, it's just moving those 1-2 hours to home instead of at the interview. If you can take time off for an hours-long interview, you have the time to complete that kind of task.
It's asymmetrical though. A reviewer can yea-or-nay my 1-hour assignment in a minute or two. If I'm desperate for work, sure I'll do it. But my position is in high enough demand that I can choose another position that doesn't require me to do homework. If they made some attempt to get to know me, talk to me, and then give me homework, I might bite. But I'm not going to do hours worth of work just for the chance to get a job.
Why would you apply for a job that you don't want? And if you want it, why wouldn't you put at least some effort into getting it?
Effort does not only include writing a cover letter that will be appealing for the company you apply to instead of just forwarding HR your resume. It also includes small activities like these.
You sound more like a spoiled brat, pardon my French, than an artist looking for new opportunities to make a difference.
You don't need to change the world, you need to make a difference. Without making a difference, you're no better than the next person of equal skill who wants a bigger TV and a fancy car.
No he sounds like a skilled artisan who is bringing his unique skill set to the table in an equal trade. He doesn't need to doff his hat and thank the kind Master for this opportunity.
Im good at my job. I solve problems that provide business value. It has nothing to do with art. If you ask me to work for 2 hours, I'm sending you an invoice.
We give these assignments but if we give you one you're already coming in for an interview. What I dislike are the companies who use it as a screen like you describe.
The problem is that in reality, what can you code in 1-2 hours? Of the coding assignments I've seen, many have started to bleed into requiring 5 hours or more.
Good design instincts takes lots of hard earned real world experience to develop. I wouldnt expect the average 21 year old programmer to really know how to design stuff maybe they can provide me crammed textbook answers but that is as "irrelevant" as whiteboarding.
I think a good approach is to not emphasize the whiteboard as much for senior hires, so people with good work experience can mostly get in on their resume and design questions and people with lesser experience still have the opportunity to break into these top companies by grinding algo questions.
I once hired a senior programmer because of his perfect resume and it was one of the worst choices I made while hiring. He was probably a net loss for the company. What difference can be in CV of two people, who both have 15 years of experience in the industry?
Good design instincts takes lots of hard earned real world experience to develop. I wouldnt expect the average 21 year old programmer to really know how to design stuff maybe they can provide me crammed textbook answers but that is as "irrelevant" as whiteboarding.
I think a good approach is to not emphasize the whiteboard as much for senior hires, so people with good work experience can mostly get in on their resume and design questions and people with lesser experience still have the opportunity to break into these top companies by grinding algo questions.
Many interviews are not pure whiteboard algos. And many interview processes are different, each company is unique.
When I interviewed a couple of years ago, I saw various kinds of interviews at companies:
* 1-2hr on site coding exercise, with the internet, alone.
* 'Here is a 100loc piece of code, fix all the problems with it'.
* Chat with product / manager / team lunch. ('Are you an asshole' test / 'culture' interview / scenario testing)
* Design / outline an app that does X, with boxes and lines on a whiteboard.
* Solve this algorithmic problem on a white board / on your computer.
The reason why everyone freaks out about the algo interview it's the only interview type that most engineers have to prepare for. The other 4 usually you don't have to do much prep work at all.
This is the amazing thing about IT. Also once you are in the computer doesn't care what your background is, what your colour is etc. It will refuse to compile the bad code of an ivy league grad.
I get your point, but I'll also point out that costs a lot more than $35 to lock yourself in a room for 3 months. Many people don't have guaranteed room and board for that long.
>Many people don't have guaranteed room and board for that long.
If a person has been in the industry for more than than three years and doesn't have 3 months cash reserve saved up[1], they are severely mismanaging their finances. _Particularly_ since we're seeing increasing warnings from the financial sector that we're in a bubble already.
Having the ability to lock yourself in a room and study for three months is an enormous privilege, particularly if the thing you're studying is only marginally useful.
Anyone who has 35 bucks can lock themselves in a room for 3 months with a copy of CTCI and a pad of paper and come out with an 80+k salary.
If anything, this just tells us what a low bar to entry the CTCI drill is.
Say you had a pool of candidates of roughly comparable intelligence, educational background, and work experience. And then you gave them 3 months of all living expenses paid† to go an "improve themselves as software development professionals."
One of them comes back and says "I cranked out an MVP (relying in part on a language I only barely knew to that point) that's actually being used by people, and getting good reviews. Here, check out he repo."
Another says, "I thought it was about time I got a handle on this machine learning stuff. Here's a project I did, based on a generalization of some ideas in such-and-such paper. Wish I had more time to spend on it, but looks like my error rate's not too bad."
And another says "It took literally 6-8 hours of devoted practice, every day. But I finally got some serious traction on Japanese. I honestly thought I'd never be able to do that."
And then there's the guy who says "Oh, I looked myself in my apartment and did every exercise in CTCI. Because, you know, I heard that's what you're supposed to do. Damn, my back is sore. I think I need a better chair or something."
I have found three different categories for tech-related interviews:
1. Algorithm and data structures trivia: which includes the classic CtCI [1],
2. In-house problems: where you have to analyze and propose multiple solutions to a problem that has been faced by the current development team in the company that is executing the interview,
3. Homework-like projects: which usually take 3-7 days to finish.
After an exhausting 2016 full of interviews I do not know which one is worse.
On one hand, having a common set of problems like the ones proposed in CtCI is good because you just prepare once (kind of) and can apply to multiple companies at the same time. On the other hand, in-house problems are good if you really want to work for a specific company and want to know what are the problems that your future co-workers are facing, the problem is that you have to be well versed in a wide range of concepts to give at least one answer that pleases the lead developer. And finally, the take-home projects, I think they are the worse in comparison, you are expected to work for free for at least one day in a problem that is either very basic or too complicated but they always expect you to have a perfect solution just by yourself. How many times have you done this in your own job? Do you start working in a solution to a problem without asking questions? Without a discussion with your colleagues? Without full specifications? Without multiple attempts? Take-home projects are really not suitable for an interview.
I understand the point of these tweets but I doubt they will change anything. Much of the craziness of nowadays interviews comes from the recruiters who have no idea about these problems, they don't know algorithms and/or data structures, they don't know what are the problems that the company is facing, they only care about the money that the company will give them if the candidate is hired. Now you know why websites like Hackerrank [2] are so popular, recruiters love to have a standard set of questions and if your answer is not the exact same that is in their sheets you lose.
From now on I will just resort to colleague's referral. With this I know that at least one member of the team will vouch for me to join the company and make the interview process as painless as possible because they already have an idea of how good of an engineer I am.
> And finally, the take-home projects, I think they are the worse in comparison, you are expected to work for free for at least one day in a problem that is either very basic or too complicated
One of the solutions my team has come across for this (also mentioned in previous HN threads) is to pay the interviewee for a day's work or give them an Amazon gift card for an equivalent amount. And we aren't looking for a perfectly crafted solution. We're just trying to get a handle on their general coding style, maintainability, coherence from variable naming and/or comments, DRY, whether they bother with edge cases in their solution, robustness/anticipate other types of errors or bugs that may pop up, etc.
237 comments
[ 2.7 ms ] story [ 303 ms ] threadI never ask people to code in front of me. I ask them questions to get a feel for where their areas of expertise are, and to get to know them.
I wrote a post about my feelings on it last year: https://medium.com/@ebbv/my-case-against-coding-tests-6b15c1...
why not have them write code, not in front of people.
give them a laptop and some time alone?
Let me be clear; it's what works best for me. It may not work best for everyone. But I feel our industry doesn't even really put much thought into interviews right now. They just find some coding puzzles online and say "Here do these." That sucks. There's always something better you could be doing IMHO.
I did a lot better than I do on whiteboard style interviews, so I'm definitely biased to preferring this :) .
Does the interviewer explain what bubble sort is, or do they expect the candidate to know it? The first is a reasonable interview, the second one isn't.
But, for the most part; I can't imagine anyone ever having to write it on the job unless they are doing super low level stuff. All the 'current' languages provide sorting algorithms built in.
It also kind of depends on the person. If you're interviewing an intern who's still in college, absolutely they should still remember bubble sort. If they don't know it off the top of their heads, they weren't paying attention in class. If you're interviewing someone who went to a boot camp then it would be expected that they wouldn't know a lot of the trivia like bubble sort that nobody really uses.
I assume that most people were taught most of the same material as I was, but in reality that may not be the case.
It's very hard to pin down what sorts of things "everyone" should be expected to know. I agree with the consensus that whiteboard coding and CS trivia is not a good way to run interviews, but I bet many/most of the people who complain the loudest about interviews have their own misconceptions about what constitutes common knowledge for programmers.
Any hiring process based on asking several current employees to take turns talking to someone for 30-60 minutes and make a hiring decision based on that is going to be deeply flawed. However, the only alternatives I've heard are so much more costly, lengthy, and elaborate that I just can't imagine many companies moving to a different process anytime soon.
This debate is also complicated because different people want very different things out of the process. Job seekers want processes that minimize false negatives (i.e., they don't want to fail the interview even though they'd make good employees), but employers want processes that minimize false positives (they don't want people who get through the hiring process that end up being bad employees that the company is now stuck with).
The current process probably doesn't do either of these things. But every more elaborate process I've heard people say they want seems to go to one extreme or the other- it either greatly favors the job seeker who hates false negatives, or greatly favors the employer who hates false positives. It's a hard problem with no good answers yet unfortunately. :(
It was covered in my first year CS courses... something like 9 years ago. I've not once needed to be able to re-implement it (or any sorting algorithm) in the intervening time finishing my degree and five years doing software dev as my day-job, much less re-implement it from memory. I would need to look it up to refresh myself on its implementation and characteristics.
I don't particularly like whiteboard interviews. How can they be discrimination if the expectations are laid out and are such that anyone can reasonably practice to reach them.
I understand you can create a test that is deliberately discriminatory (e.g. voting tests).
You can create a test and realize later that people from certain backgrounds do worse (IQ tests apparently?)
But with whiteboarding you are taking knowledge people should have (lets set aside the idiocy of some of these problems). It's foundational stuff for the industry. So the problem might be with the nature of the test, but anyone can go grab a set of whiteboard markers and start working problems. In fact this is part of why I so dislike these interviews, they are setup to make it easy to game them. So how can that really be discriminatory?
The whole meme is so stupid I want to cry.
"Asking field-relevant questions with objective answers is discriminatory, instead just have a chat. "
Mother of god.
Yes I understand the limitations and downsides of whiteboard questions, and in fact in future I'm thinking I won't do them on white boards anymore. But I'll still be asking people who claim to be programmers to do some basic programming before I hire them.
But if you don't do a whiteboard interview, everyone is by default only OK. Biases rule the day. If you do a whiteboard interview, then at least they get a chance to do great and defy your biases.
I learned how to whiteboard through peers and career programs in college. There's a lot of people that don't have that guidance, or all those people to practice with. And there's a lot of people that will believe that flailing in their first whiteboard interview means they're unqualified to be a programmer, not realizing that it's a skill in itself.
Whiteboard questions also tend to be the nonsensical algorithms that you memorize for interviews and never use in practice, which again is influenced by one's peer group - you learn, study, and practice together. This also is why race and gender matter, even when you control for income and education.
The point isn't that these are insurmountable for someone from untraditional circumstances, but it is harder. And that means you let in weaker candidates with more advantaged upbringings in favor of stronger ones with less traditional ones. This is why google offers a seminar for all its interviewees to give them interview practice - so that people can start from a more equal playing field.
Obviously, a lot of the alternatives suffer from the same issues. But like all good programmers, I believe 1. this analysis and process is more important than the specific implementations, and 2. by understanding the issues and looking at the data, we can iterate to better solutions.
'Never (work to) memorize something you can look up.'
This has the meaning of, let your (short term memory) figure out based on how often you look something up if it should be remembered or not.
But well, I only had one strange interview where I had to programm Tetris and failed. In most interviews I just got asked, if I could do the job...
There have been others come to light since, but this was two weeks ago.
I'm sure there are other examples predating it, but the following I believe is the 1st account of a Google interview gone bad for someone who very likely didn't deserve to get kicked in the butt (Max Howell, author of Homebrew for MacOS). https://twitter.com/mxcl/status/608682016205344768
Being able to understand a problem enough to successfully google an answer is the most valuable skill for a programmer.
Say you think you need a binary tree, so you can get Log(n) lookup, you look it up, implement it, it works, great. Not bad.
Then a lady comes along who actually understands the problem domain, and knows that cache locality will be a big win in this case, and she uses a sorted array with a binary search. Latency is cut in half and the company makes a brazillon dollars. Better.
Binary trees have been around for decades, and I assume there are solid libraries for all useful variants of them, and that everyone can use them.
I'm probably wrong, but in what way?
I'm kind of sick of doing CRUD apps. Maybe hearing about what you do would inspire me to explore something new.
For most developers we don't need to even consider these trade-offs. It really depends what stack you're working with. Are you a front-end engineer? Then this doesn't even make sense because you wouldn't be putting that much into storage anyways that O() is going to matter.
I'm an iOS developer and quite frankly you shouldn't be doing that much business logic on the phone. We implemented an algorithm that had a O(n^4) runtime without optimizing it because the dataset was so small it wasn't worth the time to over optimize.
You're looking for an extremely rare case and discrediting 95-99% of the time we're programming. Sure we can do speed-ups but do you really have to filter every single developer for those criteria? You really don't. Guess what? Unicorns are built off the backs of medicore developers. The codebases once they start expanding are highly unoptimized. Slack was built from contractors. Twitch apparently is still having growing pains on their iOS app because the original developer wasn't very skilled. I was turned down from thumbtack even though I knew more about iOS than the iOS team lead that interviewed me because they valued CTCI more. The iOS team lead was taking advice from me about system design. What would you rather have? Someone that knows system design or someone that memorized how to serialize an n-ary tree? What is actually going to make you the bazillion dollars? A functioning app that can scale, or someone maybe helping you serialize a data structure (with a very known solution) on an iOS app which is completely useless?
It's simply an illusion that you need to have developers who have memorized useless problems in the case that they may one day possibly might see an issue they could optimize 1% better by introducing more complicated code.
Whiteboard interviewing based on these things isn't useful unless the candidate is being considered for a position in which knowing these things is fundamentally necessary (e.g. they'll be responsible for developing new algorithms or squeezing out the last 10% of optimization available).
On the other hand, substituting requirements for having public code profiles, and the assortment of take-home projects, in-office projects, pair programming to solve a problem the company is facing (especially if unpaid) is just as bad, for different reasons.
And certainly not asking technical questions is problematic.
I just don't know what the solution is. It's one of those things where I know a bad interview when I see it.
I also ask 10,000 foot questions. "Say you're wanting to build a new app that behaves like Instagram. What's your hosting platform, your technology stack, and explain why?".
Yes. How does the person think and work? I've interviewed ops folks who are technical gurus whose walked through a problem with me and pulled up with "well, in that case the server is fine and it's not my problem". They can work somewhere else, I don't care how good they are at "making the server work".
I've also interviewed ops people who are all "well, if performance in production has degraded I want to sit down with the source control tools for your config management system and apps and work through the changesets with the devs and work out where we should start looking." I don't give a fuck if that gal needs to google some of the answers to the problems she finds, she's got what I want.
Please forgive the uncomfortable question but isn't that a tacit admission that many developers' role consists of simply gluing together libraries and frameworks built by better engineers who actually do know how fundamental algorithms, etc. work?
that being said, you're not wrong. Maybe 5% (maybe less) of the guys out there are smart enough to build libraries and frameworks. the next 45% of the guys out there are smart enough to see all the pieces and figure out how to fit them together.
the other 50% understand syntax and can write code, but aren't capable of any sort of bigger picture stuff.
and if my recent interviews are any indication, the split outside of silicon valley is more like 2/18/80.
Shoot, I'm not in that "let's build brilliant frameworks" crew. I don't think I am at least. Haven't really tried :D
"Cut and paste fragments from {SO,Github,Gist,.etc} and edit to taste"
Of course, I am idealizing the experience (eventually I really needed internet and ended up going to public libraries to use it)... but like everything, maybe we are getting too used to the instantaneous and forgetting that the alternatives are actually not so bad and sometimes empowering.
You don't need to be a genius to solve them. You just need to spend some time practicing them and reading cracking the code interview(OR you need to be a recent college graduate who did well in their classes).
I agree that the super difficult Google style questions are silly, but if an engineer can't solve an easy-mid level difficulty whiteboard question (For example, I usually ask something like, "traverse/print a tree"), well... my response to them is why didn't you prepare for it?
You really should have known that you were going to be asked to do something like this, and you didn't even bother to prepare for the easy stuff.
Do you really want to hire someone who didn't bother to put in even the tiniest, most basic level of effort into preparing for your interview?
Its like going to an onsite interview and asking "so what does this company even do anyway? I didn't check before coming here today."
Would you deny what could be the best possible candidate because they didn't do this?
Why? Because they don't care. Why would I want to hire someone who cares so little about the position that they aren't willing to put in the most basic, easiest level of effort?
If they don't care about preparing, then they probably don't care about other things, like the quality of their work.
(disclaimer) From my freelance experience in EU and the amount of "fun & satisfaction" from a the actual work is inversely proportional to the amount of drilling and grilling during interview.
I'd rather implement Bubble Sort, tbh...
which, I agree that is usually always silly
the naive solution would say, hey, this whiteboard stuff is a synthetic benchmark. just use a real one! (ex: solve a coding problem with access to the internet)
but for large companies (and this applies to school applications as well), it is not merely a process of evaluation, but of filtering.
there are more qualified candidates than there are positions. and thus whatever process is used to reduce this number will always be somewhat arbitrary
perhaps it would seem less "unfair" if it were random, but is it really?
<sarcasm> The problem with Python is the arcane, arbitrary and nonsensical syntax. It sure as hell isn't anything as obvious as length(str) or string.length like any reasonable language.
And god help you if you want to do something complicated like define a function. </sarcasm>
But seriously: I have no problems with the rest of the article, but that one just seems silly. How many times do you have to look something like that up before it sticks? For those of you not versed in the more esoteric secrets of Python, you would use the built-in function len, as in len(str).
Of course there are alternatives: http://stackoverflow.com/questions/3992192/string-length-wit...
Edit: Added <sarcasm> tags for clarity.
Each time I get to the part where I need to modify the path which the import mechanism uses to look for modules, clean up the loaded modules to make sure I'm not reimporting an already loaded module by the same name, in order to import a module which is in the SAME DIRECTORY as - get this -__file__, I need another coffee.
Also "len(...)" is not obvious.
For me the answer is "never", apparently. Then again I tend to struggle with basic syntax no matter what language I'm using or how frequently I use it.
If it makes you feel any better, I am in the same boat.
I was interested in seeing the discussion here but it looks like the article is getting penalized for some reason.
Instead, "Build X in 1-2 hours using your computer and resources available" was always a better proxy for productive engineers and largely removing both false positives and false negatives.
If we judge candidates purely on work experience that just favors the incumbents who have already broken into the industry and presumably already quite comfortable in life.
then you'll only select for people who have nothing better to do with their time than to donate it to your fun programming escapade. Experienced devs will say it's not worth their time and wont apply.
Effort does not only include writing a cover letter that will be appealing for the company you apply to instead of just forwarding HR your resume. It also includes small activities like these.
You sound more like a spoiled brat, pardon my French, than an artist looking for new opportunities to make a difference.
The only "difference" I want to make is in my checking account. I want the deposits from my employer to "differ" by greater margins as time advances.
1-2 hours on a practical assignment vs 3 months cramming suggested by the parent comment. Which of those has nothing better to do with their time?
> Experienced devs will say it's not worth their time and wont apply.
Will they? I'm an experienced dev and think this approach sounds great.
I think a good approach is to not emphasize the whiteboard as much for senior hires, so people with good work experience can mostly get in on their resume and design questions and people with lesser experience still have the opportunity to break into these top companies by grinding algo questions.
I think a good approach is to not emphasize the whiteboard as much for senior hires, so people with good work experience can mostly get in on their resume and design questions and people with lesser experience still have the opportunity to break into these top companies by grinding algo questions.
When I interviewed a couple of years ago, I saw various kinds of interviews at companies:
* 1-2hr on site coding exercise, with the internet, alone.
* 'Here is a 100loc piece of code, fix all the problems with it'.
* Chat with product / manager / team lunch. ('Are you an asshole' test / 'culture' interview / scenario testing)
* Design / outline an app that does X, with boxes and lines on a whiteboard.
* Solve this algorithmic problem on a white board / on your computer.
The reason why everyone freaks out about the algo interview it's the only interview type that most engineers have to prepare for. The other 4 usually you don't have to do much prep work at all.
If a person has been in the industry for more than than three years and doesn't have 3 months cash reserve saved up[1], they are severely mismanaging their finances. _Particularly_ since we're seeing increasing warnings from the financial sector that we're in a bubble already.
[1] barring some unexpected catastrophic expense
If anything, this just tells us what a low bar to entry the CTCI drill is.
Say you had a pool of candidates of roughly comparable intelligence, educational background, and work experience. And then you gave them 3 months of all living expenses paid† to go an "improve themselves as software development professionals."
One of them comes back and says "I cranked out an MVP (relying in part on a language I only barely knew to that point) that's actually being used by people, and getting good reviews. Here, check out he repo."
Another says, "I thought it was about time I got a handle on this machine learning stuff. Here's a project I did, based on a generalization of some ideas in such-and-such paper. Wish I had more time to spend on it, but looks like my error rate's not too bad."
And another says "It took literally 6-8 hours of devoted practice, every day. But I finally got some serious traction on Japanese. I honestly thought I'd never be able to do that."
And then there's the guy who says "Oh, I looked myself in my apartment and did every exercise in CTCI. Because, you know, I heard that's what you're supposed to do. Damn, my back is sore. I think I need a better chair or something."
Who would you rather talk to first?
† Which costs a lot more than 35 bucks, BTW.
1. Algorithm and data structures trivia: which includes the classic CtCI [1],
2. In-house problems: where you have to analyze and propose multiple solutions to a problem that has been faced by the current development team in the company that is executing the interview,
3. Homework-like projects: which usually take 3-7 days to finish.
After an exhausting 2016 full of interviews I do not know which one is worse.
On one hand, having a common set of problems like the ones proposed in CtCI is good because you just prepare once (kind of) and can apply to multiple companies at the same time. On the other hand, in-house problems are good if you really want to work for a specific company and want to know what are the problems that your future co-workers are facing, the problem is that you have to be well versed in a wide range of concepts to give at least one answer that pleases the lead developer. And finally, the take-home projects, I think they are the worse in comparison, you are expected to work for free for at least one day in a problem that is either very basic or too complicated but they always expect you to have a perfect solution just by yourself. How many times have you done this in your own job? Do you start working in a solution to a problem without asking questions? Without a discussion with your colleagues? Without full specifications? Without multiple attempts? Take-home projects are really not suitable for an interview.
I understand the point of these tweets but I doubt they will change anything. Much of the craziness of nowadays interviews comes from the recruiters who have no idea about these problems, they don't know algorithms and/or data structures, they don't know what are the problems that the company is facing, they only care about the money that the company will give them if the candidate is hired. Now you know why websites like Hackerrank [2] are so popular, recruiters love to have a standard set of questions and if your answer is not the exact same that is in their sheets you lose.
From now on I will just resort to colleague's referral. With this I know that at least one member of the team will vouch for me to join the company and make the interview process as painless as possible because they already have an idea of how good of an engineer I am.
[1] https://www.amazon.com/dp/0984782850
[2] https://www.hackerrank.com/
One of the solutions my team has come across for this (also mentioned in previous HN threads) is to pay the interviewee for a day's work or give them an Amazon gift card for an equivalent amount. And we aren't looking for a perfectly crafted solution. We're just trying to get a handle on their general coding style, maintainability, coherence from variable naming and/or comments, DRY, whether they bother with edge cases in their solution, robustness/anticipate other types of errors or bugs that may pop up, etc.