Whats funny about WBIs is that if you decide for yourself during the interview that this is not the company for you- you can derail the conversation into unfamiliar territory's and watch the interviewers struggle.
Why is that funny? Most interviewers are just doing what their company asked them, to give a whiteboarding problem to a potential candidate and figure out if they're worth hiring based on that.
I agree the process sucks and it's a hoop to jump through, but why take it out on the interviewer?
Better ways to do that than spend 5 hours in front of a whiteboard. Never gone through a whiteboard interview and then thought to myself that was time well spent, neither on the interviewer nor interviewee side.
I've written integer programs, ray tracing simulation software for fields of heliostats, custom orchestration tools and deployment pipelines, etc. and not once has anything said or done during a whiteboard interview been relevant to any of it. The actual job is pretty much orthogonal to whatever the interviewer thinks they are figuring out during a whiteboard interview and as above the stuff I've worked on goes over the heads of 90% of the programmer population so 9/10 they can't and are not qualified to judge things properly but somehow they still think through the magic of the whiteboard they'll divine the candidate's true capabilities.
When working on hard problems I go to the library and don't stand in front of a whiteboard to see if I'm smart enough to reinvent some wheel. Chances are someone else has already figured it out and I just need to read some paper and then implement it. If they haven't figured it out then that means the problem is harder than I thought and will either need to rethink my approach entirely or spend my time on something else.
Unless of course the job is to stand in front of a whiteboard and wave furiously, in which case you should be using a whiteboard to interview.
Doing their job is a perfect excuse for doing a whiteboard problem during an interview. If the company has standards for interviews, you should follow them, it would be pretty arrogant to just do it a different way because you think you know better than the standards.
Thats a missunderstanding on what standards are. Standards are minimal levels to fullfill- everyone is free to surpass them. And the first sentence of any well written standard is- "This rules may not apply universally, use your brain in case of emergencys".
I get that its a horrible thing for some to consider that there is creative freedom involved in work and not everything in life follows exact and well defined rules. Anything else, is just holding humanity back. I ve seen standards written by assembly programmers, still alive and walking the earth, torturing hundreds and undead for ever, because what is once set in motion.
Really good standards, standardize intentions.
"Every Programm shall consists of layers and components. Every component shall be sperated by a interface."
A bad standard tells you how the interface has to look like- every time.
There is kind of a symmetry to it. Get a question that the asker has studied and watch the answerer squirm. Seems like it'd be more fun, and probably more insightful, to take turns... :-)
So then you're advocating for an appeasement ritual? Most interviewers would rather feel smart than talk to another person as an equal. More than a few times I've seen candidates disqualified because they made the interviewer feel inadequate. Who exactly is the asshole in that case?
No, I'm not advocating for anything like that. But if you want to deliberately make fun or piss off your interviewer, remember that that may have later consequences on your career. Nothing you do is in vacuum and there's no such thing as "just business" when insulting behaviour is involved.
(Obviously, this goes both ways, shitty companies tend to have trouble getting good people to interview.)
I do whiteboard interviews at the whiteboard interview Mecca (Google), and detecting the digressions and putting the candidate back on the right track, is one of the skills the interviewers are explicitly trained for.
My mental model of how candidates with a CS degree do on a whiteboard interview is something like score = (grade in data structures class) + (grade in algorithms class) - k(time since undergrad) + b(time spent studying for interview). I'm curious if you think that's a good model, and if it is, would it be easier to just look at their grades in those classes, or if you actually care about the amount of time spent studying.
It's not just the time spent studying, it's the right kind of studying. "Teaching to the test" so to speak. I liken it to my driver's ed class in high school. My teacher wasn't teaching us how to drive, he was teaching us how to pass the test. Every single thing he taught us was about how to pass the test. I passed on my first try because I was very well trained.
Unfortunately, the whiteboard thing is not as standardized, so it's somewhat harder to teach to the test, but it is not impossible. People waste a lot of time I think when they can memorize a small number of basic algorithms and then learn to map the question given to one of those techniques.
But writing the code with a marker on a board definitely takes practice as well.
The available training materials are good, but they are not great. A focus on shortcuts would be nice. I remember my verbal SAT improving significantly just from the single tip to read the reading comprehension questions before reading the passage. I suspect there are similar tips that can improve your speed at figuring out the trick that solves these interview problems quickly.
There's a huge variance between different schools, and an average grade from one university may be better than good grade from some different one. Additionally, I don't really ask questions that require some particular knowledge from algorithm classes -- it would be silly to determine whether someone is a good hire based on whether they remember details of, say, Hopcroft-Karp algorithm. On the other hand, many algorithm classes are all about this kind of stuff.
So, I generally feel like it makes more sense to try to determine the actual aptitude when you have an opportunity, instead of trying to measure it by possible decades old proxy with huge and unknown variance. If you had good grade and/or spent some time studying for the interview, and really meet the hiring bar, you'll most likely do well in the interviews anyway. There's some noise in the process, and good qualified candidates are sometimes rejected, but from company's perspective, it's better to have some false negatives, than to give another metric that can be easily gamed, and get false positives. Google has enough qualified candidates applying for a job to make it not that big of a deal if you lose some due to the noise.
Any company with an interview process where the key to passing the interview is "Pay more attention in your Algorithms and Data Structures classes", is a company that I'm never taking a job at.
Why would I want to hire a candidate that paid thousands of dollars for Algorithms and Data Structures classes but didn't feel the need to pay attention? Don't you think that says something about the candidate?
I've been out of college for 10 years, writing software professionally... it's so rare to have to calculate the big-O of an algorithm, or use a data structure much fancier than an array or a hash table, or implement my own search algorithm. And if I had to do any of those things, I would look it up and leverage this collaborative thing we've created called the internet.
So why should I be expected to regurgitate trivia at a whiteboard to get your job?
It's weird to me seeing statements like this because I don't work on any esoteric or fancy code, just frontend UIs, but I'm frequently making data structure and algorithm decisions.
If I'm understanding 1_800_UNICORN's point correctly, it's more along the lines of 'any company that expects an experienced developer to recite academic details on algorithms'. Certainly my current knowledge of algorithms is very practical and I'm certain I've forgotten much of the academic stuff I learned 20 years ago. You remember that which is useful, forget that which is not (situational, of course) and when a company is looking for the latter, they're unrealistic or they want an academic programmer, which I'm not.
An essential element to the code that I write on a daily basis is a strong understanding of data structures and algorithms. Why not test this understanding?
But do you still do it using the book methods taught in college? If the interviewer is okay talking about the concepts instead of the lingo, that seems okay. But don't expect people to remember textbook details for years.
As an aside, I've done many, many interviews, and for a long time my manager insisted on jumping in and asking candidates to recite the components of SOLID and explain them. You can probably guess that very few (none that I can recall) got it on the first attempt. Good candidates could talk about the concepts if you reminded them what the acronym stands for. To me this means the question is useless. Ask about the concepts and skip the trivia.
I don't expect them to remember textbook details either. But I expect people to know how to manipulate and understand an unfamiliar data structure or understand when a given data structure is appropriate for a particular problem domain. I expect people to understand (or be able to derive) when memoization is appropriate. I expect people to understand (or be able to derive) how using different data structures or representations of these data structures might affect memory layout and cache performance.
That's essential stuff, not something to be dismissed as "textbook details".
Sure, but my experience searching for jobs indicates that those forms of questions are very rarely asked. Instead, questions like "in a N dimensional space filled with M points, find the point(s) closest to a given point" or "figure out if this list of pointers has a loop", or "implement quicksort in Java".
If you don't use these as your whiteboard questions, great. But I've not run into you or others like you in my own searching.
If an org does WBI's then it's a good indication that they are not choosing the best applicants. Instead of either getting good at them or trying to get the industry to change, we should embrace this.
If they do WBI's then they probably do lots of other things which run counter to best practices. Isn't it so much better to expose to the outside world a culture of cargo culting inability to evaluate techniques for the efficacy.
There are so many orgs who do run themselves well. The real tricky party is how to determine that quickly. With WBI's we can get a short cut.
Mostly pair-programming coding session on site or homework that is later discussed and "developed" on site. Candidate picks which option works best for her.
Conversation, half technical and half to figure out what the person would be like to work with. For example, tell me about the most interesting/difficult/satisfying problem was that you've worked on. Or, tell me what happens when you point your browser at a web site, in as much detail as you can muster. What are your go-to tools to solve problems
>If an org does WBI's then it's a good indication that they are not choosing the best applicants
I don't think the data supports this. Say what you will about Google, but they do WBIs and the average quality of their engineers is very very high. Same with FB.
I'd like to agree with you but my current gig involved a WBI and the company itself actually has stellar software development processes and tools. It might just have been that the hiring manager wasn't too comfy doing it any other way, but in any case it did not correlate strongly to the culture inside the company.
Perhaps use it as a warning sign, but you might want to talk to your potential co-workers as well and see what they're actually doing and how.
> If they do WBI's then they probably do lots of other things which run counter to best practices.
And what are the best practices for interviewing? Whiteboard interviews are employed by plenty of successful software companies (Google, Facebook etc), "best practices" implies there is something clearly accepted as more effective that they should be using instead. What is it?
I don't get the hatred towards white boarding during interviews. That's exactly what I do when brainstorming design, general architecture, state machines, explaining data structures used etc. The use of white board in an interview also covers the same topics, when I'm interviewing candidates.
Now if you say whiteboard programming, and that the candidate has to pay attention to curly braces and semicolons and typos, then I agree its not a great process.
So for me, whiteboard interviewing is perfectly fine, whiteboard programming is not.
Whiteboard interviews have a very high false negative rate (rejecting people who are good), but they also have a low false positive rate (hiring people who are not good).
On what basis is this statement made? No study or source appears to be cited. The specific false negative and false positive rates are not given. Also, what exactly is the definition of "good" used?
It's not possible to compute the false negative rate because once you turn down a candidate you get no more information. I suppose if people apply multiple times and are accepted later and then go on to do well on the job, you can count them as false negative in the first interview. The sample size for that would be small though. And I'm sure the true believers would say that these candidates improved their programming skills between the two interviews, not their interviewing skills.
I am quite interested in the false positive rate which can be computed by correlating performance reviews with interview performance.
> I suppose if people apply multiple times and are accepted later and then go on to do well on the job, you can count them as false negative in the first interview.
Isn't it more likely that they were correctly rejected multiple times and just got lucky enough to squeak through one time?
> I am quite interested in the false positive rate which can be computed by correlating performance reviews with interview performance.
You would still be excluding all the false negatives making it kind of relative performance of interview survivors. Meaning that you won't have anything to compare your false positive rate to proclaim it as low or as high.
I've never understood the hate that WBI get here and elsewhere on the internet. I see them the same way as standardized tests: without them, there's no good way to independently evaluate people's abilities in a way that is immune to cheating, dishonesty, or fraud (e.g. stealing someone's code and putting it on your github repo, being a terrible employee at previous jobs but just scraping by long enough to move onto the next company, straight up lying on a resume, etc.).
Sure, as the author notes, WBI have a high false negative rate. But if you're actually a good programmer, learning basic CS concepts should be extremely easy, and these types of problems, perhaps with a bit of practice, should be relatively trivial. I personally wouldn't want to work with someone who can't extract a path from a DFS, not out of any sense of elitism but just because such an employee likely would be unable to do many other problems properly.
I'd argue that there is a not-insignificant amount of overlap between whiteboarding and coding. Most coding in practice is not as "algorithmic" (in the sense that the problems are theoretically challenging) but having the knowledge and know-how of how to write optimal/efficient algorithms will definitely be echoed throughout any code you write.
In addition, in my experience most WBI are not particularly challenging problems that you have probably never seen before. I mentioned the ability to extract a path from a DFS; I would argue that this is such a basic CS skill that being unable to solve that problem indicates an insufficient amount of knowledge of CS.
There are definitely WBI that are poorly matched with real job requirements and knowledge. I think these are often seen at smaller companies trying to copy FB/Google without knowing what they are doing. WBI that involve using important CS constructs such as a binary tree or a directed graph IMO are testing important concepts
Sure, but you could make that argument about anything you ask candidates in interviews. At some point you have to ask something that attempts to evaluate their performance in an interview timespan.
On the other hand, people's performance in whiteboard interviews may positively correlate with later performance as employees. In which case they let you make a reasonable assumption about later performance without being able to evaluate it directly, which is really what interviews are all about.
> Sure, as the author notes, WBI have a high false negative rate.
That's why this practice should be abandoned. Give the candidates a small, two dayish homework that reveals their ability to do their would-be daily jobs. If it looks good, call them in and talk about their submission. Heck, you might even ask them do draw some diagrams or detailed explanation of their own solution.
Candidates that have offers from other companies that didn't ask them for homework will probably go for these companies first. We scheduled 8 people for interviews last week and 2 of them dropped out before their scheduled interview times because they took jobs elsewhere. Unless you're a top-tier tech employer, you don't have the luxury of asking candidates to do leg work.
These assignments are tasks that are no way could be used in production, at better places these are even paid assignments. I've seen more than one ad like that.
Small companies do it here (Europe), not sure about the US. I would happily do the leg work if I really dig the company. If they drop out from the queue, they probably prioritized you lower, not a big loss.
A friend of mine got absolutely amazing task from one company: write and publish on github extension to LLVM. It was some really simple language, nothing practical, but this whole task tests so much skills so my friend was really enjoyed doing it.
> That's why this practice should be abandoned. Give the candidates a small, two dayish homework that reveals their ability to do their would-be daily jobs.
I find the opposite. Most real world projects require a week or two of infrastructure setup (logging, error handling, IOC setup, database setup, etc) that you won't have in the two day version, yet in my experience most companies will grade you on the lack of such things.
Also, anyone already employed isn't going to invest that much time, which largely limits your candidate pool to employees no one else will hire.
Standardized tests, while a great idea on principle, leads to extreme optimizing to simply pass exams with less regard for the material that's meant to be learned. And they can be cheated:
The question is not whether the test can be gamed, but whether WBI have sufficient predictive power to be used as an interview tool.
I think this is absolutely the case at large companies which need to screen hundreds of thousands of candidates per year, perhaps less so at smaller companies.
Your assumption in the very first paragraph is that they are a good way to independently evaluate people's abilities.
It makes perfect sense to look for a way to evaluate people before we hire them. It's a real problem, and we must do something, but do we have any basis to say that the whiteboard interviews correlate with good performance. Otherwise, we go back to the classic "syllogism" to justify policy proposals
1. This is an important problem, and we must do something.
2. X is something.
3. Therefore, we must do this.
I've been developing professionally for over 15 years, and I am quite good at whiteboard interviews: 100% offer rate. That said, Many of the best developers I've worked with are not any good at whiteboard interviews, often because they lack the composure under pressure, or like to take some time to think everything through, which are correlated with scoring badly in those interviews. Therefore, no matter how good their code is, they'll have a lot of trouble getting into a company that does whiteboarding interviewing, or even practical interviews under time pressure.
I also have found that, in my list of things on what makes an effective member of a programming team, being able to come up with a DFS algorithm under pressure is nowhere near the top of the list. An attitude towards trying to learn what others bring to the table, instead of discounting them immediately because they aren't good at some specific CS shibboleth, is far closer to the top though. We just test for random algorithm questions for this because it's easy to measure, not because it's really all that helpful, and that's pretty unfortunate IMO.
Depending on the exact role and project, that can be a severe negative as well (e.g. devops). In that case, putting them through the stress of a WBI is a side benefit.
The stress of trying to land a job is different than the stress of trying to meet a deadline at a job, no?
And Im not saying that meeting deadlines is the only stress, Im just saying that being stressed out about how you will provide for yourself is different than the stress of having to perform well at a job
> I see them the same way as standardized tests: without them, there's no good way to independently evaluate people's abilities
How about straight up pay them to solve a real problem with you? If they're clearly incompetent, cut it short fast (under an hour). If they're competent, you know that they're competent at the job, not at a whiteboard interview. Yes, I've seen this work at small-medium outfits.
> I personally wouldn't want to work with someone who can't extract a path from a DFS
What if this person hacks blockchain merkle trees for breakfast, but can't be bothered to regurgitate tangential runtimes of whatever algo you decided to ask about, which they memorized and forgot 10 years ago? I've seen this to be a nontrivial intersection of skills.
We'd turn on a dime and ask them to tell us about their merkle tree hacking on the white board, and the poke at it. The goal is to get a sense of people's skill, and we need to understand that they're not regurgitating something they've memorized ahead of time. Someone we interviewed last week was hoping for fizzbuzz and couldn't answer an equally simple question!
There is no chance that I can find a problem that I need to solve that I can get a candidate to work on and finish in a day. Too much custom infrastructure and technical background. Too many problems with IP. Any code I have a candidate write will get thrown away.
So then why is a whiteboard interview different than a real problem? They don't have access to a developer environment but that's the only major difference.
One should be able to reason about running times, not memorize them. If a candidate cannot explain why their implementation of an algorithm they've never worked with is actually quadratic in disguise, for example, then I'd need to babysit their code if they were hired to make sure the system stays efficient.
> What if this person hacks blockchain merkle trees for breakfast, but can't be bothered to regurgitate tangential runtimes of whatever algo you decided to ask about, which they memorized and forgot 10 years ago? I've seen this to be a nontrivial intersection of skills.
But most runtime derivations are pretty basic math and logic. You don't need to memorize a runtime of some algorithm, because if you understand the algorithm and what runtime analysis is about, you can derive it (in an informal hand-wavy way, at least) on the spot on the whiteboard. I wouldn't expect a rigorous derivation of the master theorem during an interview, but being able to reason a bit about worst-case scenarios and the iteration bounds of nested loops or a tree traversal seems like a pretty fundamental skill if you're discussing algorithmic problems.
The cost to an employer of a false positive (hiring someone unqualified) is probably 1-2 orders of magnitude higher than the downside of missing an actually-good coder who can't pass a WBI.
Not only does it take a lot of time and bandwidth to fire someone incompetent, but now you have to deal with the damage they've done to your codebase and projects in the meantime.
Even if your WBI process rejects 10 good coders for every bad coder it weeds out, you're coming out ahead.
As for all these posts from famous or well-known hackers who assert that they don't need to show that they know their stuff -- sure, if you have a portfolio of open source projects that everyone uses, then you have the luxury of not needing to demonstrate your technical competence.
How does that help employers evaluate anyone else?
1) All new employees go on 90 day contract-to-hire. So reduce the firing overhead to 0.
2) Similarly, just build some sort of extra ring of QA around new hire code, so all their codebase changes get reviewed by a peer -specifically to assess their long-term employment at the company.-
Because nobody gets assigned a task and rushes to a white board with a non-collaborative audience. The WB wire framing etc is always a discussion amongst peers. Sit your prospects at computers spec'd like they would be using in their role with appropriate Internet access. Knowing what to Google is generally the most used function. What is googled varies with each individual.
The pressure of dumping pseudo code is a poor metric.
A good WBI should feel like a discussion with a peer. While I think the interviewer has to be careful not to direct/guide things too much, they should be giving input/feedback/questions along the way, much like a coworker on a real design problem might. Dumping pseudocode shouldn't ever be the metric to evaluate a WBI, but just a product of the conversation with the interviewer, which is a far more interesting indicator.
I agree but have yet to find one that went that way. More companies in the wild, as I alluded to, who feel that's what they should be doing with little to no understanding of how. Same for all of those brainteaser fizzbuzz-esque problems.
Good ones aren't as common as they should be...bad interviews in general are disturbingly common, though, I don't think tech interviews in general or whiteboard specifically are any different.
Brainteasers and fizzbuzz are very different things though: I'm not totally against brain teasers as a prompt for discussion, but they're not my favorite types of questions, and can be hard to judge or of little use if the person either already knows the trick, or just doesn't figure it out. Fizzbuzz on the other hand makes a lot of sense when you're interviewing someone who you don't have any other knowledge of their coding. That type of question absolutely makes a good first pass filter: the problem is easy to explain, the implementation is trivial to do in a few minutes if you are at all qualified. Sure, it's annoying for an experienced person to have to do a fizzbuzz variant for the 200th time, but given that it takes almost no time, and helps the interviewer quickly jump to more advanced stuff or cut off an interview that isn't going to go anywhere, it's a pretty minor inconvenience.
>Whiteboard interviews have a very high false negative rate (rejecting people who are good), but they also have a low false positive rate (hiring people who are not good).
That's a pretty bold assertion. Is there data to show this?
Reasoning behind why whiteboard interviews suck is based that doesn't model real job skills. Maybe our org is an outlier but but we heavily rely on whiteboards to plan out architecture(ERDs,flow), problem solving, training/knowledge sharing. Our trello cards sometimes contains images of whiteboarded diagrams. Being able to explain and whiteboard your stuff is a very important skill at organization.
Sketching out ideas and preliminary diagrams is very different from programming on a whiteboard. If a whiteboard interview were talking at a higher level about the problems at hand then I don't think it'd be such a problem. However, that doesn't seem to be the case. In whiteboard interviews you're asked to write detailed code on a board.
I'm fairly convinced much of the hate comes from people who interview at firms who do WBI incorrectly. The question I give is tailored for the following points:
You can pick up unfamiliar code/data structures easily
You can understand a given problem space without too much explaining
You can code the "easy stuff" without thinking about it too much
You can distill and discuss the difficult parts of an algorithm
All of which are traits I consider key for someone I'll be working with.
WBIs will go away if the developer community rallies around not doing them. Companies can offer crazy incentives to work there like free massage, organic lunches, and game rooms, but they can't figure out a better way to hire than conducting WBIs? I call BS. They'll figure it out when they're pressured to do so...aka they can't find good talent anymore. Let's put on the pressure.
There is no benefit for people who are good at WBI's to join you in protest, and the bad people weren't going to pass them anyways, so the companies won't care either. On top of that, anyone giving a WBI got hired with a WBI, so the cycle continues.
I am not saying WBIs are good or bad - just stating that this kind of rally won't have the desired effect.
In a similar vein, I view whiteboard interviews (and required prep) similar to professional exams, like the Bar exam for lawyers. While I think it's unreasonable to expect good professionals to be able to pass such exams without a lot of practice, it's perhaps not as unreasonable to expect good professionals to be able to pass such exams if they study. Nobody likes tests, but I suspect that a high percentage of the good programmers who belly ache about whiteboard interviewers could in fact get good at them if they put in the time.
So yes, whiteboard interviews or not optimal and they are unfair, but if you are rejecting them out of hand and refusing to study for them, you are only taking away some opportunities.
I say this as someone who hasn't had to do a whiteboard interview for 10+ years. The jury is out as to whether I'll take the pain to study for them again should I do a full job search among tech companies again.
Also: note to author, I believe, "They have little co-relation" in your opening paragraph should be "They have little correlation".
I used to believe that I got rejected after many whiteboard interviews, not because I did poorly, but because I didn’t have the same (right) background as the interviewer. I served in the military, I was a Front End Engineer, I didn’t go to a fancy school, I used the wrong language, I’ve used the wrong framework etc.
I think the author is hitting on some deep-seated psychological aversion people have to white board interviews. If you can identify that you were turned down based on your failure to do it well, you have to accept that it's your fault - not that you happened not to be good at that tech stack, or didn't have a "personality fit".
(I know there are different kinds of white board interviews - I'm assuming the generic pseudo-code algos and data structures type in keeping with the article's generality.)
Exposure to that kind of evidence of your inadequacy can be hard to accept. But it's also remediable. Get cracking and practice.
Small project, on your own time, expected to take about 3 hours or so. Automated tests can do the very basics of evaluation. Then, 1 to 2 hours explaining the decisions made, and adding one more feature to the project. It lowers stress levels of the candidate, as they get to a lot of the thinking in advance. It's practical, as it requires more detailed programming skills. It provides a better sample of the candidate, as 5 20 lines of code snippets don't compare with a larger codebase, and makes the candidate explain something they really did themselves. Total time spent by the candidate is probably about the same, but the employers saves engineering time.
I've been through that system a couple of times, it was both more enjoyable and gave people a far better picture of my skills than just how well I remember a tree traversal algorithm that I haven't had to write in forever, as data structures often handle their own traversal.
need context to determine what would be better. giving the same interview for every position you're hiring for is not usually the right thing to do. Google is very unusual in their practice of hiring into a general pool. Most companies hire for more specific openings.
In general though, when I interview it has three main sections, tailored for the role.
First, is conversational. Talk about the resume, talk about work experience, talk about technology, etc.
Second, is a practical example. Sometimes (but not always) this involves solving an algorithmic problem on a whiteboard. Sometimes it involves a guided tour of the codebase and functionality of a relevant project the candidate has worked on. Rarely it might be a take-home coding task.
Third, is a more general example usually of some kind of system design problem. A lot of candidates like to write/draw on the whiteboard for this section but it's not strictly necessary nor do I ask them to do so. There's a board in the room if they want to but I don't ask them to use it.
Have a conversation about "how would you solve this (real) problem" - the same kind of conversation you have with a peer when they're stuck on a problem. If both you and they are professional developers, you will be able to get into a nice in-depth discussion of the pros and cons, and go into nice tangental discussions along the way.
If they aren't a professional developer, it will show in no time at all; the longest I've let such a conversation go was 10 minutes just to see how long the interviewee would keep trying and snowball me.
I've found this is simply much faster and more informative about their experience with subjects that matter in the daily work of a software dev.
What I remember about much of that code I looked at in DS classes way back in the 80's was that, due to very limited memory space, all the variables were like I,J,K and X,Y,Z.
You have no idea how much more difficult that made it to follow sorts and tree algorithms. Thankfully, I seemed to have some sort of coding gift that allowed me to write the damn things at the terminal and make them work.
I got A's in all my CS classes, but now I guess I am so used to the magics of PHP object handling I've literally stopped caring about that shit.
I think one of the real problems that doesn't get discussed very much is Google-envy. A lot of companies that don't really have problems like Google, and don't really have organizations like Google, and don't really have applicant pools like Google, will yet interview the same way Google does.
It really might be the correct way to interview for Google. Certainly they've put a lot of thought into how to manage their hiring process and they still lean heavily on the whiteboard. I'm almost positive, however, that most companies are really nothing like Google and really should not be using the same interviewing techniques.
Smaller companies with more quotidian problems would be better served with a more conversational and less time-pressured interview style.
i 1000% agree with this. there is no good way to really know whether a candidate is good or not, but their ability to express their thoughts, in code, on a whiteboard is a really strong indicator that they can think through a problem and (more importantly) communicate a solution as a software engineer.
the next best approach --- a take home assignment --- is also good at this (or better, IMO, since you're free to implement that solution however you damn well please and it more closely translates into the work they'll actually be doing), but people complain that this is too invasive of their time and that they are doing free work. which is it?
I have to imagine that what most people want is to be asked about their experience, talk through it, and expect that to be enough. which is a very solid first pass filter. it's just not strong enough of a filter to extend an offer with confidence in many situations; it's too easy to bullshit and most of us engineers are damn expensive. (it's not just salary; we want dedicated offices and sit stand desks and super comfy chairs and free food and drinks and games and cadillac health insurance and 401ks and so on and so on)
i also never understood why whiteboarding was hard. if you know how to code and know your language passably well, then why can't you write code on something that isn't a computer?
I have hired a lot of developers, working for different companies in different countries. And I have never ever seen a case where a developer did well in the WBI interview, was hired, and then turned out later to not be a good developer. However I have seen plenty of examples of developers (interviewed by somebody else) who didn't do a WBI interview and later was fired because they were not good developers. My point is that if you can't do a WBI, well then perhaps you are simply not as good a developer as you think you are. Developing is about thinking. And WBI style interviews force you to think for yourself without copy-pasting a solution from Google.
I think whiteboard questions are great if you make them either pretty easy questions, or more complex questions that you just diagram and discuss. In the first case, you filter out poor coders very easily, and see how the better coders apporach the problem. In the second case, you learn about their thought process when approaching a problem where you can't just modify an algorithm you have already practiced and answer it.
Im a 'competent' programmer who has done few whiteboard interviews in my time and I have failed in most of them. Here is why ?
I consider myself a competent programmer since I have developed decent work@work and my peers recognise it too. But how would any interviewer can know about this ? My resume is like any other resume in tech world. They cannot. So I've to be put through WBI. For many years I couldn't have known what was wrong with me. Every time I failed I felt miserable to be honest. Eventually, I have narrowed down the reason. It is not just at interview I fail to problem solve - even when I try doing at home with a white board and 'explanation' I fail. English is not my region language but one would not doubt on my conversation abilities as I manage it ok. It is that I simply fail to explain solving programming puzzles in english since while programming I 'do' think in my regional language and most of the time in 'silent' mode in my head. Now 'silent' mode is again dreaded in interviews - as interviewers they want to know 'how you approach solving a problem'.
I think a better solution to WBI would be to make interviewee do programming questions on a laptop (alone) where interview is held - say make it an hour worth of exercise. Have that program run through some time bound tests (like google code jam etc) and may be do a code walk through later with interviewer. This way interviewer is sure interviewee can code.
Also, one of things that will definitely help would be to have some sort of evidence that you can code (git, code competitions etc). This gives some sort of confidence to interviewer.
If you select stringently for WBI ability, you'll get strong WBIers. However, you're also paying the opportunity cost of not selecting for other traits and skills that might matter a lot more.
+1 though for "Senior Whiteboard Engineer"--that term is genius!
100 comments
[ 3.0 ms ] story [ 185 ms ] threadI agree the process sucks and it's a hoop to jump through, but why take it out on the interviewer?
I've written integer programs, ray tracing simulation software for fields of heliostats, custom orchestration tools and deployment pipelines, etc. and not once has anything said or done during a whiteboard interview been relevant to any of it. The actual job is pretty much orthogonal to whatever the interviewer thinks they are figuring out during a whiteboard interview and as above the stuff I've worked on goes over the heads of 90% of the programmer population so 9/10 they can't and are not qualified to judge things properly but somehow they still think through the magic of the whiteboard they'll divine the candidate's true capabilities.
When working on hard problems I go to the library and don't stand in front of a whiteboard to see if I'm smart enough to reinvent some wheel. Chances are someone else has already figured it out and I just need to read some paper and then implement it. If they haven't figured it out then that means the problem is harder than I thought and will either need to rethink my approach entirely or spend my time on something else.
Unless of course the job is to stand in front of a whiteboard and wave furiously, in which case you should be using a whiteboard to interview.
I get that its a horrible thing for some to consider that there is creative freedom involved in work and not everything in life follows exact and well defined rules. Anything else, is just holding humanity back. I ve seen standards written by assembly programmers, still alive and walking the earth, torturing hundreds and undead for ever, because what is once set in motion.
Really good standards, standardize intentions. "Every Programm shall consists of layers and components. Every component shall be sperated by a interface."
A bad standard tells you how the interface has to look like- every time.
(Obviously, this goes both ways, shitty companies tend to have trouble getting good people to interview.)
... then perhaps it's best to let them know and save everybody some time (including your own).
Unfortunately, the whiteboard thing is not as standardized, so it's somewhat harder to teach to the test, but it is not impossible. People waste a lot of time I think when they can memorize a small number of basic algorithms and then learn to map the question given to one of those techniques.
But writing the code with a marker on a board definitely takes practice as well.
The available training materials are good, but they are not great. A focus on shortcuts would be nice. I remember my verbal SAT improving significantly just from the single tip to read the reading comprehension questions before reading the passage. I suspect there are similar tips that can improve your speed at figuring out the trick that solves these interview problems quickly.
So, I generally feel like it makes more sense to try to determine the actual aptitude when you have an opportunity, instead of trying to measure it by possible decades old proxy with huge and unknown variance. If you had good grade and/or spent some time studying for the interview, and really meet the hiring bar, you'll most likely do well in the interviews anyway. There's some noise in the process, and good qualified candidates are sometimes rejected, but from company's perspective, it's better to have some false negatives, than to give another metric that can be easily gamed, and get false positives. Google has enough qualified candidates applying for a job to make it not that big of a deal if you lose some due to the noise.
So why should I be expected to regurgitate trivia at a whiteboard to get your job?
As an aside, I've done many, many interviews, and for a long time my manager insisted on jumping in and asking candidates to recite the components of SOLID and explain them. You can probably guess that very few (none that I can recall) got it on the first attempt. Good candidates could talk about the concepts if you reminded them what the acronym stands for. To me this means the question is useless. Ask about the concepts and skip the trivia.
That's essential stuff, not something to be dismissed as "textbook details".
If you don't use these as your whiteboard questions, great. But I've not run into you or others like you in my own searching.
If they do WBI's then they probably do lots of other things which run counter to best practices. Isn't it so much better to expose to the outside world a culture of cargo culting inability to evaluate techniques for the efficacy.
There are so many orgs who do run themselves well. The real tricky party is how to determine that quickly. With WBI's we can get a short cut.
Long Live The White Board Interview!
I don't think the data supports this. Say what you will about Google, but they do WBIs and the average quality of their engineers is very very high. Same with FB.
How would you hire?
Perhaps use it as a warning sign, but you might want to talk to your potential co-workers as well and see what they're actually doing and how.
And what are the best practices for interviewing? Whiteboard interviews are employed by plenty of successful software companies (Google, Facebook etc), "best practices" implies there is something clearly accepted as more effective that they should be using instead. What is it?
Now if you say whiteboard programming, and that the candidate has to pay attention to curly braces and semicolons and typos, then I agree its not a great process.
So for me, whiteboard interviewing is perfectly fine, whiteboard programming is not.
Whiteboard interviews have a very high false negative rate (rejecting people who are good), but they also have a low false positive rate (hiring people who are not good).
On what basis is this statement made? No study or source appears to be cited. The specific false negative and false positive rates are not given. Also, what exactly is the definition of "good" used?
I am quite interested in the false positive rate which can be computed by correlating performance reviews with interview performance.
Isn't it more likely that they were correctly rejected multiple times and just got lucky enough to squeak through one time?
You would still be excluding all the false negatives making it kind of relative performance of interview survivors. Meaning that you won't have anything to compare your false positive rate to proclaim it as low or as high.
Sure, as the author notes, WBI have a high false negative rate. But if you're actually a good programmer, learning basic CS concepts should be extremely easy, and these types of problems, perhaps with a bit of practice, should be relatively trivial. I personally wouldn't want to work with someone who can't extract a path from a DFS, not out of any sense of elitism but just because such an employee likely would be unable to do many other problems properly.
In addition, in my experience most WBI are not particularly challenging problems that you have probably never seen before. I mentioned the ability to extract a path from a DFS; I would argue that this is such a basic CS skill that being unable to solve that problem indicates an insufficient amount of knowledge of CS.
There are definitely WBI that are poorly matched with real job requirements and knowledge. I think these are often seen at smaller companies trying to copy FB/Google without knowing what they are doing. WBI that involve using important CS constructs such as a binary tree or a directed graph IMO are testing important concepts
On the other hand, people's performance in whiteboard interviews may positively correlate with later performance as employees. In which case they let you make a reasonable assumption about later performance without being able to evaluate it directly, which is really what interviews are all about.
That's why this practice should be abandoned. Give the candidates a small, two dayish homework that reveals their ability to do their would-be daily jobs. If it looks good, call them in and talk about their submission. Heck, you might even ask them do draw some diagrams or detailed explanation of their own solution.
Not if the cost of a false positive is much higher than the cost of a false negative, which for many employers seems like the likely case.
I find the opposite. Most real world projects require a week or two of infrastructure setup (logging, error handling, IOC setup, database setup, etc) that you won't have in the two day version, yet in my experience most companies will grade you on the lack of such things.
Also, anyone already employed isn't going to invest that much time, which largely limits your candidate pool to employees no one else will hire.
http://www.straitstimes.com/world/extreme-measures-to-crack-...
https://blogs.voanews.com/student-union/2017/02/27/sat-score...
https://www.quora.com/Do-Chinese-people-have-creativity/answ...
I think this is absolutely the case at large companies which need to screen hundreds of thousands of candidates per year, perhaps less so at smaller companies.
It makes perfect sense to look for a way to evaluate people before we hire them. It's a real problem, and we must do something, but do we have any basis to say that the whiteboard interviews correlate with good performance. Otherwise, we go back to the classic "syllogism" to justify policy proposals
1. This is an important problem, and we must do something. 2. X is something. 3. Therefore, we must do this.
I've been developing professionally for over 15 years, and I am quite good at whiteboard interviews: 100% offer rate. That said, Many of the best developers I've worked with are not any good at whiteboard interviews, often because they lack the composure under pressure, or like to take some time to think everything through, which are correlated with scoring badly in those interviews. Therefore, no matter how good their code is, they'll have a lot of trouble getting into a company that does whiteboarding interviewing, or even practical interviews under time pressure.
I also have found that, in my list of things on what makes an effective member of a programming team, being able to come up with a DFS algorithm under pressure is nowhere near the top of the list. An attitude towards trying to learn what others bring to the table, instead of discounting them immediately because they aren't good at some specific CS shibboleth, is far closer to the top though. We just test for random algorithm questions for this because it's easy to measure, not because it's really all that helpful, and that's pretty unfortunate IMO.
Depending on the exact role and project, that can be a severe negative as well (e.g. devops). In that case, putting them through the stress of a WBI is a side benefit.
And Im not saying that meeting deadlines is the only stress, Im just saying that being stressed out about how you will provide for yourself is different than the stress of having to perform well at a job
How about straight up pay them to solve a real problem with you? If they're clearly incompetent, cut it short fast (under an hour). If they're competent, you know that they're competent at the job, not at a whiteboard interview. Yes, I've seen this work at small-medium outfits.
> I personally wouldn't want to work with someone who can't extract a path from a DFS
What if this person hacks blockchain merkle trees for breakfast, but can't be bothered to regurgitate tangential runtimes of whatever algo you decided to ask about, which they memorized and forgot 10 years ago? I've seen this to be a nontrivial intersection of skills.
So then why is a whiteboard interview different than a real problem? They don't have access to a developer environment but that's the only major difference.
One should be able to reason about running times, not memorize them. If a candidate cannot explain why their implementation of an algorithm they've never worked with is actually quadratic in disguise, for example, then I'd need to babysit their code if they were hired to make sure the system stays efficient.
But most runtime derivations are pretty basic math and logic. You don't need to memorize a runtime of some algorithm, because if you understand the algorithm and what runtime analysis is about, you can derive it (in an informal hand-wavy way, at least) on the spot on the whiteboard. I wouldn't expect a rigorous derivation of the master theorem during an interview, but being able to reason a bit about worst-case scenarios and the iteration bounds of nested loops or a tree traversal seems like a pretty fundamental skill if you're discussing algorithmic problems.
Not only does it take a lot of time and bandwidth to fire someone incompetent, but now you have to deal with the damage they've done to your codebase and projects in the meantime.
Even if your WBI process rejects 10 good coders for every bad coder it weeds out, you're coming out ahead.
As for all these posts from famous or well-known hackers who assert that they don't need to show that they know their stuff -- sure, if you have a portfolio of open source projects that everyone uses, then you have the luxury of not needing to demonstrate your technical competence.
How does that help employers evaluate anyone else?
1) All new employees go on 90 day contract-to-hire. So reduce the firing overhead to 0.
2) Similarly, just build some sort of extra ring of QA around new hire code, so all their codebase changes get reviewed by a peer -specifically to assess their long-term employment at the company.-
That 10:1 ratio's not set in stone.
The pressure of dumping pseudo code is a poor metric.
Brainteasers and fizzbuzz are very different things though: I'm not totally against brain teasers as a prompt for discussion, but they're not my favorite types of questions, and can be hard to judge or of little use if the person either already knows the trick, or just doesn't figure it out. Fizzbuzz on the other hand makes a lot of sense when you're interviewing someone who you don't have any other knowledge of their coding. That type of question absolutely makes a good first pass filter: the problem is easy to explain, the implementation is trivial to do in a few minutes if you are at all qualified. Sure, it's annoying for an experienced person to have to do a fizzbuzz variant for the 200th time, but given that it takes almost no time, and helps the interviewer quickly jump to more advanced stuff or cut off an interview that isn't going to go anywhere, it's a pretty minor inconvenience.
That's a pretty bold assertion. Is there data to show this?
I am not saying WBIs are good or bad - just stating that this kind of rally won't have the desired effect.
So yes, whiteboard interviews or not optimal and they are unfair, but if you are rejecting them out of hand and refusing to study for them, you are only taking away some opportunities.
I say this as someone who hasn't had to do a whiteboard interview for 10+ years. The jury is out as to whether I'll take the pain to study for them again should I do a full job search among tech companies again.
Also: note to author, I believe, "They have little co-relation" in your opening paragraph should be "They have little correlation".
I think the author is hitting on some deep-seated psychological aversion people have to white board interviews. If you can identify that you were turned down based on your failure to do it well, you have to accept that it's your fault - not that you happened not to be good at that tech stack, or didn't have a "personality fit".
(I know there are different kinds of white board interviews - I'm assuming the generic pseudo-code algos and data structures type in keeping with the article's generality.)
Exposure to that kind of evidence of your inadequacy can be hard to accept. But it's also remediable. Get cracking and practice.
I've been through that system a couple of times, it was both more enjoyable and gave people a far better picture of my skills than just how well I remember a tree traversal algorithm that I haven't had to write in forever, as data structures often handle their own traversal.
In general though, when I interview it has three main sections, tailored for the role.
First, is conversational. Talk about the resume, talk about work experience, talk about technology, etc.
Second, is a practical example. Sometimes (but not always) this involves solving an algorithmic problem on a whiteboard. Sometimes it involves a guided tour of the codebase and functionality of a relevant project the candidate has worked on. Rarely it might be a take-home coding task.
Third, is a more general example usually of some kind of system design problem. A lot of candidates like to write/draw on the whiteboard for this section but it's not strictly necessary nor do I ask them to do so. There's a board in the room if they want to but I don't ask them to use it.
Have a conversation about "how would you solve this (real) problem" - the same kind of conversation you have with a peer when they're stuck on a problem. If both you and they are professional developers, you will be able to get into a nice in-depth discussion of the pros and cons, and go into nice tangental discussions along the way.
If they aren't a professional developer, it will show in no time at all; the longest I've let such a conversation go was 10 minutes just to see how long the interviewee would keep trying and snowball me.
I've found this is simply much faster and more informative about their experience with subjects that matter in the daily work of a software dev.
You have no idea how much more difficult that made it to follow sorts and tree algorithms. Thankfully, I seemed to have some sort of coding gift that allowed me to write the damn things at the terminal and make them work.
I got A's in all my CS classes, but now I guess I am so used to the magics of PHP object handling I've literally stopped caring about that shit.
It's hard to see why I would want to again.
It really might be the correct way to interview for Google. Certainly they've put a lot of thought into how to manage their hiring process and they still lean heavily on the whiteboard. I'm almost positive, however, that most companies are really nothing like Google and really should not be using the same interviewing techniques.
Smaller companies with more quotidian problems would be better served with a more conversational and less time-pressured interview style.
the next best approach --- a take home assignment --- is also good at this (or better, IMO, since you're free to implement that solution however you damn well please and it more closely translates into the work they'll actually be doing), but people complain that this is too invasive of their time and that they are doing free work. which is it?
I have to imagine that what most people want is to be asked about their experience, talk through it, and expect that to be enough. which is a very solid first pass filter. it's just not strong enough of a filter to extend an offer with confidence in many situations; it's too easy to bullshit and most of us engineers are damn expensive. (it's not just salary; we want dedicated offices and sit stand desks and super comfy chairs and free food and drinks and games and cadillac health insurance and 401ks and so on and so on)
i also never understood why whiteboarding was hard. if you know how to code and know your language passably well, then why can't you write code on something that isn't a computer?
I consider myself a competent programmer since I have developed decent work@work and my peers recognise it too. But how would any interviewer can know about this ? My resume is like any other resume in tech world. They cannot. So I've to be put through WBI. For many years I couldn't have known what was wrong with me. Every time I failed I felt miserable to be honest. Eventually, I have narrowed down the reason. It is not just at interview I fail to problem solve - even when I try doing at home with a white board and 'explanation' I fail. English is not my region language but one would not doubt on my conversation abilities as I manage it ok. It is that I simply fail to explain solving programming puzzles in english since while programming I 'do' think in my regional language and most of the time in 'silent' mode in my head. Now 'silent' mode is again dreaded in interviews - as interviewers they want to know 'how you approach solving a problem'.
I think a better solution to WBI would be to make interviewee do programming questions on a laptop (alone) where interview is held - say make it an hour worth of exercise. Have that program run through some time bound tests (like google code jam etc) and may be do a code walk through later with interviewer. This way interviewer is sure interviewee can code.
Also, one of things that will definitely help would be to have some sort of evidence that you can code (git, code competitions etc). This gives some sort of confidence to interviewer.
+1 though for "Senior Whiteboard Engineer"--that term is genius!