Yeah I don't think OP was defending Twitter I think it was just more of a story about how crappy the interview process is at most tech companies.
My experience with Twitter was really odd. I knew someone internal and had them directly submit my resume with a recommendation. The resume was tailored for exactly the position they wanted showing me besting all of their requirements. I got an email 4 months later telling me my resume was not good enough for even a phone screening. I've never had that happen before and it's still perplexing.
Good point; never really thought about that. I just found it odd because their email response actually gave me a reason stating my resume didn't have enough experience for the position when it far exceeded everything they wanted for the position.
>You are then ushered into a room with the programmer's worst nightmare - a blank whiteboard.
Am I the only one who enjoys programming interviews? Even if you screw it up, it's still fun to try your hand at whatever problem they give you. They also don't expect you to do it perfectly; you're allowed to have sub-superhuman performance.
I believe that was the point - are you expected to do perfectly or not. There are different interviewers - some even just want to show off, probably like the strong hashing one.
It's great if you can have a reasonable discussion that actually shows your knowledge.
I interviewed at Microsoft and it was about 10 hours of this stuff and I loved it. Had a great time, and I think I did pretty well. One guy wanted me to write a shuffling program. So I came up with one which I later discovered was Fisher-Yates and optimal. But he kept saying "how can it be faster" so I'm not sure what I was missing. Most of the other stuff was pretty fun and straightforward - pointer stuff, string copying, synchronization. It felt like if you had gone through Sedgewick's algorithm book you'd be set for the most part.
Unfortunately by 7pm or so I was sorta exhausted (didn't sleep the night before 'cause I was so excited). I met the hiring manager, and he asked something fun like make a ring buffer and also write a non-recursive inorder traversal function of a binary tree. I apparently had some off-by-one bug in the ring buffer code, and then I just blanked out on the traversal question. No hire :\.
(OTOH, that team wasn't sure if they were going to be using VB6, or C++, or maybe .NET, but probably not because they wanted to ship with Office and Office severely limits dependencies. So maybe it was for the better.)
I think the idea of solving a whiteboard problem can be fun, providing it's unique. Quite a few times while looking for an internship I was asked mostly standard problems out of "Cracking the Coding Interview" or similar. Under pressure, even problems I've practiced or just understood from a contextual standpoint can be tough. Personally I have issues with the usual situation of it - interviewer staring, which is intimidating, and/or not even trying to have a mutual conversation and bounce ideas around to ease some of the pressure.
Most posts complaining about interview process are being made by people who failed those interviews. There is a ton of reverse-survivorship bias here. It's certainly not the case that companies like GOOG/FB/MS are filled with all bozzos. If anything their interview process have been rather successful considering above average quality of talent at these companies and non-trivial products they work on.
I also sense lot of "entitlement" in OP's post and comments on this thread. It's like "oh I can't answer your interview questions but I'm so good that if I don't get the job, it would be only because your interview process sucks". Most company's HR would let you know what to expect at these interviews. If you are not comfortable with CS whiteboarding questions then you should just decline at that point. It's unprofessional to blame their process after you accepted to go through it, failed and then shit all over it because you didn't get the job.
This is not to say all interviewers are good and many could be downright assholes. But that feedback should be between you and their HR. It's just professional courtesy considering that these companies don't post on Internet how badly you performed on interviews compared to their other candidates to tell other side of your story.
Personally I like solving computational challenges whether I get job or not. As a programmers we are supposed to be loving these kind of CS puzzles. If nothing else, you walk out with few CS things you didn't knew before which would have taken same amount of time to learn anyway. I'm not saying interview questions shouldn't be job related but the fact is that many of these companies are doing LOTs of things and they need to hire more generally because they give you relatively more freedom to move around once you are in. So large companies have to keep things general at some level unlike startups with one project. In any case, if you complain about having to write code and design algorithms at developer interviews then you are probably applying for the wrong job.
That was a really entertaining read. I'm not involved in the interviewing process often, but I'd like to be better prepared for the future so that I can ask meaningful questions and not waste anyone's time. I certainly won't be asking anyone to invert a binary tree.
I'm only a rising junior in CS on my second internship and I completely agree. I did really well in my Data Structs & Algorithms class (A+...) and I just hate these types of problems. I bombed Google and Dropbox's interviews, and am ready to never go through another process like that again if I can.
I've been exceedingly lucky landing gigs at great companies that don't filter with these questions. I worked for LeadGenius (YC S11!) and the technical portion of the interview was super fun and effective; I was asked to write a useful piece of software in python which I later applied to school projects! No whiteboarding, no data structures. I'm currently interning at General Motors and their interview was similarly sans whiteboarding, and I'm surrounded by brilliant interns and coworkers. Hmm. At least for me and a few of my peers, being asked to solve the type of problems presented in the article raises some red flags about a company's culture.
It's not just CS, but probably more common with CS. There are two kinds of engineers.
1. Those that love love love solving Puzzles.
2. Those that question whether solving this or that puzzle is going to bring in any money for the company.
For a places like google or deranged YC startups, whether what you are assigned has any bearing on cash flow is something way above the typical engineers paygrade. There are a lot of companies where that isn't true.
BTW: Bit of advice for anyone going into a technical fields.
Learn to write, English. Learn it well. How to clearly explain idea's and requirements. How things work, how they are broken. Justify what you did, or cover your butt. Where you are in twenty years will depend mostly on this.
Learn to speak fluently in front of a group of people.
While I'm completely, 100%, against these kinds of questions, how else can a company that receives thousands of resumes a day filter out the good from bad?
How about sitting the candidate down in front of an actual computer, or ask the candidate to bring in their own laptop? Then if you want to see how they do, pick out some open source projects that need a feature or two added, or a bug fixed, and have the candidate walk through some solution? This way, you are getting to see what their style is, and the random open source projects gets a potential improvement. And you won't get accused of having the candidate programming for you without pay, because contributing to random open source projects is kind of like charity or community volunteer work.
Or, here's a fun project. Have the candidate write up (in the language of their choice, or preferably multiple languages) a program that generates solutions for the Cracker Barrel peg board game (or variations thereof).
I like this idea for small firms, but how does this scale to the level that the parent comment is asking about? If we want to interview a few hundred candidates this month, who finds the few hundred open-source project tasks and audits them for appropriateness?
And, for what it's worth, the peg game problem sounds to me like standard interview fare: an interesting puzzle that bears little resemblance to real-world software challenges. Why is it a better alternative?
So instead of giving them an algorithm problem that they might have seen in the past (or seen something similar) you give them a random open source project -- along with all the idiosycracies it certainly contains -- that they are 100% unfamiliar with and ask them within a few minutes to make sense of the entire codebase and produce a meaningful contribution? Sorry, but that sounds even more cruel and unusual than asking a very obscure algorithm question.
This might be viable if you do it remotely: let them take their time to understand the project, understand the task, find the relevant section of the project, find the relevant section of code, understand the change they need to make, set up the development environment, and make the change.
That's at minimum a day's work for most tasks, and sometimes two if setup or debugging turns out to be more difficult than expected. It definitely doesn't scale as an in-person interview, but by doing it asynchronously you lose a lot of the live communication and thought-process data that current interviews provide…
…then again, you could take this opportunity to measure their ability to give asynchronous status updates instead: can they clearly document their progress, the challenges they faced, and how they overcame them?
I personally wouldn't be willing to spend 8 or even 16 hours working on an interview question for the chance to receive an offer. I'd take the one hour whiteboard dance.
My three favorite evaluation processes so far (from the POV of the interviewee):
1. Quick phone screen with someone technically competent enough to call me on my BS if it were painfully obvious, followed by a "come in and let's get you working as a contractor".
I went through this scenario twice, and both times it worked out rather well. Of course, there was risk involved with both parties, but it seemed to work fine for both a 5-person startup and 100-employee agency.
2. Phone call to talk generalities and big picture, followed by a lunch meeting that doubled as a conversational technical interview, followed by a take-home exercise on a paid-for-time-if-no-offer basis.
I went through this scenario once, and liked it even better. The idea behind paying me if no offer follows was that I could be (and was) given a real problem the company needs solved (small, fairly standalone feature in my case) that I could work out for them for a reasonable fee. They'd get full rights to the work, I wouldn't feel like it's a waste of my time if nothing comes of it, and it would all be nil and void if I were offered a job in the end.
3. Quick 30-minute phone call with a few team members to get a sense for what the position, team, and I are all about, followed by a 5-hour marathon in-person interview, but broken up into more digestible chunks: 45-min chat with one of the leads, 45-min chat with someone more on par with the position, 45-min whiteboard problem, 45-min session of actually solving problems at a real computer (while being encouraged to do it the way I would, using Google, Stack Overflow, asking them things as I would ask coworkers, etc.), etc.
This last one was at a BigCo in the Valley and I felt was a pretty damn solid way to go about it. I actually basically bombed the whiteboard problem, but didn't feel like it was an unfair question to ask me, as the single interviewer present was more interested in my process than my reciting a memorized answer.
It's actually not that hard -- invite them to work with you on a small project. Pay them reasonably. See how they work in RL. Then decide, and -- by all means -- give them honest feedback even when not hiring.
I don't think that was the question, though. Small firms can afford to do this, but how does this scale to companies with who want to interview a few hundred candidates this month? Where do the hundreds of real-world projects come from?
We did that one time. On the one had it worked very well. The person wasn't a good fit on the team so we declined. On the other hand it was emotionally draining because the guy was a really nice guy. It's really, really hard to work with someone, get to know them a bit and then at the end say, "Sorry, you're just not for us". I think it was even harder for the candidate who didn't clue in that they weren't doing very well. They met the team, got to know people a bit, got to feel the atmosphere and started to imagine what it would be like to work with us.
In the end it was such an upsetting experience for everybody that we agreed never to do it again. I think there are probably ways to make it work, but you need to be really careful to maintain some distance... But then, will you get the result you are looking for if you do?
This sounds like a huge time-sink on both sides. The interviewee will need to clear up a lot of time to work on the small project (which is not always possible).
This does not scale - the number of people that can be interviewed per unit time will go down drastically.
This wastes interviewers time, they could be spending the time working on real work (unless the project is real work).
If the project is a real work for hiring organisation, it likely would reveal to outsider things the company probably wants to keep secret (infrastructure, technology, processes, etc).
In Germany this is called "Probearbeiten" and is fairly common for software or design jobs: when you have passed the (mostly theoretical) interview, you're invited to work a day or up to one week to evaluate your practical skills.
Generally you tend to be given a few tasks that are representative for the work you will be doing if you get hired. This is similar to spec work (e.g. the result will typically be discarded unless it is exceptionally good and solves real-world problems) but typically you will be reimbursed if you are not hired.
For a programming job you may be given a task that doesn't require intimate knowledge of any of the company's codebases but should give some insights into how you work, followed by a short review.
It's important to note that this doesn't scale well. It works best if the candidate works on site and can work alongside future team mates, which may impact the team's productivity for the duration. This approach works best for small to medium scale companies with a small pool of viable candidates. It's beneficial for the company to only send a candidate through this process if they're very likely to hire them.
I actually prefer this approach. By the time you're invited for "Probearbeiten" both sides are fairly confident you're going to be hired and it gives both sides a chance to determine whether it's a good fit.
It should also be noted that even in companies that don't do "Probearbeiten" there's a trial period ("Probezeit") after you're hired of up to six months where you're pretty much employed "at will" and can be fired on the spot.
To change the nodes to point the other way around. So if the left edge of N1 points to N2 and the right edge points to N3 you make it so that N1's left edge points to N3 and the right edge points to N2. It's a trivial problem of changing pointers around. It's like 4 or 5 lines of code if you write a recursive function.
def invert(node):
if node is None: return
invert(node.left)
invert(node.right)
node.left, node.right = node.right, node.left
Yep, pretty much. But, to be honest, if you hadn't described it I wouldn't have had a clue what was meant by inverting a tree. Zippering it or something? I wouldn't know.
You can ask the interviewer to clarify what the question is or what the result should look like. You should always ask for more details ask if you don't understand the question, and sometimes people ask in a vague way to see if the candidate will ask for clarification for better or for worse. It will not make you look stupid.
The simple action of saying "I don't know what you mean by X" or "I don't understand what X is" builds up your credibility, not reduce it. People who aren't afraid to ask questions demonstrate credibility because when they do say they know something you know they probably actually do.
I have learned so much from not being afraid to say "What is X" when someone starts talking to me about something and assume I know what it is already.
Nobody has ever laughed at me for this. Even if they did, it wouldn't bother me, because I'd rather I look silly and then learn what it is than to pretend I know and then actually end up not learning what it was someone was talking to me about.
Finally if they did think you should have known what the topic was you'd rather they know that about you while they're evaluating you not after you're hired and end up in over your head. This is a really unlikely scenario. These interviews are not so much about what you know, but how you solve problems.
I just interviewed at a company that asked me to invert a binary tree on a (virtual) whiteboard. In fact, they asked me twice! Once in the initial technical interview, again in the follow-up!
Haven't heard back yet but I'm really disinclined to keep talking to them.
Yeah I remember the Google interview a few years back. I did it mostly for fun, and because their recruiters kept contacting me.
So there we were, building a tree out of a forest of existing smaller trees or such. That was after they couldn't find my resume, and used a 5 year old one they found some place. They seemed annoyed and tired. By that time I felt nothing short of me proving P!=NP would have help changed their mood. I mean, they didn't even ask me what I knew, what I worked on lately. Heck, I could have been herding goats for the previous 5 years, but if I could have solved that tree problem, well, Googleplex here I come, I could have been herding goats there, maybe as part of some a green eco-initiative project...
I guess half way through I just kind of decided that Google must be a pretty bad place to work at and kind of gave up. I was tempted to ask them how often they had to build the tree out of subtrees at Google. Or in this case how many times did they really have to invert the binary tree, but I am nicer that than, maybe they really did have a pretty shitty day.
Ever since then I've declined to interview at Google. I get contacted every year like clockwork "Wanna try again?", "No thanks, good luck inverting binary trees though..."
I was told by someone many years ago to always take a copy of my resume with me to an interview. That advice has paid off a couple of times. These days I also keep an email draft ready to send.
If a company doesn't have their shit together enough to coordinate an interview schedule and get the right people into the right room at the right time with my resume in their hands, ideally in their hands ahead of time, I don't want to work there. It's a basic competence test.
One reason bringing your resume is a good idea is because scumbag recruiters sometimes modify your resume, inserting the right keywords to get you the interview in the first place.
Unfortunately and apparently some recruiters have interns to convert those PDF's into their shitty doc-template. It happened to me a few times. Every time I had my CV ready and this seemed to help. Knowing LaTeX reflects very positively on you in some circles.
Lol. I do bring my own resume with me at interviews, but I've never been in the need for that. If they don't have one, they usually have me that can narrate it. Otherwise they should have done their homework.
Most people don't have to luxury of going to interviews simply to test a companies competence. They're going because they really want/need that job, and want to make it as easy as possible for the company to hire them.
Bring it anyways. If they need it, you can still count it against them, while not hurting your own chances if the job ends up being worthwhile despite failing this test.
They don't start looking at your resume or asking what you have been up to until you pass the technical interviews. Which means that if you resume or experience isn't up to snuff, the interviews are a complete waste of time for you because of a decision that takes them five minutes to make.
That seems backwards to me. The resume is supposed to help determine who to spend time interviewing with. If you do good in the interviews, what is on the resume doesn't much matter except for standard background checks.
Oh God this brings back memories. I was actually offered a job at Google (by some miracle) but turned it down. The interviews (7 in total) were an awful experience.
I had to solve problems that I had not looked at since my masters and were mostly theoretical CS things which nobody outside of academia ever has to actually use.
I had a brief read through of their coding style a few days before so I solved it in a very Google way which is what most likely got me the job. Even though doing it their way was just awful (in terms of implementation).
There is no doubt that Google's search algorithms are the best but I can't help but think the rest of their services are only high performing because they can through their enormous data centre resources at the problem.
I also hated the attitude of the interviewers and engineers I met. They were so up their own asses. Their way was the best possible way and if you didn't agree well fuck you!
I have only turned down two jobs in my career that I 100% do not regret and Google is one of them.
I actually once had to build a tree from subtrees for work (I was modelling some kind of process as a tree, given pairs of data). It was good fun, and a break from the mundane CRUD-based work I usually do. I did it with a whiteboard initially, and ended up producing a fairly robust solution.
The difference was that I was able and to try various approaches out without pressure. I had access to my old algorithms textbook as a safety blanket. Some friends dropped by and we went out for supper, and I chewed over my ideas with them.
I also had two weeks to solve the problem, and even though it took one Friday night to solve the problem, being given 2 weeks to produce a working solution put me at ease.
Doing something like that in a high-stakes interview would be nearly impossible for me.
Yes, the thing I hated the most about my Google interview experience was that there was no discussion whatsoever of my previous projects. But there was the question "What would you like to do?" and I realized later that I should have hijacked that question to talk about my past projects I enjoyed the most.
This just comes off as a bunch of bragging with the issue in the title sitting in the periphery. Also, people would be well served to realize that companies use these tests as a proxy for intelligence and education. They very much try to avoid making it a trivia contest.
I enjoy the challenge of programming interviews. When I interviewed in Mountain View, I thought it was just super cool to have been invited. It was like the mother ship had summoned me home!
I was mega-underwhelmed when I botched the last session, though. And after doing so poorly, no one escorted me out or summed things up. Efficient, I suppose -- I'd met with the recruiter at the beginning of the day and there was nothing left to discuss. But there was no denouement. A big build up all day and all of a sudden -- nothing.
That last session I was asked to implement a particular matrix manipulation. IMO a bit more utility than tree inversion. And I tried to talk through my thought process but I'll admit it just wasn't getting there. After it was over, I went back to the hotel, wrote a test case and kept coding until I made it work. I'm a little ashamed to admit that it took 1.5-2hrs to get it to work -- so probably not just intimidation/stage fright. After writing the code, I don't think I understood the design well enough to explain it -- but I did do it on my own w/o any reference. I guess I wasn't surprised when I didn't get an offer.
So I've been contacted again as a part of this latest hiring campaign. I'll give it another try, I suppose. My ego's buoyed by being contacted again and doing well on phone screens, so I suppose I can take another bruising. :)
I don't mean to personalize this to you, but your story kinda illustrates one of the problems. There's so much ego and self-worth tied up in both sides of this style of "interview". Both sides have this overwhelming need to show how smart they are.
It's understandable in some ways. It's part of what attracts people to the field, showing off intelligence. I'm guilty of it myself.
But it makes the "interview" process have an adversarial nature most of the time. That's not helpful to the actual goal of hiring people who will make your team successful.
One thing that might work along these lines- Bring in a hard problem that the interviewer doesn't know how to solve and spend some time trying to solve it with the candidate. As a team, discussing different options and problems with those options. You know, like people do in actual work situations...
> One thing that might work along these lines- Bring in a hard problem that the interviewer doesn't know how to solve and spend some time trying to solve it with the candidate. As a team, discussing different options and problems with those options. You know, like people do in actual work situations...
^This guy gets it.
I've never understood why interviews are set up to put the interviewee on edge and make him or her feel out of place, while simultaneously putting the interviewer in a temporary position of ultimate power. Said position does one of two things based on their personality: Makes the interviewer feel superior and aggressive, or makes them uncomfortable and ready to get it over with. Neither situation is good for the people involved nor the process itself. You're basically testing an employee on non-work-related problems under artificial pressure, which is pointless.
And I say all of that as both an interviewee and interviewer in the past.
Candidates naturally feel uneasy when interviewing. In the end, one hour of chatter will either get you a job you kind of desire, or not.
I think it's our responsibility, as interviewers, to make people as comfortable as possible so that they can perform at their best. It's our responsibility to prepare for the interview, smile to the candidate, downplay any hiccup they may have.
I don't believe all the BS about "taking people out of their comfort zone": 98% of people will feel uneasy at an interview and they will be well out of their comfort zone already.
Nice attitude! The fact that your curiousity drove you to figure out the problem long after its solution had immediate utility...that's always a good feature for a programmer.
I once figured out the weakness in my response to an interview problem 3 years after the interview. It was something basic too. Made me feel like an idiot, yet relieved at the same time.
I also drew a complete blank at an interview in a coding problem... and then went home an implemented it (taking quite a bit more time than I'm comfortable admitting ;-) ).
Interviewing is hard (from both sides). Sometimes it just isn't your day. Sometimes it just isn't your job. I was quite depressed when I screwed up that interview because it was the first one I had done after taking 5 years away from programming professionally. I was definitely rusty and I was secretly worried that I wasn't going to recover.
But, in retrospect, I'm glad I didn't get that job. It's a bit like the situation of being asked to TDD building Pascal's Triangle. I am an avid TDDer and I would have a hard time doing justice to that problem. Of course it is easy to get started, but then at some point there is going to be a "and then there is magic" bit because you are trying to test a function that is outputting an infinite sequence. It could make a fascinating conversation, but I think it says alot more about the interviewer than the interviewee.
Unfortunately, just like there a people who look like they would be amazing an amazing hire, but turn out to be so-so (at best), employers are the same. The the 10x employer is probably more mythical than the 10x employee.
Unit test are not just for people who write bugs (parent comment), a good start would be to test for the 9th row the problem was asking for. Since you can't test all possible values a representative sample would be good enough.
Just to be pedantic: It's not TDD if he already wrote the functional code ;)
I think that's the point, though. If I were to build this TDD, I would probably make use of the fact that that the value at any point is the sum of the values above it, in the previous row.
I could write a failing test that PT(1) = [1]. I could make it go green easily, and then write a failing test that PT(2) = [1,2,1] and make that pass, using my algorithm. Depending on how much you did in the last step, there may be no more failing tests that you could write. I can imagine maybe one more failing test, but after that you are done.
That's what I meant by "and then there is magic". The "test driven" doesn't really help the "design" at all. Your first test is the trivial case and the second test is "did it work?" I could arguably skip the first test altogether, and whether or not I wrote the second test before I wrote the production code is completely irrelevant because the test doesn't help you write the code.
Now, you might want to "refactor" this to be:
PT(n) = (n.choose(i) for i in [0..n-1])
(This is just pseudo code for a list comprehension that returns an array of n choose i for all the values of i from 0 to n-1). I could then write the code for choose(), but I'm in the same boat -- there is no benefit to writing choose() TDD because your first test will be trivial and you r second test will require the full answer.
While my "refactor" could use the tests I wrote with my "TDD" of PT(), I haven't demonstrated the value of TDD at all because the "refactor" is not related to the tests. I could just have easily written that code first and then written a test to see if it was correct.
Finally, if a bug is introduced to the code, the tests may pick it up, but are almost certain not to shine light on what went wrong because the tests are not related to the design of the code.
This is an excellent example of code that I would not write TDD (and I write 99% of everything TDD). I might not even write it test first depending on my mood.
Trying to be slightly diplomatic, the point of my post was that in all likelihood, a person who gives you this programming problem to demonstrate your TDD abilities has no clue about TDD. The alternative explanation is that they intentionally picked a problem that doesn't work well for TDD and wanted the person to explain why this was a sucky problem.
If you weren't very familiar with TDD, I could imagine this tripping you up pretty badly. You'd think, "what the hell test am I going to write?" As you point out, it doesn't really matter -- you can write any test that will sort of test that your code can grossly come up with the right answer. After that there's not actually anything beneficial you can do.
Identical experience. They had me design and implement a practical thing. It was an interesting problem, not something I would normally be able to do in 30 minutes. They wanted it in 30 minutes. After about 20 I stopped coding on the whiteboard and started explaining where I was going with it. It was not complicated, just using a hash to store things, but the removal involved moving used blocks a free list of pre-used slots so the overhead of adding a new costly element was way cheaper. This would beat a binary tree for the scale of elements they where talking about but still easily kept in ram. I am sure they didn't have a clue what I was talking about. I think they wanted an RB tree based algorithm and a more primitive solution. Right after the interview I implemented it and sent it to them. I didn't even get a reply. About two months later I got a survey. This was after three interviews. The happy ending is I have used that bit of code several times since and it rocks. :)
To fill in one gap - our hiring process has an extra layer of insulation to better control individual biases. Your interviewers all submit feedback and an independent hiring committee makes the decision, on a different day, with all due deliberation.
The reason nobody has anything to say to you at the end of the day is because none of that has happened so nobody knows what to say at that point, and I don't think it's likely to make you feel better to have somebody give you a completely generic speech that has got nothing to do with you.
As for the more general thing that keeps popping up - do people seriously believe that all the thousands of Google engineers who do interviews haven't figured out that people writing code on a whiteboard under stress aren't the same thing as people sat at their desk grinding out code?
Yes, "thousands of Google engineers who do interviews haven't figured out that people writing code on a whiteboard under stress aren't the same thing as people sat at their desk grinding out code?"
I am pretty confident they don't know that. I know people way smarter than me who flunked the google interviews and, working in the same building, know even more that got in who are very good but not great. Also, they are ageist and "don't consider experience at all" (and that is from a direct quote from a person in an interview).
Of course they know that writing code on a whiteboard under stress isn't the same thing as sitting at a desk grinding out code. The problem is that it's pretty hard to sit down all the applicants for a position for a few weeks and work with them long enough to actually get a sense of how they work, and it's quite wasteful for the 95% who don't get the job.
Some companies actually do this, eg. Coinbase requires that you take a week off from work and work with them (I think unpaid, but perhaps you get contractor wages) on a project. They've been widely criticized as exploitative here, with many people saying "Why should anyone who has any choice at all agree to that?" They also open up a rats nest of legal & IP issues.
The best way to get a full-time position at Google - or most companies, really - is to get an intern or contractor position and then convert to FTE at the end. Hiring rates are way higher for successful interns (and the interview process is shorter), because they have lots of people inside with first-hand experience grinding out code with them that goes directly into Google's systems. But getting that internship or contract itself often requires an interview...
There's not so much a shortage of talented programmers as there is a shortage of people able to identify talented programmers.
If there's anything in need of serious disruption, it's the adversarial hiring process in technology. It's all mindless cargo-culting, and it doesn't actually work.
Data counter-point: When I interviewed at Google, I used "echo foo" in my Java code through two white-board coding interviews, and I still got the job.
Sure, there are some terrible interviewers in the Valley (and I've been interviewed by some) but there are also some terrible interviewees (candidates?) out there.
A lot of these posts don't actually address the problem of hiring and filtering people. If you think whiteboard tests or logic puzzles are not effective, why do you think this and what is a better alternative?
I personally think they are ok, but should more be a means to test if the candidate can reason intelligently and it shouldn't matter much if the end answer they get is correct or not.
I always try to find out if the candidate could - with enough information - solve the puzzle.
Can you, given time, figure out a good solution to the problem I am asking? How would you go about solving the problem? What is your methodology?
I agree with you. These sorts of problems are really to understand how someone reacts to a new idea that he or she does not understand. A lot of times people tell me that they don't know they answer to a problem, which is fine. Knowing that there are things that we don't know if a sign of maturity, and the curiosity to figure out how to solve a problem is what we should be looking for.
Personally I think there are two good ways, in combination, to determining if a person is a good hire.
1. Ask questions about past projects. Probing questions to make sure their resume sounds accurate. Get a sense for how they solved problems in the past and explain them.
2. Give them something to code. On a computer. Some small example. Even better if you can try a pair programming exercise to see how they code and how they can communicate with others on a team.
Asking academic questions has been entirely useless in my opinion. I've worked with multiple people who can pass all the academic questions about algorithms and data set and couldn't write an application to save their live...and the vice versa too. I'd rather see how they can code, now and how they can communicate about projects (active and in the past).
i saw the thing on twitter earlier. what i don't get is why not ask the question of what that means? its such a simple problem once you know... and irl you don't know everything, if your interviewer expects it then you reject them for being a bad place to work.
Fact is, it just isn't true. As a Googler, I am surrounded by enormously talented engineers, the likes of which I rarely encountered on the outside. They love what they do, and it shows.
I have no doubt that there are false negatives/positives, but I am convinced that Google is doing something right.
Have you encountered anyone outside?
Sure there are talented engineers there but so are engineers in startups and other companies. I'm afraid that's a very myopic statement.
Yeah, definitely. I worked for eight years in the Toronto startup scene. Several coworkers were unquestionably awesome, but they were few and far between. The bar here is much higher.
I think we do very well at hiring the sort of people who already work at Google. The false positive rate seems to be very low, and the people I work with mostly seem very happy (I know I certainly am).
The false negative rate worries me, though, and I keep wondering if there's a way I could structure my interviews (and feedback) differently to help change that problem.
You should worry about more than the false negative rate. You should worry about the reputation Google has developed that means a lot of those who have choices opt to not even consider interviewing at Google any more because it's not worth the aggravation.
I do worry about that as well, but that's a much more difficult problem as people's opinions can be damaged by so many things along the way. What works for some people[0] is a disaster for other people. The 'disaster' nature of it can come down to not liking to be on the phone, needing answers more quickly than Google can provide them[1], or just ending up with a bad interviewer on their loop[2].
Those are much bigger issues, and there are people here trying to tackle them! I feel my effort can be best dedicated to making sure I conduct the best interviews possible. I start from by assuming my goal is to get the candidate to demonstrate the competencies we're looking for in whatever way possible. Yes, there's usually some coding. If the coding turns out to be a bit rough but the candidate can do a fantastic job of walking me through a previous project, it's design, what went well, what they would change about it now, that speaks very well of them. I can't claim they demonstrated fantastic coding, but I can claim that I think we should hire them anyway and justify that recommendation.
It's hard. I would like our process to be lighter weight (or at least, better weighted to individual candidates), but I recognize the importance of maintaining a small false positive rate. I work with really rock solid engineers, and that's one of the best things about my job. I can trust everyone around me to at least make good decisions (even if they're sometimes wrong, or they're sometimes not the decision I would have made, I can usually understand how a smart capable engineer would have made it).
[0]: My process was roughly as follows: phone screen (all technical, coding in a Google doc), on site ~3 days later (five interviews coding on a Chromebook, lunch; the usual mix of interview questions you've come to expect), offer ~5 days after that, mutual acceptance after negotiation ~3 days after that. So we're talking about two weeks end-to-end. Also, I actually really enjoy interviewing. I come out of a day of solving interview questions feeling invigorated. Like I said, we do a really good job of hiring the sort of people who already work here.
[1]: Some of the things that I think are very good about Google's process (particularly in terms of providing fairness across candidates in a way that my previous employer did not) also cause delays. A lot of effort goes into considering all of the data that's available on a candidate. The amount of discussion that goes into every candidate, even after all of the feedback is in, is staggering.
[2]: Remember that interviewers are people. We've all been told that we're representing Google, but some people take that responsibility more seriously than others. Given that interviews are typically conducted 1:1, nobody knows what happened in the room except for the interviewer and the candidate. When candidates have bad experiences, they don't necessarily report it to their recruiter, so if it's a systematic problem with a particular interviewer, unless it shows up in the feedback they're submitting, it's very difficult to detect and correct. This is really unfortunate, but of course putting two people in the room makes some candidates nervous as they feel they're being doubly judged!
I faced similar experience but here in India , since good interview questions are rare , most of the times questions are repeated from forums such as Career cup , I was asked questions irrespective of my role . I do take interviews at my current organization and ask simple practical problems with very simple solutions but most of the times candidates think in terms of complex data structures instead of simply trying to solve a problem, they do premature optimization and what not .
I don't think inverting binary trees is the problem per se. I mean, algorithms are the bread and butter of programming. You should be able to implement simple ones like that even if you'd never actually do that in practice. The real problem in my opinion is the whiteboard and time pressure.
I gave this example on Reddit this morning, but I'll repeat it. If a programmer was given an array of elements [a1, a2, a3, a4, ..., an-1, an] and asked to produce the array [a2, a1, a4, a3, ..., an, an-1] and couldn't do it, he'd be the laughing stock of /r/programming for a couple of days and we'd hear about this problem for years to come, the same way we hear about FizzBuzz. Certainly a programmer should know how to traverse an array and manipulate its elements!
For the case of inverting a binary tree, we haven't gotten a clear answer to what it means, but the concensus seems to be to swap every left and right nodes in the tree. Well, that's exactly like the array problem above: traverse the structure and manipulate its elements! The traversal is not a simple loop, you use recursion (or if you're fancier, use an explicit stack to avoid causing an overflow of the call stack), but I'd argue that recursion is a fundamental notion for a programmer to know.
A commenter on Reddit mentioned that there's a difference between programmers and computer scientists and that the programmer is responsible for writing software and the computer scientists are responsible for coming up with algorithms and data structures. But then, what do the programmers use? If we cannot expect them to know about trees, what about linked lists? Resizable arrays? Hash tables? Graphs? These data structures are extremely important tools for creating solutions, shouldn't programmers be aware of them and how to use them?
1. I think phrasing matters a lot. Saying "invert a binary tree" creates an immediate suspicion in my mind that what I'm really being asked is not "do you know how to swap elements", but rather some kind of weeding-out trivia looking for a specific, probably named-after-a-person, algorithm they're expecting me to remember from a college course (tricky for me in particular, as I didn't do CS in college).
2. A whole lot depends on the context of the interview. While I know what a binary tree is and how it works, I've never -- in 11 years of programming professionally, and several more as a recreational/amateur before that -- actually needed to implement or even explicitly use one. There are a lot of things people will deride lack of knowledge of, things that people will consider to be hugely important CS fundamentals, that can literally just never come up in a lot of programmers' careers because they are not actually universally fundamental in all fields of programming.
So jumping into things that aren't relevant to the field you're interviewing for can achieve the opposite of what the interviewer hopes for: it's putting the candidate off-balance, but in the wrong way by making them wonder if they're interviewing for the wrong job or misunderstood what the job actually involved. Which in turn is going to affect their performance.
I'll be the contrarian and claim that inverting a binary tree is a perfectly reasonable interview question. (Although not anymore now that it's famous). It combines basic background knowledge (what is a binary tree and how is it structured?), ability to reason logically (binary tree algorithms are often recursive, "inverting" a tree with no children is a no-op, otherwise you want to swap the children, and the children's children, and...), and enough familiarity in one language to write simple constructs without an IDE or stackoverflow holding your hand. None of this requires you to be Jeff Dean.
I will agree that the guy who kicked all this off has a legitimate complaint if the recruiter failed to adequately explain the interview process, and I'm sure there are lots of bad interviewers that focus on meaningless details or obscure trivia. But I don't agree that the concept of having candidates write code is fundamentally awful.
I don't get the obsession with binary trees questions on interviews. I've had several in a couple different interviews. You know how many times I've coded a binary tree algorithmn, in 15 years as a professional programmer? Zeeeeero. They just aren't that common in actual code.
Why not ask questions about, say, working with a general tree data structure, like XML or an HTML DOM? That's something that comes up all the time in many fields. Why the obsession with low-level pointer juggling?
I find myself implementing trees fairly often (and manipulating them even more often), but since I'm not using a language that requires lots of pointer juggling and lifetime reasoning, I consider implementing them a pretty basic task.
Interesting. What type of software do you work on? I ask because I write code for robotic control loops and looking to explore any possible angle. Someone who writes trees often could help broaden my horizons.
Quite a few times. If you are doing things related to computer vision, machine learning, maps, language processing, compilers, speech etc - you are going to use all these CS stuff fairly regularly. You are even going to need stats, linear algebra and probabilities (I'd to literally do bunch of math quite a few times at work on whiteboard).
Most of the crowd here seems to be doing CRUD, JS, iOS stuff. You won't need much of CS/Algorithm skills there and I understand their statements like "I have never needed to touch binary trees in 15 years". Unfortunately they are trying to make case that this is same everywhere. Sure not all jobs at GOOG/FB/MS require strong CS skills but lot of projects at these companies do. It's not reasonable expectation to walk in to these companies who literally thrive just because of their algorithms and say I don't give a damn about CS but look at this package manager I built. In my opinion, people who can't work with something as simple as binary trees won't last a day in many of the projects at these companies. Learning frameworks and languages are easy, problem solving using CS primitives is hard earned skills.
Normal web dev job? Never. Having fun with genetic algorithms? More than once.
At my day job, I did have to write a CSV parser once. Once. Don't quite remember why using an existing solution wasn't an option. I think the input wasn't exactly proper CSV format, but was very similar.
I work a pretty regular dev job (database analytics/reporting tool essentially). I of course never have to _invert_ binary trees, that's a pretty artificial example, but I had to write a method for pretty-printing tree structures for debugging something just this week.
This is a good question.
I also think it's an extremely simple question, but only if you've ever taken a data structures class.
Turns out a lot of people in our industry don't! I have a Computer Engineering degree, and know this stuff mainly due to this being my hobby (and having tried out for the IOI in high school). But there are former classmates who are only vaguely aware of these data structures, because they never got any exposure to them despite writing real programs.
I don't think there's a super strong argument that most computer engineers need to know much of anything about base data structures. There are so many other concepts that are more important to writing and deploying useful software (especially in the enterprise world) that at no point involve writing complex data structures. Data structures (or rather, knowledge of the internals) are much more domain specific than some of us would like to admit, especially in the age of RoR and Unity. There are many more design patters more important to know about than being able to implement quicksort (it took 6 years for researchers to write the first bug-free implementation!)
Google does it because they can. They're probably filtering out a lot of people who treat code as prose rather than a technical manual though. Which is probably why Google libraries look the way they look.
The biggest surprise for me is how many game programmers don't know about this stuff. It seems like the sort of stuff you'd run into pretty quickly
I'm still not entirely sure what that means. Does it mean you recursively swap left and right branches, or does it mean you create a new data structure where the bottom elements become the top elements?
Apparently it means to swap the left and right subtrees. How's that "inversion"? I'd call it "flipping" or "mirroring" or something... If this is the case, then it's a trivial solution: invert(left_child); invert(right_child);
that doesn't mean the interview question was bad, presumably in the actual google interview they defined the problem. people are confused because he didn't define "invert" in the twitter post
The last interview I did was for a node.js position. I was told to bring my laptop. When I got there, the put me in an office and handed me a packet with instructions on checking out a nearly empty git repo. Then they said "Write an API that meets as many of the following requirements as possible and push the code. We'll be back in 3 hours."
After 3 hours, I was taken to a conference room where I demonstrated the API. I also talked about a bug I had that had hung me up, but I fixed it right there in the conference room.
Got the offer. It was a great experience. I enjoyed the challenge, but it felt useful and related to the work rather than just esoteric problem that would not likely ever be required by the job, and if it was I'd have time to research it and figure it out.
That actually sounds like a decent way to do things. The only thing I'd change is making it shorter and trying to incorporate some pair programming at least to some minor degree to see how well they communicate along with developing.
In a company that does pair programming, sure, do that, because those of us that hate pair programming would then instantly tell them we're not interested and leave rather than waste time on an interview somewhere we won't want to work.
But likewise, if you're not actually doing pair programming day to day, don't do it in an interview setting.
Erm, okay, I used the wrong terminology. I simply meant two people working on the same project. I didn't mean on the same workstation, that would be terrible.
This, especially if the conference room is filled with a reasonable representation of their programmers, rather than just one or two people, and they get to ask questions and we get to talk about the code.
Hearing someone talk about their own code and explain their approach and the up and down sides of it, is a lot more valuable than simply having the correct answer roll out.
The interview process took about two weeks and on the whole was pretty reasonable for both sides.
There was a short phone screen (~30 minutes)
Then I had two technical exercises to do. For each, I was given a reasonable time period (4 hours) that started when I visited a special link to the get the problem description, and then used whatever tools I wanted to get it done, then email in the completed exercise.
The two exercises were not completely trivial, but were relatively straightforward and actually involved skills relevant to the position and not arcane puzzles or comp-sci theory. (Basically, the first was to write a Postgres schema from scratch (total 5 or 6 tables and maybe 30 columns across all the tables). The second task was to write a CSV parser (including reasonable error checking/recovery) to load data into the schema from the first problem.
Finally, there was the main interview (conducted via Facetime, this is a remote position). That ran for close to 3 hours.
This worked well, since it gave me a feel for what the work would actually be like, and I was given, as the candidate, every oppurtunity to put my best foot forward - no whiteboard exercises, no surprises.
Someone has to write that CSV parser though, don't they?
And sometimes you actually need to write things like sort algorithms. I implemented an insertion sort from scratch recently, because it needed to be done in a particular way to take advantage of the capabilities of a specific JIT compiler.
People do actually do this stuff - it's not all academic.
Did you have to do this on a whiteboard or paper as part of the interview process for your job? The point: not being able to do this on a interview in completely contrived circumstances with all the anxiety that comes with close scrutiny of a person who had the power to determine your future career is not a good gauge of your ability to do so when matters, given all the resources we as developers generally afford ourselves.
My biggest problem is that the vast majority of these tests is they only test two skills, neither of which is all that critical to software development beyond a certain minimum threshold: recollection and pattern matching known solutions to familiar problems. In my current level of competency with a decade of tenure in software development is this: if you're asking me to solve problems which I can easily "solve" with a few minutes of searching the web, you probably don't want or need someone like me whose spent the majority of their career on big projects which require a multitude of disciplines from being creative to quantifying results to forethought of future use to systematically testing and releasing at a minimum of risk....
What I find ironic is that I've never been tested on some of the few rote tasks which I find most developers struggle with: committing/branching/merging/commenting code, producing post release documentation, developing robust API functions, etc etc
Scenario: The file is a 100GB CSV file. The machine is a VPS with 500MB RAM. The task: Determine whether the value in the first column of the first row, is ever repeated in the first column of any subsequent row.
An import solution is unlikely to work. Most import solutions would try to load the rows into memory, which doesn't work here. Maybe if the imported parser can be configured to run a user-supplied callback on individual rows and then discard them...
(P.S. solving the problem with awk still counts as writing a CSV parser --- in awk)
BufferedReader r = new BufferedReader(new File(filename))
String line = r.readLine();
if (line != null) {
String first = line.split(",")[0];
while ((line = r.readLine()) != null) {
if (first.equals(line.split(",")[0])) {
return true;
}
}
return false;
}
throw new RuntimeException();
would work just fine, and very quickly, too. This sort of question simply is not hard to solve; it's far more dependent on tiny implementation details. I very much doubt that any CSV parser would try to load the file all at once; it'd be too much of a performance hit.
In other words, you write your own CSV parser, in this case using Java.
Btw, CSV files can contain values with commas and even newlines inside of them. So if your point was that you don't have to write an _entire_ CSV parser, only a partial one, unfortunately that isn't true. https://en.wikipedia.org/wiki/Comma-separated_values#Example
>I very much doubt that any CSV parser would try to load the file all at once
I agree, but a naive imported solution would likely try to store all the rows at once, possibly in some sort of list or vector or array or whatever. This is what would cause the memory failure, not the file read itself.
Possibly, but unlikely (most csv libraries will be based around iterators). In any case, the problem is stupidly underspecified. For example, what if the 100GB CSV is 100 1GB rows? What if the input is UTF16, UTF32? How do you deal with the 10 Unicode line separators?
It just tests how well the interviewee knows CSV, which is an ill specified format anyway. It's a fake problem as no sane person would parse 100GB CSVs on 500MB VPS', and in real life, you'd just try the naive solution, see why it didn't work and iterate.
this is a valid record definition from a csv file:
"b CRLF
bb","ccc" CRLF
zzz,yyy,xxx
Your code will fail on this case. CSV parsing is not as simple as it sounds. Given that the format is also not well-defined, it's even worse. (is the first line a header or not for example)
It's not even a library thing. Reading a file in this way does reads in approximately optimal (for the FS) sized blocks. The CSV parser just rides on top of stdio.
>and then used whatever tools I wanted to get it done, then email in the completed exercise.
This is a much more realistic way of testing someone's skills. Asking someone to write code while they watch is not.
I am personally horrible at coding while someone is looking over my shoulder. I am just too preoccupied with their presence and the fact that they are watching. And, unless a company is still into pair programming (is anyone these days?), it's not a valid test.
Give me a real problem, reasonable time to solve it, and the tools I'd have in the real world. Then, I can show you what I can do in that same real world vs. how well I interview in some contrived format.
Funny, I thought this was going to be "inverting binary trees considered harmful" because the sensible way to do it is to change your traversal algorithm to be right-before-left, and leave the data alone. Silly me!
I love interviewing and being interviewed. Don't know why but I do. However, I hate these types of interviews. I mean, sure if I am a new grad or you suspect I might not know my shit, ask me some probing questions. However, almost nobody these days needs to invert a binary tree or compute the hash of a string by hand.
Here is my secret go to question when interviewing someone: "can you describe how an AJAX request works, from start to finish?". The answer involves knowing that AJAX works over HTTP, same as regular page requests, knowing what IP, TCP, and HTTP are, and how client and server interact using them. It is a question most competent people can answer, yet it has lots of room for depth of detail. By discussing technical subjects, challenges and solutions I think you get a much better idea of how good someone is vs some predefined set of puzzles.
224 comments
[ 44.1 ms ] story [ 307 ms ] threadSorry my op sounded like I was accusing the author of something but this whiteboard approach is what I did at Twitter.
My experience with Twitter was really odd. I knew someone internal and had them directly submit my resume with a recommendation. The resume was tailored for exactly the position they wanted showing me besting all of their requirements. I got an email 4 months later telling me my resume was not good enough for even a phone screening. I've never had that happen before and it's still perplexing.
Am I the only one who enjoys programming interviews? Even if you screw it up, it's still fun to try your hand at whatever problem they give you. They also don't expect you to do it perfectly; you're allowed to have sub-superhuman performance.
It's great if you can have a reasonable discussion that actually shows your knowledge.
Unfortunately by 7pm or so I was sorta exhausted (didn't sleep the night before 'cause I was so excited). I met the hiring manager, and he asked something fun like make a ring buffer and also write a non-recursive inorder traversal function of a binary tree. I apparently had some off-by-one bug in the ring buffer code, and then I just blanked out on the traversal question. No hire :\.
(OTOH, that team wasn't sure if they were going to be using VB6, or C++, or maybe .NET, but probably not because they wanted to ship with Office and Office severely limits dependencies. So maybe it was for the better.)
Part of the interview process is trying to find out what type of person you are dealing with.
I also sense lot of "entitlement" in OP's post and comments on this thread. It's like "oh I can't answer your interview questions but I'm so good that if I don't get the job, it would be only because your interview process sucks". Most company's HR would let you know what to expect at these interviews. If you are not comfortable with CS whiteboarding questions then you should just decline at that point. It's unprofessional to blame their process after you accepted to go through it, failed and then shit all over it because you didn't get the job.
This is not to say all interviewers are good and many could be downright assholes. But that feedback should be between you and their HR. It's just professional courtesy considering that these companies don't post on Internet how badly you performed on interviews compared to their other candidates to tell other side of your story.
Personally I like solving computational challenges whether I get job or not. As a programmers we are supposed to be loving these kind of CS puzzles. If nothing else, you walk out with few CS things you didn't knew before which would have taken same amount of time to learn anyway. I'm not saying interview questions shouldn't be job related but the fact is that many of these companies are doing LOTs of things and they need to hire more generally because they give you relatively more freedom to move around once you are in. So large companies have to keep things general at some level unlike startups with one project. In any case, if you complain about having to write code and design algorithms at developer interviews then you are probably applying for the wrong job.
I want in implementation of a high performance octree. Do it for me. Apparently you love doing this sort of thing for people who don't pay you.
It's a good advice to go out to interviews from time to time. Just to see what's out there.
I've been exceedingly lucky landing gigs at great companies that don't filter with these questions. I worked for LeadGenius (YC S11!) and the technical portion of the interview was super fun and effective; I was asked to write a useful piece of software in python which I later applied to school projects! No whiteboarding, no data structures. I'm currently interning at General Motors and their interview was similarly sans whiteboarding, and I'm surrounded by brilliant interns and coworkers. Hmm. At least for me and a few of my peers, being asked to solve the type of problems presented in the article raises some red flags about a company's culture.
1. Those that love love love solving Puzzles. 2. Those that question whether solving this or that puzzle is going to bring in any money for the company.
For a places like google or deranged YC startups, whether what you are assigned has any bearing on cash flow is something way above the typical engineers paygrade. There are a lot of companies where that isn't true.
Learn to write, English. Learn it well. How to clearly explain idea's and requirements. How things work, how they are broken. Justify what you did, or cover your butt. Where you are in twenty years will depend mostly on this.
Learn to speak fluently in front of a group of people.
Or whatever the locally appropriate language is.
By the way, we're hiring: Visual Interaction Designer, Senior Product Engineer (Front End), and Senior Product Engineer
https://leadgenius.com/careers
Hope you're liking it over at GM!
Or, here's a fun project. Have the candidate write up (in the language of their choice, or preferably multiple languages) a program that generates solutions for the Cracker Barrel peg board game (or variations thereof).
And, for what it's worth, the peg game problem sounds to me like standard interview fare: an interesting puzzle that bears little resemblance to real-world software challenges. Why is it a better alternative?
That's at minimum a day's work for most tasks, and sometimes two if setup or debugging turns out to be more difficult than expected. It definitely doesn't scale as an in-person interview, but by doing it asynchronously you lose a lot of the live communication and thought-process data that current interviews provide…
…then again, you could take this opportunity to measure their ability to give asynchronous status updates instead: can they clearly document their progress, the challenges they faced, and how they overcame them?
My three favorite evaluation processes so far (from the POV of the interviewee):
1. Quick phone screen with someone technically competent enough to call me on my BS if it were painfully obvious, followed by a "come in and let's get you working as a contractor".
I went through this scenario twice, and both times it worked out rather well. Of course, there was risk involved with both parties, but it seemed to work fine for both a 5-person startup and 100-employee agency.
2. Phone call to talk generalities and big picture, followed by a lunch meeting that doubled as a conversational technical interview, followed by a take-home exercise on a paid-for-time-if-no-offer basis.
I went through this scenario once, and liked it even better. The idea behind paying me if no offer follows was that I could be (and was) given a real problem the company needs solved (small, fairly standalone feature in my case) that I could work out for them for a reasonable fee. They'd get full rights to the work, I wouldn't feel like it's a waste of my time if nothing comes of it, and it would all be nil and void if I were offered a job in the end.
3. Quick 30-minute phone call with a few team members to get a sense for what the position, team, and I are all about, followed by a 5-hour marathon in-person interview, but broken up into more digestible chunks: 45-min chat with one of the leads, 45-min chat with someone more on par with the position, 45-min whiteboard problem, 45-min session of actually solving problems at a real computer (while being encouraged to do it the way I would, using Google, Stack Overflow, asking them things as I would ask coworkers, etc.), etc.
This last one was at a BigCo in the Valley and I felt was a pretty damn solid way to go about it. I actually basically bombed the whiteboard problem, but didn't feel like it was an unfair question to ask me, as the single interviewer present was more interested in my process than my reciting a memorized answer.
In the end it was such an upsetting experience for everybody that we agreed never to do it again. I think there are probably ways to make it work, but you need to be really careful to maintain some distance... But then, will you get the result you are looking for if you do?
This does not scale - the number of people that can be interviewed per unit time will go down drastically.
This wastes interviewers time, they could be spending the time working on real work (unless the project is real work).
If the project is a real work for hiring organisation, it likely would reveal to outsider things the company probably wants to keep secret (infrastructure, technology, processes, etc).
Generally you tend to be given a few tasks that are representative for the work you will be doing if you get hired. This is similar to spec work (e.g. the result will typically be discarded unless it is exceptionally good and solves real-world problems) but typically you will be reimbursed if you are not hired.
For a programming job you may be given a task that doesn't require intimate knowledge of any of the company's codebases but should give some insights into how you work, followed by a short review.
It's important to note that this doesn't scale well. It works best if the candidate works on site and can work alongside future team mates, which may impact the team's productivity for the duration. This approach works best for small to medium scale companies with a small pool of viable candidates. It's beneficial for the company to only send a candidate through this process if they're very likely to hire them.
I actually prefer this approach. By the time you're invited for "Probearbeiten" both sides are fairly confident you're going to be hired and it gives both sides a chance to determine whether it's a good fit.
It should also be noted that even in companies that don't do "Probearbeiten" there's a trial period ("Probezeit") after you're hired of up to six months where you're pretty much employed "at will" and can be fired on the spot.
The simple action of saying "I don't know what you mean by X" or "I don't understand what X is" builds up your credibility, not reduce it. People who aren't afraid to ask questions demonstrate credibility because when they do say they know something you know they probably actually do.
I have learned so much from not being afraid to say "What is X" when someone starts talking to me about something and assume I know what it is already.
Nobody has ever laughed at me for this. Even if they did, it wouldn't bother me, because I'd rather I look silly and then learn what it is than to pretend I know and then actually end up not learning what it was someone was talking to me about.
Finally if they did think you should have known what the topic was you'd rather they know that about you while they're evaluating you not after you're hired and end up in over your head. This is a really unlikely scenario. These interviews are not so much about what you know, but how you solve problems.
Haven't heard back yet but I'm really disinclined to keep talking to them.
So there we were, building a tree out of a forest of existing smaller trees or such. That was after they couldn't find my resume, and used a 5 year old one they found some place. They seemed annoyed and tired. By that time I felt nothing short of me proving P!=NP would have help changed their mood. I mean, they didn't even ask me what I knew, what I worked on lately. Heck, I could have been herding goats for the previous 5 years, but if I could have solved that tree problem, well, Googleplex here I come, I could have been herding goats there, maybe as part of some a green eco-initiative project...
I guess half way through I just kind of decided that Google must be a pretty bad place to work at and kind of gave up. I was tempted to ask them how often they had to build the tree out of subtrees at Google. Or in this case how many times did they really have to invert the binary tree, but I am nicer that than, maybe they really did have a pretty shitty day.
Ever since then I've declined to interview at Google. I get contacted every year like clockwork "Wanna try again?", "No thanks, good luck inverting binary trees though..."
Truth is stranger than fiction: http://googleblog.blogspot.sg/2009/05/mowing-with-goats.html
If a company doesn't have their shit together enough to coordinate an interview schedule and get the right people into the right room at the right time with my resume in their hands, ideally in their hands ahead of time, I don't want to work there. It's a basic competence test.
They feel the same way about you having your own resume in an interview.
Well, searching is not really Google's core competency.
I had to solve problems that I had not looked at since my masters and were mostly theoretical CS things which nobody outside of academia ever has to actually use.
I had a brief read through of their coding style a few days before so I solved it in a very Google way which is what most likely got me the job. Even though doing it their way was just awful (in terms of implementation).
There is no doubt that Google's search algorithms are the best but I can't help but think the rest of their services are only high performing because they can through their enormous data centre resources at the problem.
I also hated the attitude of the interviewers and engineers I met. They were so up their own asses. Their way was the best possible way and if you didn't agree well fuck you!
I have only turned down two jobs in my career that I 100% do not regret and Google is one of them.
The difference was that I was able and to try various approaches out without pressure. I had access to my old algorithms textbook as a safety blanket. Some friends dropped by and we went out for supper, and I chewed over my ideas with them.
I also had two weeks to solve the problem, and even though it took one Friday night to solve the problem, being given 2 weeks to produce a working solution put me at ease.
Doing something like that in a high-stakes interview would be nearly impossible for me.
I was mega-underwhelmed when I botched the last session, though. And after doing so poorly, no one escorted me out or summed things up. Efficient, I suppose -- I'd met with the recruiter at the beginning of the day and there was nothing left to discuss. But there was no denouement. A big build up all day and all of a sudden -- nothing.
That last session I was asked to implement a particular matrix manipulation. IMO a bit more utility than tree inversion. And I tried to talk through my thought process but I'll admit it just wasn't getting there. After it was over, I went back to the hotel, wrote a test case and kept coding until I made it work. I'm a little ashamed to admit that it took 1.5-2hrs to get it to work -- so probably not just intimidation/stage fright. After writing the code, I don't think I understood the design well enough to explain it -- but I did do it on my own w/o any reference. I guess I wasn't surprised when I didn't get an offer.
So I've been contacted again as a part of this latest hiring campaign. I'll give it another try, I suppose. My ego's buoyed by being contacted again and doing well on phone screens, so I suppose I can take another bruising. :)
It's understandable in some ways. It's part of what attracts people to the field, showing off intelligence. I'm guilty of it myself.
But it makes the "interview" process have an adversarial nature most of the time. That's not helpful to the actual goal of hiring people who will make your team successful.
One thing that might work along these lines- Bring in a hard problem that the interviewer doesn't know how to solve and spend some time trying to solve it with the candidate. As a team, discussing different options and problems with those options. You know, like people do in actual work situations...
^This guy gets it.
I've never understood why interviews are set up to put the interviewee on edge and make him or her feel out of place, while simultaneously putting the interviewer in a temporary position of ultimate power. Said position does one of two things based on their personality: Makes the interviewer feel superior and aggressive, or makes them uncomfortable and ready to get it over with. Neither situation is good for the people involved nor the process itself. You're basically testing an employee on non-work-related problems under artificial pressure, which is pointless.
And I say all of that as both an interviewee and interviewer in the past.
I think it's our responsibility, as interviewers, to make people as comfortable as possible so that they can perform at their best. It's our responsibility to prepare for the interview, smile to the candidate, downplay any hiccup they may have.
I don't believe all the BS about "taking people out of their comfort zone": 98% of people will feel uneasy at an interview and they will be well out of their comfort zone already.
Interviewing is hard (from both sides). Sometimes it just isn't your day. Sometimes it just isn't your job. I was quite depressed when I screwed up that interview because it was the first one I had done after taking 5 years away from programming professionally. I was definitely rusty and I was secretly worried that I wasn't going to recover.
But, in retrospect, I'm glad I didn't get that job. It's a bit like the situation of being asked to TDD building Pascal's Triangle. I am an avid TDDer and I would have a hard time doing justice to that problem. Of course it is easy to get started, but then at some point there is going to be a "and then there is magic" bit because you are trying to test a function that is outputting an infinite sequence. It could make a fascinating conversation, but I think it says alot more about the interviewer than the interviewee.
Unfortunately, just like there a people who look like they would be amazing an amazing hire, but turn out to be so-so (at best), employers are the same. The the 10x employer is probably more mythical than the 10x employee.
Just to be pedantic: It's not TDD if he already wrote the functional code ;)
I could write a failing test that PT(1) = [1]. I could make it go green easily, and then write a failing test that PT(2) = [1,2,1] and make that pass, using my algorithm. Depending on how much you did in the last step, there may be no more failing tests that you could write. I can imagine maybe one more failing test, but after that you are done.
That's what I meant by "and then there is magic". The "test driven" doesn't really help the "design" at all. Your first test is the trivial case and the second test is "did it work?" I could arguably skip the first test altogether, and whether or not I wrote the second test before I wrote the production code is completely irrelevant because the test doesn't help you write the code.
Now, you might want to "refactor" this to be:
PT(n) = (n.choose(i) for i in [0..n-1])
(This is just pseudo code for a list comprehension that returns an array of n choose i for all the values of i from 0 to n-1). I could then write the code for choose(), but I'm in the same boat -- there is no benefit to writing choose() TDD because your first test will be trivial and you r second test will require the full answer.
While my "refactor" could use the tests I wrote with my "TDD" of PT(), I haven't demonstrated the value of TDD at all because the "refactor" is not related to the tests. I could just have easily written that code first and then written a test to see if it was correct.
Finally, if a bug is introduced to the code, the tests may pick it up, but are almost certain not to shine light on what went wrong because the tests are not related to the design of the code.
This is an excellent example of code that I would not write TDD (and I write 99% of everything TDD). I might not even write it test first depending on my mood.
Trying to be slightly diplomatic, the point of my post was that in all likelihood, a person who gives you this programming problem to demonstrate your TDD abilities has no clue about TDD. The alternative explanation is that they intentionally picked a problem that doesn't work well for TDD and wanted the person to explain why this was a sucky problem.
If you weren't very familiar with TDD, I could imagine this tripping you up pretty badly. You'd think, "what the hell test am I going to write?" As you point out, it doesn't really matter -- you can write any test that will sort of test that your code can grossly come up with the right answer. After that there's not actually anything beneficial you can do.
The reason nobody has anything to say to you at the end of the day is because none of that has happened so nobody knows what to say at that point, and I don't think it's likely to make you feel better to have somebody give you a completely generic speech that has got nothing to do with you.
As for the more general thing that keeps popping up - do people seriously believe that all the thousands of Google engineers who do interviews haven't figured out that people writing code on a whiteboard under stress aren't the same thing as people sat at their desk grinding out code?
https://www.google.com/about/careers/lifeatgoogle/hiringproc... is a pretty good explanation.
Some companies actually do this, eg. Coinbase requires that you take a week off from work and work with them (I think unpaid, but perhaps you get contractor wages) on a project. They've been widely criticized as exploitative here, with many people saying "Why should anyone who has any choice at all agree to that?" They also open up a rats nest of legal & IP issues.
The best way to get a full-time position at Google - or most companies, really - is to get an intern or contractor position and then convert to FTE at the end. Hiring rates are way higher for successful interns (and the interview process is shorter), because they have lots of people inside with first-hand experience grinding out code with them that goes directly into Google's systems. But getting that internship or contract itself often requires an interview...
I just meant typical American (or global?) politeness stuff -- escort your guest to the door, thank them for coming, you'll hear from us soon, etc.
If there's anything in need of serious disruption, it's the adversarial hiring process in technology. It's all mindless cargo-culting, and it doesn't actually work.
amen.
Sure, there are some terrible interviewers in the Valley (and I've been interviewed by some) but there are also some terrible interviewees (candidates?) out there.
I personally think they are ok, but should more be a means to test if the candidate can reason intelligently and it shouldn't matter much if the end answer they get is correct or not.
Can you, given time, figure out a good solution to the problem I am asking? How would you go about solving the problem? What is your methodology?
I agree with you. These sorts of problems are really to understand how someone reacts to a new idea that he or she does not understand. A lot of times people tell me that they don't know they answer to a problem, which is fine. Knowing that there are things that we don't know if a sign of maturity, and the curiosity to figure out how to solve a problem is what we should be looking for.
1. Ask questions about past projects. Probing questions to make sure their resume sounds accurate. Get a sense for how they solved problems in the past and explain them.
2. Give them something to code. On a computer. Some small example. Even better if you can try a pair programming exercise to see how they code and how they can communicate with others on a team.
Asking academic questions has been entirely useless in my opinion. I've worked with multiple people who can pass all the academic questions about algorithms and data set and couldn't write an application to save their live...and the vice versa too. I'd rather see how they can code, now and how they can communicate about projects (active and in the past).
I have no doubt that there are false negatives/positives, but I am convinced that Google is doing something right.
The false negative rate worries me, though, and I keep wondering if there's a way I could structure my interviews (and feedback) differently to help change that problem.
Those are much bigger issues, and there are people here trying to tackle them! I feel my effort can be best dedicated to making sure I conduct the best interviews possible. I start from by assuming my goal is to get the candidate to demonstrate the competencies we're looking for in whatever way possible. Yes, there's usually some coding. If the coding turns out to be a bit rough but the candidate can do a fantastic job of walking me through a previous project, it's design, what went well, what they would change about it now, that speaks very well of them. I can't claim they demonstrated fantastic coding, but I can claim that I think we should hire them anyway and justify that recommendation.
It's hard. I would like our process to be lighter weight (or at least, better weighted to individual candidates), but I recognize the importance of maintaining a small false positive rate. I work with really rock solid engineers, and that's one of the best things about my job. I can trust everyone around me to at least make good decisions (even if they're sometimes wrong, or they're sometimes not the decision I would have made, I can usually understand how a smart capable engineer would have made it).
[0]: My process was roughly as follows: phone screen (all technical, coding in a Google doc), on site ~3 days later (five interviews coding on a Chromebook, lunch; the usual mix of interview questions you've come to expect), offer ~5 days after that, mutual acceptance after negotiation ~3 days after that. So we're talking about two weeks end-to-end. Also, I actually really enjoy interviewing. I come out of a day of solving interview questions feeling invigorated. Like I said, we do a really good job of hiring the sort of people who already work here.
[1]: Some of the things that I think are very good about Google's process (particularly in terms of providing fairness across candidates in a way that my previous employer did not) also cause delays. A lot of effort goes into considering all of the data that's available on a candidate. The amount of discussion that goes into every candidate, even after all of the feedback is in, is staggering.
[2]: Remember that interviewers are people. We've all been told that we're representing Google, but some people take that responsibility more seriously than others. Given that interviews are typically conducted 1:1, nobody knows what happened in the room except for the interviewer and the candidate. When candidates have bad experiences, they don't necessarily report it to their recruiter, so if it's a systematic problem with a particular interviewer, unless it shows up in the feedback they're submitting, it's very difficult to detect and correct. This is really unfortunate, but of course putting two people in the room makes some candidates nervous as they feel they're being doubly judged!
For the case of inverting a binary tree, we haven't gotten a clear answer to what it means, but the concensus seems to be to swap every left and right nodes in the tree. Well, that's exactly like the array problem above: traverse the structure and manipulate its elements! The traversal is not a simple loop, you use recursion (or if you're fancier, use an explicit stack to avoid causing an overflow of the call stack), but I'd argue that recursion is a fundamental notion for a programmer to know.
A commenter on Reddit mentioned that there's a difference between programmers and computer scientists and that the programmer is responsible for writing software and the computer scientists are responsible for coming up with algorithms and data structures. But then, what do the programmers use? If we cannot expect them to know about trees, what about linked lists? Resizable arrays? Hash tables? Graphs? These data structures are extremely important tools for creating solutions, shouldn't programmers be aware of them and how to use them?
1. I think phrasing matters a lot. Saying "invert a binary tree" creates an immediate suspicion in my mind that what I'm really being asked is not "do you know how to swap elements", but rather some kind of weeding-out trivia looking for a specific, probably named-after-a-person, algorithm they're expecting me to remember from a college course (tricky for me in particular, as I didn't do CS in college).
2. A whole lot depends on the context of the interview. While I know what a binary tree is and how it works, I've never -- in 11 years of programming professionally, and several more as a recreational/amateur before that -- actually needed to implement or even explicitly use one. There are a lot of things people will deride lack of knowledge of, things that people will consider to be hugely important CS fundamentals, that can literally just never come up in a lot of programmers' careers because they are not actually universally fundamental in all fields of programming.
So jumping into things that aren't relevant to the field you're interviewing for can achieve the opposite of what the interviewer hopes for: it's putting the candidate off-balance, but in the wrong way by making them wonder if they're interviewing for the wrong job or misunderstood what the job actually involved. Which in turn is going to affect their performance.
I will agree that the guy who kicked all this off has a legitimate complaint if the recruiter failed to adequately explain the interview process, and I'm sure there are lots of bad interviewers that focus on meaningless details or obscure trivia. But I don't agree that the concept of having candidates write code is fundamentally awful.
Why not ask questions about, say, working with a general tree data structure, like XML or an HTML DOM? That's something that comes up all the time in many fields. Why the obsession with low-level pointer juggling?
Most of the crowd here seems to be doing CRUD, JS, iOS stuff. You won't need much of CS/Algorithm skills there and I understand their statements like "I have never needed to touch binary trees in 15 years". Unfortunately they are trying to make case that this is same everywhere. Sure not all jobs at GOOG/FB/MS require strong CS skills but lot of projects at these companies do. It's not reasonable expectation to walk in to these companies who literally thrive just because of their algorithms and say I don't give a damn about CS but look at this package manager I built. In my opinion, people who can't work with something as simple as binary trees won't last a day in many of the projects at these companies. Learning frameworks and languages are easy, problem solving using CS primitives is hard earned skills.
At my day job, I did have to write a CSV parser once. Once. Don't quite remember why using an existing solution wasn't an option. I think the input wasn't exactly proper CSV format, but was very similar.
Turns out a lot of people in our industry don't! I have a Computer Engineering degree, and know this stuff mainly due to this being my hobby (and having tried out for the IOI in high school). But there are former classmates who are only vaguely aware of these data structures, because they never got any exposure to them despite writing real programs.
I don't think there's a super strong argument that most computer engineers need to know much of anything about base data structures. There are so many other concepts that are more important to writing and deploying useful software (especially in the enterprise world) that at no point involve writing complex data structures. Data structures (or rather, knowledge of the internals) are much more domain specific than some of us would like to admit, especially in the age of RoR and Unity. There are many more design patters more important to know about than being able to implement quicksort (it took 6 years for researchers to write the first bug-free implementation!)
Google does it because they can. They're probably filtering out a lot of people who treat code as prose rather than a technical manual though. Which is probably why Google libraries look the way they look.
The biggest surprise for me is how many game programmers don't know about this stuff. It seems like the sort of stuff you'd run into pretty quickly
The fact that so many people in the previous discussion could not even tell what that means is a very bad sign.
After 3 hours, I was taken to a conference room where I demonstrated the API. I also talked about a bug I had that had hung me up, but I fixed it right there in the conference room.
Got the offer. It was a great experience. I enjoyed the challenge, but it felt useful and related to the work rather than just esoteric problem that would not likely ever be required by the job, and if it was I'd have time to research it and figure it out.
In a company that does pair programming, sure, do that, because those of us that hate pair programming would then instantly tell them we're not interested and leave rather than waste time on an interview somewhere we won't want to work.
But likewise, if you're not actually doing pair programming day to day, don't do it in an interview setting.
Hearing someone talk about their own code and explain their approach and the up and down sides of it, is a lot more valuable than simply having the correct answer roll out.
The interview process took about two weeks and on the whole was pretty reasonable for both sides.
There was a short phone screen (~30 minutes)
Then I had two technical exercises to do. For each, I was given a reasonable time period (4 hours) that started when I visited a special link to the get the problem description, and then used whatever tools I wanted to get it done, then email in the completed exercise.
The two exercises were not completely trivial, but were relatively straightforward and actually involved skills relevant to the position and not arcane puzzles or comp-sci theory. (Basically, the first was to write a Postgres schema from scratch (total 5 or 6 tables and maybe 30 columns across all the tables). The second task was to write a CSV parser (including reasonable error checking/recovery) to load data into the schema from the first problem.
Finally, there was the main interview (conducted via Facetime, this is a remote position). That ran for close to 3 hours.
This worked well, since it gave me a feel for what the work would actually be like, and I was given, as the candidate, every oppurtunity to put my best foot forward - no whiteboard exercises, no surprises.
And sometimes you actually need to write things like sort algorithms. I implemented an insertion sort from scratch recently, because it needed to be done in a particular way to take advantage of the capabilities of a specific JIT compiler.
People do actually do this stuff - it's not all academic.
My biggest problem is that the vast majority of these tests is they only test two skills, neither of which is all that critical to software development beyond a certain minimum threshold: recollection and pattern matching known solutions to familiar problems. In my current level of competency with a decade of tenure in software development is this: if you're asking me to solve problems which I can easily "solve" with a few minutes of searching the web, you probably don't want or need someone like me whose spent the majority of their career on big projects which require a multitude of disciplines from being creative to quantifying results to forethought of future use to systematically testing and releasing at a minimum of risk....
What I find ironic is that I've never been tested on some of the few rote tasks which I find most developers struggle with: committing/branching/merging/commenting code, producing post release documentation, developing robust API functions, etc etc
An import solution is unlikely to work. Most import solutions would try to load the rows into memory, which doesn't work here. Maybe if the imported parser can be configured to run a user-supplied callback on individual rows and then discard them...
(P.S. solving the problem with awk still counts as writing a CSV parser --- in awk)
Btw, CSV files can contain values with commas and even newlines inside of them. So if your point was that you don't have to write an _entire_ CSV parser, only a partial one, unfortunately that isn't true. https://en.wikipedia.org/wiki/Comma-separated_values#Example
>I very much doubt that any CSV parser would try to load the file all at once
I agree, but a naive imported solution would likely try to store all the rows at once, possibly in some sort of list or vector or array or whatever. This is what would cause the memory failure, not the file read itself.
It just tests how well the interviewee knows CSV, which is an ill specified format anyway. It's a fake problem as no sane person would parse 100GB CSVs on 500MB VPS', and in real life, you'd just try the naive solution, see why it didn't work and iterate.
this is a valid record definition from a csv file: "b CRLF bb","ccc" CRLF zzz,yyy,xxx
Your code will fail on this case. CSV parsing is not as simple as it sounds. Given that the format is also not well-defined, it's even worse. (is the first line a header or not for example)
This is a much more realistic way of testing someone's skills. Asking someone to write code while they watch is not.
I am personally horrible at coding while someone is looking over my shoulder. I am just too preoccupied with their presence and the fact that they are watching. And, unless a company is still into pair programming (is anyone these days?), it's not a valid test.
Give me a real problem, reasonable time to solve it, and the tools I'd have in the real world. Then, I can show you what I can do in that same real world vs. how well I interview in some contrived format.
Here is my secret go to question when interviewing someone: "can you describe how an AJAX request works, from start to finish?". The answer involves knowing that AJAX works over HTTP, same as regular page requests, knowing what IP, TCP, and HTTP are, and how client and server interact using them. It is a question most competent people can answer, yet it has lots of room for depth of detail. By discussing technical subjects, challenges and solutions I think you get a much better idea of how good someone is vs some predefined set of puzzles.