I think that article was pre-Win7, so it isn't really relevant anymore, except maybe in a specifically Ruby niche (which is fine for 37signals).
I can't say for sure because the post doesn't have date (day of year but no year). I wouldn't argue smart-aleck technical shibboleths with someone who can't display a date unambiguously without overflow rollover bugs.
The best interview is a pair programming session around some 1-2 hour problem. It takes longer, and it may come later in the interview process, but it's a high bandwidth way of assessing someone.
If they don't have a strong open source background (not everyone does), earlier in the process it helps to ask a couple very simple programming tasks, such as writing a function to reverse a string in place or the infamous fizzbuzz. It will weed out people with great resumes but poor skill.
Maybe, but I've come to find that pair programming is a certifiable skill that unless you've been doing it for a while needs honing. Pairing with someone great might not seem great if they have no experience with pairing.
So once again, we're back to: there is no single best-way to fully assess a candidate.
Well there is the intense XP understanding of pair programming, and then there is just sitting next to someone while they code version of pair programming. I can't believe anyone hasn't done something in that spectrum.
Even for juniors and new grads, at school people code next to each other all the time.
Then that's not pair programming. Pairing is an act of give and take, Ying and Yang, Ping and Pong, and sometimes Tweedledee and Tweedledumb. Sitting next to someone looking over their shoulder solving a problem that you know the answer to is not pair programming. There is no act of mutual discovery at all.
Why limit interview questions to problems you already know the answer to? You can have them help you with some problem in your actual codebase you're already working on.
> You can have them help you with some problem in your actual codebase you're already working on.
This is a terrible idea. If they come up with a brilliant solution can you actually use this in production? Who owns the code they write during an interview?
I have worked with many, many developers. I just don't sit down next to them watching them program, or them watching me.
I have pair programmed (over 10 years ago) and I never saw any benefit from it. I work in an environment where every line of code checked into the repository is reviewed and documented in agonizing detail (welcome to the wonderful world of Medical Device software!) and I completely fail to see the benefit of pair programming.
I don't think I've ever worked in an environment where that's been something that happens all the time. Maybe occasionally being with someone while they try to sort out a bug.
But actually creating new code with someone sitting there - that's something that's not something I've seen often
I had an interview once that had me complete 3 small tasks on my own: a debugging task, a database task, and a small web site. All three had to be completed in under 2 hours. I wasn't given much direction, other than the specs of the 3 tasks.
While I think this was a moderate evaluation of my skills, it ended up being more a test of if I could complete the tasks on time, vs. the quality of my work: I ended up not making it to the next step of the interview because I was too concerned about quality code over actually completing the tasks (with mediocre code), and didn't complete the tasks, thinking that they would rather get a feel for my quality vs. if I can complete a task. I did it remotely, so they never really got a feel for my character or personality.
Had it been in a paired session, things likely would have gone differently. However, I did learn more about where their priorities were, and ultimately feel I probably would not have been a good fit.
Unit testing is another simple topic that provides insights into how a person thinks about code.
Exercise: give them a class with empty functions, and some unit tests. Ask them to write the class and add some more tests. Also, one of the tests you provide must be subtly buggy.
fwiw we make people who are interviewing do unit tests on our production code. We have enough old gnarly code that we can always find something that doesn't have coverage. It is a great way to see how they code, how they interact with unfamiliar code and how they get on socially with the person doing the interview.
Companies like Google hire hundreds (thousands?) of developers per year, hiring policies at 37Signals are bound to be different from those at Google.
If you have to weed out thousands of candidates, trying brainteasers to estimate their problem-solving capabilities doesn't seem to be a bad strategy. If a candidate is not familiar with a particular problem, his/her solution may be suboptimal, but the process of getting to that (suboptimal) solution may be a good indicator of the candidate's problem solving skills.
Obviously, syntax-checking a candidate's on-paper Javascript code isn't going to help anyone.
google does not use "brainteasers" and in fact has a specific policy against that kind of question. they ask substantive technical questions that often involving coding on the whiteboard.
If I were applying, I'd still consider this a "brainteaser." It serves the same purpose and is likely equally abstracted from the tasks of the real job.
This was a few years ago, but I was given a brain teaser/game where I had to find the differences between two sets of Google doodles. It was supposed to prove I had attention to detail.
I applied for a developer job at Google once and eventually had two phone interviews.
I was asked to come up with a sorting algorithm that would run on 4 computers with more data than could fit on any one computer. I was then asked to do a complexity analysis on my algorithm.
The only thing I think that really tested was how long it had been since I took an algorithms course in college.
I had a similar one. Actually interviewing for a management position with no coding involved. The worst part was that a lot of the questions were imprecise and/or I could tell he expected an answer to a different question that "normal people" would look up rather than remember by rote.
One question that annoyed me in particular was "what is stored in an inode?". First problem: Lots of file systems don't have inodes, and those that do store different information in them. Ok, explained that and asked for clarification. He wants the inodes in UFS. I told him I have no idea - I've never had any reason to look into the on-disk structure of UFS. Why would I? BSD is specifically not on my resume, though I've used it now and again. I offered up the description of ext2fs instead, since I happen to roughly remember that, and talked around differences with other filesystems. But he was clearly not happy.
This was one of a series of similar
The recruiter was annoyed to no end, and managed to get the interview result thrown out, but by then I'd gotten so annoyed about the whole process (which had dragged out for weeks) that I just wasn't interested any more.
Sorry, but that's a reasonable question for assessing a particular type of skill. That you seem to think they're looking for you to remember some particular parallel sorting algorithm from college tells me that this probably isn't your strongest skill. That doesn't mean you're a bad programmer.
It's been seven years since I've taken an algorithm course, but I have a generic (assuming nothing about the input data other than it doesn't fit in a single machine's main memory and is uniformly distributed) solution in my head as well as a complexity time for it.
That said, I work on distributed systems for a living and have actually had to do similar things (manipulate data sets larger than a single machine's main memory across a cluster of machines).
On the other hand, that's not a skill 37signals needs. It's insane to think that Google, 37signals, and a random startup could and should use the same interview process.
It's testing how well you can deal with problems that don't fit on a single node -- most problems at Google don't. And if you can't do complexity analysis, you can't scale to very large datasets -- which Google has lots of.
I used to work at Google. When I interviewed I was rusty on algorithms, so I studied up on them, even delaying my on-site interview to have more time to get ready. Sure enough they asked algorithms questions and I did better on them thanks to my studying. So I think it's also testing whether, given strong evidence on what would be helpful preparation, you choose to do that preparation. (Obligatory Paul Graham essay reference - "If college applicants realized how quick and impersonal most selection processes are, they'd make more effort to sell themselves" [0])
A problem with that is that they assume they're the most attractive option. They might be to some, but I think that pool of people is dwindling. It took five attempts from different Google recruiters before one managed to "sell" Google sufficiently for me to be willing to do a phone interview.
Then I was given interview questions of a type that nobody else would give for a position at the level I interviewed for, and I quickly lost interest again as to me the interview questions advertised a lack of understanding of the skills required for the position, or a fundamental disagreement about how technical management should be carried out.
And pleading from the recruiter was not sufficient to get me to do another interview (after she got the result of the initial interview thrown out because she agreed with my assessment). The last recruiter I dealt with was good, but the impression I got of the interview process put me off for a long time.
It took several months, and in that time they'd still not been able to find anyone else. I've never, ever spent that long on a recruitment process before, and the only reason I bothered was out of curiosity and because I wasn't really looking. If I had been looking, I'd have found and taken another job long before Google would've managed to get their act together.
I make an effort to sell myself for positions I badly want, but Google doesn't seem to be interested in selling itself in the recruitment process whereas most other companies I've interviewed for are, and they certainly isn't even top 10 of list of places I want to work.
But every time I've dealt with Google recruiters, I've had to waste time dragging information out of them that I'd normally expect the recruiter to be eager to present to me.
I'm not going to make an effort if they've not gotten me to really want the job first.
Recruiters I've spoken to all tells me the people they take out of Google are very often disillusioned and thought it would be very different to their actual experience, and that dislodging Google employees have gotten much easier as it's no longer so much the pinnacle of cool places to work in the eyes of a growing number of engineers, so I think that's going to become a bigger and bigger issue for them when hiring.
All fair points. I had one friend who interviewed with about 13 different people at Google before getting an offer, which was totally absurd, and a lot of their friends got a negative impression based on that (soon after they set a cap of about 8 total interviews per candidate). So I can imagine having their interview process do other bad things, like not being forthcoming with information or trying to sell the job.
I remember reading Heroku does similar. Using applicants in a project works well.
I have always tried to push my potential employers to use me on a small real life project. I know it's time consuming (and you end up working for em for free) but it works both ways. I get to know what kind of work environment they have and they get to know me in a real deal. I find it much more comfortable because the puzzles always seem to be a hit and go. The solution may click or it may not.
I know it's a start but if this catches in the industry I expect employers to even pay for your 30-40 hours interview.
I expect employers to even pay for your 30-40 hours interview.
It's great if the new employer pays you. But what about your current employer, the one at which you've just had to spend half your annual allotment of vacation days in order to audition for a new job that you might not get?
Auditioning is definitely the best way to, well, audition people. But it puts a constraint on your candidate pool, and depending on your company and your industry that constraint can be a problem. In 37signals' situation this doesn't matter - they're built entirely on open-source tools, they don't require relocation, they can hire freelancers from a pool of freelancers that they themselves created and fostered and routinely market to, they are so famous and impressive that even in a tight labor market people will line up to try and impress them, et cetera. But there are many companies that aren't 37signals, though the 37signals evangelism team is certainly trying to change that. ;)
> The only reliable gauge I’ve found for future programmer success is looking at real code they’ve written, talking through bigger picture issues, and, if all that is swell, trying them out for size.
I agree with the latter. This is how I used to hire programmers, one month on a freelance contract first with a single project.
The importance of the code itself is also overrated IMHO. It's just one tiny part of someone you add to your team, what about creativity, taking initiative, never giving up, being a team player, etc. Each of them are equally important to plain coding skills.
This seems like it would work in environments that aren't super competitive for top talent or if you are offering a dream job, but in the face of multiple offers this is going to be far less attractive to the candidate.
I've got a family. I need stability... and health insurance. Unless it's my only offer, a substantially better offer, or a dream job, I'm inclined to go elsewhere.
It's reliable. However, it's not competitive. You're assuming that I have nothing better to do with my time than to work for you on a freelance contract for a month, with no guarantee of a full time position, probably no benefits, etc.
If I'm job hunting, I've got other offers on the table. Ones that do guarantee a salary for more than a month. Ones that include benefits. Ones that expire long before your trial hire would be over.
If you try using this method, I don't know of a top notch programmer that you would be able to attract, simply because the hiring process is too painful.
I imagine David is savvy enough to present it as a mutually beneficial arrangement.
As a prospective employee wouldn't you like to kick the tires before making the leap into a new job that you might find yourself wanting out of six weeks in? There's so many things you just can't know about a company until you're on the inside.
Edit: Remember that 37s has a lot to offer: Remote work with great people, top-tier name recognition in the tech world, and a long history of building and releasing open source tools. That's a really compelling package compared to some of the bigger names in tech.
If I'm uncertain enough to want a trial period, I'd also want to keep my options open. If the position wasn't a good fit, then by the time I find out, I've been forced to turn down my alternatives.
No matter how you look at it, you're asking me to take on significantly more risk with this process.
For developers that are in demand, it's a tough sell.
I'm sure you would like to , but it's not always practical.
Even organizations like the military who have fairly rigorous selection procedures do it in a way that can be fitted into a long weekend.
Exactly , this isn't something that would work either if somebody already has a job unless it's something they can do part time (but even that depends allot on their work hours and other commitments).
You can't really expect someone to quit their existing job just for a chance to work with you.
Don't know about the rest of HN but I found it strangely reassuring that even someone as talented as DHH can end up in front of a white board during an interview and feel stupid. I've been there...
He's obviously very talented at design (not necessarily only UI design but general program structure and framework design). But if you needed someone to write packet routing algorithms or design a high-availability database engine from scratch, that may not be his cup of tea.
OTOH, I bet there are developers who love brain teasers and puzzles who may very well excel at that type of work but couldn't implement a huge software system with many moving parts without having it collapse under its weight.
"The only reliable gauge I’ve found for future programmer success is looking at real code they’ve written"
This a thousand times over. Go through their github (or whatever) profile, look at their commits, see how they write tests, etc. It doesn't take a huge amount of time, if you are any good at what you do, you'll quickly be able to filter out a failure (like, he doesn't have code that you can browse).
Not all, in fact many, programmers won't have a public github profile. If they do, great, but it can't really be a requirement of your hiring process, unless you want to limit yourself to candidates with public profiles.
Actually that wouldn't be the worst idea in the world.
Every time I read this it frightens me, I don't have an active github account, therefore I am a failure? I'm currently working a 16 month internship before graduating with a Computer Engineering degree. My code from this work will not be available anywhere of course.
Not through any fault of that though, I realize that having a github account with projects and such that you work on is a great help, and I accept that myself not having one yet is a great disadvantage. What do you recommend to get started out on github? Is helping out open source programs still viable? Or now it seems either writing self-run projects, or forking others projects and helping them with those. Is there any clear starting point for githubbing? (and how do you squeeze it in after a 10 hour work day!)
Just questions from someone worried about my empty github profile :)
Don't worry, the "github or git out" attitude is only common among people you wouldn't want to work with anyway. They're the ones who will change languages or frameworks according to what's fashionable, rat-hole the team on fun but irrelevant projects, and not talk to you unless you dress like a hobo and swear like a sailor. They're over-represented here on HN, but real engineers know that most good code isn't on github and never will be.
My experience with gung-ho githubbers is the same. It's great to try out new technologies, but gitguys take it to an extreme. They'll shoehorn any new technology into their stack without a second thought about product stability.
Gitguy: "I used EC2, Blazboo and Chingbang to create an HA job queue that will never fail! It uses counting Bloom filters and I wrote it in Brainfuck."
Me: "What do you use it for?"
Gitguy: "Sending password reset emails."
These Rube Goldberg wannabes are generally more trouble than they're worth. You end up with a system that's a technological pastiche. The drawback is apparent when you try to hire new teammates. It turns out you can't find someone who knows the 12 esoteric packages your business is running on.
I'm glad I'm not the only one who thinks this. I really like the energetic fluorescence of some open source communities, but the punning names, over-extended metaphors and religious devotion to novelty uber alles are really starting to take their toll.
The state of programming is so awful, that I'm literally just looking to know: do you write fat controller actions..or whatever equivalent exists for the specific technology you used.
That, to me, is 100x more important than "how many zeros are in 100 factorial"
Funny, I thought github was a code repository, not a personality type...
real engineers know that most good code isn't on github and never will be
Irrelevant. Fact is, having code out there makes it easier for prospective employers to get a feel for you, there's no way around that. No one said it had to be the best code or all the code.
Now, if you are working your ass off on proprietary code and write absolutely no code outside of your work, then you won't have it. That's understandable, but doesn't change the fact that it makes it harder for someone to figure out what you can do.
Thanks for the steawman. I was explicitly referring to a mentality, not to github itself. I use github. I don't share that mentality. It's not github's fault that some users treat them as an oracle, but it does happen. I have on multiple occasions heard people say that they won't hire people unless they can check out their work on github - github specifically, and no qualification or allowance for exceptions. Every time, the speaker was exactly the kind of hipster idiot that I just described. I suppose that could just be a coincidence, or it could be that people who follow one stupid trend often follow others as well. In any case, that one stupid trend is sufficisnt to make working with them unappealing, and since there are plenty of sensible employers out there it's unnecessary as well.
Just to let you know, I agree with you in substance. If it makes a difference to someone whether you have your code on github or sourceforge, then they're obviously pushing a dogmatic view. But like you say, that's not github's fault, so conflating them and a subset of their users in an argument doesn't seem productive.
Don't you write any code outside of work? Even if the main stuff I'm working on wasn't GPL'd, I'd still have my little side projects. They're nothing fancy, but putting them out in public doesn't hurt.
Exactly...the source code for your BYOB you used to learn python say? Or even a fork and pull request to a project to fix even a small bug or add a minor feature.
On the one hand I'd agree for students/juniors it isn't a deal breaker. But students/juniors that do have it are showing a great deal of enthusiasm and passion for programming.
I would recommend to start looking into contributing to open source projects. This is already big enough to notice and become even more mainstream in future. IMHO. But, of course, it is not absolute truth. If you want to land .NET job at some big consultancy firm - they probably will not care about open source work anyway.
Funny, every time I read "puzzle" or "brainteaser", I think, "I don't know why manhole covers are round," or "Who gives a shit that it took 7.612 seconds for my program to identify the 78,498 prime numbers less that 1,000,000." (Yes I know, get a life.)
But every time I read "FizzBuzz", I drop everything and go back to refactor my latest version, aspiring to get it down to one conditional.
I revisited my FizzBuzz just now and threw together this obscene Python one-liner.
(If I write this kind of code in production, I should be a guaranteed "no-hire"!)
from itertools import chain, combinations
fizzbuzz = lambda n,terms: ((lambda d: d.update((x%d,w) for d,w in sorted(reduce(lambda (d0,w0),(d1,w1):(d0*d1,w0+w1),sorted(x)) for x in chain(*(combinations(terms.iteritems(),s) for s in xrange(1,len(terms)+1))))) or d)({0:str(x)})[0] for x in xrange(1,n))
print ', '.join(fizzbuzz(100,{3:'fizz',5:'buzz'}))
Obviously i cannot show something i'm currently working on, and everytime I look back at some code I wrote six moths ago I find myself wondering "did I really write this piece of crap?!". And it's always been this way - at least since i've been six months into programming. I hope it is a good sign.
It's a great sign as long as code you wrote 6 months ago is better than code you wrote 12 months ago--it means you're getting better and learning.
I think as long as you're applying for jobs you feel you're appropriately skilled for, you shouldn't get too self-conscience about your code. Like most personal things, it probably looks worse to you than it does to someone else. The fact that you are self-conscience about it, likely means you have enough awareness to prevent you from becoming overconfident in your skills.
For what it's worth, I expect people to have that opinion about their (older) code samples. If you get the feeling that people are judging you by the poorer sections of your code sample, speak up! Start telling them what you think is wrong with it. They might not be worth working for if they don't listen to that.
Obviously i cannot show something i'm currently working on
Is this obvious? It's not like photographers don't have a portfolio of past project that they use to showcase their work. Unless you code under an NDA, of course.
I think the best interview I had was at Mochi Media. What each interviewer did was pick some programming headache that gave them trouble in the past couple weeks and present the problem to me to see how I would solve it. It was probably the hardest interview I ever had but I think it's great way to assess wether a candidate is able to solve problems they see.
Unfortunately a large number of us work for employers with strict IP agreements and we cannot just show a potential employer some code or work on a small project for them so that they may "try [us] out". Even working on open source projects can be tricky.
I helped start the practice at ITA Software (acquired by Google last year) of using puzzles as part of the hiring process. We ultimately even put our puzzles in the Boston subway as recruiting ads. My assessment, after more than a decade of doing it:
- Good puzzles are actually a talent attractor; many smart people found out about our company via our puzzles.
- Good puzzles are ones that scale well; i.e., where the basic problem is pretty easy and can be solved by a decent programmer in a few hours, but with harder variants that can take much longer.
- "Take-home" puzzles (as opposed to in-person whiteboard tests) weed people out who don't really like to program and/or can't finish things; this is a useful filter. If someone complains that "they shouldn't have to spend three hours writing code to get an interview," that itself is a pretty good counter-indicator. (Yes, we are all busy. But you're talking about starting a relationship with the company that may last 10+ years. You can do three hours of prep work for your interview. And you have to code a fun puzzle in the language of your choice, not some subroutine in a 30-year-old COBOL banking system.)
- It seems that in-person whiteboard or locked-in-a-room tests are pretty poor indicators of success. Many good programmers put in this situation significantly underperform their true abilities.
- It's not a great idea to evaluate someone purely on the basis of puzzle-solving ability.
- Many one-liner puzzles are bad indicators, because you either need to "know the trick" or have memorized the answer. Many, but not all, Microsoft and Google interview questions I hear about -- I've never interviewed at either place myself -- sound like they fall into this category. (E.g., from a recent article I read about Google: "You're reduced to the size of a nickel and thrown into a blender. How do you get out?" I don't care if you're clever enough to answer that; I just want you to able to write enormous amounts of high-quality code for me.)
Im my experience, the best way to find out if someone is a good programmer is to talk to him/her at length about something he/she has built. Can he/she talk in detail about how it worked? What challenges were involved? What tradeoffs were made? And, above all, was there passion behind the work and these choices?
A good interview for me was one where the candidate came in having written or contributed to a large system -- for fun or work -- and was excited to tell me about it. It didn't really matter whether it was relevant to the work we did (airfare search). Anecdotally, it seemed like people who really loved to code generally worked out pretty well. People with great-looking resumes that didn't love to code -- but maybe loved other things, like arguing over which language, operating system, architecture, or business strategy was better -- usually didn't produce much.
If the first thing you want to do when you wake up in the morning is code something, you're probably going to do well as a coder on a hard-core software team. Otherwise, not. Everything I personally did on programmer-hiring strategy was a proxy for figuring out whether someone was like that or not.
but i suspect that ita were exceptional. first, your puzzles were interesting - i imagine it's quite hard to think of interesting puzzles (how did you do it?). second, ita had a very good reputation so people were very motivated to work there. third, you may have been solving harder problems (and looking for more problem-oriented programmers) than many other companies.
i have done your puzzles for fun, but i would still be annoyed at having to do half-baked puzzles if i were suddenly out of a job and applying to a bunch of places.
I agree that the puzzles definitely have to be well-designed. There were a few of us -- myself included, for a while at least -- who created most of the puzzles.
We also never put a puzzle out to the world until a bunch of us had solved it. There was a long gestation period, and there's a "puzzles-discuss" mailing list for working out the kinks (and, in many cases, rejecting certain puzzles entirely). So, yes, you have to make a real commitment to do it well.
On the other hand, developing the puzzles was a lot easier than most of the coding on our real system, so it's all relative. It was actually really fun to do this stuff, and a nice change of pace. One of ITA's best developers (Justin Boyan) would take it upon himself to create the "perl postcard" solution for each problem. Another (Jim Rees) would always make an incredibly fast C++ version; it usually ended up being not only the fastest of ours, but faster than all the submitted solutions as well. I learned things just by reading his puzzle solutions. :)
I found the process rewarding beyond the hiring benefits, and I'm sure I'm not the only one. (Well, I hope so, at least.)
I'm jealous:) sounds like a lot of fun. When they came out, I used the ITA puzzles to shake the rust of my lisp skills and even considered sending some of the solutions in and applying, but that would have been a major career change.
In any case, kudos and thanks to you - those puzzles were a lot of fun, and I always wondered where they came from.
I saw those puzzle ads frequently in Kendall/MIT and Harvard (what, 6-7 years ago?). It was very clever recruiting on your and ITA's part, and a much better way of using brain teasers than as part of an in-person whiteboard session.
I don't think you give yourself enough credit in the success of the puzzles. They were really well-conceived problems - just difficult enough without requiring too much domain knowledge. Very 'catchy', if that makes sense.
Thanks. We were very aware of the marketing aspect of the puzzles, and designed them accordingly. The ones that made it into the T were ones that had visual appeal as well. It wasn't just coders, either; marketing and graphic design worked played a big role as well.
> I don't care if you're clever enough to answer that;
Bingo.
To me, this is the kicker statement for the industry. Solving fake problems like nickel-in-a-blender is for the clever. But I sure as hell don't want clever code solutions; I want rock-solid, reliable, fundamental solutions.
Anyone who feels these questions are an indicator of programmatic success is measuring the wrong things.
To be fair, the "nickel in a blender" type of problems have been actively banned from engineering interview at Google for many years. That doesn't mean that there aren't a few holdouts that still use them, but there is an active effort to eliminate them.
I sit on an engineering hiring committee and I can't really think of the last time I saw one of the ridiculous puzzle questions show up in an engineering interview. The WSJ article was pretty ridiculous.
Wasn't implying these to be solely a Google issue, but rather something that a material number of firms in the tech industry like to follow in interviewing.
Sorry, but I'm not familiar with the WSJ article you've referenced. Link?
I got asked 3 puzzles for my current job. It was quite annoying. One, I knew off the cuff. The other 2 were bad questions. One was caught in the "if you don't make these assumptions your wrong" trap. For the other, the "correct" answer was actually wrong. It really pissed me off.
Maybe I'm weird, but I find puzzles to be a turnoff. I love writing code, and I enjoy working through a math book.
But compared to coding and math, puzzles feel sterile. I don't get to build anything, and I rarely get the kind of insights I would get from figuring out a proof. (Raymond Smullyan's puzzles are a marvelous exception to this.) Even cryptanalysis is more fun, because there's a real opponent.
I know lots of amazing programmers who love puzzles, and that's cool. But if you only hired puzzle lovers, you'd miss out on a lot of good people.
You're not weird. Puzzles can definitely be fun, but, in my experience, it doesn't mean you're a lousy programmer if you can't solve one/some.
Everyone likes to use Google as the example of the elite programming company. I've read the books about them, used the products, followed their progress over the years. So, sure, a part of their big success is hiring the 'smartest' college grads.
However, I've also worked with Ph.D.'s and even interviewed them for positions lower than ones I've held (I have not yet finished college - some day...). And I can tell you that book smarts != execution smarts.
I feel the same way about puzzles. I have never used puzzles or asked 'small' one off programming questions. Instead I've always been interested more in grep'ing the concepts and if the personality is the proper fit. In my mind, anyone can open up a book and read about specifics about O(log n). But how does a person persist at solving a problem and how do they (can they?) change direction if their solution sucks?
I feel like I value creativity over anything and they can creatively solve a puzzle, cool. But if they can creatively solve a task on the system I'm building and are a great fit to the team, then better.
Same here. My impression is that the puzzle lovers turn out to be the algorithm geeks. You need a bunch of those, but I can't see how a room for of complexity analysis experts necessarily creates well-structured software.
I'm an API design geek. I've never seen a cool puzzle about API design, or something remotely related.
i'm pretty sure one of quora's questions is/was designing an api for something in python (probably time, given how bad the current api is).
also, replying more to the other post above - you can get insights from puzzles. a lot depends on the puzzle (which is why ita is getting praise here). for example, the ita puzzle i remember taught me a lot about handling data in large numbers of dimensions (in particular, it made me think about the "curse of dimensionality" which was something i was aware of before, but hadn't considered in detail; it also led me to understanding some tools for dealing with it - locality sensitive hashing being the most interesting at the time).
Most puzzles are not very interesting. :) But some are surprisingly rich, and those are the good ones to put in front of candidates in my opinion. E.g., I worked on one puzzle before I left ITA that involved laying out train track pieces to create a complete circuit within a rectangular region. (I don't believe this one has ever been used, but I could be wrong about that.) It was pretty interesting to solve this heuristically, but it turned out to involve something much deeper: the theory of so-called "lattice animals" -- an active area of research I knew nothing about.
Lattice animals include polyominoes, which are probably familiar to most HN readers, but subsume other graph-related objects as well. I didn't know much about these, and ended up reading a bunch of cool papers on them as part of working on the puzzle. So at least for me, good puzzles have catalyzed some learning.
I don't think that's too weird. I am never very interested in solving puzzles. Myst was a major turnoff, for instance. The Project Euler problems are usually "meh".
My mathematical preference is continuous math; my preference for using mathematics is to apply them into real problems.
I'm not saying that all puzzles are pointless, but I think most interview puzzles are kind of an empty exercise that only a certain type of person likes, and that only people who know the trick will get.
Are you responsible for the puzzle involving the historical sites search with a google instant style interface? This was a long time ago and it's no longer in the archives.
I worked on that one for a looong time, and what bugged me was the screenshot of the expected interface had a result returned in like 0.05675ms or something, and I couldn't figure out how to get it that fast. It took be forever to convince myself that it was just a mock and probably should have read 0.05675s (seconds instead of milliseconds).
I didn't work on that one, no, but I wouldn't assume the time was fake either. :)
We often had internal solutions (and externally-submitted solutions as well) that were orders of magnitude faster than the average. There was a very wide range of execution times. E.g., we had solutions to the Queens & Knights puzzle (in the archive) that ran in tens of milliseconds, if I recall correctly. Typical solutions we received took orders of magnitude longer.
Usually when one solution was a million times faster than another, it was because the fast solution exploited a clever algorithm or data representation -- e.g., representing an 8x8 chess board mask as a single 64-bit register rather than as an array of bytes.
Ah thanks! However, I'm worried I may take another crack at it now!
I may have missed that the time was the time taken for the search algorithm to find results, I think I was under the impression that should be the time for the request/response to happen. Hmm.
I interviewed at ITA in 2007, and ITA did the puzzle process better than all but a few other companies I've interviewed at, for the reasons you mention. But don't discount the in-person whiteboard puzzle. I was told I had the second highest score on the puzzle solution I sent in, but I flopped the whiteboard puzzle part of the interview, and ended up not being hired. I think this was a good thing - I didn't have the algorithm skills and knowledge of the other people in the QPX group (QPX is ITA's main business, their search engine - they actually recommended in advance that I interview for their reservation system group based on my resume, but I decided to go for the more brainy one). If I didn't have the whiteboard interview, I would have gotten in and probably regretted it later (much later, I realized that I would have regretted working at ITA anyway, or anywhere in Boston, but that's totally unrelated).
During the same job search, I also noticed Facebook had puzzles. I solved one and got invited to interview but never went. I wouldn't have applied otherwise. I guess this is another lesson - if you put public puzzles out there, expect people to send solutions without necessarily being excited to work at your company.
3 hours of prep work X 5 applications (1 application a day for a week) = 15 hours of unpaid work per week, or 60 hours of unpaid work per month. Given that a typical job search lasts multiple months, this is not an insignificant workload to saddle potential employees with. However, if your company is special in some way and people feel it's worth the rigamarole to apply there, then the market will bear it.
Funnily enough, the only time I've been asked a genuine logic puzzle in an interview was at ITA (don't remember the details, but it was something about ants walking on a 1d line and bouncing off each other). It even had some kind of a "either you see it or you don't" aspect as well.
But the programming puzzles on the website were just awesome, even though I just solved them because they were fun, not for applying for a job. I've been surprised that nobody else has done the same thing, but your explanation of how much work went into the puzzles kind of explains that :-)
I once was asked the SAME puzzle at two interviews (for different companies) two weeks apart.
I'd never seen it before the first interview. In that first interview, I broke it down quickly into its component parts, and figured out where the trick had to be -- and from there quickly solved it on the spot.
In the second interview I spent a couple minutes trying to remember the solution from the first interview, and completely flubbed it. It wasn't the first part of that second interview that I flubbed, either, including arguing with one of the interviewers and basically telling him he was wrong.
Oddly enough, I got the job with the second company and worked there for almost six years. For the record, I love doing those kinds of puzzles. But I do video game development, so maybe puzzles are a bit more relevant to my industry (in that 80%+ of what we're doing is figuring out how to solve puzzles in the form of how to do something with less CPU or why a particular video card is behaving in a certain way).
"You're reduced to the size of a nickel and thrown into a blender. How do you get out?"
If I were asked such a question in an interview, my response would be: "Dunno. Let's talk about some real-world problems you're facing right now instead."
I dislike trivial questions, especially during an interview. I'm a professional, not a circus animal.
It seems you like you still hire based on puzzle as a first filter (based on on your careers page). Just wondering, what is the ratio of people who apply with only a resume vs those who solve the puzzle (I understand you probably can't reveal). And, more importantly, are those on the puzzles side show to be more successful on the interview?
But the puzzle process, as originally conceived, (in 2000!) was not necessarily supposed to be a hard filter. As an applicant, you had several ways to stand out, and doing a great job on a puzzle was one. Having someone who already worked for us say you were awesome was another. Being an open source superhero was another. Etc.
In actual implementation, though, it probably was/is used as a hard filter.
In actual implementation, though, it probably was/is used as a hard filter.
It always becomes a hard filter, there is a lot of ego in development and test are an area where ego's shine through. It's my major beef with quiz exercises in interviews, they always become the hardest filter, no matter how much we try to contain them.
Just to clarify, ITA's puzzles weren't all of mathematical / algorithmic tricks variety.
I did one of them just for fun 3-4 years ago:
"implement a scalable multi-user chat server without using a framework like Netty/ACE/EventMachine". Unfortunately, at the time I couldn't move to Boston, so I never submitted my solution.
Likewise there were puzzles that would appeal to web developers (e.g., build a typeahead web applications), machine learning/AI folks/etc...
For a good programmer, a 3 hour take home test is like a $300+ application fee. Even if the puzzles are reasonable, filters work both ways. For a recent graduate or a mediocre programmer, it's probably worth the time for a shot at the job, but talented people will have plenty of options at good companies that don't make them jump through extra hoops. Perhaps it's fine if you pay a lot higher than average or offer significant perks, but I could see a strategy like this unwittingly driving away the top tier of candidates if you aren't careful.
It isn't like that because paying money to someone in exchange for merely being evaluated is an easy way to fall for a scam: the evaluator has a significant incentive to offer this deal to people who will fail.
It is much more like having to travel 90 minutes each way to get to where they want you to interview. That could be very offputting if you do enough interviews that three hours each adds up to a big number, but at least you don't need to worry about them being tempted to rip you off.
That is a reasonable theory, and I was always personally very concerned about that, but in practice it didn't seem to be the case. It seemed that the opposite occurred: people who didn't even want jobs did the puzzles just for fun, and asked us if they got the right answer.
Of course, we couldn't easily do A/B experiments, so we'll never really know how the true attract:alienate ratio.
Trying to look at it objectively, I would probably be somewhat put off by being required to do a puzzle to "prove myself", but would then find the specific puzzles offered to be pretty interesting, and worth playing around with. Perhaps it's not tantamount to a $300 tax on ones life it is seemed fun and valuable in its own right?
I have less disdain for pre-interview development filters and have seen some companies do it pretty well. I think your assumption about people who would find it fun could hold some merit. But I do believe the parent poster is right, those really good developers are rarely going to bother. They are usually to busy in the first place. They may see your company and think oh cool I like these guys and then hit the pre-interview filter and decide it to much investment for just a chance and that's the core of it, on both sides it's a numbers game, we can say well they did not really want it or we are only getting the ones that really want it, but you just may be getting the ones that are not working the odds out in their head. Or you could be getting the ones that are desperate and will invest the time for a chance. There are a lot of assumptions in these type of tests.
I think in the end, most people just want to have their abilities evaluated fairly. And even if I'm not a total puzzle freak, I do love to code, and given time I can figure most of these puzzles out. So if I were looking for a job at your (former) company, this would not constitute a barrier for me. On the other hand, if I were verbally asked a question and told to work it out on a whiteboard, this would not be an accurate indicator of my abilities or my potential. Add to that the fact that my handwriting is poor, and I would be nervous, and I'm quite sure a verbal description of pascal's triangle would almost certainly elude me, whereas if I were shown a document with the written output, I think I could come up with a naive solution in 5-10 mins. I'm good at sitting down by myself with a written problem and working it out. As a result, I've always done well on standardized tests. In fact, I suspect this is the reason for some of the indignation at the whiteboard testing. People who have always gotten good grades, coded well, done well on standardized tests are suddenly told that they're not smart enough. I can understand how it would rankle.
I don't think I had a satisfying solution to any of the puzzles in 3 hours. In the most extreme case, I saw some people spend a couple of weeks solving the two-time pad problem. Even after knowing how to cancel out the pad.
Ah, fair enough, there must have been some simpler ones there as well. It just was not the kind of thing I would have hacked on for fun :-)
I was thinking more of all the cool search problems like Strawberry Fields, Traversing the T or Palindromic Pangram, where coming up with some solution is easy, a naive exhaustive search for the optimal solution was not feasible, and where verifying the validity of any heuristics / pruning was not necessarily trivial.
Like jsnell, I spent weeks’ worth of nights beating my head against the problem I selected, and the solution I eventually came up with was not good enough to earn me an interview.
I gather that someone with a stronger theoretical CS background than myself would have spent an hour looking at the puzzle, said “Ah, this can be modelled as a graph and solved by applying the Flibblewhiz Heuristic!”, spent another hour coding up a first draft of the solution, and a third hour optimizing and cleaning up the code.
My ego has survived the blow of learning that I am not awesome enough to work for ITA, but I wish I had figured that out sooner.
Personally, I failed the ITA puzzle. (Either I failed it or they looked at my resume post-fact and decided I wasn't smart or qualified enough... probably a combination).
So, here is my opinion as a non-halfway-decent programmer.
I thought puzzle was fairly fun, but I like programming for two reasons:
* Building useful knowledge. Learning a new language, library, or algorithm I want to play with, programming is the natural way to learn.
* Building cool stuff that people like. Having an awesome product when I'm done with it gives the feeling that I've done a good job and produced something of value.
Programming puzzles don't really do either of these things for me. My motivation for spending three hours doing a puzzle is that I would like to talk to a person. I am basically assured that the puzzle will be used as a filter for the company to avoid talking to people. It will run against a test suite and if it fails it will not be looked at by anybody. This seems very impersonal. Is this the experience I should expect from the rest of the company if I'm hired?
I've got a Git-hub account with some of my code in it. I've spent time on it and enjoyed writing all of that code. I published it to the internet so I'm (at very least) not ashamed of it. Why can't you look at that to determine if I can program? Why do I have to write code specifically for your particular company?
This isn't an attack on ITA, it seems to work for them. But they are a unique case of a company where it seemed worth it for me to spend the time to do the puzzle. Any other company I would have just skipped it unless I genuinely had nothing better or more interesting to do (of which I almost always do).
I must personally say thank you for starting those puzzles. Over the years I have implemented many of them as fun weekend projects, but one puzzle in particular stuck with me: Sling Blade Runner. On and off the last ~four years I have worked on the sling blade runner puzzle creating dozens of different approaches for the problem. From genetic algorithms, visualizers, various traveling salesmen "solutions", stuffing the graph in the cpu cache and letting it run for a month to document how far the dumb solution wouldn't get, custom algorithms doing wacky things such as spherical gravity water flow simulations and many more I have pushed that data set through a lot of things. When I read books and papers and constantly working on the SBR problem in the back of my head. Generic graph theory can be a boring topic, but for some reason with this problem and data set to play with it has been something that has stuck with me. The generic problem of solving the longest path in a directed circular graph is NP so program solutions are all about find something good enough given limited time or exploiting issues with the data set to actually solve it. There is a lot of payoff in the industry for good enough solutions to this problem (which I am not actually involved in, I hack on webkit during the day). It is the puzzle the keeps on giving and as I encounter new techniques on the job or in reading I often end up trying later in the evening to see how well/fast they apply to this problem. I moved to Boston with an existing job (same one I am at now) so I never really got to go through the whole interview process at ITA, but always thought it would be fun to help design those puzzles. So again thanks for the puzzles, especially Sling Blade Runner!
>If someone complains that "they shouldn't have to spend three hours writing code to get an interview," that itself is a pretty good counter-indicator. (Yes, we are all busy.
When I applied at ITA they didn't even acknowledge the program I had written to go along with the application. Not even a quick 'it ran' or 'had the right output but we hired somebody else'.
I'm glad I didn't get the job because not having enough time to evaluate or even run a programming test you gave is also a pretty good counter-indicator of a good company to work for. And when I wrote back to express this the guy sheepishly admitted they had so many exceptional applicants so they just went with an Ivy-league graduate. Wow.
If you are going to require a programming test, be prepared to evaluate it.
If you are going to require a programming test, be prepared to evaluate it.
I agree. It is possible the reviewer didn't think your code would run, couldn't get it to compile, etc. But, nevertheless, you should have received a fuller response than that. If I were still COO at ITA, I would ask someone to look into your particular case to understand what happened.
I saw outright that I'm not an algo guy, you already have those types of people -- namely the guy interviewing me. But they dont know how to scale systems or know overall product. If they want to look at code even I call bad, I can point them at my github account :)
I love the honesty and refreshing look your post delivers. It hadn't occurred to me until I read it that maybe there is a programmer personality type that actually likes being on display, coding on a whiteboard capable of doing recursive algorithms in 2 minutes flat. Thanks.
In my experience, I haven't found this to be true. The algo-guys I've met are also critical thinkers when it comes to designing scalable and reliable systems.
Whenever I see a firm use brainteasers during the interview process, I just wonder why they don't simply recruit the best Sudoku players. That seems about as valid a way to judge software development skills.
I made a post on HN a couple of weeks ago basically saying the same thing. It can be rather frustrating to be asked questions with seemingly no relevance to the actual position.
I understand what they're going for though - the companies that do it correctly are looking at the candidates thought process through a challenging situation.
While I think it's important to figure this out, I believe it's more important to know how well the candidate can do the actual job, that of sitting down and writing quality software. Brainteasers and writing modified sorting algorithms won't get this (unless the company is in the sort algorithm business I suppose) but having the candidate write code which could actually be used in the job would.
I would hope "you just broke the user experience for 50 million people, there are 30 internal teams screaming your name" is higher pressure and stakes than an onsite interview loop. You can make or break your career in that situation too.
I had to deal with a crisis this week that boiled down to a poorly optimized SQL query or two. Of course, the manner in which they failed meant our DB started disk swapping and hurting our availability.
Contract to hire is something we do but it is rare. An initial try out period doesn't do us much good since the learning curve is long.
I don't trust take home assignments unless their solutions can't be googled. And it's a lot of work to create and evaluate problems of that nature. You want them just long enough, not a huge time sink for you or the candidate. You want them to demand a range of solutions so there are many ways to fail and succeed.
I agree we should not be doing riddles/puzzles. For me the most straightforward way to gauge a candidate's ability in an hour is to ask them to program on the spot and see how they act. Yes, it's a game and the candidate can learn to beat the system and oversell themselves. But the vast majority of people I don't want to hire do not go to these lengths.
I suppose the type of development 37 signals will do is more concerned with product design and less to do with fast algorithms etc.
I haven't really used their products but in terms of technical sophistication they probably aren't much advanced beyond CRUD type web apps (I may be wrong here).
If you wanted to hire a games engine developer or someone to design algorithms for google then brainteaser problems might be more relevant.
I don't enjoy brainteasers. If an interview question begins, "Four people and a goat need to cross a river...", I cringe.
But when I'm interviewing, I do ask people for a pointer to their open source projects. And if they don't have open source projects, I give them a workstation and ask them to write code. I've seen too many self-proclaimed programmers who claim, "I spent the last 2 years writing Python," who can't sum a list of 10 numbers without using Google. (Hiring can break my heart.)
One way or another, you've got to see code. Real code is best, but it's better to ask people about toy data structures than it is to hire 3 team members who can't write FizzBuzz.
> who can't sum a list of 10 numbers without using Google.
I did a hiring round once for my old team when I worked at Yahoo years ago, where out of 10 or so CV's the recruiters passed me, about 5 people who were given a short list of screening questions by e-mail for them to answer at their own leisure produced answers that were obviously plagiarized from various online sources.
I got suspicious when one guy presented me with two pages of nicely formatted answer complete with section headings that didn't make any sense to a question that should have required a one line answer. He'd copied and pasted the whole thing from an on-line copy of an Oracle manual, so I started doing searches of random sentences from their answers.
Another one managed to cut and paste an answer that was flat out wrong from a forum post answering pretty much the same question, and hadn't bother to read further in the comment thread, where several people corrected the answer he'd used...
It's when they can't even answer simple stuff WITH using Google it gets really depressing, and it happened regularly enough that using screening questions by e-mail was a real time saver in weeding out undesirables, despite the ease with which people could "cheat".
On the other hand, I wouldn't mind coding a program that solved problems of the river-crossing kind, expressing it as a search through a state space with constraints.
Absolutely. At my last C++ job, I did a lot of interviews and a lot of people claimed to know the STL. I left the low-level compiler questions to my colleagues for the most part, but I would ask a few key questions.
My favourite STL-related question to ask is simple and can be done on a whiteboard in minutes: if you didn't have std::set<class T>, but you had everything else in the STL, how might you implement a set<class T> class?
There's an ideal answer[1], and then there's people who start thinking about low-level implementations. The ones who I don't want are the ones who don't have any answer at all. I'll continue to be interested in anyone who starts talking about the way to solve the problem. What matters to me is not any code that you write (although that's interesting) but your thought process. After the candidate has worked through this, if they haven't gotten the ideal answer I always give it to them: this isn't some national secret, and seeing them process the ideal answer and realize that yes, it really would work is a pleasure.
Sadly, there are cases where it doesn't work. I had one candidate who did very well on the phone screen, but didn't know the STL. I looked up what he claimed to know (Qt) and reframed the question in terms of Qt's classes—and he still couldn't come up with any answer whatsoever.
-a
[1] The ideal answer is precisely such because you're writing much less code and using stuff that people smarter than you have written. It's to use a std::map<class T, bool>.
I've never met a programmer who can't sum 10 numbers without using Google. I have however met plenty who go blank when asked a question they weren't prepared for in front of an intimidating person in a small quiet room during an interview.
Not everyone has performance anxiety, but I've met plenty of programmers who do. Unless you spend the first 15 minutes of your interview calming them down and warming them up with simple technical questions that put them in the mode of programming, you might as well be asking them to sing the star spangled banner naked in front of an audience.
I was rejected in a recent interview (2 months later I'm waiting to 'hear back' means rejection) where I wasn't competent at writing algorithms to do various tasks such as implement an insertion sort based on this image: http://en.wikipedia.org/wiki/File:Insertion_sort_animation.g...
My confusion was the job was looking for a generalist who had experience with lots of different technologies. Specifically, working with node.js and socket.io polling methods, API design, working with some python framework and backbone.js.
I'm honestly not sure what the lesson there was. Am I a bad developer? Do I navigate interviews poorly? Does my breath stink?
I wouldn't have been able to tell you what an insertion sort is, by name. But by looking at that image, I can describe it pretty well:
Have an outer loop that counts from the first position in the list to the last position (call it n...). At each step in the loop, make sure that the first n items are sorted. This probably implies an inner loop that finds where, within the first n-1 items to insert the nth item. And given that with only one item, it's already sorted, we can probably start with n on the second item.
I wouldn't take it personally. People get hired for all kinds of reasons, and not getting a specific position just means that they didn't think you were the best candidate for that position. It certainly doesn't mean you are a bad developer or anything is wrong with you. It could be they were looking for someone more experienced, less experienced, different experience, salary expectations, the job description they gave didn't accurately describe the position they are hiring for, etc...
If an employer says they will get back to you and 2 weeks go by, it's worth contacting them and asking if they have made a decision yet. If they say yes and you don't have any clue why they wouldn't have hired you, I don't think it hurts to ask them why not. You might get a BS answer or you might get some good insight.
Just keep improving your skills, and keep looking.
To DHH and team: thank you for using reasonableness and sanity in your hiring practices. Wish this was more the norm, but indeed it seems to be the exception.
Trying to evaluate programmatic skills is surely a challenge, but I've found a method that seems to work very well -- simple walkthroughs. I'll do at least two walkthroughs of code, asking a candidate to explain to me what's happening using two different artifacts:
* Some code they've written that solves some problem
* Some code we've written that solved some problem
This has proven to be quite effective in rooting out those who look great on paper and talk a great game, but fall apart when put into action. On the flip side, I've yet to find someone who could explain their code in sufficient detail that then proved incapable once in the job.
So people who handle these stressful situations well and solve these problems under pressure only represent one type of developer - which Google has in spades - the kind who does very well in a structured environment, got great grades in their undergrad program, loves to worry over interesting and difficult problems, etc.
The kind of people who don't do well in these situations (like myself) are often valuable in other ways. Some people cave under pressure, and even more people have very low self-esteems when it comes to programming and intelligence in general. Yet these same people can be highly valuable and productive given enough time to make mistakes and solve problems in their own way. These types of people may be better at seeing the bigger picture and are less likely to over-engineer a solution or fall into the not-invented-here trap.
I think it's interesting that Google finds a lot of bright engineers using these high-pressure hiring tactics, but that they often turn to wholesale talent acquisitions when it comes to building products like Google Plus. My guess is the vast majority of the talent Google has gotten through acquisitions would have failed miserably going through Google's traditional hiring process.
I think that in most cases Google will interview the employees of the company to be acquired. Either in advance to decide whether the company is worth buying at all, or after the fact to decide who can keep their jobs. The bar might be set slightly differently in these cases than for normal hiring, but certainly not so different that the vast majority of the acquihires would have been "miserable failures" otherwise.
(Also, I've never heard of G+ being in any way a product of a talent acquisition.)
My guess is the vast majority of the talent Google has gotten through acquisitions would have failed miserably going through Google's traditional hiring process.
That's a very good point that I've not seen made so well before.
This is probably a lie. The Google interview is not particularly hard. It might be hard if you absolutely refuse to ever let your eyes look at anything that could possibly be considered an "algorithms book", but otherwise... it's stuff that you will know if you care about computer programming at all. The questions are about applying big concepts, not implementation details of particular algorithms. (I couldn't write A* from memory, but I do know what an "admissible heuristic" is. And that's good enough to be hired by Google.)
When I give a live programming problem, I cut interviewees a lot of slack for not doing well under pressure, as long as they seem suitably embarrassed about it. There's a big difference to me between "I know this step has got to be really simple, but I'm having a brain cramp", and "This step is really hard, no wonder I can't get it" (when it isn't).
I'm curious why my comment has been voted down. Was "suitably embarrassed" an offensive way to phrase it, or do people disagree with the distinction I made in the last sentence? I think it's pretty valuable to have a sense, even if you can't solve something, of whether the solution is likely to be simple or not.
I think the problem people have with this approach is that they don't want to be defined by their performance on one very specific problem. Most people, myself included, want to be seen as the sum of many accomplishments, not just one superficial one in an interview.
This topic as a whole seems to be very contentious, and stuff is getting downvoted seemingly randomly. Even comments that are interesting and that nobody could surely be objecting to.
I would argue that they don't and I believe that it is due to their hiring process. Now I do believe that they have very talented individuals but there organization reflects their interviewing bias as such so does their product portfolio. There is a reason that they have to acquire new products for their portfolio and that reason is that they are not recruiting the kind of people that generate new products. As such their hiring process should be suspect.
Indeed. Hiring programmers is about hiring people who bring business value to your organization, not about bringing in people who can solve Pascal's triangle faster than the other 90% of candidates. The entire first thread on this page is a shining example of what is broken with Google's hiring process.
37signals hires for business value because, as a small company, it has to. Google is a huge company and doesn't have the same incentives. Google's size can blind it to the fact that its hiring practices optimize for mathematical knowledge and intuition over business value. And, as a business, Google is wrong and 37signals is right.
> My guess is the vast majority of the talent Google has gotten through acquisitions would have failed miserably going through Google's traditional hiring process.
I don't believe that's the case. What I've seen with talent acquisitions (at Google and elsewhere) is that interviewing the team is actually prerequisite to making the acquisition.
Of course, the interview doesn't have to be as rigorous: during the due-diligence process you can actually view the codebase and see the individuals' contributions.
What I've also seen, specifically in the case of Google, is that individuals from acquisitions would be asked to stay a year, at the end of which they'd need to interview for other positions at Google: if they couldn't pass the interview, they'd be let go with a very generous severance package.
My own experience is that if you're really interested in getting into companies with recruitment processes like this, you might want to be uninterested when talking to the recruiter, talk down your level of interest, question whether the company has anything worthwhile to offer you, be open about your reservations about their recruitment process, and detail your issues with any interviews you do with them.
Generally whenever I've done this to recruiters their interest level goes up and they tend to be far more willing to listen to your concerns about the interview and go to bat for you. And recruiters tend to have a lot more ability than you might think in getting parts of the technical interviews largely ignored by hiring managers - including at Google.
Incidentally my impression from having talked to several Google recruiters is that a decent number of them are incredibly frustrated with the way the process work, and will work hard to get you past this hurdle if you want them to and provide them with enough ammunition (e.g. explain why the question was inappropriate for the position).
We can all agree that there are different kinds of programming. There are some really hard problems out there that need to be solved. There's also the kind of programming that's a matter of getting users to interact with data. When it comes to web application development (my background), I think puzzles aren't the right way to approach the interview. To be honest, it shouldn't be the job of the company to find out what the interviewee does and doesn't know. It's the duty of the applicant to control the interview, come in with their laptop blazing, and start walking through building a web application from scratch or to show off something they are building, talk about the decisions that were made through the process and why.
445 comments
[ 3.1 ms ] story [ 429 ms ] threadhttp://david.heinemeierhansson.com/arc/000433.html
I can't say for sure because the post doesn't have date (day of year but no year). I wouldn't argue smart-aleck technical shibboleths with someone who can't display a date unambiguously without overflow rollover bugs.
http://www.codinghorror.com/blog/2008/02/douchebaggery.html
If they don't have a strong open source background (not everyone does), earlier in the process it helps to ask a couple very simple programming tasks, such as writing a function to reverse a string in place or the infamous fizzbuzz. It will weed out people with great resumes but poor skill.
So once again, we're back to: there is no single best-way to fully assess a candidate.
Even for juniors and new grads, at school people code next to each other all the time.
This is a terrible idea. If they come up with a brilliant solution can you actually use this in production? Who owns the code they write during an interview?
The last time I did anything like that was over 10 years ago. How would that skew your hiring process?
I have pair programmed (over 10 years ago) and I never saw any benefit from it. I work in an environment where every line of code checked into the repository is reviewed and documented in agonizing detail (welcome to the wonderful world of Medical Device software!) and I completely fail to see the benefit of pair programming.
But actually creating new code with someone sitting there - that's something that's not something I've seen often
I had an interview once that had me complete 3 small tasks on my own: a debugging task, a database task, and a small web site. All three had to be completed in under 2 hours. I wasn't given much direction, other than the specs of the 3 tasks.
While I think this was a moderate evaluation of my skills, it ended up being more a test of if I could complete the tasks on time, vs. the quality of my work: I ended up not making it to the next step of the interview because I was too concerned about quality code over actually completing the tasks (with mediocre code), and didn't complete the tasks, thinking that they would rather get a feel for my quality vs. if I can complete a task. I did it remotely, so they never really got a feel for my character or personality.
Had it been in a paired session, things likely would have gone differently. However, I did learn more about where their priorities were, and ultimately feel I probably would not have been a good fit.
Exercise: give them a class with empty functions, and some unit tests. Ask them to write the class and add some more tests. Also, one of the tests you provide must be subtly buggy.
If you have to weed out thousands of candidates, trying brainteasers to estimate their problem-solving capabilities doesn't seem to be a bad strategy. If a candidate is not familiar with a particular problem, his/her solution may be suboptimal, but the process of getting to that (suboptimal) solution may be a good indicator of the candidate's problem solving skills.
Obviously, syntax-checking a candidate's on-paper Javascript code isn't going to help anyone.
I was asked to come up with a sorting algorithm that would run on 4 computers with more data than could fit on any one computer. I was then asked to do a complexity analysis on my algorithm.
The only thing I think that really tested was how long it had been since I took an algorithms course in college.
One question that annoyed me in particular was "what is stored in an inode?". First problem: Lots of file systems don't have inodes, and those that do store different information in them. Ok, explained that and asked for clarification. He wants the inodes in UFS. I told him I have no idea - I've never had any reason to look into the on-disk structure of UFS. Why would I? BSD is specifically not on my resume, though I've used it now and again. I offered up the description of ext2fs instead, since I happen to roughly remember that, and talked around differences with other filesystems. But he was clearly not happy.
This was one of a series of similar
The recruiter was annoyed to no end, and managed to get the interview result thrown out, but by then I'd gotten so annoyed about the whole process (which had dragged out for weeks) that I just wasn't interested any more.
That said, I work on distributed systems for a living and have actually had to do similar things (manipulate data sets larger than a single machine's main memory across a cluster of machines).
On the other hand, that's not a skill 37signals needs. It's insane to think that Google, 37signals, and a random startup could and should use the same interview process.
[0] Two Kinds of Judgment http://www.paulgraham.com/judgement.html
Then I was given interview questions of a type that nobody else would give for a position at the level I interviewed for, and I quickly lost interest again as to me the interview questions advertised a lack of understanding of the skills required for the position, or a fundamental disagreement about how technical management should be carried out.
And pleading from the recruiter was not sufficient to get me to do another interview (after she got the result of the initial interview thrown out because she agreed with my assessment). The last recruiter I dealt with was good, but the impression I got of the interview process put me off for a long time.
It took several months, and in that time they'd still not been able to find anyone else. I've never, ever spent that long on a recruitment process before, and the only reason I bothered was out of curiosity and because I wasn't really looking. If I had been looking, I'd have found and taken another job long before Google would've managed to get their act together.
I make an effort to sell myself for positions I badly want, but Google doesn't seem to be interested in selling itself in the recruitment process whereas most other companies I've interviewed for are, and they certainly isn't even top 10 of list of places I want to work.
But every time I've dealt with Google recruiters, I've had to waste time dragging information out of them that I'd normally expect the recruiter to be eager to present to me.
I'm not going to make an effort if they've not gotten me to really want the job first.
Recruiters I've spoken to all tells me the people they take out of Google are very often disillusioned and thought it would be very different to their actual experience, and that dislodging Google employees have gotten much easier as it's no longer so much the pinnacle of cool places to work in the eyes of a growing number of engineers, so I think that's going to become a bigger and bigger issue for them when hiring.
I have always tried to push my potential employers to use me on a small real life project. I know it's time consuming (and you end up working for em for free) but it works both ways. I get to know what kind of work environment they have and they get to know me in a real deal. I find it much more comfortable because the puzzles always seem to be a hit and go. The solution may click or it may not.
I know it's a start but if this catches in the industry I expect employers to even pay for your 30-40 hours interview.
It's great if the new employer pays you. But what about your current employer, the one at which you've just had to spend half your annual allotment of vacation days in order to audition for a new job that you might not get?
Auditioning is definitely the best way to, well, audition people. But it puts a constraint on your candidate pool, and depending on your company and your industry that constraint can be a problem. In 37signals' situation this doesn't matter - they're built entirely on open-source tools, they don't require relocation, they can hire freelancers from a pool of freelancers that they themselves created and fostered and routinely market to, they are so famous and impressive that even in a tight labor market people will line up to try and impress them, et cetera. But there are many companies that aren't 37signals, though the 37signals evangelism team is certainly trying to change that. ;)
I agree with the latter. This is how I used to hire programmers, one month on a freelance contract first with a single project. The importance of the code itself is also overrated IMHO. It's just one tiny part of someone you add to your team, what about creativity, taking initiative, never giving up, being a team player, etc. Each of them are equally important to plain coding skills.
I've got a family. I need stability... and health insurance. Unless it's my only offer, a substantially better offer, or a dream job, I'm inclined to go elsewhere.
If I'm job hunting, I've got other offers on the table. Ones that do guarantee a salary for more than a month. Ones that include benefits. Ones that expire long before your trial hire would be over.
If you try using this method, I don't know of a top notch programmer that you would be able to attract, simply because the hiring process is too painful.
As a prospective employee wouldn't you like to kick the tires before making the leap into a new job that you might find yourself wanting out of six weeks in? There's so many things you just can't know about a company until you're on the inside.
Edit: Remember that 37s has a lot to offer: Remote work with great people, top-tier name recognition in the tech world, and a long history of building and releasing open source tools. That's a really compelling package compared to some of the bigger names in tech.
No matter how you look at it, you're asking me to take on significantly more risk with this process.
For developers that are in demand, it's a tough sell.
You can't really expect someone to quit their existing job just for a chance to work with you.
He's obviously very talented at design (not necessarily only UI design but general program structure and framework design). But if you needed someone to write packet routing algorithms or design a high-availability database engine from scratch, that may not be his cup of tea.
OTOH, I bet there are developers who love brain teasers and puzzles who may very well excel at that type of work but couldn't implement a huge software system with many moving parts without having it collapse under its weight.
This a thousand times over. Go through their github (or whatever) profile, look at their commits, see how they write tests, etc. It doesn't take a huge amount of time, if you are any good at what you do, you'll quickly be able to filter out a failure (like, he doesn't have code that you can browse).
Actually that wouldn't be the worst idea in the world.
Not through any fault of that though, I realize that having a github account with projects and such that you work on is a great help, and I accept that myself not having one yet is a great disadvantage. What do you recommend to get started out on github? Is helping out open source programs still viable? Or now it seems either writing self-run projects, or forking others projects and helping them with those. Is there any clear starting point for githubbing? (and how do you squeeze it in after a 10 hour work day!)
Just questions from someone worried about my empty github profile :)
Gitguy: "I used EC2, Blazboo and Chingbang to create an HA job queue that will never fail! It uses counting Bloom filters and I wrote it in Brainfuck."
Me: "What do you use it for?"
Gitguy: "Sending password reset emails."
These Rube Goldberg wannabes are generally more trouble than they're worth. You end up with a system that's a technological pastiche. The drawback is apparent when you try to hire new teammates. It turns out you can't find someone who knows the 12 esoteric packages your business is running on.
That, to me, is 100x more important than "how many zeros are in 100 factorial"
real engineers know that most good code isn't on github and never will be
Irrelevant. Fact is, having code out there makes it easier for prospective employers to get a feel for you, there's no way around that. No one said it had to be the best code or all the code.
Now, if you are working your ass off on proprietary code and write absolutely no code outside of your work, then you won't have it. That's understandable, but doesn't change the fact that it makes it harder for someone to figure out what you can do.
On the one hand I'd agree for students/juniors it isn't a deal breaker. But students/juniors that do have it are showing a great deal of enthusiasm and passion for programming.
But every time I read "FizzBuzz", I drop everything and go back to refactor my latest version, aspiring to get it down to one conditional.
I revisited my FizzBuzz just now and threw together this obscene Python one-liner.
(If I write this kind of code in production, I should be a guaranteed "no-hire"!)
After all this FizzBuzzing I guess I should start writing one up just so I can say I've done it.
Obviously i cannot show something i'm currently working on, and everytime I look back at some code I wrote six moths ago I find myself wondering "did I really write this piece of crap?!". And it's always been this way - at least since i've been six months into programming. I hope it is a good sign.
I think as long as you're applying for jobs you feel you're appropriately skilled for, you shouldn't get too self-conscience about your code. Like most personal things, it probably looks worse to you than it does to someone else. The fact that you are self-conscience about it, likely means you have enough awareness to prevent you from becoming overconfident in your skills.
Is this obvious? It's not like photographers don't have a portfolio of past project that they use to showcase their work. Unless you code under an NDA, of course.
- Good puzzles are actually a talent attractor; many smart people found out about our company via our puzzles.
- Good puzzles are ones that scale well; i.e., where the basic problem is pretty easy and can be solved by a decent programmer in a few hours, but with harder variants that can take much longer.
- "Take-home" puzzles (as opposed to in-person whiteboard tests) weed people out who don't really like to program and/or can't finish things; this is a useful filter. If someone complains that "they shouldn't have to spend three hours writing code to get an interview," that itself is a pretty good counter-indicator. (Yes, we are all busy. But you're talking about starting a relationship with the company that may last 10+ years. You can do three hours of prep work for your interview. And you have to code a fun puzzle in the language of your choice, not some subroutine in a 30-year-old COBOL banking system.)
- It seems that in-person whiteboard or locked-in-a-room tests are pretty poor indicators of success. Many good programmers put in this situation significantly underperform their true abilities.
- It's not a great idea to evaluate someone purely on the basis of puzzle-solving ability.
- Many one-liner puzzles are bad indicators, because you either need to "know the trick" or have memorized the answer. Many, but not all, Microsoft and Google interview questions I hear about -- I've never interviewed at either place myself -- sound like they fall into this category. (E.g., from a recent article I read about Google: "You're reduced to the size of a nickel and thrown into a blender. How do you get out?" I don't care if you're clever enough to answer that; I just want you to able to write enormous amounts of high-quality code for me.)
Im my experience, the best way to find out if someone is a good programmer is to talk to him/her at length about something he/she has built. Can he/she talk in detail about how it worked? What challenges were involved? What tradeoffs were made? And, above all, was there passion behind the work and these choices?
A good interview for me was one where the candidate came in having written or contributed to a large system -- for fun or work -- and was excited to tell me about it. It didn't really matter whether it was relevant to the work we did (airfare search). Anecdotally, it seemed like people who really loved to code generally worked out pretty well. People with great-looking resumes that didn't love to code -- but maybe loved other things, like arguing over which language, operating system, architecture, or business strategy was better -- usually didn't produce much.
If the first thing you want to do when you wake up in the morning is code something, you're probably going to do well as a coder on a hard-core software team. Otherwise, not. Everything I personally did on programmer-hiring strategy was a proxy for figuring out whether someone was like that or not.
i have done your puzzles for fun, but i would still be annoyed at having to do half-baked puzzles if i were suddenly out of a job and applying to a bunch of places.
(ps. and thanks for them :o)
We also never put a puzzle out to the world until a bunch of us had solved it. There was a long gestation period, and there's a "puzzles-discuss" mailing list for working out the kinks (and, in many cases, rejecting certain puzzles entirely). So, yes, you have to make a real commitment to do it well.
On the other hand, developing the puzzles was a lot easier than most of the coding on our real system, so it's all relative. It was actually really fun to do this stuff, and a nice change of pace. One of ITA's best developers (Justin Boyan) would take it upon himself to create the "perl postcard" solution for each problem. Another (Jim Rees) would always make an incredibly fast C++ version; it usually ended up being not only the fastest of ours, but faster than all the submitted solutions as well. I learned things just by reading his puzzle solutions. :)
I found the process rewarding beyond the hiring benefits, and I'm sure I'm not the only one. (Well, I hope so, at least.)
In any case, kudos and thanks to you - those puzzles were a lot of fun, and I always wondered where they came from.
I don't think you give yourself enough credit in the success of the puzzles. They were really well-conceived problems - just difficult enough without requiring too much domain knowledge. Very 'catchy', if that makes sense.
Bingo.
To me, this is the kicker statement for the industry. Solving fake problems like nickel-in-a-blender is for the clever. But I sure as hell don't want clever code solutions; I want rock-solid, reliable, fundamental solutions.
Anyone who feels these questions are an indicator of programmatic success is measuring the wrong things.
I sit on an engineering hiring committee and I can't really think of the last time I saw one of the ridiculous puzzle questions show up in an engineering interview. The WSJ article was pretty ridiculous.
Sorry, but I'm not familiar with the WSJ article you've referenced. Link?
Many of the articles about Google interview questions on sites like Business Insider, at least for software engineering positions, are total bullshit. http://www.technologywoman.com/2010/05/17/debunking-the-goog...
But compared to coding and math, puzzles feel sterile. I don't get to build anything, and I rarely get the kind of insights I would get from figuring out a proof. (Raymond Smullyan's puzzles are a marvelous exception to this.) Even cryptanalysis is more fun, because there's a real opponent.
I know lots of amazing programmers who love puzzles, and that's cool. But if you only hired puzzle lovers, you'd miss out on a lot of good people.
Everyone likes to use Google as the example of the elite programming company. I've read the books about them, used the products, followed their progress over the years. So, sure, a part of their big success is hiring the 'smartest' college grads.
However, I've also worked with Ph.D.'s and even interviewed them for positions lower than ones I've held (I have not yet finished college - some day...). And I can tell you that book smarts != execution smarts.
I feel the same way about puzzles. I have never used puzzles or asked 'small' one off programming questions. Instead I've always been interested more in grep'ing the concepts and if the personality is the proper fit. In my mind, anyone can open up a book and read about specifics about O(log n). But how does a person persist at solving a problem and how do they (can they?) change direction if their solution sucks?
I feel like I value creativity over anything and they can creatively solve a puzzle, cool. But if they can creatively solve a task on the system I'm building and are a great fit to the team, then better.
I'm an API design geek. I've never seen a cool puzzle about API design, or something remotely related.
also, replying more to the other post above - you can get insights from puzzles. a lot depends on the puzzle (which is why ita is getting praise here). for example, the ita puzzle i remember taught me a lot about handling data in large numbers of dimensions (in particular, it made me think about the "curse of dimensionality" which was something i was aware of before, but hadn't considered in detail; it also led me to understanding some tools for dealing with it - locality sensitive hashing being the most interesting at the time).
Lattice animals include polyominoes, which are probably familiar to most HN readers, but subsume other graph-related objects as well. I didn't know much about these, and ended up reading a bunch of cool papers on them as part of working on the puzzle. So at least for me, good puzzles have catalyzed some learning.
My mathematical preference is continuous math; my preference for using mathematics is to apply them into real problems.
I'm not saying that all puzzles are pointless, but I think most interview puzzles are kind of an empty exercise that only a certain type of person likes, and that only people who know the trick will get.
I worked on that one for a looong time, and what bugged me was the screenshot of the expected interface had a result returned in like 0.05675ms or something, and I couldn't figure out how to get it that fast. It took be forever to convince myself that it was just a mock and probably should have read 0.05675s (seconds instead of milliseconds).
We often had internal solutions (and externally-submitted solutions as well) that were orders of magnitude faster than the average. There was a very wide range of execution times. E.g., we had solutions to the Queens & Knights puzzle (in the archive) that ran in tens of milliseconds, if I recall correctly. Typical solutions we received took orders of magnitude longer.
Usually when one solution was a million times faster than another, it was because the fast solution exploited a clever algorithm or data representation -- e.g., representing an 8x8 chess board mask as a single 64-bit register rather than as an array of bytes.
(The sub-millisecond time was for real)
I may have missed that the time was the time taken for the search algorithm to find results, I think I was under the impression that should be the time for the request/response to happen. Hmm.
During the same job search, I also noticed Facebook had puzzles. I solved one and got invited to interview but never went. I wouldn't have applied otherwise. I guess this is another lesson - if you put public puzzles out there, expect people to send solutions without necessarily being excited to work at your company.
But the programming puzzles on the website were just awesome, even though I just solved them because they were fun, not for applying for a job. I've been surprised that nobody else has done the same thing, but your explanation of how much work went into the puzzles kind of explains that :-)
I'd never seen it before the first interview. In that first interview, I broke it down quickly into its component parts, and figured out where the trick had to be -- and from there quickly solved it on the spot.
In the second interview I spent a couple minutes trying to remember the solution from the first interview, and completely flubbed it. It wasn't the first part of that second interview that I flubbed, either, including arguing with one of the interviewers and basically telling him he was wrong.
Oddly enough, I got the job with the second company and worked there for almost six years. For the record, I love doing those kinds of puzzles. But I do video game development, so maybe puzzles are a bit more relevant to my industry (in that 80%+ of what we're doing is figuring out how to solve puzzles in the form of how to do something with less CPU or why a particular video card is behaving in a certain way).
If I were asked such a question in an interview, my response would be: "Dunno. Let's talk about some real-world problems you're facing right now instead."
I dislike trivial questions, especially during an interview. I'm a professional, not a circus animal.
But the puzzle process, as originally conceived, (in 2000!) was not necessarily supposed to be a hard filter. As an applicant, you had several ways to stand out, and doing a great job on a puzzle was one. Having someone who already worked for us say you were awesome was another. Being an open source superhero was another. Etc.
In actual implementation, though, it probably was/is used as a hard filter.
It always becomes a hard filter, there is a lot of ego in development and test are an area where ego's shine through. It's my major beef with quiz exercises in interviews, they always become the hardest filter, no matter how much we try to contain them.
I did one of them just for fun 3-4 years ago: "implement a scalable multi-user chat server without using a framework like Netty/ACE/EventMachine". Unfortunately, at the time I couldn't move to Boston, so I never submitted my solution.
Likewise there were puzzles that would appeal to web developers (e.g., build a typeahead web applications), machine learning/AI folks/etc...
It is much more like having to travel 90 minutes each way to get to where they want you to interview. That could be very offputting if you do enough interviews that three hours each adds up to a big number, but at least you don't need to worry about them being tempted to rip you off.
Of course, we couldn't easily do A/B experiments, so we'll never really know how the true attract:alienate ratio.
Trying to look at it objectively, I would probably be somewhat put off by being required to do a puzzle to "prove myself", but would then find the specific puzzles offered to be pretty interesting, and worth playing around with. Perhaps it's not tantamount to a $300 tax on ones life it is seemed fun and valuable in its own right?
Note: edited typos.
I was thinking more of all the cool search problems like Strawberry Fields, Traversing the T or Palindromic Pangram, where coming up with some solution is easy, a naive exhaustive search for the optimal solution was not feasible, and where verifying the validity of any heuristics / pruning was not necessarily trivial.
I gather that someone with a stronger theoretical CS background than myself would have spent an hour looking at the puzzle, said “Ah, this can be modelled as a graph and solved by applying the Flibblewhiz Heuristic!”, spent another hour coding up a first draft of the solution, and a third hour optimizing and cleaning up the code.
My ego has survived the blow of learning that I am not awesome enough to work for ITA, but I wish I had figured that out sooner.
So, here is my opinion as a non-halfway-decent programmer.
I thought puzzle was fairly fun, but I like programming for two reasons:
* Building useful knowledge. Learning a new language, library, or algorithm I want to play with, programming is the natural way to learn.
* Building cool stuff that people like. Having an awesome product when I'm done with it gives the feeling that I've done a good job and produced something of value.
Programming puzzles don't really do either of these things for me. My motivation for spending three hours doing a puzzle is that I would like to talk to a person. I am basically assured that the puzzle will be used as a filter for the company to avoid talking to people. It will run against a test suite and if it fails it will not be looked at by anybody. This seems very impersonal. Is this the experience I should expect from the rest of the company if I'm hired?
I've got a Git-hub account with some of my code in it. I've spent time on it and enjoyed writing all of that code. I published it to the internet so I'm (at very least) not ashamed of it. Why can't you look at that to determine if I can program? Why do I have to write code specifically for your particular company?
This isn't an attack on ITA, it seems to work for them. But they are a unique case of a company where it seemed worth it for me to spend the time to do the puzzle. Any other company I would have just skipped it unless I genuinely had nothing better or more interesting to do (of which I almost always do).
When I applied at ITA they didn't even acknowledge the program I had written to go along with the application. Not even a quick 'it ran' or 'had the right output but we hired somebody else'.
I'm glad I didn't get the job because not having enough time to evaluate or even run a programming test you gave is also a pretty good counter-indicator of a good company to work for. And when I wrote back to express this the guy sheepishly admitted they had so many exceptional applicants so they just went with an Ivy-league graduate. Wow.
If you are going to require a programming test, be prepared to evaluate it.
I agree. It is possible the reviewer didn't think your code would run, couldn't get it to compile, etc. But, nevertheless, you should have received a fuller response than that. If I were still COO at ITA, I would ask someone to look into your particular case to understand what happened.
I suggest this with tongue-in-cheek, of course, but it's about as effective as any other method.
I understand what they're going for though - the companies that do it correctly are looking at the candidates thought process through a challenging situation.
While I think it's important to figure this out, I believe it's more important to know how well the candidate can do the actual job, that of sitting down and writing quality software. Brainteasers and writing modified sorting algorithms won't get this (unless the company is in the sort algorithm business I suppose) but having the candidate write code which could actually be used in the job would.
I had to deal with a crisis this week that boiled down to a poorly optimized SQL query or two. Of course, the manner in which they failed meant our DB started disk swapping and hurting our availability.
Contract to hire is something we do but it is rare. An initial try out period doesn't do us much good since the learning curve is long.
I don't trust take home assignments unless their solutions can't be googled. And it's a lot of work to create and evaluate problems of that nature. You want them just long enough, not a huge time sink for you or the candidate. You want them to demand a range of solutions so there are many ways to fail and succeed.
I agree we should not be doing riddles/puzzles. For me the most straightforward way to gauge a candidate's ability in an hour is to ask them to program on the spot and see how they act. Yes, it's a game and the candidate can learn to beat the system and oversell themselves. But the vast majority of people I don't want to hire do not go to these lengths.
I haven't really used their products but in terms of technical sophistication they probably aren't much advanced beyond CRUD type web apps (I may be wrong here).
If you wanted to hire a games engine developer or someone to design algorithms for google then brainteaser problems might be more relevant.
But when I'm interviewing, I do ask people for a pointer to their open source projects. And if they don't have open source projects, I give them a workstation and ask them to write code. I've seen too many self-proclaimed programmers who claim, "I spent the last 2 years writing Python," who can't sum a list of 10 numbers without using Google. (Hiring can break my heart.)
One way or another, you've got to see code. Real code is best, but it's better to ask people about toy data structures than it is to hire 3 team members who can't write FizzBuzz.
I did a hiring round once for my old team when I worked at Yahoo years ago, where out of 10 or so CV's the recruiters passed me, about 5 people who were given a short list of screening questions by e-mail for them to answer at their own leisure produced answers that were obviously plagiarized from various online sources.
I got suspicious when one guy presented me with two pages of nicely formatted answer complete with section headings that didn't make any sense to a question that should have required a one line answer. He'd copied and pasted the whole thing from an on-line copy of an Oracle manual, so I started doing searches of random sentences from their answers.
Another one managed to cut and paste an answer that was flat out wrong from a forum post answering pretty much the same question, and hadn't bother to read further in the comment thread, where several people corrected the answer he'd used...
It's when they can't even answer simple stuff WITH using Google it gets really depressing, and it happened regularly enough that using screening questions by e-mail was a real time saver in weeding out undesirables, despite the ease with which people could "cheat".
My favourite STL-related question to ask is simple and can be done on a whiteboard in minutes: if you didn't have std::set<class T>, but you had everything else in the STL, how might you implement a set<class T> class?
There's an ideal answer[1], and then there's people who start thinking about low-level implementations. The ones who I don't want are the ones who don't have any answer at all. I'll continue to be interested in anyone who starts talking about the way to solve the problem. What matters to me is not any code that you write (although that's interesting) but your thought process. After the candidate has worked through this, if they haven't gotten the ideal answer I always give it to them: this isn't some national secret, and seeing them process the ideal answer and realize that yes, it really would work is a pleasure.
Sadly, there are cases where it doesn't work. I had one candidate who did very well on the phone screen, but didn't know the STL. I looked up what he claimed to know (Qt) and reframed the question in terms of Qt's classes—and he still couldn't come up with any answer whatsoever.
-a
[1] The ideal answer is precisely such because you're writing much less code and using stuff that people smarter than you have written. It's to use a std::map<class T, bool>.
Not everyone has performance anxiety, but I've met plenty of programmers who do. Unless you spend the first 15 minutes of your interview calming them down and warming them up with simple technical questions that put them in the mode of programming, you might as well be asking them to sing the star spangled banner naked in front of an audience.
My confusion was the job was looking for a generalist who had experience with lots of different technologies. Specifically, working with node.js and socket.io polling methods, API design, working with some python framework and backbone.js.
I'm honestly not sure what the lesson there was. Am I a bad developer? Do I navigate interviews poorly? Does my breath stink?
I wouldn't have been able to tell you what an insertion sort is, by name. But by looking at that image, I can describe it pretty well:
Have an outer loop that counts from the first position in the list to the last position (call it n...). At each step in the loop, make sure that the first n items are sorted. This probably implies an inner loop that finds where, within the first n-1 items to insert the nth item. And given that with only one item, it's already sorted, we can probably start with n on the second item.
Yeah, I think I could code that, given the image.
If an employer says they will get back to you and 2 weeks go by, it's worth contacting them and asking if they have made a decision yet. If they say yes and you don't have any clue why they wouldn't have hired you, I don't think it hurts to ask them why not. You might get a BS answer or you might get some good insight.
Just keep improving your skills, and keep looking.
Trying to evaluate programmatic skills is surely a challenge, but I've found a method that seems to work very well -- simple walkthroughs. I'll do at least two walkthroughs of code, asking a candidate to explain to me what's happening using two different artifacts:
* Some code they've written that solves some problem
* Some code we've written that solved some problem
This has proven to be quite effective in rooting out those who look great on paper and talk a great game, but fall apart when put into action. On the flip side, I've yet to find someone who could explain their code in sufficient detail that then proved incapable once in the job.
(Also, I've never heard of G+ being in any way a product of a talent acquisition.)
That's a very good point that I've not seen made so well before.
After so many failed Google interviews, I found it easier to build a company and sell it to them, instead.
I would argue that they don't and I believe that it is due to their hiring process. Now I do believe that they have very talented individuals but there organization reflects their interviewing bias as such so does their product portfolio. There is a reason that they have to acquire new products for their portfolio and that reason is that they are not recruiting the kind of people that generate new products. As such their hiring process should be suspect.
37signals hires for business value because, as a small company, it has to. Google is a huge company and doesn't have the same incentives. Google's size can blind it to the fact that its hiring practices optimize for mathematical knowledge and intuition over business value. And, as a business, Google is wrong and 37signals is right.
I don't believe that's the case. What I've seen with talent acquisitions (at Google and elsewhere) is that interviewing the team is actually prerequisite to making the acquisition. Of course, the interview doesn't have to be as rigorous: during the due-diligence process you can actually view the codebase and see the individuals' contributions.
What I've also seen, specifically in the case of Google, is that individuals from acquisitions would be asked to stay a year, at the end of which they'd need to interview for other positions at Google: if they couldn't pass the interview, they'd be let go with a very generous severance package.
That said, I am sure there are exceptions.
Generally whenever I've done this to recruiters their interest level goes up and they tend to be far more willing to listen to your concerns about the interview and go to bat for you. And recruiters tend to have a lot more ability than you might think in getting parts of the technical interviews largely ignored by hiring managers - including at Google.
Incidentally my impression from having talked to several Google recruiters is that a decent number of them are incredibly frustrated with the way the process work, and will work hard to get you past this hurdle if you want them to and provide them with enough ammunition (e.g. explain why the question was inappropriate for the position).
Today I Learned ....