Yeah, I'm not sure. It looks like maybe they changed their blogging backend from whatever in-house thing they used to Medium? Their blog (https://data.triplebyte.com/) is now showing Medium stories with publish dates from last month. Maybe Medium doesn't let you set arbitrary dates when migrating?
This is certainly true. All migrated stories get their publication dates set to "now." In my case, I added a footer on each migrated post stating its original post date.
My bad! I did an HN search for the link before submitting it, I find that gets around cases where people change the title. But I could have done a title search as well.
Well it's clearly not that much of a problem, HN has such an appetite for what they're doing/writing about that it's back on the front page. Also for what it's worth even though I'd previously read the article I made it about halfway through before it clicked for me that I'd already seen it.
We moved the blog to medium about a month ago. Since writing this, we've tested a bunch of new things, and now have about 20% better precision on the quiz (20% increase in the percentage that we predict will pass and go on to pass). There are some other interesting results (one is that easy programming problems are about as effective at screening as hard programming problems). I'll be writing more about this, with the new data/results in the next few months.
>As soon as we started doing them however, I saw a problem. Almost everyone was passing. Our filter was not filtering. We tried extending the duration of the interviews to probe deeper and looking at code over Google hangouts. Still, the pass rate remained too high.
To me, it seems like your team isn't that good at judging ability. To compensate, you rely on coding questions. This what the industry has been doing for years. What exactly is innovative here? I see the need for coding questions (and think that your method sounds pretty good) but too often culture fit & 'soft skills' are overlooked in favor of cramming as many technical questions as possible.
One of the best experiences I had on an interview involved no coding on-site at all. I completed a project for them, they liked it, I got an interview. 3 hours of culture/team/work philosophy. There were no surprises - if I wasn't "technical enough", I wouldn't have been let in the door. It took time up front, but at least the interview itself wasn't waste of time, PTO, and money.
So you bet your time. It's how I got my job too, but that should be the norm (unless there are guarantees).
Companies should cover the expenses of finding employees, instead of crowdsourcing the unemployed.
3 hours of soft questions seems much more preferable to 3 hours of whiteboarding data structures. Hopefully your project wasn't a bunch of questions from an undergraduate data structures/algorithms class.
They were tough, insightful soft questions. And I asked a lot of them too. Gave me a good feel for how they worked and what they valued (which is what I really want to know). The challenge was pretty tough but it wasn't just graded on performance. The team lead came in with feedback from each member which was really good. Stuff like "you could have done this with less objects" or questions for me to resolve for them. I've used the feedback to improve my code since.
What the heck is going on with the statistical explanation?
The chart's y-axis is labeled "correlation coefficient" but the text blurb analyzing it describes the definition of R^2 instead (% of variation of Y explained by X). The clarification at the end does not disambiguate this. Also, how do you numerically quantify "interview performance"/Y?
Additionally 3 out of 6 of the variables have error bars crossing 0, which means that the correlation could be positive or negative but we don't know (it would handily fail a statistical test of significance at p = 0.05 significance). Why were they included at all?
Many startups (including statistical startups!) have been playing hard and lose with statistical analysis lately and it's getting annoying.
I also would not consider an r-squared value of 0.47 indicative of strong correlation between two variables. I guess the point is that the composite quiz score is more correlated with performance than traditional methods, but I still get the impression that someone is overselling something.
I would applaud the author on trying to take a data-driven approach to discover the best hiring signals, but I must offer one major critique of the approach: These data are ultimately not too meaningful unless you correlate actual job performance with interview performance. By the author's own admission, interviews are often faulty. By correlating final-round interview performance with early-round interview performance, you are really only predicting how candidates will perform in the final-round interview, which is not necessarily a good predictor of long-term job performance.
If you are serious about this then you should be interviewing candidates, deciding on whether you think they should get a job. Then giving them a job anyway and evaluating their performance after some time interval and see how accurate your interview process was in terms of weeding out those who would fail.
In fairness, the author says at the end of the post that they do intend to rerun the analysis with actual job performance.
It seems reasonable to optimize for hiring right now since otherwise nobody would go to them to find a job, and it'd also be impossible to measure job performance when the people you collect data can not get jobs.
Which seems totally at odds with what their clients want and furthers the perception that tech interviewing has become a hazing ritual totally detached from evaluating the ability of the interviewee to actually do the job.
Absolutely, we keep in touch with companies post hiring and are gathering that data too. It won't be meaningful for a while though, interview performance gives us a tangible starting point.
The language in that description refers to "protected groups". As long as the author is not calculating selection rates for groups of candidates based on some special criteria, such as race, gender, age, etc. he should be ok, right?
"adverse impact" is designed to sniff out hiring criteria that are a proxy for discrimination based off criteria such as race, gender, and age.
For instance, if you selected for college graduates for hiring firefighters, and that causes a lot of rejected applications among African American candidates, then it would be illegal based off of an adverse impact against a protected group.
As an aside, suing employers over requiring a college degree is a seriously under-tapped and under-estimating way to charitably improve the lives of Americans (college is a positional good because college matters when employers are looking at candidates - if employers can't look at whether or not you have a college degree unless it's a bona-fide qualification, then many more people don't have to spend several years and tens of thousands of dollars)
That's not nearly enough. The 4/5ths rule is designed to identify "structural discrimination", or cases where no discriminatory act can be identified, but the outcomes are nonetheless considered discriminatory. As long as a lawyer can show that some protected group is disadvantaged by the quiz, that's enough for it to violate the rule.
I am not a lawyer, but I thought the defence was if you can show that the screening criteria correlated with on the job performance it is OK. No matter what you think of protected groups using a criteria that does not correlate with performance is not very smart anyway.
There may be some certification process you can go through to get a test approved, but mere job applicability isn't enough. For example, this firefighting test was found to violate the 4/5ths rule.
Why don't companies manage open source projects, and simply require a certain amount of contribution as a gateway to a final interview? This would allow candidates to prove their ability to write production ready code.
This article is extremely interesting, but it seems to be equating success in interview decision with competent performance at the job, which has not actually been established yet. It is interesting though to see which components of pre- and early-interview evaluation ended up being related to the decision to hire.
I've conducted a lot of programming interviews over the past three decades. I think that it's easier to do at places like Google where the objective is to avoid a Type I (false positive) error and it's expected that a lot of time and money will be spent to identify the best hires. It's harder in other contexts where Type II errors matter more.
I have a big portfolio of technical questions, but I usually stick to a small set that I consider to be well calibrated by posing them to smart coworkers and seeing what they do with them.
I've found, counterintuitively, that the best indicators are the questions that deal with "programming in the small", such as coming up with a predicate expression to describe a test for a simple relationship between a few values, because either you can do it easily or you can't do it at all. And programming in the large depends on a lot of cases of programming in the small.
I certainly hope that they don't ever assume that experienced candidates don't need a programming test.
There is a particular kind of dysfunctional candidate who once was a programmer, graduated to being a lead programmer, and then became an architect. As an architect they stop programming, and their programming becomes rusty. Once that happens, the feasibility of their proposed "architectures" becomes steadily less practical.
You can detect them by their first getting offended that they would be asked to program during an interview, and then failing to actually write working code.
By contrast people who really do program have no objection to being asked to program in an interview. To the contrary being asked to actually program is seen as a positive sign that they won't have co-workers to deal with who can't actually program. (This is an unfortunate experience that most programmers try not to repeat.)
I have declined some programming tests, when the employer wants yet another three hour challenge to build a mini app. That leaves not much time for actually thinking about a problem and rushing out a crappy solution. Not the way I like to write software. I have code online I can show them and talk to them about, and extend if they want to see me actually code.
Bashing out code is usually the simplest part in writing a software solution to a problem.
I write lot of code, but I refuse to do coding at interview. For me its sign of dysfunctional company. But I do not mind to do pair coding, and fix an actual bug.
There are more reasons, in short it takes me couple of hours/days to warm up and grasp the problem.
And secondly I refuse to waste my time, on half arsed throw-away code, with zero results.
> And secondly I refuse to waste my time, on half arsed throw-away code, with zero results.
That's a very wierd thing to say. Does that mean that each and every side-project that you do ends up being substantive? If writing few lines of code gets you a job at a good company where you can spend your time working on exciting problems how is that wasteful?
I do program a lot too, and I typically don't mind writing code in an interview. The only thing that sets me off are stupid level-order binary-tree traversal questions which test nothing but rote-memorization skills.
Most of my code is thrown away. But I like to finish stuff, so there is a chance I spend next weekend finishing/polishing the interview questions. It is great for career, but has to be managed.
Also my job is to design database structures, and there is no such thing as simple problem. Even binary tree needs thousands LOC for tests, concurrent access, writes...
I mean you should know how the basic data structures work. Not necessarily for a web dev job but for a software engineering job. Data structure interview questions are a good small concrete problem to reason about.
But they completely ignore the other 80% of the job which is design, soft skills, estimates, timelines, code reviews etc...
> I mean you should know how the basic data structures work
Oh I'm not discounting that. I'm just saying that off the bat binary-tree traversal don't really test my understanding of the data structure. Recently in an interview I was asked a problem which eventually required using a binary-search-tree to improve the running time. This, I believe, is the right way to test but it requires the employer to spend more time designing interview questions rather than relying on a problem bank[0]. (which is a strong filter for good companies in itself)
> But they completely ignore the other 80% of the job which is design, soft skills, estimates, timelines, code reviews etc...
This is true but this is really really hard to test. In the interviews that I've been giving, I've indeed been asked design questions. I assume soft skills are implicitly tested while speaking to the interviewer (to some extent, at least). But other factors, AFAIK are very hard to test. Also, I would argue that a candidate doesn't need to know everything coming in. Estimates, timelines etc can be learnt on the job.
IMO, if somehow employers could find a way to reasonably trust that the code I have on my Github is indeed written by me then I think that should be strong indicator of my competence. Evaluating candidates who don't have any code in the open is a different matter altogether to which I have no solution.
No offense, but I'm skeptical of responses like this. If you need a few hours or days to "warm up and grasp" an interview question like fizz buzz or card shuffling, then you're just not up to snuff.
For every good developer offended by coding in an interview, there are 50 idiots who can't code, who are using that as an excuse to con their way into jobs they're not qualified for. IMO, it's an acceptable trade off.
If you're that awesome you wouldn't be going through the usual interview process, anyway.
I find people that quickly do fuzbuzz type problems to be worse on average. The only way you can validate your interview assumptions is to randomly higher people you would have passed on. You could compare how important you think X is vs. how predictive it was for on the job skills, but few people actually onboard enough people to get good data.
Sometimes I've noticed I just don't "get" the problem being posed. Either it's because of ESL or well I just don't focus very well sometimes. It takes me a bit to figure out what's being asked and at that point it's usually very easy for me to go think about the solution and assumptions and such. But it does happen.
Unfortunately my bugs are typically too gnarly and need more context than a 1-2 hour interview allows, or I'd definitely incorporate them into interviews. In the best case, it's like free labour!
Even if you are exactly as you claim, MOST people who claim "I can code but don't care to demonstrate it during an interview" are actual disasters as co-workers. I've been there, done that, cleaned up the messes, and would prefer to pass on some good people to avoid that.
And yes, you do have to pass on some good people. There are competent programmers who freeze up during interviews. You do what you can to make the interview as accommodating as possible, but some people can't show what they've got. However the opportunity cost of passing a good person by is far less than the damage from hiring a bad one. This leads to being risk adverse.
I guess people get scarred both ways. I've certainly been on 20+ hour interview processes that have left me with an awful offer at the end which I just could not take.
Personally, I've never been a part of a company where the code test isn't:
1) Short (1 hour)
2) On their own time
3) A real problem (algorithms are OK in my mind as long as you help them with the core problem)
I do not expect everyone to know every algorithm... I guess it is kinda weird if you do not know Dijkstra's algorithm, but I only think it is weird because the courses I took focused heavily on it and similar algorithms.
But, do I want you to know all the same stuff I know? No, definitely not. I don't want 2 of me I want me and someone else that works well with me but ultimately has their own ideas.
The OP wrote a blanket statement stating if you follow this specific path you become "rusty" and ultimately end up not being able to code. He then states this becomes an issue when architecting solutions.
To me this is clearly non-sense as someone who's followed this path (and know others who have as well). Being an architect allows me to see the big picture and move the organization in the right direction (as we see it), while still being able to get into the trenches with my developers when something arises or help is needed. Granted, this is different for large enterprises but for startups/small/mid sized businesses, yes, us architects still get our hands dirty (or else we're irrelevant).
I said that there is a specific kind of bad candidate to watch out for who generally HAS followed that specific path.
I DID NOT say that ALL people who follow that path will become that kind of bad candidate. Or even most. Just enough that you can't afford to say, "This guy has been programming for 15+ years, is an architect, and sounds smart. I'll skip the programming test."
It is about your skill set, not your title. I have met architects who were unbelievable coders. And ones who were nightmares. But every good architect that I know was also a good programmer. They might or might not actually spend much time programming, but they clearly could do it.
My understanding of the reason is that a good architect needs to be able to mentally connect every level from abstract architecture down to actual code. If you can't, then you'll eventually absorb ideas that sound good but don't really work in practice. (A random example that I dealt with was someone who didn't understand the latency added by sequential RPC round trips. And that the latency gets worse if you're making the mistake of using XML as your on the wire format...) And the failure of said ideas can always be attributed to programmers having screwed up your brilliant architecture.
For the record, this observation about architects is not original to me. I spent a year at Google, and in interview training we were clearly told that Google does not have a position for a non-coding architect. This is not to say that Google does not have positions for architects. http://research.google.com/pubs/jeff.html certainly wears the architect hat well, and is behind the design of large chunks of fundamental Google infrastructure. See http://static.googleusercontent.com/media/research.google.co... for an example of his showing that skillset off. But they have to code. Which Jeff does. A lot.
Not always, granted, but the format really can matter.
Grabbing a random benchmark, look at http://www.maxondev.com/serialization-performance-comparison.... It is in C#, but every language is going to give similar results. An efficient on the wire format like protocol buffers is a bit faster and a bit smaller than a sane text format like JSON. But XML is several times slower and several times bigger. He didn't test pretty-printing the XML, but that makes it several times slower and bigger still than plain XML.
This adds up sooner than you'd think. For a start there is a noticeable timing difference between an RPC call that fits in one packet and one that has to be split across two.
If you've got an application that spends a sizable portion of its time in serialization/deserialization, it is obvious that the format chosen will affect maximum throughput. (Many distributed applications spend most of their CPU time doing those two things.) It is harder for the format to have a noticeable impact on latency, but it both can and I've seen it happen.
Could this not be solved by telling the candidate what you expect and then telling them that if they can't program to the expected level that they will be fired on the spot. Leave it to them to demonstrate that they can program before or after they are hired. If the candidate can then they will be happy to demonstrate after they start. Everybody is happy.
Except as the candidate, I'm not happy. Companies already hold the balance of power. Pre-threatening me just means I'm not going to leave my current job.
First of all nobody sane would take a job where they are expected to measure up to some unspecified measure or else get fired on the spot. And any competent HR department will tell you that carrying through on the threat will risk the potential for lawsuits. (They want a paper trail showing that the person should be fired.)
Secondly it takes a surprising amount of time for an incompetent programmer at an average organization to get discovered. You usually spend a period of time studying how things are set up before anyone expects you to do anything. And by the time you can be evaluated you've certainly used a bunch of other people's time with your questions. It is far more efficient to determine this in the interview process.
And thirdly, a creative company can more easily solve the problem by offering a choice of ways to demonstrate competence. See http://www.starfighters.io/ for an alternative to traditional interviews.
It is not illegal to fire someone who can't do what they were hired to do especially when they were told what was expected at the interview stage.
My thinking is the first x days on the job is spent doing programming tests. This should generate enough paperwork to satisfy any HR department. You then have the time to test all aspects of the candidates skills, rather than the limited range you can test in an interview.
In a lot of the US not only is it not illegal to fire people for incompetence, but you don't even need a reason to fire them. But stupid people when offended sometimes launch silly lawsuits alleging all kinds of crap, and those lawsuits can be expensive to defend. A lot of what HR does is stuff to reduce the frequency of such lawsuits, and to make the ones that still happen go away more cheaply.
My thinking is to move off the time and expense of testing candidates from the interview stage to as late in the process as possible. A really though test of a candidate is time consuming and to do it at the interview stage is not fair on the candidates and the company. You can provide any candidate with the full list of what they will be tested on in their first x days so they can decide if the job is for them or not (i.e. everyone is upfront with what is expected).
The only problem I see with this approach is Dunning-Kruger effects. My experience in hiring is it is pretty easy to pick candidates suffering from DK by comparing what they say with what they can demonstrated via their work history. It is the more marginal candidates where they seem non-delusional, but where they don’t have a perfect CV. These people are often overlooked by other companies, but they often turn out to be fantastic employees. Taking a chance on them can be a great.
"we’ve come up with a replacement for resume screens, and shown that it works well"
No you haven't. You might have, but you'll have no way of knowing that until they've been on the job at least 3 months.
Personally, I think it's bullshit that I can show employers all the things I've built and they still expect me to write a reverse sort btree or whatever the fuck thing that has nothing to do with web development.
College grads are perfect at passing these tests and they have no fucking idea what they are doing, but will get more offers and better pay every time.
Experience counts and if you disregard it then I'm firmly in the "Fuck you, I don't want to work for you" camp.
Unfortunately, I've resigned myself to memorizing the typical interview questions or going into management. I'd love to stay a dev, but being ruthlessly quizzed about shit that you never use in real life makes changing jobs a pain in the ass.
> Personally, I think it's bullshit that I can show employers all the things I've built and they still expect me to write a reverse sort btree or whatever the fuck thing that has nothing to do with web development.
Aye. It's a hard problem: as an employer, you want to be pretty sure that you're getting someone on team that's going to bring you value.
How do you do that? I guess ask them as much as you can.
You definitely present a good problem, it is the same they're trying to solve over at triplebyte.
> College grads are perfect at passing these tests and they have no fucking idea what they are doing, but will get more offers and better pay every time.
Yeah, you have to study for interviews. It sucks, but almost every software company asks the same questions. Just prepare, I guess, and it'll pay off with that higher salary.
> makes changing jobs a pain in the ass.
You might have nailed it here. Employee retention is good for everyone!
I think the solution is you need connections. When someone works at the company and they know you're a good dev they aren't going to ask you questions they ask other developers.
I recently did an experiment where I took 50+ Android UI libraries opened them in a modern IDE and did code audits to find glaring errors ranging from Java to Android Best practices.
Even exp will not help if you are TOO DAMN STUPID TO RESEARCH THE WARNING AND ERRORS the IDE points out to you AND RESOLVE THEM!
>>Personally, I think it's bullshit that I can show employers all the things I've built and they still expect me to write a reverse sort btree or whatever the fuck thing that has nothing to do with web development.
As a business analyst interested in a software development career, this is what worries me the most about interviews. I write a fair amount of code. I have developed apps that work really well and solve important business problems. I can talk intelligently about everything I've developed, but if you asked me to write something on the spot I'm not sure how well I'd do. Most of the time I find the information I need through exhaustive searches on the Internet as well as lots of trial and error. Is it really a deal-breaker that I can't write a tree sort algorithm off the top of my head?
Why should they expect that your experience will count for anything if you aren't willing learn a bit about data structures to pass that portion of your interview?
How do they know you actually did the work on the sites/apps you present? Dead easy to take credit for someone else's work.
I'm saying a desirable trait is that you can learn new things to solve problems. You know approximately what the problems they will give you are, if you aren't willing or able to go out and learn the necessary skills then why should they hire you?
The large portfolio comes with a large list of references. I could also show them the code for every project. I've never been asked to produce either of those in an interview. No one has ever looked at my github account either.
I never said I don't want to learn new things. Quite the contrary. I stay up to date on the latest web tech and have been frequently contracted for that exact expertise. However, I do find it frustratingly annoying that I need to learn a multitude of "interview puzzles" that I'll never use in real life.
I have to take exception to this because part of what I do for a living is teach tech companies how to hire.
The claims are:
1. Performance on our online programming quiz is a strong predictor of programming interview success
2. Fizz buzz style coding problems are less predictive of ability to do well in a programming interview
3. Interviews where candidates talk about a past programing project are also not very predictive
Those claims may be true, but it misses a key point: no matter how well someone performs in an interview, it doesn't mean they'll perform well on the job. In particular, the post talks about hard skills (and does well there), but doesn't talk about soft skills.
It's absolutely true that someone can talk a big game about hard problems they faced that they solved, but failed to mention their use of Google or StackOverflow. So unique coding tests can definitely tell us whether or not a candidate can really code (though the tests need to be tailored to the sort of real work the candidate will face). However, programming is a hard skill (one that is easy to teach), and not a soft skill (one that you tend to learn over time or have a natural inclination for).
Case in point: early on we made the mistake of hiring a very talented programmer on the basis of his reputation and a brief interview. He was a disaster because he was completely unable to work without a very hard set of requirements. In the very rapidly changing, agile, environment we put him in, he flailed badly because he was good at implementing requirements, but he was very bad at projecting himself into the customer's shoes. Later, we put him on technically demanding tasks which required considerable talent, but had no ambiguity, and he was fine. In fact, he was able to implement tasks that would have been very difficult for other developers who were fine in ambiguous environments.
Soft skills are tricky and most employers have no idea how to interview for them. They require an understanding of the actual role someone will be hired for and the set of skills necessary for them to succeed. Are there tight deadlines? If so, can the developer prioritize tasks and make a case for delaying less critical portions beyond the deadline? Is it a rapidly evolving environment which makes it hard to nail down exact behavior? Are you on a team with strong personalities who often disagree and can the dev handle that? Is the developer required to report to non-technical people who have no understanding of technology and can't appreciate what refactoring is?
Or here's a fun one for you aspiring managers: we've had one project with a superstar developer who was so monumentally productive and so brilliant at his work that some very talented developers who worked with him nonetheless had negative productivity at times because they spent so much time playing "catch up" with the superstar and understanding how the project was evolving that they spent more time keeping abreast of the code than developing it. The code was "bus sensitive" because one brilliant developer was so competent. How do you manage that?
These and other soft skills are part of what you really need to evaluate because depending on your work environment, you'll have these and other soft skills as critical skills that a dev will need, but you almost NEVER see interviewers check for these because very few people are trained in interviews.
Pro tip: start reading up about structured interviews. They've been proven via multiple studies to have a strong correlation with employees being successful. Unfortunately, they're harder to conduct and require training to understand them. As a result, companies don't know about them or worse, ignore them, because it costs more money. But what costs you more money in the long run? Taking longer to interview or hiring the wrong person?
I gave the TripleByte application a try very recently, but this step wasn't a part of it:
> 1. A fizzbuzz-like programming assignment. Applicants completed two simple problems. We tracked the time to complete each, and manually graded each on correctness and code quality.
Instead, I was presented with a single, rather convoluted, 1-hour long programming problem that was most definitely not anywhere close to the difficulty level of a FizzBuzz test (the solution they're looking for probably involved dynamic programming, which I honestly haven't had much practice with). This was before I even spoke to any humans on the phone screen or interview steps of the process.
I couldn't finish it in 1 hour (I kept going at it on my own time for another 2-3 hours before I finally arrived at a sub-optimal solution, at which point I had enough). I'm sure there are plenty of people who are much better at this than I am who can finish it in an hour. But I do want to question the wisdom of using something of this level of difficulty as a screening test.
Your clients seem to value the "Product Programmer" very highly. Using a question like that as a screener will weed out a significant portion of these "Product Programmers" before they ever get a chance to showcase their product skills. Most "Product Programmers" I know (myself included) don't do enough convoluted dynamic programming problems in their spare time to be able to solve a dynamic programming problem like that within an hour with any level of certainty (and yes, solving problems like that quickly involves either extraordinary talent or lots and lots of practice). Instead, in their spare time, these "Product Programmers" like to build things, and learn new technologies and techniques to help them build things in a simpler, more maintainable, and more productive way.
The best programmers I've worked with had this quality:
They'd always heard of everything, and they had a good idea of what everything was, even though it was quite far from the current task to hand. For instance, plenty of people have not had the opportunity to work with functional languages, yet still have a good grasp of why they're useful, what languages there are, who uses them, and so on. They tend to see coding as a vast field where you can make many valid decisions, each with their own pros and cons.
The kind of people I'd root out are the ones who merely see programming as a tool. Something along the lines of "I've got this data in a spreadsheet, I need to calculate this statistic, so here we go..." They almost always end up creating an unmaintainable mess, and they never mention any well known pattern. Quite hard to identify in an interview, though. Perhaps the lack of key words that suggest a deeper understanding?
(Hey, that's an interesting thought. You could machine learn what words good devs use vs less good ones.)
As for coding in the interview, I think this can only root out the absolute novices. It's quite hard to think of a short test that will demonstrate that someone can't write elegant code. Perhaps a take-home, open-ended test would work, but you're gonna lose people who don't want to spend a whole weekend doing it.
15 years ago you could get a software job by showing some passion, intelligence, a CS degree from a average university, and some knowledge of memory / space complexity of typical data structures and algorithms. And, if the hiring manager (usually a programmer) liked you? Job offer on the spot. Sure, you got paid 50-60k / year, but it was what it was. And, you actually got to work with an entire team of people under reasonable deadlines, 40 hours a week, with a real office, with clear objectives and scope, and an actual schedule. You got to actually use your brain and some creativity (not just google the 'best' answer or look on stack overflow or steal some code from github). Projects sometimes failed, almost always because of requirement failures, and sometimes the code was beautiful; most of the time it was average; sometimes the code was really bad.
Today, you have to go through hours and hours of phone screens, technical interviews, team meet and greets, etc. You have to know how to implement a distributed KD-Tree with skip lists in brainfuck. And, then they nitpick the code on style. And, sure, you get paid 90-150k / year now. (Wow!) But, it's such a time consuming process. And, the smart (or motivated) people know how to game the interview anyway (there are lots of resources out there on how to beat the interview process). And, even when you ace the interview it's still often difficult to get a straight answer on whether your hired or not (strangely more so if you're over 40).
They say that software developers are in high demand... I say that people just hate the idea that software developers make a respectable wage.
But, the funny thing is, after working in this field for over 15 years, after working with 100s of people and seeing millions of lines of code, the truth is that the projects are still failing, and the code is just as bad as it ever was. And, projects still fail most often because of bad requirements, poor vision, and bad management. In fact, I drink the koolaid: I hire people who ace the interview, and you know what I've found? There's a 50/50 chance that they're just as good as the person who I hired who failed parts of the interview (sometimes I'm desperate). And, those people don't bring down the project - it's the guy in the corner office.
Luckily, I was smart enough to save a lot of money over the years. I'm just too old for this shit.
82 comments
[ 2.8 ms ] story [ 150 ms ] threadhttps://news.ycombinator.com/item?id=9766816
Which makes sense given that I tried out their online quiz this summer after reading this exact post.
EDIT: Here's a link to when it was posted to LinkedIn, June 23rd, 2015:
https://www.linkedin.com/pulse/three-hundred-programming-int...
To me, it seems like your team isn't that good at judging ability. To compensate, you rely on coding questions. This what the industry has been doing for years. What exactly is innovative here? I see the need for coding questions (and think that your method sounds pretty good) but too often culture fit & 'soft skills' are overlooked in favor of cramming as many technical questions as possible.
One of the best experiences I had on an interview involved no coding on-site at all. I completed a project for them, they liked it, I got an interview. 3 hours of culture/team/work philosophy. There were no surprises - if I wasn't "technical enough", I wouldn't have been let in the door. It took time up front, but at least the interview itself wasn't waste of time, PTO, and money.
The chart's y-axis is labeled "correlation coefficient" but the text blurb analyzing it describes the definition of R^2 instead (% of variation of Y explained by X). The clarification at the end does not disambiguate this. Also, how do you numerically quantify "interview performance"/Y?
Additionally 3 out of 6 of the variables have error bars crossing 0, which means that the correlation could be positive or negative but we don't know (it would handily fail a statistical test of significance at p = 0.05 significance). Why were they included at all?
Many startups (including statistical startups!) have been playing hard and lose with statistical analysis lately and it's getting annoying.
I also would not consider an r-squared value of 0.47 indicative of strong correlation between two variables. I guess the point is that the composite quiz score is more correlated with performance than traditional methods, but I still get the impression that someone is overselling something.
If you are serious about this then you should be interviewing candidates, deciding on whether you think they should get a job. Then giving them a job anyway and evaluating their performance after some time interval and see how accurate your interview process was in terms of weeding out those who would fail.
It seems reasonable to optimize for hiring right now since otherwise nobody would go to them to find a job, and it'd also be impossible to measure job performance when the people you collect data can not get jobs.
They're optimizing for hiring, not job performance.
For instance, if you selected for college graduates for hiring firefighters, and that causes a lot of rejected applications among African American candidates, then it would be illegal based off of an adverse impact against a protected group.
As an aside, suing employers over requiring a college degree is a seriously under-tapped and under-estimating way to charitably improve the lives of Americans (college is a positional good because college matters when employers are looking at candidates - if employers can't look at whether or not you have a college degree unless it's a bona-fide qualification, then many more people don't have to spend several years and tens of thousands of dollars)
http://www.scribd.com/doc/213380090/1999-NYC-Firefighter-Wri...
You'd have a hard time convincing me that doing well on it didn't correlate with being a better firefighter.
I have a big portfolio of technical questions, but I usually stick to a small set that I consider to be well calibrated by posing them to smart coworkers and seeing what they do with them.
I've found, counterintuitively, that the best indicators are the questions that deal with "programming in the small", such as coming up with a predicate expression to describe a test for a simple relationship between a few values, because either you can do it easily or you can't do it at all. And programming in the large depends on a lot of cases of programming in the small.
There is a particular kind of dysfunctional candidate who once was a programmer, graduated to being a lead programmer, and then became an architect. As an architect they stop programming, and their programming becomes rusty. Once that happens, the feasibility of their proposed "architectures" becomes steadily less practical.
You can detect them by their first getting offended that they would be asked to program during an interview, and then failing to actually write working code.
By contrast people who really do program have no objection to being asked to program in an interview. To the contrary being asked to actually program is seen as a positive sign that they won't have co-workers to deal with who can't actually program. (This is an unfortunate experience that most programmers try not to repeat.)
Bashing out code is usually the simplest part in writing a software solution to a problem.
There are more reasons, in short it takes me couple of hours/days to warm up and grasp the problem.
And secondly I refuse to waste my time, on half arsed throw-away code, with zero results.
That's a very wierd thing to say. Does that mean that each and every side-project that you do ends up being substantive? If writing few lines of code gets you a job at a good company where you can spend your time working on exciting problems how is that wasteful?
I do program a lot too, and I typically don't mind writing code in an interview. The only thing that sets me off are stupid level-order binary-tree traversal questions which test nothing but rote-memorization skills.
Also my job is to design database structures, and there is no such thing as simple problem. Even binary tree needs thousands LOC for tests, concurrent access, writes...
But they completely ignore the other 80% of the job which is design, soft skills, estimates, timelines, code reviews etc...
Oh I'm not discounting that. I'm just saying that off the bat binary-tree traversal don't really test my understanding of the data structure. Recently in an interview I was asked a problem which eventually required using a binary-search-tree to improve the running time. This, I believe, is the right way to test but it requires the employer to spend more time designing interview questions rather than relying on a problem bank[0]. (which is a strong filter for good companies in itself)
> But they completely ignore the other 80% of the job which is design, soft skills, estimates, timelines, code reviews etc...
This is true but this is really really hard to test. In the interviews that I've been giving, I've indeed been asked design questions. I assume soft skills are implicitly tested while speaking to the interviewer (to some extent, at least). But other factors, AFAIK are very hard to test. Also, I would argue that a candidate doesn't need to know everything coming in. Estimates, timelines etc can be learnt on the job.
IMO, if somehow employers could find a way to reasonably trust that the code I have on my Github is indeed written by me then I think that should be strong indicator of my competence. Evaluating candidates who don't have any code in the open is a different matter altogether to which I have no solution.
[0] - http://leetcode.com
For every good developer offended by coding in an interview, there are 50 idiots who can't code, who are using that as an excuse to con their way into jobs they're not qualified for. IMO, it's an acceptable trade off.
If you're that awesome you wouldn't be going through the usual interview process, anyway.
Even if you are exactly as you claim, MOST people who claim "I can code but don't care to demonstrate it during an interview" are actual disasters as co-workers. I've been there, done that, cleaned up the messes, and would prefer to pass on some good people to avoid that.
And yes, you do have to pass on some good people. There are competent programmers who freeze up during interviews. You do what you can to make the interview as accommodating as possible, but some people can't show what they've got. However the opportunity cost of passing a good person by is far less than the damage from hiring a bad one. This leads to being risk adverse.
Personally, I've never been a part of a company where the code test isn't:
1) Short (1 hour) 2) On their own time 3) A real problem (algorithms are OK in my mind as long as you help them with the core problem)
I do not expect everyone to know every algorithm... I guess it is kinda weird if you do not know Dijkstra's algorithm, but I only think it is weird because the courses I took focused heavily on it and similar algorithms.
But, do I want you to know all the same stuff I know? No, definitely not. I don't want 2 of me I want me and someone else that works well with me but ultimately has their own ideas.
To me this is clearly non-sense as someone who's followed this path (and know others who have as well). Being an architect allows me to see the big picture and move the organization in the right direction (as we see it), while still being able to get into the trenches with my developers when something arises or help is needed. Granted, this is different for large enterprises but for startups/small/mid sized businesses, yes, us architects still get our hands dirty (or else we're irrelevant).
I said that there is a specific kind of bad candidate to watch out for who generally HAS followed that specific path.
I DID NOT say that ALL people who follow that path will become that kind of bad candidate. Or even most. Just enough that you can't afford to say, "This guy has been programming for 15+ years, is an architect, and sounds smart. I'll skip the programming test."
My understanding of the reason is that a good architect needs to be able to mentally connect every level from abstract architecture down to actual code. If you can't, then you'll eventually absorb ideas that sound good but don't really work in practice. (A random example that I dealt with was someone who didn't understand the latency added by sequential RPC round trips. And that the latency gets worse if you're making the mistake of using XML as your on the wire format...) And the failure of said ideas can always be attributed to programmers having screwed up your brilliant architecture.
For the record, this observation about architects is not original to me. I spent a year at Google, and in interview training we were clearly told that Google does not have a position for a non-coding architect. This is not to say that Google does not have positions for architects. http://research.google.com/pubs/jeff.html certainly wears the architect hat well, and is behind the design of large chunks of fundamental Google infrastructure. See http://static.googleusercontent.com/media/research.google.co... for an example of his showing that skillset off. But they have to code. Which Jeff does. A lot.
No it doesn't. That's not what latency is.
Grabbing a random benchmark, look at http://www.maxondev.com/serialization-performance-comparison.... It is in C#, but every language is going to give similar results. An efficient on the wire format like protocol buffers is a bit faster and a bit smaller than a sane text format like JSON. But XML is several times slower and several times bigger. He didn't test pretty-printing the XML, but that makes it several times slower and bigger still than plain XML.
This adds up sooner than you'd think. For a start there is a noticeable timing difference between an RPC call that fits in one packet and one that has to be split across two.
If you've got an application that spends a sizable portion of its time in serialization/deserialization, it is obvious that the format chosen will affect maximum throughput. (Many distributed applications spend most of their CPU time doing those two things.) It is harder for the format to have a noticeable impact on latency, but it both can and I've seen it happen.
First of all nobody sane would take a job where they are expected to measure up to some unspecified measure or else get fired on the spot. And any competent HR department will tell you that carrying through on the threat will risk the potential for lawsuits. (They want a paper trail showing that the person should be fired.)
Secondly it takes a surprising amount of time for an incompetent programmer at an average organization to get discovered. You usually spend a period of time studying how things are set up before anyone expects you to do anything. And by the time you can be evaluated you've certainly used a bunch of other people's time with your questions. It is far more efficient to determine this in the interview process.
And thirdly, a creative company can more easily solve the problem by offering a choice of ways to demonstrate competence. See http://www.starfighters.io/ for an alternative to traditional interviews.
My thinking is the first x days on the job is spent doing programming tests. This should generate enough paperwork to satisfy any HR department. You then have the time to test all aspects of the candidates skills, rather than the limited range you can test in an interview.
In a lot of the US not only is it not illegal to fire people for incompetence, but you don't even need a reason to fire them. But stupid people when offended sometimes launch silly lawsuits alleging all kinds of crap, and those lawsuits can be expensive to defend. A lot of what HR does is stuff to reduce the frequency of such lawsuits, and to make the ones that still happen go away more cheaply.
The only problem I see with this approach is Dunning-Kruger effects. My experience in hiring is it is pretty easy to pick candidates suffering from DK by comparing what they say with what they can demonstrated via their work history. It is the more marginal candidates where they seem non-delusional, but where they don’t have a perfect CV. These people are often overlooked by other companies, but they often turn out to be fantastic employees. Taking a chance on them can be a great.
If it's the former you should have no difficulty answering your question ;)
Job Offers as a Service
No you haven't. You might have, but you'll have no way of knowing that until they've been on the job at least 3 months.
Personally, I think it's bullshit that I can show employers all the things I've built and they still expect me to write a reverse sort btree or whatever the fuck thing that has nothing to do with web development.
College grads are perfect at passing these tests and they have no fucking idea what they are doing, but will get more offers and better pay every time.
Experience counts and if you disregard it then I'm firmly in the "Fuck you, I don't want to work for you" camp.
Unfortunately, I've resigned myself to memorizing the typical interview questions or going into management. I'd love to stay a dev, but being ruthlessly quizzed about shit that you never use in real life makes changing jobs a pain in the ass.
Aye. It's a hard problem: as an employer, you want to be pretty sure that you're getting someone on team that's going to bring you value.
How do you do that? I guess ask them as much as you can.
You definitely present a good problem, it is the same they're trying to solve over at triplebyte.
> College grads are perfect at passing these tests and they have no fucking idea what they are doing, but will get more offers and better pay every time.
Yeah, you have to study for interviews. It sucks, but almost every software company asks the same questions. Just prepare, I guess, and it'll pay off with that higher salary.
> makes changing jobs a pain in the ass.
You might have nailed it here. Employee retention is good for everyone!
I think the solution is you need connections. When someone works at the company and they know you're a good dev they aren't going to ask you questions they ask other developers.
>You might have nailed it here. Employee retention is good >for everyone!
I'd say having a good work environment is good for employee retention, not arbitrary limitations to prevent people from switching jobs.
Even exp will not help if you are TOO DAMN STUPID TO RESEARCH THE WARNING AND ERRORS the IDE points out to you AND RESOLVE THEM!
As a business analyst interested in a software development career, this is what worries me the most about interviews. I write a fair amount of code. I have developed apps that work really well and solve important business problems. I can talk intelligently about everything I've developed, but if you asked me to write something on the spot I'm not sure how well I'd do. Most of the time I find the information I need through exhaustive searches on the Internet as well as lots of trial and error. Is it really a deal-breaker that I can't write a tree sort algorithm off the top of my head?
Why should they expect that your experience will count for anything if you aren't willing learn a bit about data structures to pass that portion of your interview?
I'm saying a desirable trait is that you can learn new things to solve problems. You know approximately what the problems they will give you are, if you aren't willing or able to go out and learn the necessary skills then why should they hire you?
I never said I don't want to learn new things. Quite the contrary. I stay up to date on the latest web tech and have been frequently contracted for that exact expertise. However, I do find it frustratingly annoying that I need to learn a multitude of "interview puzzles" that I'll never use in real life.
The claims are:
1. Performance on our online programming quiz is a strong predictor of programming interview success
2. Fizz buzz style coding problems are less predictive of ability to do well in a programming interview
3. Interviews where candidates talk about a past programing project are also not very predictive
Those claims may be true, but it misses a key point: no matter how well someone performs in an interview, it doesn't mean they'll perform well on the job. In particular, the post talks about hard skills (and does well there), but doesn't talk about soft skills.
It's absolutely true that someone can talk a big game about hard problems they faced that they solved, but failed to mention their use of Google or StackOverflow. So unique coding tests can definitely tell us whether or not a candidate can really code (though the tests need to be tailored to the sort of real work the candidate will face). However, programming is a hard skill (one that is easy to teach), and not a soft skill (one that you tend to learn over time or have a natural inclination for).
Case in point: early on we made the mistake of hiring a very talented programmer on the basis of his reputation and a brief interview. He was a disaster because he was completely unable to work without a very hard set of requirements. In the very rapidly changing, agile, environment we put him in, he flailed badly because he was good at implementing requirements, but he was very bad at projecting himself into the customer's shoes. Later, we put him on technically demanding tasks which required considerable talent, but had no ambiguity, and he was fine. In fact, he was able to implement tasks that would have been very difficult for other developers who were fine in ambiguous environments.
Soft skills are tricky and most employers have no idea how to interview for them. They require an understanding of the actual role someone will be hired for and the set of skills necessary for them to succeed. Are there tight deadlines? If so, can the developer prioritize tasks and make a case for delaying less critical portions beyond the deadline? Is it a rapidly evolving environment which makes it hard to nail down exact behavior? Are you on a team with strong personalities who often disagree and can the dev handle that? Is the developer required to report to non-technical people who have no understanding of technology and can't appreciate what refactoring is?
Or here's a fun one for you aspiring managers: we've had one project with a superstar developer who was so monumentally productive and so brilliant at his work that some very talented developers who worked with him nonetheless had negative productivity at times because they spent so much time playing "catch up" with the superstar and understanding how the project was evolving that they spent more time keeping abreast of the code than developing it. The code was "bus sensitive" because one brilliant developer was so competent. How do you manage that?
These and other soft skills are part of what you really need to evaluate because depending on your work environment, you'll have these and other soft skills as critical skills that a dev will need, but you almost NEVER see interviewers check for these because very few people are trained in interviews.
Pro tip: start reading up about structured interviews. They've been proven via multiple studies to have a strong correlation with employees being successful. Unfortunately, they're harder to conduct and require training to understand them. As a result, companies don't know about them or worse, ignore them, because it costs more money. But what costs you more money in the long run? Taking longer to interview or hiring the wrong person?
> 1. A fizzbuzz-like programming assignment. Applicants completed two simple problems. We tracked the time to complete each, and manually graded each on correctness and code quality.
Instead, I was presented with a single, rather convoluted, 1-hour long programming problem that was most definitely not anywhere close to the difficulty level of a FizzBuzz test (the solution they're looking for probably involved dynamic programming, which I honestly haven't had much practice with). This was before I even spoke to any humans on the phone screen or interview steps of the process.
I couldn't finish it in 1 hour (I kept going at it on my own time for another 2-3 hours before I finally arrived at a sub-optimal solution, at which point I had enough). I'm sure there are plenty of people who are much better at this than I am who can finish it in an hour. But I do want to question the wisdom of using something of this level of difficulty as a screening test.
Your clients seem to value the "Product Programmer" very highly. Using a question like that as a screener will weed out a significant portion of these "Product Programmers" before they ever get a chance to showcase their product skills. Most "Product Programmers" I know (myself included) don't do enough convoluted dynamic programming problems in their spare time to be able to solve a dynamic programming problem like that within an hour with any level of certainty (and yes, solving problems like that quickly involves either extraordinary talent or lots and lots of practice). Instead, in their spare time, these "Product Programmers" like to build things, and learn new technologies and techniques to help them build things in a simpler, more maintainable, and more productive way.
They'd always heard of everything, and they had a good idea of what everything was, even though it was quite far from the current task to hand. For instance, plenty of people have not had the opportunity to work with functional languages, yet still have a good grasp of why they're useful, what languages there are, who uses them, and so on. They tend to see coding as a vast field where you can make many valid decisions, each with their own pros and cons.
The kind of people I'd root out are the ones who merely see programming as a tool. Something along the lines of "I've got this data in a spreadsheet, I need to calculate this statistic, so here we go..." They almost always end up creating an unmaintainable mess, and they never mention any well known pattern. Quite hard to identify in an interview, though. Perhaps the lack of key words that suggest a deeper understanding?
(Hey, that's an interesting thought. You could machine learn what words good devs use vs less good ones.)
As for coding in the interview, I think this can only root out the absolute novices. It's quite hard to think of a short test that will demonstrate that someone can't write elegant code. Perhaps a take-home, open-ended test would work, but you're gonna lose people who don't want to spend a whole weekend doing it.
Today, you have to go through hours and hours of phone screens, technical interviews, team meet and greets, etc. You have to know how to implement a distributed KD-Tree with skip lists in brainfuck. And, then they nitpick the code on style. And, sure, you get paid 90-150k / year now. (Wow!) But, it's such a time consuming process. And, the smart (or motivated) people know how to game the interview anyway (there are lots of resources out there on how to beat the interview process). And, even when you ace the interview it's still often difficult to get a straight answer on whether your hired or not (strangely more so if you're over 40).
They say that software developers are in high demand... I say that people just hate the idea that software developers make a respectable wage.
But, the funny thing is, after working in this field for over 15 years, after working with 100s of people and seeing millions of lines of code, the truth is that the projects are still failing, and the code is just as bad as it ever was. And, projects still fail most often because of bad requirements, poor vision, and bad management. In fact, I drink the koolaid: I hire people who ace the interview, and you know what I've found? There's a 50/50 chance that they're just as good as the person who I hired who failed parts of the interview (sometimes I'm desperate). And, those people don't bring down the project - it's the guy in the corner office.
Luckily, I was smart enough to save a lot of money over the years. I'm just too old for this shit.