HN Explorer

The Coding Test Delusion

11 points by andrewstuart ↗ HN
The coding test delusion:

Point 1: Developers are terrible at estimating how long it takes to get software projects done. This is a universal truth.

Point 2: Developers issue coding tests to job seekers and are always certain that "We confidently estimate it shouldn't take any good developer very long to complete. It certainly wouldn't take any of our team very long."

Conclusion: Your coding test takes too long but your team don't think so.

Outcome 1: Candidates lost, candidates drop your company from the hiring processes they are pursuing.

Outcome 2: Candidate completes coding test - your development team "felt it took them too long, certainly longer than it would have taken us". Candidate is marked down.

Your team's response: "Well they weren't very good because it's not a hard test, it shouldn't take long."

Guaranteed: your team feels this is not a problem for your company coding test.

What no company ever does: require every current team member to complete the coding test, measure the time and blindly assess each others test result.

13 comments

[ 5.6 ms ] story [ 44.6 ms ] thread
version_five
This is true for most tests written by people inexperienced at making tests. A common mistake made by junior university faculty is to make a test way to long and hard, having forgotten what it's like to be the test taker and making bad assumption like you say about how long it takes. Usually it gets better with experience.
theamk
When I was making interview questions, I always tested then on my teammates first. Not on all of them, that'd take too much time, but on some of them. If there are no time for that, take your estimate and multiply by 5x.

(We do not did not do take-home tests though, those were in-person coding tests).

lomcs
Does it make you feel smarter when you give people tests?
theamk
not really? The tests are fairly basic - there are just not very many tasks that are (1) can be done in <10 minutes (2) are not "aha!" questions and (3) interesting. If you can solve the programming task like this, it is basic proficiency, not something to be smart about.

An analogy: being able to do programming tests is like being able to calculate percentage ratios in the high school. Yes, there are plenty of students who cannot do that - but they are not interested in math. And for the students who are interested in math, the "percentage" problems are so simple it is not even worth mentioning.

Also, a bit of general advice: when you work in software, you don't usually want to be "the smartest person in the group" - this can easily lead to lack of professional development. You want to work with people who are smarter the you are, because you can learn from them.

raihansaputra
This is true. And the other issue is giving out tests that are low signal for skills that are actually needed day to day. I went through an FE test that's 2 algorithm tests (that requires a recursive solution) and a CSS test. I studied CS in university so I know algorithm and complexity is important, but as an FE developer, I really think deep about algo in <5% of my work.

50% of my work is coding, in which 30% knowing CSS layouts and styling (usually abstracted), 50% state and logic management (UI views/components + API, thinking about the suitable abstractions, form logic), 10% interfacing with the browser. The rest of coding (10%) is the algorithmic work + debugging. The other 50% is collaborating with the Product/Design team to explore the solution space and refine requirements (divergent and convergent), working with the BE team to shape the API, and working with QA to replicate and identify issues.

A lot of the algorithmic issues are already handled (browser implementation, framework, libraries), and I think knowing/setting up how to identify the algorithmic bottlenecks is far more important that thinking about algorithms all the time. I do identify more as a product-first dev rather than a code-first dev though.

I do agree it's hard to evaluate a developer holistically, and a company needs to weed out developers that isn't up to their standard, but I still feel the most ideal test is a test project with 2-5 times working time from internal estimate (2-3 hours? give them a week), with a showcase/interview session to demonstrate understanding how the project works. The signal to noise ratio is much better in that sense.

cwdegidio
What I find amusing is having consulting firms ignore my requests specifically for FE developers so they can hand me some Java enterprise coder who knows their DSA because every 6 months they are re-interviewing. And why do I turn away literally all of them? It's not because they failed a code test... I just watch their body language and pay attention to their speech patters when I ask them 1 question... "How is your CSS skills?" It took me until this month to finally convince the people above me to stop wasting my teams time with candidates vetted in their way and let me handle it. I don't have time to teach folks everything they need to know about FE in the life of a 10 month project. At least bring something useful to the table.
raihansaputra
Oh man that must be frustrating. And it's a bit mindboggling as there's basically a steady stream of FE developers that are willing to work, especially for a timeboxed project.
really_nobody
Yeah, tech interviews are so subjective. We have started giving take-home coding exercise to counter some of the similar issues. This way, the candidate gets to work on a relatively practical problem in a non-pressure environment.
spfzero
Dumb idea. You want to watch them work to solve. The insight gained watching is more valuable than whether they get the right answer or not.
kasey_junk
Counterpoint. The day I introduced work samples for the first time I’d done all of them for time. Then we made every existing dev take them.

Then we cut the problems because obviously interviewing is harder.

That is, these are superficial complaints that anyone who has ever thought much about work sample hiring comes to early.

lomcs
Point 3: Developers are not professors and do not have a PhD in computer science. So any test they create is based on nothing.

Leave tests to academics.

schwartzworld
Easy fix, which I have had jobs do, is to add an additional sentence to every coding challenge:

We don't expect you to complete the challenge.

It's not necessary to complete it to see if someone can code, and seeing the 75% done result and having a window to the thought process used, is just as valuable (or more) than a pass-fail evaluation.

chrisBob
I just interviewed at a few companies, and passed the technical screens without completing any of the coding assignments. Some were close, but most of the time they told me up front that they didn't expect any complete solutions, and were using them to see how I would think through and discuss a problem.

Knowing the expected outcome up front makes a big difference in how I approach a problem, so I would usually ask what they are looking for.

That being said, I ended up taking a position as an SE2 where I thought I might end up as a Senior. I am sure getting farther in the coding challenges would have helped.