Ask HN: A coding test or working for free?
The task was a data ETL problem transforming natural language text into specific parts of data. I opted to use pyparse to create a working recursive descent parser to pick out the relevant bits of text.
At this point I was wary of completing the parser for all cases found in the database table, so I decided to build it only for a few base cases which I threw in to a unit test. I reasoned I could still demonstrate my skills without giving them much free work.
The solution worked well, and I submitted it to Joe, the CTO, for review. A few hours later Joe contacted me complaining that the code crapped out after trying to run the code against their database table. He then offered me a github branch to continue work on the problem. I sent them the nicest e-mail I could muster explaining that I felt the code adequately demonstrated my skills and that the error is due to the parser not handling all cases of the problem text.
Joe's responded with "I am only trying to evaluate your work, not on just how the code looks, which is clean and well organized, but that it works correctly on the data as well. I asked advice since the program crashed on the 5th package, which didn't give me enough data to verify, and I wasn't sure if it was just a system-related issue or something quickly fixed."
Lastly I responded with "Sorry for any confusion, I was just giving you a sample of my work. The system error you mentioned is a result of the code being a sample. The cases that do work are found in the test."
Naturally they haven't gotten back to me, even though my code more than adequately met the challenge. Perhaps Joe the CTO did not understand that the other cases would be trivial to implement. In that case, perhaps I've dodged a bullet. A CTO should know better.
I'm wondering if anyone else has had similar experiences. How did you handle it? How did the interviewers respond?
91 comments
[ 2.9 ms ] story [ 177 ms ] threadRegardless, my hunch is that your instincts are right about this one.
It took several hours. After an hour of examination of the data and research into the problem at their office I opted to make it a 'take home' test. They agreed and I worked on it the next day (hence the e-mail conversation).
I still think that is the best way to interview (after a regular interview to weed out the real idiots).
I think the real thing that's suspect, is simply having to send in your code, rather than presenting your solution to a group of programmers.
Now if someone gives me a pre-interview coding test, I mentally translate it to the employer saying "I don't respect your time."
If the company asks for a coding test, I politely decline, but I offer to provide them with a code sample instead. Very rarely has that not resulted in a phone interview and I didn't have to spend hours of my time on it[0].
[0]: per company. Write once, give to many scales better than coding tests.
If all he did was ask you to "do this" at their office without picking your brains as to your thought process, then regardless of the legitimacy of the offer, I would think twice about working there.
The fact that he's asking for a solution that works on all of the data rather than a sample of it is also highly suspect. Either he is not being frank with you, or he doesn't have a firm grasp of scope (most likely the latter). Either way, he's probably not the sort of person you'd want to work for.
That conversation was there. After we spoke he left me to work.
If so, I'd say they were clearly in the (legal) wrong even before asking for the refined version.
Pay them.
So if the company wants to have you write code to prove your mettle, have them pay you for the time.
That's the case here. One employer could severely abuse paying candidates for some quick job without any contract being even mentioned between the 2.
In most places it is standard boilerplate that your employer gets approval rights on any outside work you do while employed by them. Current employees also have more restricted IP agreements typically which can put the contracting company in legal trouble as well.
For this reason, I've found paid real work projects problematic in that it biases against the currently employed.
I had a similar experience with a company in Berlin, Germany. They didn't care about the code or my skills. Only if the submitted solution worked against their existing tests.
I think it's just a sleazy way of getting things built for free. Basically, write only a test for the functionality you want. And give the actual task to job candidates. If someone succeeds, you have both the algorithm and its test for pennies. It's based on P = NP. Testing the result is easy. Designing a solution is the hard part.
I now refuse both technical interviews and take at home tests. Limits my choices a lot. But the ones that remain are usually top shelf.
I just ask because it seems like trying to get work done for free through code exercises is much more trouble than it's worth.
An ideal interview is one where I get to learn as much about who I will be working with and what I will be doing as they will be learning about me. Generally these turn into more discussion based interviews where we talk about the issues the company is having and how I might be able to best solve them. They still get a good idea about how I think and work by discussing the problem back and forth. I also get to see how they think and work.
Frankly, I white board problems very well. But when companies ask me to do so in an interview I want to see them do it as well. I want to see their ability to code as much as they want to see mine. Same goes for a take home test. If we are going to work together and you have to see code of mine, I very much want to see yours to make sure it is up to my standards.
From their end, if that was a thing they regularly did, it would be such a hassle. After all, you'd have to find a new person to do that work every time, find a way to split up the work into very small chunks, and who knows if the result from a person off the street would be usable?
My guess would be that he was trying to evaluate how you respond to bug fixes and criticism. (Still sounds like a weird interview strategy though)
The feeling I got was that they weren't developers by trade (the code quality I mentioned was a red flag), very small, early-stage startup with no money and they needed expert help, fast.
How much do you think a consultant would charge to come in for a few hours and discuss their needs, not to mention put some working code together for their problem?
I still paid it with my time.
Good luck.
On the other hand the interviewing company should avoid such situations too because of the information secrecy - unless they are mandating an NDA with each and every candidate.
Coding tests unfortunately aren't just about demonstrating skill. Then people like me could just link to their github QED. It's also about completing assignments and doing stuff because you are being instructed to.
If he's not comfortable doing that because it's restricted intellectual property, I might wonder what the real purpose of the test was.
I've been given programming tests before, but they were always for hypothetical problems that were open-ended. That way, each candidate solved the problem in a different way which was when discussed during the interview process. In your case, it sounds like they wanted you to solve a specific real world problem they are dealing with.
I have to be a bit cautious here, because our company does something similar.
We have a little programming challenge that we ask candidates to submit. We don't use those submissions in production. We find it gives people with non-traditional backgrounds a chance to show their chops, and gives us some meat to discuss during an interview.
The challenge probably takes 4-8 hours, though. Part of me does think it's a little asymmetric for us as employers to be demanding this of candidates. We've also done week-long work tests, where we pay the person market rate to work with us on real-life code.
Anecdotally, people view the paid week-long trial as demeaning, but the (unpaid) challenge as exciting and fun. We live in a bizarre world.
One big downside of trials is they exclude people who are already employed...ah well, can't win 'em all.
That's because it is demeaning. It represents a large chunk of someone's life invested in proving that they're good enough for you. It's also inherently asymmetric in that the company you work for could not possibly be economically harmed by a week-long trial period, but a single person could be financially devastated by it. Additionally, if the person discovers an illness during that period, do you provide health insurance? What about employees coming from other employers? Do they have to quit their prior jobs or use up vacation time to complete your "interview" process?
A four or eight hour programming exercise skirts some really stupid aspects of interviews and the hiring process. Of course that's welcome. A week-long commitment to work is a sizable tenure as a second-class citizen undergoing prolonged scrutiny.
Your company should be deeply ashamed of this practice.
If you're on the fence about us, give it a shot. If we're on the fence about you, your option is no offer or a week-long trial. As I identified elsewhere, a drawback of the trial is it's mostly only useful for people who are unemployed.
> Additionally, if the person discovers an illness during that period, do you provide health insurance?
We live in a country with reasonable health care coverage, so this isn't really a thing.
> A week-long commitment to work is a sizable tenure as a second-class citizen undergoing prolonged scrutiny.
This is my knee-jerk opinion, too. However, what is the alternative? We could hire you full-time right out of the gate...but we could still fire you after a week.
> Your company should be deeply ashamed of this practice.
Deeply ashamed might be overstating it. Contemplative and reflective, perhaps.
If I'm looking for a job so desperately that I'd be financially devastated by a lost week, then I'd surely jump at a paid week of work, no?
That being said, for someone who is currently unemployed, putting job interviews on hold for a week is less of a risk
Personally I find the single best question is "what's your github username?". People who are on it and active are... well, actually I've only ever had one candidate who's actually been on github and she seemed OK. The majority of people I've seen haven't been much good (and have barely heard of github).
I agree about dodging a bullet. I think the OP did exactly the right thing.
I don't feel too bad about ditching Company A, because they have contacted me through recruiters a few times. It's always the same story. I complete their programming challenge (which is really easy, because it's always the same), submit it, then never hear anything back, not even an acknowledgement that they received my solution.
I ask this because I find many of the software engineers I follow on Twitter who work at major tech companies (github, twitter, google, etc) have a pretty sparse github profile.
Just kidding, the interview itself is very short.
The problem with that is you aren't the only one doing this and when I am job searching I am applying to many companies. So now I have 10 different companies asking for an 8 hour code test to be completed within a couple business days. Meanwhile I still have a job to hold down. It's a massive time commitment to make to just one company without knowing if the work put in will result in an offer.
I much rather prefer companies that aren't lazy interviewing and will look over some of my github repos and discuss those with me to gauge my experience. There is no one size fits all answer to interviewing, but 4-8 hour code tests(plus interviews) just isn't one I can get on board with.
But I do see how having a challenge means our funnel gets narrower at the top because we're competing for people's time. We think we're worth it. :)
At the same time, we're always wondering if our process is excluding too many people. If someone came to us and articulated: "I think your challenge is trying to assess me for X, Y and Z; here's a GitHub profile that shows I have those traits" we'd likely be open to being convinced.
1. The employer is just sending the coding test as a form letter to everyone who submits a resume. I can submit the test, ace it, THEN they read my resume and decide I'm a bad fit. So I wasted my time.
I've done several coding tests where I know I aced it, and still no interview. Why am I wasting my time on this?
2. I do the test, get an interview, and now they decide I'm a "bad cultural fit".
If you're going to reject me for a flaky reason like "bad cultural fit", then you should do it BEFORE you make me waste time on a pre-interview screening test.
But as you said, the test is a useful way to filter out bad employers. If they ask me for a coding test, I don't do it, knowing I'm not missing much.
Suppose there are 5 people willing to proceed directly to a phone interview and 20 people who want me to take a 1-8 hour coding test first. As a candidate, who am I going to spend my limited job search time on?
But it doesn't only happen in this industry: my niece is a graphic designer & has been asked to provide samples of her work on projects ... to the interviewer's specification. She's done it, because she really needed the work, but felt terribly taken advantage of, and I'd agree: it's predatory.
I have the luxury of telling people that, no, I won't take their 1) IQ test, 2) Personality Test, or 3) their Coding Challenges. Not everybody does, certainly.
The appropriate thing to do is to either state up front that you'd need to be compensated for your time, or acknowledge that you're giving away work as a way to possibly get in the door & just write the best solution you can write - not leaving it incomplete, necessarily, either, as that is a sort-of halfway position that makes you look bad.
Also, that HR person is not a trained doctor. They have no business administering any kind of test of this type, and certainly have no business trying to interpret the results. It is just woo nonsense. Might as well try to do handwriting analysis, or read my tea leaves.
You are not allowed (in the states) to ask interview questions about health. Yet, somehow, the courts have found these 'personality' tests legal. I don't understand, but I will not participate. I don't take drugs, but I won't take a pee test either.*
* these results might vary depending on how close I am to foreclosure and how low my saving are. Principles are costly.
The candidate can demonstrate their work and be assured that neither does the company extort free labour from them nor is the work they do for the bin, whereas the companies can see how they behave in real-world environments.
(Actually, would you mind if I stole this idea?)
Principles and self respect have real costs. The milieu right now is a race to the bottom, and refusing to be a part of it might get you left behind.
This is further evidence of the need for unions to make a comeback.
Self respect can't feed you, but it's not so bad.
I agree with this 100% and do the same thing.
If you need something that measures my ability, then go to my github page and go over the copious amounts of projects I have there and determine if my skill level is what you need for your company.
I had one interview where I had sent over a link to my github page. As soon as I sat down, I was given a large packet (it was about 20 pages) and asked to look at the code and figure out the bugs. Most of these were in Java and .Net. I reiterated I didn't work with either language and they told me to "just look at the examples and give it a try." I asked them if they had looked at my github page and reminded them I'm a front end developer. After they said, "Uhhh no, we didn't look at your examples." I stood up and told them, "Thank you gentlemen, but this interview is over. Thanks for wasting my time." and walked out.
I've also turned down "code challenges" that would eat up a week or so of my time. I've got a myriad of recruiters calling me for gigs, why would I waste a week working for free on your "code challenge"? I guess I'm at a point in my career where my work can speak for itself. I don't need to jump through a bunch of hoops to prove my abilities anymore.
My interview for a freelance gig was wrapping up, and the COO asked if I would be willing to "fix a few bugs in the system" to see if I were a "good fit" for the project. I asked point blank if they were asking me to do work for free. He responded that no, he was not, and that the task would take no more than 30 minutes to an hour. I said that I would be happy to take a programming test, or provide some other proof of competency, but that it's pretty non-standard to ask me to fix bugs in a system for free, especially for a limited hour freelance gig.
Up until that point, we were discussing project time lines, handing over of logins, and my number of hours of availability. I got an email the next day saying that they were going to go another direction. Likely, trying to fund some other suckers to fix their bugs for free.
I'd ask to be paid for the amount of work that I'd be doing on this project or the option to get my skills assessed on some other, non-commercial test project that they wouldn't benefit from.
He got very angry when I offered to do it as a paid consulting project. If I did it for free, he wouldn't need to hire me at all!
I asked if they had a problem I could do that would be on more familiar tooling, noting that even if I could borrow an OS X machine, it would take me a while to get familiar enough to complete the task they sent me. I also expressed concern at the length. I ended up declining to participate and they moved on.
(Note: I have never used OS X besides briefly using someone else's computer. Everything I have used has been Windows and Linux, with a couple of weekends of trying some BSD-based distros thrown in.
I would view this as a blessing though. For me, interviews are a chance to evaluate how well we can communicate. If I can't be relaxed, make jokes and just feel free to talk, then I won't feel that way working for you during the next year, and that to me, is a no go.
I'd say indirectly you dodged a bullet, sounds like you two might be a high risk for constant miscommunication.