Ask HN: Recommended online coding test providers for filtering grads?

I have been issuing tests to candidates for a long time now and I have used a plethora of methods to do it. What has worked the best for me was a home rolled solution of having a server they must ssh into and then having a few bash scripts that.

    Set up the user account and a default home directory.
    Runs a job out of cron to check for the user and log them out after a time period.
    Clean up after the test has been reviewed. 

I have iterated on that base idea and ended up with a pretty nice intranet portal and self service test administration system, but then I threw it out the window and started using HackerRank. Its just easier and it does all the stuff I want it to do. There are some limitations, but if you just looking to do an initial screen they work.

I would like more companies to let the candidate submit any project that they've completed in the past 6 months, or give them a small open-ended project and let them choose how they want to build it. It gives the candidate a chance to showcase what they are capable of, and it allows you to judge their programming style.

The question I have is how can you be sure you're testing for something meaningful? By which I mean that testing new college grads on knowledge of Ruby is going to favor people with experience in Ruby and not necessarily those with a sound grasp of data structures and in the long run. Now if all you care about is the ability of someone to code up Ruby at a beginner level, then it's a good test, and in three years you may have someone who still codes Ruby at a beginner level. But if you want someone who can contribute at a higher level over the long term, then maybe taking an advanced data structures elective over a Rails elective is more important than the equivalent of a few weeks of on the job training.

Going further, a test to filter out candidates is a social act not a technical one. Some people, and perhaps a higher percentage of strong candidates who have more options, may forgo the test, or if it is easier than another company's think less of your company. Unless you're Google class as a resume builder, your pool of candidates will probably drop simply as a result of increased friction in the application process.

Sure you may get the desired effect of reducing the number of resumes on your desk, but you could do that by simply flipping a coin for each one and ignoring any that come up tails. And as crazy as that sounds at least then you would know that you were not filtering in a way that was having counter-productive results.

Good luck.

So you looking for an online service to do your job of filtering who you hire ? Don't be surprised if you don't get what you think you want.

If you're a dev company in any sense, it should be pretty straight forward to setup some kind of mock project that potentials can checkout and do something with. Investigate the results.

If you want to know what they are to work with and what skills they have, then work them...and test some skills you know are needed for the job.

Earlier this year I was in the Philippines recruiting for a development office there. Although we weren't targeting grad programmers, the market dynamics we faced were likely similar. There are a whole bunch of substandard programmers that work there, and so using an online test as an initial filter proved very useful; there was no other way we could deal with the volume.

A local company we partnered with to help us recruit suggested https://codility.com/ which we ended up using.

We choose a pair of medium level challenges for the candidates. What we discovered as we went was that their score on the task was not the most important information we got from their test. How they wrote their code was. Looking at how they break up their solutions, and even things such as choices of - or consistency with - syntax was very informative and a good predictor of quality.

My advice would be to not set the bar too high for this kind of test, use it as a filter that eliminates the candidates you definitely don't want to interview, then proceed with the normal interview process.