There doesn't appear to be any opportunity for TDD (or any kind of testing or iteration), even for something simple like checking syntax (I submitted one with a missing ';'). As a result, it does not come close enough to a real-world programming task.
Once I had looked around, and failed to find a "run" button, I decided that I'd try just pressing submit, and I got to the "we'll email you" screen. I'm sure I'm not alone. This would account for the very low pass rates.
In the real world, this would be like having all your programmers working with a plain text editor and no compiler, and having them commit untested code, in feature-sized chunks, to trunk for the CI server to build and test.
Obviously, under the right circumstances, I could do the work on my own computer, and paste the code into your form, but that sort of defeats the purpose of having it online.
Following the doctrine that programs are written for humans first, computers second, a human still needs to read the code to work out whether or not it was well written. I don't know if you make this available to screeners.
You could try applying some other metrics (e.g. max/mean function length), to give more than a pass-fail.
- You're right, I think a lot of people expect to be able to check their code first. When I was planning this, I figured if I were the candidate, I'd use my local IDE to compile and test the code first. So this was a bad assumption which I need to fix.
- Screeners do have access to the code a candidate submits. It goes in a mail to them along with the pass/fail rating and is also accessible in their accounts.
- Pass/fail is definitely on the crude side, but I was going for an 80% solution first.
This is a good idea. It needs more features and polish, but I see no good reason why someone would NOT use this for hiring. I think the especially important points will be:
1. Support for all reasonable languages.
2. LOTS of different tests--the more unique, the less likely the solutions can be copied and pasted from either someplace online, or a friend who applied and took the same test.
It would also be awesome if you told the candidate "program in this window because the employer will be able to watch you code". Then show the employer a fast-forward view of the programming process. Does this person guess and check their way to a solution? Do they plan out the whole algorithm first? What kind of mistakes did they make?
If you had all that stuff, I think this would do a reasonable job of at least filtering for raw programming ability.
- On lots of test problems: that's exactly what I intend to do. New test problems will be introduced into the private/paid queue and after they have been delivered enough times, they will be moved to the open/free queue.
- Of a fast-forward movie of the candidate coding: I'm a bit skeptical of this one. I'd really rather let people code on their own computers (in their favourite editors) first. Also, if I were the candidate, it'd annoy me a bit to be judged negatively on, for instance, typing errors and private comments while I'm fleshing the code out. But that's just what I think; if enough people ask for it, I'll add it.
>This is a good idea. It needs more features and polish, but I see no good reason why someone would NOT use this for hiring
Other than, as Joel pointed out a long time ago - the best people wouldn't put up with it because they can always get a job elsewhere.
>It would also be awesome if you told the candidate "program in this window because the employer will be able to watch you code". Then show the employer a fast-forward view of the programming process. Does this person guess and check their way to a solution? Do they plan out the whole algorithm first? What kind of mistakes did they make?
Absolutely not - I program in a real editor, and I won't be judged by some wannabe textbox.
It would depend on the company -- the "best" people would be more willing to write some code if [insert awesome company here] was using the tool, versus a company like Initech.
Would be a plus if eventually the tool could behave more like an IDE.
You might be surprised; I am not exactly the best of the best, but I personally have zero interest in applying to Google because of their ridicules process.
Of the best programmers I know, one owns his own consulting company and the other has been actively recruited through his last several jobs. He literally has not put together a resume in 20 years.
PS: The best example of the gap between these guys and the average programmer: He once released a bug into production and asked the testers what happened. They said, "We stopped testing your code last year. It seemed like a waste of time." They also requested he spent the night of Y2K in AT&T's bunker, just in case.
> the best people wouldn't put up with it because they can always get a job elsewhere.
How are you supposed to know that someone is one of the "best people"? Presumably by word of mouth and recommendations. I'm not saying that all applicants should be forced to use this system--if you know that someone is great, no need to test them. This is designed to filter out all the REST of the applicants, the ones who may be good, but don't have many references (straight out of college, etc)
> Absolutely not - I program in a real editor, and I won't be judged by some wannabe textbox.
Fair enough (I enjoy using a decent editor too), but:
1. These are not supposed to be hundred line solutions. You can live with a crappy editor for 30 minutes.
2. Maybe applicants could submit a video instead if they wanted.
3. It's not strictly necessary, it's just another nice data point.
About your second point, sure I can program is a textbox but I am going to make mistakes that I would not normally do and I don't want a potential employer to see that, because he will subconsciously think that it is bad even though he should know (and properly does) that that sort of things happen.
And at a job interview I am not going to let something like that count against me. Same reason I am going to go there in appropriate clothing, even though it shouldn't make any difference it does.
As an applicant, what do I get out of using your service? Why shouldn't I just skip ahead to the next job posting, if some would be employer demanded that I used your service?
if you are Google this does not apply, but otherwise the first problem you have is to answer "why would I ever work for that company", since most large companies are the same anyway.
Personally, from the other side, I've seen literally hundreds of people apply to jobs with only one or two open positions.
Most of those candidates turn out to be just noise, but the few that are not don't always get a fair chance because screeners/interviewers are just too swamped. A tool like this can take a large chunk of that load off, and any candidate who invests time in taking this kind of test will know that they are ahead of the 'unwashed masses' without having left the comfort of home.
I don't see that applicants are the target audience for this service -- the companies using it would be responsible for making the tests a good experience for candidates.
I really like the idea of this site, but it needs some polish.
1. As everybody has said, more languages.
2. One of the aims is to have the candidate as calm as possible. Seeing that 90% of people can't write a C program that reverses a linked list makes me nervous about the strictness of the test.
3. Rather than giving just a subset of the environment how about you make each test have to produce a binary. Then tell candidates that the test is for `gcc -o reverse reverce.c && ./reverse 1 2 3 4` to output "4 3 2 1". I presume you are running them in a sandboxed VM? That will help people run and debug the program on their own machines.
- With the 'open' tests, encouraging experimentation was the intent. It definitely needs an interactive syntax check as others have suggested.
- Each test does produce a binary, which runs in a sandbox (or rather a few nested sandboxing mechanisms). Your suggestion is interesting but I don't think I fully understand it. For one it might restrict us to solutions expressible as strings, and the formatting requirements might make the task stricter.
I guess I didn't explain myself well enough. Give candidates the entire file and tell them how it will be run. Then it's trivial for them to develop on their local machine.
I'm positive most of the failures you'll get are syntax related. Let people check their own! If you want to check their syntax for them, that is going to be a lot of work and make it harder to add new languages. Include the unit tests with what you give the candidate so you are not just limited to string output if you want, but you could give just them the formatter too. Keep up the good work though, it was actually quite fun doing a couple of the tests :)
>2. One of the aims is to have the candidate as calm as possible. Seeing that 90% of people can't write a C program that reverses a linked list makes me nervous about the strictness of the test.
really? that's about what I'd expect. Consider that the incompetent are over-represented amongst the job hunters, and it doesn't surprise me one bit that 90% of the candidates can't reverse a linked list in C.
Also, nearly all programmers are at least a little bit arrogant about their ability. I think a hard programming test, even a actual hard programming test, would scare fewer candidates than an actual face to face interview.
One of the aims is to have the candidate as calm as possible.
Why? As a potential customer of this product, I don't care how calm the candidate is. They'll have plenty of stress to manage on the job anyway. I just care if they can code. Weed out the posers and send me the rest. We can worry about the touchy/feely stuff later.
Seeing that 90% of people can't write a C program that reverses a linked list makes me nervous about the strictness of the test.
That's OP's main value proposition! A few bucks instead of a few hours to weed out the posers is an excellent selling point. Best wishes, OP.
I like this idea a lot. We routinely screen people with a really simple programming task to just make sure they aren't completely lying (which happens more than I can believe).
1. I took a test, implemented reverseWords (which is a test we often give candidates), and was rather shocked to discover that in your estimation, I failed. Yet, pasting the code into vim, and the unit test you say I failed ... I didn't fail. The fact you failed me when I should have passed, isn't great. :) More info on the failure would be helpful.
2. If I were using this on a real candidate, I would like to know how long it took them to do the test.
I think it's a great idea; I'd consider using it for pre-screening once the kinks are worked out.
Could you paste your code here or email it to me at karan@codeboff.in? I'd love to find out what happened.
In my experience code that runs on OS X often fails on linux, because gcc on OS X seems to initialize newly malloc'd memory to zero, when gcc on linux doesn't (and it shouldn't, according to malloc's man page).
Shouldn't? malloc doesn't make any guarantees that the memory is initialised or not. In the Linux malloc man page it specifically says it is not cleared, on Mac OS X it doesn't specifically mention that it is initialised. On FreeBSD it says it is initialised
Yes, programmers should make sure that memory is indeed cleared, preferably by using calloc or memset.
I just ran a test on my FreeBSD laptop, allocating 256 MB of ram (on a 64 MB physical ram laptop), and then walking through it, and with just malloc I got all 0's. I then had the program write random out to those 256 MB, free it, re-run the program, still no garbage values. Same on Mac OS X. I don't currently have access to a Linux machine so it is not possible for me to run the test there.
So apparently that way may run on Mac OS X and FreeBSD could fail on Linux.
It would be fantastic if the candidate could just click a "run" button at any time, and (provided with some input string), see the output.
Of course this would make the pass rate a lot higher... but, assuming you tracked each iteration, would give the recruiter a chance to look at how the candidate goes about identifying their mistakes.
I will add the interactive syntax feature seeing that everyone wants it.
Now about tracking iterations, from a candidate's POV, I'd be worried about how my work was coming across, at a stage when I might not be ready to show it yet. I think it's respectful of people's privacy to let them show only the final code that they want to show, and not track their every move on the test. If someone did it to me, I'd be unhappy.
You could just take the code snapshot when they choose to run it. Warn them of this, so they only end up showing the recruiter code they think will either work or is suitable for iterative development.
1. I think that it would be more interesting instead of just testing if a applicant can do a specific task, let them try and fix it if they failed. After all, it isn't such a problem in real life if someone doesn't get something right.
2. What about having them refactor some messy code?
3. Exercises where they write tests for untested code.
1. I am going to add an interactive syntax check feature as a lot of people here have recommended. (As a side-benefit, I can skip asking for an email and present the results immediately after the test).
2. Funny enough, this was my original vision for the project, but according to feedback on a blog article I wrote to test the hypothesis, people preferred to write their own code from scratch rather than fix the nearly-working test problem code.
The service looks nice and the idea is definitely a good one. One questions - you guys look very similar to http://codility.com . AFAIK they have been around for a while (see discussion here http://news.ycombinator.com/item?id=1039140), have more programming tasks, support TDD and more languages. It's unclear to me whether you just want to compete in the same space or are you anyhow different from them?
Thanks! Essentially we'd be in the same space. The ways this will differ is:
1) cheaper, one-time pricing, 2) opening up over-played questions to free, public use, 3) less complexity for test-assigners (at least I'd guess so, I haven't really used Codility to assign tests to candidates in real life).
Give it time, it's just getting warmed up. Personally I think this space will (and should) see a lot more competition. Single founder here.
Whoops: I meant pay-per-use pricing, not one-time. So it's for companies with a one-time hiring need and, for instance, individuals looking to screen outsourcing partners.
37 comments
[ 2.8 ms ] story [ 79.7 ms ] threadThere doesn't appear to be any opportunity for TDD (or any kind of testing or iteration), even for something simple like checking syntax (I submitted one with a missing ';'). As a result, it does not come close enough to a real-world programming task.
Once I had looked around, and failed to find a "run" button, I decided that I'd try just pressing submit, and I got to the "we'll email you" screen. I'm sure I'm not alone. This would account for the very low pass rates.
In the real world, this would be like having all your programmers working with a plain text editor and no compiler, and having them commit untested code, in feature-sized chunks, to trunk for the CI server to build and test.
Obviously, under the right circumstances, I could do the work on my own computer, and paste the code into your form, but that sort of defeats the purpose of having it online.
Following the doctrine that programs are written for humans first, computers second, a human still needs to read the code to work out whether or not it was well written. I don't know if you make this available to screeners.
You could try applying some other metrics (e.g. max/mean function length), to give more than a pass-fail.
- You're right, I think a lot of people expect to be able to check their code first. When I was planning this, I figured if I were the candidate, I'd use my local IDE to compile and test the code first. So this was a bad assumption which I need to fix.
- Screeners do have access to the code a candidate submits. It goes in a mail to them along with the pass/fail rating and is also accessible in their accounts.
- Pass/fail is definitely on the crude side, but I was going for an 80% solution first.
1. Support for all reasonable languages.
2. LOTS of different tests--the more unique, the less likely the solutions can be copied and pasted from either someplace online, or a friend who applied and took the same test.
It would also be awesome if you told the candidate "program in this window because the employer will be able to watch you code". Then show the employer a fast-forward view of the programming process. Does this person guess and check their way to a solution? Do they plan out the whole algorithm first? What kind of mistakes did they make?
If you had all that stuff, I think this would do a reasonable job of at least filtering for raw programming ability.
- On lots of test problems: that's exactly what I intend to do. New test problems will be introduced into the private/paid queue and after they have been delivered enough times, they will be moved to the open/free queue.
- Of a fast-forward movie of the candidate coding: I'm a bit skeptical of this one. I'd really rather let people code on their own computers (in their favourite editors) first. Also, if I were the candidate, it'd annoy me a bit to be judged negatively on, for instance, typing errors and private comments while I'm fleshing the code out. But that's just what I think; if enough people ask for it, I'll add it.
Other than, as Joel pointed out a long time ago - the best people wouldn't put up with it because they can always get a job elsewhere.
>It would also be awesome if you told the candidate "program in this window because the employer will be able to watch you code". Then show the employer a fast-forward view of the programming process. Does this person guess and check their way to a solution? Do they plan out the whole algorithm first? What kind of mistakes did they make?
Absolutely not - I program in a real editor, and I won't be judged by some wannabe textbox.
Seriously, you'll lose your better qualified candidates if you force them to use a strange editor.
Would be a plus if eventually the tool could behave more like an IDE.
Of the best programmers I know, one owns his own consulting company and the other has been actively recruited through his last several jobs. He literally has not put together a resume in 20 years.
PS: The best example of the gap between these guys and the average programmer: He once released a bug into production and asked the testers what happened. They said, "We stopped testing your code last year. It seemed like a waste of time." They also requested he spent the night of Y2K in AT&T's bunker, just in case.
How are you supposed to know that someone is one of the "best people"? Presumably by word of mouth and recommendations. I'm not saying that all applicants should be forced to use this system--if you know that someone is great, no need to test them. This is designed to filter out all the REST of the applicants, the ones who may be good, but don't have many references (straight out of college, etc)
> Absolutely not - I program in a real editor, and I won't be judged by some wannabe textbox.
Fair enough (I enjoy using a decent editor too), but: 1. These are not supposed to be hundred line solutions. You can live with a crappy editor for 30 minutes. 2. Maybe applicants could submit a video instead if they wanted. 3. It's not strictly necessary, it's just another nice data point.
And at a job interview I am not going to let something like that count against me. Same reason I am going to go there in appropriate clothing, even though it shouldn't make any difference it does.
Not to mention that you probably would wantto copy the code elsewhere anyway to test it.
if you are Google this does not apply, but otherwise the first problem you have is to answer "why would I ever work for that company", since most large companies are the same anyway.
Most of those candidates turn out to be just noise, but the few that are not don't always get a fair chance because screeners/interviewers are just too swamped. A tool like this can take a large chunk of that load off, and any candidate who invests time in taking this kind of test will know that they are ahead of the 'unwashed masses' without having left the comfort of home.
1. As everybody has said, more languages.
2. One of the aims is to have the candidate as calm as possible. Seeing that 90% of people can't write a C program that reverses a linked list makes me nervous about the strictness of the test.
3. Rather than giving just a subset of the environment how about you make each test have to produce a binary. Then tell candidates that the test is for `gcc -o reverse reverce.c && ./reverse 1 2 3 4` to output "4 3 2 1". I presume you are running them in a sandboxed VM? That will help people run and debug the program on their own machines.
- With the 'open' tests, encouraging experimentation was the intent. It definitely needs an interactive syntax check as others have suggested.
- Each test does produce a binary, which runs in a sandbox (or rather a few nested sandboxing mechanisms). Your suggestion is interesting but I don't think I fully understand it. For one it might restrict us to solutions expressible as strings, and the formatting requirements might make the task stricter.
I'm positive most of the failures you'll get are syntax related. Let people check their own! If you want to check their syntax for them, that is going to be a lot of work and make it harder to add new languages. Include the unit tests with what you give the candidate so you are not just limited to string output if you want, but you could give just them the formatter too. Keep up the good work though, it was actually quite fun doing a couple of the tests :)
really? that's about what I'd expect. Consider that the incompetent are over-represented amongst the job hunters, and it doesn't surprise me one bit that 90% of the candidates can't reverse a linked list in C.
Also, nearly all programmers are at least a little bit arrogant about their ability. I think a hard programming test, even a actual hard programming test, would scare fewer candidates than an actual face to face interview.
Why? As a potential customer of this product, I don't care how calm the candidate is. They'll have plenty of stress to manage on the job anyway. I just care if they can code. Weed out the posers and send me the rest. We can worry about the touchy/feely stuff later.
Seeing that 90% of people can't write a C program that reverses a linked list makes me nervous about the strictness of the test.
That's OP's main value proposition! A few bucks instead of a few hours to weed out the posers is an excellent selling point. Best wishes, OP.
1. I took a test, implemented reverseWords (which is a test we often give candidates), and was rather shocked to discover that in your estimation, I failed. Yet, pasting the code into vim, and the unit test you say I failed ... I didn't fail. The fact you failed me when I should have passed, isn't great. :) More info on the failure would be helpful.
2. If I were using this on a real candidate, I would like to know how long it took them to do the test.
I think it's a great idea; I'd consider using it for pre-screening once the kinks are worked out.
In my experience code that runs on OS X often fails on linux, because gcc on OS X seems to initialize newly malloc'd memory to zero, when gcc on linux doesn't (and it shouldn't, according to malloc's man page).
Thanks!
Yes, programmers should make sure that memory is indeed cleared, preferably by using calloc or memset.
I just ran a test on my FreeBSD laptop, allocating 256 MB of ram (on a 64 MB physical ram laptop), and then walking through it, and with just malloc I got all 0's. I then had the program write random out to those 256 MB, free it, re-run the program, still no garbage values. Same on Mac OS X. I don't currently have access to a Linux machine so it is not possible for me to run the test there.
So apparently that way may run on Mac OS X and FreeBSD could fail on Linux.
Of course this would make the pass rate a lot higher... but, assuming you tracked each iteration, would give the recruiter a chance to look at how the candidate goes about identifying their mistakes.
Now about tracking iterations, from a candidate's POV, I'd be worried about how my work was coming across, at a stage when I might not be ready to show it yet. I think it's respectful of people's privacy to let them show only the final code that they want to show, and not track their every move on the test. If someone did it to me, I'd be unhappy.
2. What about having them refactor some messy code?
3. Exercises where they write tests for untested code.
1. I am going to add an interactive syntax check feature as a lot of people here have recommended. (As a side-benefit, I can skip asking for an email and present the results immediately after the test).
2. Funny enough, this was my original vision for the project, but according to feedback on a blog article I wrote to test the hypothesis, people preferred to write their own code from scratch rather than fix the nearly-working test problem code.
Please see: - http://blog.codeboff.in/2010/07/03/will-the-real-programmers... - http://blog.codeboff.in/2010/07/19/bug-fixing-test-problems-...
3. Might not be important enough to take time up in a test. As I understand, everyone hates long tests.
1) cheaper, one-time pricing, 2) opening up over-played questions to free, public use, 3) less complexity for test-assigners (at least I'd guess so, I haven't really used Codility to assign tests to candidates in real life).
Give it time, it's just getting warmed up. Personally I think this space will (and should) see a lot more competition. Single founder here.