Tell HN: TechTestNow, an app to help you hire the best coders
We are launching the beta test phrase of our startup, TechTestNow. TechTestNow allows you to run online coding tests to select the best programmers to interview. You can setup a test and give the link or invite them by email. Then the candidates are asked to take a test
http://beta.techtestnow.com/
At the moment, we support 2 languages Java and Ruby. We are adding more questions and languages gradually. And as soon as we have more data, we are going to analyze and fine tune the ranking algorithms.
Here are two example tests,
Java : http://beta.techtestnow.com/en/user/testLogin/secret/TTN4c2479e455a032.22415626
Ruby : http://beta.techtestnow.com/en/user/testLogin/secret/TTN4c247a1e3fc618.66097557
To view the system as the recruiter, login as "sandaru1/hnhackers". (please don't delete/deactive above tests)
Any feedback would be appreciated.
21 comments
[ 0.20 ms ] story [ 65.3 ms ] threadhttp://beta.techtestnow.com/
Java : http://beta.techtestnow.com/en/user/testLogin/secret/TTN4c24...
Ruby : http://beta.techtestnow.com/en/user/testLogin/secret/TTN4c24...
Recruiters cannot take tests - only candidates can. Note that you can easily create a new account at the 'take test' page (please use one of your email addresses).
1. Viewing a playback of how the candidate coded the question.
Even a bad candidate might be able to 'prettify' his final code (or even craftily cut and paste code while solving the problem).
However, it is almost impossible to do so in a playback of the entire coding process.
It is like the difference in complexity between creating a fake photo vs creating a fake video !
2. Adaptive testing
Depending on the performance of the candidate, the questions presented will increase or decrease in difficulty - allowing the system to accurately assess the candidate's skill level (that is the advantage of using several small questions, as opposed to a few large questions).
3. The ability to embed the testing widget both on your website or even on a job posting on a job site - that way candidates never even have to leave your website.
However, number 2 and 3 are nice.
As you said, it is the final code which is more important. But don't you think that it is also important to view the process by which the programmer arrived at the final code ?
An experienced coder will quickly reach a solution. Although he might make several mistakes on the way, he can usually quickly detect and correct them.
Someone less experienced would take longer, make more mistakes and probably take several attempts to correct those mistakes before getting it right.
Even though the final code might be more or less the same, the playback would help you discern who has better coding skills.
Of course, this would mainly be of use when attempting to choose between two closely matched coders.
Regarding copying and pasting - at the moment the rules don't explicitly ban this. The problem with allowing it is that the likelihood that the coder might attempt to cheat (and get away with it) is vastly increased (and cutting and pasting makes the playback feature worthless !).
That is why we went to the trouble of enabling syntax highlighting, compilation and testing within the widget, so that it would be as close to an IDE as possible.
BTW - regarding testing - aside from the 2 default test cases, it is also possible to enter your own testdata in the widget.
For example someone could be great at Hibernate but not really have some a lot of work with pure JPA. Just because they are not familiar with the JPA does not mean that they have not mastered ORM concepts and would be able to do JPA development as good as the rest of us in a week or so.
Which brings me to my next point. I would throw time completely out. With the situation I just framed a senior level developer is going to do what all of us do. Fire up the browser look at the JPA api and implement it. This is what makes a good developer. Someone who can gather code together and create a solution.
On almost any given project there are going to be new API's that developers will have to familiarize themselves with. The ones that can are far more valuable developers, because they disseminate that information back to the rest of the team.
Your test does not account for that and by making it time based you are disadvantaging the very ones that posses that valuable skill. If the answer is right, it's right.
As well you should consider offering the service to recruiter and hiring line managers, not as a competitive test, but rather as a filtering tool. Competitions are regarded in academia, but in the corporate world people just want to know if you can do the job.
I myself do not possess an encyclopedic knowledge, nor will you see my fingers flying over the keyboard.
On the other hand, what I have noticed is that my understanding of the basics of coding, i.e. how to mix and match loops, arrays, conditionals, etc has vastly improved over time.
I frequently forget advanced or language specific topics - and I don't really care, since I can easily re-read about them online and refresh myself.
I have worked as an examiner for computer science university students and also as an interviewer for new hires for companies. What I have noticed is that the best candidates are similar - they have a rock solid grasp of the fundamentals of the language, and just twist and turn that knowledge to acquire the results they need.
You yourself have mentioned this - a good developer with an understanding of ORM can easily adapt to whichever framework he is required to use.
This mastery of the basics is what we are trying to assess with TechTestNow. You might notice that none of the questions require in-depth knowledge of the APIs (although that knowledge might help you solve some of them faster, there is always a solution that uses only basic constructs).
The main reason the tests are timed is because, all said and done, a good programmer does need to have a certain coding speed !
Also, in real life, a lot of time is spent not on figuring out how to solve a problem, but rather on figuring out what the problem is in the first place - what do we really need to do ? (and can it be done at all ?). Although the problems given in TechTestNow might seem contrived, they do have the advantage of being unambiguous, so that the programmer can just concentrate on writing a solution.
Note: This isn't a "wah, I did it all right and it said I was wrong!", I specifically used correct, valid answers that were unlikely to match the model you were looking for (empty for loop body for decrementing a variable to a certain value, etc.).
For example, I'll take the following question :
'Given the string variable 's' and the integer 'n', return a new string consisting of n repetitions of s.
For example, if s = "abc" and n = 3, you should return "abcabcabc".'
The code you have written is as follows :
String duplicate(String s, int n) {
}If we analyze the following code (assuming that s = "abc" and n = 3), initially, the variable 'result' is assigned the value "abc".
The loop then runs 3 times - when n = 3, when n = 2 and when n = 1. Which effectively means that "abc" is concatenated 3 more times giving the string "abcabcabcabc", which is incorrect.
I do agree that the solution is almost right. Note that you are allowed to compile your code and run it against our testdata (and even your own test cases) in order to establish it's correctness - which means that you did have the opportunity to find out that there was a bug in your code and fix it.
But in computing, how often is it acceptable to get things 'almost' right ? Would you be happy if Google returned search results which were almost right ?
I know that if a human conducted the interview, he would have helped guide you through the test - in fact, we are almost done adding the capability for recruiters to monitor test sessions in real time and converse with candidates.
I can also provide a case by case analysis of your other solutions if you wish.
Others indeed do have some errors as I quickly typed them up (off by 1, forgetting that java array length isn't the same as C++ std::vector.size). Number 3 could definitely use a better example, such as a = 12 (so the result would be {12, 11, 10}.
I admit, I checked number 2 closely, saw a horribly invalid failure reported and only glanced over the rest. Independent of the other failures, you cannot have incorrectly graded questions, and the rest of the points still stand, the requester has nothing to compare against, leaving the judgement up to your pass/fail (which is clearly not 100%) and the random assignment of questions to different test takers results in no real comparison ability between skills.
The reason different test takers get (somewhat) different question sets is because our algorithm alters the difficulty of the questions based on the test taker's performance. I.e. if you did the first few questions well, you'd start getting progressively tougher questions. On the other hand, if you do badly, you'll be given successively easier questions.