Very nice product. Much-needed quantification in tech recruiting. Maybe consider marketing it to the stackoverflow people. They have been running a job board with much fanfare since recently. (94 points for me in 7 mins, yay)
If I have an account and am taking a test that I've created, you should provide a link back to the dashboard after I've submitted so I can check how I did. Currently, I'm taken to the the "enter a password" screen and have to manually go back to the homepage and then to the dashboard.
one thing i missed, though, was the ability to write and run my own tests. perhaps a "scratchpad" that evaluates code and returns a result would be a nice addition?
Sorry, some bug on our end - it should be there. We are fixing it right now. In the meantim - if you sign up for a trial account [http://codility.com/accounts/codility-register/] it works fine there (a blue "add a test case" button in the bottom left).
thanks. another smaller point i just remembered - i was a bit surprised to see that the evaluator could see my submissions to the "validate" function. given that you do say they won't influence the score i assumed that they would be discarded.
i can see there are several reasons why this is useful to the evaluator, and i can also see that you're getting close to saturating the user with instructions, but in a perfect world it would be good to have "full disclosure" (i guess you do, in that anyone can run the example....)
Very good idea, but I don't program that way. I use a competent IDE to tell me syntax errors and obvious programming mistakes while I write. If I can't debug my algorithm with trial & error I write a test first and modify my algorithm until it complies with the test.
To think that a programmer will always come up with the exact and most optimal solution to any given problem is a bit naïve. In my 10 years as a programmer I have never seen one wonder-kid that is able to do that. The real wonder kid assets in a team also tend to be those that are more creative than syntax-anal compared to those that can write syntax-error-free-code with pen and pencil and remember large parts of a phonebook...
Well, that's one more reason you should like Codility - you can run test compilations (the VERIFY button) which will show you syntax errors. And in bottom left corner you can apply your test data sets. In other words - you work like in your every day environment. Check it out and let me know what you think.
I did, and now I see comments stating that you should be able to run tests on your code while developing it. That's awesome and what I was missing.
Next step intellisense and on-the-fly syntax checking. I hope no programmers click the "compile"-button nowadays to check if their code is correct...
To think that a programmer will always come up with the exact and most optimal solution to any given problem is a bit naïve
I have no idea where you saw this being implied. The software highlights syntax and lets you check your code as many times as desired, as well as giving hints if your result isn't correct and providing compiler output. Nothing stops you from googling something. It's not like the software will destroy your file and egg your house because your algorithm is o(n2) instead of o(n); that's up to whoever is evaluating the results.
The fact that it has a polish and chinese version should also give you a little hint about the intended audience.
Yeah, that demo problem would probably take me at least several hours to solve. I'm bad with sequences and I have a volatile short term memory which means I make mistakes and am slow to process numbers and logic. Interestingly these flaws cause me abhor any over complex logic or code where you have to remember more than 4 or 5 things at once to use properly, which makes my code clean. So maybe I'm not a good programmer (I can't write the complex stuff very easily) but the code I write is usually good. Try testing for that Codility!
I'm not sure the timer is a good idea. For one, I found the seconds counting down very distracting. It also encourages quick and inefficient solutions that are easy to hack out, over more efficient algorithms that may take longer to code.
Maybe you should time the programs, instead of the programmers.
We are timing the programs as well - with the scalability assessment. Recruiter can change the timing anyway, but we feel that a bit of pressure is good. You don't always have comfy conditions on the job after all.
Right, but maybe not down to the second. I personally didn't mind, but counting seconds causes a distracting flickering. Maybe you should consider activating it on the last minute only ?
I also found the clock annoying. If someone does this for an interview they're likely to feel enough pressure anyways. I'd let the test taker hide it and maybe redisplay it automatically with 5 minutes to go.
As for comfy conditions on the job - I would hope that default conditions are comfy. When interviewing I'm more interested in how people will perform day-to-day rather than in rare situations. Most programming jobs don't really require employees to be "cool under pressure."
Other than that, good job on the execution so far. Reminds me a lot of TopCoder, just applied to a specific niche in a good way. I would even consider using this or recommending it to colleagues for hiring.
Is the code editor they're using something publicly available? I'm looking around for a light in-browser code editor with programmer-oriented features (e.g. syntax highlighting, this also seems to auto-indent Python code to some degree).
Even I noticed that I had seen something ditto somewhere... Hmm so Codility's editor is copy of the one SPOJ uses, so which one do they both use? Can I too get it :)
You know, now that I think about it...The description of the test was pretty incoherent. If I don't know what my test data is, and if the problem description is incoherent, I can't really make it work.
It's also taking a metric fuckton of time to evaluate my solution. Which apparently fails. For reasons I can't debug because I can't see the test data.
Any company that relies on code snippets, incomplete problem statements and undefined test data to judge coders will get what they deserve.
maybe there's more than one task, but the task i took appeared to be completely specified (the one where you find a balance point - i may have remembered the name wrong - for an array, so that the sum of elements to either side is equal).
it wasn't obvious to me what to do on first read, but when i paid a little more attention, everything was there (including edge cases, as far as i could see). in particular, they gave a formal numerical expression that was unambiguous (and which didn't seem to contradict the text).
The spec was clear, but the fact that the test dataset used for the test was the very one given as an example wasn't. plus, it was a bit frustrating not being able to fiddle with the test dataset. I would have done it locally with my compiler, but the necessary boilerplate would have lost me time. So, I didn't test enough, and made a silly error which went undetected because of the trailing zero. I still got 75%, but that looks quite miraculous.
EDIT: It appears I can customize my test dataset, and I didn't see it. Silly me.
Yes, it is even possible to add your own data sets, Codility will tell you what you code returns for them. I guess we should make it more apparent in the interface.
Generally we don't want to penalize people too badly for assuming particular (but realistic) type ranges, nevertheless we think that perfect solution should be as portable as possible across different implementations of the language (e.g. one should not assume that C's int is 32 bits wide, since implementation with 16-bit int can still be Standard compliant)
That is an amazing piece of software. I hope you make a mint.
I'd give people access to their language's output facility for playing around to the solution, but that is a minor nitpick. (And I suppose I could open an irb instance on my machine for exploration and then give you code when I'm good and ready.)
My Ruby solution, in case we've got anybody who is struggling: http://pastie.org/771438 This would be my first cut if you asked me to whiteboard this.
The site's very slick. My only feedback is the histogram on the results page could be more prominent - it's cool, useful information but I ended up inspecting the html elements to work out what it was.
Also the results page seemed to keep refreshing for me while waiting for results (Safari 3). Surely that should've been AJAX'd?
def equi ( A ):
right = sum(A)
left = 0
for i,x in enumerate(A):
right -= x
if right == left:
return i
left += x
return -1
What's interesting is that all the versions I've seen so far are pretty much identical apart from very minor things. This would seem to me to be obviously the correct way to do it but I'm interested if it's just a way. Did anyone solve this in a different way of equal or greater elegance?
Mine kept one cumulative list adding left to right and another summing right to left. This is conventionally slower, but conceivably faster if you do it in parallel, leaving the final loop to be something like:
for i in xrange(len(A)):
if right[i] == left[i]:
return i
The final loop here is slightly tighter, so as long as your parallelism didn't add big overheads, this will be faster.
+1! Thanks for the reminder. I'm fairly new to python and couldn't remember the keyword. I guess this is a reasonable argument for the time deadline - to get it done quickly I didn't look it up, instead doing the simplest thing I knew would work and thus exposing the pitiful state of my python experience :-)
Edit: And won't this be a problem if interviewees can access the internet? The interviewers will have to make sure they can check the browsing history of the computer being used!
I don't mean the interviewers should shut off internet access entirely, but what I mean is that if they do allow access to the internet, they'll need to check the interviewee hasn't just googled for the solution to their codibility test (and they'll also have to make sure the interviewee can't delete their browsing history too).
For C++ it would be nice to know what compiler is being checked against. At least in the demo question I would use a 64 bit int to hold the sum if the arrays are really "large" but MSVC names the type __int64 vs G++ using long long and other such minor details. Same for the other languages aka python 2.x or 3.x etc.
We want to avoid sticking to particular compilers/interpreters. An expert in MSVC should not be penalized only because we use GCC. Also we design the tasks so that they are compiler-agnostic (actually they are even language-agnostic). It is sometimes a bit tricky due to non-standardized type sizes (e.g. C, C++, PHP), but generally the tasks can be solved using very limited and portable subset of a language.
This is quite nice, even if the exercise isn't all that difficult.
It would be nice if the boundary conditions were better specified (people seem to have been bitten by overflowing integers). This could be inferred if you told people that you're using a 32-bit machine (the sum of elements is at most SIZE_MAX * INT_MAX; if int and size_t are 32-bit, SIZE_MAX * INT_MAX < INT64_MAX.)
I was also somewhat surprised to see that my code compiles without #include <stdint.h>. I'm not sure this is correct (it could be solved by putting the code written by the candidate before the test code).
Finally, I assume you have proper sandboxing set up?
It gave me a test concerning finding the "equilibrium index" of a sequence. It said to assume that the sequence is very long. (The question is repeated here: http://forums.sun.com/thread.jspa?threadID=676596&start=...) This problem seems underspecified, because it doesn't state whether you can iterate repeatedly over the sequence in question. If you can, it's easy. If you can't I'm not sure how to do it.
Does that ever seem like a good idea? The description suggests it may be a long list. That should give you a hint that you don't need to do this to solve the problem.
The best case possible is to iterate the list once - you sum the list from left to right holding cumulative answers and if the final sum is zero, that index is a valid answer.
The worst case of the best algorithm is to iterate twice - firstly as above, and then again in the opposite direction, comparing values from the first iteration.
I assumed that "sequence is very long" meant "the time (and space?) complexity of your solution matters". Iterating over the sequence x times doesn't change the time complexity, so long as x << n (i.e. it's O(n) whether x is 1, 2 or 3).
Unfortunately, revealing whether my assumption proved correct would give away how they assess the solution to this demo problem, which some might view as a spoiler.
EDIT: since people are posting actual solutions, I suppose mere analysis of the testing methodology is okay. My assumption was correct.
As a student this is very interesting since it could also be used to measure self progress. There's a myriad of ways to test your abilities in most subjects, but if I want to find out if I'm any good at programming or to measure my progress, there are very few easy ways to do that. Are you going to sell this to just business's?
Codility tech team is from Warsaw University, we are more than familiar with TopCoder, ACM CPC, IOI etc. We deliberately keep Codility tasks at fundamental level. Codility is not a competition for 100 world's best programmers, it is a tool to tell decent programmers from poor ones. We can raise the bar arbitrarily high, if there is a customer who needs it.
Might need some way of making explicit which libraries are available?
eg. if the programmer goes off and writes a multithreaded solution, they might only realise too late that pthread isn't included. Admittedly they should have checked first using verify, but you never know with interview pressure.
Unless it is stated explicitly, we do not provide anything extra but the vanilla programming language and standard libraries. It is more clear for some languages (e.g. Java, C++) than other (e.g. PHP, JavaScript), but we try to stick to the reasonable canonical set of libraries.
Seems like a cool idea. If they were smart they'd not only target companies looking for people but people looking for jobs. e.g. Adding a "Allow the codility community to take this test" check box. Then you have the a whole world full of people taking your test.
Actually it's there - if you click on "Take a free demo test" (http://codility.com/demo/take-sample-test/) on the Codility page there is a free test and you will be given the option to tweet your result at the end.
Codility people, if you're reading, here's a bug report.
The test window's layout is fubar if you use a small window
size. I was using 800x600 and the language selection buttons were hidden behind the source code text box. Something (couldn't tell what) overlapped the first two lines of the instructions.
98 comments
[ 2.6 ms ] story [ 141 ms ] threadI took a demo quiz and it worked fine. Great idea for a product.
However I think I'm too competent for the test because I was able to run something I probably shouldn't have been able too.
best Marcin Codility
You might want to lock down functions like glob() and readfile() in PHP -- they effectively give the coder read access to your file system.
one thing i missed, though, was the ability to write and run my own tests. perhaps a "scratchpad" that evaluates code and returns a result would be a nice addition?
My bottom left corner is empty. Except when I click "verify" it shows message whether the tests passed or failed.
But I don't see the tests themselves, nor did I notice a way of writing my own. I've tried it in Firefox, IE 8 and latest dev Chrome.
So, is it possible to write my own tests when I'm taking the test, or only when I'm creating a new one?
Anyway, I like it a lot. It sure beats writing tests in front of a blackboard.
i can see there are several reasons why this is useful to the evaluator, and i can also see that you're getting close to saturating the user with instructions, but in a perfect world it would be good to have "full disclosure" (i guess you do, in that anyone can run the example....)
I have no idea where you saw this being implied. The software highlights syntax and lets you check your code as many times as desired, as well as giving hints if your result isn't correct and providing compiler output. Nothing stops you from googling something. It's not like the software will destroy your file and egg your house because your algorithm is o(n2) instead of o(n); that's up to whoever is evaluating the results.
The fact that it has a polish and chinese version should also give you a little hint about the intended audience.
That said, as soon as the test popped up, I opened my eclipse, wrote the code, cut & pasted .. viola! (in python btw)
That said: Jesus I must still suck, or maybe Math simply isn't my strong suite.
BTW 10-20% of the population is like me.
Maybe you should time the programs, instead of the programmers.
As for comfy conditions on the job - I would hope that default conditions are comfy. When interviewing I'm more interested in how people will perform day-to-day rather than in rare situations. Most programming jobs don't really require employees to be "cool under pressure."
Other than that, good job on the execution so far. Reminds me a lot of TopCoder, just applied to a specific niche in a good way. I would even consider using this or recommending it to colleagues for hiring.
Consider making the clock like on http://www.sudokuslam.com/ - with an icon right next to it to show or hide it.
https://bespin.mozilla.com/
Edit: just noticed that you'll have to wait a few days to get an embedded release that has syntax highlighting ( http://groups.google.com/group/bespin/browse_thread/thread/0... )
http://twitpic.com/x6wkq and even the code matches line by line I guess - http://twitpic.com/x6wnp
> Hmm so Codility's editor is copy of the one SPOJ uses
Please don't flame. Codility has not copied it from anybody, its EditArea-based interface is in place since 2008.
Hmm so Codility' and SPOJ use a similar editor, so which one do they both use? Can I too get it :)
Thanks for pointing out. /me looks into EditArea
http://www.cdolivet.com/index.php?page=editArea
very nice project on LPGL license.
It's also taking a metric fuckton of time to evaluate my solution. Which apparently fails. For reasons I can't debug because I can't see the test data.
Any company that relies on code snippets, incomplete problem statements and undefined test data to judge coders will get what they deserve.
it wasn't obvious to me what to do on first read, but when i paid a little more attention, everything was there (including edge cases, as far as i could see). in particular, they gave a formal numerical expression that was unambiguous (and which didn't seem to contradict the text).
EDIT: It appears I can customize my test dataset, and I didn't see it. Silly me.
I got 94 using C in 12min (didn't check for overflows of large numbers, silly haskell making me used to Integers).
I'd give people access to their language's output facility for playing around to the solution, but that is a minor nitpick. (And I suppose I could open an irb instance on my machine for exploration and then give you code when I'm good and ready.)
My Ruby solution, in case we've got anybody who is struggling: http://pastie.org/771438 This would be my first cut if you asked me to whiteboard this.
The site's very slick. My only feedback is the histogram on the results page could be more prominent - it's cool, useful information but I ended up inspecting the html elements to work out what it was.
Also the results page seemed to keep refreshing for me while waiting for results (Safari 3). Surely that should've been AJAX'd?
Original: http://pastebin.com/m6e742f56 Parallelised: http://pastebin.com/m32a60b1
I've already sent it around at the company I work for. I'm hoping we can put it to use.
http://pastebin.com/m6e742f56
Edit: And won't this be a problem if interviewees can access the internet? The interviewers will have to make sure they can check the browsing history of the computer being used!
http://codility.com/demo/results/?id=demoMPC2W9-JDG
It would be nice if the boundary conditions were better specified (people seem to have been bitten by overflowing integers). This could be inferred if you told people that you're using a 32-bit machine (the sum of elements is at most SIZE_MAX * INT_MAX; if int and size_t are 32-bit, SIZE_MAX * INT_MAX < INT64_MAX.)
I was also somewhat surprised to see that my code compiles without #include <stdint.h>. I'm not sure this is correct (it could be solved by putting the code written by the candidate before the test code).
Finally, I assume you have proper sandboxing set up?
#include --- good catch! We are not very strict about missing includes, but this is definitely not something we want to penalize.
Sandboxing -- absolutely! :-)
It isn't that difficult to solve, but I like the idea.
Does that ever seem like a good idea? The description suggests it may be a long list. That should give you a hint that you don't need to do this to solve the problem.
The worst case of the best algorithm is to iterate twice - firstly as above, and then again in the opposite direction, comparing values from the first iteration.
Unfortunately, revealing whether my assumption proved correct would give away how they assess the solution to this demo problem, which some might view as a spoiler.
EDIT: since people are posting actual solutions, I suppose mere analysis of the testing methodology is okay. My assumption was correct.
I readily concede that coding ability isn't the only thing a dev needs, but it is absolutely required for most jobs.
eg. if the programmer goes off and writes a multithreaded solution, they might only realise too late that pthread isn't included. Admittedly they should have checked first using verify, but you never know with interview pressure.
The test window's layout is fubar if you use a small window size. I was using 800x600 and the language selection buttons were hidden behind the source code text box. Something (couldn't tell what) overlapped the first two lines of the instructions.
Firefox 3.5.7/Ubuntu Linux 4.10, FWIW.