Launch HN: Litebulb (YC W22) – Automating the coding interview (litebulb.io)
I've been a software engineer at 11 companies and have gone through well over a hundred interviewing funnels. Tech interviews suck. Engineers grind LeetCode for months just so they can write the optimal quicksort solution in 15 minutes, but on the job you just import it from some library like you're supposed to. My friends and I memorized half of HackerRank just to stack up job offers, but none of these recruiting teams actually knew whether or not we were good fits for the roles. In some cases we weren't.
After I went to the other side of the interviewing table, it got worse. It takes days to create a good interview, and engineers hate running repetitive, multi-hour interviews for people they likely won't ever see again. They get pulled away from dev work to do interviews, then have to sync up with the rest of the team to decide what everyone thinks and come to an often arbitrary decision. At some point, HR comes back to eng and asks them to fix or upgrade a 2 year old interview question, and nobody wants to or has the time. Having talked with hundreds of hiring managers, VPs of eng, heads of HR, and CTOs, I know how common this problem is. Common enough to warrant starting a startup, hence Litebulb.
We don’t do LeetCode—our interviews are like regular dev work. Candidates get access to an existing codebase on Github complete with a DB, server, and client. Environments are Dockerized, and every interview's setup is boiled down to a single "make" command (DB init, migration, seed, server, client, tunnelling, etc), so a candidate can get started on coding within minutes of accepting the invite. Candidates code on Codespaces (browser-based VSCode IDE), but can choose to set up locally, though we don't guarantee there won't be package versioning conflicts or environment problems. Candidates are given a set of specs and Figma mockups (if it's a frontend/fullstack interview) and asked to build out a real feature on top of this existing codebase. When candidates submit their solution, it's in the form of a Github pull request. The experience is meant to feel the same as building a feature on the job. Right now, we support a few popular stacks: Node + Express, React, GraphQL, Golang, Ruby on Rails, Python/Django and Flask, and Bootstrap, and we’re growing support by popular demand.
We then take that PR, run a bunch of automated analysis on it, and produce a report for the employer. Of course there’s a limit to what an automated analysis can reveal, but standardized metrics are useful. Metrics we collect include linter output, integration testing, visual regression testing, performance (using load testing), cyclomatic/halstead complexity, identifier naming convention testing, event logs, edge case handling, code coverage. And of course all our interview projects come with automated tests that run automatically to verify the correctness of the candidate’s code (as much as unit and integration tests can do, at least—we’re not into formal verification at this stage!)
Right now, Litebulb compiles the report, but we're building a way for employers to do it themselves using the data collected. Litebulb is still early, so we're still manually verifying all results (24 hour turnaround policy).
There are a lot of interview service providers and automated screening platforms, but they tend to either not be automated (i.e. you still need engineers to do the interviews) or are early-funnel, mea...
195 comments
[ 4.9 ms ] story [ 82.5 ms ] threadif you succeed in making this feel "native" to both employer and the applicant, you'd be billionaires by tomorrow
Reading this comment, it occurs to me that if just anyone can sign up for a trial, then candidates may be able to game the system and prepare for the specific work sample tests that this product provides. If so, then that's unfortunate, because I like the idea behind this product.
1) This is actually why I was going back and forth on offering a trial version, and decided to use a form so that I can qualify employers who legitimately need this vs candidates just trying to game the system
2) It might not actually matter if candidates get preliminary access to Litebulb interview code bases. Our interviews aren't brainteasers that have a specific optimal solution, they're just regular feature requests, ie build a CRUD API for this new resource model, so it almost doesn't matter how long you have. From what we've seen with submissions so far, having extra time doesn't necessarily make your code quality better. Potential idea: open source all Litebulb interview codebases?
This would be such a wonderful contribution to the industry. Imagine having a hundred JS front end questions, it’s not really game-able any more. I agree with your take. Keep qualifying your leads though, for the sake of your funnel.
Potential feature: get high enough scores on a specific interview, get recommended to companies currently hiring that use this stack and get inserted mid-way through the funnel?
I would honestly pay for this without even the hope of a job, just to be able to know where I am at, feel confident if I ever get past the first screen.
The next step would be to open up that candidate-facing side, and hearing that you'd pay for just that alone is great feedback!
Take homes are great because the candidate either knows the specific frameworks, or doesn't but can learn them quick enough. Either way, if you complete it within the allocated time and get relatively high enough scores, we'd recommend moving forward.
Actually @xiphias2 what type of interviews have you seen that were really good? Like even the in-person ones, what made them good?
As an example of why for some (especially big) companies need deep algorithmic knowledge is that at Google I was working on adding new features to an ads prediction model, and even without changing the code base the new (slightly more complex) model increased the ads 99%-ile latency by 0.1ms (millions of dollars loss because of the RPC timeouts), which required director level exception there if I wanted to ramp up the experiment to total user traffic (the usual thing that people do is to optimize something else in the code base to stay in the latency budget).
The small problems that LeetCode provides for interviews are great filter for people in these companies when they are faced with much more complex optimization problems. The problem is when a small company with lots of user facing features uses the same type of interview (they shouldn't).
We also give people time to learn the framework we're using as part of the hiring process. And we encourage them to do that if they want!
People with a tremendous amount of experience in our framework-of-choice have an advantage. But they're also more likely to be immediately valuable. It's imperfect, but it works well for us. And you'd be surprised how many people float right through the technical assessment learning as they go.
I was responding to this negative part of the introduction. Actually all of the mentioned frameworks are disallowed at Google (where leet code style interviews were introduced), because they are not fit for serving billions of people in the latency budget in the search/ads stack.
If you would rephrase it that for most companies leet code style interviews don’t make sense (and probably are your customers), I would agree, but the few where it makes sense are really important companies.
And how do you prevent someone from cheating if there's no engineer there on the interview?
We are not talking some CRUD app where it is trivial to make each component work or a Leetcode problem that you can spit out in 20 minutes.
In the latter case, is it possible for a developer to operate that way and still produce a great solution given more time? I suppose so - but in my experience, the people who are not in the habit of incrementally checking their work are virtually never strong developers.
There's also a time management question here. If a person knows they only have 45 minutes, that should create extra pressure to start testing earlier. Someone who still waits until time's almost up before checking anything is probably a fairly extreme case of this antipattern.
Something I've been thinking about, not necessarily in the interviewing world, would be to have a simulation of distributed systems for non-production code: in theory I think you totally could have a simulation layer for databases / server loads and test how your code performs in such environments.
I guess it exists in production: but haven't seen it for either a) learninga bout distributed systems, or b) for interviews
1. Complex cases:
We can spin up isolated AWS or GCP environments with some resources pre-filled, and the rest is up to the candidate to build. Basically a systems design whiteboard challenge, except instead of a whiteboard, it's actually in a cloud environment. Since "input" will be defined, and specific "output" will be expected, we can then run tests against this environment to test for sufficient robustness. For example, if the prompt is to build a load balanced CRUD api, we can run create resource a million times, then try to read, update, or delete, and measure total response time. For something like testing concurrency safety, we can run a bunch of parallel threads attempting to access the same resource, then test for timeouts (deadlocks) and inconsistencies.
2. A solution that's code complete but has a minor bug that prevents compilation
For a solution like this, code completeness scores are unfortunately going to be 0, simply because there's no way to test for functionality, performance, event logs, if it doesn't compile. However, our static code analysis will still work, so they may still get high scores on code style, code complexity, naming, etc. Something we might be able to add is a way to flag to employers that a submission has high scores for everything except a specific metric, and so we'd recommend manually diving into the code.
3. Cheating
We can't prevent cheating if the candidate gets their roommate to do the interview for them. However, if someone just blatantly copies an old solution their friend sent or they found on the internet, we can check for that. We can run a diff between the submitted git history and all previous submission's git histories. 2 legitimate final solutions might be close (ie 60-70% match), but you'll never have 2 organic git histories with multiple commits be close, (it'll pretty much be either 0% or 100% match). Looking things up on the internet isn't considered cheating, I don't know of a single dev that could survive the day without SO :D
You could always have someone look at their work. Or better yet walk through it with them to try and ascertain their thinking. This is what I do with my team, multiple times week, in my actual job. Seems better than immediately excluding them
Fact: The LeetCode loop sucks. So couldn't any Director or VP in big tech just decide they are done giving that interview? They have the resources to spin up something like this in a week. Even outside big tech, companies can very well think of non LeetCode style interviews, some already do
The problem is, many engineers want new engineers forced into this type of interview, because they had to do it, so you have to do it too. Not for any reason better than that
I'm not sure a full featured well polished product like this one can change toxic engineering culture when it comes to joining the club
It doesn’t have to be leetcode, but it does have to not be “have an engineer spend an hour with everyone”
That sentence right there is a pretty solid indicator that it’s a solid business to be in, no? What VP wants to take engineer time to create an internal tool when they could pay $40/user/month or whatever? How many SaaS startups do we see right now that occupy that exact sentiment area?
As far as this being more of a culture change than a process change — well, good products can change culture. Making something much easier is a huge step on the way to getting people to do it.
The cheapest pricing for this is $600/mo, the next plan up is $2,400/mo, the enterprise plans start at $60k/mo.
Most of those would use this tool at least twice during their interview process.
So you're paying... $60k/2000 = a cost of $30 per interview. If you save your company half an hour of engineer time (at $60/hr, another comfortable lower bound) you have already made back the cost.
And that's just strictly cost reduction. If this is genuinely a better interview process, if you get more applicants and qualified candidates as a result of using the tool, of course the value of said tool is gonna be even higher.
I'm pulling numbers out of a hat, obviously, but I don't think any of this is at all unreasonable. And once you start putting it like this, I think you can start to see the justification for that kind of pricing.
Edit: well, just looking at their Enterprise pricing in a little more detail, they're pricing at 60k/200 = $300/interview, then $30/interview after that. That's... a lot steeper, and a lot less friendly to mid-tier companies. But hey, frankly, I can still see the justification for it -- if it's really good.
When it comes to changing engineering culture, I completely agree that a single tool like this isn't going to cut it, but we're seeing a big shift in new startups focusing a lot more on real work relevant take homes and/or work trials. For example, Gumroad (disclaimer: a client), does a Litebulb interview as the entrypoint, then it's straight to a 4-8 week paid trial on flexible hours with a choice of which project to work on. We're hoping that we can be a part of, and accelerate, this culture shift.
For me as someone who just finished interviewing, if you told me that I either have to work on two jobs for 4-8 weeks or quit my current job for a limited contract that may or may not work out I'd almost certainly say no. (I think most people would too, I mean people already complain about take home quizes etc.)
I doubt HR people are on a coordinated mission to revenge their own suffering by letting new candidates suffer the same horrible LeetCode fate. Instead, I rather think LeetCode is chosen because it’s the easiest way, the path of least effort, or laziness.
Some feedback on the homepage. Its current form can be improved.
My suggestion is to not try to get every visitor's email account information before you tell them how your product works. It might help raise the quality of engagements to provide your information first and then let hiring managers click on REQUEST DEMO.You expect everyone to self-host videos?
[1] CDN: Content Delivery Network, i.e. youtube, cloudflare, or any one of the others. There are multiple CDNs used by this very article's webpage.
Would you similarly be up in arms over the external JS were the site embedding a YouTube video? It also requires JS – as does most (all?) major video hosting platforms.
That means that the Youtube CDN doesn't work with modern CORS for HTML video loading of resources by offsite webpages. Other major, more modern CDNs implement this.
I'm not clear why you feel CORS factors into this.
I'm debating on just not having that interactive demo at all, and instead to be a bit clearer in the rest of the site about what we do, and how we do it.
1. Providing good programming tests with automatic env setup
2. Automatic PR reviewing and grading.
By this I mean that you could do less and still be useful, for example just run linting and tests on the PR and let me review it myself. That will match what I do at work anyway. That would let you focus on just providing great example projects and covering more languages, rather than on the automatic grading.
I conduct technical interviews routinely and very rarely do I touch on anything coding-related. That's because I'm in security, compliance, and identity.
I'd strong suggest changing changing 'technical interview', because it gave me a completely incorrect idea of your product.
Coding interviews are technical. But not all technical interviews are coding.
This is a software engineering test only, and very specific stacks at that. It would be a waste of a person's time unless that was the exact stack they used in the past, or was demanded by the job.
I'm currently tasked with hiring a senior data scientist and I'm dealing with figuring out the take home case study right now. Would be great if you guys did something similar that wasn't for pure CRUD apps, but also included a normal data science case study as well as included things like unit tests, enforced a PR workflow, and other MLOps tasks like quantization, A/B+regression testing, and monitoring, etc. I've been asking candidates questions about those topics for a while and it just feels pretty tricky to come up with what are fair and reasonable questions that aren't too easy and at the same time can be graded objectively.
Probably a lot to bite off for your team right now, but I'm just sitting here wishing I didn't have to come up with this case study from scratch atm, so there's my dream scenario :)
To be clear though, I'm not a big fan of take homes either and we're trying to limit ours to less than 2 hours total time spent and give folks a week to find those 2 hours, so it's not overly burdensome.
Your job as an interviewer isn't to come up with questions that aren't too easy or grade homework assignments. It's to assess a candidate's experience and ability to do the role you're interviewing for. Making a grind interview means you'll drive away the best candidates that have options and find your process disrespectful of their time and experience.
I thought there was some sort of machine learning involved.
“fake it till you make it”
Who can guarantee that your “reviews” are aligned to my goals and not just outsourced to the cheapest contractor available?
Some examples:
1. 7/10 unit tests passed
2. GET:/api/v1/vehicle/{id} OOM'd out at 1.5M records in the DB
3. Linter returned 10 warnings, 0 errors
4. Cyclomatic complexity counter returned max of 24
5. 5 new commits, shortest commit message 3 chars, longest commit message 45 chars
Also, the concern with outsourcing to a cheap contractor is a very legitimate concern, because we ALMOST actually did that. Instead, we decided that it was a bad move and to double down on building product + surfacing clear, unambiguous data. That's also why we're intentionally staying away from ML (at least for now), because it's inherently unpredictable.
<sarcasm> I always forget number of lines of code. </sarcasm>
Nothing against you man, but that's something I (and maybe all the people looking for a dev) don't really need.
I can run linters and static analisys tools by myself if I really want. I never did that myself when I was interviewing people and I've never seen any other interviewers doing that.
And you're asking 600$/month for this?
1. In both automated leetcode and automated "coding" interviews, most people get full score. This even turns into a negative bias as an actual good candidate who did not cheat has greater chance of getting rejected here than the cheaters.
2. Interviewers put both automated leetcode and in-person leetcode to first filter out candidates, then put them through actual interview hoops. If your product succeeds, this means another layer of "interviewing" and more wastage of engineer's time.
Sorry to say this isn't solving any problem but has the potential to create more harm for both sides.
Unless there is a solution that directly addresses hiring bias this company will fail like all the others before it for exactly the same reason: their clientele will apply bias (the actual cause of the problem), be unhappy with the result, and then blame the service provider. In sales the client is never wrong, which means the service provider is always wrong. This is why software overspends on recruiters and then invents fantasies like React developers with a year of experience are rare.
Many software companies opt to avoid leetcode-like interviewing entirely. If leetcode and prepackaged, recycled, self-contained code tests are your workplace's primary process for hiring, then I don't think this service is for you at all.
Any decent employer puts the new hire on a trial period first. During the trial period, the employer should task them with real work. When the candidate doesn’t deliver, the employer can let them go. When they do deliver, the employer can keep them.
In both cases, does it really matter if they cheated their way in or not? No. Preventing cheating is the wrong place to optimize.
While we’re here, dumb memorization of LeetCode style Q&As instead of deep understanding of underlying principles can be seen as cheating just as well. Doing actual work is better in every way.
And many employers already do exactly this coding work already, just manually. They make up a problem themselves, and give you a time limit to commit a solution without checking if you’re cheating or not.
Litebulb just automates this, by taking over the creation of the problem and the analysis of the solution. There’s lots of advantages of the centralization of this, like creating more problems, fixing and updating existing ones, etc.
Provided the problems and the analysis are as good as advertised, I think Litebulb has a great value proposition. I wish them success and hope they can have a lasting impact on the hiring process for software developers.
That would mean you waste a few months for every bad trial hire, especially if you're looking to fill just one or two positions.
If you have expertise in a field you can normally guess pretty quickly when someone bullshits their way through or actually knows what they’re talking about.
But your hiring staff usually don’t have any or deep enough expertise for the field they’re hiring for, which only gets worse the bigger and more advanced your company gets.
But your employees with which the new hire will be working with should have this expertise (or again your company has a problem). So they will find out pretty quickly if the hire is a fraud or not.
Cheating surely creates costs but not as much as you suggest. And as others have mentioned above it is surely not as widespread as you suggest, limiting the cost further.
2. Litebulb is meant to replace the initial LeetCode question + the mid-funnel technical screen. We still recommend doing a hiring manager screen at the beginning and team onsite at the end, but using Litebulb correctly should lessen an engineer's interviewing burden, not increase it.
However, because we're still early, it's not fair to ask clients to chuck out the tech eval part of their onsites, since we haven't proven effectiveness to them yet. So instead, we'll replace whatever they use mid-funnel, and also early funnel auto-interviews. We still recommend someone talk to the candidate both before and after the Litebulb interview.
In today's market you will likely see a large percentage of candidates opt out of the interview process when you send them an automated technical interview.
1. Hop on a call, do the interview in one sitting together and share thoughts/questions
2. Hop on a call, make sure set up is smooth, prompt is clear, questions are answered, then candidate goes off on their own. Hop on another call later in the day to demo the solution and talk over design decisions.
3. Do it completely async, submit the solution whenever you have time
From what we've seen, this flexibility has simultaneously reduced dropoff rate and increased candidate experience, since we're catering to the candidate's needs. Some would prefer to do it face-to-face, others prefer to work on their own.
I do agree though that some of our frontend interviews are a bit heavier on code + css and less on systems design thinking.
Btw just to clarify, we have a policy where we won't host any interviews that are just open dev tickets to build a feature that will be used in production. Like, we DO NOT tolerate companies using interviews to get free labor.
What does that change from a candidate perspective? In both cases the candidate wastes their time. I'd offer a better solution: the company pays market rate for the time needed for the test, and can then ask for anything they want - in which case an actual ticket is the best solution as it will evaluate the candidate's performance on the kinds of problems they'd actually be working day-to-day.
So kind of LinkedIn badges but with actual meaning :-D
If I could select which repos to include, that could be very interesting. Also interesting additions: open source contribution analyzer, Stack Overflow Q&A quality analyzer.
Stars should be counted but can't be the only measure.
Currently: candidate does a Litebulb interview, gets a report, that report can be shared with whoever they like, hopefully speeding up interviews at other companies.
Next up: candidate does a Litebulb interview, gets a list of actively hiring companies that use that stack and would like the candidate to be inserted mid-way into the interviewing funnel.
And if they mage to get a way to find underrated workers, they will make both sides extremely happy.
The point would not even be to try to necessarily solve the problem itself - more just to see how I got along with them. Of course some people might freeze up when asked to do this, but I think that happens in any higher pressure interview environment. And if I do not know the solution off the top of my head either and am obviously even struggling in some parts myself I think that takes a ton of the pressure off of them to be a wizard that can solve it in the moment.
As an add-on, an interesting question one of our clients asks as a conversation starter is "what's the most technically complex project you've ever worked on?". This question sets up the space such that the candidate is the expert (since they've already worked on it), and it almost becomes a session where the candidate teaches the interviewer.
I do not take personal offense to being tested in an automated fashion.
In one case I did a prerecorded interview and they later contacted everyone because something went wrong and they wanted everyone to do it again.
Done feeling like a number.
Note that it's an option, not mandatory, to do it async. If the candidate wanted to do it in a sit down session, then we prioritize making time for it to be there the entire session. The reason we offer this option is because devs that have busy lives often can't easily find a single consecutive multi-hour session to do an interview. They have full time jobs, and might have family obligations at home and over the weekends. Basically, the only way to actually do a full onsite interview is to take a day off work to go do the interview. We're giving the flexibility for the candidate to find a chunk of time here, a chunk of time there to complete the interview.
We tested this above approach, because just as you said, we did see a huge dropoff when we just emailed candidates the interview link. After we offered to hop on a call and to be available for support, we saw dropoff reduced to less than 15%.
Also, a good chunk of candidates that drop out of interview processes are senior engineers that can't be bothered with lengthy processes, which is fair. I wouldn't give a Litebulb interview to a senior engineer to begin with. Debatably I wouldn't give any kind of technical interview, actually. If they have decades of experience leading teams at big or high-growth tech companies, those achievements speak for itself. The interview process for seniors should be more geared towards finding a product-team-eng fit, rather than an evaluation.
That is the fundamental issue with technical interviews, because there is very little about that manufactured situation which is applicable to the candidate's ability to do the work. There are extremely few real-world coding situations that come anywhere close to the pressures felt while having someone hyper-analyze everything you do in real-time, particularly when that someone is responsible for deciding whether you're able to pay bills next month or feed your family next week. It's a needless, pointless and ultimately self-harming methodology that filters out exceptional candidates in favor of the most extroverted or arrogantly confident, which is a bizarre twist considering the most influential and game-changing coders are also famously some of the most introverted and "unusual" people.
On top of that, because we're all in dependency hell at this point, far more time is spent googling for framework errors that don't make any sense until you find that one undocumented command line argument that a single person posted deep inside a GitHub issue thread. If you want to test someone's ability to do the real work, then give them an obscure error code from a little-used utility and see how long it takes to find the fix.
This is actually a type of interview we're trying to support! A blocker right now is that once it's leaked, you basically have to throw the interview away. The good part about the "feature building" type interview questions is that it almost doesn't matter if the prompt + codebase is leaked, because your solution is probably still not optimal. "Good code" is super subjective and takes years of experience/learning to master, but an obscure bug fix shifts the end result to a boolean state, which is easy to leak (thus hard to detect plagiarism).
Please use your platform to spread this message to employers. I have increasingly been getting leads asking for some sort of automated testing, even when I bring 25 years of hands on and management industry experience. The tests typically require multiple hours of time investment, with no real guarantees. That's asking a lot.
Moreover, if this is necessary, I kinda assume the pay is mediocre at best. If you knew how to get enough value out of me to be able to pay me really, really well, and if I look like a good fit for the job, then again, my resume ought to justify our talking. If any of that's not the case, then it seems like the pay's gonna suck or I'm just not a good fit, so I'm out either way.
I want someone who's fishing for me with a lure, not by dragging a huge net through the water. Seems like a better sign all around.
Why? Because people tend to hire people with qualities like themselves, and overlook many things. It also then caused people to overrate/underrate the analysis of coding tests based on the previous interview (as they (dis)liked the person).
This also massively increased the diversity of our team.
But, from an employer's perspective, resumes tell you very little about a candidate and are essentially worthless. The sole usable metric we get from them is years of relevant experience.
We respond with a questionnaire that gets us more uniform answers about experience and technologies used.
Assuming the above doesn't filter the candidate out, we move on to a 90 min max "take home" exercise. In our case, it's not a coding exercise, but that's not that important for this discussion.
Only if all of the above hasn't filtered out the candidate, do we move to a Zoom meeting. In the Zoom meeting, after an initial introduction, we do a few easy coding exercises. Like, reverse a string or sum the integers in a file easy.
A few notable things from this experience:
1. The number of candidates who struggle with the Zoom interview coding exercise is surprising. It's a lot higher than expected and almost all of our candidates at that stage have 2+ years of professional development experience. 2. The number of candidates filtered out by the "take home" exercise is very high. 85+%
We literally don't have enough time to talk to every candidate that applies with the right amount of experience. And, even if we did, many/most candidates on the market aren't good developers. Our signal to noise ratio is already bad. In that case, it would be terrible.
As a developer, I sympathize with your perspective. But you may be passing on some good opportunities by not being willing to "play the game" a bit. Some employers are indeed bad actors and/or terrible at hiring. But given the current market, even those making a valiant effort to do it well, due to sheer volume, are usually going to have an exercise of some kind.
We post our salary ranges up-front, don't want to waste anyone's time that way. We also put a ton of effort into a very detailed job description and application process. The hope is that the candidate can see the effort we put in and will be willing to reciprocate. If that doesn't end up being the case, we assume, like you, it probably wouldn't have been a good for anyway.
FWIW.
Your funnel is steep because it has bullshit steps. And the only people coming through it are probably desperate, which is why your signal/noise ratio is "already terrible".
What kind of experience do you have hiring? If you have a better process, that has been proven over time to work, please share.
I'm part of a small group of leaders/owners from other development shops. We meet monthly to collaborate and learn from each other. Hiring is a frequent topic and I assure you that our steep funnel and perspective on candidate ability is not a unique experience, despite divergent hiring processes.
It is true that our process could be filtering out candidates who might otherwise apply. In that case, it's most likely working as intended.
You are very casual in claiming that there is such an intentional correlation between developers who are not willing to spend 90+ minutes on a non-technical prescreen with a 15% success rate and competent developers who can translate business needs into actionable steps. It's reasonable to say you are willing to accept that there are developers who are unwilling to go through this process, but to say you are filtering them out intentionally is needlessly contemptuous.
But, seriously, it's intentional in the sense that if someone is a fantastic developer, but can't appreciate why a process like this is valuable to them and us, then it's a good thing they aren't applying. I need our developers to be competent at development but also reasonable in their perspective on the give and take between a business and their employees.
It's deliberate that I don't want people with your perspective/attitude to apply. I don't want people who can't appreciate the others parties constraints. It's not going to be a good match in the long run. And that's true even if said person can code circles around our best developers.
To me it sounds like you needed some way to thin the stack of resumes, and this is the method you chose. I'm just skeptical that you're going to get a lot of quality candidates. How are you going to pull developers who are comfortable in their existing jobs with such onerous requirements? These are people worth hiring: they're competent, skilled, and don't need to find another job.
Though looking at your profile a little, I think the answer is clear: you don't try to. Rather, I think maybe what you mean when you say fit is "we filter for people who need this job so badly that they'll go through our hoops to get it."
perhaps its that a web dev shop can't afford to hire the best talent, so the that sort of a filter is useful.
This is definitely the most important aspect of this discussion.
So, those are the first two things we test. We give the candidate a description of the work to be done and ask them to interact with it. What questions would you ask the client and how would you break this down into stories/issues? Then, we give the schema of the existing database and ask them to modify it so the new schema supports the work requested.
Still very software development oriented, just not coding. We have more in-depth coding exercises later in the process.
I realize in some orgs, there would be a project manager or product owner of some type who would break that work down. And we have team leads who do the majority of that work. But for our org, devs who couldn't do this struggled to perform well in the development work too. So it became a skill that correlated to being able to succeed after hire.
Indeed, this simple filter is the original purpose of FizzBuzz and other simple coding tests. Circa 2007: https://blog.codinghorror.com/why-cant-programmers-program/
And yes, I agree, it seems crazy. But it's our reality and seems validated by others I know in the industry and anecdotes here on HN.
There is just so much demand, it pulls people into the industry that can't really perform as professional developers.
I think the frustrating reality is that it's usually not possible to tell the good from the bad at the portfolio/resume level, some kind of skills/competency test must be given.
I personally don't look at GitHub repos because I don't know how long it took to write that code. If it's great code, but took 4x to write than it should have, I wouldn't know. There are also a lot of applicants that just don't have much public in GH.
Finally, to be clear, I'm not saying they are all bad actors. I'm sure some are and the same for employers. Mostly a result of the current world economics.
For a 90 minute take home exercise that I have to complete before I am allowed to speak to a human being, that is on average about 10 hours of work your company is expecting me to do, for free. And then I have to do a coding assignment afterwords anyways.
I will say that the fact that you post the salary range is good. This is the sort of time investment that would only be worth it for a top end salary.
Also, I wouldn't say it's "not allowed", just not the standard process. We have, for specific reasons at the candidates request, done a Zoom meeting earlier in the process.
Candidates are also encouraged to email us with questions and we put a lot of effort into being responsive and good communicators with the applicants.
That's not what Litebulb is for though, we're mid/late funnel. As a candidate, if I have to choose between a 4 hour onsite or a 5 hour take home, I'm probably going to do the take home.
In terms of pay, I know some really good companies with a healthy eng culture and heavy pay that do take homes (not as a pre-screener, for sure).
Btw from my experience, resumes are a pretty weak indication of skill. I have interviewed people with 15+ years of experience, some at MAANG, that couldn't comprehend basic systems infra scenarios, and I've interviewed interns that had production-grade code. You need to actually try it out, one way or another, to be sure.
Those are mostly the "sweat-shops" that hire developers that can work for less for several reasons. I see that in this particular case this solution might be very welcome to assess new hires.
But for more high-profile jobs and companies, the company that eventually follow this path will probably end with low quality hires, unless of course it just create crud's anyway.
Because this is a self-fulfilling prophecy in the end. Once the candidates know what they will face, they will train until they get good in that game, which most of the time doesn't mirror the qualities required for the job.
There seem to be markets (or large enough organizations?) where coding job ads get a tonne of applicants, many of them inexperienced, and so automation is a good fit. But not when things are tipped the other way.
Our bet was for real-world tests too but it wasn't enough. A few things we missed that might help…
- Candidates don't want to be treated like cattle
- For many companies a good interview platform will be more beneficial than automation
- Companies say they care about the experience candidates receive, they'll say they're rigorous and try to be as objective as possible with the way they collate information and make decisions, they'll talk about how high their standards are, etc etc. Be weary
- There may be more gains to be made outside of the tech space
- The problem most companies complained about and is probably still the hardest: going out and finding people
I like the way you've solved the "real-world challenge" problem.
Good luck and all the best!
Also, the best engineers aren't necessarily the best engineering managers. See https://en.wikipedia.org/wiki/Peter_principle.
We don't believe tech management interviews should at all be automated. You really want these to be as qualitative and human-driven as possible.
It is a convenient myth that people who build the actual systems can be put through an automated filtering process capable of sifting wheat from chafe where as those who manage them are doing subjective, qualitative work that cannot possibly be measured.
I do think everything can be measured given the data, and hence automated. The problem in the latter case seems to me that the data is not easily accessible.
My question is around the grading examples. How do you automatically determine Functionality is Senior - Weak or Commit History is Intermediate - Strong?
I find these to be quite subjective and better understood as a whole for any given candidate. The grading mechanism provides a false sense of being unbiased, and further dehumanizes the process.
> Every metric we measure is numeric, and every test is deterministic. As a result, you can make hiring decisions based purely on merit.
I fully disagree with this for hiring, unless you’re looking for machines, not humans. Measuring things automatically does not result in purity of bias.
Instead, we're going to just expose raw data about somebody's code submission, and let you decide what those numbers mean.
Some examples:
1. 7/10 unit tests passed
2. GET:/api/v1/vehicle/{id} OOM'd out at 1.5M records in the DB
3. Linter returned 10 warnings, 0 errors
4. Cyclomatic complexity counter returned max of 24
5. 5 new commits, shortest commit message 3 chars, longest commit message 45 chars
In my experience, the metrics you've sampled aren't particularly useful for hiring. Some of my thoughts:
> 1. 7/10 unit tests passed
Automated unit tests are table stakes on all platforms like HackerRank, LeetCode, etc. Might be useful to classify/clarify the failing ones? Or even add an option for the candidate to clarify their thought process for those.
> 2. GET:/api/v1/vehicle/{id} OOM'd out at 1.5M records in the DB
As an engineer, I see the appeal of OOM checking. But as a candidate, it would be important to have the same information available during test-time, so I could fix it. Could be as simple as an accidental N+1 ORM mistake. Otherwise, hidden performance testing would make the experience very stressful, as then I need to double-check everything and over-optimise for perf.
> 3. Linter returned 10 warnings, 0 errors
I would argue that linting should be out-of-the-box and automated for the candidate. Or even if not, there should be heavy filtering of linter warnings. Every company has their own style guides, and candidates shouldn't be expected to learn that for an interview.
> 4. Cyclomatic complexity counter returned max of 24
While I do understand the concept of Cyclomatic complexity, I have /never/ seen this used IRL in any company (startup, mid-size, FAANG) that I've worked at. I would much rather judge code complexity in terms of readability and decision making.
> 5. 5 new commits, shortest commit message 3 chars, longest commit message 45 chars
I appreciate clean git commits, but these are enforced by company's internal policies during code reviews. Easy to learn once, and continue doing. But definitely not a deal-breaker in a time-constrained interview assignment.
Are these raw metrics you shared what constituted the grading from V1? I would definitely reevaluate most of these for V2, as they seem a little nitpicky than insightful.
I've interviewed hundreds of engineers in the last twenty years. I've been (and am) a hiring manager for many of them. I wouldn't touch this with a ten foot pole, and I'd run away as a candidate if my git commits were being judged as part of a coding interview.
These "metrics" are for conversation, not for automated "judgement."
I think a big reason (in addition to many others..) that companies still use LeetCode is the lack of interviewing platform alternatives, so I'm glad to see Litebulb is looking to fill that niche. I'll email you directly after work if I don't hear anything on here!
Are those really things that candidates can get a negative ding for? Those are things that can/should be handled automatically by libraries or your CI/CD (ex: Rubocop)
More importantly, how do you get more details/nuance over things like naming or cyclomatic complexity.