Agreed. We use a coding challenge that mimics the type of work we do, and we pay people for their time. We talk through their solutions and get a good sense for their ability to focus, follow written user stories / specs, and their familiarity with common tools. We also get a feel for how well they can talk about their work and the product.
We see a strong correlation between performance on this task and performance and happiness when working with us.
The other important point is that this is very similar to the work we do and the way we generally do it. If someone doesn't like or want to get paid to do this task, they're likely not going to be happy doing it every day if they get the job.
I disagree with the author. A coding challenge is a good way to judge how a candidate approaches a problem. The ability to walk through what you’re thinking while you’re doing it is a great judge of ability.
It also helps to weed out people who can’t actually program, this comes up annoyingly often.
I agree that having intentional errors in a starter repository is bad though.
You can also have a discussion one how one would approach the problem vs actually doing the coding. The discussion alone would give great insight into how the applicant understands the whole concept, and any potential pitfalls that can be anticipated, etc.
I want to move from my 2000 USD a month (this is a good salary already) in a 3rd world country to a FAANG that pays 20k USD per month.
I'm willing to go through any challenge because the reward is superb for me.
Not to mention, without objective interview questions, an introverted non-native English speaker (who doesn't speak latin-germanic-based language) like me would fail most of these subjective questions (e.g. bad culture fit, can't articulate as well as a native speaker).
That's why people do it, no doubt. Keep in mind, a lot of these questions are asked by vastly less desirable companies that ape the hiring practices of FAANG, even though they pay much less. And these companies complain bitterly about a shortage of good developers to hire.
So... I'm all onboard with your statement, but I think you only said half of it. Nobody owes me a job, and if I don't like it, I'm free to not get hired. Nobody owes a company an employee either, and if candidates don't don't like it, the company is free to not hire. Zero sympathy for companies that do this and complain about a shortage of candidates.
Do only those companies complain about a shortage of candidates? Because in the tech centers I know, the market is just empty. Lots of large companies are competing for talent, and if you can't pay bank salaries, you're going to have a hard time.
You make it sound like there's plenty of talent around but companies are being snobbish. That's not my experience at all in Germany.
In San Francisco, there's plenty of tech talent available at 500k a year. There's remarkably little available at minimum wage, though.
Complaining to congress that there is a shortage of IT talent is so common in American politics that it has become its own unique form of performance art. For instance, during the magic section of the act, companies make a labor shortage magically appear without any mention of wages, while also magically making the laws of supply and demand disappear.
“Shortage” is an economic term of art, so I won’t use it here. But it’s likely true that there is lot of valuable, economic-growth-producing work that could be profitable if a software engineer costs $100k/year instead of $500k/year. Businesses that lie on the margin of profitability are likely to feel that it’s hard to find good talent. Of course, it’s bad for us existing software engineers if the supply increases and the price goes down. It’s likely good for society overall though, if it enables a bunch of other businesses to exist. This is true in the same way that more trucks, more warehouses, and more conveyor belts would increase economic output, even if they were already available for the right price.
You just said that there's lots of valuable work that could be done, but your example of work that could be done is work isn't valuable enough to do unless software engineers get paid (in the SF Bay Area) notably less than a dental hygienist.
I like having clean teeth. If someone can make more as a dental hygienist, why on earth would it be valuable to society if they instead worked as a software engineer for less money?
If people won't work for 100k/yr as a software engineer, that means people capable of doing this work have found higher value things to do elsewhere. The higher salary elsewhere reflects the higher value produced doing something else.
This is one of many reasons why I'm so bewildered when people talk about a "shortage" of software engineers without considering the options available in other fields (fields that, perhaps, pay better and don't require taking bizarro whiteboard exams every time you change jobs).
I'd say there is a shortage of software engineers as long as most stem grads can easily raise their salary significantly by doing a few months at a coding bootcamp. That is still true, so there is a shortage of software engineers. I multiplied my salary that way, although I learned it on my own instead of a bootcamp but it is the same idea.
That makes no sense. A voluntary pay reduction is like a subsidy leading to market distortion (more supply than the market can bear) and windfall gains (employers might just keep the extra profits) - only that this subsidy is coming from your and my paycheck lol.
The idea is an employer-side argument to counter the labour-side demand for higher wages: Don't have the wages to high, otherwise we go bankrupt and can't employ you any longer.
But programmers are already well employed, and software companies aren't really suffering. Cheaper programmers won't make website less bloated, won't make us figure out what to do with GPT-3, won't find a useful application for blockchains, won't make Windows 10 less moronic, won't make ad-tech less harmful, won't make the tech recruiters and the non-technical folk around software projects more competent...it would simply kill one of the few fields of employments that still pay well!
Look at construction. Did cheap immigrant labour make rents feel low in many areas? No, market forces drive up prices and owners keep the profit.
I didn’t say anything about a voluntary reduction in pay. Any reduction in pay would be a result of market forces in response to increased supply.
> Look at construction. Did cheap immigrant labour make rents feel low in many areas? No, market forces drive up prices and owners keep the profit.
In areas that aren’t constrained by density limits, homes are actually very cheap. In terms of square footage of home per population, the US is doing great. Problems are with inequality, in terms of geography and wealth.
Market forces should have already brought in more people into software if they really are that high, bringing down the salaries. What other measure would you suggest? Importing supply from abroad? Whatever it is, it would lead to a reduction in pay, which you suprisingly support - that amounts to walking into your boss' office and asking for a pay reduction.
> In areas that aren’t constrained by density limits, homes are actually very cheap.
That's the point. The effect of cheap construction labor only affects the rent prices in some areas, in other areas owners pocket the extra profit - while making construction jobs largely untenable except for immigrants coming from poverty.
I suspect something similar could play out in the software field.
> Whatever it is, it would lead to a reduction in pay, which you suprisingly support - that amounts to walking into your boss' office and asking for a pay reduction.
I just think it’s useful to try to understand the effects of policies not just on me, but on society at large. For example, I might observe that a tax increase hurts me personally but helps others. Are you saying that instead I ought to come up with reasons why the policies that are bad for me are also bad for everyone else, to better lobby for myself, or perhaps so I feel less cognitive dissonance?
> That's the point. The effect of cheap construction labor only affects the rent prices in some areas, in other areas owners pocket the extra profit - while making construction jobs largely untenable except for immigrants coming from poverty.
> I suspect something similar could play out in the software field.
So in a region where home supply is constrained by density restrictions, there’s surplus extracted via high rents, and some of that surplus could go to construction workers (and they ought to fight to get a good wage so the developers don’t get all the surplus). Great for the construction workers, but bad for renters. In the analogy, software engineers are the construction workers and everyone else is the renters. Given the high market value of my wages, I will negotiate hard to get my share. At the same time, I can acknowledge that there would be benefits to society if there were more people qualified to write software, even though it would lower my personal wealth.
Currently there seems to be no mechanism increasing construction workers' wages or lowering rent. The extra profits are kept, and society isn't better for it (if it were, I'd be all for it like you). The reasons for this are fundamental, partly explained in an economics class, partly the result of imperfect markets and practicalities (a rich landlord might not immediately go out and build new real estate that would lower rents overall, he'd just sit on his boat).
My final point therefore is, let's keep IT salaries high for now.
In areas absent density restrictions, the US is indeed great at building tons of housing, developers don’t make huge profit margins, and rents are low. Is your argument that society would be better off if there were less construction workers, so housing costs would go up, allowing each remaining worker to be paid more and less total homes to be built? My opinion is that the higher wages for construction workers are nice, but they don’t nearly make up for the undersupply housing (we see both of these phenomenon in the Bay Area - higher wages for construction workers, who still can’t afford a home themselves).
You argued that software isn’t good for the world, so society wouldn’t benefit from more of it. This is a perspective I don’t usually get (especially here on HN) and I’m interested in hearing more. My belief is that software is part of a centuries-long trend of technology/productivity advances that have brought billions of people out of poverty, the continuance of which is necessary (although not sufficient) for a post-scarcity utopian future. I don’t know if we will ever invent a replicator, but I think more software engineers increase the odds!
This legendary meme that everyone in SF is making 300k and drives a Ferrari has taken on a life of its own, and now we even have people outside of the Bay Area believing and repeating it. This is an outlier compensation for senior people at outlier companies. Anyone would jump ship for it, but it's not like hundreds of companies are handing out these salaries at the airport when you land.
Is it? Not sure about 300k, but 200k seems within the realm of realism. I've interviewed at a few Bay Area companies (I'd say they are "mid tier", not FAANG or top tier) and expected comp ranged around 200-250k.
That said, I was actually surprised at the lack of expensive cars driven by SFBA techies. I interviewed at Netflix, which I know pays astoundingly well, and the parking lot had a distinct lack of luxury brand cars. The exception seemed to be Teslas - I did see a bunch of Model S's and X's (Model 3 had not come out yet at the time).
Compare that to many East Coast suburban office campuses of non-tech companies that pay pretty poorly - you still see tons of mid to high end BMWs, Benzes, etc.
> Lots of large companies are competing for talent, and if you can't pay bank salaries, you're going to have a hard time.
Sounds like Economics 101. The curves of supply and demand meet at some point. Some customers can't buy (enough of) the thing because the price at the intersection of the curves is too high for them. Yes, that's how free market works.
There are things I can't buy, because they are too expensive for me. Or I could buy them, but I would have to sacrifice something else I need more. This is considered normal. When some company can't afford (people like) me, it's the same thing.
They would like to have cheaper developers. Sure, who wouldn't? There are also things I would like to have cheaper. Same problem. Why am I not entitled to get things as cheaply as I want, but my potential employer would be?
In my experience many companies prefer to whine about the shortage of talent, even if they could easily hire another person by increasing the offered salary by 50%. But they won't. Then they make millions of profit, which shows that the money was there, it was just their decision not to spend it on salaries. Okay, fair, it's their money, their decision. But if it's your decision, own it, don't pretend there wasn't an alternative.
Make it known that you offer double of the market salary, and you will be surprised how many people are out there in the "empty market". At least I never heard about a company that offered lots of money and couldn't find a person for the job.
That's all well and good, but that's not the point I was trying to make. Companies aren't having a hard time finding talent because developers are so annoyed by coding interviews, they're having a hard time finding talent because they can't/don't want to pay high salaries.
So developers go to work on finance and ad-tech, and not on normal, boring, only-making-three-times-the-national-average-salary stuff. And those companies then complain that they can't find developers. Because whether they have a coding challenge or not, they can't find developers because money.
And those small companies would be able to compensate the difference in pay with "partner" status, i.e. making an employee an unfireable stakeholder that can live off profits by the time he or she turns say 45, but owners of said small companies are too greedy for that. I'd argue that opportunistic at-employement with 500k/year at Big4 is worse than guaranteed 250k/year that nobody, not even the company's owners can take from you.
I disagree with this, somewhat. Adam Smith, in Wealth of Nations, reminds us that there are a lot of factors that go into determining the desirability of a certain job, and that wages reflect this. The butcher earns more than the baker, Smith reasons, in part because most people would prefer the work of the baker. You have to pay someone enough more to convince them to be a butcher.
There are a lot of jobs that pay less than software engineer, but would be more desirable provided the pay isn't too much less. For example, someone might prefer to be a dental hygienist, figuring that the work is flexible, steady, and there's less of a possibility of age discrimination later in life. This person would become a software engineer for twice the dental hygienist salary, but not for 10% more.
In a free market society, it's not up to us to make that decision. If you want to get software engineers for in San Francisco for 10% more than a dental hygienist's salary, and you can't get any, that's the market's answer.
Corporate capitalists have discovered that if you don't like the market's answer, have the government ask for you.
> The butcher earns more than the baker, Smith reasons, in part because most people would prefer the work of the baker. You have to pay someone enough more to convince them to be a butcher.
And the surveillance capitalism developer earns more than the entire village. Ergo the market commands us to all develop surveillance stuff so Saudi Arabia can spy on dissidents and then chop them up in their embassies.
But again, that's not the point I was making, it's weird that multiple people understand my comments as "we should remove the free market". Me saying that it's the market that is making it hard for normal companies to find talent is only the background for what I'm actually trying to get across: It's not the coding challenge that is making it hard to find developers.
A company that isn't in a hyper-profitable space like ad-tech or surveillance (watching employees during WFH is booming) or fintech (figuring out new ways to make people part with their money) won't be able to attract talent by removing a coding challenge. It's just not an important factor in the equation.
I will engage with your claim that coding tests aren't making it hard to find developers. I think it is a more important factor than you do.
On a personal level, coding tests are a deterrent for me in applying for new jobs. I have my old copy of data structures and algorithms, and I've gone through "Cracking the Coding Interview." I actually enjoyed it at first. They're interesting and challenging problems. If there were an exam - maybe something similar to the actuarial calculus or probability exams - I'd probably be interested in studying hard for it and passing it.
However, I'm no longer interested in re-loading this information into short term memory in order to take a whiteboard exam. This may be a facet of how my own brain works vs others. I just don't walk around prepared to figure out which elements of a set, when combined after permutation, are equal to an existing member of the set. Or how to find all square sub-matrices with a positive determinant in a larger matrix. I can do these things - I've enjoyed doing them - but for me, it takes a lot of training and prep to re-load it in a way that I'm ready for the 45 minute whiteboard exercise. If it were a proper, lasting exam, I'd be happy to do it, but it's just too random, uncertain, and capriciously scored for me to want to do it anymore. I will note that some people don't have thae same trouble I have with this, and - I really mean this - this may actually indicate that they are better candidates than I am. In short, I need to acknowledge that I may certainly not be the "talent" that anyone is missing out on.
Money makes a difference too, but the reason I brought up Wealth of Nations is to draw your attention to the section "Of Wages and Profit in the different Employments of Labour and Stock" http://geolib.com/smith.adam/won1-10.html
I would recommend the entire article, as Smith breaks down the various factors that can lead to higher or lower wages, but you don't really need to read the whole thing, here's smith's summary:
"THE five following are the principal circumstances which, so far as I have been able to observe, make up for a small pecuniary gain in some employments, and counterbalance a great one in others: first, the agreeableness or disagreeableness of the employments themselves; secondly, the easiness and cheapness, or the difficulty and expense of learning them; thirdly, the constancy or inconstancy of employment in them; fourthly, the small or great trust which must be reposed in those who exercise them; and, fifthly, the probability or improbability of success in them."
Within this matrix, software engineering does have a lot to recommend it. For example, Law requires a very expensive and lengthy graduate degree, whereas software engineering doesn't, even though most successful SE's generally do have some kind of undergraduate, and often graduate, degree in rigorous fields that require a lot more math than law. How people will respond to this (along with other aspects of these two fields) depends a lot on their own personal wiring as well.
Within this matrix, yes, I do think that many people find in person, oral exams, at a whiteboard or at a computer under scrutiny and time pressure, on topics that they often don't know well in advance, evaluated by people whose credentials and skill they don't know in advance... yes, I think many people find that a deterrent, either for applying for new jobs, or entering the field in the first place. Many people consider these on-the-spot exams extremely stressful, especially when they are in person, at a whiteboard. And we go through days of it, and a new round of study, every time we apply for a jbo. Tech interview exams are not the only factor making it harder to attract candidates, but neither are any of the multiple factors identified and elaborated o...
In my experience the talent shortage is a giant myth - there are two thing going on: companies that can't hire or won't pay, and companies that can, but wish it were cheaper.
Evey job I've applied or hired for either a) had lots of candidates with a very selective process, or b) telegraphed that you should avoid the company or didn't pay well enough (and often still got plenty of applicants who wanted their foot in the tech door despite not being capable).
My current company would claim there's a shortage, but they actually don't know how to hire and telegraph that you probably don't want to work there in a variety of ways.
I don't know of any software company that would hire an introverted person who can't articulate themselves for 20k a month. This is how micro-service architected fully optimized solutions that solve the wrong problem happen. Do you really think the high value of a developer just comes from being able to pass coding challenges when there's millions of them out there?
Eh, each of FAANG will do exactly that (as will other tech companies!). To be charitable to the parent commenter's point, they probably mean articulate in two senses:
1. Speaking perfect English, both fluently and without a heavy accent.
2. Talking about technical topics or other shibboleths with the same cultural affect an American would, as opposed to a foreigner. For example, many Indians will call a phone screen a "telephonic interview", and I've personally seen Americans react poorly to that.
I know lots of developers who are very well paid and very effective in their jobs, but they don't speak English as well as an American and would stumble at certain cultural nuances.
Thank you for being charitable. I speak English well enough to fare in FAANG as senior engineer.
But I always notice how a native English speaker almost always speak perfect English. They pick the right word to use and many other small things that make their English superb.
No worries :) You speak English well enough that I agree you shouldn't have any problems communicating in a technical discussion. Personally I find thick accents to be a bigger obstacle to communication than stumbles in grammar. But realistically most engineers will be professional and charitable enough to work around either issue as long as you're earnest.
You seem to be confusing english speaking ability with the ability to communicate and articulate well. Communication is not about being able to speak english perfectly, it's firstly about listening and understanding the context of requirements. It's about being approachable enough so that the clients/stakeholders want to work with you, and clarify things with you, which in turn leads to a product which is close enough to something that they needed.
That's perfectly in line with expectations for a new inexperienced hire at a FAANG. If anything, it's a little high for entry-level even at these companies. Obviously much higher numbers are attainable via career progression or already having such a job to negotiate from as leverage.
I've seen messy devs who absolutely can't articulate themselves, to the extent that when they tried, everybody else would keep awkward silence, and yet they made 60k/month.
As someone who had administered dozens of "objective" coding challenges to non-native english speakers... I can guarantee you your language skills will negatively impact your performance. So those tests aren't as objective as you think.
It comes up with questions that are as simple as "Create a function that deduplicates an array of integers"
> Some people are born blind too. Shouldn't we adjust our hiring practices for them too?
Not only should we adjust our hiring practices for blind applicants, there's a legal obligation to make "reasonable adjustments" for these candidates under the Equality Act! *
Dude - you're not even making $180k/yr (gross) at FAANG and you've been there for 8 years?
What is your position...? I earn about $9k/month (net) at $170k/yr (gross) in CA as a software engineer. So, I'm already past the 2/5 mark and I'm just at a startup that doesn't even pay well! On top of that - you should be getting stock that translates directly into a good chunk of cash - not just a salary component. (Unless Netflix I guess)
Check my edit! I was talking about my actual takehome salary (post-tax, not counting RSUs). Total gross comp is just north of $300k, which is pretty much in line with levels.fyi's average for that position (a little below - which tracks. I've only been promoted into that position just under a year ago, so I'm probably under-average across the whole population, despite being in the Bay Area so I'd hope to accelerate past that average soon).
> Instead of a coding challenge, ask them to spend a short time building something they enjoy, and talk about what they did and why.
To me this would be a MUCH more difficult task. A blank canvas is much harder to work with, and downright frightening. Constraints foster creativity.
Having everyone code the same thing helps the interviewer (who is also very busy and has many other things happening in their work life) gauge the different levels of the people applying easily by comparing them on the same task.
Then pay the candidate! It might seem ridiculous but consider how much time is spent by interviewers, hiring committees etc, giving some fraction of that out even to utterly worthless candidates might not be the worst idea in the world. A side effect of this is that suddenly the cost of a bad screening becomes more intuitive
heavily dependent on the job as well. not all software engineers are tinkerers solving basic problems in their lives.
I code state of the art motion planning, localization, and robotic algorithms. some of this requires more than a day of tooling around at home building a project
I got chance to appear in a couple of coding challenges while applied for jobs. In one test I failed. It was a typical hacker rank riddle. The other I passed as I see my test ran properly yet I got a rejection letter. Another HN based job interview I was asked for a challenge but I simply refused.
That seems very taxing on the company. I'd rather screen the candidate before committing to anything via interview/coding challenge, etc... The take-home coding challenge seems like a good idea -- less pressure than coding during the interview and allows the interviewer to assess the "real life" performance.
> Ask them to build something – anything!
That's great, but a big part of working at a company is building something that you don't like or don't understand well -- talking to stakeholders, customers, figuring out specs, etc -- something that a good take-home exercise can assess
The probationary period seems incredibly stressful for the employee. If I am currently employed I would never switch to a role that uses a probationary period as a significant evaluation step because of the risks of leaving my stable employment, and not passing your probationary period. As an employer I wouldn’t want to limit my selection to people that either are unemployed or have a higher personal risk tolerance to handle such situations.
Exactly. Not to mention younger engineers will probably work 24/7 for 12 weeks while constantly worrying they could be fired tomorrow because they didn't fix some bug fast enough.
Not to mention it eliminates anyone who already has a job from the candidate pool. No company I know would be OK with you also working full time at another company simultaneously.
> That seems very taxing on the company. I'd rather screen the candidate before committing to anything via interview/coding challenge
So having a probationary period is too expensive for the company, but a several hours coding challenge for the developer is a OK?
In my experience when a candidate was a bad fit, it was because of soft skills/personality traits not technical chops (which are much easier to learn than soft skills). Are you screening for soft skills in the coding challenge?
The developer is eventually joining the company, no?
I didn't like spending probably about 5 hours total on coding challenges last time I interviewed, and I've been internally opposing them, but as the candidate who interviewed, I'm much happier having dealt with it once and gotten over it than having probationary employees show up on my team for a week every so often and having to drop what I'm working on to work with them and evaluate them.
> In my experience when a candidate was a bad fit, it was because of soft skills/personality traits not technical chops (which are much easier to learn than soft skills).
This is a good point, and argues in favor of traditional interviewing (establishing a rapport with the candidate etc.) over any sort of "challenge" IMO. I don't think I'll learn more about soft skills in a week than I would in an hour - the complex stuff generally takes several weeks for the new hire to be properly onboarded and contributing, anyway.
It's a tough problem, but eventually you have to demonstrate some sort of competency. White board, open source contributions, homework. There has to be some way of demonstrating you know what you are doing.
I'm a consultant so I've been through tons of interviews at this point in my career (21+ years).
If I had my choice? I'd pair program with my potential teammates on a bug/feature for the product I'd be working on.
Give me something real that's been in the backlog for a while, and we'll pair on it for the allotted time.
This sounds good, but it will not work for many jobs. Fixing a bug in a code base that you don't recognize is more difficult that a well defined coding problem. Candidates will be limited to the technologies and problems which they are already familiar.
This approach has some value, but it can't be the only way to evaluate candidates.
If you make about $100 an hour, a six hour coding challenge costs you about $600.
If a potential employer asked you to spend $600 to fly out and see them, on your dime, would you do it?
I would do it IF I desperately needed a job. I did it back during the dot com crash; I needed a job badly and I was willing to roll the dice.
In 2020? Oh hell no. There's a million companies who are hiring, and an employer would have to be exceptionally special for me to waste $600 on the chance they'll hire me.
Companies fly people out all the time (well, pre-COVID). They pay for the travel expenses, but not your time. I personally find this much more of an imposition than spending a few hours writing some code in the comfort of my home.
The only time I've bothered with coding challenges is what I was out of work, and zero of them, even some I spent hours working on, have ever lead me to a job. My current employer was fine with an example from current open source project I contribute to.
I agree that code challenges aren't ideal. But we've been struggling with parsing resumes and titles to identify the competency of developers. Our current coding challenge isn't super hard and based on the type of code you'd see in our product. It asks you to read a simple javascript function, explain it, and then implement new functionality that can built by refactoring the existing method. It's about 20 lines of code and 50 lines of documentation. You can use any IDE that you like.
We've had many candidates that look strong on paper (3+ years of experience, senior titles, etc) come in and struggle. Struggles ranged from basic code literacy (think for loop to traverse a nested json object) to basic programming like writing a function. We've especially noticed this more with coders coming from purely a javascript and front-end background (React/Angular, etc). They understand control structures like if/else but struggle with loops, setting variables, and basic higher level concepts like breaking a problem into smaller functions.
My philosophy is for junior candidates, any level of competency is ok. Strong mentoring and experience on job usually can allow anyone to thrive and so you can prioritize soft skills and culture. But for more advanced roles, I don't know, it's hard to parse out the quality candidates without having some sort of technical interview. Definitely would love to hear how other's learning and experiences.
Another we'd love to hear opinions on. Based on a recent study where candidates are more likely to succeed if given private space and time to work on the problem, we'll disclose the coding challenge about 15 minutes prior to the interview allow the candidate to read it before joining the call. We struggle with the right amount of time provided for private review is. We want to be fair to all candidates and not implicitly favor candidates who have more free time to dedicate to the interview process than others. but at same time provide some sort of reflection space outside of a cold start to the challenge.
I've heard a variant of this repeatedly. A company justifies technical exam style interviews on the grounds that many people who look good on paper "can't code". Meaning, can't program fizz buzz. Can't write a method or function. Can't write a loop, or an if/else conditional.
But I can do all these things. I've gotten bounced from my interviews because (this was the official reason, anyway) I didn't get far enough with the coding in finding all matching subtrees in a binary tree, in 45 minutes, at the whiteboard.
Companies should feel free to hire based on my ability to do this, they should not say they were just testing basic things like lists, loops, and conditionals, or even basic data structures and algorithms like binary trees or hash maps. These exams go way beyond that.
This may not be the case for all, and it sounds like it is not the case for your process. But in the broader industry practice, I smell a motte-and-bailey fallacy here.
In our case our its based on our actual interview experience with candidates. We've been actively evolving our process and calibrating expectation based on the candidates that have come in. We've actually simplified the interview question considerably (as I mentioned the solution doesn't even require you write new logic, you can refactor existing logic that's provided). But we still see candidates struggle with the basics.
To be fair this could also be a reflection on our ability to read resumes and experiences and our initial phone screen. But it is hard. My first job was at fortune 50 company, you could have the title of senior software engineer and not write a single line of code as part of your job.
That being said i don't won't diminish your experiences. This is just our experience from the other side of the table. Also we don't have an HR department (we're a small startup). The interviews are conducted by engineers in the company and we try our best to empathize with candidates and be conscious of creating an interview process that we'd feel comfortable being a part of as well.
we're not asking programming puzzles. i described our process above. struggling to define a function and basic code comprehension (e.g. tracing a for loop that traverses a json object in candidate's preferred language) is a strong signal that candidate is not a good fit a mid to senior level for us. If it was a junior level position, sure you can learn on the job.
in our case, the problem is a highly simplified version of functionality in our code base and is close to being representative of work they'd be doing if hired.
i actually don't disagree with the prior poster. Technical interviews are highly variable and it's frustrating to go through poor interview processes. I've got bad interview stories as well. Being on the other side of table now, we're always interested in learning about better approaches hiring. Would love hear what has worked well for others.
For advanced roles I'd suggest separating out the evaluation of coding ability from the evaluation of a particular language. Let the candidate pick their language when you want to test how well they can write new code, and then do a separate evaluation of how deep they are on the language and technical stack you are using.
I agree with the sentiment. Honestly we don't check for syntax or even require the code to compile. We let the candidate write puesdo code which usally isn't super far python and javascript implementations of our challenge. We try to pay attention to the types of questions the candidates asks about context of the code and choices they make in designing a solution.
That being said i'd be curious how you evaluate general software engineering ability (e.g. writing clean abstractions, decomposing monolithic methods, and general design). I feel most system design evaluations are a bit too high level or overly broad (e.g. design a library application or make a social network) and coding challenge are too low level.
I like to think of the higher level as creative problem solving.
Are they (me in this case) going down rabbit holes? Are they painting a clear picture? Are they communicating that picture? Are they suggesting edge cases? Are they minimizing bloat? Is it too complicated? What are the most important aspects of the design in their eyes? Can they argue for them? Do they understand where a piece fits and if not, do they understand that something has to go there? Can you both step back and definitely outline a logical flow?
There's not always one solution, but the approach counts.
Lower level is logical problem solving.
Can they reason the problem space? Do they start naively? Can they iterate? Can they optimize? Can they explain each of these steps for future iteration? Do you understand what they're writing? Would you work on that code yourself?
There's not always one solution here either, but 90% is getting a naive solution that can be optimized by taking a second look. I feel that the naive should be found in 5-15 minutes and you can spend the rest of the time talking about how to optimize.
I know I'm rehashing what you probably already know, but at a certain point interviewers are just playing games.
If you've found someone you want to work with who can also reason about problems in a timeframe you see fit, well... that to me looks like gold.
I like these questions because that's how real systems are going to be built.
Take-homes are great, but there is just so much effort wasted in the process if the product can't be understood.
One of my previous managers would set up the white board interview for a candidate by giving them a simple problem application (not a course-grading system but similar) to draft out on the whiteboard an initial design. Then they would add in additional details/requirements in about three stages. So the candidate at some point would typically have to back track and revise some of their initial design to accommodate the new requirements.
It was a good measure of how well they communicated, how they dealt with undoing some of their prior work, and the process was fairly representative of the type of work involved in the position.
If you're interviewing someone for an 'advanced role', you can talk to them and find out if they know their shit. If you can't then you shouldn't be interviewing them. Senior devs can see where other devs are at pretty easily.
I think one can be creative about it. Applying the principle "nobody wants to buy a 2 inch drill bit; what they want is a 2 inch round hole", the people in the company do not want to give you a coding test, they want to hire a competent candidate. So convince them you are competent for the job in other ways if you do not want to spend the time doing the coding challenge.
One way of going about about it is to find former colleagues that work for the company you are applying to, and ask them to provide references on your behalf.
Many of the author's suggested alternatives clearly fail on the stated reasons why coding challenges are bad. Asking pertinent questions is clearly arbitrary; probationary periods are non-inclusive to candidates who are relocating or need a visa; asking candidates to "build something - anything" would be substantially more time consuming for engineers in many specialties. (How am I supposed to build a small demo app to show off my systems programming ability?)
It's hard for me to escape the conclusion that the author's just trying to universalize their personal preferences.
I've done 5 coding challenges, I deemed each one a success on my end, none resulted in a job. A few resulted in a third interview which revealed numerous additional red flags.
At this point I consider a coding challenge a deal breaker. I'm not doing one, and if you ask, there's a near certain chance I wouldn't want to work in an environment that thinks they are a good idea.
If you're interested in my abilities, I'd be happy to spend an hour walking through something I've already worked on in the past.
A family member is a chef, and for new jobs you have to "stage" [0] where you come and cook for a (unpaid) shift to see if you are a fit, have the skills, etc. To me this is exactly the same thing, and I expect this is very common across different kinds of jobs.
I'm of the opinion that if there needs to be some objective technical challenge to determine a baseline of competency (and there are very many, very strong arguments for that) then there need to be standards.
I've encountered so many variables (no pun intended) in technical interviews that I've virtually stopped preparing. There's hardly any way to prepare when the field is so large and you never really know what you're going to be tested on.
I've had people challenge me on theoretical system-time timezone conversions (and incorrectly mark my correct answer wrong "because of physics"...), beside reimplementing native JavaScript functions, beside complex algorithmic problems, beside toy problems like building a binary calculator. [That's great if you learn I can rig a web page for adding 100 and 101, but how do you know if I know anything about databases for this back end position]
And those interviews were all for working with web tech, in a couple of the cases just front end positions.
There needs to be a common, limited standard and/or domain-specific standards.
It's really counter-intuitive to me, and I don't understand how it benefits the hiring process, if a company is looking for a web-back-end engineer for what is essentially a complex crud app and wants to propose the candidate solves a problem from a domain they've never worked in—and marking them on the correctness of the answer, rather than the work process.
I'm for objective, technical challenges. But I've seen enough that it seems apparent to me that it's broken.
(addendum: I've had several good interviews as well, and good interviewers. I had a good one recently as well that was fitting for the domain. They're not all bad, but most seem to be in my experience)
They're really not. It becomes a cat and mouse game where the interviewers know which pool of questions people are studying so they try to invent new ones to prevent people just memorizing the answers. The questions thus end up becoming more and more obscure.
yeah but they're variations on the same stuff. it just allows gaming of the whole process where if you solve enough of those types of questions, you have a better chance of getting in.
Some are variations but some are just more obscure. They definitely get into very specific greedy algoss or things like prim's/kriskals/tarjans. which can be pretty much hit or miss based on if you've seen it or not
Oh no. I've wasted plenty of time. And so have they by interviewing me and proposing ludicrous problems well outside of the problem domain of the role I was interviewing for.
I understand some problems are meant to judge your thought process, others are meant to measure your basic literacy. I'm not talking about those, however.
My experience in interviews has been that most of them are engineers pulled into the process who searched the web for "software developer interview questions technical" and cobbled together a few disparate options without a moments concern more.
Then again, those kinds of interviews have certainly cemented my desire not to work there.
I interview a lot of people for frontend positions. I ask one two-sum question to see if they have some knowledge about data structures and simple algorithms. Then I ask them to build something trivial in React, which is what we use. If you can do those things, you're probably fine. The number of people who can't is truly shocking.
>>> Given an array of integers nums and an integer target, return indices of the two numbers such that they add up to target. You may assume that each input would have exactly one solution, and you may not use the same element twice.
Seems trivial enough to me... except if the interviewer forgets the hard constraint that there is exactly one solution, then that's a shitstorm of edge cases.
I'd criticize the problem for asking for the positions of the numbers rather than the numbers. That's just super annoying for no reason.
Does the problem actually have that constraint? If so then yeah it's significantly easier. I thought the reason you'd have to use a clever algorithm is because a brute-force solution to find all index pairs would require n^2 operations, where n is the size of the input array, because you can't stop after you find a single pair.
I don't ask candidates these kinds of questions in my interviews. For a coding interview, I'd normally ask them to implement something like backend rate limiting logic for an authentication service, and continually ratchet up the complexity on each successful implementation until the end of the interview.
I was asked the two sum problem in an interview once, but I forget if it had this constraint or not. This seems much more reasonable.
Technically, if all values in the array are the same value X, and the target is 2X, you can't do better than O(n^2), because just writing writing all valid solutions is O(n^2).
If you can create a new array containing indices to the original array, sort the new array, and use two pointers starting at the beginning and the end of the sorted array to find a solution, it takes O(n×log(n)) time. It is not necessary to assume that the solution is unique, assuming it is okay to simply write one of the possible solutions.
If all values are between 1 and k, and you can make a k-sized array, you can find a solution in O(n+k).
No matter what you do, you can't get better than O(n).
The important thing is that there is more than 99% chance you will never have to solve this problem in your job; and even in the case you would, there is 95% chance that the O(n×log(n)) solution would be acceptable regardless of whether it is the best possible or not.
If we're being very charitable I guess the idea of using a HashMap for an optimal solution might slip your mind but even a freshman in CS can just loop twice over an array and solve it.
I might be misunderstanding the question; how many pairs can there be? You can't just loop over the array twice to find all qualifying pairs of indices, if there are multiple. You'd have to loop over the array n times, where n is the size of the array. Isn't that why the question is difficult? There doesn't seem to be much incentive to using a hash table, complexity wise, if you could just solve it in O(n) time and O(1) space by naively iterating over the array.
He meant "write two loops" when he said "loop twice". It seemed obvious to me that he meant that but I can see how if you're used to more unambiguous language it could be confusing.
Yeah, you just do it in O(n²) worst case because the problem is O(n²) to find all pairs that match because there can be O(n²) pairs.
However he meant one pair here too. Pretty easy to solve. Did it in under 30 s in Python on my broken cellphone. If there are multiple pairs, just replace return with yield.
Same here. Absolutely shocking how many people fail at basics, yet they pose as mids or seniors even. I'm not even talking about CS, I'm taking about simple React-based assignments designed to show you know (at least) the basics and takes as little as time as possible to be fair from our side not taking too much of interviewee's time. Take what you want from it, but I've seen it mainly in web dev related positions.
Yes, I see people fail at basics in interviews all the time. But I keep wondering how much of that is people being poorly qualified, and how much of it is interviews being poor indicators of true skill.
I've seen people barely sneak through technical interviews, whose subsequent performance justified the concerns I had at the time. I've seen others hired against my personal recommendation who turned out to be brilliant at their jobs.
Of course. That's the primary reason I'm personally against tasks on the interview itself or whiteboarding. That's usually a shitshow. We first do an interview where we talk a bit about our company and the interviewee, we get to know each other out a bit and try to find out if we're a good match for each other, based on the talk itself. If that's the case, we give a small (really small and basic, unless a senior position) assignment to take home and get back to us in five days. People have schedules, obligations, we understand. During the period we're always available for any questions, doubts, whatever you need. Assignment itself takes a couple of hours at most for someone that knows / has used the tech. At home, you can use google, stackoverflow, friends, books, I don't care. No one cares, as long as you show up with a consistently-looking code that works and that we can comment together on. You know, same as at work. Something to talk about for a bit, if at all. It's the basics. A damn good filter, it turns out.
> ...if there needs to be some objective technical challenge to determine a baseline of competency...then there need to be standards
After having been on both sides of the table, I’ve really come to appreciate the system of licensure used in the trades. I don’t have to quiz an electrician on the electrical code or ask them to wire up a dog house. I also know they have requirements to keep their license active. I’m not sure what that process could be like for software, but it would sure be nice to eliminate the enormous collective waste of time spent (poorly) establishing candidate competency (especially when they have to do it for every single company).
I usually tell interviewers I don't do per-interview assignments. I'll often e-mail them a list of current open source taks I'm working on, a paragraph with good requirements or a linked issue from an issue tracker. I'll list the programing language and framework and ask them to select from the list. I'll complete my task, with tests, and do a full presentation.
Universally I get turned down. "You have to do our assignment." Oh and I'll usually offer to do their assignment for $300. That doesn't go well either.
Only one company took me up on this offer, and it's the one I work for. (They didn't even ask me to complete one of my tasks, they just looked at the project I authored and said that was fine; and scheduled a full interview).
The few times I have actually done these assignments, I've never gotten the job.
Funny how he starts saying he never had a whiteboard interview and he never gave an interview, yet he has such strong opinions on everything that's wrong with it.
edit: read to the end, a lot of the points are absurd, easy to realize after a couple interviews.
As someone who has been programming for 40 years and had a decently successful career but is struggling with these challenges now, I can empathize.
It's not so much what the challenge is -- it's how one is asked to accomplish it.
Sitting in front of my laptop for 6 hours while people ask me random programming tasks while they watch me on camera and see every keystroke on screen while I'm editing is not how I work. I work by choosing from active tasks that fit the "space" I'm in, by multi-tasking, etc. - if I get stuck on something in the "real world" I'll go work on something else that needs to get done, etc. You'd be amazed at how many interviewers simply refuse to ask you another question in lieu of or even in addition to the one the first wanted to ask, for example.
I require them because candidates lie, not always intentionally.
The unfortunate reality is that it can be really tough to fire low-performers. I NEED to know that a candidate can do the work. We go out of our way to make our ask reasonable, relevant to the role, not spec-work, not time consuming, and well defined. It's not uncommon for us to learn that the candidate can't actually do what their resume says they can or for the candidate to learn that they grossly overestimate their own skills.
I recently talked to a Python/JS expert that led a project at a major agency to scrape a network of websites, find and extract specific data, and deliver game changing competitive intelligence. The candidate ended up admitting that their role was limited to copying and pasting a pre-built code snippet into the dev tools console and that they couldn't actually read or write any code. The proof is in the coding challenge.
Most of the alternate approaches have their own severe failings:
Ask pertinent questions - a lot of candidates are bad at immediate summation and dissemination of information. There are many good engineers who have not learned how to sell their work, themselves, or in general communicate to the intent of a question. This is an invaluable skill, largely orthogonal to writing code, but it's a skill that's often learned with experience. Learning to ask level appropriate questions is hard and evaluating answers consistently (if not objectively) even harder. It is also largely insufficient in gauging programming ability. I've had co-workers who can talk about advanced architecture-y topics seemingly competently but struggle to write even relatively simple code.
Probationary period - I've never seen this implemented (excluding internships which sort of fall into this category) and it certainly sounds like a nice option to have available but I would bet many candidates would be afraid of a short probationary period (say 3 mo) especially if they have to move their family for the job. Being a professional and being told you are on a probationary period is stressful. At many large companies it can easily take more than 6wks to get up to any sort of productivity.
Walk through existing projects - most people (vast majority) do not have any sizable or clean or even "interesting" side project they would want to walk an interviewer through. Unless you worked on open-source for your previous employer (again minority of people) or on your spare time, this is not something you can get a strong signal out of.
Ask them to build something - err, I don't see how this is that different than a coding challenge. And again, not everyone has personal projects ready to be shown to an interviewer. For example, the game I'm coding some nights after work does not look anything like code I write for my employer; there are no tests, there's a lot of experimental and often dead code, functions are not neatly factored and thought out with an eye for collaboration, etc. I would not want to be judged by this code for an interview.
Pair programming - this sounds like a coding challenge but with a more empathetic interviewer. I don't think this is an alternative, just a mild augmentation of a coding challenge.
Focus on ability to learn - companies do this but it's usually considered supplementary to coding ability. Some companies explain their product and try to see if the candidate understands where the value of the product exists or at least where some of the challenges are. This could be spun out into a product type interview or a system design interview depending on intent.
Alternatives to white-boarding or do-at-home project interviews:
- Provide a small existing but functional code base and ask the candidate to implement additional functionality. Downside is this takes a lot more effort to prepare and are limiting the candidate in regards to language they want to use.
- Provide multiple options for a coding project or question and let the candidate choose which one they want to solve. This would hopefully reduce some of the luck factor of getting a question you know how to solve.
- I can't think of any others, but I'm sure they exist
One of the big desirable goals of interviews are evaluating candidates in a consistent manner. All interviewers have biases and these biases vary between the interviewers. Trying to account for that is, I believe, important but also very hard. Using the proposed alternatives would make it even much harder.
> "I’m writing this as an interviewee, having never been on the other side of the interview table ..."
> ...
> "Most candidates will have side-projects or work they’ve done for previous employers."
As someone who has actually done a lot of interviewing, I can say with great confidence that this is simply wrong. At my current employer, we have a "homework assignment" as part of the interview process. We tell candidates that they can point us at an existing project of theirs instead of doing the homework. Few people take us up on this because few applicants have any substantial public side projects or FOSS work they can share.
And sharing "work they've done for previous employers" is a great way to court legal trouble. They shouldn't share it and as an interviewer I don't want to see it!
We use a coding challenge as well but I like your addition of "auditing" the coding part and using existing open source or side project work. Bonus points if it uses some of the target technologies too!
The author doesn't appear to understand the economics of the hiring company; it's about time & effort to filter and the cost of making a bad hire. I cannot give every application 1-2 hours of review if I get 100+ resumes. I can't give every interviewee a half-day assessment to get the very poor signal/noise ratio answers to questioning methods suggested. I can't hire easy and then fire easy. I can't invest weeks of work from myself and other mentors on non-ideal potentials. I know this means I pass over great candidates and that really sucks and is totally unfair. I'm sure that's happended to me as well. The problem is nothing presented here is a better alternative.
> I cannot give every application 1-2 hours of review if I get 100+ resumes.
This is perhaps the biggest misconception of hiring, that out of some misguided notion of "fairness" or "objectivity" you have to treat every candidate exactly the same. In theory it seems like a good principle, but the road to hell is paved in good intentions. In practice, this results in arbitrarily designed standard tests which are only superficially fair, and systematically weed out people who aren't good at test taking or auditions.
Go through all the resumes quickly, pick out the top candidates, and start from the top! Focus most of your time on researching your few top candidates, and if one of them seems good, then hire them, and end your hiring process. You may only have to interview 1, 2, maybe 3 people at most. This saves everyone time: your own time, and the time of the 100+ people who would otherwise be interviewed and rejected, because there's only 1 job opening.
Obviously if you interview your top candidates, and you find them lacking, then keep working your way down the stack. But going through the entire stack of candidates is a huge waste of time, a pointless exercise, totally counterproductive.
Don't do "screening" interviews. Only do "hiring" interviews.
Generally you don’t get all the resumes at once, this is literally the secretary problem. Unfortunately you need some objective standard of measurement across candidates that is static over at least months, and ideally takes a minimum of you and your candidates’ time. Though imperfect, real-time technical questions are a pretty good solution to this problem, especially since a lot of the best engineers prefer them to take-homes.
Also, anecdotally after interviewing hundreds of candidates, I’ve found resumes to have a relatively high false positive rate. Surprising numbers of engineers look great on paper but can’t write working code for semi-complex problems when tested, even accounting for interview anxiety.
1. I try to make the parameters of the interview as friendly as possible — language of their choice, don’t have to worry about code style or computational complexity, or talk while they’re coding, just write working code that solves the problem. They can use google freely. I’m there as a resource to talk through their solution, answer questions, and help them get unstuck if needed.
2. The algorithm to solve the question doesn’t require a trick and there are many viable solutions so most candidates can verbally explain to me a working algorithm quickly, +- a couple edge cases.
3. Candidates often get visibly flustered or hung up on irrelevant details, or unsure what exactly I want for them, and I focus on noticing this. I try to guide them down a correct path if they’re close and tell them not to over-architect if they start doing that.
Despite all this, too manly candidates do not have a dependable metaprocess for writing working code - a common scenario is they write code, it gives the wrong result, they drop some print statements to debug, but can’t effectively trace the program to figure out why they’re getting the debug output and so they guess and try to fix the wrong thing, causing further problems. I know that some of this is nerves but it can’t account for all of it.
> I know that some of this is nerves but it can’t account for all of it.
Why not?
If you've never experienced severe situational anxiety, you may not know how debilitating it can be.
And then, maddeningly, perhaps just minutes after the interview is over, you return to your "normal" self, and start to think about what you would have done in that situation if you weren't nervous.
It's basically stage fright, which can be terrifying if you're not used to the stage.
I guess I'm generally going off of an interviewee's body language, which isn't entirely fair since I'm not a mind reader. And I'm sure I'm missing some cases of stage fright, which is why again I mentioned all the techniques above that I try to mitigate this, but I agree is a really unfortunate side-effect of the interview process that I don't have a fool-proof solution for.
Still, I do have some EQ. A lot of candidates exhibit visible anxiety and then get into a rhythm and calm down when they start coding. Or they visibly calm down once I start asking them questions about how they're thinking and coax them towards viable implementation. On the flip side, a lot of candidates actually exhibit overconfidence running in the wrong direction, which I try to lightly mitigate but can't prevent in all cases without giving an unfair advantage.
Are you sure? I mean the 10 years of success spent over various companies projects on resumes have a high false rate because after they took your coding test they didn't do as well as you thought they should? Did you hire any of them and did they out perform your initial assessment.
If you didn't hire them and compare them to new hires who have bad resumes but did well on your test how would you know?
You can't objectivity judge a candidate by your test results only. Otherwise your test is just randomly filtering people. Until you do that you don't know how effective your test is. You could be flipping a coin and getting the same results.
I’ve hired people who have not passed the coding screen (but did pass subsequent interviews) and they’ve been some of my best hires. But the type of failure really matters. Did they write generally correct code but ran out of time because the algorithm they came up with under time pressure was too complicated? Or did they fail to debug a list indexing off-by-one error, even after printing the list repeatedly? It’s sometimes a judgement call but I believe there’s a lot of signal in watching people write code in real-time
Agree with this for most companies. When at an airport we were government and had to be 'fair' it was about ten solid days of work to hire for one position.
This only makes sense if they want top talent. Usually they want cheap/easy to push around talent and for that you need a screener that filters out the best and whoever remains will have the strength to deal with bully managers or lack of requirements
How do you "quickly go through" a stack of 100 resumes and choose a handful of top candidates in a way that is not subject to personal (conscious or unconscious) bias?
I remember an interview training exercise, which also covered resume screening. Each person got the same stack of resumes, and the instructor said, "You have 60 seconds to pick out the top 3 you'd like to bring in for an interview. Find them." At the end of the exercise we found that we all basically picked out people who were a lot like us. White people picked out "whiter-sounding" names, women picked more women, etc. Also, Ken Thompson's resume was one of them (with a different name) and nobody picked him.
I think the point was that human eyeballs and brains are terrible at objectively or fairly pre-filtering.
> How do you "quickly go through" a stack of 100 resumes
Well, by quickly I didn't mean only 60 seconds. :-)
> in a way that is not subject to personal (conscious or unconscious) bias?
In my opinion, a common conceit is that it's possible to design a process that is not subject to bias. It's not possible. There will be bias, no matter what process you choose. It's thought that "quiz show" style interviews are "objective", but if you look at the results, they're clearly not helping. They're not increasing diversity. The poor results speak for themselves. So then some people look at the results of this so-called "fair" process and conclude that minorities are just worse at coding, because that's what the tests say. To me, that approach and attitude is even worse than just acknowledging personal bias.
The only way to try to counteract unconscious bias is to be conscious of it. Admit it, and keep it in mind while you're going through your hiring process. Keep it in mind while you're filtering the résumés. Keep it in mind while you're interviewing. But don't try to pretend like you can eliminate it entirely from the process, because then you're just deluding yourself and everyone else.
The commenter doesn't appear to understand the economics of the job-seeking candidate; it's about time & effort to keep your current job and the cost of getting hired at a bad company. I cannot give every company I apply at hours of coding if I apply to dozens of jobs. I can't give every company a half-day assessment. I can't get hired easily and then get fired easily. I can't invest weeks of work from myself on non-ideal potentials. I know this means I pass on great jobs and that really sucks and is totally unfair. I'm sure that's happened to me as well. The problem is that nothing presented here is a better alternative.
This is a snarky parody, but put together you are both just saying that you can only afford to spend any time on job matches likely to be especially good, but that you’ll spend a lot of time on the good ones. Which just means you both are in an illiquid or conservative hiring market that is out of your control and their control to change.
> I cannot give every application 1-2 hours of review if I get 100+ resumes.
Your process has already failed if you're considering 100+ people!
First, you should never be getting 100+ resumes for a role. If you are, you need to write better job descriptions and/or get a better recruiter who knows how to match to the job description.
With that in place, you (I) are getting about 10-20 resumes at most. Then what I do is read them and rank them. Takes less than two hours.
I'll only interview the top five from this stack. I'm happy to give each of these five people however many hours of attention it takes, they and I deserve it.
This way the people who didn't get to the interview round have not wasted any of their time which is the most respectful way to handle it.
Tech job postings are the absolute worst. Anyone who's having trouble with hiring in any way needs to start there. Everyone in tech knows that job postings are a joke, which is why you're told to apply to jobs even if you're not qualified, and then companies complain about the applicant pool. It's their own fault!
The "excuse" is almost always "HR". But this just tells the applicant your company is a bureaucratic nightmare that few sane applicants would want to join.
There's a lot of worry in tech hiring about the "faker", the engineer who lies or exaggerates on their résumé.
The irony is that the tech hiring process, especially the job postings, encourage the fakers to apply. You're practically beginning fakers to apply.
The most honest, scrupulous people tend to only apply for jobs where they're 100% qualified according to the job posting. So the system, which hires people who don't actually meet the listed qualifications, is systematically selecting against the honest applicants. Honest people can even be driven out of tech in virtue of being honest.
The author doesn't appear to understand the economics of the hiring company; it's about time & effort to filter and the cost of making a bad hire. I cannot give every application 1-2 hours of review if I get 100+ resumes.
This is an entirely self-created problem, not for you personally, I mean for the industry. We are against "credentialling" so signals like personal recommendations, alma mater, or work history at prestigious firms are overlooked.
My employer tried Hackerrank for a while, we abandoned it when we realised that the candidates we really wanted (e.g. based on personal recommendations from existing employees) were dropping out of the pipeline as soon as the Hackerrank link was sent to them. And the ones we were getting were good at Hackerrank and not much else.
Me: "I've got this repo on Github with millions of downloads and 10k stars. It's at least 20k lines of my own code. Would that be good to demonstrate my coding ability?"
Them: "Haha, no, that'd be too much work for us. Please do the test instead."
Me: "Fine."
Then they rejected me because I used GET instead of POST. Clearly that shows I have no idea how to code.
That was a large (1000+ employee) YC company, by the way.
Truly absurd. Followed the link in your profile. You wrote VSCodeVim! 49,674++/14,291--. And because you used the "wrong" HTTP verb. Did they also say you should have used tabs instead of spaces?? Outrageous.
> Then they rejected me because I used GET instead of POST.
OK, I'm honestly curious here. What was the request, and why did you make such a decision? Do you think it is the right approach in this case, or that it's an insignificant mistake that does not represent a person's understanding of HTTP protocol?
I remember hearing about at least one story where using GET instead of POST lead to disastrous consequences, as some robot crawled all over the "delete" links - but I have to admit, I can't imagine how it would happen without other significant design flaws happening as well (as "delete" links being public or robot being able to crawl under authenticated user's identity). This sounds like an error that can cause irrational irritation by the reviewer, but I can't imagine it being the sole reason to decide that the author can't code.
Not the OP, but from my experience teaching web dev, one possible reason is nerves. I’ve seen some of the smartest students make that mistake on an assessment simply because they were under pressure. A simple conversation afterwards shows that they do understand the difference (as expected). In the moment though, sometimes you just have a brain fart.
Yep! I have been in similar situations where you make a mistake and you get out right rejected. Surely it's better to explain the mistake to me and then we can discuss it. This can also be a nice cultural check too, how well do I respond to my own mistakes? It's a shame most people don't follow your idea here.
This really can't be understated. The difference between your how your brain works when you're under the clock and know you're being observed vs. the "natural" flow of a calm working environment is tremendous. Anyone of us who's conducting the interview must do some kind of double-checking of obvious mistakes to fairly account for the limitations of the process.
I was definitely in the wrong about GET/POST. The company I had previously worked for simply didn't really care too much about getting the verbs right, so I didn't really fret about them. And the company I'm currently at (multiple thousands of employees, strong engineering culture if I dare say so myself) just does everything as a POST. So it really is somewhat cultural whether you care about GET/POST or not.
But, IMO, even if you don't know, it's the kind of thing you learn once in code review and then don't worry about again.
For E2E encrypted APIs, one could well argue that worrying about HTTP verbs is just wasting time for no good reason. POST (consistently) lets you attach a body too.
Just using POST for everything is a pattern I've seen becoming more popular recently.
No we can get into an endless argument whether verbs are abused or not...
IMHO, GET (as of today) gas the issue that it does not support a body. If I‘m executing a search (an idempodent action), how can I nicely pass a JSON object configuring the search? I‘ll just use POST for that... So then we‘re at the point where the url does not work anymore because we have a POST /findEmployee; POST /findContract; ... instead of GET /employee and GET /contract.
Maybe I‘m wrong here, but it doesn‘t like nice in my eyes using POST in that way.
The JS API's I used to develop in used a mix of query strings (for exact matches) mixed with JSON (for more complicated queries) all in the url parameters
What's funny is that it's literally their job. They receive money for doing it.
Doing a take home is not paid. I have no idea why these recruiters assume it's ok to ask candidates for multi hours work without any kind of compensation.
Also- thanks for VScode vim! I use it and love it. I've always wanted to reach out cause there's lots of features I wish would get implemented. Though even without those it's one of the better vim emulations I've tried, and I've tried many!
I'd say "work they've done for previous employers" is something that nearly every candidate with industry experience has. The problem is that when I ask about it, candidates often are terrible at communicating what the work consisted of: often they focus on microscopic minutiae that appears disconnected from the big picture (e.g. gloating about gluing some trivial library as if that's a big accomplishment, when the position is for a senior engineer) or they demonstrate apathy/lack of proactiveness (e.g. some variation of "I just did what I was told") or they ramble incoherently, etc.
And this is supposed to be one of those bog-standard question that interviewees are always told by job interview guides to be prepared for!
I agree that people that have OSS work to show are much rarer to come by, and FWIW, they tend to do better in interviews IME. My theory is that it's not that the OSS resume bullet point itself sets them apart, but rather that they polished their skills more through the hard work they put into their OSS projects.
Just to clarify, I thought the author of the linked piece was suggesting that people could share actual code from previous employers somehow. Of course it's reasonable to ask people about their past work in an interview and to expect them to speak about it coherently.
I think OP did suggest sharing actual work, which would make sense if said work consists of e.g. semantic html that can be looked at in a publicly accessible site.
Obviously sharing backend trade secrets would be a big no-no. I've had people say they can't talk about their work due to NDAs, and that's completely fine in my books. We just move on to another topic.
FWIW, even when a candidate mentions having an OSS project, I refrain from looking at the actual code, and especially doing code review. I don't consider an interview setting the appropriate place to be grilling people on why they used a while loop instead of a for loop or whatever, since a) nearly everyone writes "bad" code occasionally and b) often things are done in certain ways for some good reason that isn't really relevant to my goal of evaluating the candidate. At the end of the day, the interviewer's goal IMHO should be to get as many signals they can from the conversation and make a determination about whether the candidate meets/exceeds a minimum threshold of competence
Interviewers react badly if you try to describe the high-level thought processes you go through, because that's not "technical" or "coding". So you end up reaching for dumb bullshit like gluing some library or whatever the fuck, because coding is actually boring grind work that you want to do as little of as possible, as stupidly as possible, and there actually isn't anything interesting to say to these people if you did the engineering right.
If that's not you, then you need to be painfully, painfully explicit about it, and some people still won't believe you because interviewers just lie all the time.
This is on-point! I've been involved in the recruitment process at my current employer for years and 95% of people that make it past the initial recruiter meet-and-greet have absolutely nothing to show in terms of open-source contributions or personal projects. Most of them just do the homework assignment so that you can at least talk about something during the interview.
You can always say it's because the company doesn't attract good enough candidates, but good luck with that in a competitive dev market in a large european capital.
I appreciate the authors honesty in his obliviousness of being on "the other side of the table" and I think it's clear that the author is living in a bubble that doesn't correspond to reality for people recruiting.
We've done something similar and found even candidates that do have public side projects may not feel those projects fully represent the code they'd write today and preferred the takehome.
Good point. I'm sure some of the candidates who do our assignment have stuff on GitHub or elsewhere, but thought they could do better work with the assignment.
Came here to share the same exact points. He’s making a lot of assumptions never being on the other side and actually interviewing lots of candidates. Most of the time I get half assed projects with bugs that are obvious and they haven’t ever made contributions to open source software as an alternative to the take home project.
I think the at home project is the most lenient way to test someone because they aren’t pressured to solve a problem on the spot, instead they are in the comfort of their own home and can reference whatever documentation they would need, just like they would in a real position.
242 comments
[ 2.5 ms ] story [ 260 ms ] threadWhen I'm hiring, I give applicants a real-code task, something they'd really do if they were hired - and I pay for their time too.
This method has allowed me to skip a lot of time-suck-bullshit while finding candidates, and their process with me is easier/faster too.
We see a strong correlation between performance on this task and performance and happiness when working with us.
The other important point is that this is very similar to the work we do and the way we generally do it. If someone doesn't like or want to get paid to do this task, they're likely not going to be happy doing it every day if they get the job.
Most real code task require a lot of ramp up time to gain context.
It also helps to weed out people who can’t actually program, this comes up annoyingly often.
I agree that having intentional errors in a starter repository is bad though.
I want to move from my 2000 USD a month (this is a good salary already) in a 3rd world country to a FAANG that pays 20k USD per month.
I'm willing to go through any challenge because the reward is superb for me.
Not to mention, without objective interview questions, an introverted non-native English speaker (who doesn't speak latin-germanic-based language) like me would fail most of these subjective questions (e.g. bad culture fit, can't articulate as well as a native speaker).
I like coding challenges. Please do more.
So... I'm all onboard with your statement, but I think you only said half of it. Nobody owes me a job, and if I don't like it, I'm free to not get hired. Nobody owes a company an employee either, and if candidates don't don't like it, the company is free to not hire. Zero sympathy for companies that do this and complain about a shortage of candidates.
I was asked to do a coding exercise for a tiny ($2500 if memory is correct) freelance project.
You make it sound like there's plenty of talent around but companies are being snobbish. That's not my experience at all in Germany.
Complaining to congress that there is a shortage of IT talent is so common in American politics that it has become its own unique form of performance art. For instance, during the magic section of the act, companies make a labor shortage magically appear without any mention of wages, while also magically making the laws of supply and demand disappear.
I like having clean teeth. If someone can make more as a dental hygienist, why on earth would it be valuable to society if they instead worked as a software engineer for less money?
If people won't work for 100k/yr as a software engineer, that means people capable of doing this work have found higher value things to do elsewhere. The higher salary elsewhere reflects the higher value produced doing something else.
This is one of many reasons why I'm so bewildered when people talk about a "shortage" of software engineers without considering the options available in other fields (fields that, perhaps, pay better and don't require taking bizarro whiteboard exams every time you change jobs).
The idea is an employer-side argument to counter the labour-side demand for higher wages: Don't have the wages to high, otherwise we go bankrupt and can't employ you any longer.
But programmers are already well employed, and software companies aren't really suffering. Cheaper programmers won't make website less bloated, won't make us figure out what to do with GPT-3, won't find a useful application for blockchains, won't make Windows 10 less moronic, won't make ad-tech less harmful, won't make the tech recruiters and the non-technical folk around software projects more competent...it would simply kill one of the few fields of employments that still pay well!
Look at construction. Did cheap immigrant labour make rents feel low in many areas? No, market forces drive up prices and owners keep the profit.
> Look at construction. Did cheap immigrant labour make rents feel low in many areas? No, market forces drive up prices and owners keep the profit.
In areas that aren’t constrained by density limits, homes are actually very cheap. In terms of square footage of home per population, the US is doing great. Problems are with inequality, in terms of geography and wealth.
> In areas that aren’t constrained by density limits, homes are actually very cheap.
That's the point. The effect of cheap construction labor only affects the rent prices in some areas, in other areas owners pocket the extra profit - while making construction jobs largely untenable except for immigrants coming from poverty.
I suspect something similar could play out in the software field.
I just think it’s useful to try to understand the effects of policies not just on me, but on society at large. For example, I might observe that a tax increase hurts me personally but helps others. Are you saying that instead I ought to come up with reasons why the policies that are bad for me are also bad for everyone else, to better lobby for myself, or perhaps so I feel less cognitive dissonance?
> That's the point. The effect of cheap construction labor only affects the rent prices in some areas, in other areas owners pocket the extra profit - while making construction jobs largely untenable except for immigrants coming from poverty. > I suspect something similar could play out in the software field.
So in a region where home supply is constrained by density restrictions, there’s surplus extracted via high rents, and some of that surplus could go to construction workers (and they ought to fight to get a good wage so the developers don’t get all the surplus). Great for the construction workers, but bad for renters. In the analogy, software engineers are the construction workers and everyone else is the renters. Given the high market value of my wages, I will negotiate hard to get my share. At the same time, I can acknowledge that there would be benefits to society if there were more people qualified to write software, even though it would lower my personal wealth.
My final point therefore is, let's keep IT salaries high for now.
You argued that software isn’t good for the world, so society wouldn’t benefit from more of it. This is a perspective I don’t usually get (especially here on HN) and I’m interested in hearing more. My belief is that software is part of a centuries-long trend of technology/productivity advances that have brought billions of people out of poverty, the continuance of which is necessary (although not sufficient) for a post-scarcity utopian future. I don’t know if we will ever invent a replicator, but I think more software engineers increase the odds!
That said, I was actually surprised at the lack of expensive cars driven by SFBA techies. I interviewed at Netflix, which I know pays astoundingly well, and the parking lot had a distinct lack of luxury brand cars. The exception seemed to be Teslas - I did see a bunch of Model S's and X's (Model 3 had not come out yet at the time).
Compare that to many East Coast suburban office campuses of non-tech companies that pay pretty poorly - you still see tons of mid to high end BMWs, Benzes, etc.
Sounds like Economics 101. The curves of supply and demand meet at some point. Some customers can't buy (enough of) the thing because the price at the intersection of the curves is too high for them. Yes, that's how free market works.
There are things I can't buy, because they are too expensive for me. Or I could buy them, but I would have to sacrifice something else I need more. This is considered normal. When some company can't afford (people like) me, it's the same thing.
They would like to have cheaper developers. Sure, who wouldn't? There are also things I would like to have cheaper. Same problem. Why am I not entitled to get things as cheaply as I want, but my potential employer would be?
In my experience many companies prefer to whine about the shortage of talent, even if they could easily hire another person by increasing the offered salary by 50%. But they won't. Then they make millions of profit, which shows that the money was there, it was just their decision not to spend it on salaries. Okay, fair, it's their money, their decision. But if it's your decision, own it, don't pretend there wasn't an alternative.
Make it known that you offer double of the market salary, and you will be surprised how many people are out there in the "empty market". At least I never heard about a company that offered lots of money and couldn't find a person for the job.
So developers go to work on finance and ad-tech, and not on normal, boring, only-making-three-times-the-national-average-salary stuff. And those companies then complain that they can't find developers. Because whether they have a coding challenge or not, they can't find developers because money.
There are a lot of jobs that pay less than software engineer, but would be more desirable provided the pay isn't too much less. For example, someone might prefer to be a dental hygienist, figuring that the work is flexible, steady, and there's less of a possibility of age discrimination later in life. This person would become a software engineer for twice the dental hygienist salary, but not for 10% more.
In a free market society, it's not up to us to make that decision. If you want to get software engineers for in San Francisco for 10% more than a dental hygienist's salary, and you can't get any, that's the market's answer.
Corporate capitalists have discovered that if you don't like the market's answer, have the government ask for you.
And the surveillance capitalism developer earns more than the entire village. Ergo the market commands us to all develop surveillance stuff so Saudi Arabia can spy on dissidents and then chop them up in their embassies.
But again, that's not the point I was making, it's weird that multiple people understand my comments as "we should remove the free market". Me saying that it's the market that is making it hard for normal companies to find talent is only the background for what I'm actually trying to get across: It's not the coding challenge that is making it hard to find developers.
A company that isn't in a hyper-profitable space like ad-tech or surveillance (watching employees during WFH is booming) or fintech (figuring out new ways to make people part with their money) won't be able to attract talent by removing a coding challenge. It's just not an important factor in the equation.
On a personal level, coding tests are a deterrent for me in applying for new jobs. I have my old copy of data structures and algorithms, and I've gone through "Cracking the Coding Interview." I actually enjoyed it at first. They're interesting and challenging problems. If there were an exam - maybe something similar to the actuarial calculus or probability exams - I'd probably be interested in studying hard for it and passing it.
However, I'm no longer interested in re-loading this information into short term memory in order to take a whiteboard exam. This may be a facet of how my own brain works vs others. I just don't walk around prepared to figure out which elements of a set, when combined after permutation, are equal to an existing member of the set. Or how to find all square sub-matrices with a positive determinant in a larger matrix. I can do these things - I've enjoyed doing them - but for me, it takes a lot of training and prep to re-load it in a way that I'm ready for the 45 minute whiteboard exercise. If it were a proper, lasting exam, I'd be happy to do it, but it's just too random, uncertain, and capriciously scored for me to want to do it anymore. I will note that some people don't have thae same trouble I have with this, and - I really mean this - this may actually indicate that they are better candidates than I am. In short, I need to acknowledge that I may certainly not be the "talent" that anyone is missing out on.
Money makes a difference too, but the reason I brought up Wealth of Nations is to draw your attention to the section "Of Wages and Profit in the different Employments of Labour and Stock" http://geolib.com/smith.adam/won1-10.html
I would recommend the entire article, as Smith breaks down the various factors that can lead to higher or lower wages, but you don't really need to read the whole thing, here's smith's summary:
"THE five following are the principal circumstances which, so far as I have been able to observe, make up for a small pecuniary gain in some employments, and counterbalance a great one in others: first, the agreeableness or disagreeableness of the employments themselves; secondly, the easiness and cheapness, or the difficulty and expense of learning them; thirdly, the constancy or inconstancy of employment in them; fourthly, the small or great trust which must be reposed in those who exercise them; and, fifthly, the probability or improbability of success in them."
Within this matrix, software engineering does have a lot to recommend it. For example, Law requires a very expensive and lengthy graduate degree, whereas software engineering doesn't, even though most successful SE's generally do have some kind of undergraduate, and often graduate, degree in rigorous fields that require a lot more math than law. How people will respond to this (along with other aspects of these two fields) depends a lot on their own personal wiring as well.
Within this matrix, yes, I do think that many people find in person, oral exams, at a whiteboard or at a computer under scrutiny and time pressure, on topics that they often don't know well in advance, evaluated by people whose credentials and skill they don't know in advance... yes, I think many people find that a deterrent, either for applying for new jobs, or entering the field in the first place. Many people consider these on-the-spot exams extremely stressful, especially when they are in person, at a whiteboard. And we go through days of it, and a new round of study, every time we apply for a jbo. Tech interview exams are not the only factor making it harder to attract candidates, but neither are any of the multiple factors identified and elaborated o...
Evey job I've applied or hired for either a) had lots of candidates with a very selective process, or b) telegraphed that you should avoid the company or didn't pay well enough (and often still got plenty of applicants who wanted their foot in the tech door despite not being capable).
My current company would claim there's a shortage, but they actually don't know how to hire and telegraph that you probably don't want to work there in a variety of ways.
1. Speaking perfect English, both fluently and without a heavy accent.
2. Talking about technical topics or other shibboleths with the same cultural affect an American would, as opposed to a foreigner. For example, many Indians will call a phone screen a "telephonic interview", and I've personally seen Americans react poorly to that.
I know lots of developers who are very well paid and very effective in their jobs, but they don't speak English as well as an American and would stumble at certain cultural nuances.
But I always notice how a native English speaker almost always speak perfect English. They pick the right word to use and many other small things that make their English superb.
Any decent engineer joining Facebook in 2012 and sticking there until now would earn 60k a month.
It comes up with questions that are as simple as "Create a function that deduplicates an array of integers"
As it should.
If someone is a poor English speaker, at an English-speaking company, they absolutely should be penalized.
Not everyone is equal, was equal, nor will be. Some people are born blind too. Shouldn't we adjust our hiring practices for them too?
Not only should we adjust our hiring practices for blind applicants, there's a legal obligation to make "reasonable adjustments" for these candidates under the Equality Act! *
* Laws may vary in less developed jurisdictions.
As someone who's been in a FAANG for nearly 8 years and is making 2/5 of that...where the hell did you find that position!?
EDIT: oh wait. I was comparing my take-home salary. You're probably describing gross total comp? In which case - yeah, that makes sense. Good luck!
What is your position...? I earn about $9k/month (net) at $170k/yr (gross) in CA as a software engineer. So, I'm already past the 2/5 mark and I'm just at a startup that doesn't even pay well! On top of that - you should be getting stock that translates directly into a good chunk of cash - not just a salary component. (Unless Netflix I guess)
Check out https://www.levels.fyi/
> Ask them to build something – anything!
> Instead of a coding challenge, ask them to spend a short time building something they enjoy, and talk about what they did and why.
To me this would be a MUCH more difficult task. A blank canvas is much harder to work with, and downright frightening. Constraints foster creativity.
Having everyone code the same thing helps the interviewer (who is also very busy and has many other things happening in their work life) gauge the different levels of the people applying easily by comparing them on the same task.
I code state of the art motion planning, localization, and robotic algorithms. some of this requires more than a day of tooling around at home building a project
That seems very taxing on the company. I'd rather screen the candidate before committing to anything via interview/coding challenge, etc... The take-home coding challenge seems like a good idea -- less pressure than coding during the interview and allows the interviewer to assess the "real life" performance.
> Ask them to build something – anything!
That's great, but a big part of working at a company is building something that you don't like or don't understand well -- talking to stakeholders, customers, figuring out specs, etc -- something that a good take-home exercise can assess
So having a probationary period is too expensive for the company, but a several hours coding challenge for the developer is a OK?
In my experience when a candidate was a bad fit, it was because of soft skills/personality traits not technical chops (which are much easier to learn than soft skills). Are you screening for soft skills in the coding challenge?
I didn't like spending probably about 5 hours total on coding challenges last time I interviewed, and I've been internally opposing them, but as the candidate who interviewed, I'm much happier having dealt with it once and gotten over it than having probationary employees show up on my team for a week every so often and having to drop what I'm working on to work with them and evaluate them.
> In my experience when a candidate was a bad fit, it was because of soft skills/personality traits not technical chops (which are much easier to learn than soft skills).
This is a good point, and argues in favor of traditional interviewing (establishing a rapport with the candidate etc.) over any sort of "challenge" IMO. I don't think I'll learn more about soft skills in a week than I would in an hour - the complex stuff generally takes several weeks for the new hire to be properly onboarded and contributing, anyway.
I'm a consultant so I've been through tons of interviews at this point in my career (21+ years).
If I had my choice? I'd pair program with my potential teammates on a bug/feature for the product I'd be working on.
Give me something real that's been in the backlog for a while, and we'll pair on it for the allotted time.
This approach has some value, but it can't be the only way to evaluate candidates.
If you make about $100 an hour, a six hour coding challenge costs you about $600.
If a potential employer asked you to spend $600 to fly out and see them, on your dime, would you do it?
I would do it IF I desperately needed a job. I did it back during the dot com crash; I needed a job badly and I was willing to roll the dice.
In 2020? Oh hell no. There's a million companies who are hiring, and an employer would have to be exceptionally special for me to waste $600 on the chance they'll hire me.
We've had many candidates that look strong on paper (3+ years of experience, senior titles, etc) come in and struggle. Struggles ranged from basic code literacy (think for loop to traverse a nested json object) to basic programming like writing a function. We've especially noticed this more with coders coming from purely a javascript and front-end background (React/Angular, etc). They understand control structures like if/else but struggle with loops, setting variables, and basic higher level concepts like breaking a problem into smaller functions.
My philosophy is for junior candidates, any level of competency is ok. Strong mentoring and experience on job usually can allow anyone to thrive and so you can prioritize soft skills and culture. But for more advanced roles, I don't know, it's hard to parse out the quality candidates without having some sort of technical interview. Definitely would love to hear how other's learning and experiences.
But I can do all these things. I've gotten bounced from my interviews because (this was the official reason, anyway) I didn't get far enough with the coding in finding all matching subtrees in a binary tree, in 45 minutes, at the whiteboard.
Companies should feel free to hire based on my ability to do this, they should not say they were just testing basic things like lists, loops, and conditionals, or even basic data structures and algorithms like binary trees or hash maps. These exams go way beyond that.
This may not be the case for all, and it sounds like it is not the case for your process. But in the broader industry practice, I smell a motte-and-bailey fallacy here.
To be fair this could also be a reflection on our ability to read resumes and experiences and our initial phone screen. But it is hard. My first job was at fortune 50 company, you could have the title of senior software engineer and not write a single line of code as part of your job.
That being said i don't won't diminish your experiences. This is just our experience from the other side of the table. Also we don't have an HR department (we're a small startup). The interviews are conducted by engineers in the company and we try our best to empathize with candidates and be conscious of creating an interview process that we'd feel comfortable being a part of as well.
i actually don't disagree with the prior poster. Technical interviews are highly variable and it's frustrating to go through poor interview processes. I've got bad interview stories as well. Being on the other side of table now, we're always interested in learning about better approaches hiring. Would love hear what has worked well for others.
That being said i'd be curious how you evaluate general software engineering ability (e.g. writing clean abstractions, decomposing monolithic methods, and general design). I feel most system design evaluations are a bit too high level or overly broad (e.g. design a library application or make a social network) and coding challenge are too low level.
Are they (me in this case) going down rabbit holes? Are they painting a clear picture? Are they communicating that picture? Are they suggesting edge cases? Are they minimizing bloat? Is it too complicated? What are the most important aspects of the design in their eyes? Can they argue for them? Do they understand where a piece fits and if not, do they understand that something has to go there? Can you both step back and definitely outline a logical flow?
There's not always one solution, but the approach counts.
Lower level is logical problem solving.
Can they reason the problem space? Do they start naively? Can they iterate? Can they optimize? Can they explain each of these steps for future iteration? Do you understand what they're writing? Would you work on that code yourself?
There's not always one solution here either, but 90% is getting a naive solution that can be optimized by taking a second look. I feel that the naive should be found in 5-15 minutes and you can spend the rest of the time talking about how to optimize.
I know I'm rehashing what you probably already know, but at a certain point interviewers are just playing games.
If you've found someone you want to work with who can also reason about problems in a timeframe you see fit, well... that to me looks like gold.
I like these questions because that's how real systems are going to be built.
Take-homes are great, but there is just so much effort wasted in the process if the product can't be understood.
It was a good measure of how well they communicated, how they dealt with undoing some of their prior work, and the process was fairly representative of the type of work involved in the position.
One way of going about about it is to find former colleagues that work for the company you are applying to, and ask them to provide references on your behalf.
It's hard for me to escape the conclusion that the author's just trying to universalize their personal preferences.
At this point I consider a coding challenge a deal breaker. I'm not doing one, and if you ask, there's a near certain chance I wouldn't want to work in an environment that thinks they are a good idea.
If you're interested in my abilities, I'd be happy to spend an hour walking through something I've already worked on in the past.
[0] https://en.wikipedia.org/wiki/Stage_(cooking)
I've encountered so many variables (no pun intended) in technical interviews that I've virtually stopped preparing. There's hardly any way to prepare when the field is so large and you never really know what you're going to be tested on.
I've had people challenge me on theoretical system-time timezone conversions (and incorrectly mark my correct answer wrong "because of physics"...), beside reimplementing native JavaScript functions, beside complex algorithmic problems, beside toy problems like building a binary calculator. [That's great if you learn I can rig a web page for adding 100 and 101, but how do you know if I know anything about databases for this back end position]
And those interviews were all for working with web tech, in a couple of the cases just front end positions.
There needs to be a common, limited standard and/or domain-specific standards.
It's really counter-intuitive to me, and I don't understand how it benefits the hiring process, if a company is looking for a web-back-end engineer for what is essentially a complex crud app and wants to propose the candidate solves a problem from a domain they've never worked in—and marking them on the correctness of the answer, rather than the work process.
I'm for objective, technical challenges. But I've seen enough that it seems apparent to me that it's broken.
(addendum: I've had several good interviews as well, and good interviewers. I had a good one recently as well that was fitting for the domain. They're not all bad, but most seem to be in my experience)
Surely this is a good thing, no way for people to game the system and no time wasted.
I understand some problems are meant to judge your thought process, others are meant to measure your basic literacy. I'm not talking about those, however.
My experience in interviews has been that most of them are engineers pulled into the process who searched the web for "software developer interview questions technical" and cobbled together a few disparate options without a moments concern more.
Then again, those kinds of interviews have certainly cemented my desire not to work there.
>>> Given an array of integers nums and an integer target, return indices of the two numbers such that they add up to target. You may assume that each input would have exactly one solution, and you may not use the same element twice.
Seems trivial enough to me... except if the interviewer forgets the hard constraint that there is exactly one solution, then that's a shitstorm of edge cases.
I'd criticize the problem for asking for the positions of the numbers rather than the numbers. That's just super annoying for no reason.
I guess you've been giving candidates a more hardcode version of the problem, not having that constraint?
edit: nevermind, it's the other poster who was giving that problem, not you. No idea how they formulate it in their interviews.
I was asked the two sum problem in an interview once, but I forget if it had this constraint or not. This seems much more reasonable.
If you can create a new array containing indices to the original array, sort the new array, and use two pointers starting at the beginning and the end of the sorted array to find a solution, it takes O(n×log(n)) time. It is not necessary to assume that the solution is unique, assuming it is okay to simply write one of the possible solutions.
If all values are between 1 and k, and you can make a k-sized array, you can find a solution in O(n+k).
No matter what you do, you can't get better than O(n).
The important thing is that there is more than 99% chance you will never have to solve this problem in your job; and even in the case you would, there is 95% chance that the O(n×log(n)) solution would be acceptable regardless of whether it is the best possible or not.
Yeah, you just do it in O(n²) worst case because the problem is O(n²) to find all pairs that match because there can be O(n²) pairs.
However he meant one pair here too. Pretty easy to solve. Did it in under 30 s in Python on my broken cellphone. If there are multiple pairs, just replace return with yield.
I've seen people barely sneak through technical interviews, whose subsequent performance justified the concerns I had at the time. I've seen others hired against my personal recommendation who turned out to be brilliant at their jobs.
After having been on both sides of the table, I’ve really come to appreciate the system of licensure used in the trades. I don’t have to quiz an electrician on the electrical code or ask them to wire up a dog house. I also know they have requirements to keep their license active. I’m not sure what that process could be like for software, but it would sure be nice to eliminate the enormous collective waste of time spent (poorly) establishing candidate competency (especially when they have to do it for every single company).
Universally I get turned down. "You have to do our assignment." Oh and I'll usually offer to do their assignment for $300. That doesn't go well either.
Only one company took me up on this offer, and it's the one I work for. (They didn't even ask me to complete one of my tasks, they just looked at the project I authored and said that was fine; and scheduled a full interview).
The few times I have actually done these assignments, I've never gotten the job.
edit: read to the end, a lot of the points are absurd, easy to realize after a couple interviews.
It's not so much what the challenge is -- it's how one is asked to accomplish it.
Sitting in front of my laptop for 6 hours while people ask me random programming tasks while they watch me on camera and see every keystroke on screen while I'm editing is not how I work. I work by choosing from active tasks that fit the "space" I'm in, by multi-tasking, etc. - if I get stuck on something in the "real world" I'll go work on something else that needs to get done, etc. You'd be amazed at how many interviewers simply refuse to ask you another question in lieu of or even in addition to the one the first wanted to ask, for example.
More accurately, if you need to learn new things to complete it - it will be very time consuming.
Someone experienced, or that uses relevant tech/skills on a daily basis should complete it in less than the time allotted.
Therefore "homework" coding challenges are useful for discerning if a candidate has relevant, current experience.
See the article on college professor solving a 30+ of CS problems in one day.
Whiteboard challenges, on the other hand, evaluate a candidates thinking process & presentation skills.
Whiteboard challenges
The unfortunate reality is that it can be really tough to fire low-performers. I NEED to know that a candidate can do the work. We go out of our way to make our ask reasonable, relevant to the role, not spec-work, not time consuming, and well defined. It's not uncommon for us to learn that the candidate can't actually do what their resume says they can or for the candidate to learn that they grossly overestimate their own skills.
I recently talked to a Python/JS expert that led a project at a major agency to scrape a network of websites, find and extract specific data, and deliver game changing competitive intelligence. The candidate ended up admitting that their role was limited to copying and pasting a pre-built code snippet into the dev tools console and that they couldn't actually read or write any code. The proof is in the coding challenge.
Ask pertinent questions - a lot of candidates are bad at immediate summation and dissemination of information. There are many good engineers who have not learned how to sell their work, themselves, or in general communicate to the intent of a question. This is an invaluable skill, largely orthogonal to writing code, but it's a skill that's often learned with experience. Learning to ask level appropriate questions is hard and evaluating answers consistently (if not objectively) even harder. It is also largely insufficient in gauging programming ability. I've had co-workers who can talk about advanced architecture-y topics seemingly competently but struggle to write even relatively simple code.
Probationary period - I've never seen this implemented (excluding internships which sort of fall into this category) and it certainly sounds like a nice option to have available but I would bet many candidates would be afraid of a short probationary period (say 3 mo) especially if they have to move their family for the job. Being a professional and being told you are on a probationary period is stressful. At many large companies it can easily take more than 6wks to get up to any sort of productivity.
Walk through existing projects - most people (vast majority) do not have any sizable or clean or even "interesting" side project they would want to walk an interviewer through. Unless you worked on open-source for your previous employer (again minority of people) or on your spare time, this is not something you can get a strong signal out of.
Ask them to build something - err, I don't see how this is that different than a coding challenge. And again, not everyone has personal projects ready to be shown to an interviewer. For example, the game I'm coding some nights after work does not look anything like code I write for my employer; there are no tests, there's a lot of experimental and often dead code, functions are not neatly factored and thought out with an eye for collaboration, etc. I would not want to be judged by this code for an interview.
Pair programming - this sounds like a coding challenge but with a more empathetic interviewer. I don't think this is an alternative, just a mild augmentation of a coding challenge.
Focus on ability to learn - companies do this but it's usually considered supplementary to coding ability. Some companies explain their product and try to see if the candidate understands where the value of the product exists or at least where some of the challenges are. This could be spun out into a product type interview or a system design interview depending on intent.
Alternatives to white-boarding or do-at-home project interviews: - Provide a small existing but functional code base and ask the candidate to implement additional functionality. Downside is this takes a lot more effort to prepare and are limiting the candidate in regards to language they want to use. - Provide multiple options for a coding project or question and let the candidate choose which one they want to solve. This would hopefully reduce some of the luck factor of getting a question you know how to solve. - I can't think of any others, but I'm sure they exist
One of the big desirable goals of interviews are evaluating candidates in a consistent manner. All interviewers have biases and these biases vary between the interviewers. Trying to account for that is, I believe, important but also very hard. Using the proposed alternatives would make it even much harder.
> ...
> "Most candidates will have side-projects or work they’ve done for previous employers."
As someone who has actually done a lot of interviewing, I can say with great confidence that this is simply wrong. At my current employer, we have a "homework assignment" as part of the interview process. We tell candidates that they can point us at an existing project of theirs instead of doing the homework. Few people take us up on this because few applicants have any substantial public side projects or FOSS work they can share.
And sharing "work they've done for previous employers" is a great way to court legal trouble. They shouldn't share it and as an interviewer I don't want to see it!
The author doesn't appear to understand the economics of the hiring company; it's about time & effort to filter and the cost of making a bad hire. I cannot give every application 1-2 hours of review if I get 100+ resumes. I can't give every interviewee a half-day assessment to get the very poor signal/noise ratio answers to questioning methods suggested. I can't hire easy and then fire easy. I can't invest weeks of work from myself and other mentors on non-ideal potentials. I know this means I pass over great candidates and that really sucks and is totally unfair. I'm sure that's happended to me as well. The problem is nothing presented here is a better alternative.
This is perhaps the biggest misconception of hiring, that out of some misguided notion of "fairness" or "objectivity" you have to treat every candidate exactly the same. In theory it seems like a good principle, but the road to hell is paved in good intentions. In practice, this results in arbitrarily designed standard tests which are only superficially fair, and systematically weed out people who aren't good at test taking or auditions.
Go through all the resumes quickly, pick out the top candidates, and start from the top! Focus most of your time on researching your few top candidates, and if one of them seems good, then hire them, and end your hiring process. You may only have to interview 1, 2, maybe 3 people at most. This saves everyone time: your own time, and the time of the 100+ people who would otherwise be interviewed and rejected, because there's only 1 job opening.
Obviously if you interview your top candidates, and you find them lacking, then keep working your way down the stack. But going through the entire stack of candidates is a huge waste of time, a pointless exercise, totally counterproductive.
Don't do "screening" interviews. Only do "hiring" interviews.
Also, anecdotally after interviewing hundreds of candidates, I’ve found resumes to have a relatively high false positive rate. Surprising numbers of engineers look great on paper but can’t write working code for semi-complex problems when tested, even accounting for interview anxiety.
How have you accounted for interview anxiety?
1. I try to make the parameters of the interview as friendly as possible — language of their choice, don’t have to worry about code style or computational complexity, or talk while they’re coding, just write working code that solves the problem. They can use google freely. I’m there as a resource to talk through their solution, answer questions, and help them get unstuck if needed.
2. The algorithm to solve the question doesn’t require a trick and there are many viable solutions so most candidates can verbally explain to me a working algorithm quickly, +- a couple edge cases.
3. Candidates often get visibly flustered or hung up on irrelevant details, or unsure what exactly I want for them, and I focus on noticing this. I try to guide them down a correct path if they’re close and tell them not to over-architect if they start doing that.
Despite all this, too manly candidates do not have a dependable metaprocess for writing working code - a common scenario is they write code, it gives the wrong result, they drop some print statements to debug, but can’t effectively trace the program to figure out why they’re getting the debug output and so they guess and try to fix the wrong thing, causing further problems. I know that some of this is nerves but it can’t account for all of it.
Why not?
If you've never experienced severe situational anxiety, you may not know how debilitating it can be.
And then, maddeningly, perhaps just minutes after the interview is over, you return to your "normal" self, and start to think about what you would have done in that situation if you weren't nervous.
It's basically stage fright, which can be terrifying if you're not used to the stage.
Still, I do have some EQ. A lot of candidates exhibit visible anxiety and then get into a rhythm and calm down when they start coding. Or they visibly calm down once I start asking them questions about how they're thinking and coax them towards viable implementation. On the flip side, a lot of candidates actually exhibit overconfidence running in the wrong direction, which I try to lightly mitigate but can't prevent in all cases without giving an unfair advantage.
If you didn't hire them and compare them to new hires who have bad resumes but did well on your test how would you know?
You can't objectivity judge a candidate by your test results only. Otherwise your test is just randomly filtering people. Until you do that you don't know how effective your test is. You could be flipping a coin and getting the same results.
Good way to put it. Exactly.
I remember an interview training exercise, which also covered resume screening. Each person got the same stack of resumes, and the instructor said, "You have 60 seconds to pick out the top 3 you'd like to bring in for an interview. Find them." At the end of the exercise we found that we all basically picked out people who were a lot like us. White people picked out "whiter-sounding" names, women picked more women, etc. Also, Ken Thompson's resume was one of them (with a different name) and nobody picked him.
I think the point was that human eyeballs and brains are terrible at objectively or fairly pre-filtering.
Well, by quickly I didn't mean only 60 seconds. :-)
> in a way that is not subject to personal (conscious or unconscious) bias?
In my opinion, a common conceit is that it's possible to design a process that is not subject to bias. It's not possible. There will be bias, no matter what process you choose. It's thought that "quiz show" style interviews are "objective", but if you look at the results, they're clearly not helping. They're not increasing diversity. The poor results speak for themselves. So then some people look at the results of this so-called "fair" process and conclude that minorities are just worse at coding, because that's what the tests say. To me, that approach and attitude is even worse than just acknowledging personal bias.
The only way to try to counteract unconscious bias is to be conscious of it. Admit it, and keep it in mind while you're going through your hiring process. Keep it in mind while you're filtering the résumés. Keep it in mind while you're interviewing. But don't try to pretend like you can eliminate it entirely from the process, because then you're just deluding yourself and everyone else.
Why not? Maybe you should address that problem. It would certainly make things a lot easier.
Your process has already failed if you're considering 100+ people!
First, you should never be getting 100+ resumes for a role. If you are, you need to write better job descriptions and/or get a better recruiter who knows how to match to the job description.
With that in place, you (I) are getting about 10-20 resumes at most. Then what I do is read them and rank them. Takes less than two hours.
I'll only interview the top five from this stack. I'm happy to give each of these five people however many hours of attention it takes, they and I deserve it.
This way the people who didn't get to the interview round have not wasted any of their time which is the most respectful way to handle it.
Tech job postings are the absolute worst. Anyone who's having trouble with hiring in any way needs to start there. Everyone in tech knows that job postings are a joke, which is why you're told to apply to jobs even if you're not qualified, and then companies complain about the applicant pool. It's their own fault!
The "excuse" is almost always "HR". But this just tells the applicant your company is a bureaucratic nightmare that few sane applicants would want to join.
The irony is that the tech hiring process, especially the job postings, encourage the fakers to apply. You're practically beginning fakers to apply.
The most honest, scrupulous people tend to only apply for jobs where they're 100% qualified according to the job posting. So the system, which hires people who don't actually meet the listed qualifications, is systematically selecting against the honest applicants. Honest people can even be driven out of tech in virtue of being honest.
Remote roles get a _lot_ more applications than on site. In my experience getting 100 applications is quite common for such a role.
This is an entirely self-created problem, not for you personally, I mean for the industry. We are against "credentialling" so signals like personal recommendations, alma mater, or work history at prestigious firms are overlooked.
My employer tried Hackerrank for a while, we abandoned it when we realised that the candidates we really wanted (e.g. based on personal recommendations from existing employees) were dropping out of the pipeline as soon as the Hackerrank link was sent to them. And the ones we were getting were good at Hackerrank and not much else.
Me: "I've got this repo on Github with millions of downloads and 10k stars. It's at least 20k lines of my own code. Would that be good to demonstrate my coding ability?"
Them: "Haha, no, that'd be too much work for us. Please do the test instead."
Me: "Fine."
Then they rejected me because I used GET instead of POST. Clearly that shows I have no idea how to code.
That was a large (1000+ employee) YC company, by the way.
By the way, thanks for VCCodeVim!
OK, I'm honestly curious here. What was the request, and why did you make such a decision? Do you think it is the right approach in this case, or that it's an insignificant mistake that does not represent a person's understanding of HTTP protocol?
I remember hearing about at least one story where using GET instead of POST lead to disastrous consequences, as some robot crawled all over the "delete" links - but I have to admit, I can't imagine how it would happen without other significant design flaws happening as well (as "delete" links being public or robot being able to crawl under authenticated user's identity). This sounds like an error that can cause irrational irritation by the reviewer, but I can't imagine it being the sole reason to decide that the author can't code.
But, IMO, even if you don't know, it's the kind of thing you learn once in code review and then don't worry about again.
Just using POST for everything is a pattern I've seen becoming more popular recently.
IMHO, GET (as of today) gas the issue that it does not support a body. If I‘m executing a search (an idempodent action), how can I nicely pass a JSON object configuring the search? I‘ll just use POST for that... So then we‘re at the point where the url does not work anymore because we have a POST /findEmployee; POST /findContract; ... instead of GET /employee and GET /contract.
Maybe I‘m wrong here, but it doesn‘t like nice in my eyes using POST in that way.
What's funny is that it's literally their job. They receive money for doing it.
Doing a take home is not paid. I have no idea why these recruiters assume it's ok to ask candidates for multi hours work without any kind of compensation.
Also- thanks for VScode vim! I use it and love it. I've always wanted to reach out cause there's lots of features I wish would get implemented. Though even without those it's one of the better vim emulations I've tried, and I've tried many!
And this is supposed to be one of those bog-standard question that interviewees are always told by job interview guides to be prepared for!
I agree that people that have OSS work to show are much rarer to come by, and FWIW, they tend to do better in interviews IME. My theory is that it's not that the OSS resume bullet point itself sets them apart, but rather that they polished their skills more through the hard work they put into their OSS projects.
Obviously sharing backend trade secrets would be a big no-no. I've had people say they can't talk about their work due to NDAs, and that's completely fine in my books. We just move on to another topic.
FWIW, even when a candidate mentions having an OSS project, I refrain from looking at the actual code, and especially doing code review. I don't consider an interview setting the appropriate place to be grilling people on why they used a while loop instead of a for loop or whatever, since a) nearly everyone writes "bad" code occasionally and b) often things are done in certain ways for some good reason that isn't really relevant to my goal of evaluating the candidate. At the end of the day, the interviewer's goal IMHO should be to get as many signals they can from the conversation and make a determination about whether the candidate meets/exceeds a minimum threshold of competence
If that's not you, then you need to be painfully, painfully explicit about it, and some people still won't believe you because interviewers just lie all the time.
You can always say it's because the company doesn't attract good enough candidates, but good luck with that in a competitive dev market in a large european capital.
I appreciate the authors honesty in his obliviousness of being on "the other side of the table" and I think it's clear that the author is living in a bubble that doesn't correspond to reality for people recruiting.
I think the at home project is the most lenient way to test someone because they aren’t pressured to solve a problem on the spot, instead they are in the comfort of their own home and can reference whatever documentation they would need, just like they would in a real position.
And nothing gives more open ended reasons than the coding test.