As someone who's been around the tech industry for a while, I know firsthand all the sausage-making that goes into building a great technical hiring funnel.
On the flip side, as a job seeker, I also know how demoralizing it can be to go through a broken hiring process that doesn't accurately reflect your abilities.
With recent layoffs and many talented professionals on the job market, I was compelled to write a blog post about how to build an inclusive hiring culture and find exceptional engineering talent.
If you're involved in your organization's technical hiring process at any stage, I encourage you to give this a read. I share some best practices for conducting effective interviews and improving your own hiring process.
You can exclude mediocrity while also being exclusionary on other axes.
They are unrelated issues. In fact, I’ve even heard of exceptional people being abused/bullied for belonging to the wrong group to the point of being told “you couldn’t have done that” which itself is an assertion of their supposed mediocrity for exclusionary reasons.
Don’t assume you need to be bigoted to exclude mediocrity. Discriminating, yes, but not discriminatory against groups that inclusive hiring policies attempt to protect.
An inclusive interviewing process does not mean that you hire everyone. It means you reduce the weight of people's biases as part of identifying who you hire (because people turn out to be quite bad at prediction in hiring).
- The companies that make an effort get a lot closer to population baselines than the ones that just give up.
- Organizational pressures are something leadership and management should be steering. I'd rather have hiring practices be an explicit choice than something that "just happens"
if your goal is 50% women then you have to lower standards to achieve that.
You don't have to. But you'd remove them from the job market, making the pool smaller for other companies. IOW, a few companies can target 50% women, but that'll make it that much harder for other companies.
If the % of woman developers is 10% and management creates hiring goal of 50% it detorts reality. It means hr has to work harder at filling those female roles which often reduce their checkboxes while increasing the checkboxes for everyone else. Now you have to leaving positions unfilled longer in hopes of finding a candidate who matches a gender. You've turned the hiring process into a broken mess and require 1000 times more candidates.
Let's say you are successful. Let's say a class of companies are successful at this strategy. Lets use the example of faangs which are desirable places in terms of salary/brand. If faangs were successful at this that would reduce the % of female developers in other industry and assuming faangs are taking the best candidates that leaves the worst ones. Which then creates this reality where male programmers outclass female developers in these other industries. That makes it harder for women in general and makes this false impression that females are not as good as males.
To help women you really need to treat them equally. Trying to reach a goal of unhealthy unnatural % industry wide means women will left holding the bag when the music stops.
ipaddr is right that he described the actual situation with FAANGs scooping up all of high caliber underrepresented minority candidates (think black/women ivy league comp sci grads with high GPA).
But this also creates a positive feedback loop when more and more women decide to switch industries and pursue IT/Engineering jobs via bootcamps, college degrees, etc. I noticed the number of female candidates in UX/UI, fullstack, QA, Data Analytics - has increased in last several years.
Partly because the demand is still high for these professionals, partly because there is entire cottage industry of bootcamps churning out IT specialists en masse, partly these diversity hiring practices that opened up doors for women
Regardless of whether you're coming from educational elitism, you're still making sweeping claims about big groups of people based on extremely limited evidence.
I'm a bootcamp grad, and would not have gotten into the field if bootcamps did not exist. I'm about four years into my career now, currently working at a major well-reputed tech company, and haven't gotten an average-or-below annual performance review yet. (And one reason for that is that I tend to be cautious, critical, and thoughtful in my technical decisions.) There are a number of other people from my bootcamp class with similar results.
I am making a generalisation, based on having interviewed over 200 developers in the past 2 years as part of technical screening.
80%+ of the bootcampers were rubbish and shocked to be told their knowledge was way below where they thought it was.
A classic is a 6 week JavaScript bootcamp grad claiming to be an "expert in JavaScript" (their words) and couldn't explain the JS type system or basics of variable scope. That was the norm. That kind of rubbish.
I'm happy you're an exception and everyone gets a fair chance with me, regardless of background, but I am never shocked when I have to bin yet another bootcampers CV
"Everyone gets a fair chance with me, regardless of background" is an _extremely_ different statement than "anyone with an ounce of technical hiring ability avoids those bootcampers like wildfire".
If you mean "in my experience, bootcampers fail technical screens at much higher rates", then say that, instead of implying that you're stupid if you even consider hiring someone who went to a bootcamp.
Sure, but if you are lucky enough to have an entire team of interviewers who have this much experience, you're probably not having the same hiring conversation that's happening in this thread.
Thinking of it in terms of race might indicate more about how you view things. As an example, think about how frequently throughout history people in power have claimed that women "can't handle" the positions of power that men had. They cite all kinds of nonsense like "emotional" or "hysterical"... conveniently ignoring all of the hysterical and emotional men throughout history.
Think about how something like that would affect how companies are formed. Things seem much better now, but I merely wanted to highlight one of many kinds of biases that are actively affecting our society, even if they are hard to qualify.
Inclusive hiring practices, in my experience, strive to have a diverse funnel whereby under-represented groups get to be in consideration, but you still hire the best out of the pool. It may take longer to fill that pool, but many agree that it is worth it.
This take assumes a priori that inclusive hiring results in mediocrity. Sounds more like a reflection of biases, to be honest. Inclusive hiring means expanding your search criteria beyond "hire those that look like me, speak like me, have awesome education like me, and are basically smart like me". It turns out there are plenty of exceptional people outside of that narrow band.
> have awesome education like me, and are basically smart like me
At least this is meritocracy, the kind of thing that people (e.g. eugenicists) can make a serious argument for.
> look like me, speak like me
...is something that can't be justified except by terrible people. Even worse is "likes the same music and movies that I do" or "we coincidentally have mutual friends."
> Project 100,000 soldiers included those unable to speak English, those who had low mental aptitude or minor physical impairments, and those who were slightly over- or underweight.
You are moving the goalpost from mediocrity to disability.
I think structured interviews can be just as easily biased as unbiased. I'm not disagreeing necessarily about being structured with your interview practices, but I find this to be a grey area with promotions also.
As soon as people figure out the check boxes or the structured pointing system they start to check all the boxes, but it doesn't necessarily speak to the nuances between individuals that make them diverse and both valuable. In fact it can lead to a certain type of person people hired or promote, which can be on good characteristics, but I find many times turns into a "certain type" of person.
I guess what I'm saying is structure can take you so far, but you have to be willing to explore a little bit about what makes a person special, and that many times means not controlling the whole interview, and be willing to have your bias challenged through the candidate directing some of it.
In another comment I was also poking some holes into the “structured” comments, what you say above was in my head but I couldn’t quite articulate it. Well said.
I think checklist interviews miss the mark as you say. You may not have a perfect rubric, but I’m not grading students on a history exam; I am evaluating them for a role in a given position. In the limited time we have to speak, I want to use my intuition and experience as an interviewer and engineer to rapidly get to where the candidate is strong, and where they may have issues.
Agreed. It's very demoralizing to know that you are extremely capable of doing the job, but since you don't satisfy the narrowly defined fitness function you're being passed into, you fail. But more than that, companies are missing out on great people that may simply have a different way of thinking that isn't accounted for in their structure.
My take-away from interviewing is that as a candidate, getting a "you did not get the job" doesn't change anything. Since I've had the privilege of mostly being employed while interviewing for new jobs, this is a "meh, status quo" thing. Nothing changes, I have no decisions to make. If I get an offer, I have a decision to make.
Does it sting when I get a "No"? Yes, a little, but I did my best and (presumably) someone else did better. So, I take solace in that I did not have to make a (relatively large) decision.
I've also been kinda curious about the research in favor of structured interviews, because as far as I've seen they're generally drawing their conclusions based on "interviewer-predicted performance" vs "actual performance".
How the heck do you measure actual, repeatable performance? Or skill? Income / promotions / etc is very frequently a horrifically biased metric, for similar reasons to interviews, and we have much larger mountains of evidence showing that to be the case. It seems like there's a pretty good chance these studies are just measuring relative bias between interviewers and the interviewee's management, and concluding interviews are done poorly when they disagree with management. i.e. "structured interviews force people to think more like managers" rather than "structured interviews more accurately measure skill".
Using one bad measuring tool to conclude another tool is bad seems... problematic at best. I will grant that "interviews should measure what managers measure" is often what businesses want in bulk, but that does not seem like a particularly good thing to me.
Super minor language nitpick, but I thought you might want to know. There are a couple places where you seem to have mixed up empathetic and emphatic. One section is titled “Emphethatic”. I’m not sure if you were trying to make a portmanteau or just misspelled it, but I was confused by it. Other than that, I appreciate you taking the time to write up something like this. Wish it existed when I interviewed in my younger days. (MS also asked me the manhole cover question.)
Project based interviews, even paid, are so frustrating because there is no ROI if you don’t get the job. If I study leetcode, I can apply this skill to interview at a bunch of companies.
I once spent 2 hours coding Tetris for an interview. I lost to another candidate that completed 2 more features than I did in the same time period.
I agree with the conclusions, though I've seen the structured part go wrong, e.g. the interviewer is so dedicated to following the structure of the process that they forget about the empathetic part. These interviews look more like scripts than exploration of a candidate.
So I'd add another criteria: interviewers need to be trained!
My biggest recommendation is that those working directly with the new hire, peers, direct reports, subordinates, counterparts, have a vote or veto power.
HR and recruiting relationships are sparse at best.
Have to also have a challenge process. I built an infosec talent pipeline for a fintech; stakeholders get a veto but the hiring mgr can go back to the veto voter and ask them to dive deeper (and possibly perform an additional candidate call) to confirm the veto. >1 veto = no hire.
It’s working well, and has avoided at least two false negatives since implementation within the last six months.
A scenario in which I can imagine this backfiring is if the new hire and the vetoer end up working together. With an open mind that could be overcome, as in getting positively surprised, but let's be honest, how many people do you know who really have such an open mind?
No, both going great, which means (it appears) I’m balancing the org’s health and need to succeed with giving candidates an opportunity they might not otherwise have had.
The purpose of the system is what it does. If desired state is not emerging, we must adjust and observe accordingly.
It sounds good but do we have any evidence that this actually works? There’s so many of these speculative “how to interview” posts but it’s all just cargo culting.
If you really want to hire engineering talent, paying above "competitive salary" is very important.
It's a bit orthogonal to the concerns in this article, but in some ways it's much more important.
What I wonder about is given an org that is able and willing to compensate at market clearing rates, how do they get the word out well enough to get engineers interested. Because the other big BS in hiring is the whole recruiting side of things.
This is focused on finding technically skilled engineers, but I think you can get a more wholistic (holistic?) view of the person by asking them to walk you through their work history, project by project, and call a subset of the people they’ve worked with.
It’s more conversational, and you don’t have to live in hypotheticals.
We all know that skilled engineers will learn whatever skills they need to on the job, so less and less am I interested in what they can do in the interview pressure cooker.
> the best experiences were when the interviewer wanted me to succeed, was emphatic
I assume you mean empathetic. Same word is spelled “Emphethatic” later. (I tried finding a way to reach you privately, but your site “about” says you have contact methods on the left but, on mobile, there is no left…so here will have to do.)
The next time any recruiter asks me to do a "homework" assignment, I will ask them to write me a 15-page essay explaining how that homework assignment will actually be used, to what extent it will be reviewed, and what criteria it will be judged on.
If the comeback remark is something like, "if you really want this job," I will reply, "if you really want to hire me."
My current job, I told them that I was too busy to do such a thing (and I was), and got hired anyway.
Nobody w/ actual responsibilities in their life should be coerced into doing something for free, for someone they do not know.
Will an attorney give you free legal advice until you decide they are fit to represent you, or will you get free surgery until someone proves they won't completely butcher you? Can you drive a car for free (for 5 - 8 hours) until you decide that's the car you want to buy?
Why is it any different in the software industry? Because we just clack on our keyboards all day and do nothing?
> Will an attorney give you free legal advice until you decide they are fit to represent you
I agree with your point overall, but I do have to say that it is very common for lawyers to give free consultations with potential clients, often offering very useful advice. This is something I've benefited from more than once, actually.
And in fact, if as an employer all you require is a copy-pasta employee contract, you probably still want to go through a lawyer to have a professional opinion and work. But they on the other hand often just copy-pasta, and write an expensive bill.
>Will an attorney give you free legal advice until you decide they are fit to represent you, or will you get free surgery until someone proves they won't completely butcher you?
These are bad analogies because both have extremely extensive tests that are not only unpaid but the tested pays a small fortune for.
If development had the same no one would be asking you.
>I will reply, "if you really want to hire me."
It’s not about hiring you it’s about trying to prevent hiring the wrong person which is extremely expensive in time and money and takes weeks to figure out.
I understand what you're saying, however, I think that a lot of the criteria used in these homework assignments can instead be worked out via some conversation to judge the depth of knowledge and culture fit.
I've passed tech interview challenges only to fail the culture fit because I thought it would "be so easy" after the tech challenges.
And I've passed culture fits only to fail the homework assignments.
In one circumstance, not realizing that a separate recruiter was sending me to a company I had previously interviewed with, I've also seen previous work that I did in a homework assignment, given to me in a different homework assignment with a "what improvements and features would you add to this solution?," when the original homework assignment was the same task. That was a major blow, and gave me feelings they were using some of that work internally.
I get that it's extremely expensive in time and money to hire the wrong person.
On the other end, it can also be extremely expensive in time and money to not be extremely selective of whom you want to work for.
lmao do I understand correctly that you were given code that by chance you yourself happened to have written and were asked "how can this piece of crap be improved?"
>Can you drive a car for free (for 5 - 8 hours) until you decide that's the car you want to buy?
This is a perfect analogy.
I went to a Chevy dealer once, asked if I could test drive a car and they wouldn't even let me take it off the lot. I was allowed to trundle around the rows of cars at walking speed with a salesman in the passenger seat.
I went to a Honda dealer, and they let me take a test drive with a chaperone, but only around a short designated loop of streets "for insurance reasons".
I went to a Mazda dealer, and the salesman said he was busy and tossed me the keys and said have fun.
An acquaintance went to a Subaru dealer, and took the car they were considering home overnight.
Going further off topic, it amazes me how cagey some dealers can be about test drives. Same as you, I've had some just toss me the keys, without even wanting to see my drivers license, and that makes an infinitely better impression than wanting to come with me or wanting to talk about my needs first. I don't understand why it's not standard.
I think it's more up to the individual dealer than a policy of a given brand
It's sometimes useful if the salesperson can tell you things about features of the car while you're driving, but usually I know more about the car than they do.
I'm sure that some brands/locations have essentially zero risk of theft while for others it's a bigger concern. It's similar to how some places you can just pay in full with a personal check and they are fine with it (even not doing a credit check), while others won't accept that.
Insane. You've got to at least test drive on a highway.
Some years ago, a Nissan dealer offered to let me take a high-end crossover home for the night without any prompting. Ended up buying it. Would buy again from that dealer if I were still in that area.
I've bought three cars in my life and never drove any of them before buying. I know it's unusual but they've all been great. I did do a lot of research ahead of time though.
> Will an attorney give you free legal advice until you decide they are fit to represent you
All but one new legal engagement I’ve entered into started with a free consultation. (The only one that didn’t was a straightforward real estate transaction where I knew the lawyer for years beforehand.)
I don't pay $200,000 a year for a car. If I were going to then I'd like to test drive it for 5-8 hours. An ideal test drive should help both sides understand what the future relationship will be like. That saves time and disappointment on both sides. You'd like to date someone before you marry them wouldn't you. I would always be happy to pay for a test drive though. If you're not willing to offer a test drive then you'll also probably miss out on some great relationships in your life.
It is entirely normal to have a test period of 1-3 months here (inside the EU, which is not homogenous and doesn't look like the UK at all) where you can be fired immediately.
It isn't a magic bullet. I've had people perform well then tail off and while it may be a failure of my management that I couldn't help them perform better again, it isn't the norm and replacing those people isn't easy.
Rightly so, it should be hard to fire people and should require first showing evidence of trying to work together and help in good faith.
That's certainly your prerogative, and we all have to draw the line somewhere in terms of interviewer demands. Personally a reasonable homework assignment (like 2-3 hours tops) is less annoying than multiple 6+ hour interview panels stretched out over a period of months, which is also very common. At the end of the day though, the company decides the hoops you have to jump through if you want the job, and you can either take it or leave it. FAANGs can get away with onerous processes because of the technical brand and outsized comp, but startups and lesser known companies who try this are shooting themselves in the foot.
> Because we just clack on our keyboards all day and do nothing?
No, it is because some people have worked at very impressive jobs and probably just clack on keyboards all day and actually did nothing while there, so when you hire them you learn that they actually can't do the job you hired them for. You just can not rely on the CV alone.
In the hiring process there has to be some kind of skill test. If it is not a "homework" then it has to be a whiteboard/live coding/system design type of interview and there are a ton of problems with this type of skill test as well.
We usually give people the choice which route the candidate would like to do. Take a 1 hour interview or a "homework" assignment which took me about 1 hour to solve. Which is probably best because some people really prefer the homework because they get nervous in interviews.
> Will an attorney give you free legal advice until you decide they are fit to represent you, or will you get free surgery until someone proves they won't completely butcher you?
I do not know how attorneys or surgeons are hired, but my guess is that the education side of these jobs lines up much closer what is actually needed to do the job (which isn't really the case with Computer Science) and that when they claim to have done X or Y then it was actually them in court or at the operation table and not someone else from their team.
Clients select their attorney based on reputation of the attorney or the law firm they work for. For Surgeons it is probably similar. If you have the choice you want to go to the hospital with the best reputation. If the company you apply to already knew your name and reputation beforehand then the skill test is probably also not necessary, but that is not usual in my experience.
It also probably helps that both of those jobs require a Professional License to practice these jobs. Maybe if the software industry introduced a Bar Exam then these "homework" assignments would not be needed anymore.
> Can you drive a car for free (for 5 - 8 hours) until you decide that's the car you want to buy?
Usually you can. At least in my experience. Not fo 5-8 hours of course, but long enough to know.
If you know of a better way to hire people: I would be happy to listen.
An interactive whiteboard / live coding session is by far greater than any homework assignment, in my opinion.
W/ an interactive session, you get instant feedback (either verbally or via emotional cues) into what they are expecting.
With a homework assignment, it's hard to determine which path to optimize for.
If a homework assignment is necessary, it would be better if that were more of a "probationary employment" type scenario, perhaps at a very reduced amount of pay, to only imply that both parties actually have skin in the game.
Even better, for helping me judge if it's a decent fit? Show me some code you're using in production so that I can code review it on the call. Surely it's not all hyper sensitive.
I work on the matching engine of a stock exchange. So it kind of is fairly sensitive, but I get what you mean.
> interactive session, you get instant feedback (either verbally or via emotional cues) into what they are expecting.
For some people, like myself, that is a nightmare scenario. I do not do well in these type of sessions. Which are also not even close to reflecting the real work we do. There are absolutely no meetings I go into that I can't prepare for [1] and there are absolutely no meetings where I have to solve a problem on the spot.
Being on the autism spectrum also means "emotional cues" are pretty lost on me.
There is also the fact that interviews have to be during normal working hours. Some people prefer to do a "homework" assignment which they can do in an evening or weekend.
This is again why we provide the choice. Not everyone is the same and prefers different interview and assessment routes. Both types are useful for different people.
> If a homework assignment is necessary, it would be better if that were more of a "probationary employment" type scenario, perhaps at a very reduced amount of pay, to only imply that both parties actually have skin in the game.
Really? You would refuse a "homework" assignment, but would agree to "probationary employment"? The later just sounds like WAY more work on both sides.
We have discussed something like this in the company I work for, but came to the conclusion that it just is way too much work. Getting the contracts in place and working out the insurance and tax implications and all that. It just is way too much work to do legally, because it would be the same amount of work to just hire them. However: we can't just hire everyone.
Maybe if you are already a freelancer beforehand then we could work something out that way, but in the jurisdiction I am in not every software engineer is ready to accept freelance contracts. It is a simpleish process to do, but not everyone does and those that do don't apply for full time positions.
[1] You can "prepare" for interviews, but more in a scatter shot approach studying all the interview questions that could possibly be asked. That is not what I mean. There is no meeting I am going into where I do not know the precise topic that the meeting is about.
I have been caught off-guard by coding tasks in the middle of an interview that I thought was more geared for casual conversation, and it's not always a comfortable feeling, but sometimes it can prevent either party from wasting a lot of time.
However, I've also been a few hours into a homework assignment thinking that I could probably go down some rabbit-hole to try to perfect something that I may have been struggling with, and sometimes can't determine the appropriate stopping point.
The "probationary employment" would be more like, I don't know, a gift card, or something, vs. something formal.
That way, if I totally bombed out in some assessment, no big deal, here's something for taking the time to apply, and maybe I could use the card to buy a book.
Now, I get that companies aren't giving gift cards away to all of their interviewees, so this type of thing would only come after at least the first round of interviews, etc.
More often than not, a non-interested company will often not even tell why they didn't pass the assessment, and it generally feels like a waste of time.
Yeah, that is what I mean when I say that either way has problems with them.
I had someone try to implement a whole relational database when the interview task was just to read from a CSV file and provide a REST API to the contents of said file using any tech stack. Impressive for sure, but unnecessary time wasted.
> The "probationary employment" would be more like, I don't know, a gift card, or something, vs. something formal.
We discussed something like that in the company. Mainly because someone asked to be paid for the time spent on the "homework" assignment. We came to the conclusion that there is no real legal way for us to do so. We probably spent more money on discussing the possibility of paying the candidate then what 1 day of work would have cost us, but the cost wasn't even the issue.
With an in person interview there was maybe a chance. Inviting the candidate out to the exchange, giving a tour of the trading floor and then paying for transportation, lunch, dinner and hotel would be no problem.
Though interviews are online now as we are a "remote first" company anyways.
We just can't pay for work without a contract, insurance, tax and background checks in place. We can however ask them to complete a test. Which is what the "homework" assignment is.
> More often than not, a non-interested company will often not even tell why they didn't pass the assessment, and it generally feels like a waste of time.
It sucks. It just is that nothing positive can come from providing feedback and you open yourself up for a lawsuit.
The downside to 'probationary period' is that as a candidate you're limited to a single company. If you have to spend a week/month at X, you can't really consider Y, you won't be able to give them the time. Then at the end of the week/month X gives you an offer and you can't even compare it to any other offer, because you didn't have time to get any.
Another thing is, if they let you go after the probationary period, you pretty much start the search anew. If you applied to another company while you were 'employed' by X and you managed to schedule another probationary period with Y right after the end of the trial period with X, you have to refuse them last second in case X wants to hire you. If you didn't search for anything while at X, you're off work for another week/month while you search for another trial.
If X ends up giving you an offer and you want to consider Y, what are you gonna tell them? Please wait a month for me while I work for this other company and see if it's any better than you? This simply doesn't work, for both companies and candidates.
Your proposed solution works, if X is your dream company and you will accept their offer no matter what. That's not how most job searches go though.
Whiteboard coding almost always devolves into leetcode which also requires at home study. You're going to be spending evenings and weekends coding something in either case.
How many CEOs have you interviewed to run your company? How did you conduct the interview? Do you think that same interview is appropriate for a developer?
The fact of the matter is that without the skill test there are way too many people that are incompetent and you have to have some way to filter those out.
I do not know how it works with CEO positions, but I am not hiring CEOs. I do hire software engineers and would like to work with competent people.
If you know of a better way then I would like to hear it.
It's extremely intensive to onboard a new employee; and extremely disruptive to onboard a new employee and then immediately fire them. It's also very demotivating to the existing team.
I would say it's a lot more disruptive to leave someone incompetent on the team and let everyone else make up their work instead of building a solid team.
We would like to know who is competent before we hire them. The point is that we do these types of tests so that we don't hire them in the first place.
We can't hire everybody. I can't hire 20 people (then fire 19 of them) for the 1 position in the 8 Person team I want to fill. That can't work.
Just hiring someone and then firing them 2 weeks later is expensive as fuck. We can't just do that until we find someone actually competent.
Your "easy solution" is only easy if you don't think about it at all.
You're just hiring people who care to do these types of tests or are good at them, they don't mean they will be good at the job.
I'm not saying you don't do ANYTHING to hire someone. Obviously there needs to be some judgement of skill. But take home projects where you're judged essentially against how much time you put into them (I've personally had projects where they asked me to build an entire web app, it's insane) because ... it's a race to the top in a way here, if you spend more time it's going to look good, because the best projects are from people that spend a lot of time on them...
It's just ridiculous. You end up doing 8-16 or more hours of work for a job you likely won't get (it's free for you to ask me to do this test, so you ask everyone to do it).
It's not expensive to hire and fire someone. Not that expensive. It's probably worth 20hrs of 100 candidate's time, that's for sure. If you don't think so, I have a bad opinion of you.
I'd like to ask you, how many candidates a year do you have turning in take home projects?
> You're just hiring people who care to do these types of tests or are good at them, they don't mean they will be good at the job.
We align the Take home assignment very close with what the position actually entails. If you struggle with this assignment you would not succeed with actual tasks we do daily.
> Obviously there needs to be some judgement of skill. But take home projects where you're judged essentially against how much time you put into them
As I said elsewhere as well: People are free to choose to do a Technical Interview instead. If you fear that you would need 2 whole days you could take the 1 hour interview instead.
> if you spend more time it's going to look good, because the best projects are from people that spend a lot of time on them...
That is not my experience. There are people that just can't make it look good.
> It's not expensive to hire and fire someone. Not that expensive.
It absolutely is. Maybe not where you are, but where we are it is expensive. For one there is a big amount of paperwork involved, contracts, NDA's and background checks. We can't do this unless we know we actually really want to hire you.
I also can't expect a new employee to be productive right away. There is going to be a onboarding period of at least 3 months during which the employee probably needs a little bit more mentoring of a (more) senior engineer. If we would just hire 20 people for every position that we need to fill we would not do anything else then just onboard new people. So, YES, it is significantly more then 20h of 100 candidates time. Not to mention that the take home we give out takes maybe at a maximum 4h (if the candidate is fairly junior and has to look up stuff constantly, many people just do it in 1h).
Also: I think it would be HIGHLY unfair for the people to just hire 20 candidates if you only plan to keep 1. They probably quit their jobs or said no to other opportunities. We only hire people we see a long term future at the company.
> I'd like to ask you, how many candidates a year do you have turning in take home projects?
It is fairly late in our hiring process and it takes a significant amount of time from at least 2 senior engineers each time to review. I don't have the exact numbers how many times per Year, but we try to keep it very low and only if there is an actual interest on our side.
I've tried to push my employer(s) to offer the dual paths. Like you I prefer the HW-style process (and think they are a truer judgement) but I understand why some don't like it. I wish more companies would support either path.
> it is because some people have worked at very impressive jobs... they actually can't do the job
> In the hiring process there has to be some kind of skill test
Thats a contradiction:
1 - 90% of companies do skill tests
2 - you cannot trust the CV of a candidate even if the company they worked on previously also did a skill test
3 - you think your skill test will enable you to hire the right candidate
So everyone is doing tests and everyone is hiring shit candidates anyway?
That's literally proof that these tests are worthless.
Maybe if the recruiters actually knew what they need from a candidate, companies were clearer about what the job involves, they would stop hiring the wrong people for the wrong job.
There are some studies that claim a large percentage of people (40%+) lie on their CVs. As someone who hasn't done that and struggles to lie even in games, I have a hard time believing it. But even if 5-10% of people lie, hiring is a lemon market, and you definitely don't want to get stuck with the lemons.
Most lying is apparently to change start and end dates of previous employment to cover up short stints where they may have been fired for poor performance, but even if work samples/skill tests/structured interviews were 95% effective, you would still regularly hire duds.
We asked him some questions about the code when he came in for the last interview and contract signatures. He didn't have a single clue about any of what he supposedly wrote. When we talked further he admitted to using a Team in India that has been working for him for a while.
Needless to say we didn't proceed with signing the contract.
1 - I don’t know the hiring standards of every company and that 90% sounds made up to be honest.
2 - We can not trust the CV. Maybe the company did a test, maybe they didn’t. Maybe their test are to our standard, maybe they aren’t. Maybe the candidate barely passed. Maybe the candidate actually failed but made up in other ways that might be relevant to the other company, but not ours. I doubt anyone would give us this information about their current or former employees. Maybe the candidate really did work on the projects he claimed or maybe he was only tangentially involved. That is a lot of unknowns. A test clears up what the person actually can do.
3- Yes. It works for us. Do we miss out on some good candidates because of this? Yes, but it definitely prevented us from hiring bad candidates.
Again: if there is a better way then I would like to know. I have not found one yet.
The better way is to treat people's time as important as you treat your own and your resources at your company.
There is a large company based in NYC that interviews thousands of developers a year, but only hires a few hundred. Each of those devs do a take home project that takes about 16 hours to complete. The project requires nothing to ask, so this company asks everyone. 30,000 hours is a lot of time - it's about 3.5 years of someone's life - and at least this time is spent each year by this company (and they pay nothing for it), not counting the rest of the interview process, because it is cheap for them to do this.
And we wonder by productivity is down :)
Respect people's time. If it is hard, figure it out. That is your job if you want to hire people and feel good about it.
Will an attorney give you free legal advice until you
decide they are fit to represent you, or will you get
free surgery until someone proves they won't completely
butcher you?
There's a big difference between both of these industries and the software business - if you show up claiming to be a lawyer or a doctor and are bullshitting, you're in a LOT of trouble.
If you do that in software, you're probably just back on the job market...
"Can you drive a car for free (for 5 - 8 hours) until you decide that's the car you want to buy?" Yes. It's called a test drive. I think the longest I've seen is a free 48 hour test drive with an online e dealership, but free all day test drives are the norm in the UK.
Similarly most lawyers will give you a free consultation.
I think you have a highly distorted view of reality, how other professions operate, and of your own worth.
I have my own list of questions. If they answer all of them I have a pretty good idea if I'm the guy for the job. The most wonderful part is figuring out if it is an employer is looking for initiative or obedience. If you are running a sheep farm it can be very exciting to see initiative.
Maybe I'm weird but I kind of like the take-home assignments. I find them pretty fun and not at all stressful like a whiteboard interview. And I've never not gotten an offer this way because I think there's a lot less variance (whiteboard style interviews you might get asked some random BS that you don't figure out).
IME you also often get paid (albeit a relatively small amount) for doing them. And FWIW once they want to hire you many employers will actually spend hours trying to sell you on joining. Usually not in the form of an essay, but doing several hours of sell calls or lunch meetings with different people at the company is not that strange (especially if you are higher level).
You’re not weird, just lucky. I too have gotten my longest job of my career from a successful take home exercise, but I have never been paid for my work and am rapidly reconsidering just how willing I will be to do take home exercises for free in the future.
While I do agree with your general point your analogies are bad because yes, you can test drive a car, yes attorneys often have a first free session with potential clients.
That being said the come back makes perfect sense to me. When I was still on the IC track and interviewing, when I was submitted to some "dumb" questions I would ask the same kind to the interviewer. So you just asked me to write this stupid algorithm? How about you whiteboard something for me now?
It was almost always met with dead silence to which I would say "well you're trying to figure out if I'm good, I want to do the same and make sure your team has good engineers".
Of course it never went anywhere since, while they had 55 minutes to grill me I was only given the last 5 for my questions.
Because take-home exercises are both the best way to evaluate a candidate (I have never gotten a nasty "surprise" while hiring in companies that did take-home exercises: I mean it, absolute 0% "bad hire" rate, the only bad hire I got was a candidate that was allowed to skip the take home exercise) and they allow you to find absolute jewels amongst the candidate pool who are great, reliable engineers but regularly bomb regular interviews.
While there are people who are naturally great at interviewing in person or who don't mind grinding leetcode for free for months on end, there is a whole "base of the iceberg" population who a) can't spend months grinding leetcode on end but they can definitely spare a couple of afternoons for one job they are interviewing for – these are normal people with normal jobs and a normal family who don't interview every month just for kicks, they do this a couple of times every 2-3 years, and, b) just cannot code while under stress. You may scoff at b) if you've never experienced it, and while I found it really hard to understand for a while as I also have no issues with getting into deep focus while people are looking and/or trying to talk to me and there's a time limit, it is absolutely real.
Unfortunately, there is an extremely vocal minority (you) who go absolutely ballistic when asked to do a take-home exercise, who absolutely ruin it for everyone else and make hiring managers shit their pants whenever take-home exercises are suggested. I honestly don't understand why there's such an outrage, take-home exercises are the minority already, because there's a strange huge backlash.
Sure, if you're a great communicator (usually native English speaker) and grind leetcode for fun so you can shove 5 interviews in one week, you hate take-home exercises. And that's fine, really, just please apply to the leetcode grinding contests and stop poisoning the well for the rest of us who would like to cater to the large amount of extremely competent engineers that don't fit that persona.
> Nobody w/ actual responsibilities in their life should be coerced into doing something for free, for someone they do not know.
I take it you haven't interviewed for a SE in a while? The status quo requires you to perform months of unpaid labor, not days, just in the form of memorizing the solutions to every single Leetcode Hard problem. How is that better?
> I take it you haven't interviewed for a SE in a while? The status quo requires you to perform months of unpaid labor, not days, just in the form of memorizing the solutions to every single Leetcode Hard problem. How is that better?
I guess because it kinds "scales". Once you are familiar with Leetcode you can use this skill to take interviews with different companies. Take-home exercises dont' "scale".
That being said, I believe people are against take-home exercises exactly for the reason you support it: it's usable code. They worry the companies will exploit candidates by using their code for free. Leetcode is "useless" so it's safe.
For clarity, these take-home exercises that I'm referring to are completely toy problems. They're usually one or two REST APIs on top of a toy data structure (sometimes in-memory, without a DB at all), just to see how the candidate models, writes and tests code. An API that does some pricing math based on some in-memory state or the likes of it.
If we're talking about "make me a website" kind of exercises that one could throw into production after some tweaks, then that is free labor and I would absolutely refuse those.
Also, the fact that "leetcode scales" is part of the problem I was talking about. I really dislike the fact that if you train for months to develop a complete set of skills parallel and irrelevant to your actual role you can now efficiently interview for a number of companies that pay top dollar. So not only people with actual skills and experience are on equal (or worse!) footing than someone straight out of college, it also incentivises those who mastered leetcode to interview everywhere, since "it's all the same", while normal people who are good at their job get their torn to shreds after one or two interviews. So not only it's judging the wrong thing at the interview, but it's also causing a starvation situation for the candidates that are not playing that game.
Why are we creating incentives to people becoming "professional interviewers"? This would be like Google encouraging SEO spam instead of fighting against it!
> Google: 90% of our engineers use the software you wrote (Homebrew), but you can’t invert a binary tree on a whiteboard so fuck off.
— Max Howell (@mxcl)
If inverting a binary tree means swapping the left and right subtrees of every node, I wouldn't want to work with someone who can't do that either and Google is definitely right to reject him.
… but that's an interview. If you cannot, within the time, demonstrate any ability, why should you be hired?
The question above, as clarified, is not complicated, nor does it rely on memorization or some "trick": anyone purporting to be a SWE should be able to write an essentially de novo solution to it.
(And in my own technical interviews, there are multiple questions, to specifically hedge against any one being "that one question a good candidate is going to miss because it's just not their day". It doesn't happen: it's either all or nothing.)
> If you cannot, within the time, demonstrate any ability, why should you be hired?
It is more likely that the interview process is broken and missing the right candidates, than it is that the interviewees are all mediocre. Most interviews are very non-inclusive the same way that the main track of school is becoming less and less inclusive. Different people need different methods to bring out the best in them.
> It is more likely that the interview process is broken and missing the right candidates, than it is that the interviewees are all mediocre.
Here's a thought experiment for you: if the interview process is so broken, why hasn't some tech company succeeded and become famous for an improved interview process, e.g. "Moneyball style"? My guess is because the process is not actually that broken, at least from the employer's perspective. I'm sure the interview process could be changed to be less regimented and more "inclusive", but that's also likely to reduce it's predictive power (i.e. you're more likely to make bad hires, and from a company's perspective that's almost always worse than missing out on a great hire).
The layoffs that I have seen reported have been reported as being random. I've been involved now in 3 layoffs directly in my career, and 100% of them, the laid off individuals were laid off without regards to skill. The reporting in the media on layoffs happening elsewhere largely matches my experience.
Sure, an argument exists around "you shouldn't've hired that many people", but that is different from an argument of "the hiring process can't discern good hires". The former is a management & long-term planning issue, the latter is how interviews are conducted.
Sure, your day-to-day work may not involve manipulating binary trees. But presumably it does involve working with variables, objects, references, manipulating data of some kind... And if you're comfortable with the fundamentals of those, then this is something you should be able to figure out even if you've never heard of a "binary tree" before, once somebody has sketched it or shown you the definition of their TreeNode class, right?
It honestly baffles me how people consider this something which needs to be drilled or memorized.
There are absolutely algorithmic questions which would fall into that category. But if somebody considers this to be one of them - or something like "find the smallest number in an array" - then I have to question whether they have an understanding of the most fundamental concepts in programming...
Or if they get through each day solely using things they've memorized by rote, or looked up, and they don't really have any idea how any of the foundations they're building on actually operate.
Judging by Google's track record, hiring some people who have demonstrated an ability to launch and maintain a piece of software would bring a skillset they desparately need. Google engineers are incapable of keeping much going for the long term.
Wouldn't that be a "mirror" operation, while inversion would be (I dunno) swapping the direction of the edges?
I went out of my way to avoid homebrew (still do) when I worked at google because it would reliably fail to complete some key operations in a dag, hence the interest in ensuring developers know how to do CS things.
Yeah it's not clear what he was asked. Swapping the direction of every edge of a binary tree would result in a DAG that is likely no longer a binary tree though.
"Mirror" is what this question is generally understood to be asking. I don't think swapping the direction of the edges produces a tree in general although it would produce a DAG. I think "invert" could mean "mirror" or turn upside-down depending on the context.
Personally, if I were asked this, I would just say "convert the graph to a matrix, invert the matrix, and then convert the resulting inverted matrix back to a graph", and let them try to figure out if that would work for a bit before joking "oh come on, preorder traversal with a temp var, do you have a more interesting question?"
It would be more leetcode to be given an ordered binary tree and asked to reverse it O(N). It's a lot more fair to the interviewee to be given the explicit task without knowing 'the trick' unless one considers knowing recursive functions to be a trick.
The actual work for which you are hiring an engineer is building a software product/service, and the Homebrew developer has a track record of delivering great results.
Rejecting the guy because he cannot do a whiteboard brain teaser is like rejecting LeBron James because he did not make a shot at the arcade basketball game.
I'm not saying the guy would be perfect. Comparing him to LeBron James might not be a great example. Google might have other reasons to reject him.
What I'm trying to say is the current coding interview is a really poor mechanism to gauge a software engineer, especially when it comes to hiring one with real-world engineering experience.
> What I'm trying to say is the current coding interview is a really poor mechanism to gauge a software engineer.
People like to say this, but in my experience this is not true. It's just that people misunderstand the goal of technical interviews and they often are poor at evaluating their own skills.
First off, these giant tech companies have enormous economic incentives to improve their interview processes as much as possible. They also do a pretty rigorous assessment of the effectiveness of their interview process (Google, for example, has publicized some of their data). I'm not saying these tech companies interview processes are perfect, but I also have a problem believing they're so fundamentally flawed that these companies can't figure out how to fix them given the giant economic returns they get for optimizing their hiring processes.
Moreover, as some other comments mentioned, many companies (and individuals, myself included) believe it is much worse to hire someone who ends up not cutting it, than missing out on a potentially good hire. I can list out all the reasons why, but Joel Spoelsky has a pretty famous essay from a couple decades ago on the topic that explains it well [1].
Thus, it's not surprising hearing a lot people complain that they can do the job, but they aren't good at interviews. Because, from Google's/Microsoft's/etc. perspective, they're fine with a bit higher false negative rate if they can greatly reduce their false positive rate. And my experience matches that: I have never seen a candidate who did awesome in "whiteboard-style programming questions" who couldn't cut it programming-wise (they may have had other issues, but "coding productivity" wasn't one of them). Now, I certainly believe and have seen that there are some people who aren't good at these questions who can do a job well, but there are also a ton more people who can't do the job if they can't pass a technical screen, so hiring any of these folks means much more risk.
I also think that whiteboard-style coding questions help show a quality that is very important to businesses, even if those questions don't represent "real world" work. There are basically 2 types of people that do well at these questions: people who are just naturally smart and have a ton of experience to the point that they wouldn't even need to study to do well, and people who are of more "normal" intelligence/ability, but who can do well if they study a ton. Either of those two groups would likely do well in a programming role. So often I hear the complaint "I'm a busy person, I've got outside responsibilities, you can't expect me to spend all this time studying". And that may be true, but you'll be competing against people who are willing to study, so I don't think you can fault Google et al for favoring people who show a willingness to do more preparation.
1. https://www.joelonsoftware.com/2006/10/25/the-guerrilla-guid... "And in the middle, you have a large number of “maybes” who seem like they might just be able to contribute something. The trick is telling the difference between the superstars and the maybes, because the secret is that you don’t want to hire any of the maybes. Ever."
> They also do a pretty rigorous assessment of the effectiveness of their interview process
Companies do review their hiring processes, but actual experiments and data seem fairly rare. It's harder than you think. What experiment would you run? Hire a group of people entirely randomly, and compare their performance reviews after 2 years?
That's pretty much exactly my point. In the 90s, wide-scale hiring for software engineers was a relatively new thing - many companies were just figuring it out. And so they did some shit pretty early on that didn't make sense. But for all the times I hear folks pulling out the "Why are manhole covers round?" and "How many cars are there in Manhattan?" examples, I haven't heard these types of brainteaser questions being used for nearly 2 decades.
I'm not arguing that the FAANGs have some perfect, unassailable interview process that can never be improved, but I am arguing that so often I hear grumbling discontent from people who don't like the interview process, but rarely do I see much examination around why those particular hiring processes appear to work fairly well for the likes of Google, Apple, etc.
Those puzzle questions weren't just used to hire software engineers. Hiring fads sweep corporate America, despite no evidence of effectiveness, and despite economic incentives to hire well.
Yes, they did move away from it, but that doesn't mean we aren't now in the grip of equally bad fads.
> but rarely do I see much examination around why those particular hiring processes appear to work fairly well for the likes of Google, Apple, etc.
You assume they work well, but you don't have any data to support that. That's sort of assumption is basically where these hiring fads come from.
Maybe he would have been a poor fit at Google. Just because you wrote some popular open source software doesn't mean you will succeed in a corporate environment. The dynamics and skills required are quite different.
Now does being able to reverse a binary tree mean that you would fail at Google? I have no idea. But we don't really know if that was the reason he was rejected, it's just his own guess. There could have been other reasons.
Also, I don't believe that 90% of Google engineers use Homebrew. I'm not sure I'd believe 9%. Google is a Linux shop with its own internal package repo. Even if you're using a Macbook to work remotely, you're using it as a fancy terminal wrapper to connect to a Debian-based system to do your real work.
Most of us havent touched a tree structure since college, because there are other, real, problems out there. Trying to remember, or rederive it from scratch is slower and error-prone and bad for interviews
Tree structure ? json, xml, protobuf, classes, functional programming, databases with foreign key, database internals, etc ?
Oh I forgot you also serialize and deserialize data - did you forget how that works ? Tree traversal again.
Do you know how organizationl hierarchy is structure ? It's a tree.
Do you know various maps and their usages ? We use them daily - it's very very important to know their internals. Hashing vs Trees vs Linked hash vs etc.
Google maps ? n-d trees ? Comparing data - merkel trees ? etc.
Every dev out there has common work with mine. But you won't be able to solve the problems that I face on a daily basis without thinking hard & without this dsa + concurrency knowledge.
Now, is it reasonable to ask these questions ? Heck yes.
That is a fine and common opinion, but just one question: how often have you inverted a binary tree at your job? Because after nearly 20 years it hasn't come up once for me. I am sure for some roles it is a necessary skill but my issue is that most of these questions are more or less toy problems that come from academia and not business. They are a great test of your retention of a data structure class but not super relevant beyond that.
I would rather hire an engineer with a strong business or user sense - reading between the lines of requests and anticipating future issues or uses adds so much more value in a real sense.
To me, these are great entry level questions because it is a good baseline for new grads when you have little work experience to judge. Past that, it is like making a lawyer take a mini bar exam for every new job - a waste of effort if you want to hire for specific skills and experience.
Just last week I had to implement a common graph algorithm from scratch to solve a business problem (basically topological sort, with a minor variation). It's certainly much harder than reversing a binary tree (which is one of the easiest possible interview questions you could imagine).
Generally it wouldn't really make sense to reverse a tree in practice (why not just build it the other way initially?) but it has a similar structure to other tree traversal things that could actually come up so it's reasonable to ask.
For every one of you, do you think there are more engineers doing things just like you or do you think there are more engineers who never have to do a binary sort and can still lead a perfectly satisfying, fulfilling career?
So if you are just doing normal stuff, then why should anyone pay competitive TC ? That's just a normal dev, right ? What's so special or talented about you ?
Companies that pay huge TC want to hire smart people not just an average joe. Sure, you can live satisfying career and that's your pov.
What about a company's pov ? Did you ever think about it ?
Just because you haven't worked in a team that requires those skills doesn't mean they aren't valuable.
In my old team, I had to come up with a coupon distribution logic based on count, percentage, time, then generating reproducable random values that required to deep dive (algorithm) into library code & explicitly storing state in redis, then an application of dynamic programming in building as custom platform, atomic token validation, custom rate limiter algo, state machine, scheduler, distributed circuit breaker, etc.
In my current team, I had to read raft paper, zab paper, look into their implementation, make a poc with raft protocol, then autoscaling algorithms, scheduler algo's, different data structures, heck even the oss engine itself is DAG, heavy threads + concurrency stuff. Even now I come across new data structures and algorithms.
Clearly you don't know the entire industry, just because you haven't worked in such teams, doesn't mean these aren't important.
You are experienced in a bubble. The hiring bar for our team is higher than other teams and heck even for SDE3 - the requirement is higher. You would be very much surprised to know that even the senior members have research publications and deal with complex stuff.
Core teams like in AWS or GCP or Azure solve these sort of problems.
Who do you think will solve autoscaling (that's what I'm doing now) or managed scaling or network or security or any infra problems in these cloud platforms ?
As experience increases, we expect more knowledge & insights - doesn't mean to ignore basic coding stuff like arrays or linked lists or trees or graphs or simple message queues or etc.
If companies are paying competitive TC and there are multiple candidates, why not hire a smart person ? What's so special about doing regular normal stuff ? That's just a normal dev right ?
This misses 90% of what I, a startup CTO, find valuable in technical hires. What I want to know is: what type of projects have you worked on, how did you develop expertise in those systems, what level of ownership over your work did you display, how well were you able to plan and design the solution to a problem, and how did you handle the execution over the X months of work to make it go live. Demonstrate expertise, curiosity, and ownership. “System design” is like 5% of the work we do, and it’s important, but putting the designs in motion and driving value from them is 95% of our time and that’s something we do not screen well for. The way that I do this now is a process with a soft skills interview, a coding interview, then a “case study/system design” interview where I have candidates write a system design doc at home for a project they have worked on IRL and use that as a starting point for a 45 minute panel convo where we review the doc and ask questions about their choices and how execution went.
Why would you say no to a job because they want you to interview for a few hours? As a candidate I always respect the employers more if they can put together a coherent 4+ hour interview. Why would I want to work somewhere where my co-workers were only briefly vetted?
This is 3 interviews, done remote. 45 minutes per interview. Can be same day, can be over a week, candidates choice. Haven’t had any issues with people thinking this is a blocker, if anything it’s a lower load and faster process than people are going through at peer companies.
I'd love a process like that, however it's much more common for companies I've interviewed with to not only not ask that type of info, but actively avoid and discourage talking about these things.
Been treated in past like I'm avoiding the "important part" of solving their quiz and that it's some softball topic.
You want to see repeatable behavior and a general interest in going through the process. If someone takes the time to apply with homework and is able to articulate well, it gives you so much valuable signal.
Pressure cooker style interviews only reveal someone can remain focused under stress and that they studied their leetcodes.
It's not a bad post per se but we've been reading similar, anecdotal blogs like this about making the interview process kinder for decades. Yet the only companies in a position to do a rigorous statistical test - large tech cos - stick with the traditional, somewhat adversarial whiteboarding process. I would even suggest that a strictly technical whiteboarding process can be less biased than what the author describes, because you can so regularly grade everyone on the same exact rubric. That's tougher when "pair programming" or doing a take home.
Also, stop giving take home projects. Bad candidates will cheat them and good candidates will not even do them. If one of the random startup names listed on the author's site sent me a 12 hour take home project I would delete the email. Do you think they pay twice as much as the bigger company that only makes you waste 6 hours doing a whiteboard? I doubt it.
Larger tech cos with very high TC know that they will always have a pipeline of more qualified candidates, so a false negative has basically no cost to them, whereas a false positive has a significant cost. I think the reason that they run these processes is because they have a very low false positive rate, and so long as that is true, it doesn't matter to them how high the false negative rate is.
And I think that smaller companies copy this as a part of the tendency to copy large companies without thinking about whether the thing they are copying actually makes sense at their scale. In this case it can be very damaging, because false negative for a startup with a limited pipeline can be very bad.
Also, as far as I know, nobody measures false negatives in hiring. How would you even do it? Keep track of everyone you rejected, and then 5, 10, and 20 years later check on their career? I'd be fascinated to see the results of such a study. I'm sure it would be super valuable if you could find some kind of pattern where some filter is falsely excluding candidates that are actually great.
I think the goal of avoiding false positives is actually more important for smaller companies. A bad hire at a startup can significantly shorten the company's runway, while a bad hire at a bigger company tends to get isolated and managed out without doing much harm (except to morale of course).
You're right about the copycat behavior. This goes all the way to top of funnel: these small, even trivial-scale web application startups just don't have hard engineering problems. Many imagine they do, or imagine they will once they take off, but the work they're offering these high powered candidates they claim to want to hire is like, wiring up CRUD apps and making javascript buttons. It's not technically deep work, it's product work. A little humility about whether or not your tech startup is truly doing "tech" problems would, I think, fix some of the expectation/reality mismatch people are having when they complain about how hard it is to hire engineers.
And sure, lots of people join Google to work on world-scale problems and end up wiring CRUD stuff anyway. But they can at least plausibly offer some technical depth (or could anyway, perhaps Google's reputation as a great place to develop an engineering career has been fading).
(this is not to knock on "CRUD" but to highlight that a technical problem solver is an overlapping but not identical skillset to someone who can work with a fast moving team to quickly and reliably develop product changes)
I agree about the false positives. What I'm saying is that false negatives are more costly at a startup because you have less of a good candidate pipeline. Passing up on good candidates extends your hiring process and may even push people to hire someone you otherwise wouldn't have later because you need engineers. Google doesn't care because they will have 100 more candidates the next day.
A bubble sort at startup co is annoying but often less likely to cost the company millions, at Google or Amazon it very well could. Extreme example, but I think the point stands.
You should mention, what kind of teams you've worked with and what kind of stuff you've built.
Every team has different requirements and hiring bar. In my previous company the low bar caused not so good (able to understand stuff, knowledge and connect dots) people be a burden to rest of team. Heck in 2 years, 4 important people have left the team due to hiring a bad manager (has neither tech nor soft skill(s)).
There wasn't growth in that team due to mediocre hiring and eventually all the good ones - left to other companies.
My current team is an infra platform and has lot of growth as IC. Everyone is learning something in-depth and are explorers - rather than blind sheep. The bar here is higher than the one for my previous team.
Our team requires you to know about whatever you talk on, not just usage but it's internals - why ? That's what we do daily. It can be about scheduler, checkpointing, auto scaling, concurrency, different data structures & algos, integrating with ecosystem, etc.
Even soft skills - like helping others, taking feedback, communicating clearly, etc.
Yeah so, mediocre will always be a burden to team.
Since it mentions the infamous interview challenge, I'll ask: has anyone ever "inverted" (i.e. swapped left and right recursively) a binary tree in production code?
I can't think of any reason why anyone would ever do this. Just navigate the tree in the reverse of your normal direction instead.
Why not ask the much more interesting and potentially useful question of balancing a binary tree? Or do something else recursive, if that's what you're after.
> Why not ask the much more interesting and potentially useful question of balancing a binary tree?
Or even just when would you use a binary tree? Figuring out which data structure is appropriate for the problem at hand is the hard part, how to implement operations on the data structure is easy in comparison, you can just Google it.
That seems to be the wrong question though. It seems to be jeopardy style "question". You are not asking which data structure is appropriate for a problem.
Here is the answer, but what is the problem it solves.
Never in my live have I sat down and said: I don't know what problem is I need to solve, but I know the solution is a binary tree.
You usually aren't coding FizzBuzz in production code either
I would say you're constantly coding FizzBuzz. Looping, modulus arithmetic, and conditionals are all over the code I've written. At least with FizzBuzz you have a test of a person's ability to understand a task, break it down, and make sure the logic is consistent. With tree "inversion" it's not even a sensical request, it's utterly useless, and there are countless more interesting and practical ways to test understanding of recursion and trees. Knowledge that, I would bet, isn't even relevant for 90% of programmers, and if tree traversal were relevant then you'd probably want to jump to way more difficult questions (I'm thinking of the Facebook graph, for example).
I agree with the other commenter that it would make far more sense to ask questions like "you need to process data of this type, what provided data structure (e.g. C++'s STL) would you choose?"
It's not really the point of the question. The LC interview is basically a standardized test to eliminate bias as much as possible. If you prepare, you should be able to solve it. "Invert a binary tree" is actually considered an easy problem to test your basic knowledge of how trees work and tree traversal.
a standardized test to eliminate bias as much as possible
Uhhhh... It eliminates competent, skilled people who don't have the time to memorize the latest cargo cult trends in hiring. Look in the article for a glaring example.
"Invert a binary tree" is actually considered an easy problem to test your basic knowledge of how trees work and tree traversal.
So would simply printing out a tree, and at least that's something a person might actually do.
Tree "inversion" doesn't even make any sense and at this point I'm convinced that the cargo cult is choosing it because it's the extent of their own understanding of trees and somehow sounds extra technical to them.
To be clear, I didn't imply I was agreeing with it. Just saying what the justification for this kind of question is.
I agree completely it eliminates a huge swathe of people, mostly experienced and older people. FAANG employees, ime, are biased towards childless / single people with privileged backgrounds.
Tree inversion sounds weird when you hear it phrased like that, but in an interview it would be explained with an example (just swap the left and right children recursively).
I think it's more that the "inversion" interview story became a bit of a hot topic, and so is something people tend to reference in these discussions. Not that it's a specific problem people think is great to ask in interviews.
It also seems to neatly split people into camps who think "this is trivial, and totally reasonable to expect somebody to answer, even if it's a little contrived", vs those who think "this is not practical, and you'd only know the answer if you'd already practiced it, so it's not fair to ask".
> It eliminates competent, skilled people who don't have the time to memorize the latest cargo cult trends in hiring
But given that you already acknowledged that it's pretty trivial, why would memorization be necessary for a competant person?
LC interview is biased towards people who are into LC/Olympiad/competitive programming or who have enough free time and willingness/desperation to grind LC.
The point is just to ask a simple tree traversal question, to see if you know how to traverse a tree (or more generally, write recursive functions and operate on basic data structures). They could have asked some other operation instead, but this one is one of the easiest so it seems fine.
You are just ignorant. You should read about distributed systems. Google around for Zookeeper and it's zab paper.
Every (recently with raft protocol) multi master distributed system out there interacts with Zookeeper for assigning leader and maintain configuration.
Do you know how the syntax or api calls look like ? They are node path in a tree. You want to store something ? That's a tree path again. You want to listen to some change ? It's tree path again.
As I said, most people are ignorant here and don't do "true" computer science engineering in daily life.
Most devs simply convert business logic into bunch of apis + adhoc implementation.
Did you ever work at a banking firm ? I've read their codebase - they are structured as trees, every damn thing is tree. It's a headache to navigate, code, heck even the objects are literal trees.
As I said, people are ignorant and think world revolves around them.
> Why not ask the much more interesting and potentially useful question of balancing a binary tree?
Honestly? Because that's harder.
The swapping question is basically a softball / FizzBuzz-style question to test the most basic familiarity with data structures, pointers/references, and recursion.
On a smaller size (company) it's relatively easy: pay well, don't oversell the position, send a small assignment representative of work and give them ample time to solve on their own OR ask for references you can talk to from previous workplaces.
Game changes if you actively contacted someone.. if you're no BS, assumption is you know who you contacted and why, hence only thing to do, once contact established is not to oversell and pay well.
Pay-well can constitue compensation as well as time.
I think there's a fundamental misunderstanding about the purpose of the whiteboard interview. The point is to eliminate, as fast as possible, candidates who simply cannot code [0] [1].
You can't do that with a take-home (and I'm against take home as the signal to noise ratio is too low) because people will cheat and have them done by someone else.
I've heard horror story of a "senior" engineer from "his country's top school" being interviewed for a technical position by several non-technical managers and HR reps. They only included an engineer in the final round, which was basically supposed to be rubberstamped anyways. He was then asked to implement something trivial like fizzbuzz or wordcount on the whiteboard. The candidate then became extremely defensive and tried to argue that such task was "beneath him", arguing for a good 15 minutes why he shouldn't have to do it.
Then the dev just left the room and said that he used this question as a warmup with new hires and it typically takes them less than 10 minutes.
Now, a lot of folks do whiteboard interviews wrong. They often expect to get the exact implementation of an algorithm they found in a textbook and for code on the board to compile. This isn't the point of whiteboarding. Doing this only promotes rote memorization. A good whiteboard interview should be a toy problem that can be solved in several different ways by using different strategies or data-structures. The idea is to see how the candidate will break down the problem. Is the candidate able to formulate test cases, write a simple implementation, verify his code and correct the implementation should it fail a test? On the more meta side of things is the candidate able to take feedback and explain why a certain strategy was chosen? Of course it's not representative of real world engineering but it's a good way to peek at someone's ability to debug and reason about programs; these abilities translate well into debugging and design. Especially at the college level, I really can't make any assumptions on what the candidates know. I'm not judging their knowledge of the standard library of X programming language or the framework-du-jour but their ability to learn it fast.
Now the hard part isn't so much to create an interview process that works well, but to create a pipeline that feeds into this interview process that has a high signal to noise ratio. In my experience, the best predictors of a good signal to noise ratio was to select for CS fundamentals, good references and offer above market comp. The latter is especially crucial now since there's no more "local market" to speak of now that remote work is a lot prevalent. The "local market's" best devs are working for SV firms at SV salaries mentoring SV employees.
"If you wanna hire great people and have them stay working for you, you have to be run by ideas, not hierarchy. The best ideas have to win, otherwise good people don't stay."
- Steve Jobs
I think the real disconnect with the 'inclusive culture' boom comes because humans are involved so heavily in the process. The _idea_ is great, we want to be fully aware of our internal biases and avoid having them color our perception as much as possible so we do not shoot ourselves in the foot.
In practice, I have yet to see inclusivity programs at corporations be anything more than virtue signaling, and an opportunity to exclude others under the guise of "inclusivity" wink wink.
Remember 'affirmative action'? It's palpably Orwellian that inclusivity is newspeak; what we call it now.
We all want "inclusive hiring culture", "exceptional engineering talent" and a frictionless hiring process, but IMO you can't have all three. Like them or not, leetcode* interviews actually give a chance to people who can't do a home assignment, or come from a background that didn't let them have a bunch of code on Github. In that sense, it's the more fair way to test people's aptitude, and will find exceptional talent from all kinds of different backgrounds.
If you still want "exceptional talent", but not algorithmic interviews, then you end up biasing towards white guys who have a ton of projects to show you.
Actually, I think this should be verifiable. Select some companies that we think have exceptionally high bar (you could use compensation as a proxy, acknowledging it's imperfect). Then classify them based on whether they do "leetcode" interviews or not, and check their diversity reports. My bet would be that the "leetcode" companies do significantly better.
* Caveat is that companies people think do "leetcode" actually usually ban questions that appear on leetcode.
> Like them or not, "leetcode interviews" actually give a chance to people who can't do a home assignment
Not really—if I don't have time to do an hour-long take-home assignment, what makes you think I have time to practice leetcode-style questions? The take-home assignment is usually testing skills that I actually use in my job on a regular basis, so I don't need extra preparation, I just need a block of time to sit down and do it.
I agree that expecting people to be able to show side projects is a mistake, though I'm not sure why you think that the bias there would be racial—I would imagine it would be much more a filter that excludes people with families and/or non-computer hobbies.
I guess my assumption is that measuring cognitive ability is better than specific skills. The algo interviews, when done well, do that better than interviews focused on experience.
There are of course a lot of caveats - experienced candidates should be treated differently from entry level. It’s also true that, e.g. Google has a weird fetish about dynamic programming questions and other problems that people are unlikely to figure out without having taken a class in them.
On the other hand, take home assignments take up time and room you might not have. It’s easier to do those when you’re a man in a developed country than, e.g. a single mother in Alabama.
So its basic, I think algo interviews are a good way to hire junior level engineers.
While this seemed to start strong, I don’t buy into the “structured” portion of this blog post. The referenced research does not seem relevant to hiring engineers. In fact, the opener for the first reference says they researched “ 19 male applicants for life insurance sales”positions”. This is a “mountain” of evidence in hiring engineers?
My own interviews have a list of topics I want to cover (non functional requirements, data experience, app design, infrastructure, etc), so I guess there is some structure. But I mostly run the interview based on their own experiences and projects they have worked on. So we will focus on applications and systems they have worked with in the past. And then I see how deep down those rabbit holes of their own system they can go.
My primary programming languages are not allowed in leetcode interview sessions so a lot of the challenge is remembering how to use Ruby or Python on the spot, and also to think imperatively.
Lot of my thinking is based on visuals and emotions -- It's challenging for me to transcribe to English on demand and it interrupts my process -- it's somewhat like painting.
I always shine on take-homes since I'm allowed to be my authentic self. I'm enabled and have the full capacity to do my rituals, routines, and quirks.
Admittedly, this means I won't succeed in cooperative environments like pair programming. I'm better off left to my own devices.
A lot of the commenters in this thread (and elsewhere on HN) flat out refuse to do take-home assignments, live algorithms coding, 4+ hours of onsite rounds, etc. Yet every interview I've ever done in the last decade+ with FAANG and FAANG-adjacent companies have always been like this. So where are all you interviewing that pays competetively without this "traditional" interview loop?
I think this is the broken assumption—there are a lot of us who simply are willing to accept a sub-FAANG wage in exchange for a work environment/interview process that we feel respects us.
It's true, I could afford to retire now (I am <40), but I expect to keep working until I'm completely unemployable (or physically incapable). It's just more fun than anything I've found in my free time. I do work at FAANG btw so it can't be that bad ;).
Daniel Kahneman, the famous psychologist who authored the paper you're probably thinking of (the one popularized as "no increase in happiness above $75k/yr") actually just released a new paper a few days ago, finding that for the large majority of people, happiness continues to rise with income far past that point.
That's a quite an interesting paper! That said, the final results still line up with my intuition, which is that there are diminishing returns, as evidenced by the log(income) relationship that that new study identifies.
I already make a 90th percentile income for my area, and I don't feel that investing additional mental and emotional resources in maximizing salary is the best route forward for pursuing happiness. I think that at this point there are other axes to optimize on that provide greater marginal gains to happiness.
A big issue with any analysis of happiness is that it's rated on a "capped" scale. For example they ask you "how happy are you 1-10". However income is "uncapped". That means there will always inherently be a sublinear relationship between the two in any analysis, because you are compressing the top end of happiness.
I think for most people amounts up to something like 200-500k/yr (depending on COL) would provide increased happiness. Basically, if you ever have to worry about not having enough money, you could stand to make more.
Of course that doesn't mean it's necessarily worthwhile to work more to achieve that, that would be a personal decision you have to make yourself.
Uhh.. what? I've never had an interview process with a live coding session and whiteboarding that lasted longer than an hour, and my salary as a senior engineer has annually been anywhere from 150 to 200k. I might be leaving a little bit of money on the table but I highly doubt it's anywhere near 5x. Most FAANG level jobs aren't paying above 300-400k except at the most prestigious.
Level 7+ at FAANG can reach $1m+/yr. Obviously I can't say that you would be able to reach that (most don't), but it's a possibility at least. Part of the difference is that non-FAANG companies often don't have as much of a path to these high levels for ICs so you are forced into management if you want to keep raising your compensation.
Was the whole process <= 1 hour, the total amount of live coding was <= 1 hour, or any session of live coding was <= 1 hour?
All the interviews at Google are 45 minutes, and when I interviewed only 2 had coding, so there was realistically 70-80 minutes of coding that day. I did maybe 15 minutes in a phone screen on an earlier date. Even if you did only 60 minutes for the whole process, you really aren't that far off from a typical FAANG.
We don't want to do it but sometimes we do if it means getting to work on interesting things/paid competitively.
I've had interviews with heavy LC and ones where I got plain old fizzbuzz and there wasn't much difference in staff competence or how quickly we delivered.
If anything, the place with the low bar had more well rounded peers I wanted to spend time with after work.
I'm in non-FAANG tech (biotech, DoD) and no one does take homes or live coding, even for code heavy jobs. Our comp isn't in the 200k+ range, mostly due to location, but it's pretty good all the same (120k+).
I will do live or take-home algorithms (~1h) and such for any company since they are fun and help me practice anyway if I'm in interview mode.
I will do take-home assignments (assuming 4~8h of work) or 4h+ onsite only if I am (quite-to-very) interested in the company. This is either the company is famous so I know them well, or there was a good interview process and they passed all of my questions/no red flags. If I'm on the verge of rejecting a company and they ask me for a sudden 4h+ process, sorry but not.
I live in Japan so it's been interesting as there's vastly different thinking companies, you have from the most modern flexible silicon-valley-like company (few, but there are) to very traditional ones that might even be confused when you reject them (again few, but some). Last time I interviewed I told a company I wasn't interested in their offer, only to receive an email later telling me they were not interested in hiring me. I could guess HR marking me as a no-hire was a lot better for that interviewer than marking me as rejecting them, but still made me laugh a bit of how much "no, I am breaking up with you" it sounded like.
I can totally see this: can you whip up a quick breadbox while we stand here and comment on your every move? You've got to use our toolbox, work with unfamiliar materials and no measurements are allowed. You've got one hour. Oh, and we'll keep throwing in new requirements along the way.
I'm not even trying to touch on the "in person interview" stuff, more of the take home and spend like 8-16hrs on the project type stuff. Which is just crazy. That's more like "build me this table to get the job" type of thing.
The difference is it doesn't take 8 hours to make a drink or scramble eggs, it takes as long as an interview usually takes. So the equiv. there is not a take home project, it is a in-person coding question. Or getting them behind an IDE and having them type some stuff.
Lately it seems take home projects are more and more common and these do take 8+ hours. If you want the job.
Engineering and medicine require a lot more investment to get the job. Especially medicine. It's a lot more regimented, but to put things in perspective: to be hired by a hospital, a doctor needs to do stage (few months in rotations between different departments) and then residency (36-hour shifts with 12-hour sleep intervals for a few years). And then they become attending, maybe will try for a fellow, but, basically, they become a "proper" doctor, and get a permanent contract from a hospital. Sure, the procedure varies between countries and hospitals, but the process is a lot lengthier and exhausting than anything you can possibly face in programming. Hell, even karate kid punching trees with bare hands for a year fades in comparison.
You described a process required to became a doctor, not to get a job at a hospital. You don't need to do any of this when you want to move to a different place.
Here's what happens if you want to move to a "different place".
Say, you go to a different country: you have to spend upwards from a few month, but likely few years to confirm your degree. You won't need a stage, but you will not become an attending right away. In many cases there aren't even analogous positions if you move countries, unless medical systems are very similar, so, in most likelihood you will have to do at least a good chunk of residency training all over again. This will also be usually compounded by studying a new language to a very high degree as doctors are expected to produce a lot of written reports / engage in written communication, and, unlike programmers who almost universally use English regardless of the country they work in, doctors absolutely have to have good command of the local language(s).
Similarly, if you move between different medical organizations which manage hospitals. Sans the language requirements. However, within the same country hospitals will usually be more similar than between countries. Anyways, most hospitals will have fixed dates when job applications are processed, and even if you are extremely lucky and you don't need to redo any of your previous residency (both systems use the same PACS system, same or very similar internal organization etc.) you will still have to wait until the "draft" date. Typically, and due to competition, doctors will go to the hospital they intend to work at anyways before the "draft" date.
Even within the same hospital, if you want to move to a different department, you will still do residency, at least in part. I.e. say, you were already an attending in internal medicine, and you want to move to radiology: then maybe instead of 4 years, you'll do 3 years residency.
----
The above has a lot of compounding factors. Huge waiting times to get a position lead to doctors holding on to their positions with a lot more devotion than programmers. In many cases it's a job for life.
Because hospitals have to be in geographically diverse areas, they cannot, like programmers, all bunch together in one or two cities in a country and jump jobs w/o moving to a different apartment / house. A lot of hospitals thus include accommodation programs, which make it even harder to switch jobs.
It's very common for doctors to marry doctors. This makes some things easier, but it also means that if you need to switch jobs, then you have to do it in lockstep with your spouse.
Not in the least, if you move from a "less prestigious" country to a "more prestigious" country, you are almost automatically downgraded in your rank, and if you want the equivalent job, you'll have to jump through the same hoops the second time.
> Here's what happens if you want to move to a "different place".
All of this is still not true in a most simple case, so getting job at a different hospital in the same specialization. Ex. in Poland most doctors are hired in multiple hospitals at the same time.
> Ex. in Poland most doctors are hired in multiple hospitals at the same time.
Genuinely curious how does this work? Do they get paid per the number of hospitals who hired them? How do they go to work? How do they know what hospital to go to?
PS. My wife is a doctor, and I had to live through what I described. So, none of that is invented, it's just what I see happen to her and to her colleagues. To make this more concrete, she was an attending in emergency department and wanted to switch to radiology. In her case this resulted in the full 4 years of study on top of about half a year of just showing up in the hospital and tagging along with the radiology team. (This was in Israel, one of the central hospitals). One of her colleagues was a transfer from internal medicine (also an attending), and he was doing 3 years of study to get into radiology. Another was a Russian emigrant doctor with about 10 years of practice from a hospital in St. Petersburg. He was also doing a 3 year of residency.
They also had two people drop out of the residency just during the year my wife was there (before she gave it up), and that's out of a group of six residents. One was a Brazilian emigrant, who eventually decided to go back to Brazil and another one was a guy who was an Israeli, but received his degree in Romania, which was cheaper, I guess. He just couldn't pull it up, and eventually was let go from the program.
The Russian guy was also on the verge of leaving due to some bad blood between him and the head of the department. The head was actively trying to sabotage him and make him leave for god knows what reason. The Russian guy though, despite having some sort of a chronic illness was spending multiple days in a row w/o leaving the hospital.
I mean, back to my original point: I saw nothing that could come close in the programming world. And the fuss people here make about home exercises is just a sign of being way, way overly privileged compared to the majority of the workforce. By which I don't mean to say programmers should suffer like everyone else, rather everyone else has to get better conditions. It's just of all people, presently, programmer should probably show more comradery with other paid workers instead of complaining about their own issues.
> Genuinely curious how does this work? Do they get paid per the number of hospitals who hired them? How do they go to work? How do they know what hospital to go to?
They have duty schedules, so they know where they should be at a given time. They have contracts signed with each hospital, like any employee. They can also work in private healthcare at the same time. It's just the case of setting up a schedules so they won't collide.
> she was an attending in emergency department and wanted to switch to radiology. In her case this resulted in the full 4 years of study on top of about half a year of just showing up in the hospital and tagging along with the radiology team.
That's normal because this is a specialization change. Not many doctors change specs or have more than one in most cases, at least in Poland.
Academia also has the concept of tenure, which is basically "you're evaluated each year and easily fired" - if they don't like you you don't come back. This makes sense to me!
How long is the lecture? 1hr? That doesn't sound bad compared to a 20hr programming assignment!!
I won't do anything that requires more time from me than it does for them, simple as that. If they think it's worth it to waste 10-20 man hours for a day on sites, that's concerning but not disqualifying.
I refuse them. If I'm looking for a job I don't have half a day for some pointless thing. Interview me in the standard way, a few hour long interview, one at least with coding. When I interviewed at Microsoft and Google a few years ago, take home assignments were not part of the deal.
Last 10 years no take home. Today it's coding something in a vc meeting, maybe in a web browser or coding env. Maybe beginners do some coding.
Most people on hacker news and most people in tech do not work for faangs. These salaries of $250,000 or $200,000 or even $150,000 seem unreachable. But these faang salaries have *.
You signup for a new Amazon job. You are a senior developer you expect to make $400,000 with the stocks/salary. Your base outside of California is 139,000 or 129,000. After year 1 only 5% vests.. after year two 15%.. the average employment length is 1.5 years. So you end up with $140,000/150,000 for working 16 hour days. If you manage to stay 10 years you could retire..(you have to because at this point you hate life) but they don't want people staying at the same level so you need to get a promotion when the 4 year vest up or you will be at your base. Getting one takes the right project and is hard and requires a breakthrough project.
Most people 95% of developers never worked at a faang and those who have, on average worked for 1.5 years. Very few are still employed or seeking faang employment. Faangs make popular entry level position but very difficult to keep for life but if you can survive many years you usually leave the field or create your own startup because of burnout. Faang adjacent companies can be the worst of all worlds same issues worse pay/upside.
> A lot of the commenters in this thread (and elsewhere on HN) flat out refuse to do take-home assignments, live algorithms coding, 4+ hours of onsite rounds, etc.
Well, I personally have always refused to do take-home interviews but happily will do live coding and systems design interviews.
For me it's about respect and power imbalance. A company asking me to do work without them putting in equal effort sets a tone for a culture I personally don't ever want to be a part of.
Like I find take-home interviews disrespectful.
Time-bounded interviews with an interviewer also there (aka FAANG style onsites with 4 hours of interviews) is far and away my preferred process, especially if I can do them all at once. One problem I've seen in a remote friendly world is companies wanting to spread the interviews out over multiple days.
Personally, either regular tech companies in Europe who tend to be more relaxed with hiring ... but also don't pay the same astronomical salaries FAANG adjacent companies pay.
Where I'm from FAANG pays the average for the market salary. They might be more attractive as a bullet point on the resume or because of various other perks (both related to the job directly or not at all).
In the world outside HN I very rarely encountered people who'd turn away from any kind of hiring process. Maybe one in fifty candidates?.. I don't have the numbers, but I think I only met such people twice in my life.
I bailed from interviews for different reasons, but I think that homework is a legit way to test someone's skills, so I wouldn't mind that.
The reasons I cut the hiring process short in my job hunts were most commonly:
1. Employer is an MS Windows shop. Sometimes it's hard to figure this out from the job posting.
2. Employer requires employees to use company-provided tools s.a. code editor, or antivirus etc. In other words, an over-reaching IT.
3. Crazy / not very smart / borderline criminal employer. Examples include a guy who had "scrum cards" deck on his desk and essentially showed me to the door when I asked if they used this stuff for real. Another one who couldn't get my homework to run, asked for a Docker image, couldn't run that either, asked for a VM image, couldn't run that either...
----
There's one litmus test I have when interviewing that turned out to be surprisingly precise, and I don't know why. I ask potential employer if they ever use git-merge. If the answer is "no", the company turns out to be intelligent people who are nice to work with, and if the answer is anything else, it turns out to be dysfunctional in more ways than just infra. They will have toxic culture, under-the-carpet skirmishes where each department undermines another department, while at the same time trying to do as little work as possible.
As you can imagine, unfortunately, I had to take jobs where the employer answered "yes" or "sometimes" etc. That's how I know :(
All companies I worked in for the last 10 years used squash-merges exclusively, combined with trunk-based development and small commits. I don't see anything wrong with that, it's nice to have a history where every other commit is not a merge commit.
What's wrong with squash-merges? As long as your merge requests aren't enormous, squash merges keep your commit history clean, instead of being polluted with tons of commits with the comment "fixed typo" when most of the real work is in a single commit or two. When I'm reviewing someone's MR, I do not want to wade through dozens of very minor commits that are probably already fixing the things I would have complained about.
And what's wrong with merge? How do you even use git without merging?
In theory I don't mind them. In practice I've found that I more often encounter a scenario where I wish a change had been split to smaller commits rather than the scenario where there were too many commits to go through.
Anyway... I intended my example as an explicitly anecdotal evidence to counter the seemingly absurd suggestion of using Git without ever using merge. Feels like going back to subversion or CVS.
Merges make it prohibitively difficult to go back in time because they create alternative versions of history. There are many negative aspects to this, but I will use git-bisect to illustrate the problem.
For more reliable code-bases you want to do the following:
* Run tests on each commit when accepting PRs.
* Be able to remove or edit intermediate commits, if you find that they've created problems afterwards.
* Only have one path from past to the future.
This is so because if want to use git-bisect, and instead of deleting faulty code you reverted it, the command will keep failing on the code that you've already fixed, and there's nothing you can do about it. git-bisect also has to follow one and only path from the past to the future because if you don't, then, at best, you get an combinatorial explosion of possible paths git-bisect may take, and at worst, some of these paths will fail, but others will not.
So, what ends up happening is this: people who use merges are like people who never clean up their apartment. For some it will take longer, than for others, but, inevitably, the apartment will become a filthy mess. But this is just a symptom of people being afraid of not understanding their code, being afraid of making big changes, undoing things committed to long ago.
This fear is usually an indication that people aren't good at the technology they are using. They would be too afraid to delete code because "what if it breaks something?" -- and nobody can tell authoritatively "no it doesn't". In a situation like this any change in technology s.a. using a different version of the same tool, or replacing the tool altogether will be almost impossible to implement because of the fear.
It's also usually very characteristic of places like this to be afraid of knowing / learning the underlying technology, the one that supports the entire company's stack. Eg. if it's a Python shop, then they'd be opposed to writing Python modules in C, even though this is how Python typically works, because they are afraid that they won't understand this code and one day will end up with a "magical" program that sometimes fails, but nobody knows why.
It's also usually the people who won't even try an unpopular technology, even if the benefits were huge, based on their fear of not having expertise to deal with it. Eg. XML schemas are hugely superior to JSON schemas, and if you want to validate your inputs, XML is just a better tool for this, but the company I'm describing will never consider using it because they are afraid of not being able to find people willing to work with DTD / XSL / RNG.
Such a company will never consider self-hosting, and will pay through the nose for the expertise of others, being mortally scared by a prospect of running their own infrastructure.
----
And... this is the majority profile. The problem is, this is not a winner's profile. It's a scrapping-by profile. It puts an individual programmer in the situation where there's no need and no reward for bettering themselves. Where management is antagonistic to programmers because they are in a conflicting situation, where on one hand they want to give customers more stuff, but on the other hand they are too afraid to make more stuff, since it may deprive them of the stuff they already have. So programmers are punished whether they do or whether they don't. It's where cargo cult flourishes. Basically, Dillbert comic before its author went into politics.
Despite the HN bubble, most of the 2.7 million developers working in the US in banks, insurance companies or other “enterprise developers” will never see a total compensation amount (inflation adjusted) above around $170K in most of major cities in their life.
These developers may go their entire careers without ever reversing a binary tree on a whiteboard while juggling two bowling balls on a unicycle.
What bothers me about tech hiring is that tech companies overthink it. To use a housing analogy, they act like they're signing a 30 year mortgage when they're only signing a 1 year lease. Engineers come and go all the time. At present, tech companies are laying off engineers by the thousands. Think of how much time, effort, and money was spent hiring those thousands of engineers! It's a giant waste. Premature optimization is the root of all evil, and that applies not just go writing programs but also to hiring programmers.
It's funny how they claim that a bad hire is devastating, and they can't rid of them easily, but somehow they can do mass layoffs and get rid of a bunch of engineers easily.
I wholeheartedly disagree! It takes months, sometimes even half that year for engineers to fully ramp-up on teams and integrate into the culture of a company.
Yes, you are expected to hit-the-ground running on day one, but no one will immediately operate at their full potential. Even with all the shared best practices in the world, the secret sauce is the part you have to learn.
As an employer it's very hard to know if the reason for someone's uneven performance is due to ramp-up or if they are just not a good fit. Without a rigorous interview process, so many months would be wasted waiting to get a clear signal on that person.
That also doesn't account for complete cultural mismatches that cause instability in teams and hurt the impact of your other employees.
Another implied reason, good engineers want to surround themselves with other good engineers. So knowing its hard to get into a company signals to each applicant that the other employees there made it through that process.
> It takes months, sometimes even half that year for engineers to fully ramp-up on teams and integrate into the culture of a company.
Maybe that's because companies tend to hire whiteboard-master generalists rather than subject-matter specialists who may not be great at standardized technical interviews. ;-)
Also, if the company culture is ultra-bureaucratic, maybe the company should fix that instead of wasting months on every new hire.
Seriously, if a new engineer can't commit code within the first week, that's a company problem, not an engineer problem. Of course their code shouldn't go directly into production, but that's true of any new code. Give them something small to start, like some bugs to fix.
> That also doesn't account for complete cultural mismatches that cause instability in teams and hurt the impact of your other employees.
Technical interviews can't determine this.
> knowing its hard to get into a company signals to each applicant that the other employees there made it through that process.
I realize that's a signal, but it's not necessarily a good or accurate signal. I think it's mostly PR and hype. Reminds me a lot of fraternity hazing. Google engineers believe they're the best, and some of them may be, but some of them don't impress me at all. And as I mentioned, engineers tend to move from company to company anyway, so if Google engineers are "the best", they're constantly losing the best too.
The best engineers I've seen tend to ramp up pretty quickly (1 month or less). Sure there are probably exceptions, but the ones I've seen that take 6 months to 'ramp up' end up with very low performance. I think there's just a strong correlation with learning quickly and doing a good job. It's probably less true the more senior you get, but the relative effect is still present.
> It's funny how they claim that a bad hire is devastating, and they can't rid of them easily, but somehow they can do mass layoffs and get rid of a bunch of engineers easily.
I hope you realized that this should answer your own questions. Layoffs may be (relatively) easy, but firing someone for "you're just not cutting it" is much, much, much more difficult.
First off, most companies are loath to do large scale layoffs unless there are strong economic reasons to do so - many of the FAANGs have never had layoffs as big as the recent ones. So if your only chance to get rid of bad hires is every 5-10 years or so when there's an economic downturn, that's a problem.
But more importantly, while it's generally straightforward to fire someone who's flat out bad (as there is usually plenty of data to emphasize why they're bad), firing someone for cause who is just kinda mediocre is nearly impossible in the tech world in my experience. For example, if someone can do the job, but say is 50% slower than your average programmer (I've definitely seen this), it can be extremely difficult to gather enough evidence to fire that person. And it usually sucks for everyone involved, because often times these people who are slow are hard workers, but they're just not as capable as their peers.
One of the reasons you see the behaviors you see in technical interviews is precisely because hiring a kinda-OK-but-at-or-slightly-below-par is basically the worst kind of hire you can make.
> firing someone for "you're just not cutting it" is much, much, much more difficult.
It's actually not. When upper management is motivated to fire people, they get fired fast. Whether that's an individual person or a large group of people. We've seen this happen over and over. Self-imposed bureaucracy is the only thing that prevents fast firing.
> it can be extremely difficult to gather enough evidence to fire that person.
You don't need evidence. There's no such legal requirement. It's at-will employment.
And I don't want to hear about potential lawsuits. These are ghost stories, designed to scare, but ghosts don't exist. Show me the lawsuits. Incompetent people who are suddenly out of a job don't have the time or money to file frivolous lawsuits (which could get them blacklisted from the entire industry). The ratio of lawsuits to firings is close enough to zero to be negligible, and certainly big tech companies can afford to defend themselves.
Yeah. In my experience mediocre hires never get fired and just kind of coast along forever. But they can be pretty harmful to team dynamics.
I've worked with 200+ engineers and I know of exactly four that were fired for performance. But probably another 40 were quite bad and we would have been better off without them, they just didn't exactly meet the bar for 'so bad we have to fire them immediately'.
Layoffs aren't that easy, and they are very expensive. A lot of the big tech companies paid the equivalent of 6-12 months salary (if you include the various stock etc). There's also the cost in organizing the layoff, which can be millions of dollars in consulting fees, the costs of decreased productivity and morale, etc.
I don't dispute any of that. I just mean that they can legally do it, and they don't have to justify it, they don't have to put employees on PIP, they don't have to give reasons why every employee was included. I mean, Elon Musk can basically walk into Twitter and haphazardly fire a ton of people. The consequences may be bad, but it's "easy" in the sense that he can just do it whenever he wants. Even more so for individual firings as opposed to mass layoffs.
0. 45 minute homework/prescreen. Provide an (optional) pre-setup environment so it's mostly about coding and not about building/installing deps.
1. on-site where you chat about your solution, mostly an ice breaker/introduction to the team.
2. pair-programming to extend the homework or work on a simplified but real problem encountered day to day, open book
3. design review
4. code review
5. behavioral / case study
All of these can be pretty objective and don't rely on any memorization. All this should be pre-canned so individual proctors don't come up with their own questions and you're comparing candidates around the same prompts. It's amazing how few companies even manage these basic steps. I think most importantly the hiring should be done by a committee of actual practicing engineers - that means if you have checked in code in six months you aren't a vote on the committee.
Imagine the other side. You're more than likely doing this for multiple companies. Plus working your current job. Plus taking care of your kids! You have to devote minimum 5 hours per interested company. IMO that's ridiculous. I also think its why startups skew so young. I don't have the energy to put up with it anymore.
Sorry, but take home test as the first step to filter them is a big no for me. Why should i waste time on a test before even meeting anyone?
1, 2 and 5 should be more than enough. You get to talk to them about their past expertise and even combine it with some design discussion, you get a pair programming session and a final casual discussion. Why do you need everything else?
They call it a funnel for a reason - most candidates are not worth bringing onsite - which is expensive for both sides. So you need a screen anyway and homework should, if it's well designed, take similar amount of time to a screen. In my experience it takes most teams at least six months to get an engineer productive and a year before they're really hitting on all cylinders. If you're asking a company to spend six months to a year training you, giving up a day isn't (IMO) a huge ask. As a candidate you really shouldn't need to go onsite at more than a handful of companies or you're really wasting everyone's time.
572 comments
[ 5.7 ms ] story [ 992 ms ] threadWith recent layoffs and many talented professionals on the job market, I was compelled to write a blog post about how to build an inclusive hiring culture and find exceptional engineering talent.
If you're involved in your organization's technical hiring process at any stage, I encourage you to give this a read. I share some best practices for conducting effective interviews and improving your own hiring process.
Let me know what you think!
Thats a contradiction in terms. Building something exceptional always involves excluding mediocrity
You can exclude mediocrity while also being exclusionary on other axes.
They are unrelated issues. In fact, I’ve even heard of exceptional people being abused/bullied for belonging to the wrong group to the point of being told “you couldn’t have done that” which itself is an assertion of their supposed mediocrity for exclusionary reasons.
Don’t assume you need to be bigoted to exclude mediocrity. Discriminating, yes, but not discriminatory against groups that inclusive hiring policies attempt to protect.
An inclusive interviewing process does not mean that you hire everyone. It means you reduce the weight of people's biases as part of identifying who you hire (because people turn out to be quite bad at prediction in hiring).
This leads to organizational pressure to hire based on population distribution. Doing so inevitably means hiring based on attributes other than skill.
- The companies that make an effort get a lot closer to population baselines than the ones that just give up.
- Organizational pressures are something leadership and management should be steering. I'd rather have hiring practices be an explicit choice than something that "just happens"
If your goal is population distribution then you are inevitably hiring based on attributes other than skill.
For example, women don’t make up 50% of the engineering talent pool so if your goal is 50% women then you have to lower standards to achieve that.
You don't have to. But you'd remove them from the job market, making the pool smaller for other companies. IOW, a few companies can target 50% women, but that'll make it that much harder for other companies.
Let's say you are successful. Let's say a class of companies are successful at this strategy. Lets use the example of faangs which are desirable places in terms of salary/brand. If faangs were successful at this that would reduce the % of female developers in other industry and assuming faangs are taking the best candidates that leaves the worst ones. Which then creates this reality where male programmers outclass female developers in these other industries. That makes it harder for women in general and makes this false impression that females are not as good as males.
To help women you really need to treat them equally. Trying to reach a goal of unhealthy unnatural % industry wide means women will left holding the bag when the music stops.
But this also creates a positive feedback loop when more and more women decide to switch industries and pursue IT/Engineering jobs via bootcamps, college degrees, etc. I noticed the number of female candidates in UX/UI, fullstack, QA, Data Analytics - has increased in last several years.
Partly because the demand is still high for these professionals, partly because there is entire cottage industry of bootcamps churning out IT specialists en masse, partly these diversity hiring practices that opened up doors for women
They churn out ego inflated beginners who think they're experts.
The mentality peddled by bootcamps to sell their wares produces dangers "engineers" in my opinion.
I have no formal education, I'm not coming from educational elitism here.
I'm a bootcamp grad, and would not have gotten into the field if bootcamps did not exist. I'm about four years into my career now, currently working at a major well-reputed tech company, and haven't gotten an average-or-below annual performance review yet. (And one reason for that is that I tend to be cautious, critical, and thoughtful in my technical decisions.) There are a number of other people from my bootcamp class with similar results.
80%+ of the bootcampers were rubbish and shocked to be told their knowledge was way below where they thought it was.
A classic is a 6 week JavaScript bootcamp grad claiming to be an "expert in JavaScript" (their words) and couldn't explain the JS type system or basics of variable scope. That was the norm. That kind of rubbish.
I'm happy you're an exception and everyone gets a fair chance with me, regardless of background, but I am never shocked when I have to bin yet another bootcampers CV
If you mean "in my experience, bootcampers fail technical screens at much higher rates", then say that, instead of implying that you're stupid if you even consider hiring someone who went to a bootcamp.
If I could interview everyone I would, and I'd happily hire a bootcamper that seemed excellent.
The reality is that statistical likelihood of passing screening means those CVs often hit the bottom of the pile.
The two statements I made aren't contradictory
We're always hiring based on attributes other than skill. If we're lucky and purposeful, skill becomes a part of the hiring process.
People tend to substitute their biases when evaluating skills and knowledge. Some people overcome these biases through practice but everyone has them.
It's 2023, this is not new territory.
Think about how something like that would affect how companies are formed. Things seem much better now, but I merely wanted to highlight one of many kinds of biases that are actively affecting our society, even if they are hard to qualify.
I hate when companies lie about that, it massively lowers my respect for them.
At least this is meritocracy, the kind of thing that people (e.g. eugenicists) can make a serious argument for.
> look like me, speak like me
...is something that can't be justified except by terrible people. Even worse is "likes the same music and movies that I do" or "we coincidentally have mutual friends."
I have bad news for you about the people commenting in this thread.
This is wrong by definition
US army was exceptional in 1970's, did you need to graduate from harvard to join? No, they drafted everyone, even your sorry ass didn't want to join.
British industrialisation was exceptional, they didn't exclude anyone, even got children something to do by sending them into coal mines!
Amazon is exceptional, is it hard to become a worker in Amazon warehouse?
Organisation can be exceptional without any individual being exceptional.
Also you could be exceptionally bad!
You are moving the goalpost from mediocrity to disability.
It's illegal to enlist anyone with an IQ below 80 to the military.
They don't have any use for them.
Approximately 1 in 10 people have an IQ below 80.
As soon as people figure out the check boxes or the structured pointing system they start to check all the boxes, but it doesn't necessarily speak to the nuances between individuals that make them diverse and both valuable. In fact it can lead to a certain type of person people hired or promote, which can be on good characteristics, but I find many times turns into a "certain type" of person.
I guess what I'm saying is structure can take you so far, but you have to be willing to explore a little bit about what makes a person special, and that many times means not controlling the whole interview, and be willing to have your bias challenged through the candidate directing some of it.
I think checklist interviews miss the mark as you say. You may not have a perfect rubric, but I’m not grading students on a history exam; I am evaluating them for a role in a given position. In the limited time we have to speak, I want to use my intuition and experience as an interviewer and engineer to rapidly get to where the candidate is strong, and where they may have issues.
Does it sting when I get a "No"? Yes, a little, but I did my best and (presumably) someone else did better. So, I take solace in that I did not have to make a (relatively large) decision.
How the heck do you measure actual, repeatable performance? Or skill? Income / promotions / etc is very frequently a horrifically biased metric, for similar reasons to interviews, and we have much larger mountains of evidence showing that to be the case. It seems like there's a pretty good chance these studies are just measuring relative bias between interviewers and the interviewee's management, and concluding interviews are done poorly when they disagree with management. i.e. "structured interviews force people to think more like managers" rather than "structured interviews more accurately measure skill".
Using one bad measuring tool to conclude another tool is bad seems... problematic at best. I will grant that "interviews should measure what managers measure" is often what businesses want in bulk, but that does not seem like a particularly good thing to me.
I once spent 2 hours coding Tetris for an interview. I lost to another candidate that completed 2 more features than I did in the same time period.
So I'd add another criteria: interviewers need to be trained!
Plural: Criteria
SCNR
HR and recruiting relationships are sparse at best.
It’s working well, and has avoided at least two false negatives since implementation within the last six months.
The purpose of the system is what it does. If desired state is not emerging, we must adjust and observe accordingly.
It's a bit orthogonal to the concerns in this article, but in some ways it's much more important.
What I wonder about is given an org that is able and willing to compensate at market clearing rates, how do they get the word out well enough to get engineers interested. Because the other big BS in hiring is the whole recruiting side of things.
It’s more conversational, and you don’t have to live in hypotheticals.
We all know that skilled engineers will learn whatever skills they need to on the job, so less and less am I interested in what they can do in the interview pressure cooker.
I assume you mean empathetic. Same word is spelled “Emphethatic” later. (I tried finding a way to reach you privately, but your site “about” says you have contact methods on the left but, on mobile, there is no left…so here will have to do.)
If the comeback remark is something like, "if you really want this job," I will reply, "if you really want to hire me."
My current job, I told them that I was too busy to do such a thing (and I was), and got hired anyway.
Nobody w/ actual responsibilities in their life should be coerced into doing something for free, for someone they do not know.
Will an attorney give you free legal advice until you decide they are fit to represent you, or will you get free surgery until someone proves they won't completely butcher you? Can you drive a car for free (for 5 - 8 hours) until you decide that's the car you want to buy?
Why is it any different in the software industry? Because we just clack on our keyboards all day and do nothing?
I agree with your point overall, but I do have to say that it is very common for lawyers to give free consultations with potential clients, often offering very useful advice. This is something I've benefited from more than once, actually.
These are bad analogies because both have extremely extensive tests that are not only unpaid but the tested pays a small fortune for.
If development had the same no one would be asking you.
>I will reply, "if you really want to hire me."
It’s not about hiring you it’s about trying to prevent hiring the wrong person which is extremely expensive in time and money and takes weeks to figure out.
I've passed tech interview challenges only to fail the culture fit because I thought it would "be so easy" after the tech challenges.
And I've passed culture fits only to fail the homework assignments.
In one circumstance, not realizing that a separate recruiter was sending me to a company I had previously interviewed with, I've also seen previous work that I did in a homework assignment, given to me in a different homework assignment with a "what improvements and features would you add to this solution?," when the original homework assignment was the same task. That was a major blow, and gave me feelings they were using some of that work internally.
I get that it's extremely expensive in time and money to hire the wrong person.
On the other end, it can also be extremely expensive in time and money to not be extremely selective of whom you want to work for.
This is a perfect analogy.
I went to a Chevy dealer once, asked if I could test drive a car and they wouldn't even let me take it off the lot. I was allowed to trundle around the rows of cars at walking speed with a salesman in the passenger seat.
I went to a Honda dealer, and they let me take a test drive with a chaperone, but only around a short designated loop of streets "for insurance reasons".
I went to a Mazda dealer, and the salesman said he was busy and tossed me the keys and said have fun.
An acquaintance went to a Subaru dealer, and took the car they were considering home overnight.
You a Chevy.
I think it's more up to the individual dealer than a policy of a given brand
Some years ago, a Nissan dealer offered to let me take a high-end crossover home for the night without any prompting. Ended up buying it. Would buy again from that dealer if I were still in that area.
Gestures of goodwill go a long way in sales, that's why people love to pay for coffee or lunch when out with potential customers.
All but one new legal engagement I’ve entered into started with a free consultation. (The only one that didn’t was a straightforward real estate transaction where I knew the lawyer for years beforehand.)
The proceedings to jettison a bad hire can take 12 months alone, meanwhile they're still a bad hire being paid.
And if you cant tell whether they are good developer after 3 months, how is an intervoew going to help?
Rightly so, it should be hard to fire people and should require first showing evidence of trying to work together and help in good faith.
No, it is because some people have worked at very impressive jobs and probably just clack on keyboards all day and actually did nothing while there, so when you hire them you learn that they actually can't do the job you hired them for. You just can not rely on the CV alone.
In the hiring process there has to be some kind of skill test. If it is not a "homework" then it has to be a whiteboard/live coding/system design type of interview and there are a ton of problems with this type of skill test as well.
We usually give people the choice which route the candidate would like to do. Take a 1 hour interview or a "homework" assignment which took me about 1 hour to solve. Which is probably best because some people really prefer the homework because they get nervous in interviews.
> Will an attorney give you free legal advice until you decide they are fit to represent you, or will you get free surgery until someone proves they won't completely butcher you?
I do not know how attorneys or surgeons are hired, but my guess is that the education side of these jobs lines up much closer what is actually needed to do the job (which isn't really the case with Computer Science) and that when they claim to have done X or Y then it was actually them in court or at the operation table and not someone else from their team.
Clients select their attorney based on reputation of the attorney or the law firm they work for. For Surgeons it is probably similar. If you have the choice you want to go to the hospital with the best reputation. If the company you apply to already knew your name and reputation beforehand then the skill test is probably also not necessary, but that is not usual in my experience.
It also probably helps that both of those jobs require a Professional License to practice these jobs. Maybe if the software industry introduced a Bar Exam then these "homework" assignments would not be needed anymore.
> Can you drive a car for free (for 5 - 8 hours) until you decide that's the car you want to buy?
Usually you can. At least in my experience. Not fo 5-8 hours of course, but long enough to know.
If you know of a better way to hire people: I would be happy to listen.
W/ an interactive session, you get instant feedback (either verbally or via emotional cues) into what they are expecting.
With a homework assignment, it's hard to determine which path to optimize for.
If a homework assignment is necessary, it would be better if that were more of a "probationary employment" type scenario, perhaps at a very reduced amount of pay, to only imply that both parties actually have skin in the game.
Even better, for helping me judge if it's a decent fit? Show me some code you're using in production so that I can code review it on the call. Surely it's not all hyper sensitive.
> interactive session, you get instant feedback (either verbally or via emotional cues) into what they are expecting.
For some people, like myself, that is a nightmare scenario. I do not do well in these type of sessions. Which are also not even close to reflecting the real work we do. There are absolutely no meetings I go into that I can't prepare for [1] and there are absolutely no meetings where I have to solve a problem on the spot.
Being on the autism spectrum also means "emotional cues" are pretty lost on me.
There is also the fact that interviews have to be during normal working hours. Some people prefer to do a "homework" assignment which they can do in an evening or weekend.
This is again why we provide the choice. Not everyone is the same and prefers different interview and assessment routes. Both types are useful for different people.
> If a homework assignment is necessary, it would be better if that were more of a "probationary employment" type scenario, perhaps at a very reduced amount of pay, to only imply that both parties actually have skin in the game.
Really? You would refuse a "homework" assignment, but would agree to "probationary employment"? The later just sounds like WAY more work on both sides.
We have discussed something like this in the company I work for, but came to the conclusion that it just is way too much work. Getting the contracts in place and working out the insurance and tax implications and all that. It just is way too much work to do legally, because it would be the same amount of work to just hire them. However: we can't just hire everyone.
Maybe if you are already a freelancer beforehand then we could work something out that way, but in the jurisdiction I am in not every software engineer is ready to accept freelance contracts. It is a simpleish process to do, but not everyone does and those that do don't apply for full time positions.
[1] You can "prepare" for interviews, but more in a scatter shot approach studying all the interview questions that could possibly be asked. That is not what I mean. There is no meeting I am going into where I do not know the precise topic that the meeting is about.
However, I've also been a few hours into a homework assignment thinking that I could probably go down some rabbit-hole to try to perfect something that I may have been struggling with, and sometimes can't determine the appropriate stopping point.
The "probationary employment" would be more like, I don't know, a gift card, or something, vs. something formal.
That way, if I totally bombed out in some assessment, no big deal, here's something for taking the time to apply, and maybe I could use the card to buy a book.
Now, I get that companies aren't giving gift cards away to all of their interviewees, so this type of thing would only come after at least the first round of interviews, etc.
More often than not, a non-interested company will often not even tell why they didn't pass the assessment, and it generally feels like a waste of time.
I had someone try to implement a whole relational database when the interview task was just to read from a CSV file and provide a REST API to the contents of said file using any tech stack. Impressive for sure, but unnecessary time wasted.
> The "probationary employment" would be more like, I don't know, a gift card, or something, vs. something formal.
We discussed something like that in the company. Mainly because someone asked to be paid for the time spent on the "homework" assignment. We came to the conclusion that there is no real legal way for us to do so. We probably spent more money on discussing the possibility of paying the candidate then what 1 day of work would have cost us, but the cost wasn't even the issue.
With an in person interview there was maybe a chance. Inviting the candidate out to the exchange, giving a tour of the trading floor and then paying for transportation, lunch, dinner and hotel would be no problem.
Though interviews are online now as we are a "remote first" company anyways.
We just can't pay for work without a contract, insurance, tax and background checks in place. We can however ask them to complete a test. Which is what the "homework" assignment is.
> More often than not, a non-interested company will often not even tell why they didn't pass the assessment, and it generally feels like a waste of time.
It sucks. It just is that nothing positive can come from providing feedback and you open yourself up for a lawsuit.
Another thing is, if they let you go after the probationary period, you pretty much start the search anew. If you applied to another company while you were 'employed' by X and you managed to schedule another probationary period with Y right after the end of the trial period with X, you have to refuse them last second in case X wants to hire you. If you didn't search for anything while at X, you're off work for another week/month while you search for another trial.
If X ends up giving you an offer and you want to consider Y, what are you gonna tell them? Please wait a month for me while I work for this other company and see if it's any better than you? This simply doesn't work, for both companies and candidates.
Your proposed solution works, if X is your dream company and you will accept their offer no matter what. That's not how most job searches go though.
For me, I work on a lot of open-source as side projects, so there's always the coding "something" factor.
No there doesn’t. There isn’t one for the CEO is there?
I do not know how it works with CEO positions, but I am not hiring CEOs. I do hire software engineers and would like to work with competent people.
If you know of a better way then I would like to hear it.
We can't hire everybody. I can't hire 20 people (then fire 19 of them) for the 1 position in the 8 Person team I want to fill. That can't work.
Just hiring someone and then firing them 2 weeks later is expensive as fuck. We can't just do that until we find someone actually competent.
Your "easy solution" is only easy if you don't think about it at all.
I'm not saying you don't do ANYTHING to hire someone. Obviously there needs to be some judgement of skill. But take home projects where you're judged essentially against how much time you put into them (I've personally had projects where they asked me to build an entire web app, it's insane) because ... it's a race to the top in a way here, if you spend more time it's going to look good, because the best projects are from people that spend a lot of time on them...
It's just ridiculous. You end up doing 8-16 or more hours of work for a job you likely won't get (it's free for you to ask me to do this test, so you ask everyone to do it).
It's not expensive to hire and fire someone. Not that expensive. It's probably worth 20hrs of 100 candidate's time, that's for sure. If you don't think so, I have a bad opinion of you.
I'd like to ask you, how many candidates a year do you have turning in take home projects?
We align the Take home assignment very close with what the position actually entails. If you struggle with this assignment you would not succeed with actual tasks we do daily.
> Obviously there needs to be some judgement of skill. But take home projects where you're judged essentially against how much time you put into them
As I said elsewhere as well: People are free to choose to do a Technical Interview instead. If you fear that you would need 2 whole days you could take the 1 hour interview instead.
> if you spend more time it's going to look good, because the best projects are from people that spend a lot of time on them...
That is not my experience. There are people that just can't make it look good.
> It's not expensive to hire and fire someone. Not that expensive.
It absolutely is. Maybe not where you are, but where we are it is expensive. For one there is a big amount of paperwork involved, contracts, NDA's and background checks. We can't do this unless we know we actually really want to hire you.
I also can't expect a new employee to be productive right away. There is going to be a onboarding period of at least 3 months during which the employee probably needs a little bit more mentoring of a (more) senior engineer. If we would just hire 20 people for every position that we need to fill we would not do anything else then just onboard new people. So, YES, it is significantly more then 20h of 100 candidates time. Not to mention that the take home we give out takes maybe at a maximum 4h (if the candidate is fairly junior and has to look up stuff constantly, many people just do it in 1h).
Also: I think it would be HIGHLY unfair for the people to just hire 20 candidates if you only plan to keep 1. They probably quit their jobs or said no to other opportunities. We only hire people we see a long term future at the company.
> I'd like to ask you, how many candidates a year do you have turning in take home projects?
It is fairly late in our hiring process and it takes a significant amount of time from at least 2 senior engineers each time to review. I don't have the exact numbers how many times per Year, but we try to keep it very low and only if there is an actual interest on our side.
Ah, I missed this, and I'm sorry. My experience has not been this over 100+ technical interviews over the last 10yrs.
Almost all of them have asked for crazy take-home projects. I'm pretty bitter about the fucked process.
> In the hiring process there has to be some kind of skill test
Thats a contradiction:
1 - 90% of companies do skill tests
2 - you cannot trust the CV of a candidate even if the company they worked on previously also did a skill test
3 - you think your skill test will enable you to hire the right candidate
So everyone is doing tests and everyone is hiring shit candidates anyway?
That's literally proof that these tests are worthless.
Maybe if the recruiters actually knew what they need from a candidate, companies were clearer about what the job involves, they would stop hiring the wrong people for the wrong job.
Most lying is apparently to change start and end dates of previous employment to cover up short stints where they may have been fired for poor performance, but even if work samples/skill tests/structured interviews were 95% effective, you would still regularly hire duds.
We had one person who literally hired a team in India to do it for him. Presumably planning to give his work to them later if he was hired.
You sound so jaded, I don't blame you, but I hope you're not ever my hiring manager.
Needless to say we didn't proceed with signing the contract.
2 - We can not trust the CV. Maybe the company did a test, maybe they didn’t. Maybe their test are to our standard, maybe they aren’t. Maybe the candidate barely passed. Maybe the candidate actually failed but made up in other ways that might be relevant to the other company, but not ours. I doubt anyone would give us this information about their current or former employees. Maybe the candidate really did work on the projects he claimed or maybe he was only tangentially involved. That is a lot of unknowns. A test clears up what the person actually can do.
3- Yes. It works for us. Do we miss out on some good candidates because of this? Yes, but it definitely prevented us from hiring bad candidates.
Again: if there is a better way then I would like to know. I have not found one yet.
There is a large company based in NYC that interviews thousands of developers a year, but only hires a few hundred. Each of those devs do a take home project that takes about 16 hours to complete. The project requires nothing to ask, so this company asks everyone. 30,000 hours is a lot of time - it's about 3.5 years of someone's life - and at least this time is spent each year by this company (and they pay nothing for it), not counting the rest of the interview process, because it is cheap for them to do this.
And we wonder by productivity is down :)
Respect people's time. If it is hard, figure it out. That is your job if you want to hire people and feel good about it.
If you do that in software, you're probably just back on the job market...
I have my own list of questions. If they answer all of them I have a pretty good idea if I'm the guy for the job. The most wonderful part is figuring out if it is an employer is looking for initiative or obedience. If you are running a sheep farm it can be very exciting to see initiative.
IME you also often get paid (albeit a relatively small amount) for doing them. And FWIW once they want to hire you many employers will actually spend hours trying to sell you on joining. Usually not in the form of an essay, but doing several hours of sell calls or lunch meetings with different people at the company is not that strange (especially if you are higher level).
That being said the come back makes perfect sense to me. When I was still on the IC track and interviewing, when I was submitted to some "dumb" questions I would ask the same kind to the interviewer. So you just asked me to write this stupid algorithm? How about you whiteboard something for me now?
It was almost always met with dead silence to which I would say "well you're trying to figure out if I'm good, I want to do the same and make sure your team has good engineers".
Of course it never went anywhere since, while they had 55 minutes to grill me I was only given the last 5 for my questions.
While there are people who are naturally great at interviewing in person or who don't mind grinding leetcode for free for months on end, there is a whole "base of the iceberg" population who a) can't spend months grinding leetcode on end but they can definitely spare a couple of afternoons for one job they are interviewing for – these are normal people with normal jobs and a normal family who don't interview every month just for kicks, they do this a couple of times every 2-3 years, and, b) just cannot code while under stress. You may scoff at b) if you've never experienced it, and while I found it really hard to understand for a while as I also have no issues with getting into deep focus while people are looking and/or trying to talk to me and there's a time limit, it is absolutely real.
Unfortunately, there is an extremely vocal minority (you) who go absolutely ballistic when asked to do a take-home exercise, who absolutely ruin it for everyone else and make hiring managers shit their pants whenever take-home exercises are suggested. I honestly don't understand why there's such an outrage, take-home exercises are the minority already, because there's a strange huge backlash.
Sure, if you're a great communicator (usually native English speaker) and grind leetcode for fun so you can shove 5 interviews in one week, you hate take-home exercises. And that's fine, really, just please apply to the leetcode grinding contests and stop poisoning the well for the rest of us who would like to cater to the large amount of extremely competent engineers that don't fit that persona.
> Nobody w/ actual responsibilities in their life should be coerced into doing something for free, for someone they do not know.
I take it you haven't interviewed for a SE in a while? The status quo requires you to perform months of unpaid labor, not days, just in the form of memorizing the solutions to every single Leetcode Hard problem. How is that better?
I guess because it kinds "scales". Once you are familiar with Leetcode you can use this skill to take interviews with different companies. Take-home exercises dont' "scale".
That being said, I believe people are against take-home exercises exactly for the reason you support it: it's usable code. They worry the companies will exploit candidates by using their code for free. Leetcode is "useless" so it's safe.
If we're talking about "make me a website" kind of exercises that one could throw into production after some tweaks, then that is free labor and I would absolutely refuse those.
Also, the fact that "leetcode scales" is part of the problem I was talking about. I really dislike the fact that if you train for months to develop a complete set of skills parallel and irrelevant to your actual role you can now efficiently interview for a number of companies that pay top dollar. So not only people with actual skills and experience are on equal (or worse!) footing than someone straight out of college, it also incentivises those who mastered leetcode to interview everywhere, since "it's all the same", while normal people who are good at their job get their torn to shreds after one or two interviews. So not only it's judging the wrong thing at the interview, but it's also causing a starvation situation for the candidates that are not playing that game.
Why are we creating incentives to people becoming "professional interviewers"? This would be like Google encouraging SEO spam instead of fighting against it!
If inverting a binary tree means swapping the left and right subtrees of every node, I wouldn't want to work with someone who can't do that either and Google is definitely right to reject him.
The question above, as clarified, is not complicated, nor does it rely on memorization or some "trick": anyone purporting to be a SWE should be able to write an essentially de novo solution to it.
(And in my own technical interviews, there are multiple questions, to specifically hedge against any one being "that one question a good candidate is going to miss because it's just not their day". It doesn't happen: it's either all or nothing.)
It is more likely that the interview process is broken and missing the right candidates, than it is that the interviewees are all mediocre. Most interviews are very non-inclusive the same way that the main track of school is becoming less and less inclusive. Different people need different methods to bring out the best in them.
Here's a thought experiment for you: if the interview process is so broken, why hasn't some tech company succeeded and become famous for an improved interview process, e.g. "Moneyball style"? My guess is because the process is not actually that broken, at least from the employer's perspective. I'm sure the interview process could be changed to be less regimented and more "inclusive", but that's also likely to reduce it's predictive power (i.e. you're more likely to make bad hires, and from a company's perspective that's almost always worse than missing out on a great hire).
Hiring is guessing. Firing is knowing. If the hiring process worked, we wouldn't have layoffs like we do.
Sure, an argument exists around "you shouldn't've hired that many people", but that is different from an argument of "the hiring process can't discern good hires". The former is a management & long-term planning issue, the latter is how interviews are conducted.
Sure, your day-to-day work may not involve manipulating binary trees. But presumably it does involve working with variables, objects, references, manipulating data of some kind... And if you're comfortable with the fundamentals of those, then this is something you should be able to figure out even if you've never heard of a "binary tree" before, once somebody has sketched it or shown you the definition of their TreeNode class, right?
It honestly baffles me how people consider this something which needs to be drilled or memorized.
There are absolutely algorithmic questions which would fall into that category. But if somebody considers this to be one of them - or something like "find the smallest number in an array" - then I have to question whether they have an understanding of the most fundamental concepts in programming...
Or if they get through each day solely using things they've memorized by rote, or looked up, and they don't really have any idea how any of the foundations they're building on actually operate.
How complex is homebrew ? Can no one else replicate it ? Why should a company hire for something you did that's simple ?
What are the skills he posses that no one else has ?
Learn your basica dsa stuff for gods sake people.
I went out of my way to avoid homebrew (still do) when I worked at google because it would reliably fail to complete some key operations in a dag, hence the interest in ensuring developers know how to do CS things.
Here's an example:
https://leetcode.com/problems/invert-binary-tree/
It's an 'easy' question. The solution is <10 lines.
Rejecting the guy because he cannot do a whiteboard brain teaser is like rejecting LeBron James because he did not make a shot at the arcade basketball game.
I'm not saying the guy would be perfect. Comparing him to LeBron James might not be a great example. Google might have other reasons to reject him.
What I'm trying to say is the current coding interview is a really poor mechanism to gauge a software engineer, especially when it comes to hiring one with real-world engineering experience.
People like to say this, but in my experience this is not true. It's just that people misunderstand the goal of technical interviews and they often are poor at evaluating their own skills.
First off, these giant tech companies have enormous economic incentives to improve their interview processes as much as possible. They also do a pretty rigorous assessment of the effectiveness of their interview process (Google, for example, has publicized some of their data). I'm not saying these tech companies interview processes are perfect, but I also have a problem believing they're so fundamentally flawed that these companies can't figure out how to fix them given the giant economic returns they get for optimizing their hiring processes.
Moreover, as some other comments mentioned, many companies (and individuals, myself included) believe it is much worse to hire someone who ends up not cutting it, than missing out on a potentially good hire. I can list out all the reasons why, but Joel Spoelsky has a pretty famous essay from a couple decades ago on the topic that explains it well [1].
Thus, it's not surprising hearing a lot people complain that they can do the job, but they aren't good at interviews. Because, from Google's/Microsoft's/etc. perspective, they're fine with a bit higher false negative rate if they can greatly reduce their false positive rate. And my experience matches that: I have never seen a candidate who did awesome in "whiteboard-style programming questions" who couldn't cut it programming-wise (they may have had other issues, but "coding productivity" wasn't one of them). Now, I certainly believe and have seen that there are some people who aren't good at these questions who can do a job well, but there are also a ton more people who can't do the job if they can't pass a technical screen, so hiring any of these folks means much more risk.
I also think that whiteboard-style coding questions help show a quality that is very important to businesses, even if those questions don't represent "real world" work. There are basically 2 types of people that do well at these questions: people who are just naturally smart and have a ton of experience to the point that they wouldn't even need to study to do well, and people who are of more "normal" intelligence/ability, but who can do well if they study a ton. Either of those two groups would likely do well in a programming role. So often I hear the complaint "I'm a busy person, I've got outside responsibilities, you can't expect me to spend all this time studying". And that may be true, but you'll be competing against people who are willing to study, so I don't think you can fault Google et al for favoring people who show a willingness to do more preparation.
1. https://www.joelonsoftware.com/2006/10/25/the-guerrilla-guid... "And in the middle, you have a large number of “maybes” who seem like they might just be able to contribute something. The trick is telling the difference between the superstars and the maybes, because the secret is that you don’t want to hire any of the maybes. Ever."
But we have examples where the companies themselves have admitted that their past interview practices turned out not to work: https://business.time.com/2012/10/23/no-brainer-brainteaser-...
> They also do a pretty rigorous assessment of the effectiveness of their interview process
Companies do review their hiring processes, but actual experiments and data seem fairly rare. It's harder than you think. What experiment would you run? Hire a group of people entirely randomly, and compare their performance reviews after 2 years?
That's pretty much exactly my point. In the 90s, wide-scale hiring for software engineers was a relatively new thing - many companies were just figuring it out. And so they did some shit pretty early on that didn't make sense. But for all the times I hear folks pulling out the "Why are manhole covers round?" and "How many cars are there in Manhattan?" examples, I haven't heard these types of brainteaser questions being used for nearly 2 decades.
I'm not arguing that the FAANGs have some perfect, unassailable interview process that can never be improved, but I am arguing that so often I hear grumbling discontent from people who don't like the interview process, but rarely do I see much examination around why those particular hiring processes appear to work fairly well for the likes of Google, Apple, etc.
Yes, they did move away from it, but that doesn't mean we aren't now in the grip of equally bad fads.
> but rarely do I see much examination around why those particular hiring processes appear to work fairly well for the likes of Google, Apple, etc.
You assume they work well, but you don't have any data to support that. That's sort of assumption is basically where these hiring fads come from.
Now does being able to reverse a binary tree mean that you would fail at Google? I have no idea. But we don't really know if that was the reason he was rejected, it's just his own guess. There could have been other reasons.
Tree structure ? json, xml, protobuf, classes, functional programming, databases with foreign key, database internals, etc ?
Oh I forgot you also serialize and deserialize data - did you forget how that works ? Tree traversal again.
Do you know how organizationl hierarchy is structure ? It's a tree.
Do you know various maps and their usages ? We use them daily - it's very very important to know their internals. Hashing vs Trees vs Linked hash vs etc.
Google maps ? n-d trees ? Comparing data - merkel trees ? etc.
Every dev out there has common work with mine. But you won't be able to solve the problems that I face on a daily basis without thinking hard & without this dsa + concurrency knowledge.
Now, is it reasonable to ask these questions ? Heck yes.
I would rather hire an engineer with a strong business or user sense - reading between the lines of requests and anticipating future issues or uses adds so much more value in a real sense.
To me, these are great entry level questions because it is a good baseline for new grads when you have little work experience to judge. Past that, it is like making a lawyer take a mini bar exam for every new job - a waste of effort if you want to hire for specific skills and experience.
(I'm not sure Homebrew is all that well engineered, actually. Hard to tell, but I've had trouble with it and avoid it.)
I think what it comes down to is that nobody really knows what to interview for.
Generally it wouldn't really make sense to reverse a tree in practice (why not just build it the other way initially?) but it has a similar structure to other tree traversal things that could actually come up so it's reasonable to ask.
Companies that pay huge TC want to hire smart people not just an average joe. Sure, you can live satisfying career and that's your pov.
What about a company's pov ? Did you ever think about it ?
In my old team, I had to come up with a coupon distribution logic based on count, percentage, time, then generating reproducable random values that required to deep dive (algorithm) into library code & explicitly storing state in redis, then an application of dynamic programming in building as custom platform, atomic token validation, custom rate limiter algo, state machine, scheduler, distributed circuit breaker, etc.
In my current team, I had to read raft paper, zab paper, look into their implementation, make a poc with raft protocol, then autoscaling algorithms, scheduler algo's, different data structures, heck even the oss engine itself is DAG, heavy threads + concurrency stuff. Even now I come across new data structures and algorithms.
Clearly you don't know the entire industry, just because you haven't worked in such teams, doesn't mean these aren't important.
You are experienced in a bubble. The hiring bar for our team is higher than other teams and heck even for SDE3 - the requirement is higher. You would be very much surprised to know that even the senior members have research publications and deal with complex stuff.
Core teams like in AWS or GCP or Azure solve these sort of problems.
Who do you think will solve autoscaling (that's what I'm doing now) or managed scaling or network or security or any infra problems in these cloud platforms ?
As experience increases, we expect more knowledge & insights - doesn't mean to ignore basic coding stuff like arrays or linked lists or trees or graphs or simple message queues or etc.
If companies are paying competitive TC and there are multiple candidates, why not hire a smart person ? What's so special about doing regular normal stuff ? That's just a normal dev right ?
They have different values. Different expectations of what is normal or important. "Culture", "team fit" and other bs.
Everything changes. New people come who don't know the past.
Been treated in past like I'm avoiding the "important part" of solving their quiz and that it's some softball topic.
You want to see repeatable behavior and a general interest in going through the process. If someone takes the time to apply with homework and is able to articulate well, it gives you so much valuable signal.
Pressure cooker style interviews only reveal someone can remain focused under stress and that they studied their leetcodes.
Also, stop giving take home projects. Bad candidates will cheat them and good candidates will not even do them. If one of the random startup names listed on the author's site sent me a 12 hour take home project I would delete the email. Do you think they pay twice as much as the bigger company that only makes you waste 6 hours doing a whiteboard? I doubt it.
And I think that smaller companies copy this as a part of the tendency to copy large companies without thinking about whether the thing they are copying actually makes sense at their scale. In this case it can be very damaging, because false negative for a startup with a limited pipeline can be very bad.
You're right about the copycat behavior. This goes all the way to top of funnel: these small, even trivial-scale web application startups just don't have hard engineering problems. Many imagine they do, or imagine they will once they take off, but the work they're offering these high powered candidates they claim to want to hire is like, wiring up CRUD apps and making javascript buttons. It's not technically deep work, it's product work. A little humility about whether or not your tech startup is truly doing "tech" problems would, I think, fix some of the expectation/reality mismatch people are having when they complain about how hard it is to hire engineers.
And sure, lots of people join Google to work on world-scale problems and end up wiring CRUD stuff anyway. But they can at least plausibly offer some technical depth (or could anyway, perhaps Google's reputation as a great place to develop an engineering career has been fading).
(this is not to knock on "CRUD" but to highlight that a technical problem solver is an overlapping but not identical skillset to someone who can work with a fast moving team to quickly and reliably develop product changes)
There wasn't growth in that team due to mediocre hiring and eventually all the good ones - left to other companies.
My current team is an infra platform and has lot of growth as IC. Everyone is learning something in-depth and are explorers - rather than blind sheep. The bar here is higher than the one for my previous team.
Our team requires you to know about whatever you talk on, not just usage but it's internals - why ? That's what we do daily. It can be about scheduler, checkpointing, auto scaling, concurrency, different data structures & algos, integrating with ecosystem, etc.
Even soft skills - like helping others, taking feedback, communicating clearly, etc.
Yeah so, mediocre will always be a burden to team.
I can't think of any reason why anyone would ever do this. Just navigate the tree in the reverse of your normal direction instead.
Why not ask the much more interesting and potentially useful question of balancing a binary tree? Or do something else recursive, if that's what you're after.
Or even just when would you use a binary tree? Figuring out which data structure is appropriate for the problem at hand is the hard part, how to implement operations on the data structure is easy in comparison, you can just Google it.
That seems to be the wrong question though. It seems to be jeopardy style "question". You are not asking which data structure is appropriate for a problem.
Here is the answer, but what is the problem it solves.
Never in my live have I sat down and said: I don't know what problem is I need to solve, but I know the solution is a binary tree.
What's easier?
You usually aren't coding FizzBuzz in production code either
I would say you're constantly coding FizzBuzz. Looping, modulus arithmetic, and conditionals are all over the code I've written. At least with FizzBuzz you have a test of a person's ability to understand a task, break it down, and make sure the logic is consistent. With tree "inversion" it's not even a sensical request, it's utterly useless, and there are countless more interesting and practical ways to test understanding of recursion and trees. Knowledge that, I would bet, isn't even relevant for 90% of programmers, and if tree traversal were relevant then you'd probably want to jump to way more difficult questions (I'm thinking of the Facebook graph, for example).
I agree with the other commenter that it would make far more sense to ask questions like "you need to process data of this type, what provided data structure (e.g. C++'s STL) would you choose?"
Uhhhh... It eliminates competent, skilled people who don't have the time to memorize the latest cargo cult trends in hiring. Look in the article for a glaring example.
"Invert a binary tree" is actually considered an easy problem to test your basic knowledge of how trees work and tree traversal.
So would simply printing out a tree, and at least that's something a person might actually do.
Tree "inversion" doesn't even make any sense and at this point I'm convinced that the cargo cult is choosing it because it's the extent of their own understanding of trees and somehow sounds extra technical to them.
I agree completely it eliminates a huge swathe of people, mostly experienced and older people. FAANG employees, ime, are biased towards childless / single people with privileged backgrounds.
Tree inversion sounds weird when you hear it phrased like that, but in an interview it would be explained with an example (just swap the left and right children recursively).
It also seems to neatly split people into camps who think "this is trivial, and totally reasonable to expect somebody to answer, even if it's a little contrived", vs those who think "this is not practical, and you'd only know the answer if you'd already practiced it, so it's not fair to ask".
> It eliminates competent, skilled people who don't have the time to memorize the latest cargo cult trends in hiring
But given that you already acknowledged that it's pretty trivial, why would memorization be necessary for a competant person?
Every (recently with raft protocol) multi master distributed system out there interacts with Zookeeper for assigning leader and maintain configuration.
Do you know how the syntax or api calls look like ? They are node path in a tree. You want to store something ? That's a tree path again. You want to listen to some change ? It's tree path again.
As I said, most people are ignorant here and don't do "true" computer science engineering in daily life.
Most devs simply convert business logic into bunch of apis + adhoc implementation.
Did you ever work at a banking firm ? I've read their codebase - they are structured as trees, every damn thing is tree. It's a headache to navigate, code, heck even the objects are literal trees.
As I said, people are ignorant and think world revolves around them.
Honestly? Because that's harder.
The swapping question is basically a softball / FizzBuzz-style question to test the most basic familiarity with data structures, pointers/references, and recursion.
Game changes if you actively contacted someone.. if you're no BS, assumption is you know who you contacted and why, hence only thing to do, once contact established is not to oversell and pay well.
Pay-well can constitue compensation as well as time.
You can't do that with a take-home (and I'm against take home as the signal to noise ratio is too low) because people will cheat and have them done by someone else.
I've heard horror story of a "senior" engineer from "his country's top school" being interviewed for a technical position by several non-technical managers and HR reps. They only included an engineer in the final round, which was basically supposed to be rubberstamped anyways. He was then asked to implement something trivial like fizzbuzz or wordcount on the whiteboard. The candidate then became extremely defensive and tried to argue that such task was "beneath him", arguing for a good 15 minutes why he shouldn't have to do it.
Then the dev just left the room and said that he used this question as a warmup with new hires and it typically takes them less than 10 minutes.
Now, a lot of folks do whiteboard interviews wrong. They often expect to get the exact implementation of an algorithm they found in a textbook and for code on the board to compile. This isn't the point of whiteboarding. Doing this only promotes rote memorization. A good whiteboard interview should be a toy problem that can be solved in several different ways by using different strategies or data-structures. The idea is to see how the candidate will break down the problem. Is the candidate able to formulate test cases, write a simple implementation, verify his code and correct the implementation should it fail a test? On the more meta side of things is the candidate able to take feedback and explain why a certain strategy was chosen? Of course it's not representative of real world engineering but it's a good way to peek at someone's ability to debug and reason about programs; these abilities translate well into debugging and design. Especially at the college level, I really can't make any assumptions on what the candidates know. I'm not judging their knowledge of the standard library of X programming language or the framework-du-jour but their ability to learn it fast.
Now the hard part isn't so much to create an interview process that works well, but to create a pipeline that feeds into this interview process that has a high signal to noise ratio. In my experience, the best predictors of a good signal to noise ratio was to select for CS fundamentals, good references and offer above market comp. The latter is especially crucial now since there's no more "local market" to speak of now that remote work is a lot prevalent. The "local market's" best devs are working for SV firms at SV salaries mentoring SV employees.
[0] https://blog.codinghorror.com/why-cant-programmers-program/
[1] https://economictimes.indiatimes.com/tech/ites/95-engineers-...
- Steve Jobs
I think the real disconnect with the 'inclusive culture' boom comes because humans are involved so heavily in the process. The _idea_ is great, we want to be fully aware of our internal biases and avoid having them color our perception as much as possible so we do not shoot ourselves in the foot.
In practice, I have yet to see inclusivity programs at corporations be anything more than virtue signaling, and an opportunity to exclude others under the guise of "inclusivity" wink wink.
Remember 'affirmative action'? It's palpably Orwellian that inclusivity is newspeak; what we call it now.
If you still want "exceptional talent", but not algorithmic interviews, then you end up biasing towards white guys who have a ton of projects to show you.
Actually, I think this should be verifiable. Select some companies that we think have exceptionally high bar (you could use compensation as a proxy, acknowledging it's imperfect). Then classify them based on whether they do "leetcode" interviews or not, and check their diversity reports. My bet would be that the "leetcode" companies do significantly better.
* Caveat is that companies people think do "leetcode" actually usually ban questions that appear on leetcode.
Not really—if I don't have time to do an hour-long take-home assignment, what makes you think I have time to practice leetcode-style questions? The take-home assignment is usually testing skills that I actually use in my job on a regular basis, so I don't need extra preparation, I just need a block of time to sit down and do it.
I agree that expecting people to be able to show side projects is a mistake, though I'm not sure why you think that the bias there would be racial—I would imagine it would be much more a filter that excludes people with families and/or non-computer hobbies.
This seems off to me as well. Wealth, and schooling, are not relvant here. Just grab an old computer, install Linux for free, and off you go.
Loads of free tool stacks, github is free, etc.
There are of course a lot of caveats - experienced candidates should be treated differently from entry level. It’s also true that, e.g. Google has a weird fetish about dynamic programming questions and other problems that people are unlikely to figure out without having taken a class in them.
On the other hand, take home assignments take up time and room you might not have. It’s easier to do those when you’re a man in a developed country than, e.g. a single mother in Alabama.
So its basic, I think algo interviews are a good way to hire junior level engineers.
My own interviews have a list of topics I want to cover (non functional requirements, data experience, app design, infrastructure, etc), so I guess there is some structure. But I mostly run the interview based on their own experiences and projects they have worked on. So we will focus on applications and systems they have worked with in the past. And then I see how deep down those rabbit holes of their own system they can go.
Lot of my thinking is based on visuals and emotions -- It's challenging for me to transcribe to English on demand and it interrupts my process -- it's somewhat like painting.
I always shine on take-homes since I'm allowed to be my authentic self. I'm enabled and have the full capacity to do my rituals, routines, and quirks.
Admittedly, this means I won't succeed in cooperative environments like pair programming. I'm better off left to my own devices.
I think this is the broken assumption—there are a lot of us who simply are willing to accept a sub-FAANG wage in exchange for a work environment/interview process that we feel respects us.
Could I use more money? Probably. But would I be happier with more? The research suggests I wouldn't.
https://www.pnas.org/doi/10.1073/pnas.2208661120
I already make a 90th percentile income for my area, and I don't feel that investing additional mental and emotional resources in maximizing salary is the best route forward for pursuing happiness. I think that at this point there are other axes to optimize on that provide greater marginal gains to happiness.
I think for most people amounts up to something like 200-500k/yr (depending on COL) would provide increased happiness. Basically, if you ever have to worry about not having enough money, you could stand to make more.
Of course that doesn't mean it's necessarily worthwhile to work more to achieve that, that would be a personal decision you have to make yourself.
All the interviews at Google are 45 minutes, and when I interviewed only 2 had coding, so there was realistically 70-80 minutes of coding that day. I did maybe 15 minutes in a phone screen on an earlier date. Even if you did only 60 minutes for the whole process, you really aren't that far off from a typical FAANG.
https://twitter.com/patio11/status/1627276892226322432
I've had interviews with heavy LC and ones where I got plain old fizzbuzz and there wasn't much difference in staff competence or how quickly we delivered.
If anything, the place with the low bar had more well rounded peers I wanted to spend time with after work.
I will do take-home assignments (assuming 4~8h of work) or 4h+ onsite only if I am (quite-to-very) interested in the company. This is either the company is famous so I know them well, or there was a good interview process and they passed all of my questions/no red flags. If I'm on the verge of rejecting a company and they ask me for a sudden 4h+ process, sorry but not.
I live in Japan so it's been interesting as there's vastly different thinking companies, you have from the most modern flexible silicon-valley-like company (few, but there are) to very traditional ones that might even be confused when you reject them (again few, but some). Last time I interviewed I told a company I wasn't interested in their offer, only to receive an email later telling me they were not interested in hiring me. I could guess HR marking me as a no-hire was a lot better for that interviewer than marking me as rejecting them, but still made me laugh a bit of how much "no, I am breaking up with you" it sounded like.
Of course many carpenters work as contractors so that's why it seems a bit silly.
My buddy just got a job at a high-end cocktail bar as a bartender. Part of the interview process was asking him to mix a drink.
Gordon Ramsay has talked about how he'll interview chefs by asking them to make scrambled eggs.
Actors, even famous ones, generally have to 'read' for roles in order to land them.
Musicians interview for seats in symphonies by playing music.
MBAs have to do case studies to land jobs at high end consulting firms.
Hell I applied to Taco Bell as a kid and they made me take a short math test to prove I knew how to make change.
I could give similar examples for dozens of other jobs.
The cases where you don't need to demonstrate some skill in order to get the job generally fall into a few categories:
- There's some outside certifying body like the Bar, CPA, PE, various tradesmen unions, or all the licenses like a CDL.
- The jobs are undifferentiated so the workers are fungible (no special skills required).
- Job skill is immediately apparent (less than two weeks to know for certain if someone can do the job or not).
- The cost of a bad hire is low so you're willing to eat the cost and just cut the workers how don't work out.
Lately it seems take home projects are more and more common and these do take 8+ hours. If you want the job.
Here's what happens if you want to move to a "different place".
Say, you go to a different country: you have to spend upwards from a few month, but likely few years to confirm your degree. You won't need a stage, but you will not become an attending right away. In many cases there aren't even analogous positions if you move countries, unless medical systems are very similar, so, in most likelihood you will have to do at least a good chunk of residency training all over again. This will also be usually compounded by studying a new language to a very high degree as doctors are expected to produce a lot of written reports / engage in written communication, and, unlike programmers who almost universally use English regardless of the country they work in, doctors absolutely have to have good command of the local language(s).
Similarly, if you move between different medical organizations which manage hospitals. Sans the language requirements. However, within the same country hospitals will usually be more similar than between countries. Anyways, most hospitals will have fixed dates when job applications are processed, and even if you are extremely lucky and you don't need to redo any of your previous residency (both systems use the same PACS system, same or very similar internal organization etc.) you will still have to wait until the "draft" date. Typically, and due to competition, doctors will go to the hospital they intend to work at anyways before the "draft" date.
Even within the same hospital, if you want to move to a different department, you will still do residency, at least in part. I.e. say, you were already an attending in internal medicine, and you want to move to radiology: then maybe instead of 4 years, you'll do 3 years residency.
----
The above has a lot of compounding factors. Huge waiting times to get a position lead to doctors holding on to their positions with a lot more devotion than programmers. In many cases it's a job for life.
Because hospitals have to be in geographically diverse areas, they cannot, like programmers, all bunch together in one or two cities in a country and jump jobs w/o moving to a different apartment / house. A lot of hospitals thus include accommodation programs, which make it even harder to switch jobs.
It's very common for doctors to marry doctors. This makes some things easier, but it also means that if you need to switch jobs, then you have to do it in lockstep with your spouse.
Not in the least, if you move from a "less prestigious" country to a "more prestigious" country, you are almost automatically downgraded in your rank, and if you want the equivalent job, you'll have to jump through the same hoops the second time.
All of this is still not true in a most simple case, so getting job at a different hospital in the same specialization. Ex. in Poland most doctors are hired in multiple hospitals at the same time.
Genuinely curious how does this work? Do they get paid per the number of hospitals who hired them? How do they go to work? How do they know what hospital to go to?
PS. My wife is a doctor, and I had to live through what I described. So, none of that is invented, it's just what I see happen to her and to her colleagues. To make this more concrete, she was an attending in emergency department and wanted to switch to radiology. In her case this resulted in the full 4 years of study on top of about half a year of just showing up in the hospital and tagging along with the radiology team. (This was in Israel, one of the central hospitals). One of her colleagues was a transfer from internal medicine (also an attending), and he was doing 3 years of study to get into radiology. Another was a Russian emigrant doctor with about 10 years of practice from a hospital in St. Petersburg. He was also doing a 3 year of residency.
They also had two people drop out of the residency just during the year my wife was there (before she gave it up), and that's out of a group of six residents. One was a Brazilian emigrant, who eventually decided to go back to Brazil and another one was a guy who was an Israeli, but received his degree in Romania, which was cheaper, I guess. He just couldn't pull it up, and eventually was let go from the program.
The Russian guy was also on the verge of leaving due to some bad blood between him and the head of the department. The head was actively trying to sabotage him and make him leave for god knows what reason. The Russian guy though, despite having some sort of a chronic illness was spending multiple days in a row w/o leaving the hospital.
I mean, back to my original point: I saw nothing that could come close in the programming world. And the fuss people here make about home exercises is just a sign of being way, way overly privileged compared to the majority of the workforce. By which I don't mean to say programmers should suffer like everyone else, rather everyone else has to get better conditions. It's just of all people, presently, programmer should probably show more comradery with other paid workers instead of complaining about their own issues.
They have duty schedules, so they know where they should be at a given time. They have contracts signed with each hospital, like any employee. They can also work in private healthcare at the same time. It's just the case of setting up a schedules so they won't collide.
> she was an attending in emergency department and wanted to switch to radiology. In her case this resulted in the full 4 years of study on top of about half a year of just showing up in the hospital and tagging along with the radiology team.
That's normal because this is a specialization change. Not many doctors change specs or have more than one in most cases, at least in Poland.
How long is the lecture? 1hr? That doesn't sound bad compared to a 20hr programming assignment!!
Last 10 years no take home. Today it's coding something in a vc meeting, maybe in a web browser or coding env. Maybe beginners do some coding.
You signup for a new Amazon job. You are a senior developer you expect to make $400,000 with the stocks/salary. Your base outside of California is 139,000 or 129,000. After year 1 only 5% vests.. after year two 15%.. the average employment length is 1.5 years. So you end up with $140,000/150,000 for working 16 hour days. If you manage to stay 10 years you could retire..(you have to because at this point you hate life) but they don't want people staying at the same level so you need to get a promotion when the 4 year vest up or you will be at your base. Getting one takes the right project and is hard and requires a breakthrough project.
Most people 95% of developers never worked at a faang and those who have, on average worked for 1.5 years. Very few are still employed or seeking faang employment. Faangs make popular entry level position but very difficult to keep for life but if you can survive many years you usually leave the field or create your own startup because of burnout. Faang adjacent companies can be the worst of all worlds same issues worse pay/upside.
Well, I personally have always refused to do take-home interviews but happily will do live coding and systems design interviews.
For me it's about respect and power imbalance. A company asking me to do work without them putting in equal effort sets a tone for a culture I personally don't ever want to be a part of.
Like I find take-home interviews disrespectful.
Time-bounded interviews with an interviewer also there (aka FAANG style onsites with 4 hours of interviews) is far and away my preferred process, especially if I can do them all at once. One problem I've seen in a remote friendly world is companies wanting to spread the interviews out over multiple days.
You see people on HN balk at 500k+ engineering jobs even existing, so I think that's your answer.
Or startups. :)
In the world outside HN I very rarely encountered people who'd turn away from any kind of hiring process. Maybe one in fifty candidates?.. I don't have the numbers, but I think I only met such people twice in my life.
I bailed from interviews for different reasons, but I think that homework is a legit way to test someone's skills, so I wouldn't mind that.
The reasons I cut the hiring process short in my job hunts were most commonly:
1. Employer is an MS Windows shop. Sometimes it's hard to figure this out from the job posting.
2. Employer requires employees to use company-provided tools s.a. code editor, or antivirus etc. In other words, an over-reaching IT.
3. Crazy / not very smart / borderline criminal employer. Examples include a guy who had "scrum cards" deck on his desk and essentially showed me to the door when I asked if they used this stuff for real. Another one who couldn't get my homework to run, asked for a Docker image, couldn't run that either, asked for a VM image, couldn't run that either...
----
There's one litmus test I have when interviewing that turned out to be surprisingly precise, and I don't know why. I ask potential employer if they ever use git-merge. If the answer is "no", the company turns out to be intelligent people who are nice to work with, and if the answer is anything else, it turns out to be dysfunctional in more ways than just infra. They will have toxic culture, under-the-carpet skirmishes where each department undermines another department, while at the same time trying to do as little work as possible.
As you can imagine, unfortunately, I had to take jobs where the employer answered "yes" or "sometimes" etc. That's how I know :(
For all I know, I once joined a company where the policy was to only do squash-merges. I left from there at the brink of mental breakdown.
And what's wrong with merge? How do you even use git without merging?
In theory I don't mind them. In practice I've found that I more often encounter a scenario where I wish a change had been split to smaller commits rather than the scenario where there were too many commits to go through.
Anyway... I intended my example as an explicitly anecdotal evidence to counter the seemingly absurd suggestion of using Git without ever using merge. Feels like going back to subversion or CVS.
For more reliable code-bases you want to do the following:
* Run tests on each commit when accepting PRs.
* Be able to remove or edit intermediate commits, if you find that they've created problems afterwards.
* Only have one path from past to the future.
This is so because if want to use git-bisect, and instead of deleting faulty code you reverted it, the command will keep failing on the code that you've already fixed, and there's nothing you can do about it. git-bisect also has to follow one and only path from the past to the future because if you don't, then, at best, you get an combinatorial explosion of possible paths git-bisect may take, and at worst, some of these paths will fail, but others will not.
So, what ends up happening is this: people who use merges are like people who never clean up their apartment. For some it will take longer, than for others, but, inevitably, the apartment will become a filthy mess. But this is just a symptom of people being afraid of not understanding their code, being afraid of making big changes, undoing things committed to long ago.
This fear is usually an indication that people aren't good at the technology they are using. They would be too afraid to delete code because "what if it breaks something?" -- and nobody can tell authoritatively "no it doesn't". In a situation like this any change in technology s.a. using a different version of the same tool, or replacing the tool altogether will be almost impossible to implement because of the fear.
It's also usually very characteristic of places like this to be afraid of knowing / learning the underlying technology, the one that supports the entire company's stack. Eg. if it's a Python shop, then they'd be opposed to writing Python modules in C, even though this is how Python typically works, because they are afraid that they won't understand this code and one day will end up with a "magical" program that sometimes fails, but nobody knows why.
It's also usually the people who won't even try an unpopular technology, even if the benefits were huge, based on their fear of not having expertise to deal with it. Eg. XML schemas are hugely superior to JSON schemas, and if you want to validate your inputs, XML is just a better tool for this, but the company I'm describing will never consider using it because they are afraid of not being able to find people willing to work with DTD / XSL / RNG.
Such a company will never consider self-hosting, and will pay through the nose for the expertise of others, being mortally scared by a prospect of running their own infrastructure.
----
And... this is the majority profile. The problem is, this is not a winner's profile. It's a scrapping-by profile. It puts an individual programmer in the situation where there's no need and no reward for bettering themselves. Where management is antagonistic to programmers because they are in a conflicting situation, where on one hand they want to give customers more stuff, but on the other hand they are too afraid to make more stuff, since it may deprive them of the stuff they already have. So programmers are punished whether they do or whether they don't. It's where cargo cult flourishes. Basically, Dillbert comic before its author went into politics.
These developers may go their entire careers without ever reversing a binary tree on a whiteboard while juggling two bowling balls on a unicycle.
I quite like how things work currently. It's easy to tell the difference.
It's funny how they claim that a bad hire is devastating, and they can't rid of them easily, but somehow they can do mass layoffs and get rid of a bunch of engineers easily.
Yes, you are expected to hit-the-ground running on day one, but no one will immediately operate at their full potential. Even with all the shared best practices in the world, the secret sauce is the part you have to learn.
As an employer it's very hard to know if the reason for someone's uneven performance is due to ramp-up or if they are just not a good fit. Without a rigorous interview process, so many months would be wasted waiting to get a clear signal on that person.
That also doesn't account for complete cultural mismatches that cause instability in teams and hurt the impact of your other employees.
Another implied reason, good engineers want to surround themselves with other good engineers. So knowing its hard to get into a company signals to each applicant that the other employees there made it through that process.
Maybe that's because companies tend to hire whiteboard-master generalists rather than subject-matter specialists who may not be great at standardized technical interviews. ;-)
Also, if the company culture is ultra-bureaucratic, maybe the company should fix that instead of wasting months on every new hire.
Seriously, if a new engineer can't commit code within the first week, that's a company problem, not an engineer problem. Of course their code shouldn't go directly into production, but that's true of any new code. Give them something small to start, like some bugs to fix.
> That also doesn't account for complete cultural mismatches that cause instability in teams and hurt the impact of your other employees.
Technical interviews can't determine this.
> knowing its hard to get into a company signals to each applicant that the other employees there made it through that process.
I realize that's a signal, but it's not necessarily a good or accurate signal. I think it's mostly PR and hype. Reminds me a lot of fraternity hazing. Google engineers believe they're the best, and some of them may be, but some of them don't impress me at all. And as I mentioned, engineers tend to move from company to company anyway, so if Google engineers are "the best", they're constantly losing the best too.
I hope you realized that this should answer your own questions. Layoffs may be (relatively) easy, but firing someone for "you're just not cutting it" is much, much, much more difficult.
First off, most companies are loath to do large scale layoffs unless there are strong economic reasons to do so - many of the FAANGs have never had layoffs as big as the recent ones. So if your only chance to get rid of bad hires is every 5-10 years or so when there's an economic downturn, that's a problem.
But more importantly, while it's generally straightforward to fire someone who's flat out bad (as there is usually plenty of data to emphasize why they're bad), firing someone for cause who is just kinda mediocre is nearly impossible in the tech world in my experience. For example, if someone can do the job, but say is 50% slower than your average programmer (I've definitely seen this), it can be extremely difficult to gather enough evidence to fire that person. And it usually sucks for everyone involved, because often times these people who are slow are hard workers, but they're just not as capable as their peers.
One of the reasons you see the behaviors you see in technical interviews is precisely because hiring a kinda-OK-but-at-or-slightly-below-par is basically the worst kind of hire you can make.
It's actually not. When upper management is motivated to fire people, they get fired fast. Whether that's an individual person or a large group of people. We've seen this happen over and over. Self-imposed bureaucracy is the only thing that prevents fast firing.
> it can be extremely difficult to gather enough evidence to fire that person.
You don't need evidence. There's no such legal requirement. It's at-will employment.
And I don't want to hear about potential lawsuits. These are ghost stories, designed to scare, but ghosts don't exist. Show me the lawsuits. Incompetent people who are suddenly out of a job don't have the time or money to file frivolous lawsuits (which could get them blacklisted from the entire industry). The ratio of lawsuits to firings is close enough to zero to be negligible, and certainly big tech companies can afford to defend themselves.
I've worked with 200+ engineers and I know of exactly four that were fired for performance. But probably another 40 were quite bad and we would have been better off without them, they just didn't exactly meet the bar for 'so bad we have to fire them immediately'.
> the costs of decreased productivity and morale
I don't dispute any of that. I just mean that they can legally do it, and they don't have to justify it, they don't have to put employees on PIP, they don't have to give reasons why every employee was included. I mean, Elon Musk can basically walk into Twitter and haphazardly fire a ton of people. The consequences may be bad, but it's "easy" in the sense that he can just do it whenever he wants. Even more so for individual firings as opposed to mass layoffs.
0. 45 minute homework/prescreen. Provide an (optional) pre-setup environment so it's mostly about coding and not about building/installing deps.
1. on-site where you chat about your solution, mostly an ice breaker/introduction to the team.
2. pair-programming to extend the homework or work on a simplified but real problem encountered day to day, open book
3. design review
4. code review
5. behavioral / case study
All of these can be pretty objective and don't rely on any memorization. All this should be pre-canned so individual proctors don't come up with their own questions and you're comparing candidates around the same prompts. It's amazing how few companies even manage these basic steps. I think most importantly the hiring should be done by a committee of actual practicing engineers - that means if you have checked in code in six months you aren't a vote on the committee.
1, 2 and 5 should be more than enough. You get to talk to them about their past expertise and even combine it with some design discussion, you get a pair programming session and a final casual discussion. Why do you need everything else?