Article author - I'm on the writing team at Triplebyte. Most of what we do is summarize candidates' technical performance for their introduction to companies, but we also send feedback to everyone who takes our two-hour interview. I took this responsibility over from our first engineer, who built a bunch of software to make the process faster - it lets me quickly autogenerate emails by clicking all the resources I want to include, and then highlights the things that require more careful review. (So the people who accuse me of being a robot are half-right, I guess.)
Isn't giving detailed feedback a liability nightmare? Everything I've ever heard on this is to say as little as possible.
It's certainly great as a candidate to get detailed feedback (would have really appreciated it back in the day as a co-op student), but I just wonder if the concerns over it have any merit or are overblown.
> The number one reason companies cite for not sending feedback is legal risk. Interestingly, I don’t think this is true. Companies put themselves at legal risk if they are rejecting candidates for illegitimate reasons, like race, gender, or a disability. If they send feedback which tells candidates, truthfully, that they were rejected because they didn’t get very far on the coding project, then if anything a company reduces their legal risk: they have a transparent track record of evaluating candidates based only on their skills. I recently talked with an employment lawyer about this, and he didn’t think that specific feedback on technical performance put companies at risk. So legal risk, despite being frequently cited, seems unlikely to be the real driver of policies here.
Then, an explanation that legal risk isn't the same thing as lawsuit risk:
> Even if your process isn’t biased, if you send feedback that creates the perception you’re biased, that’s enough for a costly lawsuit. So while legal risk isn’t a reason not to send detailed, honest technical feedback (as long as you’re not discriminating), it’s a very good reason not to send carelessly compiled feedback through a haphazard process (even if you’re not discriminating).
Right, the liability is not "we are going to send a letter that says you're too old/you're a woman" or something blatant like that, the liability is that a very well-meaning person sends a thoughtful rejection letter and it can be inferred in the language - true or not, and this is the kicker - that there is discrimination going on.
You are doing a nice thing by being detailed, but you are basically introducing nearly unbounded downside for the upside of being nice. Most companies don't find much value in this calculus. It doesn't make it the right thing to do, but it is understandable.
I doubt that liability is much more of an excuse- how often do disgruntled candidates actually sue corporations, in any context? If a company doesn't want a candidate, there's not much value lost by them burning bridges with them, by refusing to send them feedback, or even a rejection notice. This is especially true for big companies who receive a lot of candidates. And for smaller startups, they simply lack the time and energy to give detailed responses to people they pass on. A failed candidate is of marginal use to a company.
My understanding is that as long as you're not discriminating against candidates on the basis of race, gender or some other protected category membership, and as long as your feedback reflects that by being focused on the technical abilities the candidate demonstrated during the interview, you're not actually at all that much risk. Of course, if you are illegally discriminating, or if your feedback suggests that you are by giving feedback on candidate appearance or something (never do that), then you're absolutely better off not sending it.
How do you imagine they would give feedback on something that's neither technical nor illegal? I'm imagining everything from "we found you too arrogant" to "you smelled awful when you came in"...
For arrogant, I'd try to make the feedback as concrete and specific as possible - "sometimes you gave confidently wrong answers. If you're guessing, it's better to tell your interviewer that. Interviewers typically won't hold it against you if you guess and guess rightly, but if you don't acknowledge you're guessing and get things wrong, it raises questions about whether you know what you don't know." or "sometimes it's great to ignore the spec because you have something better in mind, but on an interview it's typically better to demonstrate your creativity and knowledge while still building to the spec - it makes it easier for us to evaluate you" or "when talking about your last company, you said some things that came across as disdainful about your coworkers. It'd be better to highlight your achievements."
All of those are ways being arrogant can manifest, but they're much more actionable than 'you were arrogant' and unsurprisingly get received a lot better.
I wouldn't comment on smell - yes, that's valuable feedback a candidate really ought to hear from someone, but the risk of really angering them is too high for me to feel comfortable with it.
The thing is you still reduced arrogance to technical correctness. However, what I was trying to get at was, what about cases where the technical correctness is just fine? If it's their attitude or hygiene or something else that you don't like, how do you tell them that?
I was trying to get at the same thing you just said, which is that, like you, most people would become uncomfortable providing feedback on at least some of these. Meaning that you would have to turn away these candidates without any concrete feedback. Now how do you imagine they'll react when they realize most people do get feedback but they didn't? Is their reaction (which might result in bad publicity) a risk you and your company really want to take? For what gain?
So the thing is, I think arrogance is typically reflected in actual deficiencies in interview performance. If it isn't - if it's just a vibe that the interviewer got with no concrete implications for how they work with others, solve problems, or communicate - then I worry taking it into account is introducing bias. If I can't think of a concrete implication that the arrogance had, then I don't think I want to take it into account. (You almost always can identify concrete effects, though.)
I think it's a pretty good starting point. I also like Cracking the Coding Interview and I think there's definitely a place for timed coding challenge sites like leetcode - especially if you've been in a role where you're mostly working on larger-scale problems rather than on producing smart, working code quickly on the fly.
Can you tell me what companies find attractive about: "...producing smart, working code quickly[;] on the fly," (emphasis mine)?
I understand there are a lot of substandard programmers in the marketplace. However, why is it that this specific criteria is the one the industry is so attracted to? Could it be that kids are proficient at these kind of games? Because I can tell you that in the 1980s, 1990s, and early 2000s, I wasn't being asked to write a regular expression parser under time pressure.
Others in this thread say they've done Triplebyte take-home tests, only to end up in a "go fast-fast-fast!" interview in the end.
Why is speed so important? Every popular [aA]gile methodology today is implicitly -- if not explicitly -- against such "machismo" programming. If you're pair programming, how is this ever relevant?
Whenever I see someone say leetcode, hackerrank, and Cracking the Coding Interview is the "answer" it translates to "only the young need apply" in my head.
It's interesting that you pat yourself on the back about it, I think the whole Triplebyte interview process is poorly thought out, including your email comms.
I did your online code quiz and got sent an email about doing a 2-hour technical interview, without really knowing much about what the job I was supposed to be applying for was.
On the interview, since I didn't really want to waste 2-hours on something I didn't want to do, I asked the guy a few questions about the company only to learn he's actually a freelancer interviewer, has little direct relationship with triplebyte and doesn't really know anything about me.
I carried on for a 2-hour quick-fire interview with a guy that was obviously trying to fill in a questionnaire rather than actually gauge my ability, questions designed by people who likely have no real-world experience in the scenarios they describe ("how would you architect the amazon.com frontpage?" is not a 2-minute answer)
About 15 minutes in, I was sure that even if I had wanted the job in the first place I wouldn't have taken it; and I had forgotten about it when I got an incredibly patronising email explaining how, if I do some online code puzzles and study hard, I too can get a job. Gee, thanks.
Granted: a bored, funemployed, grumpy dev is probably not your target audience, and I'm sure this interview style works to filter out people fresh off college, but the email was definitely the most ridiculous part.
Yeah, another disadvantage of feedback is that some people really resent suggestions on how to improve - they can come across as condescending. I definitely would rather get feedback to someone who wants it even if this annoys someone who didn't, but I think lots of companies are making the opposite tradeoff - and that's part of why feedback is so rare in the industry.
I don't think the suggestions are bad per se. The reason they come across as condescending is because they're obviously very generic and they're sent to people who, in the current market, can be picky about who they want to work with.
Ultimately your whole business model is competing with tech recruiters, and my recruiters will call me to give me feedback from a role application, you have reduced that human touch aspect to an email with a few links to hackerrank.
I did a degree in symbolic systems (CS + philosophy + linguistics; I have some CS background but less than I'd have gotten from a CS degree), I did one software internship, and Triplebyte was my first job when I graduated. I'm sure every candidate I send feedback is a stronger engineer than I am, but I do have some technical background. Most engineers want to write code all day, not emails, but I think a technical background does help us do our jobs.
At least you get a response even if it is a rejection letter. I have heard - not experienced - that sometimes you just don't hear back from the company at all, which seems worse to me.
Just write up a review on glassdoor. Specifically calling them out for a crappy interview process. Or if you are really bold and bother with linkedin, make a post.
Just like with any "crowdsourced" review site, I find value in the reviews, even if I don't, strictly, trust them.
Of particular value are multiple reviews that corroborate certain details, especially in the absence of reviews providing contradictory details.
At the very least, with something like "ghosting" during the interview process, it can help set expectations and inform behavior such as follow-up. For example, if interview reports complain of ghosting, I might still apply, but I wouldn't exert as much (if any) effort in soliciting a post-interview reply.
It absolutely is worse to not hear back at all. As someone who has dealt with this recently, the one positive thing I can tell myself about the experience is that I would not work for a company that cannot be bothered to write a rejection letter.
It reminds me of a restaurant I worked at in my youth - because the owner and her daughter were so conflict averse - rather than fire someone, they would just slowly write them off the schedule. Sad.
> because the owner and her daughter were so conflict averse - rather than fire someone, they would just slowly write them off the schedule.
Firing someone (without cause) leaves you open to having unemployment claims filed against you. Writing them off the schedule so they're forced to quit on their own accord negates that leverage; underemployment claims are harder for complainants to pursue/win.
Restaurant owners being the stingy type, I guarantee you this wasn't done to avoid hurting employee feelings.
Hourly workers who have their hours dropped below a minimum threshold qualify for unemployment in many states, as that is frequently treated as a de-facto layoff.
You make a strong, valid point. However, in this case, as someone who worked there for a long time, and knew both the owner, and her family rather well.... I can say with absolute certainty it was to avoid the conflict / hard feelings.
This happened to me. Several rounds of interviews with a well known colored-hat company, then... nothing. No response to follow-ups either.
What was especially frustrating to me was that up to that point, the tone of both the conversations and email exchanges was very positive and cordial. I would have expected a "Thanks, but no thanks" follow-up at least, especially considering I was an internal referral from a Sr. Mgr. But... nothing. Made my reconsider my view of that particular company.
When this has happened to me before with bigger companies, I finally get a rejection letter months after the fact. My guess is that it’s them hedging and not wanting to reject until they’ve filled the position.
If in 2-3 weeks you haven't heard from the recruiter, you should ping them back, if no response don't was further time and move along.
Interviews often seem like hit and miss. I would recommend training at geeksforgeeks.org just to refresh dynamic programming, etc. But beyond that you're better off applying multiple places.
Yeah, I was referred to a company and went through the full interview process, ended with a cheerful sounding "talk to you soon" and then zip. No messages and no replies to my emails.
It took my friend there a week to find out that I'd been rejected.
I had this happen on hires I was responsible for, but on "continuously open positions" or when the position is left vacant and the headcount repurposed to another position.
I didn't find a useful or correct way to inform all the potential candidates about that change. That happened two times and I only sent the more generic e-mail we send for rejection telling the people that had had at least one face to face to apply for other positions if they were still interested in the company.
For the easier case of filling the headcount up with someone and not wanting the rest it's easier to send a rejection e-mail, it's just not what always happens with every job opening.
I haven't had much interviewing experience lately, since I was at my last job for 5 years and at my current job for 6, but over the past 15 years or so, I came to the conclusion that not hearing back was the norm rather than the exception. I think it's ludicrously unprofessional, but that's typical these days.
I used to write rejection emails for companies at the end of YC interview days. It always sucked, especially when it was borderline decision (which it often was) but PG made doing these well an important part of internal YC culture. We couldn't leave until they were finished and each one was reviewed by another partner before sending.
In hindsight I'm glad we did this. In the years since I've had multiple people tell me the rejection was a positive turning point and the only honest feedback they'd received.
Getting rejected at Triplebyte was actually a pretty good investment time wise. Guess the whole thing costed me three hours in total and I got quite a list of things to improve and how. It was clear it was tailored towards the interview not just a larger generic mail.
There are tons of companies that give you a generic email after you completed an IQ test, a questionary, open questions and of course the 8hours+ homework. That's just perverse.
But they will. The interview process is stochastic; exactly the same performance from you will lead to different results on different attempts.
> Getting rejected at Triplebyte was actually a pretty good investment time wise. Guess the whole thing costed me three hours in total and I got quite a list of things to improve and how. It was clear it was tailored towards the interview not just a larger generic mail.
For another perspective, here's the entire feedback I got when they rejected my application:
> This was a tough decision and one that we were on the fence about. We really appreciate you taking the time to work on the take home project. We're aware this requires a substantial time commitment and we are really grateful that you invested the time in completing it. We thought you wrote a great, very full featured regular expression matcher. It was especially impressive how much you dug into the academics behind regular languages.
> However we made the decision because we felt that while going through the project together during the interview, we didn't see the fluency of programming when adding to it that we had hoped for. While we specifically designed the take home project track to help overcome the difficulties of coding under time pressure with someone watching, we do still need to see a certain level of programming during the interview. This didn't seem to be the case here, where making changes to the project seemed to be slower and more difficult than we'd have liked.
That's quite a bit less than I got. Perhaps you're just better than me? ;) But I didn't know there might be another take home assignment.
> But they will. The interview process is stochastic; exactly the same performance from you will lead to different results on different attempts.
Yeah I thought about that. But "try x months later" really means that you didn't pass some kind of bar. If I would be one of the five people that passed there would be no problem if I applied next week again, right?
All the ways I discussed in the article - more nuanced, more thorough, more detailed, more focused on constructive advice you can take to your next interview. Plus, we've just improved our process in general so you aren't tested on skills you don't need. It's easier to give constructive feedback about an interview process you have a lot of confidence in, and much of the work we've put in as a company over the last few years has been designing an interview we think really works.
> It's easier to give constructive feedback about an interview process you have a lot of confidence in, and much of the work we've put in as a company over the last few years has been designing an interview we think really works.
It's hard to take this at face value, as 100% of the TripleByte messaging I've seen since before I received that email focused heavily on how confident they were that their interviews were thorough, high-quality assessments.
I think a lot of the dichotomy here is that nobody really knows what a thorough, high-quality assessment for a software engineer really is. I mean, yes, the standard "work sample über alles" line that comes from research is great, but what really constitutes a valid work sample? How do you set up expectations so the candidate knows where the bar is, much less how to get over it? Things like that.
Edit: You also need to consider that TripleByte's idea of what works is probably different from yours. Their idea is probably more along the lines of "people pass our assessment and get hired." All you really need to do to hit that (not that it's a trivial thing at all), is conduct an assessment that's similar enough to what the hiring companies are doing. And, many of us know that hiring companies frequently aren't very good at interviewing.
We don't have the project track anymore, because most candidates weren't willing to do it and it wasn't successfully predicting hires. (There's actually a blog post about this upcoming). We have generalist, front-end and mobile tracks but none of them involve take-home work anymore.No matter which one you do, the feedback will be a lot more thorough than we were able to be two years ago.
The bigger difference that I see is your statement that "Every few months, I check that all our recommended resources are still up-to-date, available, and free."
In some unfruitful further communication with TripleByte, I asked how I was supposed to improve, given that their feedback seemed to be "we have no qualms with your ability, but we don't think you can pass an interview". They recommended that I sign up on interviewing.io. I did that, and a year later, I was still on the wait list. (I eventually got off the waitlist when complaining about this same sequence of events on HN, and an interviewing.io employee saw the complaint. I don't think that approach scales well.)
Also, this looks like a feedback email after a take home project? I guess the rejection emails after the technical interview should be much more detailed.
While we specifically designed the take home project track to help overcome the difficulties of coding under time pressure with someone watching, we do still need to see a certain level of programming during the interview.
“How to set up a test that works and then completely ruin the results by doing nonsensical things during a dumb ritual that our entire industry seems set on preserving.”
Question for Triplebyte: when’s the last time that a single hour of coding — while being watched — determined whether you’d get to keep your job? Not acquire a job, but keep it.
I went through TripleByte 2 years ago and got much the same feedback. I was left with the same feelings, on top of the feeling of having wasted a bunch of time. TripleByte only makes sense if it can fast track you to enough companies to make the investment worthwhile. And, even then, I think, at the time (maybe still now), they only let you skip technical phone screens. Never mind that their process is more or less equivalent to the standard tech on-site (which, IMO, means it should at least count for something at the client companies.)
You need something that confirms the take-home test was really done by the candidate.
I have received take-home tests, passed via recruiters, where the recruiters literally sent me the solutions given by previous successful candidates "for reference".
Although that seems like a valid reason, consider how many justifications have been used throughout history to scare people into submission. “There May be cheaters out there” is of the same form as “there may be <insert statistically unlikely thing> out there, so you’d better <unnecessary overreaction>.”
If there is data to support that most people are cheaters, that’s fine. But at Matasano I believe the statistics were ~30 candidates who were invited for an on-site interview after a take home test, and ~30 happy hires.
The on-site interview was also mostly a formality; the fact that they could do the work was enough to all but guarantee an offer.
And when you reduce it to those terms, it seems ludicrous that the world should be any other way. You can either do the work or you can’t. And if you can, nothing else should matter.
There are other counter arguments: what if someone is a huge introvert and not suited to working in a team environment? Bring on the introvets, I say. You won’t regret it. The most talented coworker I’ve ever seen was also someone who had stammering problems and would barely talk. But he was very nice, and much more skilled than I was at the time.
You don’t know someone until you work with them. And if they’re a bad hire, it’ll work itself out within the first month. It’s far better to deal with a possible cheater than to miss out on a skilled, solid hire. The latter makes or breaks companies; the former are just a temporary thorn.
> But they will. The interview process is stochastic; exactly the same performance from you will lead to different results on different attempts.
I agree that's in practice true for many places. But I hope companies have the good sense to be too embarrassed to admit that their inability to be consistent is the reason to reapply.
I think the traditional reason for the "try again in three months" bit is that paper-based processes make it hard to find and re-contact promising candidates who didn't happen to make the grade this time. But if any place really thinks, "We won't hire this developer now but they would be good in the future," I think a much better solution is for the hiring company to make a list of people to re-contact next time they're hiring.
I did the project (back when that was an option on TripleByte) and failed it. They said I could still try the pure technical interview option, but then ignored me when I asked for that (albeit more than a year later).
Even a year later, you should definitely have had the chance to try again if you'd like. I can reactivate your account; email me at kelsey@triplebyte.com
Id rather get a rejection email than just not hear back. If you dont bother responding with a rejection after an interview you are a coward. And shouldnt be interviewing people.
my triplebyte rejection was surprisingly insightful. I don't know whether it's cookie-cutter, entirely handwritten, or a combination of a few macros, but it makes sense to me.
> When someone can’t answer an interview question about relational databases, it might be that ... they know them inside-and-out but aren’t used to answering questions on the fly, or that our question didn’t use the vocabulary they’re familiar with, or that they misheard the question and answered a different one.
Ultimately, that means your interviews have bias. (Even though it appears you try very, very, very hard to avoid bias.)
Honestly, I don't think interview feedback is a good idea. It just encourages gaming the system. I'd rather that feedback come through a neutral 3rd party. We just haven't set our field up to do that.
Why neutral 3rd party? Because of the above situation! The 3rd party could just say things like, "looks like this was just a bad interview. Don't read too deep into it, and keep trying." The 3rd party could also push back on the employer if the interview ran poorly.
Seriously, we need something written on why rejection sucks?
Rejection sucks because we all like to imagine that we are good enough and the only reason we applied is because we believed we are good enough so it stinks to be told that we are not good enough.
Doesn't matter how you phrase the email, might be nice to give a feedback, but for anyone who receives it, it stings. The only difference is that some folks have a positive mindset, they get over the sting and work towards getting better. Feedback or not.
I was lead dev in a company, I felt strongly that I don't want to reject people without providing them with clear feedback what they did well and didn't do well.
VP hated the idea and very quickly was abandoned. We got a lot of bad candidates tbh, so it was hard to tell them what they did wrong (they bombed pretty much).
I still think, done well, it provides great benefit to candidates being considered.
Thing that worked well for me, I had elaborate set of topics/knowledge I want my developers to know and be rated on, it wasn't arbitrary selection. Still, when someone bombs, it is hard to relay they did bad.
> I work at Triplebyte, and over the last year I’ve written over 3,000 detailed, individual rejection emails.
Wait, what? That means you're interviewing on average 12 people every single working day of the year. Even for someone who's job title was "Technical Recruiter" that would be a TON, let alone for someone who is a Team Lead and presumably has other duties. How is that possible?
It's possible his job is JUST to write these rejection letters. He gets feedback forms from all the interviewers, compiles them, reviews them, condenses them, touches them up, then sends them out.
That could easily be a full-time job for a large company.
When someone can’t answer an interview question about relational databases, it might be that they don’t know anything about relational databases.
I thought you all were better than this. Why are you asking questions about relational databases? Why not just have candidates accomplish the thing you're assessing with an actual relational database? I know you're work-sample-literate! But if your feedback emphasizes communication, doesn't that imply a lot about your process is subjective? After all, and to extend an analysis used in this blog post: it could be that the candidates couldn't effectively communicate knowledge about RDMBS's. Or, it could be that the interviewer wasn't effectively listening to what the candidate was saying.
Because sometimes understanding the base concept is important. I cannot possibly ask you to solve 1,000 scenarios in 45 minutes. But if you get the basic idea of how the concept works, I can be sure that when you encounter scenario #945, you'll have the basic grasp on the concept to at least know where to look, and when scenario #487 comes up, you'll know the basic idea of how to handle it.
Asking you just to do one thing in an interview risks accidentally hitting one of the ten-twenty things you do know how to do, leaving me with no proof that you can solve the other 980-ish possible problems.
Isn't a lot of the job of software engineering to quickly and concisely communicate complicated concepts? Might this actually be an accurate work sample? How else might you measure something like this?
Also the next few lines kinda address what you are saying:
> It also might be that they know them inside-and-out but aren’t used to answering questions on the fly, or that our question didn’t use the vocabulary they’re familiar with, or that they misheard the question and answered a different one. It made quite a difference just to phrase the feedback in a way which acknowledged all those possibilities.
>Isn't a lot of the job of software engineering to quickly and concisely communicate complicated concepts?
No, I write code daily but I might have to distill down technical concepts once a week or so (usually not even that much) and even then if I happen to be the only one who can distill it down.
If your devs are often explaining basic stuff like 'what is a relational database?', you need to hire someone specifically to do that. It's not a good use of time especially when they can go google/wikipedia those concepts and figure it out themselves.
Sure, people don't need to explain RDBMS frequently, but that code you wrote last week? Or that reason you can't do exactly what the PM wants? YMMV but I spend a LOT of time communicating difficult concepts to other engineers (not only mentoring juniors, but also just doing hand-offs and stuff to others), to product managers, sales people etc.
Some engineers really do sit in a quiet room all day writing code, but my experience has been that it is an extremely communication-centric job.
Either way: if you do work in the kind of environment where you need to work with other engs and teams frequently you do need to test communication skills, and simply coding is not always sufficient.
>YMMV but I spend a LOT of time communicating difficult concepts to other engineers (not only mentoring juniors, but also just doing hand-offs and stuff to others), to product managers, sales people etc.
This is a good use of time though. You won't find this on wikipedia, obviously, and is not considered basic technical information.
If your devs are often explaining basic stuff like 'what is a relational database?', you need to hire someone specifically to do that. It's not a good use of time especially when they can go google/wikipedia those concepts and figure it out themselves.
If you can’t communicate ideas and basic concepts to non technical people, you will both limit your career opportunities and not be able to get your ideas implemented while someone with worse ideas will.
Developers underestimate the amount of influence you can have just by being able to communicate effectively.
If they come to me for basic stuff, I tell them to go research it on their own. I'm not going to regurgitate wikipedia if they haven't put in some effort.
At some point, we need to start demanding basic technical competence from the people around software developers.
Otherwise, people will just be interrupting you all day and how much have we collectively written about that problem?
If they come to me for basic stuff, I tell them to go research it on their own. I'm not going to regurgitate wikipedia if they haven't put in some effort.
And that’s why developers don’t get ahead....
There are basically “three levers of power” in an organization - relationship, expert, and role in that order.
The developer who knows how to build relationships is the one that doesn’t get his silly bug put on blast by the QA and gets an unofficial Skype message and doesn’t get official very visible tickets when something blows up in production and gets a quick Slack message so that he can be prepared to explain it.
It’s also the different between a developer who has to submit a ticket to netops and wait three days for a VM and one that can send an email, get it set up within 30 minutes and then create the ticket as a formality.
Once you get to a certain level in your career, part of your job is to be the go to person that explains things, mentors, spends way too much time in meetings and just greases the wheels. The heads down developer is not seen as the multiplier like the team lead/architect is and they get paid accordingly. I have my office days where I expect to get interrupted and my work from home days where I don’t.
No one gets promoted by constantly telling coworkers and management to RTFM.
>Once you get to a certain level in your career, part of your job is to be the go to person that explains things, mentor, spend way too much time in meetings and just grease the wheels. The heads down developer is not seen as the multiplier like the team lead/architect is and they get paid accordingly.
Great. That's exactly what I said in my original post -- hire someone specifically to do that. Problem solved. Now your junior/mids don't have to explain that. But that's not the career trajectory of every developer, let's be honest.
If someone's going to deny me a promotion for linking a wikipedia page that answers a basic question and completely ignore my technical contributions, I absolutely do not trust that place has the best interests of its developers in mind and is likely driven more by politics than anything else.
If someone's going to deny me a promotion for linking a wikipedia page that answers a basic question and completely ignore my technical contributions, I absolutely do not trust that place has the best interests of its developers in mind and is likely driven more by politics than anything else.
No place has the "best interest of its developers in mind". That's true for any industry. It's a lot easier to replace the on the floor factory worker (i.e. the developer) than the foreman (the architect) and they get paid accordingly.
Don't get caught up on the title, role power is the least effective type of power in an organization. If you leverage relationships and can be seen as the expert, you can easily punch above your weight.
Why do I need to punch above my weight? So I can finally tell people 'no' when they try to take my time away with stuff that they can fix themselves?
I think it's clear we have two very different motivating factors in careers. You want to climb the corporate ladder and get power and influence, and I'm content building things.
That's just the point I've been making "climbing the corporate ladder" is about gaining role power. Role power is the least effective method of getting things done in an organization.
I wanted to "build things" my entire career and was stymied by management and team leads who I couldn't convince to see my "vision", net ops and dev ops who made me go through mounds of red tape to get anything done and coworkers who had their own agendas and wanted to get noticed and get the prime projects just as much as I did. It led to a lot of frustration.
The best way to be able to "build things" the way you want is to have the influence to do so by getting people on your side through relationship building and getting people to trust your expertise.
I like to "build things" too and have no desire for management. But, the way to stand out and make more money than the average, heads down developer is to have soft skills.
What you say makes complete sense. It is correct as per my observations. However, one big issue is that sociopaths (manipulators, idea\effort-appropriators...) have an edge. Also, such people accumulate and ruin workplace.
Once a week (or even every two weeks) is still frequent enough to count as a core job responsibility. And the raw frequency understates its significance, considering that it can be a blocker for other employees' contributions.
Now, you're right, the technical distillation is not going to be on the level of "explain relational databases to a Joe off the street starting from square one", but it's effectively the same as the skill of "demystifying arbitrary misunderstandings and knowledge gaps other have", which is important.
"or that our question didn’t use the vocabulary they’re familiar with"
I lost out a job interview in 1997 because I wasn't familiar with "state management" for websites. The guy was pretty insistent I needed to know what "state management" was. I'd already sent over a project using session management (and he'd created an account, logged in, I sent him the code, etc), but... I didn't know what 'state management' was, so I was passed over. I wasn't strong enough on the phone (and was in a different country at the time - was worrying about the long distance charges!) and... it fell apart. I was essentially a perfect fit (had had an interview before - this was second interview with someone else), but I choked on that phrase, and they passed me by...
Maybe good oral communication skills are part of these particular requirements? We don't know the details of the interview. But I don't understand why it should be wrong to ask a candidate a question that he should be able to answer and see how he reacts.
For some jobs it may even be appropriate to ask questions that the candidate cannot answer and see the reaction. Does he admit that he does not know? How does he phrase it? Is he making things up to cover for his lack of knowledge? And so on.
I have been doing mysql database type things for 15 years solid. Once during a job interview at digg.com, someone [ok ok doxx removed] asked: "what is a having statement in sql?". I jumped in explained how you can filter aggregated sets performed by a 'group by' and rambled on and on. He stopped me and said, "You can use a having statement without a group by, are you sure you know how they work." I have never seen this or done this in practice, so I just stood there stumped. He rolled his eyes and ended the interview. 4 hours of an amazing interview going well: ruined. At least this guy went on to ruin digg.com. "Let's just re-write everything over again better" kinda guy.
"Doxxing", in the original sense, involves publishing more private info, info like address, home phone, cell phone, SSN, etc. A link to someones public LinkedIn page is not a "dox."
That is an absurd definition of doxxing. Reporting the names of you have had in-person interactions with, in cases where it would not expose someone's pseudonym, is in no way, shape, or form doxxing.
If I understand correctly from this post on dba [1], HAVING without GROUP BY would have the same effect as WHERE. Seems like they were just being pedantic for no good reason.
If one of the columns you're SELECTing has some sort of complicated expression in it (e.g. `UNIX_TIME(colA) - UNIX_TIME(colB) AS duration`), I think you can refer to that column by name in HAVING, but not in WHERE. It's a pretty niche thing though; I can't remember if I've ever actually used it.
Do you realize you dodged a bullet at that moment? That was a classic case of "smartest-guy-in-the-room" syndrome: from your description, he wasn't there to interview you, he was there to show off and stroke his own ego.
Be glad you didn't get that job, because that would have been the future of your days at work -- constantly listening to Mr. Smartypants compensate for his own sense of inadequacy, where every different opinion is treated as a personal insult or a challenge. No thanks. That personality type is infectious (not in a good way) and it damages an organization.
Hm, I'm honest: If I get along well with an interviewee, I might throw technical oddballs at them like that. I wouldn't have known that specific one.
But at that point, I'm more interested in their personal reaction. Pondering, and/or asking "Why the fuck do you even do that!" would've been fine there to me.
This person probably wanted you to say that there might be a performance problem in filtering a result set using HAVING, instead if selecting rows using WHERE, if the query optimizer fails to see the equivalence. And the only reason why they knew was because they fixed that very same problem one week before, after debugging it for three days, as it is often the case with bad interview questions.
I suspect a lot of my feeling of whether these people are good or crazy comes down to whether they can contextualize their questions. Most of the questions I get asked are for code I would veto in a PR because there are battle tested alternatives.
I’ve been trying to do something about that when it’s my turn to ask interview questions. For instance, too many front end people struggle with basic data manipulation workflows. I want to virtue signal that it’s better to move this kind of logic to the backend, but I need to test them anyway.
So I create a plausible scenario, maybe this is a POC to see if it’s worth sort or grouping functionality to the backend.
Sure! It's an objective work-sample process if you can say these things about the challenge:
* It's the same for all candidates.
* It captures facets of the work as it is done on the actual job.
* It's objectively evaluated (ideally, it has a rubric established a priori, such that results can be evaluated by someone other than the person who proctors the challenge).
It's possible to devise work-sample challenges that assess communication skills. I have friends who've done it at their companies for customer service and sales roles. I'm saying, the process described in this blog post does not appear to be that.
I think you're missing my point. Qualifying candidates based on relational database skill is not a problem; if it's something they'll be expected to do on the job, you should evaluate their ability. I'm saying the kind of Socratic interview alluded to in this article isn't the best way to accomplish that.
Why not just have candidates accomplish the thing you're
assessing with an actual relational database?
My employer interviews like this, and I can tell you one reason it's not very common: It's a pain in the ass.
After all, it'd be unfair to judge someone on a platform they weren't familiar with - so now you gotta maintain a fleet of laptops with a really wide range of tools. And these have to sorta float outside the usual IT management system because they aren't really issued to a single person, and you gotta be online enough that people can google stuff, and you can't have hiring managers let other people use their login, that'd be bad security practice. And if you didn't confirm in advance what platform the candidate wanted to be tested on, you gotta haul three laptops to the interview. Oh, they're pretty good developer laptops and one went missing? We really ought to have people sign those out...
And even after that, you _still_ have to apply subjective measures like "were their variable names clear?" and judge them on communication - like if they see an opportunity to refactor the code for clarity, but they say they're focusing on completing the task before spending time on that.
I'm not saying it's a bad thing to do this, just that I can understand why many companies don't.
It is harder to build a work-sample regimen than to just send candidates to interviewers, sure. But then, the point of Triplebyte is that they're eating all that work for you.
Regarding "fleets of laptops" and environments and all that jazz: these seem like unforced obstacles. Just have the candidates use their own machines. Here's a crazy idea: have them use their own offices/couches, too.
Regarding security practices... come on. We have an interview process that involves giving remote developers read/write access to an entire AWS environment. These are simple engineering problems. If they're the only thing stopping you from having a better interviewing process, and you hire regularly, just go solve the problems.
We tried inviting candidates to bring their own laptops, and it turns out often they didn't _have_ laptops - or we'd tell them there was a Java-based test and (perhaps due to a miscommunication or because this is an uncommon interview practice) they'd arrive with a laptop without a working Java VM or compiler.
Needless to say, you can't objectively compare two candidates' progress if one of them spends half the interview trying to get their environment set up!
Regarding security practices... come on. [...]
These are simple engineering problems.
Perhaps I wasn't clear: My employer is a medium-sized company, and consequentially IT security is actually a complex political problem rather than a simple engineering problem.
You've also made a pretty big shift in the goalposts there, from asking "why don't companies have candidates accomplish the thing they're assessing?" to asking "why don't companies have candidates accomplish the thing they're assessing, and perform remote video interviews, and have the ability to give spin up clean AWS environments and give remote semi-trusted interview candidates access to them?"
If you insist on doing the interview in person, why not just tell them ahead of time what their environment needs to do when they get there? Give people instructions and a script (formal or just a numbered list of steps) that determines they're ready to go.
Better yet, just don't make people do that stuff in your office.
Why bother videorecording interviews at all? I'd have problems writing a line of code with someone breathing down my neck. If you did it to me on the job, I'd chase you out of the room. I feel like a lot of these problems are, like I said, unforced.
Agree to disagree about the degree of difficulty of getting clean environments to candidates. You're either serious about recruiting as an engineering problem or you're not. "Not" is fine, but then don't pretend like there's some kind of rigor in deciding which corners you're willing to cut.
I'm not making this stuff up; this is how we've been hiring people for about 10 years now, and every time I hear someone explain how challenging or untenable our process is, I keep wondering, "what am I doing wrong to make this work so well for us?"
We don't record interviews. By "remote video interviews" I meant "remote interviews by videoconferencing such as Skype or Google Hangouts" which I assumed was what you meant when you proposed candidates use their own office or couch.
why not just tell them ahead of time what their
environment needs to do when they get there?
We did this, including a github test project they could check out and build to make sure everything was working. Still, about 50% of candidates arrived without a working environment.
We chose to supply a known-good laptop instead of rejecting such candidates instantly when they've spent time travelling to us etc.
every time I hear someone explain how challenging or
untenable our process is, I keep wondering, "what am
I doing wrong to make this work so well for us?"
I'm not saying your process is challenging for _you_ given _your_ situation; I'm sure it works very well for you! I'm saying it's challenging for _us_ given _our_ situation :)
Maybe I'm being unclear. I'm asking: why does there need to be video or telephonic oversight of any sort for a work-sample challenge? Why are you assigning yourself that problem? We've never done that and never had a problem.
Some people involved in our hiring process don't like them, they say experienced candidates stop responding when sent take-home tests. Their theory is employed people with families don't have the time - although we don't have hard empirical evidence for this for obvious reasons.
People don't do take-home tests because companies give them in addition to interview loops. I'm saying, just do the at-home test, and cut out the interviews.
> Needless to say, you can't objectively compare two candidates' progress if one of them spends half the interview trying to get their environment set up!
If most of the issues revolve around having roughly the same environment for candidates, just create ready to go VM images, for example VirtualBox, and share that with them. Or use a cloud desktop VM.
All of which make it easier for someone to succeed.
I agree with you generally, but you'd better have loaner machines ready. Not every candidate currently has a working, dev-capable laptop. Economic bias. (Hell, my current laptop is only 70% functional because I can't decide if I want to repair or replace it.)
If economic bias is your concern, then you shouldn't be asking people to travel onsite for a tech-out interview; you should be doing everything you can to make tech qualification remote, so that by the time you need to call them to your office, you and the candidate have a pretty good idea it's worth disrupting their work and home life with the trip.
After hundreds of interviews (I have no idea; lots, over 10 years of almost continuous hiring) I've never run into a candidate that couldn't do a tech challenge remotely.
And I don't know that that's better than "walk me through the query you'd use to do X". Because the reality is they may need to look up the exact syntax for something, and I can't gauge how much they actually know based on their googling; someone who knows nothing might stumble on a good search and seem to get it really quick, someone who knows all the fundamentals might spend 5 minutes trying to find a keyword that's just slipped his mind ('having', for instance, which I've used maybe...once? :P), who will seem like they know nothing.
Instead, just describe it to me. Best guess it. We'll dive into that.
> I thought you all were better than this. Why are you asking questions about relational databases?
You write about hiring from the perspective of someone with hiring authority. TripleByte doesn't have hiring authority, or even sufficient reputation to get their candidates out of doing another technical interview at the companies to which they apply.
There are two problems you might solve:
- Joe Nerd needs a job. He knows everything about relational databases, but no interviewer has ever noticed this. His limp, effeminate handshake leaves them unimpressed.
- IBM needs a database engineer. They really want to hire someone, but they're having trouble filling the opening; their existing network of friends-of-current-employees is tapped out.
That is to say, you could try to optimize for finding people who will be good employees, and then bully companies into hiring those people, or you could try to optimize for finding people who will pass an existing hiring gauntlet, and then introduce them to companies where the magic will happen naturally.
The first approach solves the candidate's problem and would logically charge fees to the candidate. The second approach solves the company's problem and would logically charge fees to the company. TripleByte wants to get money from companies, and follows the second strategy.
But... they like to send messages as if they were following the first strategy, because that strategy solves the candidate's problem and those messages therefore attract candidates to TripleByte. I don't like this.
My experience with trying this for a year at a previous startup which shall remain nameless is that no matter how I approached candidate qualification, work-sample or interview or ritual chicken sacrifice, I'd still have the same problem of clients rejecting candidates by default. Recruiters mostly all work on your "second" model. So why add crappy tech qualification to your problems? Do at least that right!
According to my mental model of the world, if I'm trying to find people who will pass their interviews at IBM, then the more my qualifying interview looks like IBM's interview, the better I'm doing.
I find it very plausible that your experience ("acceptance rate doesn't seem to change no matter how I personally vet the candidates") is more realistic than my armchair model, but I suspect the model is intuitive to a lot of people and will go a long way toward explaining "why are you asking questions about relational databases?".
You're ignoring the sentence that came immediately after that one:
> It also might be that they know them inside-and-out but aren’t used to answering questions on the fly, or that our question didn’t use the vocabulary they’re familiar with, or that they misheard the question and answered a different one. [...] People are generally open to hearing that, one way or another, they didn’t manage to demonstrate that they understood a topic.
The author is actively acknowledging that being unable to answer a question about RDMSs doesn't mean they don't actually know anything about them.
And the point, I think, stands. An interview isn't a passive process where by some magic algorithm they determine good candidates from bad and the candidate just sits there hoping the right question will be asked. You have to actually communicate to the interviewer your knowledge and experience because they don't know.
No, I get that! I think the author does a good job of implicitly recognizing the weakness of interview processes. I'm really just saying 2 things:
1. They missed a failure mode: in addition to (a) lack of knowledge and (b) lack of communication skills, there is also (c) lack of interviewing skill.
2. It's possible to design an interview process that is resilient to both (b) and (c), and I figured Triplebyte, "who has just one job", would do that.
In addition, on this thread, I've tried (badly) to point out that while (b) is maybe a reasonable thing to check candidates for, it's better to do that explicitly, with an actual test of communication skill, rather than something that can easily get confounded with (a) and (c) (and all the attendant stress that situation generates!).
Thanks for the opportunity to clarify; I'm doing too many things at once today.
> Why are you asking questions about relational databases? Why not just have candidates accomplish the thing you're assessing with an actual relational database?
It's not the same thing. Browse around the various SQL tags in StackOverflow and you'll see plenty of candidates who can "accomplish" things using relational databases yet have positively no idea of how they work.
When shit hits the fan they're asking strangers to optimize their thorny queries. But a modicum of understanding of how a relational database works would have led them to a better way to do things to begin with.
I think the better example of a problem would be asking them how MySQL handles something when you really care about relational databases/SQL in general.
Obviously if the job is highly MySQL specific and they need to know all the quirks that’s relevant.
Are you trying to say that there's some technical detail about solving a problem with an RDBMS that can only be expressed in an interview question? I call "shenanigans" on that. If the most obvious challenges are too easy for candidates to solve, so that they can just copy the answer from Stack Overflow, come up with better questions.
Not saying that at all. Merely that a mere of understanding of a few basics, such as how indexes work and when to use one, goes a long way towards being more effective at interacting with a database.
"Here's some schema information. This complicated query meant to do X is running too slowly. Can you recommend ways to speed things up and explain your reasoning?"
I've done the TripleByte interview, so the decision to ask questions about relational databases instead of asking for a practical exercise makes sense in context.
It's because you can only fit so much into an already long interview (2 hours). A big chunk of that time is already spent on an exercise about reading/writing/debugging complex code. You can't fit everything in, so database stuff is moved to the non-coding section. Also, the questions aren't "guess the right answer" questions, the interviewer keeps digging with open ended questions to see how deep you can go.
> it could be that the candidates couldn't effectively communicate knowledge about RDMBS's. Or, it could be that the interviewer wasn't effectively listening to what the candidate was saying.
You could certainly get a bad interviewer, but that's a strawman here. If it's not TripleByte judging the candidate's communication skills, then it's the hiring team judging that. The suggestion was about how to give feedback about communication skills. And there are definitely stronger and weaker communicators, and it definitely makes a big difference in day to day work.
I don't think legal risk is the main reason companies refrain from elaborating on why they chose not to hire. That's just a convenient thing to point to (in other words, lazy). It's more like, there's all downside and no upside. I can't think of one benefit that would accrue to an employer for doing that, but I can think of a lot of bad (or just undesirable) things that could potentially come from it.
Companies hire the candidate they like best, they don't even spend mental energy 'rejecting' the other candidates. If you don't get hired, it says more about the candidate they chose and little about you.
I'd settle for "do you at least send a rejection email of any kind to the candidate?" vs. the "dropping someone on the floor and not responding for weeks/months/ever". Even without detailed or personalized feedback, it's a huge improvement. Also if an internal person refers someone, you should let the internal person know if you're passing, I think (depends on other factors, though.)
"Ghosting" seems like a common phenomenon across different types of human interaction. There's dating, interviewing, and even pitching investors.
This was a common experience for me pitching my company last year:
1. Investor likes my co enough to schedule an in-person meeting
2. I meet investor in person to pitch
3. I send them a followup email
4. Radio silence
I'd read that investors like to keep you in limbo instead of passing, I just didn't realize these well-respected professionals would value someone highly enough to give them an hour of their day, but low enough to neglect all followup communication. In retrospect I don't think it's a big deal, but I felt bad about it at the time.
This is the biggest US/Europe cultural difference I've noticed. I can't speak for every country in Europe or every industry, but for every job in Europe I've personally applied for, I got a rejection letter if I wasn't offered the job. Sometimes just a minimal notification, but always something. Not in America though...
The best is for European academic jobs. Often there's a schedule up front for when they'll make a decision, usually a 2-stage thing like: we will shortlist candidates by Sept 15, interview in the following 3-4 weeks, and make a hiring decision by Oct 15. So if you didn't get shortlisted, you get informed early and don't have to wonder whether your resume is still under consideration or what. American universities, though, leave you guessing what their schedule is, may take months to get back to you even if they're interested, and usually don't send a rejection letter if they aren't.
So, good article, but another point occurred to me while reading it that wasn't covered. I think that writing a good email explaining the rejection, might be a good exercise for the company doing the rejecting. "Well, why are we rejecting this person, exactly?" Of course, this will only be true if the email is honest (if diplomatic). It could serve as an institutionalized review of whether candidates are getting rejected for the right reason. Building feedback loops into a company's internal process can be a very powerful thing, even when there's no bonus or penalty involved for the person doing the writing.
I wonder, does Triplebyte have any kind of annual summary of why candidates are getting rejected?
The other reason specificity is a legal issue is that it provides data that can be disputed. For example, if you reject someone black and say the reason they were rejected because hey didn’t get far enough in the coding test — you’d better not hire the white guy who only got just as far (which might happen for a variety of reasons).
It’s almost the same reason you stay quiet when held by police. Even something seemingly innocuous may end being used against you in the future.
If you/your company doesn't have a policy of sending personal feedback please consider doing this at least for junior people. Volunteer your after work time if needed.
Trying to find the first job is extremely stressful process. A junior person has no notion of his worth on the market. Each rejection even if only by a lack of any response ("I'm sorry, I'm afraid we are looking for a bit more experienced person" would suffice) can be like a kick on the face when you're just barely learning to walk and most likely is a burned bridge.
I've mentored my girlfriend for 3 years from almost 0 to getting her first job in a company run by a React Native core developer. She had the skill, great attitude, really solid work ethic and very analytical thinking. It'd trust her more with any task than significant number of my past and current senior coworkers. It's hard to prove and no one expects that so naturally her applications had been ignored or rejected. With each one I saw her confidence, self-esteem and enthusiasm crumb. With each positive reply/invitation she was invigorated until the next step came. I'm pretty sure for some the roller-coaster or even worse, being rejected over and over again can be a life defining experience.
Any reply is great, personal is even better. If you spend time describing what was missing from the expectations of your company (don't say "You don't know enough", say "We need someone with more knowledge") and sincerely wish the person well you can be sure they'll be grateful, remember you, work on the gaps and who knows... maybe some day become part of your team.
Please feel free to reach out if you want some example for inspiration.
Edit: Please don't do that against the policy of your company. But if there are no reasons against just ask if you could provide some feedback and resources for the rejection letter.
If someone told me that in my country (or maybe just outside US) I would simply not believe them at first. It might just be the choice of companies I've worked with though.
You're willing to bet your own employment on that? We'd all love to live in a world of butterflies and sugar plums, but risking your own for that is silly don't you think?
Absolutely I would. Employees create the culture too. If you cant push back against stupid policies then it is probably good to be pushed out and move on. Plus any environment that has a policy like that is probably a very sterile workplace.
Off topic, but I find it amusing that we all do in fact live in a world with both butterflies and sugar plums. They are both very cheap and plentiful as well.
Managerial CYA, and fear of the potential for litigation are crippling corporations more that I think we'd like to admit. Financial risk is something people seem comfortable accepting, but the specter of unknowable legal risk cause so many management anti patterns and so much passive aggressive behavior is it incredible.
As a relatively junior person in management, it is amazing the kind of phantom fears I've been cautioned against. Some of which don't even have any legal precedent at all!
I think a lot of it is trotted out as managerial "emergency hypothesis" for why someone doesn't want to do something, and so invents some plausible legal risk to justify their decision. But, honestly, it can't be only that.
For a swath of business responsibilities, this "phantom legal risk" is the Most Available Excuse (TM).
We see Most Available Excuses in product feedback ("it's too hard to use" is easier to say than "I didn't see how this would help me accomplish anything I actually care about"), in social engagements ("sorry, too busy this week"), and many other areas.
I'm a natural skeptic so I maintain a mental set of Most Available Excuses, and when I hear one I treat it as a dodge, not an answer. Why don't they want to do it? How might I make them more comfortable with me so they can explain how they really feel?
Except when it’s really the reason for not saying. Should I tell my 48 candidates that I didn’t take them because they’re foreigners and I don’t want to do the visa parkour? Is it illegal to select on this criteria? Do I want the antiracism unions on my back? Ok let’s tell them a generic reason. Here again, the legal excuse is the genuine reason for not telling.
PS: my first employee is already a foreigner, so don’t lecture me on racism.
Why don't they want to do it? How might I make them more comfortable with me so they can explain how they really feel?
I'd be very cautious with this approach. Politeness is an extremely important social defense mechanism for most people. By ignoring the standard polite response and trying to get at the truth you're undermining a person's attempt to save face. By doing so, you're attacking their autonomy and agency as a human being.
I'm not suggesting directly asking these questions of the other person. I'm suggesting these as topics for you to think about if you're getting the polite excuse and want the actual reason. To get that you have to actually make them more comfortable; calling them out on a polite lie obviously isn't going to do that.
> fear of the potential for litigation are crippling corporations more that I think we'd like to admit.
Sure, I sometimes see it that way until I'm in the seat of a employer. This isn't a one-sided "corporations suck" because guess who's doing the suing in these cases? That's right, the potential employee. So where is the onus in this negative externality? I'd argue both sides.
Agree. I remember having applied to McKinsey many years ago as a graduate. I got rejected after the first round of interview, but one of the interviewers would call me to explain their rationale. I was very grateful but I can imagine that it must be painful when the candidate doesn't react well or sees that as an opportunity to put the foot back in the door.
I've rejected dozens of people right after face to face coding session of a simple task (third meeting, first technical) explaining the reasons, providing them some guidance on what to tackle next, if I saw hope invite them to try again once they feel they filled the gaps or at least describe the progress and ask if we think it's enough to try and/or what could be next.
I've most likely never had a person who left without a handshake with a sincere smile on his/her face and most of them expressed their gratitude.
Sometimes you have to reject a person on what you feel is a gut feeling. It's because over time you developed an intuition which is picking small details in a less conscious manner. In the end there are some reasons your intuition is shaped that way not the other and you can find something that presented within the context of your company and expectations will resonate with the candidate and he won't feel like he's been scammed.
Having to give that kind of feedback just sounds horribly draining to me. I know there are people who would accept it. There are people it would really help.
Some would already know it and just say ‘thanks’.
But the people who argue with you that you’re wrong, or they need another try, or ‘I forgot to mention X’, or ‘you just hate me because Y’... the people who feel hopeless and you’re just ‘confirming’ their fears (even if they applied for something way outside their qualifications) or falling apart over it.
I would never want having to deal with that be a big part of my job.
> If you/your company doesn't have a policy of sending personal feedback please consider doing this at least for junior people.
If you want to keep your job, don't follow this advice. Honestly, I would love nothing more than giving junior people (all people in fact) feedback on why they didn't get the job, but people will sue at even the slightest hint of any type of potentially litigious situation. It's even worse when the person is in a position of desperation ("I can't pay rent because I can't find a job...oh what this lawyer is going to take my case on a performance basis...heck ya, let's sue those assholes!"). Most of these cases get settled because no one wants to deal with them and it's easier to just pay the problem to go away, so they're easy wins.
Seriously, most good honest HR people WANT to give rejected candidates feedback, but asking them to benevolently provide feedback at the risk of their own job won't earn you many friends.
I didn't say to do that against the policy. What I meant was if your company simply doesn't have a habit of doing it just say: "Hey, can I write some feedback and provide some resources to that candidate in your rejection message?".
Edit: My intuition is most companies don't provide the feedback because simply they don't see a value in the time spent or simply never really considered that as there are always other things to do than think about people you'll probably never meet again.
And I didn't say anything about this having to do with policies of companies. Regardless of whether you have policies in place or not, doing what you suggested without any counsel is a very dangerous way of losing your job. Companies will make it very easy to fire a person who costs them a lawsuit (and the subsequent negative PR).
> My intuition is most companies don't provide the feedback
Your intuition is probably right, but it doesn't mean that people who DO want to give feedback should all of the sudden ignore sound advice. Bad HR teams who don't want to give feedback will always use legality as a shield for an excuse, but that doesn't mean that good HR teams who are also taking legal advice are not interested in doing so. As many people have cited in this thread - the long term effects of giving feedback to rejected candidates (they tell their friends, they come back later, etc) are numerous, but the downside risk makes it very difficult for any HR team to make this investment.
Why not make candidates sign a non-liability agreement to receive feedback? Of course, choosing to ask for such feedback would be optional - but if you want it, sign the form so you agree you won't sue.
Most people would find this information quite valuable to correct their mistakes for next time. It makes the whole process much more efficient for all parties if there's quick, specific feedback.
You can't absolve a company from liability for US labor laws. So even after you sign the non-liability agreement, you can still sue. It's not a real shield for the company.
If they send feedback which tells candidates, truthfully, that they were rejected because they didn’t get very far on the coding project, then if anything a company reduces their legal risk: they have a transparent track record of evaluating candidates based only on their skills
I think the problem comes when he talks to his friend of another race/gender and that friend said "Yeah, I couldn't finish that either, but they still hired me". The company may have had a legit reason to overlook the coding project (like the second candidate had experience in some other technology), but when you tell candidate X that they didn't get the job because they didn't complete the coding exercise, but then you hire candidate Y despite him not completing it, it provides candidate X with some concrete evidence of discrimination.
>I heard back from many of the engineers I write to, and often, they were furious
Bingo. I've opted to share specific team feedback via phone and although candidate feedback was generally positive and thankful, once in awhile the reaction would be extremely negative. I now opt for the much more (emotionally) safe route.
Triplebyte is more incented to provide candidate feedback because if the candidate improves, Triplebye may be more likely to place them in the future. With companies, this incentive is less apparent.
Yeah, this sounds like as much of an issue as the legal risk to me. Imagine if you reject someone and they blog about the letter you sent them and the interview process, and the post makes Hacker News or Reddit.
Now you have to decide whether to fight in public with someone you didn't hire, normally bad form, or say nothing.
I'm going to go against the grain and say that interview feedback is overrated (at least for senior people). If you got offered the job, then there's your feedback. If not, the interviewing company isn't going to tell you any more than you could already discern yourself by playing back your answers and conversations with the interviewers. If you honestly feel you aced the interview, then other factors are in play - perhaps they realized they are overstaffed, layoffs are imminent/hiring freezes, or just a slightly better and more personable candidate came along.
One time I interviewed for a position that I wanted badly. I studied and prepped for the interview, then during the interview I nailed every question. I waited a week but never heard back. After a few weeks of silence and giving up hope, I searched the company on LinkedIn, and found the person they hired for the position. It turns out he had more backend experience, which is what they were looking for. It was a painful truth, but them sending me a rejection email telling me this wouldn't have helped me at all.
Precisely. Often times a rejection isn't saying that the candidate didn't meet the job requirements or wouldn't do well in the position, it says that the employer found someone who fit the role even better or knew that they could given past hiring experience.
I agree with you, except that the company should send out a rejection letter as soon as they can. If I were hiring, I would only send a non-generic response if the rejected candidate specifically asked me if I would be willing to give her or him constructive feedback, or otherwise indicate why they did not get the position.
What if they are not rejecting you? What if they like you as a candidate but decided to move forward with someone else (and it was a tough decision), and want to wait to see if that person will work out. It sucks, but it happens.
A rejection brings closure. If I spend four hours answering these silly Big O questions, the least the company could do is give me a definitive yes/no in a timely fashion.
the hours of interview time, scheduling, and travel shouldn't be discounted so easily. not that the company owes you anything, but surely some of the smartest business minds in the world can come up with a way to safely provide some feedback. i'd probably sign a waiver to at least get something for my time
I once got a detailed, feedback-driven rejection email, stating very clearly and professionally where I shined in my interview, and where I didn't. I was so appreciative of the message that I made sure to follow up with the manager working on my application and thanked them.
9 months later, I found myself in a bad management situation at another company, thinking about looking for another gig when the company that rejected me reached out asking if I'd be willing to come back and interview again. I did and accepted the offer.
By giving good and candid feedback, they wound up saving months of searching for a new dev when they reached back out knowing I was a good fit for what they needed at that time. I was essentially a lead they'd already warmed months prior. It made me wonder why more companies don't think of this.
When I was a hiring manager, I used to always include a personal note that included suggestions and constructive criticism for the candidate. In a couple of cases, those people replied to me, demonstrated some actions towards those goals, and I hired them later, when I had more available positions.
And those that I didn't hire, I encountered them at other companies. It was flattering to hear them say they remembered me and had a positive impression of our recruiting process, even though they were rejected.
I've always believed that the recruiting process is a great way to sell one's company. Even if the candidate isn't a good match, that candidate may recommend peers to the role if they have a positive experience with you.
I appreciate people like you but unfortunately most companies' policy mandates not talking about the reasons due to some sort of legal risk of law suit.
I interviewed for an internal position once and got similar feedback, by phone, from the HR person. She was on the phone with me for about an hour and, in all honesty, that rejection phone call made that one of the best interviews I've ever had.
I didn't get that job, but it gave me a lot of constructive advice and I ended up getting the next one I interviewed for.
That's nice and helpful, especially since tech interviews can be taxing. Sounds like they actually liked you and probably would've hired you under different circumstances.
I had terribly frustrating experiences interviewing. Mostly just taking a bunch of tests and interviewing two, three times, and not hearing back for months. What sticks out was a post-interview for a large company that aggressively recruited from my uni. When I asked how I could have improved the answer was, "You ask too many specific questions about the company and software platform. Be more focused on the interviews."
"For example?"
"It isn't appropriate to discuss how wages are adjusted according to location or salaried overtime policies or the tech stack... in an interview..."
I took that one as the, "not gonna drink the Kool Aid." box being checked. Dodged a bullet there, though, seeing as her answers did not exactly inspire faith.
> In a couple of cases, those people replied to me, demonstrated some actions towards those goals, and I hired them later, when I had more available positions.
I've seen this being automated in enterprise recruiting systems as "Candidate Relationship Management" using terms like "silver medalist" to identify and re-engage folks who didn't quite make the cut for positions they interviewed for but may be good fits for other open positions or for future pre-vetted candidates.
I once sent 49 rejection letters (for an internship!): « Here are statistics about the 50 applications I’ve received. » Received big thanks from at least 50% of them.
Last internship position I opened, I received more than 100 applications. Going through all of them and replying (even with a standard template) was grueling.
I did give feedback to everyone we interviewed, though.
A good 50% of the resumes were flat out wrong for the position or missing critical information. A standard rejection letter with stats like these and basic recommendations might be useful in the future. It might also trigger a lot of self-righteous justification though... Don’t know if I’m up to receiving another 100 re-submissions with cooked CVs.
>When I was a hiring manager, I used to always include a personal note that included suggestions and constructive criticism for the candidate
That's seriously awesome. I would so love that. For me most of the time they just stop responding, even right after "I'll get back to you by the end of the day!" type conversations.
I don't care if it is a no, I just want to get a message, and feedback would be even better.
One company I wanted to work at recently did exactly what I described .... all hyped up meeting, we all got along, good stuff, we'll get back to you by the end of the day. Then nothing, I called a bit later, emailed, nothing.... My impression was pretty negative.
I have a job now, I'm excited to start it, that other company, very negative feelings towards that other company ... if they just sent an email even to say no I'd feel better about it, but nothing.
Perhaps I'm being too broad but I think that rejection emails suck because generally, as a society, we aren't valuing communication or we don't think we can afford the time for actual feedback.
It triggers one of those "how much better would the world be" feelings, if more people took more time to give each other genuine feedback. I mean, maybe giving good feedback (for candidates that took time to apply and clearly made effort) could help people learn, it might even ultimately address unemployment, homelessness, or other root cause problems.
I understand the legal concerns - and there would be candidates who would exploit the process of genuine feedback as well - but I think it would serve to help people more than it would hurt. It does require time and resources, so organizations / institutions would have to look at it as a sort of a social benefit cost or something. But I do wonder how much good it might really do.
Actually this is in my opinion the main advantage of headhunters. They will typically call the recruiter after the interview and seek feedback. And it is easier to provide feedback to the headhunter than to the interviewee directly.
425 comments
[ 1.6 ms ] story [ 479 ms ] threadIt's certainly great as a candidate to get detailed feedback (would have really appreciated it back in the day as a co-op student), but I just wonder if the concerns over it have any merit or are overblown.
> The number one reason companies cite for not sending feedback is legal risk. Interestingly, I don’t think this is true. Companies put themselves at legal risk if they are rejecting candidates for illegitimate reasons, like race, gender, or a disability. If they send feedback which tells candidates, truthfully, that they were rejected because they didn’t get very far on the coding project, then if anything a company reduces their legal risk: they have a transparent track record of evaluating candidates based only on their skills. I recently talked with an employment lawyer about this, and he didn’t think that specific feedback on technical performance put companies at risk. So legal risk, despite being frequently cited, seems unlikely to be the real driver of policies here.
Then, an explanation that legal risk isn't the same thing as lawsuit risk:
> Even if your process isn’t biased, if you send feedback that creates the perception you’re biased, that’s enough for a costly lawsuit. So while legal risk isn’t a reason not to send detailed, honest technical feedback (as long as you’re not discriminating), it’s a very good reason not to send carelessly compiled feedback through a haphazard process (even if you’re not discriminating).
You are doing a nice thing by being detailed, but you are basically introducing nearly unbounded downside for the upside of being nice. Most companies don't find much value in this calculus. It doesn't make it the right thing to do, but it is understandable.
All of those are ways being arrogant can manifest, but they're much more actionable than 'you were arrogant' and unsurprisingly get received a lot better.
I wouldn't comment on smell - yes, that's valuable feedback a candidate really ought to hear from someone, but the risk of really angering them is too high for me to feel comfortable with it.
I was trying to get at the same thing you just said, which is that, like you, most people would become uncomfortable providing feedback on at least some of these. Meaning that you would have to turn away these candidates without any concrete feedback. Now how do you imagine they'll react when they realize most people do get feedback but they didn't? Is their reaction (which might result in bad publicity) a risk you and your company really want to take? For what gain?
What are your recommended resources for technical improvements in coding interviews?
I think it's a pretty good starting point. I also like Cracking the Coding Interview and I think there's definitely a place for timed coding challenge sites like leetcode - especially if you've been in a role where you're mostly working on larger-scale problems rather than on producing smart, working code quickly on the fly.
I understand there are a lot of substandard programmers in the marketplace. However, why is it that this specific criteria is the one the industry is so attracted to? Could it be that kids are proficient at these kind of games? Because I can tell you that in the 1980s, 1990s, and early 2000s, I wasn't being asked to write a regular expression parser under time pressure.
Others in this thread say they've done Triplebyte take-home tests, only to end up in a "go fast-fast-fast!" interview in the end.
Why is speed so important? Every popular [aA]gile methodology today is implicitly -- if not explicitly -- against such "machismo" programming. If you're pair programming, how is this ever relevant?
Whenever I see someone say leetcode, hackerrank, and Cracking the Coding Interview is the "answer" it translates to "only the young need apply" in my head.
I did your online code quiz and got sent an email about doing a 2-hour technical interview, without really knowing much about what the job I was supposed to be applying for was.
On the interview, since I didn't really want to waste 2-hours on something I didn't want to do, I asked the guy a few questions about the company only to learn he's actually a freelancer interviewer, has little direct relationship with triplebyte and doesn't really know anything about me.
I carried on for a 2-hour quick-fire interview with a guy that was obviously trying to fill in a questionnaire rather than actually gauge my ability, questions designed by people who likely have no real-world experience in the scenarios they describe ("how would you architect the amazon.com frontpage?" is not a 2-minute answer)
About 15 minutes in, I was sure that even if I had wanted the job in the first place I wouldn't have taken it; and I had forgotten about it when I got an incredibly patronising email explaining how, if I do some online code puzzles and study hard, I too can get a job. Gee, thanks.
Granted: a bored, funemployed, grumpy dev is probably not your target audience, and I'm sure this interview style works to filter out people fresh off college, but the email was definitely the most ridiculous part.
Ultimately your whole business model is competing with tech recruiters, and my recruiters will call me to give me feedback from a role application, you have reduced that human touch aspect to an email with a few links to hackerrank.
(caveats re trusting glassdoor, blah blah)
Of particular value are multiple reviews that corroborate certain details, especially in the absence of reviews providing contradictory details.
At the very least, with something like "ghosting" during the interview process, it can help set expectations and inform behavior such as follow-up. For example, if interview reports complain of ghosting, I might still apply, but I wouldn't exert as much (if any) effort in soliciting a post-interview reply.
It reminds me of a restaurant I worked at in my youth - because the owner and her daughter were so conflict averse - rather than fire someone, they would just slowly write them off the schedule. Sad.
Firing someone (without cause) leaves you open to having unemployment claims filed against you. Writing them off the schedule so they're forced to quit on their own accord negates that leverage; underemployment claims are harder for complainants to pursue/win.
Restaurant owners being the stingy type, I guarantee you this wasn't done to avoid hurting employee feelings.
What was especially frustrating to me was that up to that point, the tone of both the conversations and email exchanges was very positive and cordial. I would have expected a "Thanks, but no thanks" follow-up at least, especially considering I was an internal referral from a Sr. Mgr. But... nothing. Made my reconsider my view of that particular company.
If in 2-3 weeks you haven't heard from the recruiter, you should ping them back, if no response don't was further time and move along.
Interviews often seem like hit and miss. I would recommend training at geeksforgeeks.org just to refresh dynamic programming, etc. But beyond that you're better off applying multiple places.
It took my friend there a week to find out that I'd been rejected.
I didn't find a useful or correct way to inform all the potential candidates about that change. That happened two times and I only sent the more generic e-mail we send for rejection telling the people that had had at least one face to face to apply for other positions if they were still interested in the company.
For the easier case of filling the headcount up with someone and not wanting the rest it's easier to send a rejection e-mail, it's just not what always happens with every job opening.
After 2 full days on-site with said company, I got radio silence. Not even a quick "thanks but no" email.
In hindsight I'm glad we did this. In the years since I've had multiple people tell me the rejection was a positive turning point and the only honest feedback they'd received.
There are tons of companies that give you a generic email after you completed an IQ test, a questionary, open questions and of course the 8hours+ homework. That's just perverse.
"Try again in three months"
Why? I wouldn't do anything different.
> Why? I wouldn't do anything different.
But they will. The interview process is stochastic; exactly the same performance from you will lead to different results on different attempts.
> Getting rejected at Triplebyte was actually a pretty good investment time wise. Guess the whole thing costed me three hours in total and I got quite a list of things to improve and how. It was clear it was tailored towards the interview not just a larger generic mail.
For another perspective, here's the entire feedback I got when they rejected my application:
> This was a tough decision and one that we were on the fence about. We really appreciate you taking the time to work on the take home project. We're aware this requires a substantial time commitment and we are really grateful that you invested the time in completing it. We thought you wrote a great, very full featured regular expression matcher. It was especially impressive how much you dug into the academics behind regular languages.
> However we made the decision because we felt that while going through the project together during the interview, we didn't see the fluency of programming when adding to it that we had hoped for. While we specifically designed the take home project track to help overcome the difficulties of coding under time pressure with someone watching, we do still need to see a certain level of programming during the interview. This didn't seem to be the case here, where making changes to the project seemed to be slower and more difficult than we'd have liked.
> But they will. The interview process is stochastic; exactly the same performance from you will lead to different results on different attempts.
Yeah I thought about that. But "try x months later" really means that you didn't pass some kind of bar. If I would be one of the five people that passed there would be no problem if I applied next week again, right?
It's hard to take this at face value, as 100% of the TripleByte messaging I've seen since before I received that email focused heavily on how confident they were that their interviews were thorough, high-quality assessments.
Edit: You also need to consider that TripleByte's idea of what works is probably different from yours. Their idea is probably more along the lines of "people pass our assessment and get hired." All you really need to do to hit that (not that it's a trivial thing at all), is conduct an assessment that's similar enough to what the hiring companies are doing. And, many of us know that hiring companies frequently aren't very good at interviewing.
The bigger difference that I see is your statement that "Every few months, I check that all our recommended resources are still up-to-date, available, and free."
In some unfruitful further communication with TripleByte, I asked how I was supposed to improve, given that their feedback seemed to be "we have no qualms with your ability, but we don't think you can pass an interview". They recommended that I sign up on interviewing.io. I did that, and a year later, I was still on the wait list. (I eventually got off the waitlist when complaining about this same sequence of events on HN, and an interviewing.io employee saw the complaint. I don't think that approach scales well.)
“How to set up a test that works and then completely ruin the results by doing nonsensical things during a dumb ritual that our entire industry seems set on preserving.”
Question for Triplebyte: when’s the last time that a single hour of coding — while being watched — determined whether you’d get to keep your job? Not acquire a job, but keep it.
I’m gusssing “never.” It’s a fake ritual.
I have received take-home tests, passed via recruiters, where the recruiters literally sent me the solutions given by previous successful candidates "for reference".
If there is data to support that most people are cheaters, that’s fine. But at Matasano I believe the statistics were ~30 candidates who were invited for an on-site interview after a take home test, and ~30 happy hires.
The on-site interview was also mostly a formality; the fact that they could do the work was enough to all but guarantee an offer.
And when you reduce it to those terms, it seems ludicrous that the world should be any other way. You can either do the work or you can’t. And if you can, nothing else should matter.
There are other counter arguments: what if someone is a huge introvert and not suited to working in a team environment? Bring on the introvets, I say. You won’t regret it. The most talented coworker I’ve ever seen was also someone who had stammering problems and would barely talk. But he was very nice, and much more skilled than I was at the time.
You don’t know someone until you work with them. And if they’re a bad hire, it’ll work itself out within the first month. It’s far better to deal with a possible cheater than to miss out on a skilled, solid hire. The latter makes or breaks companies; the former are just a temporary thorn.
I agree that's in practice true for many places. But I hope companies have the good sense to be too embarrassed to admit that their inability to be consistent is the reason to reapply.
I think the traditional reason for the "try again in three months" bit is that paper-based processes make it hard to find and re-contact promising candidates who didn't happen to make the grade this time. But if any place really thinks, "We won't hire this developer now but they would be good in the future," I think a much better solution is for the hiring company to make a list of people to re-contact next time they're hiring.
https://pastebin.com/AGPyzmgU
Ultimately, that means your interviews have bias. (Even though it appears you try very, very, very hard to avoid bias.)
Honestly, I don't think interview feedback is a good idea. It just encourages gaming the system. I'd rather that feedback come through a neutral 3rd party. We just haven't set our field up to do that.
Why neutral 3rd party? Because of the above situation! The 3rd party could just say things like, "looks like this was just a bad interview. Don't read too deep into it, and keep trying." The 3rd party could also push back on the employer if the interview ran poorly.
And no, recruiters are not neutral 3rd parties.
Rejection sucks because we all like to imagine that we are good enough and the only reason we applied is because we believed we are good enough so it stinks to be told that we are not good enough.
Doesn't matter how you phrase the email, might be nice to give a feedback, but for anyone who receives it, it stings. The only difference is that some folks have a positive mindset, they get over the sting and work towards getting better. Feedback or not.
VP hated the idea and very quickly was abandoned. We got a lot of bad candidates tbh, so it was hard to tell them what they did wrong (they bombed pretty much).
I still think, done well, it provides great benefit to candidates being considered.
Thing that worked well for me, I had elaborate set of topics/knowledge I want my developers to know and be rated on, it wasn't arbitrary selection. Still, when someone bombs, it is hard to relay they did bad.
Wait, what? That means you're interviewing on average 12 people every single working day of the year. Even for someone who's job title was "Technical Recruiter" that would be a TON, let alone for someone who is a Team Lead and presumably has other duties. How is that possible?
That could easily be a full-time job for a large company.
I thought you all were better than this. Why are you asking questions about relational databases? Why not just have candidates accomplish the thing you're assessing with an actual relational database? I know you're work-sample-literate! But if your feedback emphasizes communication, doesn't that imply a lot about your process is subjective? After all, and to extend an analysis used in this blog post: it could be that the candidates couldn't effectively communicate knowledge about RDMBS's. Or, it could be that the interviewer wasn't effectively listening to what the candidate was saying.
Asking you just to do one thing in an interview risks accidentally hitting one of the ten-twenty things you do know how to do, leaving me with no proof that you can solve the other 980-ish possible problems.
Also the next few lines kinda address what you are saying:
> It also might be that they know them inside-and-out but aren’t used to answering questions on the fly, or that our question didn’t use the vocabulary they’re familiar with, or that they misheard the question and answered a different one. It made quite a difference just to phrase the feedback in a way which acknowledged all those possibilities.
No, I write code daily but I might have to distill down technical concepts once a week or so (usually not even that much) and even then if I happen to be the only one who can distill it down.
If your devs are often explaining basic stuff like 'what is a relational database?', you need to hire someone specifically to do that. It's not a good use of time especially when they can go google/wikipedia those concepts and figure it out themselves.
Some engineers really do sit in a quiet room all day writing code, but my experience has been that it is an extremely communication-centric job.
Either way: if you do work in the kind of environment where you need to work with other engs and teams frequently you do need to test communication skills, and simply coding is not always sufficient.
This is a good use of time though. You won't find this on wikipedia, obviously, and is not considered basic technical information.
This is what I want to spend my time explaining.
If you can’t communicate ideas and basic concepts to non technical people, you will both limit your career opportunities and not be able to get your ideas implemented while someone with worse ideas will.
Developers underestimate the amount of influence you can have just by being able to communicate effectively.
At some point, we need to start demanding basic technical competence from the people around software developers.
Otherwise, people will just be interrupting you all day and how much have we collectively written about that problem?
And that’s why developers don’t get ahead....
There are basically “three levers of power” in an organization - relationship, expert, and role in that order.
The developer who knows how to build relationships is the one that doesn’t get his silly bug put on blast by the QA and gets an unofficial Skype message and doesn’t get official very visible tickets when something blows up in production and gets a quick Slack message so that he can be prepared to explain it.
It’s also the different between a developer who has to submit a ticket to netops and wait three days for a VM and one that can send an email, get it set up within 30 minutes and then create the ticket as a formality.
If you want to be the go-to guy/gal that gets constantly interrupted with this sort of stuff, your time won't be respected.
Plus, you're teaching them to go research things on their own. Why is that a bad thing?
No one gets promoted by constantly telling coworkers and management to RTFM.
Great. That's exactly what I said in my original post -- hire someone specifically to do that. Problem solved. Now your junior/mids don't have to explain that. But that's not the career trajectory of every developer, let's be honest.
If someone's going to deny me a promotion for linking a wikipedia page that answers a basic question and completely ignore my technical contributions, I absolutely do not trust that place has the best interests of its developers in mind and is likely driven more by politics than anything else.
No place has the "best interest of its developers in mind". That's true for any industry. It's a lot easier to replace the on the floor factory worker (i.e. the developer) than the foreman (the architect) and they get paid accordingly.
Don't get caught up on the title, role power is the least effective type of power in an organization. If you leverage relationships and can be seen as the expert, you can easily punch above your weight.
I think it's clear we have two very different motivating factors in careers. You want to climb the corporate ladder and get power and influence, and I'm content building things.
I wanted to "build things" my entire career and was stymied by management and team leads who I couldn't convince to see my "vision", net ops and dev ops who made me go through mounds of red tape to get anything done and coworkers who had their own agendas and wanted to get noticed and get the prime projects just as much as I did. It led to a lot of frustration.
The best way to be able to "build things" the way you want is to have the influence to do so by getting people on your side through relationship building and getting people to trust your expertise.
I like to "build things" too and have no desire for management. But, the way to stand out and make more money than the average, heads down developer is to have soft skills.
Now, you're right, the technical distillation is not going to be on the level of "explain relational databases to a Joe off the street starting from square one", but it's effectively the same as the skill of "demystifying arbitrary misunderstandings and knowledge gaps other have", which is important.
I lost out a job interview in 1997 because I wasn't familiar with "state management" for websites. The guy was pretty insistent I needed to know what "state management" was. I'd already sent over a project using session management (and he'd created an account, logged in, I sent him the code, etc), but... I didn't know what 'state management' was, so I was passed over. I wasn't strong enough on the phone (and was in a different country at the time - was worrying about the long distance charges!) and... it fell apart. I was essentially a perfect fit (had had an interview before - this was second interview with someone else), but I choked on that phrase, and they passed me by...
For some jobs it may even be appropriate to ask questions that the candidate cannot answer and see the reaction. Does he admit that he does not know? How does he phrase it? Is he making things up to cover for his lack of knowledge? And so on.
Seemed pretty cut and dry to me. What about it disqualified it as a not-doxxing situation?
[1] https://dba.stackexchange.com/questions/57445/use-of-having-...
However because HAVING takes place after all the results are fetched and processed (so it can do GROUP BYs) HAVING can’t use indexes.
So while the two will give you identical results if you’re not using a GROUP BY, the HAVING version could be thousands of times slower. Or worse.
Be glad you didn't get that job, because that would have been the future of your days at work -- constantly listening to Mr. Smartypants compensate for his own sense of inadequacy, where every different opinion is treated as a personal insult or a challenge. No thanks. That personality type is infectious (not in a good way) and it damages an organization.
But at that point, I'm more interested in their personal reaction. Pondering, and/or asking "Why the fuck do you even do that!" would've been fine there to me.
Edit: shouldn't assume gender
There you go, now it's a work-sample for your senior developer.
I’ve been trying to do something about that when it’s my turn to ask interview questions. For instance, too many front end people struggle with basic data manipulation workflows. I want to virtue signal that it’s better to move this kind of logic to the backend, but I need to test them anyway.
So I create a plausible scenario, maybe this is a POC to see if it’s worth sort or grouping functionality to the backend.
* It's the same for all candidates.
* It captures facets of the work as it is done on the actual job.
* It's objectively evaluated (ideally, it has a rubric established a priori, such that results can be evaluated by someone other than the person who proctors the challenge).
It's possible to devise work-sample challenges that assess communication skills. I have friends who've done it at their companies for customer service and sales roles. I'm saying, the process described in this blog post does not appear to be that.
Why wouldn't you ask questions about relational databases? I would expect any decent dev to know the fundamentals of relational databases.
After all, it'd be unfair to judge someone on a platform they weren't familiar with - so now you gotta maintain a fleet of laptops with a really wide range of tools. And these have to sorta float outside the usual IT management system because they aren't really issued to a single person, and you gotta be online enough that people can google stuff, and you can't have hiring managers let other people use their login, that'd be bad security practice. And if you didn't confirm in advance what platform the candidate wanted to be tested on, you gotta haul three laptops to the interview. Oh, they're pretty good developer laptops and one went missing? We really ought to have people sign those out...
And even after that, you _still_ have to apply subjective measures like "were their variable names clear?" and judge them on communication - like if they see an opportunity to refactor the code for clarity, but they say they're focusing on completing the task before spending time on that.
I'm not saying it's a bad thing to do this, just that I can understand why many companies don't.
Regarding "fleets of laptops" and environments and all that jazz: these seem like unforced obstacles. Just have the candidates use their own machines. Here's a crazy idea: have them use their own offices/couches, too.
Regarding security practices... come on. We have an interview process that involves giving remote developers read/write access to an entire AWS environment. These are simple engineering problems. If they're the only thing stopping you from having a better interviewing process, and you hire regularly, just go solve the problems.
Needless to say, you can't objectively compare two candidates' progress if one of them spends half the interview trying to get their environment set up!
Perhaps I wasn't clear: My employer is a medium-sized company, and consequentially IT security is actually a complex political problem rather than a simple engineering problem.You've also made a pretty big shift in the goalposts there, from asking "why don't companies have candidates accomplish the thing they're assessing?" to asking "why don't companies have candidates accomplish the thing they're assessing, and perform remote video interviews, and have the ability to give spin up clean AWS environments and give remote semi-trusted interview candidates access to them?"
Better yet, just don't make people do that stuff in your office.
Why bother videorecording interviews at all? I'd have problems writing a line of code with someone breathing down my neck. If you did it to me on the job, I'd chase you out of the room. I feel like a lot of these problems are, like I said, unforced.
Agree to disagree about the degree of difficulty of getting clean environments to candidates. You're either serious about recruiting as an engineering problem or you're not. "Not" is fine, but then don't pretend like there's some kind of rigor in deciding which corners you're willing to cut.
I'm not making this stuff up; this is how we've been hiring people for about 10 years now, and every time I hear someone explain how challenging or untenable our process is, I keep wondering, "what am I doing wrong to make this work so well for us?"
We chose to supply a known-good laptop instead of rejecting such candidates instantly when they've spent time travelling to us etc.
I'm not saying your process is challenging for _you_ given _your_ situation; I'm sure it works very well for you! I'm saying it's challenging for _us_ given _our_ situation :)Some people involved in our hiring process don't like them, they say experienced candidates stop responding when sent take-home tests. Their theory is employed people with families don't have the time - although we don't have hard empirical evidence for this for obvious reasons.
If most of the issues revolve around having roughly the same environment for candidates, just create ready to go VM images, for example VirtualBox, and share that with them. Or use a cloud desktop VM.
All of which make it easier for someone to succeed.
I agree with you generally, but you'd better have loaner machines ready. Not every candidate currently has a working, dev-capable laptop. Economic bias. (Hell, my current laptop is only 70% functional because I can't decide if I want to repair or replace it.)
After hundreds of interviews (I have no idea; lots, over 10 years of almost continuous hiring) I've never run into a candidate that couldn't do a tech challenge remotely.
Instead, just describe it to me. Best guess it. We'll dive into that.
You write about hiring from the perspective of someone with hiring authority. TripleByte doesn't have hiring authority, or even sufficient reputation to get their candidates out of doing another technical interview at the companies to which they apply.
There are two problems you might solve:
- Joe Nerd needs a job. He knows everything about relational databases, but no interviewer has ever noticed this. His limp, effeminate handshake leaves them unimpressed.
- IBM needs a database engineer. They really want to hire someone, but they're having trouble filling the opening; their existing network of friends-of-current-employees is tapped out.
That is to say, you could try to optimize for finding people who will be good employees, and then bully companies into hiring those people, or you could try to optimize for finding people who will pass an existing hiring gauntlet, and then introduce them to companies where the magic will happen naturally.
The first approach solves the candidate's problem and would logically charge fees to the candidate. The second approach solves the company's problem and would logically charge fees to the company. TripleByte wants to get money from companies, and follows the second strategy.
But... they like to send messages as if they were following the first strategy, because that strategy solves the candidate's problem and those messages therefore attract candidates to TripleByte. I don't like this.
I find it very plausible that your experience ("acceptance rate doesn't seem to change no matter how I personally vet the candidates") is more realistic than my armchair model, but I suspect the model is intuitive to a lot of people and will go a long way toward explaining "why are you asking questions about relational databases?".
> It also might be that they know them inside-and-out but aren’t used to answering questions on the fly, or that our question didn’t use the vocabulary they’re familiar with, or that they misheard the question and answered a different one. [...] People are generally open to hearing that, one way or another, they didn’t manage to demonstrate that they understood a topic.
The author is actively acknowledging that being unable to answer a question about RDMSs doesn't mean they don't actually know anything about them.
And the point, I think, stands. An interview isn't a passive process where by some magic algorithm they determine good candidates from bad and the candidate just sits there hoping the right question will be asked. You have to actually communicate to the interviewer your knowledge and experience because they don't know.
1. They missed a failure mode: in addition to (a) lack of knowledge and (b) lack of communication skills, there is also (c) lack of interviewing skill.
2. It's possible to design an interview process that is resilient to both (b) and (c), and I figured Triplebyte, "who has just one job", would do that.
In addition, on this thread, I've tried (badly) to point out that while (b) is maybe a reasonable thing to check candidates for, it's better to do that explicitly, with an actual test of communication skill, rather than something that can easily get confounded with (a) and (c) (and all the attendant stress that situation generates!).
Thanks for the opportunity to clarify; I'm doing too many things at once today.
It's not the same thing. Browse around the various SQL tags in StackOverflow and you'll see plenty of candidates who can "accomplish" things using relational databases yet have positively no idea of how they work.
When shit hits the fan they're asking strangers to optimize their thorny queries. But a modicum of understanding of how a relational database works would have led them to a better way to do things to begin with.
Obviously if the job is highly MySQL specific and they need to know all the quirks that’s relevant.
"Here's some schema information. This complicated query meant to do X is running too slowly. Can you recommend ways to speed things up and explain your reasoning?"
It's because you can only fit so much into an already long interview (2 hours). A big chunk of that time is already spent on an exercise about reading/writing/debugging complex code. You can't fit everything in, so database stuff is moved to the non-coding section. Also, the questions aren't "guess the right answer" questions, the interviewer keeps digging with open ended questions to see how deep you can go.
> it could be that the candidates couldn't effectively communicate knowledge about RDMBS's. Or, it could be that the interviewer wasn't effectively listening to what the candidate was saying.
You could certainly get a bad interviewer, but that's a strawman here. If it's not TripleByte judging the candidate's communication skills, then it's the hiring team judging that. The suggestion was about how to give feedback about communication skills. And there are definitely stronger and weaker communicators, and it definitely makes a big difference in day to day work.
This was a common experience for me pitching my company last year:
1. Investor likes my co enough to schedule an in-person meeting
2. I meet investor in person to pitch
3. I send them a followup email
4. Radio silence
I'd read that investors like to keep you in limbo instead of passing, I just didn't realize these well-respected professionals would value someone highly enough to give them an hour of their day, but low enough to neglect all followup communication. In retrospect I don't think it's a big deal, but I felt bad about it at the time.
The best is for European academic jobs. Often there's a schedule up front for when they'll make a decision, usually a 2-stage thing like: we will shortlist candidates by Sept 15, interview in the following 3-4 weeks, and make a hiring decision by Oct 15. So if you didn't get shortlisted, you get informed early and don't have to wonder whether your resume is still under consideration or what. American universities, though, leave you guessing what their schedule is, may take months to get back to you even if they're interested, and usually don't send a rejection letter if they aren't.
I wonder, does Triplebyte have any kind of annual summary of why candidates are getting rejected?
It’s almost the same reason you stay quiet when held by police. Even something seemingly innocuous may end being used against you in the future.
Trying to find the first job is extremely stressful process. A junior person has no notion of his worth on the market. Each rejection even if only by a lack of any response ("I'm sorry, I'm afraid we are looking for a bit more experienced person" would suffice) can be like a kick on the face when you're just barely learning to walk and most likely is a burned bridge.
I've mentored my girlfriend for 3 years from almost 0 to getting her first job in a company run by a React Native core developer. She had the skill, great attitude, really solid work ethic and very analytical thinking. It'd trust her more with any task than significant number of my past and current senior coworkers. It's hard to prove and no one expects that so naturally her applications had been ignored or rejected. With each one I saw her confidence, self-esteem and enthusiasm crumb. With each positive reply/invitation she was invigorated until the next step came. I'm pretty sure for some the roller-coaster or even worse, being rejected over and over again can be a life defining experience.
Any reply is great, personal is even better. If you spend time describing what was missing from the expectations of your company (don't say "You don't know enough", say "We need someone with more knowledge") and sincerely wish the person well you can be sure they'll be grateful, remember you, work on the gaps and who knows... maybe some day become part of your team.
Please feel free to reach out if you want some example for inspiration.
Edit: Please don't do that against the policy of your company. But if there are no reasons against just ask if you could provide some feedback and resources for the rejection letter.
But at the same time requested to not give any feedback because of fear from litigation. Sad world.
It is sad indeed.
As a relatively junior person in management, it is amazing the kind of phantom fears I've been cautioned against. Some of which don't even have any legal precedent at all!
I think a lot of it is trotted out as managerial "emergency hypothesis" for why someone doesn't want to do something, and so invents some plausible legal risk to justify their decision. But, honestly, it can't be only that.
We see Most Available Excuses in product feedback ("it's too hard to use" is easier to say than "I didn't see how this would help me accomplish anything I actually care about"), in social engagements ("sorry, too busy this week"), and many other areas.
I'm a natural skeptic so I maintain a mental set of Most Available Excuses, and when I hear one I treat it as a dodge, not an answer. Why don't they want to do it? How might I make them more comfortable with me so they can explain how they really feel?
PS: my first employee is already a foreigner, so don’t lecture me on racism.
I'd be very cautious with this approach. Politeness is an extremely important social defense mechanism for most people. By ignoring the standard polite response and trying to get at the truth you're undermining a person's attempt to save face. By doing so, you're attacking their autonomy and agency as a human being.
Sure, I sometimes see it that way until I'm in the seat of a employer. This isn't a one-sided "corporations suck" because guess who's doing the suing in these cases? That's right, the potential employee. So where is the onus in this negative externality? I'd argue both sides.
I've most likely never had a person who left without a handshake with a sincere smile on his/her face and most of them expressed their gratitude.
Sometimes you have to reject a person on what you feel is a gut feeling. It's because over time you developed an intuition which is picking small details in a less conscious manner. In the end there are some reasons your intuition is shaped that way not the other and you can find something that presented within the context of your company and expectations will resonate with the candidate and he won't feel like he's been scammed.
Some would already know it and just say ‘thanks’.
But the people who argue with you that you’re wrong, or they need another try, or ‘I forgot to mention X’, or ‘you just hate me because Y’... the people who feel hopeless and you’re just ‘confirming’ their fears (even if they applied for something way outside their qualifications) or falling apart over it.
I would never want having to deal with that be a big part of my job.
If you want to keep your job, don't follow this advice. Honestly, I would love nothing more than giving junior people (all people in fact) feedback on why they didn't get the job, but people will sue at even the slightest hint of any type of potentially litigious situation. It's even worse when the person is in a position of desperation ("I can't pay rent because I can't find a job...oh what this lawyer is going to take my case on a performance basis...heck ya, let's sue those assholes!"). Most of these cases get settled because no one wants to deal with them and it's easier to just pay the problem to go away, so they're easy wins.
Seriously, most good honest HR people WANT to give rejected candidates feedback, but asking them to benevolently provide feedback at the risk of their own job won't earn you many friends.
Edit: My intuition is most companies don't provide the feedback because simply they don't see a value in the time spent or simply never really considered that as there are always other things to do than think about people you'll probably never meet again.
And I didn't say anything about this having to do with policies of companies. Regardless of whether you have policies in place or not, doing what you suggested without any counsel is a very dangerous way of losing your job. Companies will make it very easy to fire a person who costs them a lawsuit (and the subsequent negative PR).
> My intuition is most companies don't provide the feedback
Your intuition is probably right, but it doesn't mean that people who DO want to give feedback should all of the sudden ignore sound advice. Bad HR teams who don't want to give feedback will always use legality as a shield for an excuse, but that doesn't mean that good HR teams who are also taking legal advice are not interested in doing so. As many people have cited in this thread - the long term effects of giving feedback to rejected candidates (they tell their friends, they come back later, etc) are numerous, but the downside risk makes it very difficult for any HR team to make this investment.
I think the problem comes when he talks to his friend of another race/gender and that friend said "Yeah, I couldn't finish that either, but they still hired me". The company may have had a legit reason to overlook the coding project (like the second candidate had experience in some other technology), but when you tell candidate X that they didn't get the job because they didn't complete the coding exercise, but then you hire candidate Y despite him not completing it, it provides candidate X with some concrete evidence of discrimination.
Bingo. I've opted to share specific team feedback via phone and although candidate feedback was generally positive and thankful, once in awhile the reaction would be extremely negative. I now opt for the much more (emotionally) safe route.
Triplebyte is more incented to provide candidate feedback because if the candidate improves, Triplebye may be more likely to place them in the future. With companies, this incentive is less apparent.
Now you have to decide whether to fight in public with someone you didn't hire, normally bad form, or say nothing.
One time I interviewed for a position that I wanted badly. I studied and prepped for the interview, then during the interview I nailed every question. I waited a week but never heard back. After a few weeks of silence and giving up hope, I searched the company on LinkedIn, and found the person they hired for the position. It turns out he had more backend experience, which is what they were looking for. It was a painful truth, but them sending me a rejection email telling me this wouldn't have helped me at all.
9 months later, I found myself in a bad management situation at another company, thinking about looking for another gig when the company that rejected me reached out asking if I'd be willing to come back and interview again. I did and accepted the offer.
By giving good and candid feedback, they wound up saving months of searching for a new dev when they reached back out knowing I was a good fit for what they needed at that time. I was essentially a lead they'd already warmed months prior. It made me wonder why more companies don't think of this.
And those that I didn't hire, I encountered them at other companies. It was flattering to hear them say they remembered me and had a positive impression of our recruiting process, even though they were rejected.
I've always believed that the recruiting process is a great way to sell one's company. Even if the candidate isn't a good match, that candidate may recommend peers to the role if they have a positive experience with you.
I didn't get that job, but it gave me a lot of constructive advice and I ended up getting the next one I interviewed for.
I had terribly frustrating experiences interviewing. Mostly just taking a bunch of tests and interviewing two, three times, and not hearing back for months. What sticks out was a post-interview for a large company that aggressively recruited from my uni. When I asked how I could have improved the answer was, "You ask too many specific questions about the company and software platform. Be more focused on the interviews." "For example?" "It isn't appropriate to discuss how wages are adjusted according to location or salaried overtime policies or the tech stack... in an interview..." I took that one as the, "not gonna drink the Kool Aid." box being checked. Dodged a bullet there, though, seeing as her answers did not exactly inspire faith.
I've seen this being automated in enterprise recruiting systems as "Candidate Relationship Management" using terms like "silver medalist" to identify and re-engage folks who didn't quite make the cut for positions they interviewed for but may be good fits for other open positions or for future pre-vetted candidates.
I applaud you for making things more human!
A good 50% of the resumes were flat out wrong for the position or missing critical information. A standard rejection letter with stats like these and basic recommendations might be useful in the future. It might also trigger a lot of self-righteous justification though... Don’t know if I’m up to receiving another 100 re-submissions with cooked CVs.
That's seriously awesome. I would so love that. For me most of the time they just stop responding, even right after "I'll get back to you by the end of the day!" type conversations.
I don't care if it is a no, I just want to get a message, and feedback would be even better.
One company I wanted to work at recently did exactly what I described .... all hyped up meeting, we all got along, good stuff, we'll get back to you by the end of the day. Then nothing, I called a bit later, emailed, nothing.... My impression was pretty negative.
I have a job now, I'm excited to start it, that other company, very negative feelings towards that other company ... if they just sent an email even to say no I'd feel better about it, but nothing.
It triggers one of those "how much better would the world be" feelings, if more people took more time to give each other genuine feedback. I mean, maybe giving good feedback (for candidates that took time to apply and clearly made effort) could help people learn, it might even ultimately address unemployment, homelessness, or other root cause problems.
I understand the legal concerns - and there would be candidates who would exploit the process of genuine feedback as well - but I think it would serve to help people more than it would hurt. It does require time and resources, so organizations / institutions would have to look at it as a sort of a social benefit cost or something. But I do wonder how much good it might really do.