“Was this statistically significant, or are you conducting pseudoscience? Definitely pseudoscience.”
That’s all you need to know. They quantified interviewer quality in terms of offer rate. This article is about self-celebrating a hiring binge without regard to the impact on the team or company.
No it's not. The pair of charts in the post are really illustrative of how inconsistent some of the interviewers were; this post contains some valuable insights which merit further study.
Noise. They didn’t have enough data to run the analysis they wanted to run. I would guess they need 50-250 actual hires per interviewer/group to get meaningful results.
It might be noise; I wouldn't try to publish these results in an academic journal, but I'd keep digging.
The title uses the word 'superforecasters', which makes me think the author has read some of Philip Tetlock's work, and is trying to apply it to their situation, which I find admirable.
That's hard to do, considering any fixed test has a ceiling on how much it can test for, and you don't get access to rejected candidates as a control group. Although, testing current team members with the same test could offer a sort of pseudo-control group, I suppose.
They quantified phone screen effectiveness by yield of candidates who passed onsites. I don’t know if they said, but I think those were conducted by different engineers.
That’s not infallible, but they’re committed to it being an ok metric—-it’s already what determines who they’re hiring. If you think that their decisions in on-sites are sufficiently bad, you probably should give up on the idea of optimizing how phone screens are conducted--it would be wasted effort, and you'd have no idea how to make sure it worked.
You can easily imagine that these results are just noise, but it’s more difficult (though hardly impossible) to spin a story in which it’s making their process worse. I think it would require that these interviewers have a knack for selecting weaker candidates who can still pass the on-site.
Phone screens are tests of sociability. Sociability is an inherently bipolar or multipolar activity, where participants speech, body language and tone are interpreted by other participants, digested, then fed back to the original participant in the form of more speech, body language and tone. We deem those who would score high in sociability to be charming, to have charisma. We deem them so because they have the ability to raise the sociability of those around them. By dent of their mannerisms, they make others feel confident in expressing themselves, and they are willingly to let minor faux pas slide.
We would thus expect a candidate being interviewed by a charismatic interviewer to score more highly than they would in the presence of a social dolt. This state of affairs brings into question the utility of phone interviews as measures of job performance.
I don't think I've ever taken or given a phone screen where sociability was the metric being tested for. If I was far below a certain baseline of sociability I'm sure I'd fail, but beyond that I don't think phone screens were distinguishing my sociability or charisma whatsoever.
Really? I generally found that in a sequence of ~3 interviews for a position, the order will generally be
* a personal interview (i.e. the one for sociability - mostly with HR people)
* a more technical interview (generally with tech people who don't answer questions about non-technical stuff)
* an interview with stakeholders (i.e. a potential future boss, or a department head who can answer questions about long-term career development, perks, etc.. This also tests for sociability)
That sounds nice! It definitely seems like a better process than 5 interviews of coding + system design and a half assed behavioral thrown in, which is what I'm used to.
Ah yes, that's one of the worst systems I've encountered as well.
(For full disclosure, I should mention that I received an "You passed 4 out of 5 of these interviews, and got a 'maybe' on the fifth" and thus received no offer).
When I used to conduct a ton of interviews, I would explicitly call out candidates I found exceedingly charismatic in my feedback so the hiring committee could watch out for other interviewers lowering their expectations. This pattern popped up regularly when I would compare my feedback against feedback from others.
When you meet these people, it can really be its own kind of drug.
This is often a factor. I'm hiring right now and I notice that the product manager judges engineers skills inaccurately. Fortunately I'm able to point it out to him and explain why B is probably a better engineer than A
I wonder how much of this comes down to problem selection, rather than effectiveness of the interviewer? Interviewers usually stick to a couple of problems, so their predictive power might depend primarily on the problem set.
I've been going through some technical phone screens lately, and am pretty surprised by how bad the questions are that some people choose. Questions that I wouldn't think would provide any signal whatsoever for hiring
Personally speaking, our phone screens are more focused on if an "on-site" (we really need a new name) interview will just be a complete waste of time, especially for the interviewee themselves!
When I was interviewing candidates at a company that didn't have a structured interviewing process, I used to ask people to write "the world's shittiest JSON encoder." What that is is a very limited JSON encoder that could handle strings, numbers, None, dicts, and arrays, without regard to performance, on their laptop or a loaner, with full access to the internet, and a pinky promise that the candidate wouldn't just look up "how to write JSON encoder," or anything similar.
I would give the problem in stages, starting with primitive types and working up to arrays and dicts. If we finished that, we could talk about other stuff, like how to write the world's shittiest JSON encoder, adding additional types, testing, and maybe a little optimization.
I thought it was a good problem, and we got a good candidate or 2 out of it through interview loops I participated in.
There were lots of twists and turns to the problem, but it wasn't some commonly asked question like LRU cache, and it didn't demand a lot of code.
I assume you wouldn't have classified me as a "hazer," right?
No that sounds great. Definitely not hazing. Hazing is like where you get asked to solve problems and are expected to produce an algorithm that was originally someone’s PHD thesis and there’s no way you would come up with it unless you memorized it from leetcode and literally did the same problem already. Oh and you gotta also write all the code out by hand on a whiteboard while the person smirks and makes tsk tsk sounds.
The idea that anyone should be able to solve a problem like this in 45 minutes as a demonstration of their skill as a software engineer is just ludicrous. If you can solve this problem it means you have studied this or similar problems. All it means is that you have the luxury of study. This has almost nothing to do with being a software engineer.
No. Don't make candidates practice an otherwise nearly useless skill they will most likely never use on the job, when they can just look this algorithm up on the internet if they need it.
Experienced candidates shouldn't have to spend any time preparing for the technical portion of an interview - if they have to study something new to "pass" the interview, how does that reflect on their past or current abilities? All it does is show that they have time (or made time) to study for the interview. So either they are company megafan (which would be a good thing for the company) or more likely, they are after they money the company pays - so they aren't going to be the best employee.
Personally, I think these sort of approaches are the problem with the tech job market. We currently have a massive shortage in talent to do the job, yet complains seem to think we need to prove we're worth their time when in realitiy there are more companies looking to hire than developers so the competition is not with developers with other developers for jobs but with companies for talent. So that is the first part I think is fundmentally wrong. I've literally had companies tell me people spend days on their tech test, I instantly told them I wouldn't be doing that tech test and if they wanted to test me I could come in again and they could give me a production issue to fix, code review, etc but any test would need to have a reasonable time limit. They never got back to me and honestly I'm really happy they didn't.
The next part is the entire tech screening process. I remember back in the day when we used to just ask people if they could do FizzBuzz now we're looking at if they used an interface to wrap their ORM usage. That's something that can get picked up at code review and taught. Not everyone is going to code by default at the style of a company but they can learn to do the things the company wants.
Then there is the obivous, lets test people for things they aren't going to do. Google and co made this the in thing and the fact we now have books upon books just designed so people who are good at tech can pass a tech interview is a sign in itself that there is something rotten. Hell, there was one guy that took almost a year off to study for the Google exam, err I mean interview. In a world where the shortage of tech employee means they're on the eligble for a visa list in pretty much every country in the world.
I've literally had someone be failed on a test before they did far more than expected therefore didn't have time to finish the test. No, that's the person you 100% want. They exceeded expecations.
I've made pretty much every mistake in interviewing others, I've had these mistakes while I was interviewing. I've come to realise what is important is, can they do the basic level programming required? If so get them in and if they really suck let them go during the trial period. Our worst developer at the company I work for interviewed amazingly, he falls asleep during meetings.
A bunch of companies keep trying to improve the hiring in tech, the problem is, they are the problem.
"We currently have a massive shortage in talent to do the job"
no way. i just cannot believe that's true. i'm based in the U.S and i'm currently interviewing ... and spending my evenings studying to be better prepared for upcoming interviews. they all do the same thing: crazy programming puzzles.
i believe what's happening is that companies have become extremely choosy because there is a never-ending stream of candidates. they subject to insane technical interviews and reject good candidate after good candidate with the hope that maybe the next will be just a tiny bit better. And if not, well, there's more after that.
i'm convinced that in 2020 the software engineering profession is absolutely, positively supersaturated.
i'd say more, but i need to stop procrastinating and get back to the video about dynamic programming that i was taking notes on.
Yes this. If the employers are acting as if engineers are a dime a dozen, then isn't it safe to assume that that's the case? I've been trying to get interviews as well and haven't got one yet.
It must be a regional/industry thing. I've had a former manager and colleague contact me out of the blue really desperate for staff. I've never gotten more recruiting spam, it's at double or triple the usual level.
I was a new dev around 2008 and while there were software jobs around, fewer companies were willing to hire newbies. That could be playing out again.
Well, there's a slight difference. Candidates are a dime a dozen. In my opinion, from my perspective at my non-prestigious-employer, most candidates are not qualified.
It's a matter of perspective as well though, so both sides are correct. "The job" here can talk about very different engineering roles and levels, which will have a different hiring pool by market and by saturation.
From my own experience running interviews, even in the same "community's" hiring pool in the same location you can expect large differences depending on what position and skill set you're hiring for at that moment.
I find it more worrying that over years I've seen that some engineers are attempting to optimise the interviewing process on their end. Obviously instead interviews should optimise to appeal to candidates and make the process equitable, quick, and accurate, without nonsense like coding challenges that don't relate to the job.
As we go up the engineering career ladder and land on more specialised roles with the necessary experience, both from a technical and managerial perspective, the funnel definitely shrinks. I won't deny that that looks different from place to place, but it's hard to deny that it will take years still for any saturation we see to take place on the higher part of the career ladder as people specialise more as they get there.
Specifically once we started looking for engineering management and principal level roles that still bring the technical expertise we'd expect the hiring pool was definitely not saturated.
> We currently have a massive shortage in talent to do the job
This is not at all true for the entry level market. This might be case for mid-level or even seniors but the entry level market is so saturated that these tests are bound to be necessary to cull the pool in a sense.
We do not have a massive shortage of talent. I don't understand why people keep repeating this. If there were, interviews would be much easier. There wouldn't be this interviewing arms race.
I interviewed ~30 people total between 2018 and 2019, and I have to agree with you. There is no shortage of applicants. Just the shortage of qualified people.
Side note - I do NOT use whiteboard technique in my interviews. I ask the candidates questions about the stuff they mention in their resumes and go deep in the technical discussion based on their responses (or I told them to walk me through one of their recent projects be it a work project or their side project). Using that 'simple' technique, you can easily filter out so many pretenders (as I'd like to call them, who have splendid resumes but little to back things up when grilled the details about their resumes).
This is the method I like to use as an interviewer.
But asking about work projects specifically can be tricky, since you have to avoid any proprietary bits, to avoid any possible appearance of IP taint.
If they don't have open source projects to dig into, hypothetical plausible system problems (not leetcode exercises, nor puzzles) are an option.
You still have to watch out for people who know a lot of the right engineering things to say, and/or have done prep memorizing and drilling, rather than speaking from experience. (Example from one of the links in this thread, which had an example of a repetition flash card app, saying that UDP is connectionless. I don't know what good purpose would be served by having that info on a flash card for interview prep, other than to help someone who knows absolutely nothing about it to bluff their way through an interview that asks about it.)
Very similar experiences here. And I was interviewing people we've tried to pouch from best paying companies like Google, and had stream of newgrads from the top-notch schools. So I'm assuming candidates I've been dealing with were far above the average.
From what I can tell, there's a sea of people who can barely do anything with some hand-holding, then each level of knowledge, competence and experience you want to go higher, you have to slice the pool by 100. By the time you get to level where people can competently drive non-trivial projects, there's a severe shortage.
We weren't "rejecting most of candidates from Google" - I never said that. We were trying to pouch some really good people from them. The standards weren't the problem because we were hiring a lot of SWEs. Literally hundreds in few years. And companies like Amazon, Google, Facebook, etc. do have a large pool of above average pool of engineers. But it's not like just because someone works there makes them necessarily a hire. Same with schools.
Since I am getting downvoted, I am going to lay out why I think there is no talent shortage.
Jobs that have actually had talent shortages do not operate this way. They hire candidates based on degree alone. Interviews are a formality. If they are really desperate they offer to pay the training of unqualified but interested people. The idea that they would reject multiple candidates with degrees and years of experience based on interview performance is absurd. No one is rejecting math teachers with years of experience because they failed to score 100% on a pop quiz about math, covering everything from set theory, to calculus and geometry. Sitting on the hiring side of the table at a big company, I saw us reject so many qualified candidates. The most absurd rejection was when someone with 8 years of experience at a FAANG was rejected for a Senior Frontend Role because they couldn't answer one of the interviewers pet questions about differences between MYSQL and Postgres. A situation like that would not happen if there was actually a talent shortage.
Also, the idea that there are all of these unfilled jobs strikes me as fairly suspect as well. I know someone who runs a startup and noticed that they had the same job posting on their site for a year. I asked them if they were having trouble finding someone and if I could help them by reaching out to people who might be interested. He admitted that they had no intention of hiring anyone, it was simply advertising. It makes them looks successful, like they were "growing". Having been at bigger companies where we rejected qualified candidate after qualified candidate, I suspect this practice is widespread. Better to project growth than to admit you are actually in a hiring freeze.
The last point I will make is that if there was a talent shortage, the bar would stay the same or decrease. From multiple discussions with people at FAANG, they have said the bar has increased markedly in the past 3 years. Again this is not something that would happen if there was actually a talent shortage, but would happen if there was a talent surplus.
There isn't a talent shortage because the candidates you interview don't talk about their projects up to your arbitrary bar.
>I remember back in the day when we used to just ask people if they could do FizzBuzz now we're looking at if they used an interface to wrap their ORM usage. That's something that can get picked up at code review and taught. Not everyone is going to code by default at the style of a company but they can learn to do the things the company wants.
>Then there is the obivous, lets test people for things they aren't going to do. Google and co made this the in thing and the fact we now have books upon books just designed so people who are good at tech can pass a tech interview is a sign in itself that there is something rotten.
So which is it? Should interviews be related to the work at the job or not? The Google type interviews evolved precisely from the philosophy in your first paragraph I quoted: that a lot of domain knowledge can be learned easily and that it was most important to test for more fundamental knowledge and abilities as well as general problem solving skills.
Ultimately I think it's a lot easier to criticize the interview process than to come up with a good alternative, as well as establishing some basic criteria about what should be selected for and against. Not that I think Google style interviews are perfect or anything, mind.
> So which is it? Should interviews be related to the work at the job or not? The Google type interviews evolved precisely from the philosophy in your first paragraph I quoted: that a lot of domain knowledge can be learned easily and that it was most important to test for more fundamental knowledge and abilities as well as general problem solving skills.
If a company wants to test someones basic logical knowledge and fundmental programming knowledge, seems fair enough.
There is a massive difference between writing FuzzBuzz and knowing the differences and uses from all the different sort algorithms and being to write optmisied computer science code. One tests programming and logical skills to a childs level (FuzzBuzz is a kids game), the other tests computer science skills to a high education level.
If someone wants me to write a basic sorting function, then I would have no issues. But if someone expects me write an optimised quicksort function, nah.
The list of the things these Google style questions ask is big enough thath the books for these interviews is quite big. (Lots of fun to read tho, highly recommend them for people wanting to nerd out)
>Ultimately I think it's a lot easier to criticize the interview process than to come up with a good alternative, as well as establishing some basic criteria about what should be selected for and against. Not that I think Google style interviews are perfect or anything, mind.
Well, I think I did come up with a good alternative, basic interview with work trials. You want to know how good they work, not how good they interview and the two actually are different skills.
> If someone wants me to write a basic sorting function, then I would have no issues. But if someone expects me write an optimised quicksort function, nah.
Let me be a devils attorney here. Understanding efficient sorting and how quicksort works is fundamental CS knowledge. I would say the idea is if you know how quicksort works, you should be able to write the code, no? If one is okay with writing a basic sort function but not quicksort - it could mean (1) that person does not know quicksort, (2) knows quicksort but not a good enough engineer to write code for it or (3) idealistic.
That said, I personally don't believe in testing whether a candidate can write the code for quicksort. I have however found that 'general problem solving skills' typically correlate with good engineers at places where one isn't just building yet another MEAN stack app.
> Well, I think I did come up with a good alternative, basic interview with work trials.
Not practical enough. Do you expect, in the current scenario where there is a demand surplus, for candidates to happily sign up for a trial stint - that is for the "interview" to last one or three months? I personally won't be okay with it because it messes up with a persons stability, their ability to interview at multiple places.
Even if you expect this to be AFTER the onsite interviews, like a probation period, both the company and candidate will want to maximize the # of candidates staying on, which means it doesn't really solve the problem you intended to solve.
> Let me be a devils attorney here. Understanding efficient sorting and how quicksort works is fundamental CS knowledge. I would say the idea is if you know how quicksort works, you should be able to write the code, no? If one is okay with writing a basic sort function but not quicksort - it could mean (1) that person does not know quicksort, (2) knows quicksort but not a good enough engineer to write code for it or (3) idealistic.
Companies are hiring software engineer not computer scientists. If someone wrote a sort algorithm in a normal job I would expect that not to be used and a library to be used. This is also knowledge that people who studied and aced their tests in Uni forget, because no one in their right mind writes a quicksort.
> Not practical enough. Do you expect, in the current scenario where there is a demand surplus, for candidates to happily sign up for a trial stint - that is for the "interview" to last one or three months? I personally won't be okay with it because it messes up with a persons stability, their ability to interview at multiple places.
People actually sign contracts like that on a daily basis. That is literally what you sign up for. Here is the actual process: CV screening -> Tech Test -> Phone Screening -> Interview -> Trial or "probation".
And the rest of it about people looking for jobs, well that's why companies should be selling/recruiting. The whole "but people won't do that" is clearly wrong since they already do and often jump at their first offer.
> Companies are hiring software engineer not computer scientists. If someone wrote a sort algorithm in a normal job I would expect that not to be used and a library to be used. This is also knowledge that people who studied and aced their tests in Uni forget, because no one in their right mind writes a quicksort.
Typically these questions are asked to entry-level software engineers and not ones with some experience.
Again, I don't believe asking people to write quicksort provides any signal but I was arguing that there isn't any difference between asking a candidate to write a "basic sort" vs "quick sort" in an interview.
I also mentioned this is not a good question to ask if all you are hiring for is some MEAN stack app - but is an appropriate question if you do any development where some optimization is required.
> People actually sign contracts like that on a daily basis. That is literally what you sign up for. Here is the actual process: CV screening -> Tech Test -> Phone Screening -> Interview -> Trial or "probation".
Yes, typically for roles where it is difficult to vet a candidate's skills appropriately during the interview. It is also not an improvement for Software Engineering, imo. If you ask Software Engineers if they might prefer this process over the current process, I think, most would not.
> Yes, typically for roles where it is difficult to vet a candidate's skills appropriately during the interview. It is also not an improvement for Software Engineering, imo. If you ask Software Engineers if they might prefer this process over the current process, I think, most would not.
I think you would be surprised, I am sure there are lots of people who wouldn't be interested in such a methodogly. But you need to remember what the purposal is. Do you want to be interviewed and tested on your skills in an interview and then sign a job contract or would you rather do a possible 2 day tech test. You also need to remember the job market, if I have 5 companies in my job pipeline (I stop adding companies at 5) why would I want to do 5 1 day tech tests?
Yes, its certainly an improvement over the whole-day-take-home-test. I was comparing it with the leetcode style format for junior-mid level engineers and system design for senior level. I would much rather prefer that over contract thing; and also my social bubble. Maybe I am biased.
Although, I do agree that sometimes engineers ask questions that they don't expect to work on at the company. When I interview I ask questions (leetcode style because of the nature of systems involved) based on what problems we typically solve. I never go into advanced graph etc. But good understanding of basic data structures, understanding time complexity, testing is utmost important.
Sorting is fundamental. Quicksort is not. It’s a single algorithm out of many.
Quick, without looking it up, in 45 minutes, write an implementation of CC-Radix, cache-conscience radix sort. It is a sort, it deals with performance, therefore it’s fundamental, right? After all, in my last Google and Facebook interviews, the questions were all about performance.
While we’re talking about performance, let’s talk about cache. It’s a fundamental part of computing after all. I tell you that you can pick any architecture you like, except one that is four way set-associative. What percentage of candidates can answer that? 50? 25? What percentage who did not go to certain schools can answer that? Of those who could answer it, how many saw this issue ever come up?
I love hardware, but these days most things are out of my control. Either I’m running on my Mac or on some VM instance I didn’t spin up.
Our field is VAST. It is a natural tendency to be biased towards what you know. Many things can be argued to be fundamental, even writing AND, OR, NOR, and NOT gates from NAND. That’s about as fundamental as it gets. It provides zero insight into the candidate for most groups.
If companies are treating potential hires as if it's an employer's market, then isn't it safe to say that it IS an employer's market? Isn't this a sign that there is not a shortage of software developers?
There is a shortage of senior talent. For entry level I don’t think there is any shortage. And for seniors I do see much more flexibility in waiving these types of interview requirements.
Based on my experience, system design questions (in the spirit of [0]) are getting more popular/important for senior candidates. For instance, my M1/M2 interview at Facebook had only one coding round, but three system design rounds (one phone screen, two onsite), and I was explicitly told up-front that that coding round was weighed less than the others (though it still mattered and can filter you out).
Personally, I thought this was shift in focus was pretty reasonable. I didn't prepare much at all for system design and relied on my real-world experience, which feels like how an interview for a senior candidate should go.
Its a given that if you are trying to get into faang you are going to need to spend a considerable amount of time trying to game the interview process by doing as many leetcode problems and mock interviews as you can before the interview. And then still there is no guarantee, even if you ace all rounds and pass all the coding challenges.
Please don’t implement this in your hiring pipeline. What’s missing from this is an analysis of the candidates that didn’t make it past the screens of these superscreeners. What kind of bias are these superscreeners introducing into the candidate pool? Are they good predictors because they inadvertently exploit a bias in the interviewers?
While the author didn't put that data in the article, the very last bullet point claimed that this method increased diversity. I would be a bit skeptical of that claim without any evidence, but it doesn't look like they ignored that could be a problem.
And also they are optimizing for TP while ignoring the FN. Is it possible that some of the candidates they rejected could have been interviewed by other interviewers and made it to offer stage?
It probably doesn't matter to them as long as they get their positions filled, but its prolonging their hiring.
So they selected phone screeners with the same bias as the on-site screeners? Shouldn't the aim be to reduce correlation in order to increase the signal? Doing the same interview 5 times would make hiring even cheaper, but that isn't really the most important metric you should optimize for.
76 comments
[ 3.9 ms ] story [ 145 ms ] threadThat’s all you need to know. They quantified interviewer quality in terms of offer rate. This article is about self-celebrating a hiring binge without regard to the impact on the team or company.
The title uses the word 'superforecasters', which makes me think the author has read some of Philip Tetlock's work, and is trying to apply it to their situation, which I find admirable.
But I agree what's presented here is noise.
Nor did they test their system by doing blind interviews where already hired and well performing employees would go through the pipeline.
I don't know if they actually wanted to take a good hard look at their own system.
That’s not infallible, but they’re committed to it being an ok metric—-it’s already what determines who they’re hiring. If you think that their decisions in on-sites are sufficiently bad, you probably should give up on the idea of optimizing how phone screens are conducted--it would be wasted effort, and you'd have no idea how to make sure it worked.
You can easily imagine that these results are just noise, but it’s more difficult (though hardly impossible) to spin a story in which it’s making their process worse. I think it would require that these interviewers have a knack for selecting weaker candidates who can still pass the on-site.
We would thus expect a candidate being interviewed by a charismatic interviewer to score more highly than they would in the presence of a social dolt. This state of affairs brings into question the utility of phone interviews as measures of job performance.
* a personal interview (i.e. the one for sociability - mostly with HR people)
* a more technical interview (generally with tech people who don't answer questions about non-technical stuff)
* an interview with stakeholders (i.e. a potential future boss, or a department head who can answer questions about long-term career development, perks, etc.. This also tests for sociability)
(For full disclosure, I should mention that I received an "You passed 4 out of 5 of these interviews, and got a 'maybe' on the fifth" and thus received no offer).
When you meet these people, it can really be its own kind of drug.
I've been going through some technical phone screens lately, and am pretty surprised by how bad the questions are that some people choose. Questions that I wouldn't think would provide any signal whatsoever for hiring
When I was interviewing candidates at a company that didn't have a structured interviewing process, I used to ask people to write "the world's shittiest JSON encoder." What that is is a very limited JSON encoder that could handle strings, numbers, None, dicts, and arrays, without regard to performance, on their laptop or a loaner, with full access to the internet, and a pinky promise that the candidate wouldn't just look up "how to write JSON encoder," or anything similar.
I would give the problem in stages, starting with primitive types and working up to arrays and dicts. If we finished that, we could talk about other stuff, like how to write the world's shittiest JSON encoder, adding additional types, testing, and maybe a little optimization.
I thought it was a good problem, and we got a good candidate or 2 out of it through interview loops I participated in.
There were lots of twists and turns to the problem, but it wasn't some commonly asked question like LRU cache, and it didn't demand a lot of code.
I assume you wouldn't have classified me as a "hazer," right?
The idea that anyone should be able to solve a problem like this in 45 minutes as a demonstration of their skill as a software engineer is just ludicrous. If you can solve this problem it means you have studied this or similar problems. All it means is that you have the luxury of study. This has almost nothing to do with being a software engineer.
The next part is the entire tech screening process. I remember back in the day when we used to just ask people if they could do FizzBuzz now we're looking at if they used an interface to wrap their ORM usage. That's something that can get picked up at code review and taught. Not everyone is going to code by default at the style of a company but they can learn to do the things the company wants.
Then there is the obivous, lets test people for things they aren't going to do. Google and co made this the in thing and the fact we now have books upon books just designed so people who are good at tech can pass a tech interview is a sign in itself that there is something rotten. Hell, there was one guy that took almost a year off to study for the Google exam, err I mean interview. In a world where the shortage of tech employee means they're on the eligble for a visa list in pretty much every country in the world.
I've literally had someone be failed on a test before they did far more than expected therefore didn't have time to finish the test. No, that's the person you 100% want. They exceeded expecations.
I've made pretty much every mistake in interviewing others, I've had these mistakes while I was interviewing. I've come to realise what is important is, can they do the basic level programming required? If so get them in and if they really suck let them go during the trial period. Our worst developer at the company I work for interviewed amazingly, he falls asleep during meetings.
A bunch of companies keep trying to improve the hiring in tech, the problem is, they are the problem.
The problem was "graph algorithms" used to be like "C pointer indirections"--a task that indicated that you were one notch above the average.
Of course, once Google made this a metric, it lost all meaning.
If you put a condition between someone and $500K, they WILL figure out how to convince you that they passed the condition.
no way. i just cannot believe that's true. i'm based in the U.S and i'm currently interviewing ... and spending my evenings studying to be better prepared for upcoming interviews. they all do the same thing: crazy programming puzzles.
i believe what's happening is that companies have become extremely choosy because there is a never-ending stream of candidates. they subject to insane technical interviews and reject good candidate after good candidate with the hope that maybe the next will be just a tiny bit better. And if not, well, there's more after that.
i'm convinced that in 2020 the software engineering profession is absolutely, positively supersaturated.
i'd say more, but i need to stop procrastinating and get back to the video about dynamic programming that i was taking notes on.
I was a new dev around 2008 and while there were software jobs around, fewer companies were willing to hire newbies. That could be playing out again.
It's a matter of perspective as well though, so both sides are correct. "The job" here can talk about very different engineering roles and levels, which will have a different hiring pool by market and by saturation.
From my own experience running interviews, even in the same "community's" hiring pool in the same location you can expect large differences depending on what position and skill set you're hiring for at that moment.
I find it more worrying that over years I've seen that some engineers are attempting to optimise the interviewing process on their end. Obviously instead interviews should optimise to appeal to candidates and make the process equitable, quick, and accurate, without nonsense like coding challenges that don't relate to the job.
You seem to be saying there is an area of softwre engineering that is not saturated candidate-wise.
Well, don't hold us in suspense -- go ahead and tell us specifically what sort of engineering role you were referring to here!
As we go up the engineering career ladder and land on more specialised roles with the necessary experience, both from a technical and managerial perspective, the funnel definitely shrinks. I won't deny that that looks different from place to place, but it's hard to deny that it will take years still for any saturation we see to take place on the higher part of the career ladder as people specialise more as they get there.
Specifically once we started looking for engineering management and principal level roles that still bring the technical expertise we'd expect the hiring pool was definitely not saturated.
This is not at all true for the entry level market. This might be case for mid-level or even seniors but the entry level market is so saturated that these tests are bound to be necessary to cull the pool in a sense.
Side note - I do NOT use whiteboard technique in my interviews. I ask the candidates questions about the stuff they mention in their resumes and go deep in the technical discussion based on their responses (or I told them to walk me through one of their recent projects be it a work project or their side project). Using that 'simple' technique, you can easily filter out so many pretenders (as I'd like to call them, who have splendid resumes but little to back things up when grilled the details about their resumes).
But asking about work projects specifically can be tricky, since you have to avoid any proprietary bits, to avoid any possible appearance of IP taint.
If they don't have open source projects to dig into, hypothetical plausible system problems (not leetcode exercises, nor puzzles) are an option.
You still have to watch out for people who know a lot of the right engineering things to say, and/or have done prep memorizing and drilling, rather than speaking from experience. (Example from one of the links in this thread, which had an example of a repetition flash card app, saying that UDP is connectionless. I don't know what good purpose would be served by having that info on a flash card for interview prep, other than to help someone who knows absolutely nothing about it to bluff their way through an interview that asks about it.)
From what I can tell, there's a sea of people who can barely do anything with some hand-holding, then each level of knowledge, competence and experience you want to go higher, you have to slice the pool by 100. By the time you get to level where people can competently drive non-trivial projects, there's a severe shortage.
Jobs that have actually had talent shortages do not operate this way. They hire candidates based on degree alone. Interviews are a formality. If they are really desperate they offer to pay the training of unqualified but interested people. The idea that they would reject multiple candidates with degrees and years of experience based on interview performance is absurd. No one is rejecting math teachers with years of experience because they failed to score 100% on a pop quiz about math, covering everything from set theory, to calculus and geometry. Sitting on the hiring side of the table at a big company, I saw us reject so many qualified candidates. The most absurd rejection was when someone with 8 years of experience at a FAANG was rejected for a Senior Frontend Role because they couldn't answer one of the interviewers pet questions about differences between MYSQL and Postgres. A situation like that would not happen if there was actually a talent shortage.
Also, the idea that there are all of these unfilled jobs strikes me as fairly suspect as well. I know someone who runs a startup and noticed that they had the same job posting on their site for a year. I asked them if they were having trouble finding someone and if I could help them by reaching out to people who might be interested. He admitted that they had no intention of hiring anyone, it was simply advertising. It makes them looks successful, like they were "growing". Having been at bigger companies where we rejected qualified candidate after qualified candidate, I suspect this practice is widespread. Better to project growth than to admit you are actually in a hiring freeze.
The last point I will make is that if there was a talent shortage, the bar would stay the same or decrease. From multiple discussions with people at FAANG, they have said the bar has increased markedly in the past 3 years. Again this is not something that would happen if there was actually a talent shortage, but would happen if there was a talent surplus.
There isn't a talent shortage because the candidates you interview don't talk about their projects up to your arbitrary bar.
>Then there is the obivous, lets test people for things they aren't going to do. Google and co made this the in thing and the fact we now have books upon books just designed so people who are good at tech can pass a tech interview is a sign in itself that there is something rotten.
So which is it? Should interviews be related to the work at the job or not? The Google type interviews evolved precisely from the philosophy in your first paragraph I quoted: that a lot of domain knowledge can be learned easily and that it was most important to test for more fundamental knowledge and abilities as well as general problem solving skills.
Ultimately I think it's a lot easier to criticize the interview process than to come up with a good alternative, as well as establishing some basic criteria about what should be selected for and against. Not that I think Google style interviews are perfect or anything, mind.
If a company wants to test someones basic logical knowledge and fundmental programming knowledge, seems fair enough.
There is a massive difference between writing FuzzBuzz and knowing the differences and uses from all the different sort algorithms and being to write optmisied computer science code. One tests programming and logical skills to a childs level (FuzzBuzz is a kids game), the other tests computer science skills to a high education level.
If someone wants me to write a basic sorting function, then I would have no issues. But if someone expects me write an optimised quicksort function, nah.
The list of the things these Google style questions ask is big enough thath the books for these interviews is quite big. (Lots of fun to read tho, highly recommend them for people wanting to nerd out)
>Ultimately I think it's a lot easier to criticize the interview process than to come up with a good alternative, as well as establishing some basic criteria about what should be selected for and against. Not that I think Google style interviews are perfect or anything, mind.
Well, I think I did come up with a good alternative, basic interview with work trials. You want to know how good they work, not how good they interview and the two actually are different skills.
Let me be a devils attorney here. Understanding efficient sorting and how quicksort works is fundamental CS knowledge. I would say the idea is if you know how quicksort works, you should be able to write the code, no? If one is okay with writing a basic sort function but not quicksort - it could mean (1) that person does not know quicksort, (2) knows quicksort but not a good enough engineer to write code for it or (3) idealistic.
That said, I personally don't believe in testing whether a candidate can write the code for quicksort. I have however found that 'general problem solving skills' typically correlate with good engineers at places where one isn't just building yet another MEAN stack app.
> Well, I think I did come up with a good alternative, basic interview with work trials.
Not practical enough. Do you expect, in the current scenario where there is a demand surplus, for candidates to happily sign up for a trial stint - that is for the "interview" to last one or three months? I personally won't be okay with it because it messes up with a persons stability, their ability to interview at multiple places.
Even if you expect this to be AFTER the onsite interviews, like a probation period, both the company and candidate will want to maximize the # of candidates staying on, which means it doesn't really solve the problem you intended to solve.
Companies are hiring software engineer not computer scientists. If someone wrote a sort algorithm in a normal job I would expect that not to be used and a library to be used. This is also knowledge that people who studied and aced their tests in Uni forget, because no one in their right mind writes a quicksort.
> Not practical enough. Do you expect, in the current scenario where there is a demand surplus, for candidates to happily sign up for a trial stint - that is for the "interview" to last one or three months? I personally won't be okay with it because it messes up with a persons stability, their ability to interview at multiple places.
People actually sign contracts like that on a daily basis. That is literally what you sign up for. Here is the actual process: CV screening -> Tech Test -> Phone Screening -> Interview -> Trial or "probation".
And the rest of it about people looking for jobs, well that's why companies should be selling/recruiting. The whole "but people won't do that" is clearly wrong since they already do and often jump at their first offer.
Typically these questions are asked to entry-level software engineers and not ones with some experience.
Again, I don't believe asking people to write quicksort provides any signal but I was arguing that there isn't any difference between asking a candidate to write a "basic sort" vs "quick sort" in an interview.
I also mentioned this is not a good question to ask if all you are hiring for is some MEAN stack app - but is an appropriate question if you do any development where some optimization is required.
> People actually sign contracts like that on a daily basis. That is literally what you sign up for. Here is the actual process: CV screening -> Tech Test -> Phone Screening -> Interview -> Trial or "probation".
Yes, typically for roles where it is difficult to vet a candidate's skills appropriately during the interview. It is also not an improvement for Software Engineering, imo. If you ask Software Engineers if they might prefer this process over the current process, I think, most would not.
I think you would be surprised, I am sure there are lots of people who wouldn't be interested in such a methodogly. But you need to remember what the purposal is. Do you want to be interviewed and tested on your skills in an interview and then sign a job contract or would you rather do a possible 2 day tech test. You also need to remember the job market, if I have 5 companies in my job pipeline (I stop adding companies at 5) why would I want to do 5 1 day tech tests?
And 1-2 tech test is extremely common.
Although, I do agree that sometimes engineers ask questions that they don't expect to work on at the company. When I interview I ask questions (leetcode style because of the nature of systems involved) based on what problems we typically solve. I never go into advanced graph etc. But good understanding of basic data structures, understanding time complexity, testing is utmost important.
Quick, without looking it up, in 45 minutes, write an implementation of CC-Radix, cache-conscience radix sort. It is a sort, it deals with performance, therefore it’s fundamental, right? After all, in my last Google and Facebook interviews, the questions were all about performance.
While we’re talking about performance, let’s talk about cache. It’s a fundamental part of computing after all. I tell you that you can pick any architecture you like, except one that is four way set-associative. What percentage of candidates can answer that? 50? 25? What percentage who did not go to certain schools can answer that? Of those who could answer it, how many saw this issue ever come up?
I love hardware, but these days most things are out of my control. Either I’m running on my Mac or on some VM instance I didn’t spin up.
Our field is VAST. It is a natural tendency to be biased towards what you know. Many things can be argued to be fundamental, even writing AND, OR, NOR, and NOT gates from NAND. That’s about as fundamental as it gets. It provides zero insight into the candidate for most groups.
Personally, I thought this was shift in focus was pretty reasonable. I didn't prepare much at all for system design and relied on my real-world experience, which feels like how an interview for a senior candidate should go.
[0] https://github.com/donnemartin/system-design-primer
Its a given that if you are trying to get into faang you are going to need to spend a considerable amount of time trying to game the interview process by doing as many leetcode problems and mock interviews as you can before the interview. And then still there is no guarantee, even if you ace all rounds and pass all the coding challenges.
It probably doesn't matter to them as long as they get their positions filled, but its prolonging their hiring.