Another clarification: this data only applies to general programming exercises and not domain-specific knowledge. For example, if your role requires bio-informatics knowledge (or ML or AI etc.), then by all means ask about bio-informatics—even if it means asking harder questions. (We do, however, recommend keeping the content of general programming exercises as vanilla as possible, so that you don't filter out someone because they lack mastery of a subject that you don't actually intend to measure.)
,,Hard questions do filter out bad engineers, but they also filter out good engineers (that is, they have a high false-negative rate). Easy questions, in contrast, produce fewer false-negatives but more false-positives''
The philosophy at Google is that it's better to filter out 3 good engineers than to let in a bad one. The consequence of this is that it's really hard to get kicked out of Google.
The other part (whether it's more important to work on long easier questions to see how the candidate works on a large code base) is orthogonal reasoning, and that part may be true, depending on what type of engineers somebody is looking for.
For me not being afraid to get kicked out of my workplace was a good thing. Also most of my colleagues were amazing, I miss talking to them, but at the same time I don't miss what the top management has become.
The philosophy at Google is that it's better to filter out 3 good engineers than to let in a bad one.
Corollary: "If I follow (what I've heard to be) Google's hiring practices (despite not having their brand recognition or candidate pool or a comparable engineering environment to offer) -- then my company is on the way to becoming another Google!"
Sure, it's something not to be copied by a small company. But it's important to understand for good developers that they can be great and still fail the process (they just have to prepare again adn reapply)
> The philosophy at Google is that it's better to filter out 3 good engineers than to let in a bad one.
Google's problem nowadays is that they have such strict standards only for grunt developers, but not for management. In fact, it looks like that it's more like opposite, i.e. it's better letting in/promoting 3 bad managers/directors than 1 good one.
This is sadly true, but even more true for PMs, the average engineering skill required to be inside the organization went down. When Eric Schmidt was selected as the CEO, having a strong engineering background was a requirement for him, and he was great at connecting with engineers inside the organization (probably not that great with connecting with sales, they had to accept that Google was an engineering driven organization in the past, now it's a mix).
> The philosophy at Google is that it's better to filter out 3 good engineers than to let in a bad one.
Just fire the bad ones. You're going to get bad ones anyway. At larger companies you might never notice whether someone is good or bad.
From the way they structure their interviews, it seems like they'll still get plenty of bad ones - it's just they'll get bad ones that are great at algorithms, with unknown skill at everything else (like the actual work done).
Who and what decides what a bad software engineer is (note this is very different than deciding who is a good)? Is it a manager? Managers are single people so shouldn't be the sole deciding point. So you need to gather feedback, aggregate it and decide what constitutes bad. Then have that feedback reviewed by independent people to ensure that no one is being unfair. That takes time and bureaucracy.
Think it’s more a by-product than by-design. At the risk of making a blanket statement, think it’s fair to say most software engineers have average-to-poor people skills, leading them to the fallacy that “logic tests” are a good way to evaluate people. That in turn leads to a bias you describe “did you see how badly the candidate failed the test!?!”
What do you mean by "than you need to"? I'm guessing for many companies (especially the ones with this kinds of interviews), the limit is the number of hires they can do, not the number of candidates that apply... So by definition, you need to reject all but n (the number of open jobs)... Why wouldn't you reject them based on performance on interviews (as opposed to, by their CV or luck or something)?
In many cases companies don't interview all the candidates before making a decision, it's more on a rolling basis. You consider each candidate separately - you interview a candidate and then decide whether to hire him/her or not.
In plenty of cases companies reject candidates who later perform successfully at similar roles, and this is the point of the parent comment. However this is kind of a desired effect, because not hiring a right candidate has lower cost than hiring a wrong candidate:
- if you skip good candidate, because you're not sure whether they'll perform well, you just wait for more candidates to apply, it just slows down the process
- if you hire wrong candidate, the candidate joins the team, underperforms and eventually is let go, but during the time that person works for you, you don't look for the right candidate for that role, which costs you more money and time
I did seven interviews for a top US bank. I didn't pass. They needed to fill dozens of positions. Two years later they opened the gates and hired a notorious complainer ex-coworker. That's the current state of affairs.
This might be true at Amazon, Google, or Netflix. But many companies have openings that sit vacant for months, even though perfectly great candidates applied for the job.
Months? Years :). Know couple of teams in my current place (very well known company) that couldn’t hire for a year... and lost position, as business argued - if you can’t hire for a whole year you don’t really need t
Don't forget that they introduce market friction to reduce developer turnover and reduce salaries.
I suspect one of the reasons Google is so open about their process and the need to study is so that everyone follows suit. Thereby forcing people to take days off and do homework for even the most mediocre of positions, causing the switching costs of interviewing anywhere to become higher.
Interesting idea. I got an email from Facebook with invitations for the interview. Replied: “I can do it but i’m Letting you know, I will not spend a minute preparing for it. So, we can do it today”.
I never heard from this person again, which brings a question - is probability of passing it without putting personal time is so low it makes no point in even trying?
That's a nonsensical line of reasoning. Hard interviews don't reduce salaries, they increase them. Why do you think FAANG companies pay developers 300k+/year when there are so many lower-status companies who pay <100k/year and still manage to find hires? Because the latter are willing to hire candidates who have worse resumes and less impressive interview skills that the FAANG companies consider beneath them.
You might want to re-read your own post because whether 300k/year is a lot of money or not is a non sequitor. You claimed that companies ask hard interview questions to depress salaries, which is a trivially falsifiable claim when you note that companies that don't ask hard interview questions can get away with paying less money than companies that do because they are less picky about who they hire.
Specific examples of what classifies as a “hard” or “easy” interview question would be very helpful to have reference points and assess one’s own interview process.
Easy - reverse a string, determine if a string is a palindrome, reverse the digits of an integer, determine if one string is an anagram of another.
Hard - implement a subset of regex match in optimal time+space, find the operations required to turn 1 word into another word given a list of transitory words, find the median of 2 sorted arrays in optimal time, find the next permuted value.
I think those "easy" examples are too easy to get any meaningful signal from. If they struggle, they have no idea what they're doing, and if they don't, you don't really learn anything about how they work because there's not much to them.
Other people have mentioned Triplebyte using console tic tac toe as a question; that seems like a better sort of "easy" question that still lets the interviewee have a chance to show off their problem solving and factoring skills.
examples of hard (subjective), recursive algorithms, anything that requires dynamic programming, generating permutations/subsets, problems that require a "de facto" memorized algorithm such as tree/graph traversals, coloring of subsets etc.
And here's the problem. People don't agree what's hard/easy. If I ask you to find duplicate values in a nested data structure (arrays of arrays) ... that calls for recursion. I think that's easy.
I wish this comment were higher in the thread, because it's essential. I'm very interested in this finding from triple byte, but we have no guidelines for what hard and easy questions are.
I know these questions are part of triplebyte's product, so a full, repeatable study isn't in the cards. But if someone from triplebyte could just post a few examples of each, I'd be able to get a lot more out of this result.
A while ago we had a job applicant who had travelled very far and long to reach us. (literally from the other side of the world.)
So, as a courtesy we figured, why not spend a few hours extra with this applicant in the programming test. We set up a laptop with a clean Ubuntu install, devised a programming test that was quite involved. Not algorithmic hard, just more complex than what can normally be done within a 20-minute whiteboard interview. We expected it to take at least 2-3 hours. Google/Stack overflow/etc access was allowed and encouraged. "Just act as like you would normally do when solving a problem."
We spent like 2x4 hours devising this problem, based on our codebase (cutting out something somewhat easily digestible and making it able to run standalone).
It took like one hour to get productive. Explaining the problem, setting up editors, compilers, etc.
We took turns, but most of the time someone in the interview team (of two) sat next to the guy. We did give him some alone time.
This is probably nothing new in terms of interviewing techniques, but to us it was such a revelation. We learned so much more about the applicant. Perhaps it worked well with this guy because he happened to be a bit more outgoing than our typical successful applicant. We'd never felt so confident about giving someone an offer before.
I'm really looking forward towards testing out this approach with local candidates to see if we can replicate this "data gathering success".
This can be problematic, you waste a lot of time setting up the PC / explaining the problem.
Last time I was involved with this interview style it always seemed to take an hour to get setup which meant a long interview of 3-4 hours particularly if a candidate went down the wrong path.
In the end we optimised for SOLID principles with a blackbox dll that had a function that slept for 2 seconds and a calling class that had mixed responsibilities (logging and calling the dll). We started folks off with a test or two and hoped they'd inject a mock to get rid of the delay and split logging off into a separate class.
I'm not saying it was a great test but you could do something within an hour or so then maybe spend half an hour talking through what techniques they'd use for a more complicated scenarios.
If you can afford the time then more realistic testing is great and I do think you should try.
I've never been giving given a practical programming test. Nearly all are useless algorithms. Create a function that takes a integer greater than 0. Build an array equal in size to the argument. Fill the array with numbers that when summed together equal zero.
It was worded far worse than that. What exactly is that telling you about the engineer?
On the flip side I'm asked to code full fledged applications but not to spend too much time on them... okay...
Another time I was asked to code a luhn algorithm. Oh and do it while a room of people watches you on giant screen cause that's what your day to day job will look like... I failed miserably and still got the job. What?!?!?
FWIW, I regularly pass interview candidates who "fail" at certain questions. The point of those questions isn't to complete it perfectly (and sometimes we get less information from those, honestly), it's to see how you go about getting there, how you fail, and how you deal with that failure.
For instance, one of the problems I frequently ask has a structure that really encourages people to try inventing heuristics to solve the problem, even though ultimately all of those heuristics fail. Seeing how people react to "but what if your input looks like this?" questions is often very enlightening- can they rethink their approach? Do they just keep glomming on more special cases? Can they deal with someone pointing out that sort of flaw?
I got bit in the ass by this one as triplebyte itself. They asked me to make a tic tac toe game, and gave me iirc 30 minutes (less?) to do it. Except, it wasn't "build a tic tac to" game, first it was "draw a board to the console," "take user input from the console," etc a bunch of instructions in a convoluted path that perhaps another engineer would do when knowing from the outset that the goal was to build a tic tac toe game in 30 minutes, but not me.
So we'd get to a portion where I'd be writing a quick test on user input, or extrapolating something to a function, and the interviewer would say "don't worry about that, just worry about {getting the grid to print to console or whatever}."
Later on I got my feedback and they said they were disappointed with my user input tests and repeated, extractable code in the tic tac toe portion.
Triplebyte is trying to do good things in the interview space but I think they're still learning. All in all my interview with them was about as positive an experience as a harried and bad interview could be, from my perspective.
Telling people not to worry about parts of the problem is often a time management strategy. They want you to move on so you have a chance to show strength in other parts of the problem, or at least come away with the positive experience of finishing something, even though it may count against you.
I took shortcuts in my triplebyte coding porting of the interview, assuming we would iterate and expand on the solution. Instead we just moved on to algorithms and I got heavily dinged on my feedback for the shortcuts I took (things like hard-coded strings in some places to get the UI working before wiring up to real data models).
Same experience with the Tic Tac Toe, then again with the rest of the interview.
There were a lot of Googlable boilerplate questions (e.g. "what does malloc return?", "what's a bloom filter?") that, as a product engineer, never come up.
Then there were the classic Big-O notation queries that for most use cases don't come up until much later stage. It felt like the founders were classically trained in CS and over-optimizing for things that aren't practically relevant for the large majority of early/mid-stage startups.
Am I familiar with these concepts—e.g. can I go back and refresh myself when they come up?—absolutely. But often times the skills you'd want in an engineer are:
1. Knowing when to optimize
2. Knowing how to profile and identify bottlenecks
3. Familiarity with the available solutions
4. Ability to dig in and evaluate which is the right tool for the jon
This is particularly pernicious, because it's a trick question, too. On linux, malloc always returns, it will never return NULL. Even if you ask for 4 petabytes of memory on a 128mb system, malloc will hand you back a valid pointer for the memory.
If I were asked that, I would say something like "malloc attempts to allocate some memory on the heap and return a pointer to it." If pressed for more details, I would say it depends on the allocator and that we should look at the documentation for it to find out.
If that was an unacceptable answer, then I'd consider the interview a waste of time.
You failed your Triplebyte interview because you neglected an extremely important aspect of the job: communication. You made assumptions about the ask which turned out to be grossly out of tune with those of the interviewer. In the real world, engineers are often left holding the bag when other participants of the process leave out important details. It’s our job to ask questions and establish the boundaries of each problem before diving into a solution.
> engineers are often left holding the bag when other participants of the process leave out important details
That sounds like a process problem and not an engineering problem. If I've been given requirements, I'm going to trust that my project managers and stakeholders have done the due diligence to understand their request.
Also you've got to realize that different developers tackle issues in different ways. Not every engineer is going to be super talkative while they're in the mud trying to get something to work until they've hit a wall that they don't feel like they have enough information to overcome. I think expecting an engineer to sit there and talk through every aspect of their reasoning WHILE working is fundamentally counter to the way that most engineers perform their day-to-day jobs.
I had the same tic-tac-toe q, I felt that the instructions were pretty clear: “Accomplish this one thing, as simple as possible, then generalize to actually work.” To me it felt like I was getting specs to write a class and that’s what I did.
I had a pretty great experience and would recommend TripleByte.
The less talked about absurdity is that successfully passing programming interviews is a skill itself. It's especially absurd because the time I spend developing that skill is less time spent developing skills and knowledge more directly relevant to my job. Yet, programming interview skill is more relevant to progressing my career.
edit: now if you'll excuse me, I need to do some dynamic programming problems.
This is kind off strange for a non US resident to grasp. I've been employed as a programmer three times and noone tested my coding abilites what so ever on the interviews. No samples, nothing. Never heard of any collegue doing that either.
The massive FAANG-types of companies receive so many millions of job applications, they championed these types of interviews to have a more objective way to filter through all the applicants. That practice has waned somewhat at the larger companies (not totally, but it's changing), but has trickled down to smaller companies.
I received a cold call from a Lyft recruiter recently for a senior machine learning position. I asked why they were reaching out to me and the recruiter mentioned that it’s an especially hard time to locate experienced machine learning candidates, they don’t have enough applicants.
I said I was interested in interviewing but that I would only agree to a process that evaluates me based on my previous work history, and not any onsite or takehome coding projects, system design questions or whiteboard coding questions.
The recruiter said she would run it by the manager, but thought it would not be possible, and a few days later I got a rejection email.
Not really though. If your price is working in a quiet, private office with a door that shuts, or having humane treatment when you’re being interviewed, then it seems nobody would pay it.
Let’s take their word for it that they are desperate for a machine learning engineer. Then it suggests they care more about mandating workspace conditions (since financial cost even to provide thousands of workers with private offices in dense urban areas is not a realistic excuse not to do it) or trivia during interviews than about business needs.
This is such an important observation. Claims of limited talent pool don't align with reality. If demand were truly inelastic, companies would negotiate.
In some cases I do think there is a limited talent pool. Unwillingness to negotiate in those cases usually means the company would rather simply suffer along with worse business outcomes, or try to invest in a totally different area of work, than to accept they are at a negotiating disadvantage and that it may be the case that painful company culture changes have to be made (like giving some people private offices and not others).
“The market can stay irrational longer than you can stay solvent,” seems apt for this.
It's actually not as common as it sounds from reading HN. There is definitely a bias based on the fact that the big tech companies do it and startups tend to do it. Which many commenters here have experience with.
A lot of more traditional software development roles do not have much testing. Sometimes they will have a quick online timed test or a simple question on an intitial phone screen. That isn't to say that all companies do not have testing. But it's definitely the startup and big tech worlds that have the majority of it.
>This is kind off strange for a non US resident to grasp.
I've worked for two non-technical companies as a software developer and one highly technical company, and interviewed at a few Silicon Valley companies.
The difference between the interview processes is staggering; my current job's interview was two hours of conversation, no code tests, just a general assessment of "do you know what you're doing" by the hiring manager and a couple other members of the team. The highly technical company had a code assessment then the in-person interviews had zero coding.
The SV companies must have a good reason for this, but golly the amount of coding in those interviews is nuts. I'm a process over code speed kind of coder, and I've failed every SV-level test because of it; my code comes from talking to non-technical users like medical researchers and study operations managers and tossing something together in Python or a cloud service that makes their lives easier. Needless to say, I don't go over algorithm fundamentals on a regular basis, and I generally fall out after the first or second interview.
It's especially odd that interviews are so intensely focused on those couple hours since I personally don't see any dev or any resource for that matter contributing in any meaningful way in so fast a time, or even within 90 days. I'm not sure how this problem could be solved with the limited time companies can dedicate to interviews, though; maybe rely more on portfolios?
Well it started off with Microsoft in the 90's, then Google in 2000's and then it just became common for every company in SV to conduct these programming interviews.
Go start a company. Hire people without doing any sort of coding interviews. Report back in a year.
The reality is that these are extremely desirable positions with a staggering number of applicants who _do not know how to code_. Not as in “I can’t solve a dynamic programming problem without studying up on it”, but as in literally don’t know what a for loop is.
The process is far from perfect and the frustration is understandable, but it works well enough as a filter from these companies’ point of view.
I imagine technical screening is necessary, but in-person coding assessments are nonsensical. It's not like programming is a spectator sport, so why are we tested on it live? I suppose the process is self-selecting, because I personally have given up responding to SV recruiters or interviewing with those companies.
It's quite frustrating, I've submitted my portfolio of open source projects on GitHub for interviews. I specifically told the recruiters, HR personnel, hiring managers and some of the developers that the projects contain a large enough body of work to see examples of my code. These projects are quite comprehensive and not one person looked at them or mentioned them during the interviews.
Unfortunately, people in general rarely try to first understand what the candidate offers. It's more often ONLY about whether candidates uunderstand the exact way the company uses certain technology.
Knowing what good work looks like is a plus, but I wouldn't be convinced by cherry-picked successes. That doesn't tell me how much you struggled with them, how many others you failed, or how much of the work was actually yours.
I think most of your concerns could be answered fairly quickly in a conversation, though. I can look over a Github repo and check out its history and ask specific questions about changes and why they were made or why design decisions were made.
The cherry-picked successes thing is certainly a problem, though, and not just for coding. Maybe it's the candidates I've asked it but when asked "tell me about when you made a mistake in a project" they tend to answer with a strength and try to re-frame it as a weakness. "Oh, I worked too hard on this project and it made me tired" isn't a weakness. "I worked too hard on this project and that made me neglect business requirements since I was too myopic to notice" is a weakness.
Sorry if that's a tangent, it's been a pet peeve of mine since I started interviewing that not many people are humble enough or have thought enough about what their weaknesses actually are, and how that affects the success of their work.
This is also my experience interviewing for embedded systems and (hardware) test automation work at electronics manufacturers. I've never been asked to code anything. (Nor have I been asked to draw any schematics.) Just conversation.
Probing someone with very technical conversation about past technical projects is a much stronger filter to prevent unqualified candidates than passing CoderPad tests, whiteboard algorithms, etc.
The conditional probability you are hopelessly lacking software skills to do a job given that you nonetheless passed a TripleByte exam or something is quite high. Overfitting & memorization for the sake of the test is extremely common.
But it’s much, much harder to fake competence when needing to dynamically and verbally explain technical details in a conversational interview about past work experience.
I can, with quite a high level of competence, speak about projects I had no part in implementing or that don't yet exist.
It's far, far easier to fake expertise when you have more context than the person asking questions. Technical interview questions make sure that the question giver has more context than the recipient, ignoring pathological cases.
I’m sorry but you cannot do what you are claiming. It has nothing to do with whether the candidate has more context or not.
In fact, conversational interviewing like this has very little to do with any of the domain specifics of the project. The point is to recursively keep probing for deeper technical specifics, so they have to explain at finer and finer technical levels what were the tradeoffs, why exactly were certain decisions made or how were certain problems overcome.
It is precisely the situation when someone did not have to dig into the technical weeds of a project for themselves that they will not be able to fake or fast talk their way through this type of interview.
That is the number one, defining characteristic of this way of interviewing.
>I’m sorry but you cannot do what you are claiming.
But I have. Not in the specific context of a job interview, but I have absolutely convinced technical experts that my level of expertise in a field is above my actual level of expertise. Ironically, if this were a technical interview, I'd fail it, not because I lack the skills to convince technical experts of my non-existent abilities, but because you don't find me trustworthy.
>The point is to recursively keep probing for deeper technical specifics, so they have to explain at finer and finer technical levels what were the tradeoffs, why exactly were certain decisions made or how were certain problems overcome.
But without context, you can't effectively do that. I, the candidate, am in control. I can steer the conversation to avoid areas where I don't have expertise by answering all kinds of things: "investigating that was someone else's responsibility", "well we never tried anything else and our current implementation works well enough that we never needed to", etc. You can't know if those are lies or not. There are all kinds of completely valid non-technical reasons for decisions that may be completely outside of a candidate's control.
You're either forced to completely trust the candidate, or attempt to verify their authenticity during the interview, at which point you quickly venture into the land of bias and subjectivity.
> “But without context, you can't effectively do that.”
This is simply false. You don’t need context to understand if the breakdown of a technical problem into constituent trade-offs was appropriate or not — by definition that very breakdown into constituent technical details is the context.
> “investigating that was someone else's responsibility"
This just confirms to me that you are not correct in asserting people can just skate by these discussions. If someone tells me something like that, I’ll ask them what did that other person find when they investigated? If you say anything like, “I don’t know; that was their job not mine,” then you’ve lost credibility because you didn’t put that other person’s conclusions through strong skepticism until you were satisfied you knew the details well enough that you could own or support them if you had to. That is exactly the sort of thing that indicates bullshitting.
> “attempt to verify their authenticity during the interview, at which point you quickly venture into the land of bias and subjectivity.”
This is just wrong. You don’t ever “just trust” the candidate, that’s the opposite of this interview style. Further, you never “attempt” to verify authenticity.. you just do verify it, since it does not require context or domain specialty to analyze reductionist decomposition of any engineering work down into primitive constituent tasks or decisions that are universal.
This approach is far less biased or subjective than appraising “how a candidate thinks” while they solve tricky puzzles in a foreign environment with unrealistic time pressure.
>This is simply false. You don’t need context to understand if the breakdown of a technical problem into constituent trade-offs was appropriate or not — by definition that very breakdown into constituent technical details is the context.
I mean, yes you do. Context tells you which decisions are important. You're asking for the context you need to assess someone's technical competency from the person whose technical competency you're attempting to assess. They have every incentive to lie, mislead, or stretch the truth to make the context they give you highlight their skills more than the real context did. And no, you can't verify that.
>If someone tells me something like that, I’ll ask them what did that other person find when they investigated? If you say anything like, “I don’t know; that was their job not mine,” then you’ve lost credibility because you didn’t put that other person’s conclusions through strong skepticism until you were satisfied you knew the details well enough that you could own or support them if you had to. That is exactly the sort of thing that indicates bullshitting.
But now you're punishing someone for organizational things possibly beyond their control. If my job was to build a thing that made use of some blackbox algorithm, and John developed the algorithm, why should I put the algorithm under strong skepticism, perhaps that's how things work in your workplace, but there's no clear reason that mine work the same way. This is just a bias against people whose development practices aren't the same as yours.
>This approach is far less biased or subjective than appraising “how a candidate thinks” while they solve tricky puzzles in a foreign environment with unrealistic time pressure.
Except that, as I've literally just proven, you've failed to verify the authenticity of me because you've wrongly concluded that I'm not authentic. This comment thread exactly demonstrates just how you can fail to identify a good candidate with this method because if you dislike what they're saying, you'll unconsciously convert that to them being less authentic.
So again, either you take the candidate at face value in a situation where they have every incentive to lie to you, or you attempt to verify their authenticity, at which point that analysis is subject to a multitude of biases that have nothing to do with the candidates skill level.
> “I mean, yes you do. Context tells you which decisions are important.”
No, you definitely don’t need to come into it knowing about this, and even if you don’t know about this ahead of time, it won’t imply “just trusting” the candidate or being overly subjective. You will ask the candidate to explain why various decisions were important, and not stop at the top line answer but recursively probe into it, breaking it down into concepts and trade-offs that are universal in any kind of applied problem solving.
> “But now you're punishing someone for organizational things possibly beyond their control. If my job was to build a thing that made use of some blackbox algorithm, and John developed the algorithm, why should I put the algorithm under strong skepticism, perhaps that's how things work in your workplace, but there's no clear reason that mine work the same way.”
I’m sorry but this also just isn’t true. If you are describing your contributions to projects and all that keeps happening is you hit walls in your explanation where someome else did the work and you did not review that work at a high level of depth, then you’re just being misleading about your contributions at work.
Your job as an engineer in a company is to solve problems for your stakeholders, whether that means building tooling for other engineers, assisting designers with prototypes, designing algorithms for core product functionality, sales engineering for client stakeholders, etc.
It doesn’t matter how your company is structured, it doesn’t matter how the work was divided up. Your job is to know about the stakeholder problem you are solving, at a deep level, and when you represent your work to other people and you fail to offer technical depth about the trade-offs needed to solve stakeholder problems, that’s a clear mark against you as a candidate.
It’s bewildering to me that anyone would think that the way their current employer organizes assignments should reduce their burden of knowing how to represent their projects in significant technical depth. That is an always-on, never mitigated, constant responsibility for all employees anywhere. You’re not holding anything against someone if they can’t provide that in an interview... no, you’re just uncovering what they’ve lied about or embellished on a resume.
> “Except that, as I've literally just proven, you've failed to verify the authenticity of me because you've wrongly concluded that I'm not authentic.”
I see no such proof at all, and the cheeky rhetoric just makes me feel more entrenched that you are bullshitting hugely in this thread.
> “So again, either you take the candidate at face value in a situation where they have every incentive to lie to you, or you attempt to verify their authenticity, at which point that analysis is subject to a multitude of biases that have nothing to do with the candidates skill level.”
You are doing nothing but gainsaying here. You’ve made no argument that would support any of these strong conclusions, especially not any reason why this interview method faces the false dilemma between either just trusting the candidate or else succumbing to biases.
You are just asserting things, but they do not seem to be connected to or bolstered by any of the other things you’ve written.
> You are just asserting things, but they do not seem to be connected to or bolstered by any of the other things you’ve written.
Let me lay it out clearly: I asserted that I can, and have, inflated my abilities to people who have technical know how. This was in response to you stating that "I’m sorry but you cannot do what you are claiming."
So to be clear, at this point, one of two things is true:
1. You are wrong
2. I am a liar
To you, it is clear that point 2 is the true one. To most readers, this is not as obvious. Once you have decided that (2) is true and I am a liar, nothing I say can or will convince you otherwise. But you haven't decided that based on anything factual. In fact, (1) is true here. I am not lying. I can, and have, done the things I claim to have done in this case.
I'm using this to demonstrate that your ideas about such a conversational interview don't work, by pointing out that in the conversational interview that we are having right now, you've decided, based on a preconception, that what I say cannot be true! I could be the world's most successful conman, but because you're preconceptions lead you to believe that your preferred interview process is effective and is less biased than your non-preferred one, you won't accept evidence to the contrary.
>I see no such proof at all, and the cheeky rhetoric just makes me feel more entrenched that you are bullshitting hugely in this thread.
Right, and my point is you're wrong and unwilling to accept that. And that is a demonstration of you not being able to effectively figure out whether or not someone is bullshitting from a conversation with them. You've decided that I'm bullshitting because the alternative would require you to do a lot of introspection about how and why you analyze candidates the way you do. So it's easier to just say "you're bullshitting" and then not put in the effort. And that's certainly your prerogative, but its not at all a good look for your interviewing capabilities.
That you're so prone to cognitive biases that you're willing to completely write off someone's experience because it forces you to rethink something you hold dear is not a selling point of the process you espouse. It demonstrates, like I've said, that the process is prone to cognitive bias and is therefore decidedly not objective.
That is, there are two possibilities:
1. You are wrong, and you're refusal to accept that is coloring your perceptions of our interactions in such a way that you are not able to be objective about my experiences and abilities, as I claim.
2. I'm completely making everything I've said up and haven't ever been able to inflate my abilities to anyone. Your person-analytical skills are infallible and you've caught me.
I subscribe to (1), you continue to wrongly believe (2). This is expected, its why your process isn't as objective as you claim.
>It doesn’t matter how your company is structured, it doesn’t matter how the work was divided up. Your job is to know about the stakeholder problem you are solving, at a deep level, and when you represent your work to other people and you fail to offer technical depth about the trade-offs needed to solve stakeholder problems, that’s a clear mark against you as a candidate.
This is, again, your opinion of how engineering should be done. Not every engineer has the opportunity to work in a workplace where that's how things work. Are you going to write off everyone whose experience has been in a PM led environment because they haven't had the opportunity to develop using the process you prefer? If so that's again your prerogative, but you're probably filtering out a bunch of good engineers.
Are you talking about TripleByte exams from experience, or are you speculating that they are similar to other software interviewing processes? The questions they asked me covered what I think of as a staggeringly wide range of knowledge, including sysadmin stuff, detailed knowledge of four or five programming languages, POSIX semantics, high-level scalable systems architecture, and so on. It seemed to me that it would be very difficult to "memorize for the sake of the test".
Maybe we just see it differently, but I’d consider everything you’ve listed as exactly the sort of stuff that just gets memorized. I don’t even think the breadth of what you listed is that bad actually. Especially for recent grad brogrammer types with no serious life obligation time commitments, memorizing an entire program of study like that is probably only an investment of ~1 month of time, and has practically no correlation to on the job effectiveness once hired.
I’m speaking from ~10 years of experience running my team’s recruiting in a quant finance firm, where many interview requirements / tests / etc., came down from executive managers, so I got to see a wide range of performance on tests of all sorts, riddles, hardcore algo trivia, etc.
The sum total of all that leads me to believe quite strongly that the best signal to noise comes from super careful and tedious resume selection followed by conversational and behavioral interviews that recursively probe into more specific technical details.
I agree that any one of the questions could have been memorized, but it seems to me like it would take a year or two of constant memorization, like more than 8 hours a day, to get all of it, even assuming there was a place that had the relevant information conveniently formatted for memorization — which, as far as I know, there isn't. But maybe I'm just bad at memorizing things?
I note that you didn't answer my question. Have you taken the TripleByte exam or not?
I've been in technical positions in the UK and Hong Kong, and in both cases the interviews were very technical. Hours and hours of technical assessment and questions.
true that, and I wonder, based on the little info I've about employment in the US, it's so expensive to fire someone once you realize they are not a good fit?
Even in my country (Arg), that has huge amounts of protections to workers, the first 3 months are consider trial, so you can fire someone without additional cost of the payed salaries.
Most workers in the US are under the "at will" regime: they can be let go at any time in theory. In practice, companies (especially big ones) usually try to "build a case" to justify the move.
This is why this whole interview thing is so absurd. The amount of days lost by engineers to relearn obscure algorithms and training on leetcode while we could be coding for things that are actually useful.
obscure algorithms are useful.
it isn't a perfect system but its better than what most people propose as alternatives, which is to just have an ad hoc conversation.
testing whether someone is willing to prepare for a thing is a relevant work skill test too.
There seems to be an assumption in these discussions that everyone has to prepare for these things in equal amount. But that's not the case.
(The counter to that is that if people can relatively-easily (single-digit days) cram for your interview, you're still not going to be effectively screening for at-hand pre-existing familiarity/knowledge.)
There’s a vast difference between, say, preparing for a school exam and preparing for a job interview. In the former, you know in advance what material will be tested. Job interviews are more like PhD comprehensive exams, or professional licensing exams. I know a lot of working professionals who are good at their jobs, yet admit they wouldn’t be able to pass the license exam today.
It is not illegal to use IQ tests as part of a job screening (in the US). It is illegal to screen in a way that is both discriminatory and not proven to correlate with job performance, but that applies equally to both IQ tests and algorithms questions.
Obviously I do not think that is false; you can get sued for literally anything. Do I think you would lose? Depends on whether you've done the research showing that 135 is an important cut-off.
To turn this around, can you point at the law that makes IQ a protected class or whatever you're claiming it is? I'm not going to have much luck proving a negative - Russell's teapot and all that.
Go search for "Knights on a Keypad" (a formerly-common Google interview question). Trying to imagine any situation in which the solution would be useful is harder than the problem itself.
I have actually implemented memoized graph traversals for work (unlike most programmers). To begin with, I see no point in asking this when I could ask how to implement trait checking in Rust (what I had to use memoized graph traversals for), which a practitioner will find much more intuitive.
Moreover, memoized graph traversals don't get you full credit on this question. There's a dynamic programming solution, and in fact the ideal solution is one using matrix math, which is ludicrously divorced from anything most programmers would ever see.
If you're trying to find a situation where that exact solution is going to be useful, you're not thinking about the problem from the right perspective.
Handled correctly, a problem like this answers several questions:
1) Can you correctly break down a problem like this into its components parts?
2) Can you recognize the overall class of problems that this falls into?
3) Can you transform this specific problem into the more general class so that you can solve it in a known fashion?
4) Can you think about and implement the movement?
5) Can you communicate while you're doing the above?
No one cares about solving that particular problem. But the answers to the above really are relevant. Being able to map novel problems onto known solutions is absolutely a skill that any competent software engineer needs to have. "Oh, you want me to do X? That looks a lot like Y, this thing we've already solved; maybe I can just implement it in the same fashion (or re-use our existing system!)"
I'm not saying that this particular problem is a wonderful example, or that I'd use it in my own interviews. But this overall class of problems really does have a place in interviewing when it's handled well by the interviewers, and arguments against it on the basis of the specific problem being irrelevant are really rather missing the point.
If it's true that this type of problem is relevant (I doubt it), then why not ask about a specific problem you've actually had to solve?
I don't do many interviews anymore, but I used to, and I had no short supply of problems that I actually had to solve in the course of my work that I could ask about. I don't think whiteboard interviewing is great in general, but if you're going to do it you can at least try to keep it relevant.
As a matter of fact, I did have to do memoization of graph traversals at least once for work (most programmers never have to do this), and I find that problem a lot more interesting (trait matching for Rust). I could easily give a talk about that problem. As for that interview question, though? I don't always do well with time pressure, and so I can't guarantee I'd be able to answer it to your satisfaction.
Setting up the same problem in terms of trait matching for Rust is quite a bit more difficult than this toy problem when you're in a limited-duration interview setting. All of the time that you spend laying out the problem is time the candidate doesn't spend solving it, and time you spend not getting a signal. The simpler the problem setup the better, in my experience.
I've done the "ask a real problem that I've solved before" thing, and I find that it usually gets hung up on details and context around the problem to the detriment of actually solving the problem at hand. That's not to say that such a conversation isn't itself very valuable, but that's a different interview conversation, at least on my team. We find it important to maintain some level of focus just to ensure that we're covering the various signals that we'd like to get from the candidate.
I am not at all confident that I would be able to answer Knights on a Keypad to your satisfaction, due to some combination of nervousness and time pressure. I probably could if I did many hours of interviewing practice, though. Absent that, I think there would be a good chance you would reject me on the grounds that I don't know how to do memoized graph traversals, despite the fact that I'm in a small minority of programmers that have shipped memoized graph traversals to production (and in fact a few teams at Google rely on my implementation of memoized graph traversals, ironically enough).
My view is that Google's interview process seems to work because Google gets so many applicants that they can afford to randomly reject most of them. It's not because it's a good process.
Using an actual problem you solved can also work against the candidate because most problems we encounter in our daily jobs require a lot more time than the standard 45 minutes reserved for an interview (actually only 35 minutes out of 45 because there's 5 minutes of intro/icebreaker and 5 minutes reserved at the end to answer any of the candidate's questions).
So it isn't fair to ask the candidate to solve a real bug or implement a real feature in only 35 minutes unless they've seen something similar before.
This is why big companies like Google are limited to whiteboarding interviews because they need to have an interview process efficient enough to properly vet and filter >1 million applicants Google receives each year.
Personally, I think a better interview process is a pair-programming or work audition for a day. But that is not even close to matching the scale of Google.
Let's say out of a million applications, maybe only 25% are qualified. That is still almost 1000 candidates to interview per day (number of U.S. business days in 2019 is 261; 250,000 candidates / 261 business days = ~957 candidates per business day). Pair-programming or full day work audition will not be able to accommodate 957 candidates every day.
>testing whether someone is willing to prepare for a thing is a relevant work skill test too.
Is it, though? I could understand if algo questions had some relation to the work you're doing, but your comment on why they're good is independent of the actual material. If we replaced the algorithm question interview with an interview testing obscure presidential facts, would it really be a useful test to have devs take? I guess it is a relevant work skill test in that you get people willing to put the work in/game the system, but it doesn't seem to be much of (if at all) an improvement from ad hoc conversations.
Knowing the existence of, and understanding the performance characteristics of a broad range of algorithms is, to my mind, a much more useful set of working knowledge than the details of any individual algorithm. The latter can always be looked up. It is harder to find out the former and much better to say “yes, there is an algorithm such-and-such that may be applicable here, let me spend 5 minutes checking that.” In my experience, anyway.
Broadly speaking, that used to be how design patterns were often treated in interviews. Could you recognise, describe, code, discuss pros, cons and applicability for pattern X in given circumstance. Pretty easy stuff to cram unfortunately, which may be why they may have fallen out of favour in recent times as proxy measure for brains, but depending on the role in question that sort of material was often more likely to be useful in the job than a similarly broad knowledge of algorithms
So you are saying that companies are hiring people that are willing to prove that they can put "a lot of work" into preparing for interviews that have no relevance to the actual work? I can see that as a valid point, as a way to filter lazy individuals. (In the same way that a college degree is more a way to prove that you can sustain X years of learning things without dropping out).
But it will also filter out everyone that is opinionated enough to not do that stupid preparation work, and you will end up with sheep coders that will always follow the rules.
Looking at Google, Facebook etc, this might already be the case. I will even go so far to say that they prefer those type of obedient coders than the ones that ask too many questions and get too creative.
No relevance would be like when I took the GRE for grad school. I studied things that had zero relevance to the CS program I ended up attending.
Reviewing algorithms for interview prep at least has some relevance to programming. While a candidate may not use that exact algorithm in their day to day job, they are creating ad hoc algorithms all day long. With that said, I don't think time pressure, white boarding algorithms is a very good job performance predictor.
I've worked with some folks who were decent engineers who originally didn't get offers from FANG companies because they blew the interview; they later studied their ass off on leetcode and got offers. This did not make them better engineers at all.
Ultimately, its just studying for the test, very much like the ACT/SAT in high school. You can be great at taking tests but ultimately a terrible student or vice versa.
It feels very much like that to me which is why it’s strange there isn’t a Kaplan equivalent.
There’s interview cake and leetcode, but I think people would pay $2k for a class that focuses on the questions and in person whiteboard practice.
They could collect information about the interviews at the major companies and then use those to create the program. For payment could also help candidates negotiate and then take a cut of the signing bonus.
If this isn’t part of lambda school already I think it should be.
I already work at a competitive tech company, but I’d sign up for this in a second to help me stay competitive for interviews.
App Academy was working on something like this, last time I checked. I believe it's more focused on people who've already been through their program working on getting the next-step-up job from their first so I don't know if it's generally advertised/ available
I've noticed several groups that offer SE interview specific training out there now that I'm stepping through this interview process after the landscape has changed since my last position.
It's funny that after doing some searching, many proponents of this widely adopted and poorly researched interviewing methodology appear to be running businesses selling training for these interview processes... surprise, surprise.
The only way I would consider paying for a course like this is if, upon successfully completing it, I would not have to do the technical portion of the interview at companies I applied to. They would accept this cert and just do the soft skills interview.
One could argue that CS undergraduate degrees are perfectly suited for this purpose. FAANG & Co don't seem to care for that. You've successfully passed a few dozen exams at MIT or Stanford? Well, let's make the reasonable assumption that you have no clue about algorithms and data structures and start with our third whiteboarding session, that should be a way better proxy for your knowledge.
They may very well be, but is the quality of signal truly worse than that which an interviewer can extract from a few hours of your writing code on the board?
I think that’ll never happen because the incentives of the cert granting institution and the company hiring will never be aligned (or aligned enough to matter).
The major ones seem to be Outco and Interview Kickstart. But agreed, there's far fewer whiteboarding/interview process-specific bootcamps compared to web/mobile bootcamps.
Yes! I felt I was noticeably better at selling myself in interviews, which was my primary goal for going through them. I didn't feel I had enough of a support network to properly prepare otherwise. There's a lot of technical stuff, that goes by very quickly, and also equal amount of non-technical, this-is-how-recruiting-works content.
I understand that FANG have themselves come to the conclusion that brain teasers are not necessarily very predictive for engineering performance.
But CS/programming questions for CS/programming roles? That seems sensible.
You can study lots of vocabulary to achieve better results on the verbal section of the GRE or similar tests. But afterwards, you will, in fact, have better vocabulary, I submit.
I am surprised that you categorically deny that they were better engineers after studying CS/programming questions.
>You can study lots of vocabulary to achieve better results on the verbal section of the GRE or similar tests. But afterwards, you will, in fact, have better vocabulary, I submit.
Yeah, but tests like the GRE claim to predict academic performance, not vocabulary. So unless you think augmenting one's vocabulary alone will make one a substantially better student, there's still a gap there.
Oh, you'll certainly be better at solving programming puzzles type of problems, but the thing is that those questions usually don't translate well into real-life problems that you'll encounter at work later. You can't architect a good solution IRL by trying to figure our what the test author wanted you to do, like with programming puzzles. And for implementing some algorithm keeping every detail of it in head definitely helps, but it's not a significant advantage over someone who just googled all those details 10 minutes ago. So both types of tests are not testing the real dev capabilities, they test how much they prepared for the test. IMO only tasking the candidate to build some real piece of software or even better refactor some real code, with enough time and full access to google and stack overflow, and being fully able to ask other people for suggestions & help will show you the realistic picture of the candidate's future performance as a part of the dev team. Put them in a real situation and give them the real type of problems, and you'll get the real results. Everything else is like giving sudoku tests hoping to choose the best mathematician.
As someone trying to break into the field, I'm very intimidated by coding challenges. The ones on leetcode take me an afternoon and some change to get through the easy ones. Some people seem to take no issue with googling the result then moving forward but I thought that wasn't in the spirit of doing these challenges.
> You can study lots of vocabulary to achieve better results on the verbal section of the GRE or similar tests. But afterwards, you will, in fact, have better vocabulary, I submit.
Maybe for a few days after the test anyways. And then...you push out those words you never use or read to make room for actual useful stuff.
If you're FANG you kind of want to hire that guy though -- the one who sees there's a task and a set of things he can do to accomplish that task. Same thing with the ACT/SAT, colleges want good test-takers because they're going to be giving a lot of tests.
I go back and forth about how I feel about doing heavy algorithm/data-structure stuff in interviews. On the one hand, I have never once written any kind of sort algorithm or LRU cache by hand for a production system, because why would I? Pretty much every language's standard library has a fairly-optimized sort and caching thing built in, and if they don't then there's still probably a million outside libraries to do it for me.
On the other hand, I genuinely do feel theory is really important. While not knowing the minutia of a tim-sort doesn't indicate that you'll be a bad engineer, not knowing the runtime efficiency of a sort can lead to some really awful code. Not knowing when to use a hash table instead of a nested-for loop can be a sign that you don't really know what you're doing, and not knowing some rough theory on concurrency indicates that I might be stuck debugging your race conditions or deadlock.
I try to not be a complete jerk and I won't do stuff like give out an NP-complete problem (which an interviewer gave me once), nor will I ask for intimate details of how one would implement CSP, but I do tend to focus on theory-heavy questions more than my peers, but I try to give a fairly-generous amount of hints so that people don't get too stuck.
I'd be curious to see if there is any value in questions that focus on understanding of the basic ideas behind algorithms and data structures as a subject. Time complexity, sure, but what I would really want to know is, given a description of a data structure and some algorithms for manipulating it, can you identify the invariants that should be true of these rules?
That sort of thing might give a better sense of if someone has an instinct for how to verify whether the code is working. And, by extension, if they have a grasp of some concepts that go a long way toward helping a person come up with more reliable and maintainable designs.
I think the important thing here is the difference between understanding complexity and implementing a sorting algorithm.
If I were hiring a carpenter I would want to know they understand their tools and when to use them. I don't care if they don't know how to make them. But of course knowing how to make them suggests an intimate appreciation for the craft (looking your way Matthias Wandel)
The flip side would be asking a carpenter to do trigonometry problems or something... Anyway, what's most important is measure twice and cut once. That and following a blueprint and going the extra mile.
> Not knowing when to use a hash table instead of a nested-for loop
On the flip side, the O(1) look-up nature of hash tables make them the no-brainer data structure to use, at least for passing programming interviews. Perhaps more interesting test questions would be when not to use hash tables.
Fully agreed. Most questions I got when I started out as a data scientist interviewing for jobs were pretty simple: "use a hash table to count stuff, join stuff, lookup stuff". However not once did I get asked, "why do databases not exclusively use hash tables if they're so good?" That's a much more interesting question, though perhaps out of scope for a data scientist. I'd add that I've never heard that question being asked of engineers, I'd love to hear of people out there getting it though.
I actually got that exact question once, or pretty close to it. I'm paraphrasing, but the question was "Hash tables are cool, but what's a data structure they might use in a DB besides that?". This was pretty early in my career, so it stumped me, and he pointed out that a binary tree is used semi-often because of ordering.
> On the other hand, I genuinely do feel theory is really important.
My beef with this in interviewing is that a huge chunk of modern programming work is developing UIs - and user interface theory and skills are treated like some softball thing. You ask UI-related questions to interviewers and half the time you get some shrug "oh we use whatever", etc.
This depends highly on the domain. I'm the only person on my team with any level of frontend work in the past ~2 years. And when I did, I was able to consult with UX experts and designers.
I can't say that theory is more universal than frontend, though anecdotally, for me, it is. But ui isn't universal.
> On the other hand, I genuinely do feel theory is really important. While not knowing the minutia of a tim-sort doesn't indicate that you'll be a bad engineer, not knowing the runtime efficiency of a sort can lead to some really awful code. Not knowing when to use a hash table instead of a nested-for loop can be a sign that you don't really know what you're doing, and not knowing some rough theory on concurrency indicates that I might be stuck debugging your race conditions or deadlock.
Theory is important but what's more important is how someone applies the theory to actual problem solving.
Reversing a binary tree isn't testing your knowledge of theory as much as it's really just testing memorization of a very very specific application. Knowing how to reverse a binary tree or how a hash map works is pointless if the person can't identify when to use them when solving an actual higher level problem. No one is ever given a binary tree and told to reverse it in the real world, they are given a business problem that you identify can be solved efficiently by modelling it as a binary tree and reversing it.
I'd bet most of the good people who fail the "reverse a binary tree" type of questions would succeed if you give them a realistic problem to solve without forcing a very specific solution onto them. Either they will come up to the realization that the solution is to think of the problem in terms of binary tree or they won't.
And neither is a terribly bad answer either. If they recognize you gave them a problem that can be represented as a binary tree and efficiently solved by reversing it then there you go, not only did you prove they knew the "theory" but they knew how to apply it as well. If they don't recognize it you gain valuable insight into their line of thinking and they might find novel ways to represent the problem and apply other theoretical concepts that solve it efficiently (maybe more maybe less).
I don't think you're wrong in general, but in one specific case I actually can give an example where me not knowing how a hash-map worked was really bad. I was working on a very limited set of memory for a small embedded thing when I was an intern, and was using hashmaps all over the place because they're magic, and kept getting out of memory exceptions.
I learned later that hashmaps are inherently O(n) (or O(log n)), or they take a lot of memory; internal arrays can get super huge if you're not careful.
In this particular case, an ugly nested for-loop was the immediate solution, and eventually I was able to cheat a little and have hard-coded integer indexes and was able to use an array.
Anyway, your point is valid, I just figured I'd give an example where knowing the internals for a hash table would have saved me a lot of time.
> On the other hand, I genuinely do feel theory is really important. While not knowing the minutia of a tim-sort doesn't indicate that you'll be a bad engineer, not knowing the runtime efficiency of a sort can lead to some really awful code.
OK but why don’t these interviewers ever ask “internals” questions? How does a CPU works, or what the different levels of memory hierarchy are, what an interrupt request is, what is pipelining, what is SIMD, what is a GPU, etc. Or how about compiler internals, what are the different stages in a compilers, what are the different grammars and parsers, what is an AST, what is interpretation, compilation, bytecode, JIT, and how does it work? How about database internals? How is a database implemented, what is relational algebra, what is normalization, what is a data model, how do indices work? Similar questions can be asked about operating system internals, networking, floating point arithmetic, etc. In my experience, no interviewer has ever asked me these questions. And as an interviewer, I have consistently been disappointed with candidates’ inability to answer basic and relevant “internals” questions. Ironically, the algorithms and data structures questions that are commonly asked in interviews are outdated ways of thinking which do not take the underlying hardware reality into account (memory hierarchy and parallelism). It could be because a lot of these FANG(-like) “engineers” are themselves not knowledgeable to ask such questions. Or they are aware that new comp sci graduates are unlikely to really understand anything at a deep level, but all comp sci students are drilled on algorithms and data structures. If you are targeting a very specific age group (0-5 years experience) and want a standardized test then I guess asking CS201 exam questions in a job interview make sense. Companies prefer the 0-5yrs experience segment because they’re cheap, don’t have children, are easy to exploit, and they are easier to “mold” into your corporate culture.
> In my experience, no interviewer has ever asked me these questions
Because not everyone writing Java code has had a background that exposes them to compiler internals or grammars and parsers.
But everyone writing Java code should at least be a little bit arsed to figure out what common data structures do - or how to write two for loops that print out "1 2 fizz 4 buzz"...
What's more, it's funny reading that Google sends out prep packets saying to learn/relearn these algorithms. So they don't expect their recruits to know them, just be able to memorize them for the interview.
This resonates strongly with me. I know a few engineers who are bad to mediocre software developers, but excellent interviewees. That skill alone lands them offers at any place they want to work. They can whiteboard algorithms like there is no tomorrow, but they can't manage software complexity.
Managing complexity is a valuable skill that should also be screened for at interview. At most top companies you are there for at least 4-5 hours so there should be plenty of time to evaluate that skill.
I think whiteboard questions are good, I want to know that this person is capable of writing difficult code if we need them to.
I also think we probably ask too many of them.
I have been on the hiring side and my experience so far has been that almost always the feedback is close to identical across multiple whiteboard questions. The questions are also so abstract that asking multiple to "prevent bias" seems ineffectual. What bias could there be, you either solve the problem or you don't. Bias is more likely to come in on the behavioral interviews. There should be multiple of those for sure.
Generally someone is either a good enough coder or not and they will display that consistently across all the interviews. You will see the same stuff throughout (good or bad variable naming, good or bad communication etc) thus asking > 1 coding question by default is a waste of everyone's time.
It's not a false dichotomy because I am not suggesting that both skills are mutually exclusive. I'm highlighting some personal experiences where I've seen one skill is vastly overvalued compared to another.
>Managing complexity is a valuable skill that should also be screened for at interview
I've never been part of an interview, on either side, where I've seen testing for managing software complexity. Good interfaces, function design, side effects, state management, etc, are all second class citizens to finding the appropriate algorithm to solve the interview question. I've seen it over and over. The only time I've been close to a complexity management question was on a systems design question, but even then, it was only a very high-level systems discussion. I don't think most places know how to screen for it.
I've interviewed tons of engineers that ace the interview and you have no choice but to pass but you see later on writing abysmal code. The problem is that design, quality, and maintainability are not valued. As a result, you end up with extremely mediocre "hacky" engineers at larger companies that can spit out tons of valid code but lead to a tangled mess shortly thereafter.
Worse, there's this huge emphasis on "big O" with zero focus on clearly egregious bad practices (tons of copies, outrageous memory usage, casting between strings and numbers all the time). I've rarely seen clearly bad algorithms be deployed but I have seen plenty of unperformant code go out.
I had a coding test once where I was given someone else's code with an introduced bug, and I had to fix the bug. To do so, you were basically documenting the code along the way as you traced your way through possible trouble spots. It felt far more meaningful than the typical "implement some list traversal algorithm that you won't actually ever use at this job."
I've had a similar interview and although I did not solve the bug, I think the interview was much more effective than most algorithm based interviews. Not only did my interviewers get to see how I navigate a codebase and use my IDE/other tools, they also saw how I approached known unknowns and discovered information.
I myself too have witnessed a similar thing but with not-so-algorithmic-people. I think it all comes down to the company culture and the person actually being able to let go of their ego and improve their programming skills. It is similar to a senior developer who never learned to properly program and because of hir position is incapable of seeing faults in their code. Once you've gotten too accustomed to shitty code, it's hard to change your style which is understandable.
Probably biggest influence on this I think is the culture, if nobody tells you that your code is shit (in a kind way) you'll never learn better. But then again, some people are so hard-headed and full of themselves that they get defensive and never actually admit their faults. Though giving and receiving feedback is not easy, can give you an identity crisis once you realize you've been doing something wrong for years.
And then you have a bunch of Ivy League graduates who have spent years learning algorithms and are burning to use them but there's no actual problems that really need them.
Note that one often-cited counterargument to its seeming "absurdity" is that it evens the playing field in a fairer, more meritocratic way.
No matter your background, what school you went to, or what randomized experience you got in previous jobs, every person has equal opportunity to study and practice the same algorithms on their own (as opposed to being lucky enough to be able to afford a top-tier $$$ CS education, or to being lucky enough to have the connections or chance to get certain previous jobs).
And thus, when applying to jobs, it becomes something more akin to a raw-ability IQ test, which you can argue is "fairer", especially when management realistically knows developers might be shuffled around all the time, and that the extensive SQL experience they were hired for will mean nothing when project requirements switch to a basic key-value store.
On the other hand, if you are interviewing for a highly specialized position that is fairly certain not to undergo change, then it makes sense that specialized experience could rightly count for far more than any kind of generalized intelligence or ability.
> every person has equal opportunity to study and practice the same algorithms on their own
Well, that's just not the cause - there are many groups of people who lack the opportunity to study and practice. Couple of examples: people with kids, people working 12 hour shifts, people without access to teaching materials, people without a sufficiently advanced machine to run dev environments, etc.
It's "fairer" if you're a recent college grad, probably single, definitely no kids or other family obligations, fresh out of school, and without much to lose if you spend all your free time studying for a "fair, meritocratic" test.
Interviewing is a skill regardless of what field you are in. I agree that algorithm are not optimal for testing engineering skills, but there's a whiff of entitlement when engineers get indignant about having to spend time preparing for an interview. Objective evaluation methods for evaluating candidates are hard to come by, and I don't see obviously better alternatives.
I'd suggest kattis and CodeForces. I use kattis mostly, but know a number of people who like codeforces better.
Here's a few for you to try. Some of these are pretty hard, but you should be able to find solution sketches online if you google the contests they are from.
I don't give questions like this primarily because:
- Workers are going to be around for 15 months or less and they have domain expertise on 1 stack already and I don't need to screen for how they would hypothetically function across all stacks
- Worker's process and resource finding skills are more indicative of the time they will spend on a task
- Worker's process includes collaborate use of version control and code reviews, if they pass the screening but can't really integrate on these things then thats what will get them booted from the team
It isn't always more expensive to have a not great developer. Look in your organization and see if what I experience is true for you, and you'll save everyone a lot of time.
I think that asking a candidate to perform a code review can be an effective method of evaluating quite a few desirable qualities. Can they understand someone else's code? Can they engage in constructive critical discussion? Are they able to effectively refactor something to make it better? Can they spot mistakes and do they have an opinion about how to avoid such mistakes?
I know you're joking but... If i had never heard of a binary tree, saw it for the first time in my interview, and didn't at least have an idea of how to attempt solving it, I'm probably a lousy hire. The quality of attempt is what these hard questions are really about.
Looks like I am gonna have to look at the solution as I don't fully get how to achieve that structure. Recursion is good for a few standard problems but those are just standard problems. We shouldn't try doing every possible thing using recursion, even if it is possible to do so. Also, debugging an iterative solutions comes naturally to me compared to recursion. I guess I don't have the right mental model for how to track recursion/recursive calls.
There's a classical test of giving a never seen before programming language (invented for the test), then the candidates need to reason about some code. Never seen that in the industry though.
That would be a rather interesting challenge for an interview; I'd love to be able to try it someday.
I recently did something similar, that was of my own devising.
At one point several years back I wrote down some "code" in my own "shorthand" form; it was meant to implement a library and some test code for a microcontroller project I was contemplating at the time. I basically wrote it in such a fashion so that I was quick to get my main ideas down without being too "wordy" (whether in code or otherwise).
Then I put it away, and didn't revisit it again - until recently.
A couple of weeks back I found that code again, and looked at it - worried that I wouldn't be able to recall my shorthand or what I was thinking; in short, worried that my ideas would be "lost".
I looked over the code, walked thru it in my mind - and after a few minutes it all came back to me, and I was able to understand again what I had originally created (and where I could make improvements as well). It both left me feeling optimistic about the process, as well as a bit excited that I was able to remember it and improve on it - that I didn't have to worry about it, and that my shorthand "pseudo-code" was legitimate enough that it could have been real code for all that it mattered.
Yeah that's what we do. As a developer you spend a large amount of time reading/reviewing existing code and making changes to it. I'm surprised there aren't more companies that test for that skill.
i did something similar with junior candidates a few years ago. put them in front of a piece of code and ask them to fix the problem. not a code review, but debugging. that covers many similar qualities.
i didn't expect critical discussion because they were not native english speakers. (in fact they hadn't had an opportunity to even use english outside of talking to their teacher, so when they were able to engage in friendly arguments over how to solve a problem, that was quite an accomplishment, even more so for asian culture which is generally rather submissive (you don't argue with your boss))
A few jobs back I was tasked with hiring new developers to bolster a thin front-end team. The job was very CSS/JavaScript heavy, so I asked questions that were pertinent to what the candidate would be doing if hired. Of the five candidates, only one answered all the questions perfectly, and he turned out to be the biggest bust for us.
The other candidates, after answering some of the harder questions incorrectly, seemed very upset with themselves. They knew they were cracking a bit under pressure, but actually showed that they knew the answers when we chatted further. I hired 3/4 of those people because of how well I felt they'd do given the opportunity. All three became leads within a year and a half.
I think personality has a lot to do with outcomes. If you are someone who shows they are hungry to learn and knows how to improve their skills, I will never dismiss you for screwing up a few coding questions.
Personality has a huge amount to do with outcomes. Especially in teams, where the best teams have a mix of complementary talents.
If you're hiring for Generic Developer Skills you're going to get generic developers - and much less development than a more flexible approach would give you.
One interview of mine asked something i didn't know yet I said there's no doubt I figured it out via Google. The interview pretty much ended there and I'm glad it did! Any place or interviewer that says you shouldn't use Google of OverFlow to get your work done is no place I want to work for.
But, that interviewer is the type of developer I don't want to work with. A know it all who only acts like he knows it all and uses Google secretly. Who puts others down to make themselves look good. AKA an insecure P word I want nothing to do with.
Another solution to this problem is contract to hire. I realize this is kicking the can down the road to the contracting firm but hear me out: that's the business the contracting firm is in. They can get really good at their hiring practice since that's their core business. That's not our core business. We've been doing this for the past two years and it's worked out great. Now you can see how well people do the actual job and if you're not satisfied, which happens from time to time, just get someone else.
How does this help? You still have to find people to contract and so you still have to screen the contractors. Every one that fails you was a huge waste of your time. You've got to spend time telling them what you want them to do, explaining to them your systems, your APIs, training them to your way of doing things or whatever and if they fail than all your time is in the toilet.
The only advantage I can see is you can get rid of them easily by not renewing their contract but that in itself is not a solution to finding good people in the first place. Plus it's likely to be hugely limiting. I'm not going to give up my current job and move to your city for a contract you might cancel in 3 months.
Why would I leave my W2 job for your contract to hire position? All I can see here is a company that doesn’t want to fire people who aren’t working out, possibly to save on benefits and unemployment insurance premiums. And why do you have so many people not working out that you have to do this? None of this inspires any confidence.
Yeah, in my experience asking the basic question of why I would even entertain moving from my W2 job to a contract position where I assume 100% of the risk is met with dead air on the other end, followed by a hasty assurance that we don't really have to go the contract route.
It gives me the feeling that I will be working with people who either don't know their own value and so are willing to agree to this kind of deal, or are so poor performing they don't have the leverage to insist on a regular employment contract.
I'm glad it works out for you. But I would not want to work at a company where a significant number of my coworkers only last a couple months, and I think I'm far from alone in that perspective.
This is dangerous because it will just create a shadow workforce of contractors without any health benefits or rights as an employee. We already have this issue in tech.
Many companies are "contract to hire" but never hire and keep contractors on as long term employees.
Based off some of the responses here I just want to clarify some things in order to make this work.
- Yes, you have to have vetted your contracting companies. In our case that's something we'd already done (roughly 20%-30% of our staff are contractors)
- This is contact to hire, which is different than bringing on contractors for staff augmentation. The intent is to hire and the applicant is made aware of that up-front, to the extent that their employee salary is negotiated and employee benefits package is discussed.
- W2 employees jump at the chance to be given an opportunity to prove themselves in a new domain. Younger and older employees are especially attracted to this option.
- You're still doing an interview, but the nature of the interview is different since you know the contracting firm has vetted their basic skills.
- So far we've only had to let one person go, they simply never could "gel" with the team. We've had two others we had to let go as we were scammed - the person that did the interview wasn't the person that showed up to work. Yes, it happens. What we don't have is a "revolving door" situation where people are continually rotating in-and-out.
Now, the ads ask simpler things like floating point precision and function variable scoping (https://www.facebook.com/triplebyte/ads/?ref=page_internal ); legit problems, but not sure if they are an indicator of how good a developer they'd be in the real world working on a CRUD app.
You nailed it. For someone who needs to determine the endianness of a machine on the fly, the question is trivial to solve; for everyone else, it’s trivial in the sense of “unimportant.”
Their pre-screening quiz uses a bunch of questions on various technologies to ensure that some subset of the questions tests you on technologies you're familiar with. Which is good, since not everyone is a web engineer.
I don't think there would be too much correlation between CS degree and ability to answer that question. There are lots of CS degree programs that don't deliver such low level instruction, and there are plenty of people without CS degrees who can answer that question because they were curious about how computers work or have been exposed to low level programming.
The bigger problem is the inconsistency amongst interviewers when judging candidates. All these articles from TripleByte and Gayle (who've built business on the flaws of interviewing) focus on the questions instead. Doesn't matter how hard the question is if the interviewer knows what they're looking for, are experienced enough, show no nepotism and have good communication skills.
My worst interview ever was with Facebook when a non-native, new college grad gave me a Leetcode hard problem in half-broken english and went back to his work without even looking up or walking with me through the problem.
This article makes a lot of good points. After going through the "implement a red-black tree on this whiteboard" experience as a more junior dev, I always promised myself I would never use this kind of stupid questions to hire.
Now, 13 years later, I mostly rely on "homework" type exercises. I think they address most of the issues. They are more "real world", no time pressure, etc. However, even those now are being heavily criticized. What's left to be used?
There is a problem with homework, candidate spends couple of hours and you spend couple of minutes to assess candidates. It's not fair. Google doesn't do that, Netflix doesn't do that. Why anyone would believe random startup and spend their weekend unless you're desperate looking for any job?
It is both true and false. Yes, the candidate may spend 1 hour (I don't give difficult homework exercises) while it takes me 15 to 20 minutes to review the submission, but I have to do it for maybe 5 candidates.
I totally understand the main criticism for homework, it takes time, the company may never call you back after you poured 2 hours into their stupid exercise. But it is an attempt at fixing all other alternatives:
* the onsite whiteboarding is BS
* using open source contribution is totally unfair to candidates who don't participate
* the "contract for 1 month and then maybe we'll hire you" is also total BS in my book. Who would leave a FT job with benefits for a contract that may end?
I'm not sure I can think of any alternative that has 0 drawbacks.
and at the end it turns out to be 5-10 hours. Companies say "you should be able to finish it in 1-2 hours", but it's almost never true.
I think alternative is to have 1 hour coding session on a good problem. Most problems are complicated, but there could be something else. I was once asked by Uber to implement timer in JavaScript that will update DOM, also create APIs to stop/start/pause. This kind of a challenge doesn't involve any algorithm, but it shows your ability to code.
It should be enough to bring you onsite.
But I open source should work. If you contributed to Linux Kernel and it was accepted - onsite no questions asked. If you have github repo of 1000+ stars - onsite no questions asked. If you have already passed Google tech screen and was invited onsite - invite onsite no questions asked.
Homework is a bit of a catch-22. I agree that they can be made more real world than typical interview problems (although not all are). OTOH, why should I do your “4 hour” project, in which I’m competing with people who spend 8+ hours on it, just to get to the same onsite interview I could get with another company after a recruiter chat and a 45-60 minute technical phone screen? It’s not a good use of my time.
Just to clarify first, the exercise should not take more than 1 hour. And I mean it. It is not difficult, or tricky or anything. It is used 1) to make sure the candidate can do very elementary things, 2) as support for follow up during the on-side interview.
Regarding "getting the on site", if you don't do the exercise before you would get the "whiteboard" BS. And then you may be competing with people who spent 2 weeks reading "cracking the tech interview". How is that different?
If your exercise truly takes 1 hour, and you're evaluating it on the basis of 1 hours' worth of effort, then fair enough. I would say that roughly equates to the technical phone screen in this case, except without all the annoying "let's talk about every little thing we're doing while we're doing it, so it takes twice as long" BS.
And there's nothing wrong with whiteboarding, if done appropriately. Whiteboarding a system design question is totally appropriate. Asking a candidate to write down code for finding the longest palindromic substring of an input string in linear time is not. I've never come to a situation where writing anything more than the barest pseudocode outline for what to do on a whiteboard was the best way to get a real world task accomplished.
It seems like this article would be stronger with some examples. What are some examples of good and overly-difficult questions, according to TripleByte?
I've seen this borne out in practice after administering dozens of phone screens and in-person interviews over the years. I started off with questions that were a little too hard and couldn't get a good read of the candidate unless they happened to have already done the problem on some interview prep site.
Switching to more practical, simpler problems allowed me to really observe how they work and solve a coding problem. As the article said, I was also able to add requirements or features to the problem which let me see how the candidate adapts to changing requirements, or refactors their own solution to handle a new edge case. Simpler is generally better if you are timeboxed to 45 minutes.
I wonder if hard questions, especially hard algorithm questions, has high false positive rate as well. Being good at solving those questions simply requires you to do tons of practice on leetcode, and those who are willing to spend tons of time are probably those who find it difficult to get jobs (or new grads). I don't think the skills for solving hard algorithm questions correlate well with the actual performance at work. Especially if someone is a new grad, they could be good at thise questions but don't have a clue about how real software systems are built.
too hard?!
please...
just becuse it may have been to hard for you dosen't mean all programers are idiots/...
at least an example of what is considerd hard should'v been given....
you wanna know how to tell if a programer is good or bad in thir job...
give them questions from the eular project...
and start from page 6 or so... from questions 200-300-to start with..
after he answerd it all correctly and...
he should be able to solve at least 1 problem from page 13... only then hire the man....
just because you may be stupid/... find somthing too hard dosen't mean it is too hard!
man...
539 comments
[ 6.3 ms ] story [ 310 ms ] threadThe philosophy at Google is that it's better to filter out 3 good engineers than to let in a bad one. The consequence of this is that it's really hard to get kicked out of Google.
The other part (whether it's more important to work on long easier questions to see how the candidate works on a large code base) is orthogonal reasoning, and that part may be true, depending on what type of engineers somebody is looking for.
Corollary: "If I follow (what I've heard to be) Google's hiring practices (despite not having their brand recognition or candidate pool or a comparable engineering environment to offer) -- then my company is on the way to becoming another Google!"
Google's problem nowadays is that they have such strict standards only for grunt developers, but not for management. In fact, it looks like that it's more like opposite, i.e. it's better letting in/promoting 3 bad managers/directors than 1 good one.
Just fire the bad ones. You're going to get bad ones anyway. At larger companies you might never notice whether someone is good or bad.
From the way they structure their interviews, it seems like they'll still get plenty of bad ones - it's just they'll get bad ones that are great at algorithms, with unknown skill at everything else (like the actual work done).
That's Facebook. Hire fast, fire fast.
Who and what decides what a bad software engineer is (note this is very different than deciding who is a good)? Is it a manager? Managers are single people so shouldn't be the sole deciding point. So you need to gather feedback, aggregate it and decide what constitutes bad. Then have that feedback reviewed by independent people to ensure that no one is being unfair. That takes time and bureaucracy.
* Rejecting far more candidates than you need to -- so you can feel like you're hiring "the top 1 percent"
* Giving yourself the feeling that you have an objective hiring process (when really you don't)
* Making your own team members feel like they're super brilliant and special when really they're not
That's what the modern hiring process is designed to do. And in fact it works quite well, to serve this purpose.
What do you mean by "than you need to"? I'm guessing for many companies (especially the ones with this kinds of interviews), the limit is the number of hires they can do, not the number of candidates that apply... So by definition, you need to reject all but n (the number of open jobs)... Why wouldn't you reject them based on performance on interviews (as opposed to, by their CV or luck or something)?
In plenty of cases companies reject candidates who later perform successfully at similar roles, and this is the point of the parent comment. However this is kind of a desired effect, because not hiring a right candidate has lower cost than hiring a wrong candidate:
- if you skip good candidate, because you're not sure whether they'll perform well, you just wait for more candidates to apply, it just slows down the process - if you hire wrong candidate, the candidate joins the team, underperforms and eventually is let go, but during the time that person works for you, you don't look for the right candidate for that role, which costs you more money and time
I suspect one of the reasons Google is so open about their process and the need to study is so that everyone follows suit. Thereby forcing people to take days off and do homework for even the most mediocre of positions, causing the switching costs of interviewing anywhere to become higher.
I never heard from this person again, which brings a question - is probability of passing it without putting personal time is so low it makes no point in even trying?
Your assertion that low paying positions don't also cargo cult these test-as-interview processes is pure speculation.
Hard - implement a subset of regex match in optimal time+space, find the operations required to turn 1 word into another word given a list of transitory words, find the median of 2 sorted arrays in optimal time, find the next permuted value.
Other people have mentioned Triplebyte using console tic tac toe as a question; that seems like a better sort of "easy" question that still lets the interviewee have a chance to show off their problem solving and factoring skills.
easy.. anything that is not hard :-)
So I'm curious how they define hard/easy.
I know these questions are part of triplebyte's product, so a full, repeatable study isn't in the cards. But if someone from triplebyte could just post a few examples of each, I'd be able to get a lot more out of this result.
So, as a courtesy we figured, why not spend a few hours extra with this applicant in the programming test. We set up a laptop with a clean Ubuntu install, devised a programming test that was quite involved. Not algorithmic hard, just more complex than what can normally be done within a 20-minute whiteboard interview. We expected it to take at least 2-3 hours. Google/Stack overflow/etc access was allowed and encouraged. "Just act as like you would normally do when solving a problem."
We spent like 2x4 hours devising this problem, based on our codebase (cutting out something somewhat easily digestible and making it able to run standalone).
It took like one hour to get productive. Explaining the problem, setting up editors, compilers, etc.
We took turns, but most of the time someone in the interview team (of two) sat next to the guy. We did give him some alone time.
This is probably nothing new in terms of interviewing techniques, but to us it was such a revelation. We learned so much more about the applicant. Perhaps it worked well with this guy because he happened to be a bit more outgoing than our typical successful applicant. We'd never felt so confident about giving someone an offer before.
I'm really looking forward towards testing out this approach with local candidates to see if we can replicate this "data gathering success".
Last time I was involved with this interview style it always seemed to take an hour to get setup which meant a long interview of 3-4 hours particularly if a candidate went down the wrong path.
In the end we optimised for SOLID principles with a blackbox dll that had a function that slept for 2 seconds and a calling class that had mixed responsibilities (logging and calling the dll). We started folks off with a test or two and hoped they'd inject a mock to get rid of the delay and split logging off into a separate class.
I'm not saying it was a great test but you could do something within an hour or so then maybe spend half an hour talking through what techniques they'd use for a more complicated scenarios.
If you can afford the time then more realistic testing is great and I do think you should try.
It was worded far worse than that. What exactly is that telling you about the engineer?
On the flip side I'm asked to code full fledged applications but not to spend too much time on them... okay...
Another time I was asked to code a luhn algorithm. Oh and do it while a room of people watches you on giant screen cause that's what your day to day job will look like... I failed miserably and still got the job. What?!?!?
For instance, one of the problems I frequently ask has a structure that really encourages people to try inventing heuristics to solve the problem, even though ultimately all of those heuristics fail. Seeing how people react to "but what if your input looks like this?" questions is often very enlightening- can they rethink their approach? Do they just keep glomming on more special cases? Can they deal with someone pointing out that sort of flaw?
I got bit in the ass by this one as triplebyte itself. They asked me to make a tic tac toe game, and gave me iirc 30 minutes (less?) to do it. Except, it wasn't "build a tic tac to" game, first it was "draw a board to the console," "take user input from the console," etc a bunch of instructions in a convoluted path that perhaps another engineer would do when knowing from the outset that the goal was to build a tic tac toe game in 30 minutes, but not me.
So we'd get to a portion where I'd be writing a quick test on user input, or extrapolating something to a function, and the interviewer would say "don't worry about that, just worry about {getting the grid to print to console or whatever}."
Later on I got my feedback and they said they were disappointed with my user input tests and repeated, extractable code in the tic tac toe portion.
Triplebyte is trying to do good things in the interview space but I think they're still learning. All in all my interview with them was about as positive an experience as a harried and bad interview could be, from my perspective.
There were a lot of Googlable boilerplate questions (e.g. "what does malloc return?", "what's a bloom filter?") that, as a product engineer, never come up.
Then there were the classic Big-O notation queries that for most use cases don't come up until much later stage. It felt like the founders were classically trained in CS and over-optimizing for things that aren't practically relevant for the large majority of early/mid-stage startups.
Am I familiar with these concepts—e.g. can I go back and refresh myself when they come up?—absolutely. But often times the skills you'd want in an engineer are:
1. Knowing when to optimize
2. Knowing how to profile and identify bottlenecks
3. Familiarity with the available solutions
4. Ability to dig in and evaluate which is the right tool for the jon
This is particularly pernicious, because it's a trick question, too. On linux, malloc always returns, it will never return NULL. Even if you ask for 4 petabytes of memory on a 128mb system, malloc will hand you back a valid pointer for the memory.
https://scvalex.net/posts/6/
(And apparently other conditions: https://stackoverflow.com/questions/2248995/is-there-a-need-... )
If that was an unacceptable answer, then I'd consider the interview a waste of time.
That sounds like a process problem and not an engineering problem. If I've been given requirements, I'm going to trust that my project managers and stakeholders have done the due diligence to understand their request.
Also you've got to realize that different developers tackle issues in different ways. Not every engineer is going to be super talkative while they're in the mud trying to get something to work until they've hit a wall that they don't feel like they have enough information to overcome. I think expecting an engineer to sit there and talk through every aspect of their reasoning WHILE working is fundamentally counter to the way that most engineers perform their day-to-day jobs.
Me: "I'm going to just put some user input checks here, no guarantees they'll actually input X or O, yea?"
Interviewer: "Oh, don't worry about that, assume for now that you'll get X on X's turn, O on O's turn."
Later:
Me: "Normally I'd extrapolate this to a function, so let me just - "
Interviewer: "For now, just focus on getting the program to response to the next user input."
Me: "Ok... well the fastest way to do that right now is just copy paste this code down here."
Interviewer: "That's fine."
But, it turns out, it was not fine.
I felt I was bamboozled.
When she says "don't worry about it" or "that's fine", you can't take her word for it.
You need to be psychic and understand her needs without her saying anything.
I had a pretty great experience and would recommend TripleByte.
edit: now if you'll excuse me, I need to do some dynamic programming problems.
I said I was interested in interviewing but that I would only agree to a process that evaluates me based on my previous work history, and not any onsite or takehome coding projects, system design questions or whiteboard coding questions.
The recruiter said she would run it by the manager, but thought it would not be possible, and a few days later I got a rejection email.
Let’s take their word for it that they are desperate for a machine learning engineer. Then it suggests they care more about mandating workspace conditions (since financial cost even to provide thousands of workers with private offices in dense urban areas is not a realistic excuse not to do it) or trivia during interviews than about business needs.
“The market can stay irrational longer than you can stay solvent,” seems apt for this.
A lot of more traditional software development roles do not have much testing. Sometimes they will have a quick online timed test or a simple question on an intitial phone screen. That isn't to say that all companies do not have testing. But it's definitely the startup and big tech worlds that have the majority of it.
I've worked for two non-technical companies as a software developer and one highly technical company, and interviewed at a few Silicon Valley companies.
The difference between the interview processes is staggering; my current job's interview was two hours of conversation, no code tests, just a general assessment of "do you know what you're doing" by the hiring manager and a couple other members of the team. The highly technical company had a code assessment then the in-person interviews had zero coding.
The SV companies must have a good reason for this, but golly the amount of coding in those interviews is nuts. I'm a process over code speed kind of coder, and I've failed every SV-level test because of it; my code comes from talking to non-technical users like medical researchers and study operations managers and tossing something together in Python or a cloud service that makes their lives easier. Needless to say, I don't go over algorithm fundamentals on a regular basis, and I generally fall out after the first or second interview.
It's especially odd that interviews are so intensely focused on those couple hours since I personally don't see any dev or any resource for that matter contributing in any meaningful way in so fast a time, or even within 90 days. I'm not sure how this problem could be solved with the limited time companies can dedicate to interviews, though; maybe rely more on portfolios?
In fact, no, nobody has a good reason for it.
The reality is that these are extremely desirable positions with a staggering number of applicants who _do not know how to code_. Not as in “I can’t solve a dynamic programming problem without studying up on it”, but as in literally don’t know what a for loop is.
The process is far from perfect and the frustration is understandable, but it works well enough as a filter from these companies’ point of view.
Unfortunately, people in general rarely try to first understand what the candidate offers. It's more often ONLY about whether candidates uunderstand the exact way the company uses certain technology.
The cherry-picked successes thing is certainly a problem, though, and not just for coding. Maybe it's the candidates I've asked it but when asked "tell me about when you made a mistake in a project" they tend to answer with a strength and try to re-frame it as a weakness. "Oh, I worked too hard on this project and it made me tired" isn't a weakness. "I worked too hard on this project and that made me neglect business requirements since I was too myopic to notice" is a weakness.
Sorry if that's a tangent, it's been a pet peeve of mine since I started interviewing that not many people are humble enough or have thought enough about what their weaknesses actually are, and how that affects the success of their work.
The conditional probability you are hopelessly lacking software skills to do a job given that you nonetheless passed a TripleByte exam or something is quite high. Overfitting & memorization for the sake of the test is extremely common.
But it’s much, much harder to fake competence when needing to dynamically and verbally explain technical details in a conversational interview about past work experience.
It's far, far easier to fake expertise when you have more context than the person asking questions. Technical interview questions make sure that the question giver has more context than the recipient, ignoring pathological cases.
In fact, conversational interviewing like this has very little to do with any of the domain specifics of the project. The point is to recursively keep probing for deeper technical specifics, so they have to explain at finer and finer technical levels what were the tradeoffs, why exactly were certain decisions made or how were certain problems overcome.
It is precisely the situation when someone did not have to dig into the technical weeds of a project for themselves that they will not be able to fake or fast talk their way through this type of interview.
That is the number one, defining characteristic of this way of interviewing.
But I have. Not in the specific context of a job interview, but I have absolutely convinced technical experts that my level of expertise in a field is above my actual level of expertise. Ironically, if this were a technical interview, I'd fail it, not because I lack the skills to convince technical experts of my non-existent abilities, but because you don't find me trustworthy.
>The point is to recursively keep probing for deeper technical specifics, so they have to explain at finer and finer technical levels what were the tradeoffs, why exactly were certain decisions made or how were certain problems overcome.
But without context, you can't effectively do that. I, the candidate, am in control. I can steer the conversation to avoid areas where I don't have expertise by answering all kinds of things: "investigating that was someone else's responsibility", "well we never tried anything else and our current implementation works well enough that we never needed to", etc. You can't know if those are lies or not. There are all kinds of completely valid non-technical reasons for decisions that may be completely outside of a candidate's control.
You're either forced to completely trust the candidate, or attempt to verify their authenticity during the interview, at which point you quickly venture into the land of bias and subjectivity.
This is simply false. You don’t need context to understand if the breakdown of a technical problem into constituent trade-offs was appropriate or not — by definition that very breakdown into constituent technical details is the context.
> “investigating that was someone else's responsibility"
This just confirms to me that you are not correct in asserting people can just skate by these discussions. If someone tells me something like that, I’ll ask them what did that other person find when they investigated? If you say anything like, “I don’t know; that was their job not mine,” then you’ve lost credibility because you didn’t put that other person’s conclusions through strong skepticism until you were satisfied you knew the details well enough that you could own or support them if you had to. That is exactly the sort of thing that indicates bullshitting.
> “attempt to verify their authenticity during the interview, at which point you quickly venture into the land of bias and subjectivity.”
This is just wrong. You don’t ever “just trust” the candidate, that’s the opposite of this interview style. Further, you never “attempt” to verify authenticity.. you just do verify it, since it does not require context or domain specialty to analyze reductionist decomposition of any engineering work down into primitive constituent tasks or decisions that are universal.
This approach is far less biased or subjective than appraising “how a candidate thinks” while they solve tricky puzzles in a foreign environment with unrealistic time pressure.
I mean, yes you do. Context tells you which decisions are important. You're asking for the context you need to assess someone's technical competency from the person whose technical competency you're attempting to assess. They have every incentive to lie, mislead, or stretch the truth to make the context they give you highlight their skills more than the real context did. And no, you can't verify that.
>If someone tells me something like that, I’ll ask them what did that other person find when they investigated? If you say anything like, “I don’t know; that was their job not mine,” then you’ve lost credibility because you didn’t put that other person’s conclusions through strong skepticism until you were satisfied you knew the details well enough that you could own or support them if you had to. That is exactly the sort of thing that indicates bullshitting.
But now you're punishing someone for organizational things possibly beyond their control. If my job was to build a thing that made use of some blackbox algorithm, and John developed the algorithm, why should I put the algorithm under strong skepticism, perhaps that's how things work in your workplace, but there's no clear reason that mine work the same way. This is just a bias against people whose development practices aren't the same as yours.
>This approach is far less biased or subjective than appraising “how a candidate thinks” while they solve tricky puzzles in a foreign environment with unrealistic time pressure.
Except that, as I've literally just proven, you've failed to verify the authenticity of me because you've wrongly concluded that I'm not authentic. This comment thread exactly demonstrates just how you can fail to identify a good candidate with this method because if you dislike what they're saying, you'll unconsciously convert that to them being less authentic.
So again, either you take the candidate at face value in a situation where they have every incentive to lie to you, or you attempt to verify their authenticity, at which point that analysis is subject to a multitude of biases that have nothing to do with the candidates skill level.
You've just demonstrated this perfectly.
No, you definitely don’t need to come into it knowing about this, and even if you don’t know about this ahead of time, it won’t imply “just trusting” the candidate or being overly subjective. You will ask the candidate to explain why various decisions were important, and not stop at the top line answer but recursively probe into it, breaking it down into concepts and trade-offs that are universal in any kind of applied problem solving.
> “But now you're punishing someone for organizational things possibly beyond their control. If my job was to build a thing that made use of some blackbox algorithm, and John developed the algorithm, why should I put the algorithm under strong skepticism, perhaps that's how things work in your workplace, but there's no clear reason that mine work the same way.”
I’m sorry but this also just isn’t true. If you are describing your contributions to projects and all that keeps happening is you hit walls in your explanation where someome else did the work and you did not review that work at a high level of depth, then you’re just being misleading about your contributions at work.
Your job as an engineer in a company is to solve problems for your stakeholders, whether that means building tooling for other engineers, assisting designers with prototypes, designing algorithms for core product functionality, sales engineering for client stakeholders, etc.
It doesn’t matter how your company is structured, it doesn’t matter how the work was divided up. Your job is to know about the stakeholder problem you are solving, at a deep level, and when you represent your work to other people and you fail to offer technical depth about the trade-offs needed to solve stakeholder problems, that’s a clear mark against you as a candidate.
It’s bewildering to me that anyone would think that the way their current employer organizes assignments should reduce their burden of knowing how to represent their projects in significant technical depth. That is an always-on, never mitigated, constant responsibility for all employees anywhere. You’re not holding anything against someone if they can’t provide that in an interview... no, you’re just uncovering what they’ve lied about or embellished on a resume.
> “Except that, as I've literally just proven, you've failed to verify the authenticity of me because you've wrongly concluded that I'm not authentic.”
I see no such proof at all, and the cheeky rhetoric just makes me feel more entrenched that you are bullshitting hugely in this thread.
> “So again, either you take the candidate at face value in a situation where they have every incentive to lie to you, or you attempt to verify their authenticity, at which point that analysis is subject to a multitude of biases that have nothing to do with the candidates skill level.”
You are doing nothing but gainsaying here. You’ve made no argument that would support any of these strong conclusions, especially not any reason why this interview method faces the false dilemma between either just trusting the candidate or else succumbing to biases.
You are just asserting things, but they do not seem to be connected to or bolstered by any of the other things you’ve written.
Let me lay it out clearly: I asserted that I can, and have, inflated my abilities to people who have technical know how. This was in response to you stating that "I’m sorry but you cannot do what you are claiming."
So to be clear, at this point, one of two things is true:
1. You are wrong
2. I am a liar
To you, it is clear that point 2 is the true one. To most readers, this is not as obvious. Once you have decided that (2) is true and I am a liar, nothing I say can or will convince you otherwise. But you haven't decided that based on anything factual. In fact, (1) is true here. I am not lying. I can, and have, done the things I claim to have done in this case.
I'm using this to demonstrate that your ideas about such a conversational interview don't work, by pointing out that in the conversational interview that we are having right now, you've decided, based on a preconception, that what I say cannot be true! I could be the world's most successful conman, but because you're preconceptions lead you to believe that your preferred interview process is effective and is less biased than your non-preferred one, you won't accept evidence to the contrary.
>I see no such proof at all, and the cheeky rhetoric just makes me feel more entrenched that you are bullshitting hugely in this thread.
Right, and my point is you're wrong and unwilling to accept that. And that is a demonstration of you not being able to effectively figure out whether or not someone is bullshitting from a conversation with them. You've decided that I'm bullshitting because the alternative would require you to do a lot of introspection about how and why you analyze candidates the way you do. So it's easier to just say "you're bullshitting" and then not put in the effort. And that's certainly your prerogative, but its not at all a good look for your interviewing capabilities.
That you're so prone to cognitive biases that you're willing to completely write off someone's experience because it forces you to rethink something you hold dear is not a selling point of the process you espouse. It demonstrates, like I've said, that the process is prone to cognitive bias and is therefore decidedly not objective.
That is, there are two possibilities:
1. You are wrong, and you're refusal to accept that is coloring your perceptions of our interactions in such a way that you are not able to be objective about my experiences and abilities, as I claim.
2. I'm completely making everything I've said up and haven't ever been able to inflate my abilities to anyone. Your person-analytical skills are infallible and you've caught me.
I subscribe to (1), you continue to wrongly believe (2). This is expected, its why your process isn't as objective as you claim.
>It doesn’t matter how your company is structured, it doesn’t matter how the work was divided up. Your job is to know about the stakeholder problem you are solving, at a deep level, and when you represent your work to other people and you fail to offer technical depth about the trade-offs needed to solve stakeholder problems, that’s a clear mark against you as a candidate.
This is, again, your opinion of how engineering should be done. Not every engineer has the opportunity to work in a workplace where that's how things work. Are you going to write off everyone whose experience has been in a PM led environment because they haven't had the opportunity to develop using the process you prefer? If so that's again your prerogative, but you're probably filtering out a bunch of good engineers.
I’m speaking from ~10 years of experience running my team’s recruiting in a quant finance firm, where many interview requirements / tests / etc., came down from executive managers, so I got to see a wide range of performance on tests of all sorts, riddles, hardcore algo trivia, etc.
The sum total of all that leads me to believe quite strongly that the best signal to noise comes from super careful and tedious resume selection followed by conversational and behavioral interviews that recursively probe into more specific technical details.
I note that you didn't answer my question. Have you taken the TripleByte exam or not?
testing whether someone is willing to prepare for a thing is a relevant work skill test too.
(The counter to that is that if people can relatively-easily (single-digit days) cram for your interview, you're still not going to be effectively screening for at-hand pre-existing familiarity/knowledge.)
If I require all prospective engineering hires to prove explicitly that they have an IQ of at least 135, I will get sued. Do you think this is false?
To turn this around, can you point at the law that makes IQ a protected class or whatever you're claiming it is? I'm not going to have much luck proving a negative - Russell's teapot and all that.
I find that hard to believe.
Moreover, memoized graph traversals don't get you full credit on this question. There's a dynamic programming solution, and in fact the ideal solution is one using matrix math, which is ludicrously divorced from anything most programmers would ever see.
Handled correctly, a problem like this answers several questions:
1) Can you correctly break down a problem like this into its components parts? 2) Can you recognize the overall class of problems that this falls into? 3) Can you transform this specific problem into the more general class so that you can solve it in a known fashion? 4) Can you think about and implement the movement? 5) Can you communicate while you're doing the above?
No one cares about solving that particular problem. But the answers to the above really are relevant. Being able to map novel problems onto known solutions is absolutely a skill that any competent software engineer needs to have. "Oh, you want me to do X? That looks a lot like Y, this thing we've already solved; maybe I can just implement it in the same fashion (or re-use our existing system!)"
I'm not saying that this particular problem is a wonderful example, or that I'd use it in my own interviews. But this overall class of problems really does have a place in interviewing when it's handled well by the interviewers, and arguments against it on the basis of the specific problem being irrelevant are really rather missing the point.
I don't do many interviews anymore, but I used to, and I had no short supply of problems that I actually had to solve in the course of my work that I could ask about. I don't think whiteboard interviewing is great in general, but if you're going to do it you can at least try to keep it relevant.
As a matter of fact, I did have to do memoization of graph traversals at least once for work (most programmers never have to do this), and I find that problem a lot more interesting (trait matching for Rust). I could easily give a talk about that problem. As for that interview question, though? I don't always do well with time pressure, and so I can't guarantee I'd be able to answer it to your satisfaction.
I've done the "ask a real problem that I've solved before" thing, and I find that it usually gets hung up on details and context around the problem to the detriment of actually solving the problem at hand. That's not to say that such a conversation isn't itself very valuable, but that's a different interview conversation, at least on my team. We find it important to maintain some level of focus just to ensure that we're covering the various signals that we'd like to get from the candidate.
My view is that Google's interview process seems to work because Google gets so many applicants that they can afford to randomly reject most of them. It's not because it's a good process.
So it isn't fair to ask the candidate to solve a real bug or implement a real feature in only 35 minutes unless they've seen something similar before.
This is why big companies like Google are limited to whiteboarding interviews because they need to have an interview process efficient enough to properly vet and filter >1 million applicants Google receives each year.
Personally, I think a better interview process is a pair-programming or work audition for a day. But that is not even close to matching the scale of Google.
Let's say out of a million applications, maybe only 25% are qualified. That is still almost 1000 candidates to interview per day (number of U.S. business days in 2019 is 261; 250,000 candidates / 261 business days = ~957 candidates per business day). Pair-programming or full day work audition will not be able to accommodate 957 candidates every day.
Is it, though? I could understand if algo questions had some relation to the work you're doing, but your comment on why they're good is independent of the actual material. If we replaced the algorithm question interview with an interview testing obscure presidential facts, would it really be a useful test to have devs take? I guess it is a relevant work skill test in that you get people willing to put the work in/game the system, but it doesn't seem to be much of (if at all) an improvement from ad hoc conversations.
But it will also filter out everyone that is opinionated enough to not do that stupid preparation work, and you will end up with sheep coders that will always follow the rules. Looking at Google, Facebook etc, this might already be the case. I will even go so far to say that they prefer those type of obedient coders than the ones that ask too many questions and get too creative.
Reviewing algorithms for interview prep at least has some relevance to programming. While a candidate may not use that exact algorithm in their day to day job, they are creating ad hoc algorithms all day long. With that said, I don't think time pressure, white boarding algorithms is a very good job performance predictor.
Ultimately, its just studying for the test, very much like the ACT/SAT in high school. You can be great at taking tests but ultimately a terrible student or vice versa.
There’s interview cake and leetcode, but I think people would pay $2k for a class that focuses on the questions and in person whiteboard practice.
They could collect information about the interviews at the major companies and then use those to create the program. For payment could also help candidates negotiate and then take a cut of the signing bonus.
If this isn’t part of lambda school already I think it should be.
I already work at a competitive tech company, but I’d sign up for this in a second to help me stay competitive for interviews.
It’s part of classes for our existing students, but it isn’t a single offering. One day.
It's funny that after doing some searching, many proponents of this widely adopted and poorly researched interviewing methodology appear to be running businesses selling training for these interview processes... surprise, surprise.
How do you know?
I understand that FANG have themselves come to the conclusion that brain teasers are not necessarily very predictive for engineering performance.
But CS/programming questions for CS/programming roles? That seems sensible.
You can study lots of vocabulary to achieve better results on the verbal section of the GRE or similar tests. But afterwards, you will, in fact, have better vocabulary, I submit.
I am surprised that you categorically deny that they were better engineers after studying CS/programming questions.
Yeah, but tests like the GRE claim to predict academic performance, not vocabulary. So unless you think augmenting one's vocabulary alone will make one a substantially better student, there's still a gap there.
Maybe for a few days after the test anyways. And then...you push out those words you never use or read to make room for actual useful stuff.
On the other hand, I genuinely do feel theory is really important. While not knowing the minutia of a tim-sort doesn't indicate that you'll be a bad engineer, not knowing the runtime efficiency of a sort can lead to some really awful code. Not knowing when to use a hash table instead of a nested-for loop can be a sign that you don't really know what you're doing, and not knowing some rough theory on concurrency indicates that I might be stuck debugging your race conditions or deadlock.
I try to not be a complete jerk and I won't do stuff like give out an NP-complete problem (which an interviewer gave me once), nor will I ask for intimate details of how one would implement CSP, but I do tend to focus on theory-heavy questions more than my peers, but I try to give a fairly-generous amount of hints so that people don't get too stuck.
That sort of thing might give a better sense of if someone has an instinct for how to verify whether the code is working. And, by extension, if they have a grasp of some concepts that go a long way toward helping a person come up with more reliable and maintainable designs.
If I were hiring a carpenter I would want to know they understand their tools and when to use them. I don't care if they don't know how to make them. But of course knowing how to make them suggests an intimate appreciation for the craft (looking your way Matthias Wandel)
On the flip side, the O(1) look-up nature of hash tables make them the no-brainer data structure to use, at least for passing programming interviews. Perhaps more interesting test questions would be when not to use hash tables.
Although these days one should talk about "learned index structures" instead, I guess :)
My beef with this in interviewing is that a huge chunk of modern programming work is developing UIs - and user interface theory and skills are treated like some softball thing. You ask UI-related questions to interviewers and half the time you get some shrug "oh we use whatever", etc.
I can't say that theory is more universal than frontend, though anecdotally, for me, it is. But ui isn't universal.
Theory is important but what's more important is how someone applies the theory to actual problem solving.
Reversing a binary tree isn't testing your knowledge of theory as much as it's really just testing memorization of a very very specific application. Knowing how to reverse a binary tree or how a hash map works is pointless if the person can't identify when to use them when solving an actual higher level problem. No one is ever given a binary tree and told to reverse it in the real world, they are given a business problem that you identify can be solved efficiently by modelling it as a binary tree and reversing it.
I'd bet most of the good people who fail the "reverse a binary tree" type of questions would succeed if you give them a realistic problem to solve without forcing a very specific solution onto them. Either they will come up to the realization that the solution is to think of the problem in terms of binary tree or they won't.
And neither is a terribly bad answer either. If they recognize you gave them a problem that can be represented as a binary tree and efficiently solved by reversing it then there you go, not only did you prove they knew the "theory" but they knew how to apply it as well. If they don't recognize it you gain valuable insight into their line of thinking and they might find novel ways to represent the problem and apply other theoretical concepts that solve it efficiently (maybe more maybe less).
I learned later that hashmaps are inherently O(n) (or O(log n)), or they take a lot of memory; internal arrays can get super huge if you're not careful.
In this particular case, an ugly nested for-loop was the immediate solution, and eventually I was able to cheat a little and have hard-coded integer indexes and was able to use an array.
Anyway, your point is valid, I just figured I'd give an example where knowing the internals for a hash table would have saved me a lot of time.
OK but why don’t these interviewers ever ask “internals” questions? How does a CPU works, or what the different levels of memory hierarchy are, what an interrupt request is, what is pipelining, what is SIMD, what is a GPU, etc. Or how about compiler internals, what are the different stages in a compilers, what are the different grammars and parsers, what is an AST, what is interpretation, compilation, bytecode, JIT, and how does it work? How about database internals? How is a database implemented, what is relational algebra, what is normalization, what is a data model, how do indices work? Similar questions can be asked about operating system internals, networking, floating point arithmetic, etc. In my experience, no interviewer has ever asked me these questions. And as an interviewer, I have consistently been disappointed with candidates’ inability to answer basic and relevant “internals” questions. Ironically, the algorithms and data structures questions that are commonly asked in interviews are outdated ways of thinking which do not take the underlying hardware reality into account (memory hierarchy and parallelism). It could be because a lot of these FANG(-like) “engineers” are themselves not knowledgeable to ask such questions. Or they are aware that new comp sci graduates are unlikely to really understand anything at a deep level, but all comp sci students are drilled on algorithms and data structures. If you are targeting a very specific age group (0-5 years experience) and want a standardized test then I guess asking CS201 exam questions in a job interview make sense. Companies prefer the 0-5yrs experience segment because they’re cheap, don’t have children, are easy to exploit, and they are easier to “mold” into your corporate culture.
Because not everyone writing Java code has had a background that exposes them to compiler internals or grammars and parsers.
But everyone writing Java code should at least be a little bit arsed to figure out what common data structures do - or how to write two for loops that print out "1 2 fizz 4 buzz"...
Managing complexity is a valuable skill that should also be screened for at interview. At most top companies you are there for at least 4-5 hours so there should be plenty of time to evaluate that skill.
I think whiteboard questions are good, I want to know that this person is capable of writing difficult code if we need them to.
I also think we probably ask too many of them.
I have been on the hiring side and my experience so far has been that almost always the feedback is close to identical across multiple whiteboard questions. The questions are also so abstract that asking multiple to "prevent bias" seems ineffectual. What bias could there be, you either solve the problem or you don't. Bias is more likely to come in on the behavioral interviews. There should be multiple of those for sure.
Generally someone is either a good enough coder or not and they will display that consistently across all the interviews. You will see the same stuff throughout (good or bad variable naming, good or bad communication etc) thus asking > 1 coding question by default is a waste of everyone's time.
It's not a false dichotomy because I am not suggesting that both skills are mutually exclusive. I'm highlighting some personal experiences where I've seen one skill is vastly overvalued compared to another.
>Managing complexity is a valuable skill that should also be screened for at interview
I've never been part of an interview, on either side, where I've seen testing for managing software complexity. Good interfaces, function design, side effects, state management, etc, are all second class citizens to finding the appropriate algorithm to solve the interview question. I've seen it over and over. The only time I've been close to a complexity management question was on a systems design question, but even then, it was only a very high-level systems discussion. I don't think most places know how to screen for it.
Worse, there's this huge emphasis on "big O" with zero focus on clearly egregious bad practices (tons of copies, outrageous memory usage, casting between strings and numbers all the time). I've rarely seen clearly bad algorithms be deployed but I have seen plenty of unperformant code go out.
Probably biggest influence on this I think is the culture, if nobody tells you that your code is shit (in a kind way) you'll never learn better. But then again, some people are so hard-headed and full of themselves that they get defensive and never actually admit their faults. Though giving and receiving feedback is not easy, can give you an identity crisis once you realize you've been doing something wrong for years.
And then you have a bunch of Ivy League graduates who have spent years learning algorithms and are burning to use them but there's no actual problems that really need them.
No matter your background, what school you went to, or what randomized experience you got in previous jobs, every person has equal opportunity to study and practice the same algorithms on their own (as opposed to being lucky enough to be able to afford a top-tier $$$ CS education, or to being lucky enough to have the connections or chance to get certain previous jobs).
And thus, when applying to jobs, it becomes something more akin to a raw-ability IQ test, which you can argue is "fairer", especially when management realistically knows developers might be shuffled around all the time, and that the extensive SQL experience they were hired for will mean nothing when project requirements switch to a basic key-value store.
On the other hand, if you are interviewing for a highly specialized position that is fairly certain not to undergo change, then it makes sense that specialized experience could rightly count for far more than any kind of generalized intelligence or ability.
Well, that's just not the cause - there are many groups of people who lack the opportunity to study and practice. Couple of examples: people with kids, people working 12 hour shifts, people without access to teaching materials, people without a sufficiently advanced machine to run dev environments, etc.
Where are you learning DP problems from? I am very bad at those and need a few good references so that it sticks in my memory.
Here's a few for you to try. Some of these are pretty hard, but you should be able to find solution sketches online if you google the contests they are from.
https://open.kattis.com/problems/increasingsubsequence
https://open.kattis.com/problems/maximumsubarrays
https://open.kattis.com/problems/tray
https://open.kattis.com/problems/dinnerbet
https://open.kattis.com/problems/hyperpyramids
I've solved all of these as well, so if you get really stuck feel free to reply here and I'll try to guide you through them.
https://open.kattis.com/problems/theescape
- Workers are going to be around for 15 months or less and they have domain expertise on 1 stack already and I don't need to screen for how they would hypothetically function across all stacks
- Worker's process and resource finding skills are more indicative of the time they will spend on a task
- Worker's process includes collaborate use of version control and code reviews, if they pass the screening but can't really integrate on these things then thats what will get them booted from the team
It isn't always more expensive to have a not great developer. Look in your organization and see if what I experience is true for you, and you'll save everyone a lot of time.
I recently did something similar, that was of my own devising.
At one point several years back I wrote down some "code" in my own "shorthand" form; it was meant to implement a library and some test code for a microcontroller project I was contemplating at the time. I basically wrote it in such a fashion so that I was quick to get my main ideas down without being too "wordy" (whether in code or otherwise).
Then I put it away, and didn't revisit it again - until recently.
A couple of weeks back I found that code again, and looked at it - worried that I wouldn't be able to recall my shorthand or what I was thinking; in short, worried that my ideas would be "lost".
I looked over the code, walked thru it in my mind - and after a few minutes it all came back to me, and I was able to understand again what I had originally created (and where I could make improvements as well). It both left me feeling optimistic about the process, as well as a bit excited that I was able to remember it and improve on it - that I didn't have to worry about it, and that my shorthand "pseudo-code" was legitimate enough that it could have been real code for all that it mattered.
i didn't expect critical discussion because they were not native english speakers. (in fact they hadn't had an opportunity to even use english outside of talking to their teacher, so when they were able to engage in friendly arguments over how to solve a problem, that was quite an accomplishment, even more so for asian culture which is generally rather submissive (you don't argue with your boss))
The other candidates, after answering some of the harder questions incorrectly, seemed very upset with themselves. They knew they were cracking a bit under pressure, but actually showed that they knew the answers when we chatted further. I hired 3/4 of those people because of how well I felt they'd do given the opportunity. All three became leads within a year and a half.
I think personality has a lot to do with outcomes. If you are someone who shows they are hungry to learn and knows how to improve their skills, I will never dismiss you for screwing up a few coding questions.
If you're hiring for Generic Developer Skills you're going to get generic developers - and much less development than a more flexible approach would give you.
But, that interviewer is the type of developer I don't want to work with. A know it all who only acts like he knows it all and uses Google secretly. Who puts others down to make themselves look good. AKA an insecure P word I want nothing to do with.
I'm being facetious and agree with your sentiment.
The only advantage I can see is you can get rid of them easily by not renewing their contract but that in itself is not a solution to finding good people in the first place. Plus it's likely to be hugely limiting. I'm not going to give up my current job and move to your city for a contract you might cancel in 3 months.
It gives me the feeling that I will be working with people who either don't know their own value and so are willing to agree to this kind of deal, or are so poor performing they don't have the leverage to insist on a regular employment contract.
Many companies are "contract to hire" but never hire and keep contractors on as long term employees.
- Yes, you have to have vetted your contracting companies. In our case that's something we'd already done (roughly 20%-30% of our staff are contractors)
- This is contact to hire, which is different than bringing on contractors for staff augmentation. The intent is to hire and the applicant is made aware of that up-front, to the extent that their employee salary is negotiated and employee benefits package is discussed.
- W2 employees jump at the chance to be given an opportunity to prove themselves in a new domain. Younger and older employees are especially attracted to this option.
- You're still doing an interview, but the nature of the interview is different since you know the contracting firm has vetted their basic skills.
- So far we've only had to let one person go, they simply never could "gel" with the team. We've had two others we had to let go as we were scammed - the person that did the interview wasn't the person that showed up to work. Yes, it happens. What we don't have is a "revolving door" situation where people are continually rotating in-and-out.
Now, the ads ask simpler things like floating point precision and function variable scoping (https://www.facebook.com/triplebyte/ads/?ref=page_internal ); legit problems, but not sure if they are an indicator of how good a developer they'd be in the real world working on a CRUD app.
It proxies the reader having a CS degree, which is incidentally what TripleByte's testing is supposed to deemphasize.
My worst interview ever was with Facebook when a non-native, new college grad gave me a Leetcode hard problem in half-broken english and went back to his work without even looking up or walking with me through the problem.
Now, 13 years later, I mostly rely on "homework" type exercises. I think they address most of the issues. They are more "real world", no time pressure, etc. However, even those now are being heavily criticized. What's left to be used?
I totally understand the main criticism for homework, it takes time, the company may never call you back after you poured 2 hours into their stupid exercise. But it is an attempt at fixing all other alternatives: * the onsite whiteboarding is BS * using open source contribution is totally unfair to candidates who don't participate * the "contract for 1 month and then maybe we'll hire you" is also total BS in my book. Who would leave a FT job with benefits for a contract that may end?
I'm not sure I can think of any alternative that has 0 drawbacks.
> after you poured 2 hours
and at the end it turns out to be 5-10 hours. Companies say "you should be able to finish it in 1-2 hours", but it's almost never true.
I think alternative is to have 1 hour coding session on a good problem. Most problems are complicated, but there could be something else. I was once asked by Uber to implement timer in JavaScript that will update DOM, also create APIs to stop/start/pause. This kind of a challenge doesn't involve any algorithm, but it shows your ability to code.
It should be enough to bring you onsite.
But I open source should work. If you contributed to Linux Kernel and it was accepted - onsite no questions asked. If you have github repo of 1000+ stars - onsite no questions asked. If you have already passed Google tech screen and was invited onsite - invite onsite no questions asked.
Just to clarify first, the exercise should not take more than 1 hour. And I mean it. It is not difficult, or tricky or anything. It is used 1) to make sure the candidate can do very elementary things, 2) as support for follow up during the on-side interview.
Regarding "getting the on site", if you don't do the exercise before you would get the "whiteboard" BS. And then you may be competing with people who spent 2 weeks reading "cracking the tech interview". How is that different?
And there's nothing wrong with whiteboarding, if done appropriately. Whiteboarding a system design question is totally appropriate. Asking a candidate to write down code for finding the longest palindromic substring of an input string in linear time is not. I've never come to a situation where writing anything more than the barest pseudocode outline for what to do on a whiteboard was the best way to get a real world task accomplished.
Switching to more practical, simpler problems allowed me to really observe how they work and solve a coding problem. As the article said, I was also able to add requirements or features to the problem which let me see how the candidate adapts to changing requirements, or refactors their own solution to handle a new edge case. Simpler is generally better if you are timeboxed to 45 minutes.
you wanna know how to tell if a programer is good or bad in thir job... give them questions from the eular project...
and start from page 6 or so... from questions 200-300-to start with.. after he answerd it all correctly and... he should be able to solve at least 1 problem from page 13... only then hire the man....
just because you may be stupid/... find somthing too hard dosen't mean it is too hard! man...