I thought this is an extremely inaccurate portrayal of CS PhDs who want to get into good startups. Pretty much all the PhDs I know can hack prodigious amounts of code and are not totally incompetent socially. Where they lack are mature sw dev skills (unit test, fancy design patterns, comments in their code, variable names). This is not because they are lazy or unawares. Most systems grad student projects have a specific structure that makes this a reasonable choice. Many students don't work with teams of devs during their PhD education. So please .. do not think most PhDs can't code.
I went through a similar struggle at my first interview out of school, at least in regards to an interview.
As I did non-majors courses at a community college before an intense 2.75 years at a full course university, I sometimes had to compromise on my course selection, as some things were taught in semesters I was unable to take them in.
In the interview(consisting of multiple interviewers, this was my first) in my first I was given the task to create a small class that would hand out read or write access, allowing many reads, and one write, and never a read and a write at the same time.
I thought for a moment, and started dictating my solution. Unfortunately, due to the above, I was never able to take an Operating Systems or Parallel Programming course. Thus, I had never even heard of a semaphore (I now consider that inexcusable, but perhaps understandable). As I set to work implementing two locking list structures, the interviewer would constantly interrupt me, trying to steer me elsewhere - a place I could not have possibly even known about - an unknown unknown if you will. This ultimately resulted in what was, to me, and I'm sure to him, a very frustrating 15 minutes and maybe 3 lines of code.
All this to say, I think sometimes there are just vastly different expectations on each side of the table, and it can result in mutually poor experiences in the interview room.
I deal with this a lot as a programmer who doesn't have a comp sci degree. I know what a semaphore is, I just don't call it a semaphore and I didn't know that it was called a "semaphore". I have been happily chugging along calling these "control variables"
I do not agree with the definition of "production quality software" in this post, which exclusively focuses on purported readability and extensibility by others.
No. No. No.
The difference is that production quality software needs to be usable by other people -- often many other people -- instead of just the author or a small group of researchers. In many cases it must work close to 24 hours/7 days per week under highly variable conditions.
This means it generally must have fewer and less severe bugs than a proof-of-concept or research prototype. This often requires extensive testing by end users or QA folks standing in for end users.
There are many examples of shipping commercial products and widely used open source programs that clearly meet this criterion but are certainly not easy to read, maintain or extend code.
Modularity in particular often fails to produce either readable or extensible code, rather producing numerous layers of confusing modules and/or objects -- "lasagna code" instead of "spaghetti code." It also can fail to produce "production quality software" as I define it above.
> The startup pace is fast, and you need to be creative, think on your feet, and come up with solutions as quickly as possible.
A more accurate description is: The startup risk tolerance is extremely high, thus it rewards quick and dirty over well thought out and solid.
This is often opposite to academic work, which needs to be well thought out and rigorous to an extent, though not necessarily in code quality. The code just needs to work for the experiment and be correct. It doesn't need to be maintained, extended, and debatably, communicated. This is also probably why startups tend to hire younger people, rather than older developers who've worked for places where the risk tolerance is very low and thus they take the time to make sure things are very solid.
What? A decent PhD needs to have a payback horizon (time between start to commercialization) of over 5 years. Startups cannot afford that level of risk. PhDs should be much riskier than startup pitches.
Maybe it is an artifact of the fact that I work on computer systems research (where the focus is on finding the simplest solution that works) but the comments here make me wonder where you (collectively) are finding your PhD candidates!
I do agree that is often OK for PhD code to be terrible ("least publishable unit"), but that is also true of non-core production code at startups! ("Minimum viable product")
Furthermore, the article under consideration speaks to that:
>Many PhD students can be perfectionists. But they need to remember that there’s an important compromise that needs to be made between the best solution and a good solution that can be used in practice.
Code quality doesn't seem to be a concern for Dr. O at least.
> This is also probably why startups tend to hire younger people, rather than older developers who've worked for places where the risk tolerance is very low and thus they take the time to make sure things are very solid.
Appropriate older developers should have an even better handle on when (and how) to write good code and when (and how) to write bad code. Moreover, their bad code will be better, and produced faster.
Young coders are hired mostly because they are cheap and don't understand stock options.
Perhaps you misunderstand the agile process? Quality isn't something you ever actually achieve through the Feynman Algorithm (write down the problem -> think real hard -> write down the solution) regardless of whether you spend two hours thinking real hard or two years. Quality is achieved through iterative, continual improvement and polish. Otherwise, waterfall software would be the pinnacle of quality.
Software quality isn't the easiest thing to define. In some cases, it means "no bugs", and it certainly looks to me as though the places turning out really-low-defect-rate code for, e.g., aerospace applications are doing a fair amount of up front planning (plus other things -- reviews, dedicated QA groups, etc.)
In other contexts "nice to use" is a better measure of quality, and I definitely agree that some iteration can help there. Even so, I'd caution that iterative development and a planned overall architecture aren't necessarily incompatible. And when following an iterative approach you have to be pretty careful to optimise for something that most users like, rather than a just bolting on everyone's pet feature.
I love how the assumption is that the problem is with the interviewee rather than the startup - so confident of this, in fact, that they feel obliged to write a blog post asserting this.
Defensive anyone?
Notice the assumption here, that their 20-minute task is a perfectly reliable indicator of how the interviewee would behave in general.
The interviewee was right to leave them without saying a word.
God I get sick of the smugness. You can cut it with a knife.
Ignoring the tone of the piece, the article does make valid criticisms regarding those entering the workforce with higher education. Most people usually get these experience and lessons from working in the workforce, but those in higher education sacrifices that gain further professional education.
I think the interviewee did make a mistake during his interview, in that he didn't read the room. They weren't looking for anything "cutting edge" and they know that they can't understand the in-depth and complicated work you did for your thesis. So what the interviewee should have done was focus on simpler and easier methods to start off with and kept it general. They already know you're smart from your degrees and from your CV, the focus of that segment of their interview was to see if the candidate had the communication skills available to express their ideas.
We have the same meetings with our stakeholders and during our project meetings. I don't explain the complex methods and algorithms used in-depth. I just keep it general and tell them how everything fits together (if I have performance indices then I also add that in to show accuracy). If the stakeholders want more detailed information like specifically how it breaks down and why I used X method instead of Y, then I can answer their questions and (if they showed continued interest) be willing to put together a report for them detailing the literature reviews and information related to that topic.
20 minutes isn't a reliable indicator of how someone can do their job. However, as the interviewee or the person in the "hot seat", it is your job to know how to sell yourself and to use your time effectively to maximize your chances of getting an offer. The decision makers (or in this case, the interviewers) has the hard task of figuring out which candidate is right for them. Your job is to make that easier. Once you get the offer, then you get to decide if you want to accept it or not.
And then after you get the offer, you reflect on your experience with that company so far. If the interviewers seemed to be focusing on different things than you're interested in or if it seems like they're going to be miserable to work with, then you politely decline the offer and move on to the next thing. If it seems like they have a condescending tone towards you and your work, then you can always just say "thank you for your offer but I'll be declining".
To be honest, the fact they don't tell us the actual problem they set their candidate on makes it hard for us to evaluate it either way. In fact, it could lead one to wonder why they don't just tell us what it was.
That is true. Like in any project, research, or work, the devil is in the details. I personally think everyone could have been more understanding of the situation and clarified specifics. However, in the end no one won. The talent has left and the company still needs to fill a position with a capable candidate.
Interviewer here. Apologies for the tone, totally not intended. The true intend was to help PhDs be more prepared, not to express smugness.
Here are some details that were cut out of the post:
- we have an extensive engineering interview process, typically 3-4 sessions. First chat, tech screen, take home assignment, and half day hackathon with a few engineers
- sometimes we do the initial chat and tech screen together so we had a 40 min chat, and 20 min simple tech screen in this case
- proposed of simple tech screen is to decide if this person knows basic coding/problem solving (eg Josephus problem)
- goal is not to write code on the white board, but rather talk and discuss approach, general problem solving
This candidate was specialized in OR, so we asked an optimization related question (given an array of products and prices, how to spend all your money exactly). Instead of a discussion, the candidate dove into writing mathematical proofs, and after framing the goal of the exercise and a few hints, he ended up writing a 10-level deeply nested for loop...
Obviously I'm not trying to make generalizations, but I've interviewed over 100 candidates in my career, and there is a strong negative correlation with PhDs in particular, hence the article. Understandably, because they are trained in other methods, typically unsuitable to startups, as others have commented. But if they want to get into startups, I hope these tips would help.
Alternatively, they might have looked at their publications / spoke with them and determined they have the research chops and want to figure out if they will destroy the code base or not in the process.
It's unclear what problem they gave the PhD was about. It could have been a hard optimization problem or some coding test.
+eleventy on that comment. It sounds like they want a clone of Dr O that has industry experience and can bang out code quickly that solves an artificial problem given in an interview.
What I got from this article is that the company shouldn't be looking at PhDs but rather at coders. At least that's what their interview seems to be geared towards.
Being sat down with a group of new people and told to work, observed, or a novel algorithmic problem against a time limit does not feel like a good way to evaluate people for a job which likely has none of these challenges.
Don't get me wrong, I kind of enjoy doing those things personally, but it doesn't seem like a great way to evaluate developers.
This is a great article. As someone who just interviewed 100 engineers for my startup, the lessons resonate with me, and can be generalised beyond PhDs:
Come up with any solution, no matter how bad or inefficient, and then improve it. That's better than trying to find an optimal solution and not being able to come up with any solution at all before you're out of time. Then, you've demonstrated an inability to solve the problem. To put it in black and white terms, you've failed. An inefficient solution is much better than no solution at all. And it may be good enough if the data set is small.
Also, if you give me an answer in 10 seconds, efficient or not, you will impress me as someone able to make rapid progress. You can then improve it, putting you in a better position than someone who ended up with the same final answer but took longer to give me any solution at all. Working code is critical.
Discussion is important. If you've chosen an array over an ArrayList, I want to know why, such as an ArrayList of a primitive type being inefficient, or that we know the size ahead of time. Conveying that you know what you're doing is more important than a specific choice. Because if you know what you're doing, I'm confident that you can handle a different problem that turns up if I hire you.
"Also, if you give me an answer in 10 seconds, efficient or not, you will impress me as someone able to make rapid progress."
It sounds like you are passing on good candidates due to them not being fluent in blabling. Which is fine if the job requires that ability (half selling to customers position), but is not if you need engineers to build system.
And then startups complain that there are no good people - they exists but get ruled out due to non technical presumptions like this.
You're wrong in dismissing a correct but inefficient solution as babbling. I won't respond to your other wrong conclusion that follows from this wrong starting point.
The ability you test there is saying half throught things out loud while you are still evaluating them. You yourself talked about saying things "bad or inneficient" within 10 seconds.
That ability is not the same as being good engineer or problem solver. It is pretty much orthogonal. E.g. You can be good and both have it or not have it.
Basically,you test social skill for position that does not necessary requires that skill.
It's not half thought out if the answer correctly solves the problem, even if inefficient. Rather, it's the attitude of finding a working solution first, and then making it better, rather than the attitude of perfectionism ("I'll take a lot of time to find a great answer"), which the blog post talks about as well.
It is half thought, because at the time of speaking you haven't evaluated yet whether it works or not. It is about being ok with uncertainty when speaking.
It has more to do with whether you think out loud (which I do and have a bit of advantage in such hiring process) and less to do with overall attitude of perfectionism when giving away finished work. I would not hire such person as consultant or support, but I worked with such people on positions that don't require much speaking with customers and they were great to work with.
Going back to the example of a doubly nested loop to determine if there are any two numbers in an array that sum to zero, it should take at most 1 minute for a smart engineer to figure out that it works. They don't have to speak before they've evaluated whether it works. The metric I use is: how long did they take to get to a solution that works, even if inefficient?
I didn't say I want quick but wrong answers. I said I prefer a quick but inefficient solution to a problem, as a starting point.
If I ask to determine whether an array contains two numbers that add to zero, a competent engineer should be able to give the naive answer -- a doubly nested loop -- in ten seconds, or at most a minute. If someone takes a lot of time to give me any answer at all, it's a concern.
Sort the array by absolute value, and then a check to see if two adjacent values are a negative/positive pair?
But seriously, giving programming problems like these to a PhD is probably going to scare the hell out of them, not because they think it's hard, but because they'll find it indicative of the work they'll be doing if they get and accept the position. Hopefully such faker filter questions are followed up with ones with real technical meat.
If the candidate does great, I'd definitely move on to more complex problems.
But this question lets me learn a lot about the candidate:
- How quickly did they come up with any working solution?
- How long did it take them to reach the optimal solution?
- Do they know time complexity?
- If they're using Java, did they use a plain array or ArrayList? Do they understand boxing and performance?
- Do they understand the range of an integer? I'd tell them that the array represents the population of the earth every year from 1900 till today, and see if they realise that they need a 64-bit integer.
>"If I ask to determine whether an array contains two numbers that add to zero, a competent engineer should be able to give the naive answer -- a doubly nested loop -- in ten seconds, or at most a minute."
A competent engineer or just one who's practiced HackerRank questions and mastered the cliche choreography of the SV interview format?
"Think out loud! We want to know how you think! Ask clarifying questions! Start with the naive solution! But oh also please do all of these things within an arbitrary time restriction known only by me if you want to impress me."
But these skills are also very useful during oral exams.
If you don't know the answer to a question, start talking about what you know in the general vicinity of it and see what happens.
You might get a hint for a direction, or the person giving the test might drop the question entirely because you brought up something more interesting (so careful there!), or they might decide that, while you cannot answer that particular question you're still sufficiently knowledgeable.
As long as you keep talking good things will happen compared to silence. A bad answer is a lot better than no answer, always. Also good is asking suitable questions back. If you're lucky you can derail the professor from giving a test to having an inspiring conversation about the subject, without grade loss.
So, this is a skill that is (indirectly) taught and practiced in academia. Thus I don't get the point of academia people having problems with this.
>"This is a great article. As someone who just interviewed 100 engineers for my startup, the lessons resonate with me, and can be generalized beyond PhDs:"
Let me get this straight, you invited 100 engineers in for an onsite after qualifying them with a phone screen and their resume? You conducted 100 Google style whiteboard algorithmic puzzle interviews just to find 1 single engineer to work for your startup?
These facts also resonate and could be generalized as something might also be terribly wrong with your "precious" interview process as well.
While I agree with the base message in the post (speed is important, keep it simple, discuss before expanding, etc.), I find the post itself made assumptions about PhD that can be clarified upon.
I'm an engineering Ph.D. candidate focused on optimization and forecasting (tldr: large amounts of data analytics). I also code extensively in R. As a Ph.D. candidate, you're supposed to be the expert with the cutting-edge knowledge in your field. You're supposed to know what options are currently in development, the pros and cons of each methodology, and the significance of the final output and how it fits in to the big picture.
Most PhDs focuses on developing and implementing best methods (theoretically) and continuously improving accuracy of the final results. Methods that haven't matured enough to be implemented today, but will tomorrow. Timing and speed is still important in academia, but (and this is my opinion) more focus is spent on implementing the theoretical approach correctly.
From my observations, industry environment focuses on implementation speed (not the theoretical approach development but rather just "coding it in") and making it work to "solve" the problem. Focus is spent on implementation speed over maximizing accuracy. Therefore, simpler and less computationally intensive (and in most cases, easier) methods are preferred over more complicated methods.
I guess to sum up my point, academia focuses on cutting edge technology and continual development of some of the best methods available. That's how you get more of your research published and further your academic career. Private industry would love to implement cutting edge technology, but focuses more on implementation speed which can sometimes impact model accuracy. From my understanding of the article, it seems like the candidate they interviewed wanted to show the extent of his knowledge whereas the interviewers were looking for the general "overview" and discussion approach and thought process (his mistake probably for not reading the situation right, but just what I gathered).
Oh also to add as a P.S., most PhDs spend time focusing on theoretical problems and methods. Which usually means that knowledge regarding full stacks, infrastructure design, supplemental technology (e.g. D3), etc. can be a bit lacking at first compared to someone who spent the equivalent 4/5 years working in the industry and who has gathered experience using those technologies. Also PhD is really a "solo" mountain/task to conquer. While you do have colleagues and lab mates with you, everyone is usually doing their own research. So unless you spend time outside of your "work time" contributing to team coding projects (which is highly improbable as you'll spend most of your waking time working on your thesis), you probably won't get too many people with coding collaborative experience.
Overall, the post is pretty standard to what everyone needs. I'd chalk this up to inexperience in the workforce.
Maybe the problem is that you are trying to hire a PhD (whose entire value proposition is the notion of a complex, well-thought-out thesis) and you are trying to shoehorn them into the agile workflow, where people produce potentially crappy solutions (albeit something) and quickly iterate.
The whole point of a PhD is to NOT approach problems like that. Someone with a PhD has literally been trained to approach problems in a methodical, thoughtful manner, writing a thesis on the singular topic over a long period of time.
Is the problem here a fundamental problem with the PhD manner of approaching problems, or is it a problem with your startup mentality and hiring processes?
You make PhD sound like robots, only able to follow what they've been programmed to do. Persons don't have "entire value propositions", and they may welcome a different environment than what they've been doing in academia.
Ah, well, some PhDs do work with an agile mindset (fail fast with research ideas, most of them suck anyways). PhDs seem to be all over the map in how they approach problems.
I dunno what kind of a PhD the guy mentioned in the article had, but the most important thing I learned from my unfinished PhD was how to actually convey your ideas to others. Go from an idea to a paper, put it into a few slides, entertain the audience and make them follow you... And I had to do it several times, for each publication along the way.
Seriously this. Biggest thing I learned when I started my PhD was that communication skill is key. The interviewers in the blog post weren't looking for cutting-edge best theoretical approach, they were just interested in his process of thinking.
Isn't this basically the same advice you can find in hundreds of books and blog posts about how to approach a tech company interview? Is there something about having a Ph.D. that makes a person incapable of googling "what to expect in a coding interview" or buying a copy of CTCI before they head out the door to try to find a job?
As an interviewer, it's important to set expectations ahead of time so the candidate knows what they're getting into. It sounds like that wasn't done here. But it seems like the cat should be pretty far out of the bag about this style of question for anyone who bothered to prepare in advance.
> Don’t worry if you come up with a shitty answer. Just come up with something.
This is such a blatantly false statement. Nobody cares if you come up with a shitty answer in an interview. They expect working code which is optimal. You won't pass the interview if you write a naive, working solution. It's literally all or nothing. Maybe it's time companies make it clear to candidates.
I do find it remarkable the extent to which knowledge workers are discouraged from sitting and thinking about stuff. We seem to have a generation of managers who've more-or-less entirely conflated "work" and "collaboration".
While it looks like this company wasn't the place to try it, I do find myself admiring the student who had the fortitude to keep on working through the problem while the interviewers looked on. I hope he finds a better opportunity soon.
> I do find it remarkable the extent to which knowledge workers are discouraged from sitting and thinking about stuff.
That's such an uncharitable interpretation of the post.
It's a job interview, not a final exam. You're not supposed to sit there thinking for half an hour in order to produce a flawless solution. Maybe the interviewer was making progress, maybe they completely misunderstood the scope of the problem, who knows? The interviewers can't read your mind. Ask clarifying questions, make sure what would be the output of the algorithm in simple cases, then think about it enough so you can outline a solution.
I think this is great way to search for PhDs with programming skills for free. I am really getting sick of this kind of covered advertising method. It devalues hn.
PhDs are people too. They can have very different backgrounds. If the biggest, boldest, thing you see in the candidates with PhDs is always them having PhDs, you are painting everyone with a PhD with the same brush, you are having incorrect prejudices, and probably making a mistake.
Similarly, considering the author's self-declaration of being a TechStars alum, I also could pass judgement about them being a second-rate company since if you are willing to go through an accelerator, and you have not gone into YC, that probably means something. Luckily I know better than that. But I bet this superficial judgement would be more statistically accurate than painting all PhDs with the same brush.
We also found a mild negative correlation between "years spent in academia" and "can get shit done".
But rather than whinging about it, we opened a free "Student Incubator" program [0], where we work on ML projects with selected talented students, teaching them about collaboration in fast-paced SW projects.
>"The moment the PhD student stood up and bolted out of our office was the moment I knew we needed to write a blog post."
There is so much wrong with this sentence.
I think that most people and companies would have been quite concerned with an interview ending in this fashion. I also think most people would have taken a moment to reflect about the sequence of events that led to an interview ending in such a manner.
Not these folks though, their first thought was we "we need to write a blog post." It's amazing that they chose to take a one-sided interpretation of an awkward event as a marketing opportunity, complete with a picture of "Dr. O"
This blog post reads a lot like "users are using our product incorrectly, so we wrote them a blog post". Of course, the problem is almost never the user and almost always the product's UX design.
Just like with UX design, you should design your interview process with the people you're interviewing in mind. The interview process isn't valuable in and of itself; it only matters to the extent that it fulfills your actual goals—evaluating and recruiting suitable talent. If you are rejecting or scaring away candidates with PhDs and you believe it's because they approach your interview incorrectly (even though they might actually be a great fit to the company), change your interview process.
In this case, it feels like the company came up with an interview process and then decided that the interview was an end unto itself. That is not a good approach to building a strong team. Just imagine how a similar outlook would look like in your sales process—would you ever write a blog post about how clients with PhDs weren't understanding your sales pitch correctly?
Of course, it's possible that the candidates you rejected would not have been a good fit. But if you believe that, you would not write a blog post like this—instead, you would focus on improving your upstream recruitment process.
Either way, the important attitude is the same: you should treat this as a defect in your process and you should debug it. Find the root cause of the issue and fix it by changing your system.
We all understand that PhDs don't have to apply for startups. There are definitely other options to explore if you don't like speed and working in agile. But, when you apply for one and get an interview, you should be prepare to adapt. It is just simply a matter of researching into what you are expected and whether you'd want to work in that environment or not. It applies to anyone who applies for startup jobs tbh.
62 comments
[ 2.7 ms ] story [ 117 ms ] threadSounds like two weeks before a paper deadline!
I went through a similar struggle at my first interview out of school, at least in regards to an interview.
As I did non-majors courses at a community college before an intense 2.75 years at a full course university, I sometimes had to compromise on my course selection, as some things were taught in semesters I was unable to take them in.
In the interview(consisting of multiple interviewers, this was my first) in my first I was given the task to create a small class that would hand out read or write access, allowing many reads, and one write, and never a read and a write at the same time.
I thought for a moment, and started dictating my solution. Unfortunately, due to the above, I was never able to take an Operating Systems or Parallel Programming course. Thus, I had never even heard of a semaphore (I now consider that inexcusable, but perhaps understandable). As I set to work implementing two locking list structures, the interviewer would constantly interrupt me, trying to steer me elsewhere - a place I could not have possibly even known about - an unknown unknown if you will. This ultimately resulted in what was, to me, and I'm sure to him, a very frustrating 15 minutes and maybe 3 lines of code.
All this to say, I think sometimes there are just vastly different expectations on each side of the table, and it can result in mutually poor experiences in the interview room.
I would like to hear his side of the story. Maybe he was like "what kind of ridiculous problem do these people ask, I cannot wait to get out of here"
I deal with this a lot as a programmer who doesn't have a comp sci degree. I know what a semaphore is, I just don't call it a semaphore and I didn't know that it was called a "semaphore". I have been happily chugging along calling these "control variables"
No. No. No.
The difference is that production quality software needs to be usable by other people -- often many other people -- instead of just the author or a small group of researchers. In many cases it must work close to 24 hours/7 days per week under highly variable conditions.
This means it generally must have fewer and less severe bugs than a proof-of-concept or research prototype. This often requires extensive testing by end users or QA folks standing in for end users.
There are many examples of shipping commercial products and widely used open source programs that clearly meet this criterion but are certainly not easy to read, maintain or extend code.
Modularity in particular often fails to produce either readable or extensible code, rather producing numerous layers of confusing modules and/or objects -- "lasagna code" instead of "spaghetti code." It also can fail to produce "production quality software" as I define it above.
A more accurate description is: The startup risk tolerance is extremely high, thus it rewards quick and dirty over well thought out and solid.
This is often opposite to academic work, which needs to be well thought out and rigorous to an extent, though not necessarily in code quality. The code just needs to work for the experiment and be correct. It doesn't need to be maintained, extended, and debatably, communicated. This is also probably why startups tend to hire younger people, rather than older developers who've worked for places where the risk tolerance is very low and thus they take the time to make sure things are very solid.
Maybe it is an artifact of the fact that I work on computer systems research (where the focus is on finding the simplest solution that works) but the comments here make me wonder where you (collectively) are finding your PhD candidates!
I do agree that is often OK for PhD code to be terrible ("least publishable unit"), but that is also true of non-core production code at startups! ("Minimum viable product")
>Many PhD students can be perfectionists. But they need to remember that there’s an important compromise that needs to be made between the best solution and a good solution that can be used in practice.
Code quality doesn't seem to be a concern for Dr. O at least.
Appropriate older developers should have an even better handle on when (and how) to write good code and when (and how) to write bad code. Moreover, their bad code will be better, and produced faster.
Young coders are hired mostly because they are cheap and don't understand stock options.
Perhaps you misunderstand the agile process? Quality isn't something you ever actually achieve through the Feynman Algorithm (write down the problem -> think real hard -> write down the solution) regardless of whether you spend two hours thinking real hard or two years. Quality is achieved through iterative, continual improvement and polish. Otherwise, waterfall software would be the pinnacle of quality.
In other contexts "nice to use" is a better measure of quality, and I definitely agree that some iteration can help there. Even so, I'd caution that iterative development and a planned overall architecture aren't necessarily incompatible. And when following an iterative approach you have to be pretty careful to optimise for something that most users like, rather than a just bolting on everyone's pet feature.
Defensive anyone?
Notice the assumption here, that their 20-minute task is a perfectly reliable indicator of how the interviewee would behave in general.
The interviewee was right to leave them without saying a word.
God I get sick of the smugness. You can cut it with a knife.
I think the interviewee did make a mistake during his interview, in that he didn't read the room. They weren't looking for anything "cutting edge" and they know that they can't understand the in-depth and complicated work you did for your thesis. So what the interviewee should have done was focus on simpler and easier methods to start off with and kept it general. They already know you're smart from your degrees and from your CV, the focus of that segment of their interview was to see if the candidate had the communication skills available to express their ideas.
We have the same meetings with our stakeholders and during our project meetings. I don't explain the complex methods and algorithms used in-depth. I just keep it general and tell them how everything fits together (if I have performance indices then I also add that in to show accuracy). If the stakeholders want more detailed information like specifically how it breaks down and why I used X method instead of Y, then I can answer their questions and (if they showed continued interest) be willing to put together a report for them detailing the literature reviews and information related to that topic.
20 minutes isn't a reliable indicator of how someone can do their job. However, as the interviewee or the person in the "hot seat", it is your job to know how to sell yourself and to use your time effectively to maximize your chances of getting an offer. The decision makers (or in this case, the interviewers) has the hard task of figuring out which candidate is right for them. Your job is to make that easier. Once you get the offer, then you get to decide if you want to accept it or not.
And then after you get the offer, you reflect on your experience with that company so far. If the interviewers seemed to be focusing on different things than you're interested in or if it seems like they're going to be miserable to work with, then you politely decline the offer and move on to the next thing. If it seems like they have a condescending tone towards you and your work, then you can always just say "thank you for your offer but I'll be declining".
Here are some details that were cut out of the post:
- we have an extensive engineering interview process, typically 3-4 sessions. First chat, tech screen, take home assignment, and half day hackathon with a few engineers - sometimes we do the initial chat and tech screen together so we had a 40 min chat, and 20 min simple tech screen in this case - proposed of simple tech screen is to decide if this person knows basic coding/problem solving (eg Josephus problem) - goal is not to write code on the white board, but rather talk and discuss approach, general problem solving
This candidate was specialized in OR, so we asked an optimization related question (given an array of products and prices, how to spend all your money exactly). Instead of a discussion, the candidate dove into writing mathematical proofs, and after framing the goal of the exercise and a few hints, he ended up writing a 10-level deeply nested for loop...
Obviously I'm not trying to make generalizations, but I've interviewed over 100 candidates in my career, and there is a strong negative correlation with PhDs in particular, hence the article. Understandably, because they are trained in other methods, typically unsuitable to startups, as others have commented. But if they want to get into startups, I hope these tips would help.
If you are hell bent on the algorithms and data structures interview model, you just might not have PhD level problems to solve.
It's unclear what problem they gave the PhD was about. It could have been a hard optimization problem or some coding test.
What I got from this article is that the company shouldn't be looking at PhDs but rather at coders. At least that's what their interview seems to be geared towards.
Don't get me wrong, I kind of enjoy doing those things personally, but it doesn't seem like a great way to evaluate developers.
Come up with any solution, no matter how bad or inefficient, and then improve it. That's better than trying to find an optimal solution and not being able to come up with any solution at all before you're out of time. Then, you've demonstrated an inability to solve the problem. To put it in black and white terms, you've failed. An inefficient solution is much better than no solution at all. And it may be good enough if the data set is small.
Also, if you give me an answer in 10 seconds, efficient or not, you will impress me as someone able to make rapid progress. You can then improve it, putting you in a better position than someone who ended up with the same final answer but took longer to give me any solution at all. Working code is critical.
Discussion is important. If you've chosen an array over an ArrayList, I want to know why, such as an ArrayList of a primitive type being inefficient, or that we know the size ahead of time. Conveying that you know what you're doing is more important than a specific choice. Because if you know what you're doing, I'm confident that you can handle a different problem that turns up if I hire you.
It sounds like you are passing on good candidates due to them not being fluent in blabling. Which is fine if the job requires that ability (half selling to customers position), but is not if you need engineers to build system.
And then startups complain that there are no good people - they exists but get ruled out due to non technical presumptions like this.
That ability is not the same as being good engineer or problem solver. It is pretty much orthogonal. E.g. You can be good and both have it or not have it.
Basically,you test social skill for position that does not necessary requires that skill.
It has more to do with whether you think out loud (which I do and have a bit of advantage in such hiring process) and less to do with overall attitude of perfectionism when giving away finished work. I would not hire such person as consultant or support, but I worked with such people on positions that don't require much speaking with customers and they were great to work with.
Going back to the example of a doubly nested loop to determine if there are any two numbers in an array that sum to zero, it should take at most 1 minute for a smart engineer to figure out that it works. They don't have to speak before they've evaluated whether it works. The metric I use is: how long did they take to get to a solution that works, even if inefficient?
Back in the day my grandma was testing my foreign vocabulary. She had to learn that I knew the translations, just not at the speed she was used to.
Not every brain works in the same way.
If I ask to determine whether an array contains two numbers that add to zero, a competent engineer should be able to give the naive answer -- a doubly nested loop -- in ten seconds, or at most a minute. If someone takes a lot of time to give me any answer at all, it's a concern.
But seriously, giving programming problems like these to a PhD is probably going to scare the hell out of them, not because they think it's hard, but because they'll find it indicative of the work they'll be doing if they get and accept the position. Hopefully such faker filter questions are followed up with ones with real technical meat.
But this question lets me learn a lot about the candidate:
- How quickly did they come up with any working solution?
- How long did it take them to reach the optimal solution?
- Do they know time complexity?
- If they're using Java, did they use a plain array or ArrayList? Do they understand boxing and performance?
- Do they understand the range of an integer? I'd tell them that the array represents the population of the earth every year from 1900 till today, and see if they realise that they need a 64-bit integer.
A competent engineer or just one who's practiced HackerRank questions and mastered the cliche choreography of the SV interview format?
"Think out loud! We want to know how you think! Ask clarifying questions! Start with the naive solution! But oh also please do all of these things within an arbitrary time restriction known only by me if you want to impress me."
If you don't know the answer to a question, start talking about what you know in the general vicinity of it and see what happens.
You might get a hint for a direction, or the person giving the test might drop the question entirely because you brought up something more interesting (so careful there!), or they might decide that, while you cannot answer that particular question you're still sufficiently knowledgeable.
As long as you keep talking good things will happen compared to silence. A bad answer is a lot better than no answer, always. Also good is asking suitable questions back. If you're lucky you can derail the professor from giving a test to having an inspiring conversation about the subject, without grade loss.
So, this is a skill that is (indirectly) taught and practiced in academia. Thus I don't get the point of academia people having problems with this.
Let me get this straight, you invited 100 engineers in for an onsite after qualifying them with a phone screen and their resume? You conducted 100 Google style whiteboard algorithmic puzzle interviews just to find 1 single engineer to work for your startup?
These facts also resonate and could be generalized as something might also be terribly wrong with your "precious" interview process as well.
I'm an engineering Ph.D. candidate focused on optimization and forecasting (tldr: large amounts of data analytics). I also code extensively in R. As a Ph.D. candidate, you're supposed to be the expert with the cutting-edge knowledge in your field. You're supposed to know what options are currently in development, the pros and cons of each methodology, and the significance of the final output and how it fits in to the big picture.
Most PhDs focuses on developing and implementing best methods (theoretically) and continuously improving accuracy of the final results. Methods that haven't matured enough to be implemented today, but will tomorrow. Timing and speed is still important in academia, but (and this is my opinion) more focus is spent on implementing the theoretical approach correctly.
From my observations, industry environment focuses on implementation speed (not the theoretical approach development but rather just "coding it in") and making it work to "solve" the problem. Focus is spent on implementation speed over maximizing accuracy. Therefore, simpler and less computationally intensive (and in most cases, easier) methods are preferred over more complicated methods.
I guess to sum up my point, academia focuses on cutting edge technology and continual development of some of the best methods available. That's how you get more of your research published and further your academic career. Private industry would love to implement cutting edge technology, but focuses more on implementation speed which can sometimes impact model accuracy. From my understanding of the article, it seems like the candidate they interviewed wanted to show the extent of his knowledge whereas the interviewers were looking for the general "overview" and discussion approach and thought process (his mistake probably for not reading the situation right, but just what I gathered).
Oh also to add as a P.S., most PhDs spend time focusing on theoretical problems and methods. Which usually means that knowledge regarding full stacks, infrastructure design, supplemental technology (e.g. D3), etc. can be a bit lacking at first compared to someone who spent the equivalent 4/5 years working in the industry and who has gathered experience using those technologies. Also PhD is really a "solo" mountain/task to conquer. While you do have colleagues and lab mates with you, everyone is usually doing their own research. So unless you spend time outside of your "work time" contributing to team coding projects (which is highly improbable as you'll spend most of your waking time working on your thesis), you probably won't get too many people with coding collaborative experience.
Overall, the post is pretty standard to what everyone needs. I'd chalk this up to inexperience in the workforce.
The whole point of a PhD is to NOT approach problems like that. Someone with a PhD has literally been trained to approach problems in a methodical, thoughtful manner, writing a thesis on the singular topic over a long period of time.
Is the problem here a fundamental problem with the PhD manner of approaching problems, or is it a problem with your startup mentality and hiring processes?
As an interviewer, it's important to set expectations ahead of time so the candidate knows what they're getting into. It sounds like that wasn't done here. But it seems like the cat should be pretty far out of the bag about this style of question for anyone who bothered to prepare in advance.
This is such a blatantly false statement. Nobody cares if you come up with a shitty answer in an interview. They expect working code which is optimal. You won't pass the interview if you write a naive, working solution. It's literally all or nothing. Maybe it's time companies make it clear to candidates.
The truth is if you can develop an optimal solution super quick during the interview, you're better than someone who can't. It might be like
optimal solution, quick > working solution, quick > no solution.
The problem is if you can't do optimal quick, then you end up spinning your wheels and end up with no solution in the allotted time.
While it looks like this company wasn't the place to try it, I do find myself admiring the student who had the fortitude to keep on working through the problem while the interviewers looked on. I hope he finds a better opportunity soon.
That's such an uncharitable interpretation of the post.
It's a job interview, not a final exam. You're not supposed to sit there thinking for half an hour in order to produce a flawless solution. Maybe the interviewer was making progress, maybe they completely misunderstood the scope of the problem, who knows? The interviewers can't read your mind. Ask clarifying questions, make sure what would be the output of the algorithm in simple cases, then think about it enough so you can outline a solution.
Similarly, considering the author's self-declaration of being a TechStars alum, I also could pass judgement about them being a second-rate company since if you are willing to go through an accelerator, and you have not gone into YC, that probably means something. Luckily I know better than that. But I bet this superficial judgement would be more statistically accurate than painting all PhDs with the same brush.
But rather than whinging about it, we opened a free "Student Incubator" program [0], where we work on ML projects with selected talented students, teaching them about collaboration in fast-paced SW projects.
[0] https://rare-technologies.com/incubator/
There is so much wrong with this sentence.
I think that most people and companies would have been quite concerned with an interview ending in this fashion. I also think most people would have taken a moment to reflect about the sequence of events that led to an interview ending in such a manner.
Not these folks though, their first thought was we "we need to write a blog post." It's amazing that they chose to take a one-sided interpretation of an awkward event as a marketing opportunity, complete with a picture of "Dr. O"
Just like with UX design, you should design your interview process with the people you're interviewing in mind. The interview process isn't valuable in and of itself; it only matters to the extent that it fulfills your actual goals—evaluating and recruiting suitable talent. If you are rejecting or scaring away candidates with PhDs and you believe it's because they approach your interview incorrectly (even though they might actually be a great fit to the company), change your interview process.
In this case, it feels like the company came up with an interview process and then decided that the interview was an end unto itself. That is not a good approach to building a strong team. Just imagine how a similar outlook would look like in your sales process—would you ever write a blog post about how clients with PhDs weren't understanding your sales pitch correctly?
Of course, it's possible that the candidates you rejected would not have been a good fit. But if you believe that, you would not write a blog post like this—instead, you would focus on improving your upstream recruitment process.
Either way, the important attitude is the same: you should treat this as a defect in your process and you should debug it. Find the root cause of the issue and fix it by changing your system.