I've often thought about this, and based on my own experiences this is the process I would like:
1. Test language-specific knowledge - it's not a bad thing to not know all the small quirks or details of a language since they're generally not too useful, but when someone does know them it tends to be a good sign that they really enjoy coding and learning. And of course there is a certain minimum amount of knowledge that should be required - listing Java as "proficient" on your resume while not knowing the difference between abstract classes and interfaces might indicate a problem.
2. Design question - just a simple toy problem like "how would you design the book management system for a library?". It's easy to use such a problem to probe into some different aspects of programming like concurrency and databases, just to see how much the person knows.
3. A not-crazy-hard algorithms questions - people often say algo questions are irrelevant to actual work, and I agree with that. But I think algorithms are really a core part of the CS curriculum at every school, so getting completely stuck on a medium-difficulty algorithm question should raise some flags.
4. Resume-specific things - it's always nice for an interviewer to show that they've actually read your resume, and it can be a good way to convey some strengths that aren't otherwise evident.
I guess my philosophy is to interview in a way that can test the depth of a candidate's knowledge while not being obnoxiously tedious or memorization-focused. i.e. someone who has written a lot of production code should do better in an interview than someone who's just memorized every problem in Cracking the Coding Interview. So, ideally with the screening round clearing the first part (language-specific knowledge), and then 2 or 3 subsequent interviews that cover design and algos.
That’s not true. If you have ten years experience with Python, but don’t know about, say, @property decorators or how variable scoping works, that raises some red flags.
I think it depends how you do it. When I interview, part of what I do is try to find quirks that they know about, but not count the ones they don’t know about against them.
I think you can quickly get a sense of depth, potential, curiosity, and passion if you can get them talking about quirks and their opinions on those quirks.
For example, if someone says that they once attached a debugger to the JVM to confirm a pathological issue arising from benign looking Java code and found that the JIT compiler falls apart when a certain construct is used, I’m probably going to hire that person, even if we have no Java code in the company - assuming I can also confirm that they are productive and don’t just waste time going down deep rabbit holes.
How has that been working out for you? I used to think the same and I'd ask a "programming passion question" right before a simple coding question, and then I'd run into these amazing bullshitters who could talk shop but couldn't write fizzbuzz. But they'd come up with a bunch of excuses as they were writing the code as to why ("oh my company uses this other framework so I forgot how to write a for loop"). So sadly I stopped paying attention to that part. I only ask it to make the candidate comfortable before we dive into code.
It seems you're doubting my story. Maybe you think my threshold for "that level of detail" isn't sufficient, or whatever candidate sourcing we use doesn't filter these people out.
The only thing I'd caution (particularly about #1) is that being able to parrot facts — even ones you know cold — on-demand in a high-intensity interview setting can be very different from using those same facts day-to-day. If someone can't describe the difference between abstract classes and interfaces, how confident are you it's because they don't understand it and not just because they're horrible at being put on the spot in an interview-style environment?
Hell, I can barely "hello world" in languages I've been actively and productively writing for months or even years, without context (i.e. existing code) to draw on and/or tool assistance. I've got a better than 50% offer rate on interviews in an over 15 year long career but in one of the ones I flubbed I blanked on the syntax for invoking methods on an object in the language I knew best, at the time, and wrote some other (more common) invocation syntax instead, which can't have helped. Ask me JS scoping trivia and I guarantee I'll screw it up, despite having written JS and TS as my main language(s) for 3ish years now. Look, I stick to a subset of that language precisely so that I can ignore that shit. We can just skip the interview and go straight to the rejection if you're gonna do that. Doubly so if you don't tell me you're gonna do that so I can drill on it before hand.
Coworker and manager feedback, and my compensation over the years, say I'm pretty damn good at adding value despite this. I've been considered the "smart one" and the go-to guy for hard or weird problems. But I will struggle to produce something that will even come close to compiling on your whiteboard, in a language I wrote 300 useful lines of yesterday, unless I spend a couple days basically flash-card drilling for your test. Go figure.
Not being able to describe the difference between abstract classes and interfaces is a huge red flag for anyone working in OO Software Development. You can't be serious?
Only dipshits working on shitty OO projects care about the difference between an abstract class and an interface. Let me guess - you also think algorithmic complexity analysis is overrated.
They're functionally different tools and knowing when to use each is important. I actually studied theoretical CS so of course I think complexity analysis is important. Your completely tangential tirade is noted though (Y)
For #3, it’s not really a red flag. I am guessing you are a fairly recent grad?
Most people who learn algos & DS never have to implement them on the job, and often haven’t actually written their own since their CS classes. If the question is any deeper than “what are some good and bad choices of algorithms for this problem”, you’re selecting more for who has just graduated or has been grinding leetcode and doesn’t have performance anxiety than you are for competence, expertise, and productivity.
I know folks who got rejected from the leetcode companies and went on to build major products from scratch. The 0.5% of work that requires them to have to implement something like this manually, they’ll just grab their CLRS book and refresh their memory long enough to solve the problem and probably forget it again a week later.
I also know folks at these companies who started fresh out of college, passed that style of interview, worked there for 7-10 years, moved up to senior or higher in the engineering track, and openly admit that they would be a deer in the headlights if they got any of the interview questions. Without going back to study and practice those, they know wouldn’t pass an interview for a junior dev, let alone one for their current job (where they are currently considered high performers).
The big problem is that this style of interview filters for people who pass this style of interview, and then those new hires are asked to interview others, so they interviewed in the style that they were given and passed.
Yeah, I'm currently a grad student so there's definitely some bias haha.
The kind of question I had in mind would be along the lines of "say you have a tree, how could you find the nearest common ancestor of two nodes in that tree?". I actually got this question during an interview recently, and I felt it was of a suitable difficulty where I could figure out a solution without any specific knowledge other than what a tree is (which most developers should know).
I'm sure I'm still young and naive, but my line of thinking is that a question like that is good for having a person show their thought process when figuring out a problem they haven't seen before. Design questions are a bit different because they tend to just require a rehash of a codebase you've worked with before.
A more realistic question is this. Italy is mandating XML business to business invoices starting from 2019-01-01. You're given the spec and a database schema. The question is "which fields are we missing and how should we ask customers for those information?" Extra bonus if you know GDPR and suggest which fields are sensible information, not that I'm expecting that somebody out of school can make that assessment.
In the 20+ years since I left university with a CS degree, not once have I directly used any of the algorithmic stuff I learnt commercially. (Obviously I've used sort routines in libraries.)
It should be clear about what they're looking for, should be collaborative (I want to see if I can work with them), it should be respectful (ending it early for not a good fit is not a good reason), they should allow for you to talk to your strengths and not let it be a we are pushing you to failure.
I like asking candidates to give me a code review on a pull request. Because of IP concerns, I normally do this with an OSS project I’m involved with.
Sure, they don’t know the codebase, but they won’t know the codebase that they’ll be working on in a month if hired either. I want to see how they think about creating software and whether they notice potential defects or tricky areas.
My apologies in advance that this may not be the response you were looking forward to on forum question.
But I discovered an older post online on the Hacker News site some time ago for a Remote Junior Programmer/Assistant at Luma. I noticed you were looking to make your first hire and find someone that has some knowledge in technologies such as Python and Linux.
Well, I had completely forgot about viewing it and so I was checking out Hacker News job board today and saw it was for a remote Part/Full-time opening available at the Luma in the NYC area.
Well I'm someone who enjoys coding and learning with Python as far as learning purposes/hobbies go. Plus, I'm someone who has been going the self taught route trying to break into the IT field/Python Development world. I also have interests in some of the things mentioned in the post such as finance/trading and business.
But, more than anything.. I'd looking to further learn and grow in my skills as far as Python development goes. I don't have much experience with Django Web development but feel I can learn and pick up on it as well as with any other technology requirements quickly.
After quickly reading your post on Hacker News I was encouraged and interested to reach out and contact you for more info.
So with that said, here I am..and I wanted to inquire to find out if this opening is available? Or has it been filled? Also would you allow for training of junior developers to get up to speed? I had to ask because but I wasn't sure if you meant junior or for more seasoned developers.
Also, you have a contact number to learn more about this position or information on the things you require in regards to the nature of the job to increase one's chances to be a part of your startup team? Any help in this matter will be greatly appreciated. Thank you
Feel free to contact me at: pydeveloper22@gmail.com
Most interviews have asymmetric costs. There’s a potential payoff for both parties, but costs are higher for the interviewee (time off, prep, work done).
I recommend trying to make the cost/benefit more symmetric. One way of doing this, is offering compensation for real work. Essentially, hire interviewees to do a small amount of useful (a day?) and evaluate them based on that.
This isn’t going to work for all interviewees, but where possible it feels like a better way of doing things to me.
Not going to happen - I think the supply of software developers has started to outweigh the demand, so there's no way that companies are going to offer compensation to interviewees. If anything, the difficulty bar is just going to continue to go up. If there were a shortage there wouldn't be so many of these silly hoops to jump through.
It's not really worth it to interview nowadays. Practicing leetcode for months, flying, doing all day onsites, when all the company has to do is interview you for a few hours.
We've been issuing take-homes lately, and we ask that candidates spend no more than 3 hours on them. Well, it turns out almost no one spends only 4 hours on them... The most recent candidate we interviewed spent about 3 days on theirs judging from the commit logs. It was a fully fleshed out application with unit tests and everything. After all that they still have to pass the all day on-site, lol. Anyway I spend all of 2 minutes looking at each submission
>Not going to happen - I think the supply of software developers has started to outweigh the demand, so there's no way that companies are going to offer compensation to interviewees.
I was thinking the opposite - that no employed engineer is going to bother with a multi-day paid contract for a new position, so the only engineers you will attract are those without a job and no pending interviews.
Your company might be inadvertently attracting poor candidates. I imagine a lot of good candidates (like myself) would just pass on a 4 hours of homework for a random programming job. And good candidates almost certainly won’t spend 3 days on it.
Remember Joel Splosky’s classic observation: all the good programmers already have good jobs. The ones on the job market who apply to jobs are usually the bad ones.
It's more like, given a good job market and low barriers to entry, the percentage of candidates who are not qualified will greatly outnumber the good candidates. Good programmers might be looking for work for any number of legitimate reasons, even in a hot market. Doubly so if you are offering a premium on compensation or otherwise trying to make yourself more attractive to applicants (which four hours of take home is the opposite of). The signal to noise ratio is just lower the less attractive you make applying to your company.
I would argue this in reverse. The costs are all on that side of the table. I love interviewing. Love it. I am honestly not being sarcastic.
I am more than happy to take the time off just for the opportunity. If it doesn't go well or the position is not a good fit then I go back to my current job.
In my current role I had what I consider the best interview I have had to date. I had a brief non technical video
interview with the software director. Then I was assigned a programming task, essentially to create a site focused on a specific task using a given set of technologies. Once done, I sent the interviewer the url to the site and a link to the code on git. The next day I had a code review and video interview with the original director and several Sr. engineers to review my code and answer relatively technical questions on my design decisions and code.
I thought this was perfect, they did not waste much time on the original interview, essentially just determined I was not insane and would be an ok fit for the team. Then via the assignment they were able to see me build something from start to deployment including database development and server deployment. One of the required technologies was a javascript framework I had never used before, they knew that but I spent 8 hours (I really wanted the job) reading a book on it before starting the project and they appreciated it. I had a couple of options on server side stack and they were able to observe and question my decisions. In the code review they were able to see how I handled criticism and reacted to stress.
Whole process from scheduling the first interview to the job offer took 4 days. No white boarding, no obscure technical jargon. They met me (virtually), learned if I could code to their standards and made a decision. In my mind it was great. Via the same process they interviewed someone else I recommend and very professionally and politely turned them down, someone who in my mind was a pretty solid engineer but he just lacked the specific skill set they wanted. While creating the requested application, he knew he was not going to get the offer, which is great as you as a developer are then able to determine for yourself if you are a good fit for the role. No mess no fuss.
I've considered this approach, but I worry that it consumes too much of the candidate's time. That it ends up favoring people who have the time and the financial freedom to do a full day of unpaid work.
This is a valid criticism; however, they did not specify I had to turn it in the next day, I could have kept it over the weekend and not taken a day off of work. Several people kept it over a weekend with no penalty. I also could have worked on it at night after work. I definitely preferred this option as opposed to having to take a full day off for an in person interview and another for the follow up. I only took an hour off for the first interview as it was at 5 and I just ran home and threw on a suit jacket and a couple hours for the final interview as it was close to the same time.
If I had already worked in the js stack they asked me to use the entire project build would have been 5 or 6 hours and that's only because I really wanted to impress, I could have knocked it out once my kids went to sleep.
But as in everything it depends on the person and their life circumstances, we all have different things going on.
I have been interviewing recently and started turning down any coding assignment that will take more than a couple of hours. To get a reasonable number of offers to enable you to negotiate requires going through the interview process for several companies since some percentage of them will turn you down. After a phone screen I don’t have enough information about the role to be able to potentially waste 10-20 hours on a toy problem.
A few companies gave a simple api project or a timed hacker rank and those feel reasonable. An entire web app with database, rest api, front end and CI pipeline following production code best practices does not.
While technically correct, I think most people agree that it is better to field several offers simultaneously so you can choose the best culture fit for yourself as well as being able to provide a concrete example of someone offering you a higher salary to most easily justify your counter-offer.
Negotiating a salary is a situation I have never been in as I have always told them up front before interviewing how much I want to get paid after finding out the salary range. Whenever a recruiter calls, I ask the salary first and tell them where I want to be on that range before even sending a recent resume. Not that negotiating for more if I have several offers would not be great, just not something I have experienced. I interview, they already know my number as I have already asked the hiring salary range and told them what I want. If I pass the interview they offer me my original request and we move forward. I could see how if one was interviewing with multiple companies and had multiple offers how one could seek to play them against one another for a potentially higher amount. Just hasn't happened to me.
I’ve always gone through local recruiters that have relationships with the hiring manager. They know the salary range, the interview process, and where other candidates fail because they get feedback from both the candidate and the hiring manager.
How is that any different from taking a whole day to do a normal onsite interview? Doesn’t that also favor those with the ability to take a day to jump through hoops?
"Normal" for who? It's normal for fresh graduates interviewing at the biggest tech companies, but that's not normal anywhere else in the world or for other jobs.
There is an entire class of developers you really want to hire, that don't want to spend that amount of time unless you are one of 10 companies on their list.
My most enjoyable interview was for an internship in college. I had a take home coding challenge where I had to write some simple code to fetch information from an API using whatever language I liked. I was given a week to do it, but it only took me an hour or so to meet all of their explicit requirements. I liked that there was no time pressure in that regard.
After the week was up I went into the onsite and in the "technical" portion of my interview two engineers went over the code I had written for the assignment and asked me about design choices I had made and what I would do if constraint X was added or feature Y. It was all very open ended and much more of a true discussion than an interview which I really appreciated.
I think this kind of format is ideal for interviews. The assignment requirements were simple enough that you could fulfill them easily and in your most comfortable language without any time pressure, but you could also go above and beyond and show that you really knew your stuff. For example, in the instructions they didn't explicitly ask for error handling on the input, but both of the engineers I interviewed with really liked that I had included it. You weren't penalized if you did just the bare minimum because you had the opportunity extend and build on the assignment during the onsite. I felt enabled to showcase my knowledge and justify my design decisions and that that effort was rewarded.
I would love to be interviewed like this. Even though the candidate does end up spending time doing work outside of work, they'd end up doing that anyway if prepping for an algorithms interview. At least with an assignment you can be finished instead of doing endless leetcode prep. Plus it gives you a chance to practice relevant software engineering skills in a different environment.
until you have to endure the fifth rejection and realize you have to multiply this effort for every interviews. Whereas you can prep for an algorithms interview once, and any additional prep is incremental and focused by prior interviews.
All the coding challenges I ask of my candidates have been real-world problems I had to solve in the past where I though, "huh, this would probably be a good interview question". I also pay candidates for their time, except for the initial screening test.
While I appreciate that it can be a waste of time, the way I tended to look at this was that if the coding test / task was minimal, it was an acceptable trade-off for both company and candidate. Like, that task should take an hour or so.
The problem it's trying to solve is just the surprising number of bad candidates that apply. We're talking about people with decent CVs, a bit of work experience at different companies, some open-source work etc., and who had a good chat on the phone – sometimes they just completely mess it up when faced with a task like "combine these JSON files to make output that looks like this sample". Things like code that literally doesn't work, or code that uses totally inappropriate tools. We want to avoid wasting both their time and our time by bringing them onsite for an interview that isn't going to go anywhere.
The risk with being "hired to do real, paid work remotely" is that it relies on finding people who are in a position to do that, and this ends up excluding a bunch of good candidates. Not everybody is in an employment situation that will allow them to do external work, and it disincentives people with families or other commitments.
The process of applying for a position shouldn't be over-burdening on a candidate. I agree that it's unfair to expect lots of stupid work, multiple-day interview processes and so on. But I don't think it's unreasonable to expect that a candidate who wants to work with a company can afford to spend a few hours in total as part of that process.
This is exactly how we do it at my company. The first interview is mainly for the candidate to get to know the company, products, tech. If we think there is a fit, we send them home with a coding task, choose 1 of 5, not expecting more than 100 lines of code, or more than a couple of hours (though all candidates have submitted more and spent more time). There is no time expectation, but they usually finish in a week or less.
They then come back for a code review, walk the team through their code and explain it. This tells us a lot about their ability, but also their communication style and clarity, how they approach problem solving in general and even in which problem they chose.
I chose this approach because 'coding under duress' during an interview does not offer a valid sample of what the candidate can do. Taking the time they need and working in a comfortable environment will give a much more realistic result. Coupling the take-home with an in-person code review weeds out those who may just Google up or borrow code from elsewhere.
This sounds exactly like one of the best interviews I had when I was searching for my first dev job, and I've tried to emulate it whenever I've been involved in hiring new people.
A good template is:
- Simple yet relevant take home coding test with a relaxed timeframe
- Technical portion of the interview where the candidate is asked to add a simple feature to their existing tech test.
- This is followed by a more general interview.
None of these sections should take more than 1-2 hours.
That's great to hear – my previous company did something pretty similar to this, except the onsite "technical" portion involved actually pairing with one of the team for an hour to add one of those features, then discussing that process with a second engineer to describe what we'd done together and why.
I generally think this works well for a few reasons:
1. The take-home task lets us see if candidates meet a minimum standard of competency and generally follows good practice – do they test the solution, document it, write clear… of course, I think it's important that the size of this task is limited to avoid wasting candidates' time.
2. The on-site interview helps to make sure the candidate is able to discuss the work they're doing, collaborate with other technical staff, and communicate technical ideas well.
It feels like the overall goal of an interview process is to ensure that the candidate is at the skill level they have communicated, and that they can work with others.
The coding challenge sounds a lot like what my company does (including the time contraints and how long it really takes to code it), but the interview is definitely different. Our coding "challenge" isn't a challenge at all, it's just a test of basic developer competency, including following the spec and making things work. There isn't a lot of room to ask about why things were done a certain way.
You've got me thinking that maybe I need something a little more complicated that will give room for those questions.
With this approach you'll miss good senior people; I have dependants and very little free time outside work. Last time I was looking for a job I skipped everyone who wanted me to do prep/homework. If the other interviews hadn't worked out I'd have gone back to them, but I didn't need to.
Senior people will be able to do the two hour coding homework in way less than two hours. If someone can't manage this as part of an interview process, it might be a sign of their time management abilities.
It means you don't do unnecessary work, you avoid pitfalls and traps you have seen before, you don't over-engineer but you keep it as simple as possible. It also means that you push back against non-constructive requests from business and management and focus your time and effort on what matters.
I also avoid anyone who wants me to do a coding exercise for a few hours over the weekend.
If you want pay me for that time, and make it not a dig and fill hole exercise - find a OSS project that needs some contributions. Decide that you and your team will do thsoe via your ongoing interview process that hires people before the need arises, so you can all take your time selecting and onboarding people.
>you don't over-engineer but you keep it as simple as possible
It should also mean you don't under engineer, but that is extremely hard to test for because that involves knowing the business and your customers and having a reasonable estimate of where future needs will be. I'm not sure how you would test for lack of under engineering, especially since any interview task would be a perfect case where you practically can't under engineer since it is guaranteed throw away work. Maybe asking during code review for how you would've done the solution different if you knew that in the next quarter you would likely have to implement either feature A and B or C and D (but you didn't know which yet).
Contributions to a new OSS project that show significant programming ability are very time-intensive. Do you know anyone who has successfully implemented this scheme?
I could set up one of our OSS projects with first-contribution issues and help walk someone through but I suspect most people would rather not spend the time. If it takes 4 hours with back-and-forth in the PR (simply because I may be asleep when you commit and you when I comment) that's a big request of any person.
The main antagonism here is that the hiring company wants to minimise its effort in getting great devs, and the devs want to minimise their effort in getting great jobs.
The first point to note is that constant marketing is the first, best solution to this. I would definitely put more effort into joining a well-marketed company (StackOverflow?) than J.Random Inc. So both me and J.Random Inc had better put some effort into standing out from the crowd.
OSS is one seemingly good way to do this - and having a reasonable Github account is something I would say should get you through most interview filters (ie if you are looking for a Python dev, and they have commits to say a flask extension or bits of salt-stack, then you can blip over the Fizzbuzz and whiteboards.)
But marketing is simply a way to get past other people's filters. "Networking" is good, whatever that might mean, and being a good citizen works. but these are as always, long term, constant application efforts. And of course, costly.
Secondly get over the "we only hire the best of the best of the best". If that's true then like the SEALs you clearly have a six month training programme that takes the best and shapes them into effective teams, fully paid while they learn - yes? And the training staff for this programme are pulled off current fee-paying projects to keep up the standards yes?
Otherwise your best-of-the-best is so much auto-trumpet blowing. So please leave that at home. Focus on process not heroes.
The interview by takehome project is a big problem.
Yes it clearly weeds out candidates - mostly by making the ones with options elsewhere go elsewhere. (Even Google, with its firehose of applicants, seemed to realise this was a dumb idea and instead used the MIT graduate program as a filter instead).
My main issue with interviews-by-homework is it is usually a hole-dig-and-fill session as the problem has been done by dozens of people before (often you can find their solutions on github). My time is then valued at less than zero. As a filter before we get to interview, it really is just an artifical hurdle. All you are saying is that "we have soooo much choice we can make you dance before talking to you" - if it works for you great. Your marketing is working (see above)
IMO a better approach is to find different existing properties that you can filter to get applicants to the interview.
Sometimes these are "Alpha Male" filters like 1st class honours at MIT. Other people on HN stand by different filters - I seem to remember someone saying one of their best hires had taught themselves coding whilst getting off drugs in a Glasgow slum. How you filter for that is harder than the MIT problem but I bet its a less tapped seam than MIT.
Overall, a good interview process is probably focusing on the wrong end of the problem. If you have a new open position and then you start looking its too late. Be a good OSS citizen, be involved in the community of developers (OSS and elsewhere), find ways to reach out to unusual developers, or people not marketing themselves, and keep the process going constantly, with suitable investment. The general Big Co idea of "you cannot put out job ads before you have a signed off budget and position" tends to make this a problem. A solution would be to take the expected Agency percentages for each new hire (between 5-20%) and put it into a centralised "developer evangelist" team that just gets people through the door.
This way when you need to open a new position, you will have four people in mind you already want to hire, and you can just go to Starbucks for an interview.
An extra point - senior means FU money. Often people have degree of authority and sway with others in authority - often by right of tenure, organisational position and friendships.
But coming fresh and new into a company, it's hard to just have your opinion or taste taken seriously - especially where that opinion will be unpopular (ie your code base needs rewriting from scratch)
So FU money is needed to avoid the tendency "just to go along with it for a bit"
Hire rich people is hardly a great line, but it will have benefits.
Hiring desperate people with no other options does not sound like you are winning of course
They can, but why would they? They're probably busy, possibly with family. These aren't graduates desperate for a job, if they're the ideal employee you're looking for they'll have a comfortable fallback in continuing what they're doing and plenty of other companies that don't want them to do homework.
What if 9 great companies give just 2 hours of homework and 1 gives none?
My experience is quite opposite. I used to be much faster 5+ years ago. I would just jump to a conclusion, execute, and see what happens.
I rarely start anything these days without thinking about the problem thoroughly. It results in longer development times, but better designed, and more robust software, or so I hope.
This is the difference between building a solution that works today and building a solution that works tomorrow. A good interviewer would prefer the latter, but would recognise that it takes longer.
> With this approach you'll miss good senior people
Aren't you already spending many hours researching a job as the part of the process? Aren't you already taking a full day off work for the interviews? Why is that time all fine to spend, but not fine to spend a couple of hours on doing an assignment?
I have the exact opposite opinion: a coding test is for a candidate you already want to hire and introduce to your development team, to have a look at their coding style.
You don't want to get your developers together for any random loser who'd never want in your team anyway. So the coding is a late stage filter.
If life's too short to continue with the final stages of an interview process, I don't know what to say. The whole idea, if you do it right, at least, is that you only give this test to developers that you already know you want. That means that for those developers, this is not one of dozens of tests they need to do, it's the one test; or one of a few, if they're that lucky.
And if they're not interested in putting any time or effort into a new job, it seems to me they're probably not that interested in that new job in the first place.
But it seems weird to me that so many people here consider a whole day of interviews to be totally acceptable, while some actual coding is not. Compared to some of the interview horror stories I often read here, this seems vastly preferable. I actually get to show what I can do in a realistic setting, and I get to explain why I do what I do. I get to meet the other developers and talk to them.
And with the best programming tests, I actually get to learn something new. For the best one, I had to learn React (which I had no experience in), build a game board on which you can put obstacles, and write an algorithm to find the shortest part through the resulting maze. (I ended up rejecting that job because it was too far away and not enough pay, but I'm still really happy I did that coding challenge.)
Depends for many people there is not a shortage of jobs to apply for, so coding a path through a maze which is just regurgitation rote memorised algos is a filter for us.
I do agree about a daylong set of interviews with 4 or 5 people that's also a filter id use.
I guess it's also an issue of what you want. Are you fine with just any job, or do you want something specific, or something challenging? Do you want to be the smartest person in the room or do you want competent co-workers?
In my experience, these sort of tests do often seem to be used by the more interesting, more competent kind of companies. The first company where I encountered this (14 years ago), had only highly competent developers, many of them committers to various open source projects. About the recent one where I wrote a customised version of A* (because of the presence of wormholes, which made it not remotely rote for me or for most people, I'd expect), I of course don't know how it would have turned out had I taken the job, but if they select for this kind of thinking, I see that as a very positive signal about the kind of work they do and the kind of people they hire.
At companies that didn't have this kind of test, the quality of their developers was a lot more hit-and-miss.
At least for permanent employees; I've never encountered this for freelancers.
Not really. I've never needed to research a job beyond a bit of Linkedin/glassdoor stalking and reading of corporate guff.
I've never had a full day interview for a company I'd actually want to work for (based on what I've learned while grinding through the interviews) so I'm beginning to take requests for those (along with requests for advance coding exercises) as a recruiting smell.
Interesting. Would you mind sharing in which country you work (or if America, which state), what your field of interest is, and how long you've been in the business?
My team has full-day interviews and I'm curious to see who has been self-selecting out of this.
To offer a counter-point, I have a level of self-confidence, worth and respect, I wouldn't join a company without a full day of interviewing with them to get an understanding for the company.
I don't know if people are getting hung up on the term "full day" but I'm not sure I've ever heard of people being hired for senior (or even not so senior) professional roles without having multiple in-person interviews.
I suppose there are distributed companies that just do things over video link these days (and I've interviewed people over a video link when necessitated by travel schedules or people being in different locations). But some number of in-person interviews over the course of 4 or 5 hours is absolutely standard in my experience.
I'm a freelancer, so that's a bit different, but I admit I'm surprised at how brief and superficial the intake to hire me usually is.
For permanent jobs, it's usually an hour long interview followed by a programming assessment; either take-home and then presenting to their developers, or codility. But for freelance work, none of that. Just a talk and they hire me. But of course if I turn out to be useless, a freelancer is easy to fire.
Video interviews are definitely less good than in-person from my perspective. But between the fact that the teams asking me to interview candidates are pretty distributed and have people who travel a lot as I do, it's pretty hard to get all the "right" people into an office on the same day.
As for hiring freelancers, I'm not involved in hiring programmers but we use external people for various other things. We've try to use people we have experience with but it's mostly not a big deal. If we don't like the work they do, maybe we're out a few bucks but we just don't hire them again.
"Full day" absolutely matters though, especially if you want to attract passive candidates.
If I'm unemployed, then full day is fine.
If I'm really keen to get out of my current job, or really keen to work for you, then I might do it.
But if I'm just exploring opportunities, or I have other options in progress, then I'm not taking a day off just to interview.
With any interview when it might be really obvious 10 minutes in that it's not going to work. If I've scheduled that during my lunch break, or for an hour before/after work, then that's a small cost.
If I take a day off work to interview, then that's costing me in the order of $1000. If I don't know whether I want to work for you, then why would I do it?
And I definitely can't do that for 5-10 different roles that I might apply for.
A full day is also quite hard work. Interviews are stressful. Dealing with people you don't know, trying to make sure you don't do/say something stupid, it gets exhausting.
I generally expect 4-6 hours worth of interviews before an offer, but the typical process stretches those over a few weeks, which allows the candidate to fit them into available blocks of free time, and gives the candidate oppotunities to think about how things went, what questions to ask, whether this seems like the right fit, and pull out at any point.
Thank you. I've definitely seen a greater resistance to the full day interview process when hiring in London. I never did learn to hire properly for London. I think there's a cultural difference between there and the Bay.
Wow. I live in Sweden, and I have the same expectations: I would not really want to spend much time on an interview process, and I have never, ever heard of day-long interviews either!
We have had a perpetual shortage of developers though, it sounds much harder to land a job in the US.
To clarify, full day means 4 sessions of an hour each bridged by lunch (1 hr). The hope is to give the candidate sufficient information about the team while the team gathers info about the candidate.
That sounds pretty typical. In practice, it's not unusual to do one or more of the interviews by video because of travel schedules but four "in person" interviews is probably the norm.
I guess if you're hiring junior people to stick in a standardized box, it's mostly just a numbers game. Way back when I got a job offer out of school strictly based on my resume (took another job).
But even when the interview process was mostly a formality because I knew many of the people and they knew my work, I still had some on-site interviews.
I'm pretty sure on-site interviews are the norm for the vast majority of professional jobs. Including, or perhaps especially, for the most senior positions.
Netherland here. Interviews take an hour. I don't think I've ever had a full-day interview, though I may have had two interviews in a row. Not sure if they were two half-hour or two full-hour interviews. I have had an on-site assessment of some sort many years ago. Didn't like it much.
I'd much rather do a take-home test than spend a full day getting grilled in interviews.
imagine trying to convince a sernior candidate to do your assignment, and imagine all the other offers she or he has who aren't asking that much of their free time. it's not about how much time it would take, it's about the other choices they have.
This is what I am wondering. The process at my company is a personal interview, coding challenge, then technical interview.
So essentially about 4 hours of time. What makes this different from a company that wants 4 interviews from different managers, which likely takes up more time plus all of it being fixed time slots, versus having one hour of work that you can work on at your own pace?
An interview is a two-way process. I typically find out a lot about the company and the people I'll be working with. Even a 30 minutes phone call is as much for me to be able to filter them out.
I get where you're coming from, and am in a similar position. On the other hand, I've used a coding test to drop a company before (completed the test and refused to submit it, told them the test gave me enough information to know I didn't want to work there). It gives you a hint as to what they think is important technically.
In theory there shouldn’t be any, assuming it’s limited to an hour, both in description and in reality.
In practice there’s a pervasive phenomenon called asymmetry of effort. The hiring manager may crib a task set either from a perfunctory google search or their own body part. The sum total of time spent on their part is often five minutes, including coming up with the problem and their review of your code.
This follows no industry standard, and I can count multiple experiences where they were too inept to “grade” the homework assignment, thinking it didn’t work when it did, or that the problem set was impossible.
The reason senior engineers in particular turn down homeworks is because they’ve been burned by them before, and the average hiring engineer tends to know less than they do (but as the hiring manager, can’t assess it, mistaking competence for arrogance or ineptitude).
When a company is not willing to invest the same amount of time to interview as they ask of the candidate, it’s a strong signal of how serious they are as a quality employer.
When job hunting as a senior, hearing about take homes (esp. early stage ones) is the young person’s dating equivalent of considering a relationship with someone divorced twice and a felony. You just get conditioned to pass up kissing frogs.
Everything else is simply busy work for all parties because it doesn’t tell you anything you don’t already know, plus it scares away quality candidates.
Hmm, interesting. I don't know if it's legal for me to run different interview tracks for immigrant and domestic workers. If I'm hiring away an H-1B from somewhere else, probationary isn't going to work. They will not join since their residency depends on employment. But it appears that some domestic engineers prefer the shorter test series with a bigger opportunity to prove themselves.
Does anyone know if having different tracks is legal?
Seconding this. Read the candidate's actual resume, spend 30 minutes trying to suss out any deception on it, then a couple of hours trying to gauge compatibility, and also sell the candidate on working for you.
You don't even really need a probationary period in an "at will" state.
At some point, you either trust the credentials from universities and certification bodies, and take what the resume says at face value, or you have to try to reinvent some wheels, and re-discover that eliminating the phonies and mimics is a very difficult problem that even the experts can't solve entirely.
I don't know what the conversion factor is between running a fruitless job search filled with cargo-culted bozo tasks, and just randomly picking the first vaguely suitable candidate off the stack and letting them run for a few months, but I suspect that filling the position quickly and painlessly is probably worth about one month of having a 0%ile employee, or six months of a 20%ile, or keeping a 40%ile employee until they retire. This rests on the presumption that some work done right now is worth more than the same amount done some time in the future, and that higher %ile-ranked candidates will be less tolerant of any stupid hiring games.
Or when they ask a seriously difficult np hard problem (probably looking for an algorithmic solution to their real world problem) and never reply whatsoever when you send them a solution for a slightly simplified version.
I am big defender of take home tasks as part of the interview process, but if someone asks me some incredibly hard problem to be solved or anything that seems like could be part of their actual work, I would ask compensation for the time. Take home tasks are not supposed to be real work...
Seems like you have a problem with the implementation of the idea, not the idea itself.
At my current employer, our recruiting manager has a preliminary talk with the candidate and if there seems to be mutual interest in continuing in the process, she sends a simple task. The task was defined by the technical team, it should be short (no more than an hour) and it has nothing to do with our work.
Once she receives the submitted task, we ask everyone in the tech team to make a review. We do have a "gold standard implementation", but we know that no one will get it and we don't care. We had cases of candidates that misunderstood the task and we had no issue with giving them a second chance or simply asking what was unclear and what would they do differently had they had got the intended requirements.
We just want to filter the obvious "no hires" from the ones with potential. The recruiting manager communicates this even before the task is submitted to them. I am yet to hear any story of any candidate that was interested in the position but walked away due to the way that the process is conducted.
> I am yet to hear any story of any candidate that was interested in the position but walked away due to the way that the process is conducted.
Thank you for making my point. Why would you ever hear about it?
This is the "British Rails" fallacy in action. (A long time ago, the British Rail organization had heard complaints that residents of a local community weren't satisfied with the once-daily train service it received. Its response was there was no need for a second train, because at the times considered, no one was standing on the platform waiting to board it).
> Thank you for making my point. Why would you ever hear about it?
Seems like the one that does not want to listen is you.
The take home task is only communicated during the preliminary interview with the recruiter (i.e, it is not something that is required to start the application process) and we do get feedback from candidates that have withdrawn their application. And while we had cases of people that withdrawn for various reasons (company size/culture, process adopted, being too "startupy" for a German company, or even simply getting a better offer somewhere else) the take-home task is yet to be mentioned as an issue.
In the end, your reasons for being against take-home tasks are not really against the idea of take-home tasks, just some of the implementations you've been through. Your "point" does not stand.
Think of it as a small (hopefully) fun mini project in your own pace versus a scheduled phone/onsite screening. Of course it requires the task to be small and open enough.
If the project only takes an hour for a junior it shouldn't be a problem. I would be worried that their senior position problems might not be as quick to solve (which is similar to the real world), but then this interview technique might just be best for junior positions and they do something different for senior positions given the different in skill sets and what problems they are expected to solve.
this comment misses the point and is why we don't give take-home work to senior developer candidates. they are in high demand and will not spend their free time on homework, when many other companies will gladly hire them without making them jump through hoops.
1. Require developers to code on demand during an interview (personally, I have a hard time with this; the pressure of an interview is not a normal working situation).
2. Allow someone to do it on their own time, under no time pressure. (I advocate paying them for their time at a fair market rate).
Do you see another way of a senior developer demonstrating their abilities without hitting either of the two above?
Ask the candidate to thoroughly review code during the interview and offer insight. See if they can spot undefined behavior, if they can improve an already-good solution, and ask what their approach would be to refactor it. If they are truly senior programmers, their actual job will be reviewing more than they write anyway, and you’re not hiring them for their ability to crank out a quicksort implementation.
For the junior folks, sure: have them burn through homework.
Furthermore, just because you can bang out something in an hour or two doesn't mean that you will spend only an hour or two when you know you'll be competing against people who put a lot of effort into the task.
In my case, I do a lot of writing these days and, if someone were to ask me to write a thousand word analysis on some topic I was familiar with, I certainly could knock that off in a few hours. But, assuming I agreed to do it at all as part of the interview process--I'd be far more inclined to just give them a bunch of links to my work--I'm going to put the effort in to come up with a tight, polished product.
Is it really normal for a developer to need to apply to that many positions to find a decent match? I've only ever done a dozen or so at once, and each one I spend a decent amount of time researching the company for tailoring a cover page (if they accept them) or for the interview. I find those making the hiring decisions are quite impressed by candidates who have done their homework (and more than once I've made the hiring manager a bit worried about how I found so much information just through google searches; they don't realize how much their employees let leak through Linked In resumes and such).
That's why it shouldn't be the first step of the interview process, but it's totally fine as the last step. When the hiring party thinks: "we want this candidate if they're as good as they seem to be", then it's a good time to have a look at their code.
I've seen companies hire truly awful senior developers who ended up costing the company quite a lot after they were fired and the entire search had to begin again 3 months later. If the position is in high demand, you still need to gate it to make sure you get a decent chance at a good hire.
Also, I highly suspect companies who are having trouble hiring senior devs are having issues stemming from not paying enough for their given situation.
As a senior developer I have no problem with take-home work. Obviously I won't do it for just any job or project; I'm picky and I'm looking for something special. If I find something that seems to meet my criteria, I'm not going to dismiss it just because they give me take-home work. In fact, I like them because they give me insight into the sort of problems I may have to solve there, and it lets me show what I can do.
I'm not going to jump through stupid hoops, but I love jumping through interesting hoops.
Might be location specific but out of 300+ candidates for 15 positions mid to senior level over 8 years in London recruited using this technique I've only ever had one person turn it down. We ask for fairly simpler requirements and for people to limit themselves to 4-6 hours over the course of a week with a focus on comments and recording their thoughts in a readme to suggest how they'd scale things if it was a real project. We don't really bother with many technical questions outside of the project anymore. It really is the single most important thing to get a glimpse into a persons ability to deliver, let them code at their own time in a setting they are comfortable with and then do a peer review with them on location and discuss the implementation and potential enhancements. The rest after that is generally just team fit and culture alignment checks.
I don't turn them down - bridge burning is for pyromaniacs - I just make non-committal noises and don't usually get back to them.
I should clarify that I was thinking more of outfits that respond to my enquiry with "Hi $candidate, please do this generic exercise after which we'll deign to look at your CV". Bonus points awarded/deducted if the requested exercise is for something already available on my github profile.
And 4-6 hours is typically my free time (not free computer time, total free time) for a week.
The ideal process is having one that actually exists.
Most startups basically try to wing this without any definition of how they want the interview process to work. IMHO this is about 50% of the problem. On the flip side, large companies appear to be overburdened by process (anecdote example - I've heard from several people that getting into Google takes 6 months on average, along with the notorious b-tree whiteboard process).
That being said, I think Aline Lerner[0] and Triplebyte[1] have some good ideas on the topic:
- Clear timelines on the interviewing process. Some companies show no respect for your time. Telling me to pop by the office to "chat" at a specific time and then telling me your agenda for a 4 round technical interview on the spot is not going to inspire any sort of trust.
- Ask me questions that actually apply to the job. If I'm building an iPhone app, please don't ask me to run through a gauntlet of white-boarding/coding challenges that don't apply to the job. Have me walk through something I've built or talk about how I would build a theoretical iPhone app. Most of us aren't building software for self driving cars so please quit acting like we are.
- If I'm writing code or solving problems, give me an ideal scenario to do my best. Let me use my own computer (not one that you just handed me setup to your preferences.) Potentially let me do it at home (where I am comfortable and not in an unfamiliar place.)
- Gauge confidence on the technical stack
- Don't say we'll be in touch shortly and then ghost. You can say goodbye to someone without false promises. I'm likely to tell fellow developers about how the process went.
- Have people with decent social skills interview
- If I ask questions like, "What should I be prepared for?" please give me a basic agenda or guidance. It'll go much better for both of us.
No whiteboard code, no algorithms questions. If you want them to code, let them do it on their own time in a comfortable environment. The industry needs to stop the bullshit leetcode meme that millenials are propagating.
I’m a little confused — can you explain how this is a generation’s fault?
It certainly seems that millennials are subjugated to this type of process, and it’s possible that millennials are the ones giving these types of interviews more often than not, but as far as I can tell these interviews propagated because large, well-regarded, highly successful companies (Google etc.) started performing them and then smaller companies just copied their formula. I’m not quite sure where the generational connection is.
I would give them opportunities to show their skills at real-world tasks:
- Version control. Given a terminal (or Explorer with TortoiseSomething) and an existing project, make some simple changes and commit.
- Testing. Given a simple piece of code, its tests and either a bug report or feature request, explain at least in high level terms how to move forward.
- Code review. Give constructive feedback on how to improve any aspect of a diff.
These tasks indicate an understanding of code and facility with communication beyond the very basics, and you'll never finish learning them, so I believe they are appropriate for any non-entry level position. They are also sufficiently open ended that the candidate has to prioritize getting at least something done on all of them.
After that, unless they are completely hopeless I'd arrange coffee or lunch (on company money of course) with at least part of the team they'd be working with, so they can tell if they are at all compatible.
Needless to say, give the candidate plenty of options for a suitable time, let them know exactly how long it'll take, coordinate the time with the team, show up on time with a computer—which the admin team has wiped and you've just had to copy a few files onto—and find somewhere quiet and comfortable for them to work.
Yes. It really is. A couple weeks ago I interviewed a self-described C# expert who claimed to know everything about C# but couldn't even begin to describe the difference between methods and fields and properties, and did not seem aware that methods have return types. I'm not sure he had ever written a line of real code in C#, or any other language for that matter. This was a recent CS grad from a reasonable quality engineering school, so I'm not sure how he even got through college.
And for the record, I try to interview people on their strengths, giving them the opportunity to show off what they know about the area where their greatest interest or experience lies. A shocking percentage of people (>50%) nevertheless fail to demonstrate even basic knowledge of syntax or language features in the language of their own choice. This is without even considering actual workplace skills like git, PR review, tickets, etc. as discussed above.
> Version control. Given a terminal (or Explorer with TortoiseSomething) and an existing project, make some simple changes and commit.
Does this really tell you anything about a candidate? If you sat me down in front of a terminal and said "here's a git shell, make a change and commit it" I'd fail the test immediately. I've used perforce for the last 9 years, and the terms are not the same, and the commands are _not_ discoverable. So you're eliminating anyone who doesn't know git basics (which can be taught in 5 minutes, or with a GUI of their choice), or you're giving a test for some criteria that you won't judge people on, so it's a waste of time.
> - Code review. Give constructive feedback on how to improve any aspect of a diff.
I like this idea actually. Have the person I'm interviewing review a piece of code (maybe a diff? I'm not sure about different diff formats, maybe it just needs to be some text files, or maybe they just have full internet access and the ability to download their own tools), and judge based on that.
I also balked at it, but for the opposite reason. Everyone should be able to sit down to a terminal, make a change to a repository, and commit it.
I mean even if you don't know the command (or they are using a different version control system then you are used to) you can always check the help or man page. It's still a trivial task.
The intricate of GIT commands are not obvious in the man page. I'd be more interested in a talk about how often to commit, branch, merge, review, ect. If you got that right, the VSC is just a tool you learn if you don't know it.
> Everyone should be able to sit down to a terminal, make a change to a repository, and commit it
Disagree - You're testing can someone use the basics of git in a terminal.
> you can always check the help or man page.
Assuming you know how to do that, and what you're looking at. Say I'm sat down in front of a terminal, and I'm told "here's a terminal, make a change and commit it"
I type in "git", and I get: "usage: git" - If you're used to using shell tools, then sure you can make sense of it, If you're not, then you're done.
You said "commit" and the description for that in the "git" command is:
> grow, mark and tweak your common history
> commit Record changes to the repository
Right, I don't know what that means, but lets try this `git commit`
> fatal: not a git repository (or any of the parent directories): .git
Ok, No idea here. I somehow figure out that git init will give me a .git folder. Now that I've got that out of the way, I try git commit , I get:
> nothing added to commit but untracked files present
Ok, how do I track files? "git help" doesn't mention tracking files. I'll try "git commit hello.txt", which gives me:
> error: pathspec 'Hello.txt' did not match any file(s) known to git
I give up at this point really.
(by the way, I got this far by doing this walkthrough this morning, and googling "how to use git" - which told me the answer in 3 seconds).
Not knowhing how to use a terminal vcs, or knowing the commands to perform even the basics doesn't mean you can't use version control, it just means you can't use a terminal for git. Is my 9 years of C++ and perforce experience because I didn't know that commit was analogous to submit, or because I've used a graphical interface for all that time?
> I type in "git", and I get: "usage: git" - If you're used to using shell tools, then sure you can make sense of it, If you're not, then you're done.
That's part of the test, to see if they're familiar with the command line and if they know how to open the man page. It's trying to weed out the people that can only work in the confines of an IDE and gui tools. That said I wouldn't expect anyone to know git from the man page, I would however expect anyone for a senior role to be familiar with what is a de-facto industry standard.
And source control in general is a great topic for interviews on both sides. Many devs (and companies) don't know what a branch is or what you'd use one for. Many companies make it hard/impossible to create feature branches, either by policy or crazy mono-repo stuff. Even their choice of SCM says a great deal about them, I'd avoid anyone that uses clear case or TFS.
> That's part of the test, to see if they're familiar with the command line and if they know how to open the man page.
That doesn't tell you anything about how good a programmer they are. I don't need to use a command line or man pages for 99.999% of my work, so I'm not going to waste time learning to use more tools.
> It's trying to weed out the people that can only work in the confines of an IDE and gui tools
Ah, so anyone who uses a terminal is superior to someone who uses an IDE or a GUI?
> I would however expect anyone for a senior role to be familiar with what is a de-facto industry standard.
In _your_ industry. As I've mentioned before, I use Perforce (which is standard in my industry).
> And source control in general is a great topic for interviews on both sides
Agreed, but asking somone to rattle off `git init git add . git commit -m"I can remember three lines"` doesn't tell you anything about how much they know about source control. Talk to them about branching/workflows to find out how much they know about source control, or let them use the tools they're comfortable with, but plonking someone in front of a terminal to rattle off some commands is the equivalent of looking for a "culture fit"
> That doesn't tell you anything about how good a programmer they are. I don't need to use a command line or man pages for 99.999% of my work, so I'm not going to waste time learning to use more tools.
I don't want to offend but you sound like exactly the type of developer I try to filter out. IME there is a large correlation between at least rudimentary command line proficiency and being a good developer. Aside from that I want someone to know about the world outside of the IDE and what options are available, because a unix like environment offers far more power than an IDE.
I expect developers should be able to automate common tasks, work with build tools, grep through logs, remote into servers, debug on servers without an IDE installed and a million other things that are very common.
Maybe this doesn't apply to your specific industry but it does to everyone I've worked in to varying degrees. Not memorizing git I could forgive if you could explain branching workflows, but living life in the IDE I wouldn't.
I've encountered people who didn't know the concept of a man page exists, or that -h / -? / --help is a semi-standard construct.
Yet they carried mid/senior-level titles.
You should absolutely test for the things that you think are annoyingly trivial if this person is to be a close peer or a direct report, because your level of disappointment will be so much greater after that person becomes an employee.
I define "annoyingly trivial" as the things that you feel everyone should know "at this stage", and you would be annoyed at having a conversation about said topic for more than 5 minutes.
My personal opinion is Git definitely falls into that category.
> I've encountered people who didn't know the concept of a man page exists, or that -h / -? / --help is a semi-standard construct.
I can count on one hand the number of times I've relied on man pages for descriptions. Using git as the example, compare https://man.cx/git-commit(1) to https://www.atlassian.com/git/tutorials/saving-changes for someone who has never used version control. One of them talks about saving, with examples. The other talks about storing an index in a log.
> My personal opinion is Git definitely falls into that category.
There are plenty of developers with working knowledges of branching/merging workflows, version control, using visual tools, or non-git tools.
I specified "TortoiseSomething" rather than "TortoiseGit" because I specifically did not expect the developer to be familiar with all VCSes. This needs to be agreed upon in advance - it shouldn't be hard to create an environment for whichever VCS the candidate is familiar with.
Obviously not a complete answer, but I think it's a component that isn't being mentioned here. Disclaiming that I don't have much experience (still in university).
One of my internships went from "applying -> interviewing -> accepting" in under 24 hours. Impressed the hell out of me. It was really the reason why I ended up accepting their job. 24 hours is obviously abnormally fast, but a week or two doesn't need to be.
One of my friends ended up rejecting an offer both from Google and Apple because the interview process took too long with no responses. They got a good offer in the mean time, and after waiting over a month for a response they decided they had to go with it. (They followed through with the rest of the Google/Apple interview process anyways for the experience... which was basically just host matching and getting an offer).
That can only work if you have a large number of available positions (typically, low-paid interns). In other cases, you have one position and try to hire the best candidate for it. Interviewing enough candidates takes some time, if only because not everybody is available at the same time.
I notice a lot of these answers focus on the technical or coding challenge. As a hiring manager, that's critical, but technical competency only accounts for about a third of the qualities my team is evaluating in a candidate.
I've given this question a lot of thought over the last couple years as I've lead teams that have needed to get organized and expand quickly. Here's my summary taken from a Google Slides presentation I put together titled "Hiring in a Time of Cargo Culturalism".
It starts with Principles and Guidelines:
- Hiring cycles will be structured and as short as possible.
- When we start a hiring cycle, we will finish it by hiring the most qualified applicant who accepts our offer.
- Every applicant will receive a response within 48 hours and be updated on the status of their application at each step asap.
- The hiring process will be as transparent as possible.
- Objective and fair-minded measures will displace biased and bigoted ones.
- Every applicant will appreciate their experience, even the rejected ones.
- The process will be agile and adapt over time to improve and meet the specific needs of the organization.
- Onboarding will begin with hiring.
Then an outline of my team's current Methodology:
- A thoughtful and literate job posting will accurately describe the job and foreshadow the company culture.
- Simple challenges and honeypots will filter serious candidates from the applicant bots.
- At the end of every step, we will inform the applicant what comes next. Courteous templated responses will be promptly delivered.
- Two interviews. No more than three. The coding challenge will represent a genuine work sample. It will be no more than one or two hours.
- Candidates will be evaluated using a simple quantitative assessment of core competencies (see Ch. 21 of Kahneman’s Thinking Fast and Slow).
- Final decision will be a collective decision of the hiring team.
- After hiring cycle is complete, hiring team will hold a retrospective.
I've hired over a dozen developers this year. They haven't all been homeruns but no strikeouts either. A few singles or walks. A lot of solid doubles. And that's mostly what my company needs.
In the final interview, the candidate meets with 3 to 4 of us on the team and we talk for about an hour. Our team comes in with a set of prepared questions which address past work experience, company culture, team practices and relationships. We make sure it's the same group of interviewers for a given position and we don't robotically stick to the script but try to cover the same ground with each candidate so we're making fair comparative evaluations.
Also, I suspect our job description and pre-interview process (requesting a cover letter, asking a couple short-answer questions in a personal message as soon as we receive an application) succeed in filtering out candidates who aren't thinking about a job as a social inter-personal commitment.
I've spent alot of time thinking about this, and I've concluded with this:
Hire local people minimum wage to learn and teach each other coding structured around your company's codebase. Once they get to know the basics, have them work on your company's open source projects. Identify the ones who actively help others and convert them as a full time software engineer.
It takes an average person about 1 year to learn enough basics to contribute to a codebase, and paying someone minimum wage only costs about 30k / year so it works out financially on both sides.
For reference, this only works in places where the minimum wage is also a reasonable amount.
In the UK, the minimum wage is laughably poor and in Switzerland it's significantly below the amount needed to live.
A better threshold would be to look at the cost of living _in the area_ and pay a reasonable starting wage that actually lets people live there comfortably.
Speed and no trivia / bullshit interview question. Interviews should be assessing me on the skills I’ll need to get the job done, not recanting some data structure or algorithm to solve a problem that’s unlike anything I’ll ever actually deal with.
Ask me about my past projects and decisions I made and mistakes I made and what I’d do differently. Sit down with me and actually code with me and get a feel for what that’s like. Ask me to how I’d design some actually realistic system, and drill into the details of each.
I shouldn’t have to train to pass an interview. Prepare, yes. Practice a bunch of problems unrelated to the job at hand, no thank you.
Counterintuitively I think the problem with hiring is actually a problem with firing. Hear me out.
Everyone is so focused on finding this mythical perfect candidate, instead of giving people a chance. Which is really another way of saying "We're afraid to fire people."
Culturally, we should hire and fire more freely.
I once got a call from a company that said I was an absolute perfect fit for what they were doing-- would I consider a 6-month contract-to-hire? I had been in my job 10 years and told them no. They were so flabbergasted they called back and asked if I wouldn't consider it, that the CTH was simply to see if I was a "culture fit." I told them in plain language that if I was such a perfect candidate, they could hire me straight out. That I'd been in a job 10 years, was obviously happy, why would I jump ship so they could dangle employment as a carrot in front of me?
I can't tell you how many jobs I've not gotten because I didn't get some gotchya question-- or in some cases understood more than the person interviewing me.
The abusive hours need to end, as does the concept of "culture fit" which is just a proxy for age, race, religion, binger drinker status, etc.
We need to stop judging people and trying to feel better about ourselves by dismissing people who can't answer questions we just googled ourselves. Almost every company I've ever talked to claims they only hire the best candidates. That simply, cannot be true.
How is anyone supposed to grow if you can only get a job you're an expert in? And when what you're supposed to be an expert in changes every two years?
So instead of giving you a 6 month trial run followed by employment they should give you a permanently temporary gig? Because a job where they hire and fire freely sounds a lot like how companies treat contractors.
You can't fire people for no reason in the UK and probably the rest of Europe as well or you'll end up being taken to court for unfair dismissal. Roles can be made redundant, but it's a lengthy process.
At my company employment is also at will so they can terminate, but they try very hard not to. In practice contractors are let go much more frequently than employees, and when employees are let go they tend to get a soft landing. But maybe my job is the exception.
I was, of course, using the story to illustrate the absurdity of the overall situation.
I don't want to leave a secure job for an insecure situation because they're difficult to find. They won't consider a direct hire for a "perfect candidate" -- proving my fears.
My point which perhaps I should have been made more clearly is that they're asking me to leave a direct hire situation for a temp job. They're asking me for a commitment they're unwilling to make themselves.
I see, thank you for explaining. It just seemed odd that you yourself sought a secure situation but in you also said "Culturally, we should hire and fire more freely" which to me sounded like we shouldn't have these secure situations at all.
Culturally, we should sue people less frequently. Until that happens, you won't find companies being more willing to fire people, and therefore won't be willing to give them a chance unless they look really good.
The interviewer should ask the candidate, in the candidate's own words, to walk through solving a problem proposed by the interviewer. No whiteboard. Just a simple conversation. The candidate must be provided the opportunity to organize their thoughts with notes on paper and have a short pause of time to think through an answer.
Problem solving is a fairly universal thought experience not limited to writing code. The goal is to ascertain whether the candidate can break down the complexity of a problem into simple steps, organize their thoughts into a clear flow, communicate clearly, and finally recommend a valid solution.
You don't need to test whether the candidate can actually write code, because this is built into the nature of the exercise as qualified by the feasibility of the proposed solution. This exercise also implicitly tests confidence, creativity, experience, and approach style.
Most importantly though, it separates the competent from the incompetent. No amount of framework foolishness and dependency baggage will communicate a solution for you.
I strongly second this approach. I would also check some code but only as a tie breaker. I've seen people writing such messy code with useless random variable names, inconsistent style, etc. That can be fixed. Attitude at problem solving is harder to teach.
I've interviewed many folks over the years who talk the talk and just simply ace problem solving, but they can't write a unit test in their preferred language.
A process that I've seen have some success (years ago):
Start with a very, very simple initial phone screen or take-home test, intended to basically verify whether the candidate can write any code, at all. Max 1 hour, weeds out more people than you'd think.
For the first in-house interview, ask the candidate to code up a problem that is representative of your company's work and requires coding a significant amount, ideally 100+ LOC. The problem should not require any major leaps of intuition, dynamic programming, or recursion – all of these are areas where people do way worse when they're nervous, and this is an engineering interview not special forces training. Let them bring their own laptop, give them the prompt, and have them code, although they can ask the interviewer questions at any time. When they're done, go over the question in detail with the expectation that their code compiles and runs, discuss extensions, etc. Max 1 hour. This interview should answer the binary question "can this person promptly produce meaningful working code and discuss it intelligently?"
For the next in-house interview, do a deep dive into a technical project that the candidate worked on that they're proud of. They describe it and you ask questions. Keep asking questions, especially getting at the "why" behind different decisions, for as long as you can – you're trying to get to the borders of their knowledge and intelligence. Look for mastery of the area, thoughtful decisions, and communication skills. Max 1 hour. This interview should answer the question "is this person a thoughtful, effective, smart contributor on a project?" A good answer should make you think "damn, that's really smart, I wonder if I would have thought of that?" at least once.
End with a final behavioral interview, intended to sell the candidate. This is also a last gut check on whether they're insane, dangerous to themselves or others, extremely arrogant, etc. Also use this time to ask the candidate questions about what really matters to them to improve your closing rate. 30 minutes, and can be combined with the step above.
I've liked this system, YMMV. It's a relatively efficient process, doesn't have weird tricks, and based upon a longterm analysis of candidate outcomes was quite effective (this included an analysis of candidates whom we rejected and who rejected us).
Are you saying max 1hr to understand a problem and write 100 loc? Seems unreasonable unless you are working in languages with a bad signal to noise ratio.
Ideal process for me is being placed in front of other engineers and discussing what I did in the past and how I understand their challenges. Meeting the team I’m working with is a plus to see if we are a match. I don’t do tests and stop the process before it gets to that. I also consider talking to HR a waste of time. The company should sell itself to me.
288 comments
[ 2.4 ms ] story [ 252 ms ] thread1. Test language-specific knowledge - it's not a bad thing to not know all the small quirks or details of a language since they're generally not too useful, but when someone does know them it tends to be a good sign that they really enjoy coding and learning. And of course there is a certain minimum amount of knowledge that should be required - listing Java as "proficient" on your resume while not knowing the difference between abstract classes and interfaces might indicate a problem.
2. Design question - just a simple toy problem like "how would you design the book management system for a library?". It's easy to use such a problem to probe into some different aspects of programming like concurrency and databases, just to see how much the person knows.
3. A not-crazy-hard algorithms questions - people often say algo questions are irrelevant to actual work, and I agree with that. But I think algorithms are really a core part of the CS curriculum at every school, so getting completely stuck on a medium-difficulty algorithm question should raise some flags.
4. Resume-specific things - it's always nice for an interviewer to show that they've actually read your resume, and it can be a good way to convey some strengths that aren't otherwise evident.
I guess my philosophy is to interview in a way that can test the depth of a candidate's knowledge while not being obnoxiously tedious or memorization-focused. i.e. someone who has written a lot of production code should do better in an interview than someone who's just memorized every problem in Cracking the Coding Interview. So, ideally with the screening round clearing the first part (language-specific knowledge), and then 2 or 3 subsequent interviews that cover design and algos.
I think you can quickly get a sense of depth, potential, curiosity, and passion if you can get them talking about quirks and their opinions on those quirks.
For example, if someone says that they once attached a debugger to the JVM to confirm a pathological issue arising from benign looking Java code and found that the JIT compiler falls apart when a certain construct is used, I’m probably going to hire that person, even if we have no Java code in the company - assuming I can also confirm that they are productive and don’t just waste time going down deep rabbit holes.
In my experience curious developers who like to dive deep light up and love discussing that kind of stuff.
Testing random trivia is dumb.
Coworker and manager feedback, and my compensation over the years, say I'm pretty damn good at adding value despite this. I've been considered the "smart one" and the go-to guy for hard or weird problems. But I will struggle to produce something that will even come close to compiling on your whiteboard, in a language I wrote 300 useful lines of yesterday, unless I spend a couple days basically flash-card drilling for your test. Go figure.
Most people who learn algos & DS never have to implement them on the job, and often haven’t actually written their own since their CS classes. If the question is any deeper than “what are some good and bad choices of algorithms for this problem”, you’re selecting more for who has just graduated or has been grinding leetcode and doesn’t have performance anxiety than you are for competence, expertise, and productivity.
I know folks who got rejected from the leetcode companies and went on to build major products from scratch. The 0.5% of work that requires them to have to implement something like this manually, they’ll just grab their CLRS book and refresh their memory long enough to solve the problem and probably forget it again a week later.
I also know folks at these companies who started fresh out of college, passed that style of interview, worked there for 7-10 years, moved up to senior or higher in the engineering track, and openly admit that they would be a deer in the headlights if they got any of the interview questions. Without going back to study and practice those, they know wouldn’t pass an interview for a junior dev, let alone one for their current job (where they are currently considered high performers).
The big problem is that this style of interview filters for people who pass this style of interview, and then those new hires are asked to interview others, so they interviewed in the style that they were given and passed.
The kind of question I had in mind would be along the lines of "say you have a tree, how could you find the nearest common ancestor of two nodes in that tree?". I actually got this question during an interview recently, and I felt it was of a suitable difficulty where I could figure out a solution without any specific knowledge other than what a tree is (which most developers should know).
I'm sure I'm still young and naive, but my line of thinking is that a question like that is good for having a person show their thought process when figuring out a problem they haven't seen before. Design questions are a bit different because they tend to just require a rehash of a codebase you've worked with before.
Algorithms, who cares. Maybe as a tie breaker.
Sure, they don’t know the codebase, but they won’t know the codebase that they’ll be working on in a month if hired either. I want to see how they think about creating software and whether they notice potential defects or tricky areas.
My apologies in advance that this may not be the response you were looking forward to on forum question.
But I discovered an older post online on the Hacker News site some time ago for a Remote Junior Programmer/Assistant at Luma. I noticed you were looking to make your first hire and find someone that has some knowledge in technologies such as Python and Linux.
Well, I had completely forgot about viewing it and so I was checking out Hacker News job board today and saw it was for a remote Part/Full-time opening available at the Luma in the NYC area.
Well I'm someone who enjoys coding and learning with Python as far as learning purposes/hobbies go. Plus, I'm someone who has been going the self taught route trying to break into the IT field/Python Development world. I also have interests in some of the things mentioned in the post such as finance/trading and business.
But, more than anything.. I'd looking to further learn and grow in my skills as far as Python development goes. I don't have much experience with Django Web development but feel I can learn and pick up on it as well as with any other technology requirements quickly.
After quickly reading your post on Hacker News I was encouraged and interested to reach out and contact you for more info.
So with that said, here I am..and I wanted to inquire to find out if this opening is available? Or has it been filled? Also would you allow for training of junior developers to get up to speed? I had to ask because but I wasn't sure if you meant junior or for more seasoned developers.
Also, you have a contact number to learn more about this position or information on the things you require in regards to the nature of the job to increase one's chances to be a part of your startup team? Any help in this matter will be greatly appreciated. Thank you
Feel free to contact me at: pydeveloper22@gmail.com
--K
I recommend trying to make the cost/benefit more symmetric. One way of doing this, is offering compensation for real work. Essentially, hire interviewees to do a small amount of useful (a day?) and evaluate them based on that.
This isn’t going to work for all interviewees, but where possible it feels like a better way of doing things to me.
It's not really worth it to interview nowadays. Practicing leetcode for months, flying, doing all day onsites, when all the company has to do is interview you for a few hours.
We've been issuing take-homes lately, and we ask that candidates spend no more than 3 hours on them. Well, it turns out almost no one spends only 4 hours on them... The most recent candidate we interviewed spent about 3 days on theirs judging from the commit logs. It was a fully fleshed out application with unit tests and everything. After all that they still have to pass the all day on-site, lol. Anyway I spend all of 2 minutes looking at each submission
I really wish there were a better signal.
I was thinking the opposite - that no employed engineer is going to bother with a multi-day paid contract for a new position, so the only engineers you will attract are those without a job and no pending interviews.
Remember Joel Splosky’s classic observation: all the good programmers already have good jobs. The ones on the job market who apply to jobs are usually the bad ones.
I am more than happy to take the time off just for the opportunity. If it doesn't go well or the position is not a good fit then I go back to my current job.
I thought this was perfect, they did not waste much time on the original interview, essentially just determined I was not insane and would be an ok fit for the team. Then via the assignment they were able to see me build something from start to deployment including database development and server deployment. One of the required technologies was a javascript framework I had never used before, they knew that but I spent 8 hours (I really wanted the job) reading a book on it before starting the project and they appreciated it. I had a couple of options on server side stack and they were able to observe and question my decisions. In the code review they were able to see how I handled criticism and reacted to stress.
Whole process from scheduling the first interview to the job offer took 4 days. No white boarding, no obscure technical jargon. They met me (virtually), learned if I could code to their standards and made a decision. In my mind it was great. Via the same process they interviewed someone else I recommend and very professionally and politely turned them down, someone who in my mind was a pretty solid engineer but he just lacked the specific skill set they wanted. While creating the requested application, he knew he was not going to get the offer, which is great as you as a developer are then able to determine for yourself if you are a good fit for the role. No mess no fuss.
If I had already worked in the js stack they asked me to use the entire project build would have been 5 or 6 hours and that's only because I really wanted to impress, I could have knocked it out once my kids went to sleep.
But as in everything it depends on the person and their life circumstances, we all have different things going on.
A few companies gave a simple api project or a timed hacker rank and those feel reasonable. An entire web app with database, rest api, front end and CI pipeline following production code best practices does not.
You only need 1 offer to negotiate.
I’ve always gone through local recruiters that have relationships with the hiring manager. They know the salary range, the interview process, and where other candidates fail because they get feedback from both the candidate and the hiring manager.
After the week was up I went into the onsite and in the "technical" portion of my interview two engineers went over the code I had written for the assignment and asked me about design choices I had made and what I would do if constraint X was added or feature Y. It was all very open ended and much more of a true discussion than an interview which I really appreciated.
I think this kind of format is ideal for interviews. The assignment requirements were simple enough that you could fulfill them easily and in your most comfortable language without any time pressure, but you could also go above and beyond and show that you really knew your stuff. For example, in the instructions they didn't explicitly ask for error handling on the input, but both of the engineers I interviewed with really liked that I had included it. You weren't penalized if you did just the bare minimum because you had the opportunity extend and build on the assignment during the onsite. I felt enabled to showcase my knowledge and justify my design decisions and that that effort was rewarded.
I don't do challenges any more, life is too short to waste solving problems that don't exist for someone else without getting paid.
A simple 100 line code can be curated and taught by a friend/relative in a very elaborate way. This approach can negate that issue in someway.
That excludes anyone with a no moonlighting agreement in their current contract, though.
The problem it's trying to solve is just the surprising number of bad candidates that apply. We're talking about people with decent CVs, a bit of work experience at different companies, some open-source work etc., and who had a good chat on the phone – sometimes they just completely mess it up when faced with a task like "combine these JSON files to make output that looks like this sample". Things like code that literally doesn't work, or code that uses totally inappropriate tools. We want to avoid wasting both their time and our time by bringing them onsite for an interview that isn't going to go anywhere.
The risk with being "hired to do real, paid work remotely" is that it relies on finding people who are in a position to do that, and this ends up excluding a bunch of good candidates. Not everybody is in an employment situation that will allow them to do external work, and it disincentives people with families or other commitments.
The process of applying for a position shouldn't be over-burdening on a candidate. I agree that it's unfair to expect lots of stupid work, multiple-day interview processes and so on. But I don't think it's unreasonable to expect that a candidate who wants to work with a company can afford to spend a few hours in total as part of that process.
They then come back for a code review, walk the team through their code and explain it. This tells us a lot about their ability, but also their communication style and clarity, how they approach problem solving in general and even in which problem they chose.
I chose this approach because 'coding under duress' during an interview does not offer a valid sample of what the candidate can do. Taking the time they need and working in a comfortable environment will give a much more realistic result. Coupling the take-home with an in-person code review weeds out those who may just Google up or borrow code from elsewhere.
A good template is:
- Simple yet relevant take home coding test with a relaxed timeframe
- Technical portion of the interview where the candidate is asked to add a simple feature to their existing tech test.
- This is followed by a more general interview.
None of these sections should take more than 1-2 hours.
I generally think this works well for a few reasons:
1. The take-home task lets us see if candidates meet a minimum standard of competency and generally follows good practice – do they test the solution, document it, write clear… of course, I think it's important that the size of this task is limited to avoid wasting candidates' time.
2. The on-site interview helps to make sure the candidate is able to discuss the work they're doing, collaborate with other technical staff, and communicate technical ideas well.
It feels like the overall goal of an interview process is to ensure that the candidate is at the skill level they have communicated, and that they can work with others.
You've got me thinking that maybe I need something a little more complicated that will give room for those questions.
It means you don't do unnecessary work, you avoid pitfalls and traps you have seen before, you don't over-engineer but you keep it as simple as possible. It also means that you push back against non-constructive requests from business and management and focus your time and effort on what matters.
I also avoid anyone who wants me to do a coding exercise for a few hours over the weekend.
If you want pay me for that time, and make it not a dig and fill hole exercise - find a OSS project that needs some contributions. Decide that you and your team will do thsoe via your ongoing interview process that hires people before the need arises, so you can all take your time selecting and onboarding people.
Value your own time, or no one will
It should also mean you don't under engineer, but that is extremely hard to test for because that involves knowing the business and your customers and having a reasonable estimate of where future needs will be. I'm not sure how you would test for lack of under engineering, especially since any interview task would be a perfect case where you practically can't under engineer since it is guaranteed throw away work. Maybe asking during code review for how you would've done the solution different if you knew that in the next quarter you would likely have to implement either feature A and B or C and D (but you didn't know which yet).
I could set up one of our OSS projects with first-contribution issues and help walk someone through but I suspect most people would rather not spend the time. If it takes 4 hours with back-and-forth in the PR (simply because I may be asleep when you commit and you when I comment) that's a big request of any person.
The main antagonism here is that the hiring company wants to minimise its effort in getting great devs, and the devs want to minimise their effort in getting great jobs.
The first point to note is that constant marketing is the first, best solution to this. I would definitely put more effort into joining a well-marketed company (StackOverflow?) than J.Random Inc. So both me and J.Random Inc had better put some effort into standing out from the crowd.
OSS is one seemingly good way to do this - and having a reasonable Github account is something I would say should get you through most interview filters (ie if you are looking for a Python dev, and they have commits to say a flask extension or bits of salt-stack, then you can blip over the Fizzbuzz and whiteboards.)
But marketing is simply a way to get past other people's filters. "Networking" is good, whatever that might mean, and being a good citizen works. but these are as always, long term, constant application efforts. And of course, costly.
Secondly get over the "we only hire the best of the best of the best". If that's true then like the SEALs you clearly have a six month training programme that takes the best and shapes them into effective teams, fully paid while they learn - yes? And the training staff for this programme are pulled off current fee-paying projects to keep up the standards yes? Otherwise your best-of-the-best is so much auto-trumpet blowing. So please leave that at home. Focus on process not heroes.
The interview by takehome project is a big problem.
Yes it clearly weeds out candidates - mostly by making the ones with options elsewhere go elsewhere. (Even Google, with its firehose of applicants, seemed to realise this was a dumb idea and instead used the MIT graduate program as a filter instead).
My main issue with interviews-by-homework is it is usually a hole-dig-and-fill session as the problem has been done by dozens of people before (often you can find their solutions on github). My time is then valued at less than zero. As a filter before we get to interview, it really is just an artifical hurdle. All you are saying is that "we have soooo much choice we can make you dance before talking to you" - if it works for you great. Your marketing is working (see above)
IMO a better approach is to find different existing properties that you can filter to get applicants to the interview.
Sometimes these are "Alpha Male" filters like 1st class honours at MIT. Other people on HN stand by different filters - I seem to remember someone saying one of their best hires had taught themselves coding whilst getting off drugs in a Glasgow slum. How you filter for that is harder than the MIT problem but I bet its a less tapped seam than MIT.
Overall, a good interview process is probably focusing on the wrong end of the problem. If you have a new open position and then you start looking its too late. Be a good OSS citizen, be involved in the community of developers (OSS and elsewhere), find ways to reach out to unusual developers, or people not marketing themselves, and keep the process going constantly, with suitable investment. The general Big Co idea of "you cannot put out job ads before you have a signed off budget and position" tends to make this a problem. A solution would be to take the expected Agency percentages for each new hire (between 5-20%) and put it into a centralised "developer evangelist" team that just gets people through the door.
This way when you need to open a new position, you will have four people in mind you already want to hire, and you can just go to Starbucks for an interview.
But coming fresh and new into a company, it's hard to just have your opinion or taste taken seriously - especially where that opinion will be unpopular (ie your code base needs rewriting from scratch)
So FU money is needed to avoid the tendency "just to go along with it for a bit"
Hire rich people is hardly a great line, but it will have benefits.
Hiring desperate people with no other options does not sound like you are winning of course
What if 9 great companies give just 2 hours of homework and 1 gives none?
I rarely start anything these days without thinking about the problem thoroughly. It results in longer development times, but better designed, and more robust software, or so I hope.
Aren't you already spending many hours researching a job as the part of the process? Aren't you already taking a full day off work for the interviews? Why is that time all fine to spend, but not fine to spend a couple of hours on doing an assignment?
Coding tests are just an early stage filter for experienced people - so those companies get dropped early on.
You don't want to get your developers together for any random loser who'd never want in your team anyway. So the coding is a late stage filter.
At least for me it's always been.
And Martin any professional should adapt to the systems in place - and I wouldn't use terms like "random luser" might have some blowback.
And if they're not interested in putting any time or effort into a new job, it seems to me they're probably not that interested in that new job in the first place.
But it seems weird to me that so many people here consider a whole day of interviews to be totally acceptable, while some actual coding is not. Compared to some of the interview horror stories I often read here, this seems vastly preferable. I actually get to show what I can do in a realistic setting, and I get to explain why I do what I do. I get to meet the other developers and talk to them.
And with the best programming tests, I actually get to learn something new. For the best one, I had to learn React (which I had no experience in), build a game board on which you can put obstacles, and write an algorithm to find the shortest part through the resulting maze. (I ended up rejecting that job because it was too far away and not enough pay, but I'm still really happy I did that coding challenge.)
I do agree about a daylong set of interviews with 4 or 5 people that's also a filter id use.
In my experience, these sort of tests do often seem to be used by the more interesting, more competent kind of companies. The first company where I encountered this (14 years ago), had only highly competent developers, many of them committers to various open source projects. About the recent one where I wrote a customised version of A* (because of the presence of wormholes, which made it not remotely rote for me or for most people, I'd expect), I of course don't know how it would have turned out had I taken the job, but if they select for this kind of thinking, I see that as a very positive signal about the kind of work they do and the kind of people they hire.
At companies that didn't have this kind of test, the quality of their developers was a lot more hit-and-miss.
At least for permanent employees; I've never encountered this for freelancers.
I certainly didn't get asked coding Questions when I went to BHRA (ranked 1 or 2 in its field) research organisation.
I was told in my first week "oh there is a book on Fortran in the library" teach yourself it
But it's very likely different for research institutes. I'm talking about companies.
I've never had a full day interview for a company I'd actually want to work for (based on what I've learned while grinding through the interviews) so I'm beginning to take requests for those (along with requests for advance coding exercises) as a recruiting smell.
My team has full-day interviews and I'm curious to see who has been self-selecting out of this.
Anyone with any level of self-confidence, worth and respect.
I suppose there are distributed companies that just do things over video link these days (and I've interviewed people over a video link when necessitated by travel schedules or people being in different locations). But some number of in-person interviews over the course of 4 or 5 hours is absolutely standard in my experience.
For permanent jobs, it's usually an hour long interview followed by a programming assessment; either take-home and then presenting to their developers, or codility. But for freelance work, none of that. Just a talk and they hire me. But of course if I turn out to be useless, a freelancer is easy to fire.
(I don't like video interviews much.)
As for hiring freelancers, I'm not involved in hiring programmers but we use external people for various other things. We've try to use people we have experience with but it's mostly not a big deal. If we don't like the work they do, maybe we're out a few bucks but we just don't hire them again.
If I'm unemployed, then full day is fine. If I'm really keen to get out of my current job, or really keen to work for you, then I might do it. But if I'm just exploring opportunities, or I have other options in progress, then I'm not taking a day off just to interview.
With any interview when it might be really obvious 10 minutes in that it's not going to work. If I've scheduled that during my lunch break, or for an hour before/after work, then that's a small cost.
If I take a day off work to interview, then that's costing me in the order of $1000. If I don't know whether I want to work for you, then why would I do it? And I definitely can't do that for 5-10 different roles that I might apply for.
A full day is also quite hard work. Interviews are stressful. Dealing with people you don't know, trying to make sure you don't do/say something stupid, it gets exhausting.
I generally expect 4-6 hours worth of interviews before an offer, but the typical process stretches those over a few weeks, which allows the candidate to fit them into available blocks of free time, and gives the candidate oppotunities to think about how things went, what questions to ask, whether this seems like the right fit, and pull out at any point.
We have had a perpetual shortage of developers though, it sounds much harder to land a job in the US.
But even when the interview process was mostly a formality because I knew many of the people and they knew my work, I still had some on-site interviews.
I'm pretty sure on-site interviews are the norm for the vast majority of professional jobs. Including, or perhaps especially, for the most senior positions.
I'd much rather do a take-home test than spend a full day getting grilled in interviews.
So essentially about 4 hours of time. What makes this different from a company that wants 4 interviews from different managers, which likely takes up more time plus all of it being fixed time slots, versus having one hour of work that you can work on at your own pace?
In practice there’s a pervasive phenomenon called asymmetry of effort. The hiring manager may crib a task set either from a perfunctory google search or their own body part. The sum total of time spent on their part is often five minutes, including coming up with the problem and their review of your code.
This follows no industry standard, and I can count multiple experiences where they were too inept to “grade” the homework assignment, thinking it didn’t work when it did, or that the problem set was impossible.
The reason senior engineers in particular turn down homeworks is because they’ve been burned by them before, and the average hiring engineer tends to know less than they do (but as the hiring manager, can’t assess it, mistaking competence for arrogance or ineptitude).
When a company is not willing to invest the same amount of time to interview as they ask of the candidate, it’s a strong signal of how serious they are as a quality employer.
When job hunting as a senior, hearing about take homes (esp. early stage ones) is the young person’s dating equivalent of considering a relationship with someone divorced twice and a felony. You just get conditioned to pass up kissing frogs.
Everything else is simply busy work for all parties because it doesn’t tell you anything you don’t already know, plus it scares away quality candidates.
Does anyone know if having different tracks is legal?
You don't even really need a probationary period in an "at will" state.
At some point, you either trust the credentials from universities and certification bodies, and take what the resume says at face value, or you have to try to reinvent some wheels, and re-discover that eliminating the phonies and mimics is a very difficult problem that even the experts can't solve entirely.
I don't know what the conversion factor is between running a fruitless job search filled with cargo-culted bozo tasks, and just randomly picking the first vaguely suitable candidate off the stack and letting them run for a few months, but I suspect that filling the position quickly and painlessly is probably worth about one month of having a 0%ile employee, or six months of a 20%ile, or keeping a 40%ile employee until they retire. This rests on the presumption that some work done right now is worth more than the same amount done some time in the future, and that higher %ile-ranked candidates will be less tolerant of any stupid hiring games.
At my current employer, our recruiting manager has a preliminary talk with the candidate and if there seems to be mutual interest in continuing in the process, she sends a simple task. The task was defined by the technical team, it should be short (no more than an hour) and it has nothing to do with our work.
Once she receives the submitted task, we ask everyone in the tech team to make a review. We do have a "gold standard implementation", but we know that no one will get it and we don't care. We had cases of candidates that misunderstood the task and we had no issue with giving them a second chance or simply asking what was unclear and what would they do differently had they had got the intended requirements.
We just want to filter the obvious "no hires" from the ones with potential. The recruiting manager communicates this even before the task is submitted to them. I am yet to hear any story of any candidate that was interested in the position but walked away due to the way that the process is conducted.
Thank you for making my point. Why would you ever hear about it?
This is the "British Rails" fallacy in action. (A long time ago, the British Rail organization had heard complaints that residents of a local community weren't satisfied with the once-daily train service it received. Its response was there was no need for a second train, because at the times considered, no one was standing on the platform waiting to board it).
Nice coconut earmuffs.
Seems like the one that does not want to listen is you.
The take home task is only communicated during the preliminary interview with the recruiter (i.e, it is not something that is required to start the application process) and we do get feedback from candidates that have withdrawn their application. And while we had cases of people that withdrawn for various reasons (company size/culture, process adopted, being too "startupy" for a German company, or even simply getting a better offer somewhere else) the take-home task is yet to be mentioned as an issue.
In the end, your reasons for being against take-home tasks are not really against the idea of take-home tasks, just some of the implementations you've been through. Your "point" does not stand.
1. Require developers to code on demand during an interview (personally, I have a hard time with this; the pressure of an interview is not a normal working situation).
2. Allow someone to do it on their own time, under no time pressure. (I advocate paying them for their time at a fair market rate).
Do you see another way of a senior developer demonstrating their abilities without hitting either of the two above?
For the junior folks, sure: have them burn through homework.
In my case, I do a lot of writing these days and, if someone were to ask me to write a thousand word analysis on some topic I was familiar with, I certainly could knock that off in a few hours. But, assuming I agreed to do it at all as part of the interview process--I'd be far more inclined to just give them a bunch of links to my work--I'm going to put the effort in to come up with a tight, polished product.
Also, I highly suspect companies who are having trouble hiring senior devs are having issues stemming from not paying enough for their given situation.
Companies like Square and Stripe have project-centered rounds during their onsite that allows them to assess these skills without giving a take home.
I'm not going to jump through stupid hoops, but I love jumping through interesting hoops.
I should clarify that I was thinking more of outfits that respond to my enquiry with "Hi $candidate, please do this generic exercise after which we'll deign to look at your CV". Bonus points awarded/deducted if the requested exercise is for something already available on my github profile.
And 4-6 hours is typically my free time (not free computer time, total free time) for a week.
Most startups basically try to wing this without any definition of how they want the interview process to work. IMHO this is about 50% of the problem. On the flip side, large companies appear to be overburdened by process (anecdote example - I've heard from several people that getting into Google takes 6 months on average, along with the notorious b-tree whiteboard process).
That being said, I think Aline Lerner[0] and Triplebyte[1] have some good ideas on the topic:
[0] - http://blog.interviewing.io/
[1] - https://triplebyte.com/blog
I don't think implementing either one of their services is a silver bullet, but are likely n% better than what most companies are doing.
I can only speak to my relatively recent experience, but it was <5 weeks from application to offer for me. No idea if that's typical.
- Ask me questions that actually apply to the job. If I'm building an iPhone app, please don't ask me to run through a gauntlet of white-boarding/coding challenges that don't apply to the job. Have me walk through something I've built or talk about how I would build a theoretical iPhone app. Most of us aren't building software for self driving cars so please quit acting like we are.
- If I'm writing code or solving problems, give me an ideal scenario to do my best. Let me use my own computer (not one that you just handed me setup to your preferences.) Potentially let me do it at home (where I am comfortable and not in an unfamiliar place.)
- Gauge confidence on the technical stack
- Don't say we'll be in touch shortly and then ghost. You can say goodbye to someone without false promises. I'm likely to tell fellow developers about how the process went.
- Have people with decent social skills interview
- If I ask questions like, "What should I be prepared for?" please give me a basic agenda or guidance. It'll go much better for both of us.
It certainly seems that millennials are subjugated to this type of process, and it’s possible that millennials are the ones giving these types of interviews more often than not, but as far as I can tell these interviews propagated because large, well-regarded, highly successful companies (Google etc.) started performing them and then smaller companies just copied their formula. I’m not quite sure where the generational connection is.
- Version control. Given a terminal (or Explorer with TortoiseSomething) and an existing project, make some simple changes and commit.
- Testing. Given a simple piece of code, its tests and either a bug report or feature request, explain at least in high level terms how to move forward.
- Code review. Give constructive feedback on how to improve any aspect of a diff.
These tasks indicate an understanding of code and facility with communication beyond the very basics, and you'll never finish learning them, so I believe they are appropriate for any non-entry level position. They are also sufficiently open ended that the candidate has to prioritize getting at least something done on all of them.
After that, unless they are completely hopeless I'd arrange coffee or lunch (on company money of course) with at least part of the team they'd be working with, so they can tell if they are at all compatible.
Needless to say, give the candidate plenty of options for a suitable time, let them know exactly how long it'll take, coordinate the time with the team, show up on time with a computer—which the admin team has wiped and you've just had to copy a few files onto—and find somewhere quiet and comfortable for them to work.
And for the record, I try to interview people on their strengths, giving them the opportunity to show off what they know about the area where their greatest interest or experience lies. A shocking percentage of people (>50%) nevertheless fail to demonstrate even basic knowledge of syntax or language features in the language of their own choice. This is without even considering actual workplace skills like git, PR review, tickets, etc. as discussed above.
Does this really tell you anything about a candidate? If you sat me down in front of a terminal and said "here's a git shell, make a change and commit it" I'd fail the test immediately. I've used perforce for the last 9 years, and the terms are not the same, and the commands are _not_ discoverable. So you're eliminating anyone who doesn't know git basics (which can be taught in 5 minutes, or with a GUI of their choice), or you're giving a test for some criteria that you won't judge people on, so it's a waste of time.
> - Code review. Give constructive feedback on how to improve any aspect of a diff.
I like this idea actually. Have the person I'm interviewing review a piece of code (maybe a diff? I'm not sure about different diff formats, maybe it just needs to be some text files, or maybe they just have full internet access and the ability to download their own tools), and judge based on that.
I mean even if you don't know the command (or they are using a different version control system then you are used to) you can always check the help or man page. It's still a trivial task.
Disagree - You're testing can someone use the basics of git in a terminal.
> you can always check the help or man page.
Assuming you know how to do that, and what you're looking at. Say I'm sat down in front of a terminal, and I'm told "here's a terminal, make a change and commit it"
I type in "git", and I get: "usage: git" - If you're used to using shell tools, then sure you can make sense of it, If you're not, then you're done.
You said "commit" and the description for that in the "git" command is: > grow, mark and tweak your common history > commit Record changes to the repository
Right, I don't know what that means, but lets try this `git commit`
> fatal: not a git repository (or any of the parent directories): .git
Ok, No idea here. I somehow figure out that git init will give me a .git folder. Now that I've got that out of the way, I try git commit , I get:
> nothing added to commit but untracked files present
Ok, how do I track files? "git help" doesn't mention tracking files. I'll try "git commit hello.txt", which gives me:
> error: pathspec 'Hello.txt' did not match any file(s) known to git
I give up at this point really.
(by the way, I got this far by doing this walkthrough this morning, and googling "how to use git" - which told me the answer in 3 seconds).
Not knowhing how to use a terminal vcs, or knowing the commands to perform even the basics doesn't mean you can't use version control, it just means you can't use a terminal for git. Is my 9 years of C++ and perforce experience because I didn't know that commit was analogous to submit, or because I've used a graphical interface for all that time?
That's part of the test, to see if they're familiar with the command line and if they know how to open the man page. It's trying to weed out the people that can only work in the confines of an IDE and gui tools. That said I wouldn't expect anyone to know git from the man page, I would however expect anyone for a senior role to be familiar with what is a de-facto industry standard.
And source control in general is a great topic for interviews on both sides. Many devs (and companies) don't know what a branch is or what you'd use one for. Many companies make it hard/impossible to create feature branches, either by policy or crazy mono-repo stuff. Even their choice of SCM says a great deal about them, I'd avoid anyone that uses clear case or TFS.
That doesn't tell you anything about how good a programmer they are. I don't need to use a command line or man pages for 99.999% of my work, so I'm not going to waste time learning to use more tools.
> It's trying to weed out the people that can only work in the confines of an IDE and gui tools
Ah, so anyone who uses a terminal is superior to someone who uses an IDE or a GUI?
> I would however expect anyone for a senior role to be familiar with what is a de-facto industry standard.
In _your_ industry. As I've mentioned before, I use Perforce (which is standard in my industry).
> And source control in general is a great topic for interviews on both sides
Agreed, but asking somone to rattle off `git init git add . git commit -m"I can remember three lines"` doesn't tell you anything about how much they know about source control. Talk to them about branching/workflows to find out how much they know about source control, or let them use the tools they're comfortable with, but plonking someone in front of a terminal to rattle off some commands is the equivalent of looking for a "culture fit"
I don't want to offend but you sound like exactly the type of developer I try to filter out. IME there is a large correlation between at least rudimentary command line proficiency and being a good developer. Aside from that I want someone to know about the world outside of the IDE and what options are available, because a unix like environment offers far more power than an IDE.
I expect developers should be able to automate common tasks, work with build tools, grep through logs, remote into servers, debug on servers without an IDE installed and a million other things that are very common.
Maybe this doesn't apply to your specific industry but it does to everyone I've worked in to varying degrees. Not memorizing git I could forgive if you could explain branching workflows, but living life in the IDE I wouldn't.
Yet they carried mid/senior-level titles.
You should absolutely test for the things that you think are annoyingly trivial if this person is to be a close peer or a direct report, because your level of disappointment will be so much greater after that person becomes an employee.
I define "annoyingly trivial" as the things that you feel everyone should know "at this stage", and you would be annoyed at having a conversation about said topic for more than 5 minutes.
My personal opinion is Git definitely falls into that category.
I can count on one hand the number of times I've relied on man pages for descriptions. Using git as the example, compare https://man.cx/git-commit(1) to https://www.atlassian.com/git/tutorials/saving-changes for someone who has never used version control. One of them talks about saving, with examples. The other talks about storing an index in a log.
> My personal opinion is Git definitely falls into that category.
There are plenty of developers with working knowledges of branching/merging workflows, version control, using visual tools, or non-git tools.
Or even better, this one about git: https://xkcd.com/1597/
It's good to use version control, but I wouldn't rate it too high in judging developer competence.
Obviously not a complete answer, but I think it's a component that isn't being mentioned here. Disclaiming that I don't have much experience (still in university).
One of my internships went from "applying -> interviewing -> accepting" in under 24 hours. Impressed the hell out of me. It was really the reason why I ended up accepting their job. 24 hours is obviously abnormally fast, but a week or two doesn't need to be.
One of my friends ended up rejecting an offer both from Google and Apple because the interview process took too long with no responses. They got a good offer in the mean time, and after waiting over a month for a response they decided they had to go with it. (They followed through with the rest of the Google/Apple interview process anyways for the experience... which was basically just host matching and getting an offer).
I've given this question a lot of thought over the last couple years as I've lead teams that have needed to get organized and expand quickly. Here's my summary taken from a Google Slides presentation I put together titled "Hiring in a Time of Cargo Culturalism".
It starts with Principles and Guidelines:
- Hiring cycles will be structured and as short as possible.
- When we start a hiring cycle, we will finish it by hiring the most qualified applicant who accepts our offer.
- Every applicant will receive a response within 48 hours and be updated on the status of their application at each step asap.
- The hiring process will be as transparent as possible.
- Objective and fair-minded measures will displace biased and bigoted ones.
- Every applicant will appreciate their experience, even the rejected ones.
- The process will be agile and adapt over time to improve and meet the specific needs of the organization.
- Onboarding will begin with hiring.
Then an outline of my team's current Methodology:
- A thoughtful and literate job posting will accurately describe the job and foreshadow the company culture.
- Simple challenges and honeypots will filter serious candidates from the applicant bots.
- At the end of every step, we will inform the applicant what comes next. Courteous templated responses will be promptly delivered.
- Two interviews. No more than three. The coding challenge will represent a genuine work sample. It will be no more than one or two hours.
- Candidates will be evaluated using a simple quantitative assessment of core competencies (see Ch. 21 of Kahneman’s Thinking Fast and Slow).
- Final decision will be a collective decision of the hiring team.
- After hiring cycle is complete, hiring team will hold a retrospective.
I've hired over a dozen developers this year. They haven't all been homeruns but no strikeouts either. A few singles or walks. A lot of solid doubles. And that's mostly what my company needs.
Also, I suspect our job description and pre-interview process (requesting a cover letter, asking a couple short-answer questions in a personal message as soon as we receive an application) succeed in filtering out candidates who aren't thinking about a job as a social inter-personal commitment.
Hire local people minimum wage to learn and teach each other coding structured around your company's codebase. Once they get to know the basics, have them work on your company's open source projects. Identify the ones who actively help others and convert them as a full time software engineer.
It takes an average person about 1 year to learn enough basics to contribute to a codebase, and paying someone minimum wage only costs about 30k / year so it works out financially on both sides.
In the UK, the minimum wage is laughably poor and in Switzerland it's significantly below the amount needed to live.
A better threshold would be to look at the cost of living _in the area_ and pay a reasonable starting wage that actually lets people live there comfortably.
Ask me about my past projects and decisions I made and mistakes I made and what I’d do differently. Sit down with me and actually code with me and get a feel for what that’s like. Ask me to how I’d design some actually realistic system, and drill into the details of each.
I shouldn’t have to train to pass an interview. Prepare, yes. Practice a bunch of problems unrelated to the job at hand, no thank you.
Everyone is so focused on finding this mythical perfect candidate, instead of giving people a chance. Which is really another way of saying "We're afraid to fire people."
Culturally, we should hire and fire more freely.
I once got a call from a company that said I was an absolute perfect fit for what they were doing-- would I consider a 6-month contract-to-hire? I had been in my job 10 years and told them no. They were so flabbergasted they called back and asked if I wouldn't consider it, that the CTH was simply to see if I was a "culture fit." I told them in plain language that if I was such a perfect candidate, they could hire me straight out. That I'd been in a job 10 years, was obviously happy, why would I jump ship so they could dangle employment as a carrot in front of me?
I can't tell you how many jobs I've not gotten because I didn't get some gotchya question-- or in some cases understood more than the person interviewing me.
The abusive hours need to end, as does the concept of "culture fit" which is just a proxy for age, race, religion, binger drinker status, etc.
We need to stop judging people and trying to feel better about ourselves by dismissing people who can't answer questions we just googled ourselves. Almost every company I've ever talked to claims they only hire the best candidates. That simply, cannot be true.
How is anyone supposed to grow if you can only get a job you're an expert in? And when what you're supposed to be an expert in changes every two years?
The answer is, culturally, you're not.
I was a formal full-time employee at NCC Group until just a month ago, when they fired me with no warning, no explanation, and no severance.
I don't want to leave a secure job for an insecure situation because they're difficult to find. They won't consider a direct hire for a "perfect candidate" -- proving my fears.
My point which perhaps I should have been made more clearly is that they're asking me to leave a direct hire situation for a temp job. They're asking me for a commitment they're unwilling to make themselves.
Power asymmetries suck.
No, you could also find this if the legal system stopped encouraging meritless suits.
Suing would be less frequent then, but as an effect, not a cause.
Problem solving is a fairly universal thought experience not limited to writing code. The goal is to ascertain whether the candidate can break down the complexity of a problem into simple steps, organize their thoughts into a clear flow, communicate clearly, and finally recommend a valid solution.
You don't need to test whether the candidate can actually write code, because this is built into the nature of the exercise as qualified by the feasibility of the proposed solution. This exercise also implicitly tests confidence, creativity, experience, and approach style.
Most importantly though, it separates the competent from the incompetent. No amount of framework foolishness and dependency baggage will communicate a solution for you.
Start with a very, very simple initial phone screen or take-home test, intended to basically verify whether the candidate can write any code, at all. Max 1 hour, weeds out more people than you'd think.
For the first in-house interview, ask the candidate to code up a problem that is representative of your company's work and requires coding a significant amount, ideally 100+ LOC. The problem should not require any major leaps of intuition, dynamic programming, or recursion – all of these are areas where people do way worse when they're nervous, and this is an engineering interview not special forces training. Let them bring their own laptop, give them the prompt, and have them code, although they can ask the interviewer questions at any time. When they're done, go over the question in detail with the expectation that their code compiles and runs, discuss extensions, etc. Max 1 hour. This interview should answer the binary question "can this person promptly produce meaningful working code and discuss it intelligently?"
For the next in-house interview, do a deep dive into a technical project that the candidate worked on that they're proud of. They describe it and you ask questions. Keep asking questions, especially getting at the "why" behind different decisions, for as long as you can – you're trying to get to the borders of their knowledge and intelligence. Look for mastery of the area, thoughtful decisions, and communication skills. Max 1 hour. This interview should answer the question "is this person a thoughtful, effective, smart contributor on a project?" A good answer should make you think "damn, that's really smart, I wonder if I would have thought of that?" at least once.
End with a final behavioral interview, intended to sell the candidate. This is also a last gut check on whether they're insane, dangerous to themselves or others, extremely arrogant, etc. Also use this time to ask the candidate questions about what really matters to them to improve your closing rate. 30 minutes, and can be combined with the step above.
I've liked this system, YMMV. It's a relatively efficient process, doesn't have weird tricks, and based upon a longterm analysis of candidate outcomes was quite effective (this included an analysis of candidates whom we rejected and who rejected us).