I would think that the other interpretation of this result is just that many engineers suck at writing resumes, which is totally plausible. In fact, one could analyze this in the current data set by looking at the total number of correct predictions versus the different resumes in the set. A well-written resume would then be one which results in correct predictions most frequently. You'd then have two dimensions for each resume - quality as judged by the author and signaling accuracy in the resume itself.
I've seen resumes in all corners of that plot, so I can't say that I'd throw out the resume straight away, but it does indicate that there is some large gap between what hirers want and what candidates think they want.
I think that's a stronger statement than is intended in the article - theoretically if people wrote resumes differently they could be good, but the way they are used now, they are bad. You're providing a theory as to why they are bad now, but not really contradicting anything in the article.
Probably just arguing semantics at this point. I think "engineers need to all learn how to write resumes differently from how any of them do now" qualifies as a pretty fundamental change. Although I guess you really only need to get the good ones to do it. :-)
From my conclusions, that's also a valid take-away.
Looks like it goes both ways. Companies will gain better candidates if they stopped using resumes, and engineers will land better jobs if they started creating resumes that are different from what everybody else writes.
If a large number of people suck at writing resumes enough to throw off a search, the question is raised of why you are first and foremost evaluating candidates for their ability to write resumes.
Since technical writing is part of an engineers job requirements, I would argue that an engineer incapable of writing a decent resume is already at a disadvantage.
Resumes are inherently hard to prepare because the requirements are often deliberately withheld. Granted, if a resume has basic formatting, spelling or grammar problems that's a big red flag, but other than basic linguistic elements it's not really a fair way to evaluate someone's technical communication skills. Some companies treat the job posting itself as a type of puzzle you're expected to guess at with your cover letter and resume.
Also, a lot of job postings are terribly written, extremely vague or otherwise problematic, so it's not a one-sided problem. It's hard to know if you're meeting the employer's needs when their posting is asking for someone who could engineer a mission to Mars on their lunch break and will work for $50,000 and 1 week of vacation.
In the intro of the article, it stated that the strongest signal of applicant quality found on resumes was the number of typos and grammatical errors. Perhaps that was an exaggeration or even a joke, but if not, I would agree that this should be a red flag, regardless of a person's qualifications. (I would similarly judge a potential employer on the quality of their job posting. I know when I write a job posting I go to a great deal of effort to make sure it reads well and reflects our company as well as possible.)
"Granted, if a resume has basic formatting, spelling or grammar problems that's a big red flag"
I agree. The same could also be said about a companies job posting. If the job posting is "vague or problematic", it also indicates the company has no idea of what skill set they need.
This has many parallels with a client not properly being able to specify his requirements. Usually if a client cannot properly specify his requirements, he also underestimate the amount of work / cost it will require to develop.
The same with a job posting, if they cant properly specify a job posting, an employer also might think the job is easy or he can pay someone peanuts to do it.
I propose an alternative or complimentary idea. That cultural inertia guides not only that employers expect the same kinds of data, but also that employees do the same.
If employers wanted to, they could start the ball rolling by simply requesting for more predictive data up-front. Candidates often make specific designs to the company they are applying to. Then companies can filter these candidates out to control the load on expensive interviews.
I'd also say that if a company wants an introspective hiring system, they would naturally pursue a strategy where you identify the kinds of predictors (exclusive or inclusive) you want and request them in resume or some screening test that goes with the resume.
I feel like this data is a small piece of the age long debate about hiring methods and competence/success indicators.
There's been a lot of data about how even people who think they are experts at finding good candidates are really no better than someone flipping a coin.
How then, if the line between "strong/less strong" candidates are so blurred, does Aline draw the line between "strong/less strong" herself? Isn't the way she splits the candidates' resumes also part of the exact thing she's trying to measure?
> "...then even if my criteria for whether each resume belonged to a strong candidate wasn’t perfect, the results would still be compelling"
I don't think you actually demonstrate this statement:
> if the line between "strong/less strong" candidates are so blurred
It is hard to tell prospectively whether someone will be a good programmer. It is not too difficult to tell retrospectively. (Any research that tries to measure competence/success indicators must be using something as its dependent variable, right?) The line you quote is specifically about the fact that resumes could be meaningful even if her competence assignments were not. Resumes seem to fail at that test as well.
The biggest threat to validity would probably be that different people will be good/bad developers in different environments, and that this effect is very strong. Then each reviewer could be good at selecting for success in their own company, but they would only agree with each or with aline to the extent that their environment was similar to that of other companies or the ones that aline placed those engineers at. I am pretty comfortable rejecting this theory out of hand myself.
It's hard to gauge a person's qualities from a resume. Things like culture fit (extremely important IMO), fast thinker, quick learner - are all aspects that can't be garnered from a resume. IMO, resumes are only a decent filter for experience, and provides a baseline for the interview.
There are many I've interviewed that padded their resumes and said they are experienced in something they really aren't, and there are some who dont have much experience but are smart and quick learners.
Without a resume, what else can we go with when it comes to hiring? In my career, the best hires have always been from two sources:
- someone i've worked with and trust before
- someone who was recommended by someone i worked with and trust before
Great... if you're hiring writers. Writing skill is uncorrelated with everything else. I'm a great writer. Many engineers I admire and respect are not.
I've got no magic bullet. Hiring people you know is good, and I strongly recommend informal face-to-face interviews. Two of the best interviews I've ever had (one where I was the interviewer, the other where I was the interviewee) were mostly about language design, which turns out to be a really revealing topic for discovering attitudes about process, standards, and software design generally.
But how to screen people in the pipeline so you can avoid interviewing everyone who applies? No idea.
Is it actually uncorrelated with everything else? I think it's worth investigating (and I hope the addendum at the top of her article means that she will let us know how it works out!) In my experience great engineers who are bad writers definitely exist, but are more the exception than the rule.
I think having the candidate speak to his favorite project or past work is a great gauge of passion and communication skills. I always ask that, albeit in the face to face interview. It might be good to bump that step further up the funnel, but a lot of 'passion' gets lost in translation when it comes to writing vs talking.
I hate to talk on the phone (aka talk without seeing the other person) so I hate phone screening.
To complicate the situation, most of phone screenings I did were made by young recruiters that don't understand anything of what they are asking about, making all this thing silly and awkward. A skype interview is much more relevant but probably scary for the (non technical) recruiter.
By the other hand if I can get a face to face interview, normally I'm in.
I think you hit the nail on the head here and probably why so many engineering jobs are gained through personal referrals instead of cold resumes. I know that's how I got hired at most of the companies I've worked at.
The company I work at building a way to capture those recommendations and that concept of trust through peer-to-peer reviews: https://www.roikoi.com/
The site asks you to play a anonymously game comparing your most relevant coworkers to one another and a users' score is generated by their aggregate results from their coworkers playing the game. It essentially crowdsources gauging those qualities that can't be gathered from a resume.
The nice thing about ROIKOI (disclaimer: I also work there) is that we don't need all signals to be from your current company. We get signals from previous companies, people you've gone to school with, and people you may know in your city.
All of the leaderboards for the companies that I worked for are all top notch folks I would hire in a heartbeat, so so far our output has been fantastic.
This doesn't allow for new people to enter the system, like new college graduates. While hiring within the circle is great for consistency, it also means you may be missing out on great talent. At fast enough growth rates, you'll quickly tap out on the friends-of-friends network and will need a solid plan for hiring unknowns.
I wrote exactly these two tags "fast thinker" / "quick learner" in my CV / cover letter / LinkedIn profile and I removed them a few months later because from my experience it looks like they drive recruiters away or scare them.
I can only speak for myself but terms like that set off my BS meter and probably make me think at least marginally less of whoever sticks them on their resume. They're vague and unquantifiable. They're of a class with words like acclaimed, industry-leading, and so forth in copy describing products.
Yes, for that reason I took it off, still the question is how can one communicate that one is? Should you write "constantly programmed faster and leaner and with less errors that his colleagues?". I don't think there is a way to communicate this but I still think it matters.
You can easily determine if some one is a BS guy in a technical interview anyway, so I don't think, BSers are very frequent in IT and much less so in programming.
Results, achievements, accomplishments. Frankly I don't really care if you were better than the guy in the next cube. He might be an idiot or a complete slacker.
And I'm not sure I wouldn't get negative vibes from someone who felt they had to position themselves by de-positioning the people they worked with.
>You can easily determine if some one is a BS guy in a technical interview anyway, so I don't think, BSers are very frequent in IT and much less so in programming.
I'm not very surprised that resumes aren't a good indicator of performance. They're real only useful as an indicator of experience. In other words they tell you about quantity and not quality. I'd say if they seem reasonable for the role then move them on to the next step of the hiring process. Every interview process that I've been involved with has had a series of increasing expensive steps designed to find specific information and weed people out as early as possible.
It's also worth noting that several of the criteria people used to reject candidates in the article aren't good predictors. For example, lack of pedigree doesn't tell you much. People from a variety of backgrounds can be good or bad as coders. A good or bad pedigree can be a useful indicator.
>For each resume, I had a pretty good idea of how strong the engineer in question was, and I split resumes into two strength-based groups. To make this judgment call, I drew on my personal experience — most of the resumes came from candidates I placed (or tried to place) at top-tier startups. In these cases, I knew exactly how the engineer had done in technical interviews, and, more often than not, I had visibility into how they performed on the job afterwards. The remainder of resumes came from engineers I had worked with directly.
Does anyone else think this is a huge caveat getting thrown into a blender of numbers? Why does this person's opinion even matter? Why should I take this at face value? I'm not trying to say resumes are a good indicator of anything, but neither is this "study".
It goes further than just that - the stuff about Fleiss' kappa shows that not only do these people not agree with the author; they also don't agree with each other. Maybe all development shops are so different that there is no significant correlation between success at one place and success at another, but I highly doubt it!
Thank you. This entire 'study' is nonsense because this recruiter has no idea how 'strong' these candidates really are. She only thinks she knows, just like every arrogant jerk who's ever conducted a technical interview.
> To try to understand whether people really were this bad at the task or whether perhaps the task itself was flawed, I ran some more stats. One thing I wanted to understand, in particular, was whether inter-rater agreement was high. In other words, when rating resumes, were participants disagreeing with each other more often than you’d expect to happen by chance? If so, then even if my criteria for whether each resume belonged to a strong candidate wasn’t perfect, the results would still be compelling
The result of the Fleiss' kappa test subsequently run was negative, i.e. people didn't agree with each other either. So maybe the author's judgement was wrong, but that doesn't affect the conclusions.
I don't think the article posits a cause for why resumes are useless - just argues that they are useless. Your guess about why is definitely a reasonable option. From a hiring manager's perspective, it doesn't matter all that much.
I think it's even simpler just to observe that the way someone chooses to write and target a resume has no particular relationship to how well they would do the job.
Interestingly, why didn't she do some kind of text mining/classification to try to show what features gave the highest discrimination between the two classes?
I am not sure how I would feel if my resume was posted here. I've already found someone I was able to quickly identify by looking at the resumes. I would have expected more discretion on the recruiters part.
> As it turned out, people were pretty bad at filtering resumes across the board, and after running the numbers, it began to look like resumes might not be a particularly effective filtering tool in the first place.
Her theory - people suck at filtering because the tool they use is poor.
> it may not be a matter of being good or bad at judging resumes but rather a matter of the task itself being flawed — at the end of the day, the resume is a low-signal document.
Resumes suck for hiring all people. But that's not the purpose of a resume -- it's a low-signal filter to weed out the initial batch of candidates. If you treat hiring like a sales funnel (and you should), then a resume is a great way to eliminate unnecessary time for your team to assess someone's cultural fit. Expecting the resume to be high signal filter is pretty naive.
> If you treat hiring like a sales funnel (and you should), then a resume is a great way to eliminate unnecessary time for your team to assess someone's cultural fit.
Where's the evidence that it is great at this? Certainly, its a basis for narrowing the set of candidates that get further review, but is it a good basis?
> Where's the evidence that there is a better alternative?
Where's the evidence that resumes are better than random selection among applicants into the next round, much less enough better to be a net positive given the resources that go into reviewing them?
Huh? I think you're confusing the concept of "top of the funnel". Are you suggesting that I randomly select people to interview from a pile of resumes? What if I get a marketing person who has no software experience in the pile for a software developer job?
>Are you suggesting that I randomly select people to interview from a pile of resumes? What if I get a marketing person who has no software experience in the pile for a software developer job?
Well, yeah. If you don't have any better metric, the best result comes from filtering out everyone who just blatantly submitted a resume for the wrong job entirely, and then selecting from the remaining pool at random.
dragonwriter suggested "don't use resumes, just seldect at random". You're suggesting "use resumes, then select randomly". You appear to be agreeing with mbesto.
I am not suggesting anything, I am asking where the evidence is that the practice of asking for and reviewing résumés has positive expected meet utility as a filtering mechanism, and your responses suggest to me that for you that practice is not evidence based but based in unchallenged intuition.
It's definitely not a solution for an employer who is just like scraping resumes off of Monster or something, but it's not much different from the uncontroversial practice of asking for a cover letter.
No, you shouldn't treat hiring like a sales funnel, you really shouldn't. Treat hiring like an investment, because it is. Treat it like a competitive advantage, because it is. Treat it like a skill your company and everyone in it needs to constantly be improving at, because they do. Treat it like one of the most important things you do, because it is.
A company is made up of people, hiring is to a company as mitosis is to the human body, it's fundamental. The better you can be at hiring the better your company will be. More talented, more capable employees translates to better execution on projects, better ideas, better products, more revenue, a higher profile, and a greater pull for more talent out in the industry. If you treat it like boiler room telemarketing grabass then you'll end up with a company to match those ideals.
It also works the other way around. Some of the best people I know have horrible resumes or no resume at all. Resumes are probably subject to the Dunning-Kruger effect that tends to plague software development.
for anyone relating with every "engineers suck at writing resumes" comment; I've been using my stackoverflow profile as a resume and it's been working out great.
The startup where I serve as in-house recruiter, FutureAdvisor, recently ran an experiment in hiring without resumes. It was called StaffupWeekend. The Chronicle's careers blogger wrote about it here:
We recognized that resumes were poor indicators of performance. In fact, we believe that the only thing that correlates with performance -- is performance. A study that Google released last year supports that.
The solution we came up with was to invite people to work on projects for a weekend at no cost other than their time. They chose what they wanted to work on together.
In a sense, we were batch processing applicants while gathering much richer information than resumes provide. Strong candidates we didn't hire, we shared in the Sequoia network.
We ended up making eight interview offers (several candidates decided to work elsewhere), and we hired one engineer.
This sounds like a really interesting idea. I probably takes a weekend to prepare a proper application anyway, so spending the time on a project (which may be fun even) sounds much better.
This is a good interview process for a coding gig. I am a devops engineer who is mainly a system administrator. I code but I code as a tool not as a goal. So for this kind of positions looking for coding performance does not give good indicators. Or you can not do this kind of experiment for other kinds of positions.
>The solution we came up with was to invite people to work on projects for a weekend at no cost other than their time.
I see more and more companies doing this small projects type of initial screening. I like it. But time is actually an expensive cost. Suppose your candidate makes $130,000 per year, which isn't even a high salary in bay area. Then if he/she spends 15 hours on this project, then we are talking about 1000 bucks. Now that is quite a cost, isn't it?
Honestly compared to a white board programming puzzle, such projects always make more sense to me. But most companies will still do white-boards during on-site after the initial project. And although I like most of those projects, even in my case, if I happen to be busy, then I do feel like they are annoying. I'd rather spend my time doing what I was busy doing, or learning new stuff, or working on my pet projects, rather than working on a project that may or may not make sense to me, just to show to other people that I can code, and I can write clean code. Let alone I happen to know some friends who just hate those projects and will never do them unless it's from their dream company. Plus, there are people who are better communicator when they talk face-to-face. Then it's really hard to say if such project is really better than white-boarding for that type of candidates. But it's still way better than those 30-minutes online coding over a phone interview. That's the one I hate most.. Even Skype interview is better than phone interview.
I almost spit out my coffee when the OP stated "for free", as if not having to pay anything but time, my most valuable asset, was some perk.
I'm sure the OP means well, and it may have been a success for them, but I don't know anyone in my circle who would participate in something like this, unless they were utterly desperate for a job. I thought there was a serious talent crunch? Who is courting whom?
I wonder what sort of engineers self-select in a process like this?
Ones that are young enough not to know any better. This is a double-win for a company that only wants to hire 22 year olds anyway, as it avoids having to hide under the hazy fog of "culture fit", and they get free work out of the deal to boot.
One reason why engineers and companies have trouble discussing the job market is that talented people who know how to craft their resumes don't imagine that any other kind of candidate exists.
Unemployed people have lots of time. Understand that and you'll understand why a weekend like this makes sense.
One reason they're unemployed is that they don't know how to demonstrate their value. A lot of engineers are actually poor at communicating, and not that great at resumes.
We invited people to come up with ways to improve the job market, which is clearly broken. Some chose to do that, while others had their own apps already in mind.
One group made contactr.io, an extension for Google Chrome that shows you the email addresses of the founders of the company website you're visiting.
Another group is working on a project called livelyhood, which will be a sort of LinkedIn 2.0 where candidates can share their work easily.
The participants worked for themselves, helped each other, and walked away with a larger network of people who recognized their qualities.
This weekend was specifically designed for people whose resumes would not automatically "command" a $120K salary. They were the ones languishing in the slush pile.
We held the event on a weekend so that people who already had jobs, albeit jobs they were unhappy with, could join us.
If you like your weekends more than good work, then it definitely would not have been the right fit.
Your blood is boiling because your reading comprehension is low.
We had several positions to fill, and we helped good candidates whom we couldn't hire get in touch with other employers. The chances of an interview were much higher than 1 in 15. In fact, we made eight interview offers.
The weekend was totally voluntary, not cheap for us, and everyone knew exactly what they were getting into. It was one weekend, not 15 weekends.
If you think weekends like that are inefficient, consider the job market as it operates currently. If you can think of a more efficient way for it to work, I would love to hear it.
Interestingly, Google itself still extensively uses resumes. In fact, I've been asked by Google recruiters to include "Everything you can" in the resume.
Resumes are just a broad filter for skills and a way to find useful questions to later flesh out a candidate and their competence/experience level. It shouldn't be shocking that code samples and/or a short programming test and a phone call will tell you more about the candidate, and if they should come in for a full interview, than blind keyword bingo.
As someone trying to hire, I find myself going through resumes for the firs ttime...
And honestly, my primary metric is if the word "rails" appears anywhere (It's a ruby on rails job). I might even let that go if I saw a cover letter explaining why they liked rails (but maybe don't have professional experience in it).
Honestly this is a pretty low bar. And the way I've been doing it, the phone screen that follows is pretty softball. But after that, I give out the coding exercise...
If you complete it, you get an in person interview for the most part. But all the rest of it - looking at resumes, the phone screen - sort of seems like a waist of time. I don't care how big a game you can talk, if you can't do the coding exercise (which, btw, is the most common outcome - they beg a couple extensions and then give up), then... oh well.
Honestly, if hiring permenant staff for long term occupation, I probably wouldn't even require that much. If they have php, perl or python on their CV, then they'll have had the opportunity to know everything else you'll want them to know.
This article should be titled, "Resumes that suck suck for hiring engineers".
Resumes that are clear help someone massively, or at the very least do not get in the way. At NemCV we have 100% success rate finding people jobs, and a CV/resume may not be everything needed to get the job, but is a bare expected minimum. Every employer no matter how they find a candidate always asks for a resume.
What I feel right now is that the hiring process is more and more based on personnal branding. If you want to have a job interview in top startup you have to have a good github, smart blog posts, a good stackoverflow, a good kaggle, some certificats from coursera, some conf etc.
It's a good thing for hiring process, you could judge a candidat more easely if you could read some of his code or thoughts. You could have more interesting job offers from people that actually understand what you do. The problem is that you have to spend lot of time in improving these things. Some are easy because it's natural to do some work on a open source project that you're interested in, the personnal branding thing it's just a bonus, but some other need works...
I believe that in all automatic job applying process that I tried recently there were fields for linkedin, github, personnal blogs, websites etc.
A company that relies on GitHub and blog posts will miss tons of highly-qualified candidates who don't (or cannot due to their employment agreement) work on open source and who don't spend their weekends blogging and posting to StackOverflow. Sure, I can see interesting programming side projects being a nice bonus, but it's not a replacement for a resume + technical interview.
Sadly one of the biggest problems I have found regarding Resumes/CV's in the IT industry and companies is the rise of HR departments who will just word filter the candidates and even remove people they deem not suitable. Even the classic case of filtering out people they deem `overqualified`, which sadly means the IT manager most often does not get to even see or known about the better candidates and ends up with a selection picked by HR people who know nothing of the work involved.
I for one welcome phone interaction and tests, though not all tests perfect and found many flawed tests and errors in tests in the past. Some companies appreciate when you tell them of mistakes and others will hold it against you, though again, found HR departments if you point out a mistake will not take it onboard and take it as criticism and in a negative way and hold that against you.
So for me, I do feel HR departments play too large a part in recruitment in IT than their skillsets entail.
Not saying a golden solution, but I do feel that many recruitment processes have gone down hill with HR wedging themselves into the process more and more and filtering out candidates that the manager doing the recruiting, never even knows about.
Though this is all based upon my experience and I'm sure others had different views. But for me, having been turned down for a job for being overqualified and case of you actually knowing the manager and he never even got to see your CV, well, sadly I know it is very true and happens.
Yeah agree. Am trying to bootstrap a startup around crowd sourcing the interview process as a potential solution to the disconnect between recruiters and hiring managers. Could always use some more thought partners on it for the interested (see profile for my contact). Sorry, no website yet (busy on an mvp) but hopefully soon.
Yes, but you can't call everybody that sent you a CV. RH filtering sucks, but is needed usually. Of course there are false positives and negatives
Yes, RH might miss a J2EE expert for a position requiring Java experience, but these days companies search from big databases (Linkedin/Indeed) so this type of grepping is usually needed.
Because most of us wouldn't want to work for a company with such a broken hiring process, it's a big red flag for a lot of other things being wrong in they way they treat their tech people.
And those who do want to get in should be smart enough to play the game and cater their resumes to those HR practices. If you can't or won't, you probably wouldn't happy or successful working for such a company.
As far as I'm concerned, any recruitment process is essentially self-selecting: it will attract and pass the candidates that are a good fit for the company. If a company finds it can't get the right candidates, they should take a step back and look at their entire company culture.
Broken HR processes are a symptom, not the root cause. And it's a filter that works both ways, preventing naive candidates form inadvertently getting hired for jobs that will make them miserable.
I agree, though for many companies recruiting those fields of work have long established institutions and recognised exams, be those lawyers or accountants. now IT has its own institutions and exams but the difference is, those change so frequently or that anything with credibility has changed or indeed expired due to technology changes at a frequency higher than the educational process too learn those skills fully.
Accountant exams and lawyers as the best examples have not changed. So for HR to see filter is not only easy but obvious process in such stable measures. In IT those change so much that it is mostly recruitment agency input that has those factored into requirements and as for agencies. Well some good at filtering and know the trades and others mostly work upon (1) get CV's (2) get companies (3) push CV's and effort done by wine and dine approach with HR.
So not only is it the HR department but the agency factor and in the web based direction the added level of filtering does get down to keywords still and little if not mostly none at all human sanity by somebody who knows the role and can see the signs of somebody who knows the skills.
Then keywords do not bestow what level of competence and that varies greatly as we all know.
This all said sometimes the gains outway the broken HR if you get an offer and not every time do you have the luxury of saying no.
Though as a personal anecdote I once had a job interview with a company and got there on time, left waiting and eventually HR bod came down and said fill this in (form which was all details upon my CV), so I ask can I just attach my CV you already have and told no. So I had to fill out in pen on a form that you just know somebody will have to type into a system, details already upon my CV. Did that, still waiting for HR bod to return. They return then have interview with HR. Was very belittling with much attitude of thats a lot for money for somebody your age and in short really off putting attitude.
I then saw the boss of the IT department, was interview with him and IT contractor, not only aced the interview and blew them away but taught them things as well like undocumented debug modes in software they used (dataease so talking 90's here). Given the contractor was a `specialist` in this product he was mind blown. Also the small bonus skills I totally aced, in short a perfect interview at every level.
Not only was I offered the job before I got home from the interview but they had offered 20% more than the agency had put me forward for and that HR person had been so much attitude towards me (I was early 20's). It just totally put me off, and turned it down, just due to that HR experience. Though have many other HR anecdotes, some good and some bad, as we all do and shall not mention the posture nazi's for want of a better term.
As a hiring manager, I want those resumes as an initial filter. No, they don't say much about engineering skills, but they tell me quite a lot about a number of other things.
Things like communication skills, priorities, attention to detail and motivation are strongly reflected by someones resume.
The only people who can afford to not put serious effort into their resumes are people that have an online (Github etc) or offline (network) footprint that speaks for itself. But those also tend to be the people that don't get recruited via the formal application process.
You know what's frustrating? When you don't even make it to the interview stage because hiring managers think that "Red Hat" and "Linux" are two different things.
I was recently passed over because my 15 years' of experience with Linux has been with multiple distributions (including RPM-based ones) and my resume didn't have a line that just said "Red Hat" on it.
Them: "Oh, I'm sorry, we can't hire you because you don't have any JavaScript experience..."
Candidate: "What do you think all of this stuff with JS in the name on my resume means?!"
Had the same experience. These days I tailor my resume for every single job application. I make sure it uses the language the job description uses. If it says Red Hat, my resume says Red Hat. If it says JavaScript, my resume says JavaScript. It takes time, but its been worth it.
Where a skill is required that I don't have, I don't mention it. Where a skill applies to a version I'm not experienced with, I leave the version number out. When the job spec requires experience of say, Red Hat, and I have experience of multiple distributions, I lie and call it Red Hat. The point of the resume is to land the interview, so I make sure my resume doesn't fail me in achieving that objective.
> These days I tailor my resume for every single job application
You absolutely need to do this unless you're in crazy demand. Every generic resume I've sent out has yielded exactly zero phone calls. Every tailor-made resume I've sent out has gotten at least to the in-person interview stage.
Survivor bias? Maybe, but 5 jobs in since college and I'm never sending generic resumes anywhere ever again.
While I understand the sentiment, I only used a single resume for all of my applications this past fall. But I think the reason it worked is that I knew what positions I wanted to apply for and they were all similar(ish). So my resume matched all of them.
I did find that when I was applying to the bigger SV companies (and Seattle), I wasn't necessarily applying for a specific position, so even if I had wanted to tailor it, it would have been more of a challenge. That's just my experience though. I do believe that if I was going for a very specific position, I would definitely spend the extra time to make sure it was perfect.
Treat CV writing as a SEO task: "JavaScripted the JavaScript scripts using the JavaScript MVC framework Angular.js (JavaScript)." instead of "Implemented using Angular.js".
I got an interview with Google without having "JavaScript" in my resume, and with tons of AngularJS on it. I did link to my GitHub though, where it shows contributions to Angular, various Angular plugins, and other random libraries.
I ended up turning down the interview after accepting another offer though.
I've got a blatant http://... hyperlink on the top of my resume. Almost nobody clicks on it. I went to an interview where they requested code samples which they could have already reviewed beforehand if they had just followed that link.
I've recently written a resolution engine for keywords when matching cv's to job descriptions. it also helps finding patterns in other skills that may be transferrable. i hope to show HN as soon as i finish some performance improvements (which includes writing a brand new db driver :'()
I think that I won't be compatible in a culture where such process is acceptable. If my first touch-point at a company is profound incompetence, I've already wasted enough of my time.
I do not look forward to being on a team with people who were compatible with such a brain-damaged arbitrary system. I've been engaged with too many clients where every day of work is a theater of the absurd.
This was a state government position where the "hiring manager" is "the person who looks at resumes and decides who to call to be interviewed" rather than the direct supervisor of the new hire.
Maybe I used the wrong phrase or maybe it just means different things in different places.
The interesting part is that recruiters seem more in agreement when identifying low performers. When faced with a high performer their scores differ wildly.
It wouldn't especially surprise me if that were the case. It's a lot easier to identify people who don't meet the requirements of a position than to intuit how good a job they'll do based on paper qualifications. You'll make mistakes in both cases, but more in the latter.
Testing like SATs are similar. A low score is probably a much better predictor of (lack of) success at an elite school than a high score is a predictor of success.
171 comments
[ 2.5 ms ] story [ 208 ms ] threadI've seen resumes in all corners of that plot, so I can't say that I'd throw out the resume straight away, but it does indicate that there is some large gap between what hirers want and what candidates think they want.
"...at the end of the day, the resume is a low-signal document."
"Longer-term, how engineers are filtered fundamentally needs to change."
The conclusion seems to be that resumes are fundamentally flawed.
Looks like it goes both ways. Companies will gain better candidates if they stopped using resumes, and engineers will land better jobs if they started creating resumes that are different from what everybody else writes.
Also, a lot of job postings are terribly written, extremely vague or otherwise problematic, so it's not a one-sided problem. It's hard to know if you're meeting the employer's needs when their posting is asking for someone who could engineer a mission to Mars on their lunch break and will work for $50,000 and 1 week of vacation.
I agree. The same could also be said about a companies job posting. If the job posting is "vague or problematic", it also indicates the company has no idea of what skill set they need.
This has many parallels with a client not properly being able to specify his requirements. Usually if a client cannot properly specify his requirements, he also underestimate the amount of work / cost it will require to develop.
The same with a job posting, if they cant properly specify a job posting, an employer also might think the job is easy or he can pay someone peanuts to do it.
If employers wanted to, they could start the ball rolling by simply requesting for more predictive data up-front. Candidates often make specific designs to the company they are applying to. Then companies can filter these candidates out to control the load on expensive interviews.
I'd also say that if a company wants an introspective hiring system, they would naturally pursue a strategy where you identify the kinds of predictors (exclusive or inclusive) you want and request them in resume or some screening test that goes with the resume.
>I'm a real engineer guys!
There's been a lot of data about how even people who think they are experts at finding good candidates are really no better than someone flipping a coin.
How then, if the line between "strong/less strong" candidates are so blurred, does Aline draw the line between "strong/less strong" herself? Isn't the way she splits the candidates' resumes also part of the exact thing she's trying to measure?
> "...then even if my criteria for whether each resume belonged to a strong candidate wasn’t perfect, the results would still be compelling"
> if the line between "strong/less strong" candidates are so blurred
It is hard to tell prospectively whether someone will be a good programmer. It is not too difficult to tell retrospectively. (Any research that tries to measure competence/success indicators must be using something as its dependent variable, right?) The line you quote is specifically about the fact that resumes could be meaningful even if her competence assignments were not. Resumes seem to fail at that test as well.
The biggest threat to validity would probably be that different people will be good/bad developers in different environments, and that this effect is very strong. Then each reviewer could be good at selecting for success in their own company, but they would only agree with each or with aline to the extent that their environment was similar to that of other companies or the ones that aline placed those engineers at. I am pretty comfortable rejecting this theory out of hand myself.
There are many I've interviewed that padded their resumes and said they are experienced in something they really aren't, and there are some who dont have much experience but are smart and quick learners.
Without a resume, what else can we go with when it comes to hiring? In my career, the best hires have always been from two sources:
- someone i've worked with and trust before
- someone who was recommended by someone i worked with and trust before
I've got no magic bullet. Hiring people you know is good, and I strongly recommend informal face-to-face interviews. Two of the best interviews I've ever had (one where I was the interviewer, the other where I was the interviewee) were mostly about language design, which turns out to be a really revealing topic for discovering attitudes about process, standards, and software design generally.
But how to screen people in the pipeline so you can avoid interviewing everyone who applies? No idea.
To complicate the situation, most of phone screenings I did were made by young recruiters that don't understand anything of what they are asking about, making all this thing silly and awkward. A skype interview is much more relevant but probably scary for the (non technical) recruiter.
By the other hand if I can get a face to face interview, normally I'm in.
The company I work at building a way to capture those recommendations and that concept of trust through peer-to-peer reviews: https://www.roikoi.com/
The site asks you to play a anonymously game comparing your most relevant coworkers to one another and a users' score is generated by their aggregate results from their coworkers playing the game. It essentially crowdsources gauging those qualities that can't be gathered from a resume.
All of the leaderboards for the companies that I worked for are all top notch folks I would hire in a heartbeat, so so far our output has been fantastic.
I wrote exactly these two tags "fast thinker" / "quick learner" in my CV / cover letter / LinkedIn profile and I removed them a few months later because from my experience it looks like they drive recruiters away or scare them.
You can easily determine if some one is a BS guy in a technical interview anyway, so I don't think, BSers are very frequent in IT and much less so in programming.
And I'm not sure I wouldn't get negative vibes from someone who felt they had to position themselves by de-positioning the people they worked with.
>You can easily determine if some one is a BS guy in a technical interview anyway, so I don't think, BSers are very frequent in IT and much less so in programming.
Of course they're frequent. They are everywhere.
It's also worth noting that several of the criteria people used to reject candidates in the article aren't good predictors. For example, lack of pedigree doesn't tell you much. People from a variety of backgrounds can be good or bad as coders. A good or bad pedigree can be a useful indicator.
Does anyone else think this is a huge caveat getting thrown into a blender of numbers? Why does this person's opinion even matter? Why should I take this at face value? I'm not trying to say resumes are a good indicator of anything, but neither is this "study".
But what I or my company consider a quality candidate may differ from the author.
On another note, I've found that for recent grads, GPA, school, and major (all items on the resume) are excellent predictors of interview performance.
> To try to understand whether people really were this bad at the task or whether perhaps the task itself was flawed, I ran some more stats. One thing I wanted to understand, in particular, was whether inter-rater agreement was high. In other words, when rating resumes, were participants disagreeing with each other more often than you’d expect to happen by chance? If so, then even if my criteria for whether each resume belonged to a strong candidate wasn’t perfect, the results would still be compelling
The result of the Fleiss' kappa test subsequently run was negative, i.e. people didn't agree with each other either. So maybe the author's judgement was wrong, but that doesn't affect the conclusions.
Her theory - people suck at filtering because the tool they use is poor.
> it may not be a matter of being good or bad at judging resumes but rather a matter of the task itself being flawed — at the end of the day, the resume is a low-signal document.
Resumes suck for hiring all people. But that's not the purpose of a resume -- it's a low-signal filter to weed out the initial batch of candidates. If you treat hiring like a sales funnel (and you should), then a resume is a great way to eliminate unnecessary time for your team to assess someone's cultural fit. Expecting the resume to be high signal filter is pretty naive.
Where's the evidence that it is great at this? Certainly, its a basis for narrowing the set of candidates that get further review, but is it a good basis?
I was slightly perturbed by the original author as she offered no real solution.
Where's the evidence that resumes are better than random selection among applicants into the next round, much less enough better to be a net positive given the resources that go into reviewing them?
Well, yeah. If you don't have any better metric, the best result comes from filtering out everyone who just blatantly submitted a resume for the wrong job entirely, and then selecting from the remaining pool at random.
A company is made up of people, hiring is to a company as mitosis is to the human body, it's fundamental. The better you can be at hiring the better your company will be. More talented, more capable employees translates to better execution on projects, better ideas, better products, more revenue, a higher profile, and a greater pull for more talent out in the industry. If you treat it like boiler room telemarketing grabass then you'll end up with a company to match those ideals.
http://blog.sfgate.com/gettowork/2014/10/31/staffup-offers-a...
We recognized that resumes were poor indicators of performance. In fact, we believe that the only thing that correlates with performance -- is performance. A study that Google released last year supports that.
The solution we came up with was to invite people to work on projects for a weekend at no cost other than their time. They chose what they wanted to work on together.
In a sense, we were batch processing applicants while gathering much richer information than resumes provide. Strong candidates we didn't hire, we shared in the Sequoia network.
We ended up making eight interview offers (several candidates decided to work elsewhere), and we hired one engineer.
The whole event was inspired by this essay: http://brookeallen.com/pages/archives/1234
I see more and more companies doing this small projects type of initial screening. I like it. But time is actually an expensive cost. Suppose your candidate makes $130,000 per year, which isn't even a high salary in bay area. Then if he/she spends 15 hours on this project, then we are talking about 1000 bucks. Now that is quite a cost, isn't it?
Honestly compared to a white board programming puzzle, such projects always make more sense to me. But most companies will still do white-boards during on-site after the initial project. And although I like most of those projects, even in my case, if I happen to be busy, then I do feel like they are annoying. I'd rather spend my time doing what I was busy doing, or learning new stuff, or working on my pet projects, rather than working on a project that may or may not make sense to me, just to show to other people that I can code, and I can write clean code. Let alone I happen to know some friends who just hate those projects and will never do them unless it's from their dream company. Plus, there are people who are better communicator when they talk face-to-face. Then it's really hard to say if such project is really better than white-boarding for that type of candidates. But it's still way better than those 30-minutes online coding over a phone interview. That's the one I hate most.. Even Skype interview is better than phone interview.
Instead of presenting the "screening project", they present their open source project and its source code.
As in working for you for free, and on a WE to boot?
For a student or someone who is unemployed, that might work.
For someone with a job and a family life, I don't think so. Especially if that person can command a $120k+ salary.
I'm sure the OP means well, and it may have been a success for them, but I don't know anyone in my circle who would participate in something like this, unless they were utterly desperate for a job. I thought there was a serious talent crunch? Who is courting whom?
I wonder what sort of engineers self-select in a process like this?
Unemployed people have lots of time. Understand that and you'll understand why a weekend like this makes sense.
One reason they're unemployed is that they don't know how to demonstrate their value. A lot of engineers are actually poor at communicating, and not that great at resumes.
This weekend was for them.
We invited people to come up with ways to improve the job market, which is clearly broken. Some chose to do that, while others had their own apps already in mind.
One group made contactr.io, an extension for Google Chrome that shows you the email addresses of the founders of the company website you're visiting.
Another group is working on a project called livelyhood, which will be a sort of LinkedIn 2.0 where candidates can share their work easily.
The participants worked for themselves, helped each other, and walked away with a larger network of people who recognized their qualities.
This weekend was specifically designed for people whose resumes would not automatically "command" a $120K salary. They were the ones languishing in the slush pile.
We held the event on a weekend so that people who already had jobs, albeit jobs they were unhappy with, could join us.
If you like your weekends more than good work, then it definitely would not have been the right fit.
Seriously????
15 weekends of hard work just to change job? Yes, very efficient.
The arrogance of your approach makes my blood boil.
We had several positions to fill, and we helped good candidates whom we couldn't hire get in touch with other employers. The chances of an interview were much higher than 1 in 15. In fact, we made eight interview offers.
The weekend was totally voluntary, not cheap for us, and everyone knew exactly what they were getting into. It was one weekend, not 15 weekends.
If you think weekends like that are inefficient, consider the job market as it operates currently. If you can think of a more efficient way for it to work, I would love to hear it.
The bit where you call me stupid...
And honestly, my primary metric is if the word "rails" appears anywhere (It's a ruby on rails job). I might even let that go if I saw a cover letter explaining why they liked rails (but maybe don't have professional experience in it).
Honestly this is a pretty low bar. And the way I've been doing it, the phone screen that follows is pretty softball. But after that, I give out the coding exercise...
If you complete it, you get an in person interview for the most part. But all the rest of it - looking at resumes, the phone screen - sort of seems like a waist of time. I don't care how big a game you can talk, if you can't do the coding exercise (which, btw, is the most common outcome - they beg a couple extensions and then give up), then... oh well.
Resumes that are clear help someone massively, or at the very least do not get in the way. At NemCV we have 100% success rate finding people jobs, and a CV/resume may not be everything needed to get the job, but is a bare expected minimum. Every employer no matter how they find a candidate always asks for a resume.
Zubair, NemCV
It's a good thing for hiring process, you could judge a candidat more easely if you could read some of his code or thoughts. You could have more interesting job offers from people that actually understand what you do. The problem is that you have to spend lot of time in improving these things. Some are easy because it's natural to do some work on a open source project that you're interested in, the personnal branding thing it's just a bonus, but some other need works...
I believe that in all automatic job applying process that I tried recently there were fields for linkedin, github, personnal blogs, websites etc.
I for one welcome phone interaction and tests, though not all tests perfect and found many flawed tests and errors in tests in the past. Some companies appreciate when you tell them of mistakes and others will hold it against you, though again, found HR departments if you point out a mistake will not take it onboard and take it as criticism and in a negative way and hold that against you.
So for me, I do feel HR departments play too large a part in recruitment in IT than their skillsets entail.
Not saying a golden solution, but I do feel that many recruitment processes have gone down hill with HR wedging themselves into the process more and more and filtering out candidates that the manager doing the recruiting, never even knows about.
Though this is all based upon my experience and I'm sure others had different views. But for me, having been turned down for a job for being overqualified and case of you actually knowing the manager and he never even got to see your CV, well, sadly I know it is very true and happens.
Yes, but you can't call everybody that sent you a CV. RH filtering sucks, but is needed usually. Of course there are false positives and negatives
Yes, RH might miss a J2EE expert for a position requiring Java experience, but these days companies search from big databases (Linkedin/Indeed) so this type of grepping is usually needed.
Because most of us wouldn't want to work for a company with such a broken hiring process, it's a big red flag for a lot of other things being wrong in they way they treat their tech people.
And those who do want to get in should be smart enough to play the game and cater their resumes to those HR practices. If you can't or won't, you probably wouldn't happy or successful working for such a company.
As far as I'm concerned, any recruitment process is essentially self-selecting: it will attract and pass the candidates that are a good fit for the company. If a company finds it can't get the right candidates, they should take a step back and look at their entire company culture.
Broken HR processes are a symptom, not the root cause. And it's a filter that works both ways, preventing naive candidates form inadvertently getting hired for jobs that will make them miserable.
Accountant exams and lawyers as the best examples have not changed. So for HR to see filter is not only easy but obvious process in such stable measures. In IT those change so much that it is mostly recruitment agency input that has those factored into requirements and as for agencies. Well some good at filtering and know the trades and others mostly work upon (1) get CV's (2) get companies (3) push CV's and effort done by wine and dine approach with HR.
So not only is it the HR department but the agency factor and in the web based direction the added level of filtering does get down to keywords still and little if not mostly none at all human sanity by somebody who knows the role and can see the signs of somebody who knows the skills.
Then keywords do not bestow what level of competence and that varies greatly as we all know.
This all said sometimes the gains outway the broken HR if you get an offer and not every time do you have the luxury of saying no.
Though as a personal anecdote I once had a job interview with a company and got there on time, left waiting and eventually HR bod came down and said fill this in (form which was all details upon my CV), so I ask can I just attach my CV you already have and told no. So I had to fill out in pen on a form that you just know somebody will have to type into a system, details already upon my CV. Did that, still waiting for HR bod to return. They return then have interview with HR. Was very belittling with much attitude of thats a lot for money for somebody your age and in short really off putting attitude. I then saw the boss of the IT department, was interview with him and IT contractor, not only aced the interview and blew them away but taught them things as well like undocumented debug modes in software they used (dataease so talking 90's here). Given the contractor was a `specialist` in this product he was mind blown. Also the small bonus skills I totally aced, in short a perfect interview at every level.
Not only was I offered the job before I got home from the interview but they had offered 20% more than the agency had put me forward for and that HR person had been so much attitude towards me (I was early 20's). It just totally put me off, and turned it down, just due to that HR experience. Though have many other HR anecdotes, some good and some bad, as we all do and shall not mention the posture nazi's for want of a better term.
Arrogance, disrespect and plain unpreparedness.
They're the first step in the funnel, if you can't get through them your resume usually needs fixing.
And of course the interviewing flaws that are discusses several times on HN play a part as well.
Things like communication skills, priorities, attention to detail and motivation are strongly reflected by someones resume.
The only people who can afford to not put serious effort into their resumes are people that have an online (Github etc) or offline (network) footprint that speaks for itself. But those also tend to be the people that don't get recruited via the formal application process.
I was recently passed over because my 15 years' of experience with Linux has been with multiple distributions (including RPM-based ones) and my resume didn't have a line that just said "Red Hat" on it.
Them: "Oh, I'm sorry, we can't hire you because you don't have any JavaScript experience..."
Candidate: "What do you think all of this stuff with JS in the name on my resume means?!"
Where a skill is required that I don't have, I don't mention it. Where a skill applies to a version I'm not experienced with, I leave the version number out. When the job spec requires experience of say, Red Hat, and I have experience of multiple distributions, I lie and call it Red Hat. The point of the resume is to land the interview, so I make sure my resume doesn't fail me in achieving that objective.
You absolutely need to do this unless you're in crazy demand. Every generic resume I've sent out has yielded exactly zero phone calls. Every tailor-made resume I've sent out has gotten at least to the in-person interview stage.
Survivor bias? Maybe, but 5 jobs in since college and I'm never sending generic resumes anywhere ever again.
I did find that when I was applying to the bigger SV companies (and Seattle), I wasn't necessarily applying for a specific position, so even if I had wanted to tailor it, it would have been more of a challenge. That's just my experience though. I do believe that if I was going for a very specific position, I would definitely spend the extra time to make sure it was perfect.
I'm sure if I was trying to get a job at Buffer or someplace like that where you are mostly applying to the company, I'd need to rethink my strategy.
(I have no affiliation with Buffer whatsoever).
I don't think Google can give a good match to a Javascript CV with only Angular.js written
Yes, it's stupid. Yes, do it.
I ended up turning down the interview after accepting another offer though.
Yeah, I think you made the right choice turning them down.
I do not look forward to being on a team with people who were compatible with such a brain-damaged arbitrary system. I've been engaged with too many clients where every day of work is a theater of the absurd.
If a recruiter or HR rep doesn't know Red Hat from Linux they need to find another field to recruit in.
(a hiring manager is usually the person a new hire directly reports to, not the person doing initial resume screening)
Maybe I used the wrong phrase or maybe it just means different things in different places.
Testing like SATs are similar. A low score is probably a much better predictor of (lack of) success at an elite school than a high score is a predictor of success.