Terrific analysis, thank you! I love reading your posts.
I also think that the dynamics differ based on the mode of interviewing viz. Faceless (like http://interviewing.io does) vs face-ful (like normal onsite or video interviews, or the way http://interviewkickstart.com does).
I have an observation to share for the latter kind of interviews:
When I was hiring at Box for one of my teams (or even now), I'd generally go towards the end, to meet the candidate and talk to them about how the day went. I'd ask them about specific interviews, the interviewers and how they think they did on the given problems.
I found that it was surprisingly rare to get a candidate who was aware enough of how things went (positive or otherwise).
Digging into it, I realized that the correlation of what the candidate perceived, was simply to how expressive the interviewER was. If the interviewer was friendly and chatty, the candidate would feel like they have done well (or at least forgivably well). If the interviewer was more silent, the candidate would feel they haven't done very well.
Now that I do interview training for a living, I try and hammer this point home: that interviews are like a date. It depends a lot on the other person (in this case, the interviewer). Understanding this well often works like magic.
> Hopefully this gets more love than it did last time
We invited the submitter to repost it for that reason. I'll try to merge the comments from there into here.
Invited reposts are mostly deprecated now in favor of re-ups [1], but when it looks like the submitter might also be the author (as e.g. with Show HNs), we still send them. It's nice for an author to know that their post may still get discussed, and it's good for HN when an author jumps into the thread.
Eh - humans generally have a hard time rating things on a 5 star rating system. That's mainly because the discrete intervals between 1 and 5 stars aren't well defined so everyone will have a different opinion of what 3 stars "mean". I encountered this when I ran a rating system in Mechanical Turk. If you ask turkers to do something arbitrary like "rate the quality of this picture" it takes quite a number of ratings on the same image to get an accurate consensus.
It would be interesting to see if you can replicate experiment but replace the 5 stars with 5 yes/no questions regarding their performance and see if those match up
There are text descriptions under the stars, which are coincidentally rated out of 4, not 5. 3/4 is "Solid" for someone's technical skills. If someone does below average or amazing it's unlikely that the interviewer is going to select 3/4, regardless of their internal idea of what 3/4 means, when it says "Solid" underneath that.
Even worse because 1 star and 5 stars mean different things to different people, or even sometimes to the same person under different circumstances. If you asked me to rate my interview performance without first defining some objective criteria, for example, I'd have to spent a few minutes deciding whether (1 vs 5) stars is ("didn't get the job" vs "got hired with a nice title and negotiated a bunch of extra pay") or ("was arrested for assaulting the interviewer" vs "got the job").
Whats worse is that any type of rating system has an utter bias towards the very very upper end of the scale by it's consumers.
If you have a true rating system then 2.5/3 out of 5 means average or more or less what you should be looking for in most cases 4 is out standing and 5 is bloody icecream pooping unicorn.
You see this every where a rating system is implemented heck in entertainment movies and games don't fight from 0 to 100 anymore but form the 90-100 if you see any score below 90 or 4 stars people usually avoid that product.
I've found this with myself as well. In one of my worst-ever interviews, I had the flu really badly the entire time, and read the signals of all of my interviewers to be both negative regarding my technical chops and negative toward my weakened state. I got the job, and was very well liked. In another trainwreck of an interview where I seriously considered standing up and leaving after getting extreme negative signals, I was offered the job as well-- but I should have paid attention to the negative signals, as they were indicative of larger problems with that group.
In other interviews, I've massively overestimated my own performance, seeing positive signals everywhere and being very impressed with myself-- of course, these didn't pull through, and my confidence was dashed. In another incidence, I did get the job offer.
The most confusing are the interviews in which I don't have a strong feeling of doing well or poorly either way-- not the median case, but certainly it happens frequently enough that I've gotten a job where I didn't think that I stood out. If anything, these cases are a consolation that others may see something positive that I don't see in myself.
This tells me that my ability to correctly predict my interview performance is no more efficacious than random chance. I think that there are a few confounding factors which make interviewee measuring interview performance as perceived by the interviewer difficult. Interviews are a time of endless posturing, flagrant lies, propaganda, and overt deception hopefully mingled in with actual mutual interest, excitement, and good will. There are many social signals to keep track of in addition to self-monitoring to ensure the correct outcome. For people who are not the strongest socially, I think that these social signals tend to get dropped intermittently as attention shifts inward (don't say the wrong thing) or outward (get this technical problem correct).
It's a fools errand to try and predict interview performance. I've been interviewing for a short while now and haven't gotten an offer yet despite feeling really great (and for some, really crappy) about my interviews. Feedback from the interviewers is equally inaccurate. On my most recent interview, I was told that my solution to their coding challenge was one of the best they'd ever seen, and that the feedback so far was extremely positive. Two days later I got the "another direction" email. I don't really take rejection to heart anymore but I hate not knowing what I need to do better.
I ask myself after a job interview if I really like my potential tasks, the people and the work environment. If "yes", then I will not get the job. If I don't like it then I will get an offer. This metric worked for me the last 5 out of 5 times.
I've yet to be met with success but I'm getting closer. I'm refactoring the way I present myself to emphasize the value I bring. I believe I was too focused on how the company could benefit me. Not salary or perks, but mentorship and training since I'm still early in my career.
The thing is that I already bring a lot of value - I could definitely bring more, but I failed to emphasize what I already have.
Sometimes it has nothing to do with your performance. Budget cuts, downsizing, sudden hiring freezes, or possibly you were the last candidate but the VP is already sold on another guy. Sometimes job parameters shift, they are looking for different areas of expertise and they don't realize it til after they interview a bunch of people.
That is absolutely false. I graduated with a BSEE and worked as an electrical engineer for over four years without ever attempting the FE. Nobody I knew, regardless of discipline, who graduated in the mid-90s or later has done so either.
An FE is required in order to be hired at a firm providing engineering services "to the public", such as a civil engineering or architecture practice. It is not required, or in some cases even valued, at product companies.
What is it with software developers who take such offense at the use of the word "engineer"? Clearly, you're not a "real" engineer, or you'd know that your statement about the use of the word "engineer" in titles was wrong in the US (and Canada doesn't use the FE exam).
I've never met a engineer who gave a crap what my title was, software developer or software engineer. Which isn't a surprise, since we pose no threat to the jobs of "real" engineers. Not sure why the word is so threatening to some software developers.
Actually I'm a mechanical engineer. Webdevs calling themselves engineers is an insult to the amount of work and effort that I've put in to get where I am now. It's like medical technician calling themselves a doctor because they took a training prep course.
A technician could call themselves a "doctor", but they couldn't call themselves a "Medical Doctor". A developer can call themselves an "engineer", but they cannot call themselves a "Professional Engineer". If I started putting "PE" after my name, I'd expect the NCEES to come after me. No one but you can decide what you take as an insult, of course, but choosing industry-standard (and legally uncontrolled) terminology like "software engineer" to take offense at seems like a losing battle.
Webdevs calling themselves engineers is an insult to
the amount of work and effort that I've put in to get
where I am now
1. "web dev" is nebulous. At what part of the stack does it cease to be "web dev?"
2. Definition of engineer:
"An engineer is a practitioner of engineering,
concerned with applying scientific knowledge,
mathematics, and ingenuity to develop solutions for
technical, societal and commercial problems."
Your statements misunderstand the definition of an engineer and devalue the accomplishments of people who apply math, science, and ingenuity to [animate things on websites] [make a website load in requisite time] [bend the constraints set by the browser to achieve certain objectives] [...insert other software engineering tasks].
3. It sounds to me that you feel hurt over credentials. What you are projecting: "I worked hard for this thing and I perceive that you are devaluing it." My response is, credentials are different in software engineering, sorry. And nobody is devaluing your mechanical engineering degree here; if it's something in your field that is valuable, that's good for you.
But mechanical engineering isn't software engineering, and you can't say x isn't engineering, you have to understand what the accepted definition is.
I once worked at a company (2007-2010) where almost every non-manager had "engineer" in their job title.
Aside from Software Engineers, our QA staff were Test Engineers, our customer support people were Customer Support Engineers, our tech writer was the Technical Documentation Engineer (that was a late addition: his predecessor was simply a Sr. Technical Writer), our pentesters were Vulnerability Research Engineers, our sysadmin was the IT Systems Engineer, and probably a few more I can't remember. The only exceptions were the tech leads in development, QA, and support, who all had the title Technical Leader (except one guy who was a Software Lead for reasons I don't know).
There was also a push for non-technical employees to all be managers, even if they didn't manage anyone. Our accountant was the Accounting Manager, person responsible for training sessions was the Technical Training Manager, the people who wrote our marketing collateral were the Marketing Communications Manager and the Sr. Manager of Technical Marketing, etc.
Basically, everybody below the director level was either an engineer, a manager, or a technical leader.
If you want software engineering to be certified by a test, you should be advocating a relevant test. My career predates the web by years and yet I've never needed to know the shear strength of copper or the resonant frequency of an RC circuit to make a server work.
I actually interpret their data quite differently. There is a statistically robust and positive correlation between perceived and actual ratings:
- 44% of the time, the actual and perceived ranking was identical.
- 91% of the time, the difference between the rankings was at most one star.
- Only in 9% of cases was there was a strong disagreement about how well the interview went (2 or more stars out of four).
- There is also a bias towards engineers thinking they did slightly worse than they actually did.
Even in an alternative universe where the same interview was repeated, you would not expect rankings by the same individual to be perfect. A more meaningful benchmark (though probably unattainable) would be interviewer/interviewee consistency relative to interviewer self-consistency.
That's what I thought as well. I'm not sure it's a great idea to fit a straight line and then look at the R2. After all there's only three other values it can be, rather than a normal distribution.
I think it's arguable that, when the question you're asking is 'how well do X and Y two things correlate with each other', the statistic you should be giving is r (which here is 0.49), not r^2. r^2 is the proportion of the variance of Y that's explained by X -- not that that isn't an interesting thing, but ITSM the correlation coefficient is more to what people expect a measure of relationship to be. (IANAStatistician)
The problem with that interpretation is that it ignores how much something deviates from the expected random. If everything was completely random:
- 25% of the time, the actual and perceived ranking would be identical.
- 62.5% of the time, the difference would be at most one.
- Only in 37.5% of cases would there be a strong difference in scores (2 or more stars out of four).
Would you say that 44% is a good score on a multiple choice test? Obviously candidates did better than random guessing, but not by enough to call it a strong correlation. Determining the strength of a correlation is what R-squared is for, which makes it the right choice for this analysis.
But the problem with R-squared in this case is that a linear model isn't appropriate for ordinal data. These rankings are ordered categories, not numbers.
The comment I was replying to was not about ordinal data and I stand by it in its context.
But on your point, the Jury's still out on what to do with Likert-type data like this. Purists do say that it's ordinal and therefore you can't do any numeric analysis on it, but practically speaking you can learn a lot from it by treating it as interval data.
In my opinion, statistical analysis like this is ALWAYS two parts speculation and one part science (if you're lucky). It's designed to give insight without declaring fact so you're allowed to bend the rules a little if it's in the spirit of the math.
My "gut" feeling agrees with the original post. If we treat the data as ordinal data, then we can only look at the accuracy, and 44% is, if anything, more compelling.
Cards on the table: I was involved in analyzing the data in the first place and have a vested interest in things-- but I am saying what I believe without any intentional bias.
Nobody can judge their own interview performance - engineer or not. The hard part is that you as an interviewee don't really know what the interviewer is looking for. You may have nailed the back half of the interview, but you don't have the one skill they're looking for. Or sometimes you don't blow it at all and they love you, but there's one other person they interviewed for the position that blew them away.
Trying to guess how well you did in an interview is folly. Hell, one interview tactic I use with people is to push them until they break: I keep asking technical questions until I reach a point where you say "I don't know". This sometimes makes people think they "failed" the interview, when all I'm really trying to do is see how much you know while also seeing how much you'll bullshit me on topics you don't know as well. Nobody can be an expert on everything, and interviews are as much about behavioral profiling as they are subject matter expertise.
I never cared about my own "interview performance".
But I always cared about my interviewer's assessment of me.
So much so that I've always reserved the last 10 minutes of every interview just to find that out.
They almost always ask if I have any questions, but even if they don't I ask them anyway:
Based on what you've just learned:
- How well would you say I did in this interview?
- How well do my skills fill your requirement?
- Where do you see me in the first month?
- Where do you see me in 3 years?
- What's the first thing you'd have me work on?
- What gaps do you think I have?
- What are your 3 biggest concerns about me?
- What can I do to address those concerns?
and of course:
- What is the next step?
I have never had an interviewer afraid to honestly (or appear to be honest) answer these questions. See how easy?
When I'm an interviewee I do the same thing. Opportunities to get interview feedback are already scarce, best to maximize them with questions like this.
When I'm interviewing, I always do a portion at the end to tell them at least one thing that impressed me and one thing that gave me pause.
The last 10 minutes where you ask questions is a bad time to waste asking someone to judge your 3 year plan when they just met you. Most of these questions are better asked when you fail an interview and request feedback as to why.
I specifically state when I perform interviews that I will not answer the question "How well would you say I did in this interview?" The interview isn't over yet, and that's impossible to answer.
You should really spend this time to see how much you would want to work at the company. Ask about their processes, the things that would absolutely turn you off to a company or the things that would absolutely turn you on to a company and make you forget about all their other bullshit. Ask about the problems they are trying to solve. Ask about how smart the other people are who you'll be working with.
Don't waste time in the interview checking if you passed. You'll know very soon anyway and there is always the chance to get this feedback afterward.
> Don't waste time in the interview checking if you passed. You'll know very soon anyway and there is always the chance to get this feedback afterward.
I think that would be a mistake. If the interviewer isn't able to answer those questions satisfactorily, it might be a sign that the hiree isn't a good fit. Nobody wants to change jobs to something that doesn't work out.
Just to add another datapoint, some companies don't allow their interviewers to give feedback on how well you did right there. "How well would you say I did in this interview" would not be a question they could answer.
And "How well do my skills fill your requirement?" may not apply for some companies, where their interviewing practices don't look for a specific requirement but go for a more general set of characteristics.
Not commentating on whether this is effective or not, just that it is the case at several of the large companies you might imagine applying for these days.
I might point out that it's not in the company's best interest to answer any questions about your performance. If an interviewer gives you feedback, and they say something wrong, it could open the company up to a discrimination lawsuit.
You're better off asking questions about the business and trying to determine whether they're a good fit for you than you are deciding whether they're impressed by you or not. By assessing their business and their fit for you, you'll decide whether you want to accept any offer they may or may not decide to offer. If they don't offer, they weren't impressed, and they still shouldn't provide you much feedback because of aforementioned liability issues. If they do offer, then they were impressed, and you can ask those questions.
Maybe resume's specifically do suck, but surely it's beneficial to have at least some way to see an overview of what a candidate has accomplished in the past. Especially when you're hiring for more senior roles where you need to place more value on experience and a proven track record than raw technical excellence. I can't really see these "anonymous" interviewing platforms working well for those kinds of roles.
I have seen hundreds of CVs that show no achievements of note - and tend to be more along the lines of describing the duties of the job and little else besides. I've also had hundreds of subsequent conversations with those people that revealed significant accomplishments.
CVs are better than nothing. But there's no set way to write them that's widely known, and people tend to be rather reticent about their own accomplishments - perhaps not even thinking of them as such. There's a lot of value being left on the table.
Beat me to it. When I interviewed at Google there were multiple interviewers and then there were multiple committees and people who look at their feedback.
Comparing interviewers to other interviewers makes a "He did above average" from a person who's hard to impress as good as a "She did really great!" from someone else.
Being easy to work with is certainly something interviewers should look for, but software doesn't care how nice you are. Your technical competency should ideally be measured outside of any subjectivity. Of course that's not possible but companies try their best.
It's incredibly difficult to measure technical competency quickly. If you don't spend a lot of time writing string functions then don't ask about them in interviews.
The problem is very few things are short term tasks in isolation. So, someone that takes a while to solve 6h problems can be really efficient in the long term. Worse, short term problems need to fit the same type as what your actually doing.
PS: As soon as someone says "i don't know" you need to stop and realize they would spend a few minutes on google. Spending more time on the topic at this point is pointless.
> PS: As soon as someone says "i don't know" you need to stop and realize they would spend a few minutes on google. Spending more time on the topic at this point is pointless.
Or, give them the answer and see what they can do with the information. IE, pretend that they DID lookup the answer and go from there. Did they grasp the concept? Could they make logical inferences from that new concept? If they can't do it when you tell them the answer, why would you think they could if Google told them it instead?
Googling gives someone time not just an answer. Sure, for trivia an answer is all you need, but trivia is a waste of interview time. On the other hand if someone looks up a formula they can spend 5 minutes actually understanding it which feels like an eternity in an interview.
Remember the context is the Job not the interview. On the job, having not done something for 3 years is a minimal setback for most people as the retraining time is almost meaningless. In an interview that 'highly' skilled person is going to take a long time to get back up to speed.
a) At my uni 15 yrs ago. Write a code that listens on certain input. You upload the source code, a bot compiles it, runs a bunch of functional test, then it tells your score. It didn't help what you missed, just a score, like Your solution passed 225 test out of 250.
b) Find a bug in a source code. Describe the end-user captured bug, and measure the time and the quality of the solution.
c) Simulate a distributed code review and measure the found problems. This can be extended to UX as well.
Your coworkers definitely care how nice you are, and the degree to which your technical competency matters is gated by your ability to work effectively with the team.
I was once in an interview where the interviewer was a hard ass. Asking all kids of hard questions about languages I knew, then he got deeper and asked about specifics into those languages. Then started asking about really abstract stuff like different sorting algorithms, OOP concepts like polymorphism, etc. I felt I was holding my own but as the interview went on the interviewer seemed to grow more impatient and irritable. Needless to say I left the interview thinking I blew it. Two days later I got a call from the recruiter saying I was their top choice and got an offer soon after.
P.S. It turned out that guy was just grouchy because he had interviewed so many underwhelming candidates. He admitted he was just trying to break me.
Trying to break a candidate seems to be very related issue to the one the article is talking about. Most people will take that kind of interviewer attitude as a negative experience that reflects poorly on the company and would make them less interested in taking the job.
The goal of the interviewer should be to evaluate a candidate, not dig and dig until you find a weakness. If it takes that much effort it's not worth doing in the first place.
So many of you 20-somethings interviewing really make me cringe! Basically, many of you are a stark mirror in which I can see how terrible I was as an interviewer in my 20's and 30's.
Be honest now, do you do this?
1) See something or think of something
2) Decide that it's the most important point
3) Go on a "fishing" expedition to try and get the interviewee to see it
4) If the interviewee can't figure out what you're hinting at, decide
s/he's completely worthless! (Or if she gets it, decide she's the ultimate!)
A lot of you do this. Cut it out. It's idiotic. It presupposes a pretty arrogant view of the relative value of your own perceptions. What you really need to be doing is listening.
I had the experience of someone doing this to me with concurrency, then concluding I'm an impostor, while I'm telling him about 3 concurrent systems I've worked on. The disconnect? He was "fishing" for the current undergrad spiel about scheduling. Sorry, we covered it, but that wasn't emphasized in the same way when I was an undergrad, and the poorly thought out puzzle you came up with on the spot wasn't such a great hint.
Given the number of times I was mis-identified as an impostor programmer during interviews (a minority of instances, but still) there is clearly something going on. You guys are just like the young Ivy league grads in Mad Men. You just have different clothes and different language, but you are looking out for the signs of your own tribe just the same.
>And by extension, it means that in every interview cycle, some portion of interviewees are losing interest in joining your company just because they didn’t think they did well, despite the fact that they actually did.
I hate the assertion of causation here. As always, this relationship means that A causes B, B causes A, and/or A and B have a common cause (or even more possibilities, given that we're essentially conditioning on this event taking place). Aline argues that "some portion of interviewees are losing interest in joining your company just because they didn’t think they did well," but I think it's more plausible that there is something that happens in the interview that simultaneously makes the interviewer think they did poorly and makes them think poorly of the company. If you're rating me on my ability to name country capitals in a statistics interview, I'm going to think you don't want to hire me and I'll think you're an idiot I'd hate to work for. Or if you're being kind of a jerk or being impatient and checking your cell phone every 30 seconds, I'll think I'm doing poorly and that you're rude and I wouldn't like to work with you. Common cause seems much more likely to me.
74 comments
[ 4.1 ms ] story [ 92.0 ms ] threadI also think that the dynamics differ based on the mode of interviewing viz. Faceless (like http://interviewing.io does) vs face-ful (like normal onsite or video interviews, or the way http://interviewkickstart.com does).
I have an observation to share for the latter kind of interviews:
When I was hiring at Box for one of my teams (or even now), I'd generally go towards the end, to meet the candidate and talk to them about how the day went. I'd ask them about specific interviews, the interviewers and how they think they did on the given problems.
I found that it was surprisingly rare to get a candidate who was aware enough of how things went (positive or otherwise).
Digging into it, I realized that the correlation of what the candidate perceived, was simply to how expressive the interviewER was. If the interviewer was friendly and chatty, the candidate would feel like they have done well (or at least forgivably well). If the interviewer was more silent, the candidate would feel they haven't done very well.
Now that I do interview training for a living, I try and hammer this point home: that interviews are like a date. It depends a lot on the other person (in this case, the interviewer). Understanding this well often works like magic.
I found the analysis very interesting and consistent with my personal bias (I almost always think I did worse than I actually did).
We invited the submitter to repost it for that reason. I'll try to merge the comments from there into here.
Invited reposts are mostly deprecated now in favor of re-ups [1], but when it looks like the submitter might also be the author (as e.g. with Show HNs), we still send them. It's nice for an author to know that their post may still get discussed, and it's good for HN when an author jumps into the thread.
[1] All the details at https://news.ycombinator.com/item?id=10705926 and previous posts linked there.
It would be interesting to see if you can replicate experiment but replace the 5 stars with 5 yes/no questions regarding their performance and see if those match up
If you have a true rating system then 2.5/3 out of 5 means average or more or less what you should be looking for in most cases 4 is out standing and 5 is bloody icecream pooping unicorn.
You see this every where a rating system is implemented heck in entertainment movies and games don't fight from 0 to 100 anymore but form the 90-100 if you see any score below 90 or 4 stars people usually avoid that product.
In other interviews, I've massively overestimated my own performance, seeing positive signals everywhere and being very impressed with myself-- of course, these didn't pull through, and my confidence was dashed. In another incidence, I did get the job offer.
The most confusing are the interviews in which I don't have a strong feeling of doing well or poorly either way-- not the median case, but certainly it happens frequently enough that I've gotten a job where I didn't think that I stood out. If anything, these cases are a consolation that others may see something positive that I don't see in myself.
This tells me that my ability to correctly predict my interview performance is no more efficacious than random chance. I think that there are a few confounding factors which make interviewee measuring interview performance as perceived by the interviewer difficult. Interviews are a time of endless posturing, flagrant lies, propaganda, and overt deception hopefully mingled in with actual mutual interest, excitement, and good will. There are many social signals to keep track of in addition to self-monitoring to ensure the correct outcome. For people who are not the strongest socially, I think that these social signals tend to get dropped intermittently as attention shifts inward (don't say the wrong thing) or outward (get this technical problem correct).
The thing is that I already bring a lot of value - I could definitely bring more, but I failed to emphasize what I already have.
That is unless you're a webdev, in which case you can call yourself head fruitloop and it would still be acceptable, apparently
An FE is required in order to be hired at a firm providing engineering services "to the public", such as a civil engineering or architecture practice. It is not required, or in some cases even valued, at product companies.
At the very least, not in the US you don't.
/has title inflated job title of Engineer, not a web developer.
I've never met a engineer who gave a crap what my title was, software developer or software engineer. Which isn't a surprise, since we pose no threat to the jobs of "real" engineers. Not sure why the word is so threatening to some software developers.
Do railroad engineers have to put up with this?
2. Definition of engineer:
Your statements misunderstand the definition of an engineer and devalue the accomplishments of people who apply math, science, and ingenuity to [animate things on websites] [make a website load in requisite time] [bend the constraints set by the browser to achieve certain objectives] [...insert other software engineering tasks].3. It sounds to me that you feel hurt over credentials. What you are projecting: "I worked hard for this thing and I perceive that you are devaluing it." My response is, credentials are different in software engineering, sorry. And nobody is devaluing your mechanical engineering degree here; if it's something in your field that is valuable, that's good for you.
But mechanical engineering isn't software engineering, and you can't say x isn't engineering, you have to understand what the accepted definition is.
I once worked at a company (2007-2010) where almost every non-manager had "engineer" in their job title.
Aside from Software Engineers, our QA staff were Test Engineers, our customer support people were Customer Support Engineers, our tech writer was the Technical Documentation Engineer (that was a late addition: his predecessor was simply a Sr. Technical Writer), our pentesters were Vulnerability Research Engineers, our sysadmin was the IT Systems Engineer, and probably a few more I can't remember. The only exceptions were the tech leads in development, QA, and support, who all had the title Technical Leader (except one guy who was a Software Lead for reasons I don't know).
There was also a push for non-technical employees to all be managers, even if they didn't manage anyone. Our accountant was the Accounting Manager, person responsible for training sessions was the Technical Training Manager, the people who wrote our marketing collateral were the Marketing Communications Manager and the Sr. Manager of Technical Marketing, etc.
Basically, everybody below the director level was either an engineer, a manager, or a technical leader.
Not to mention the not insignificant number of EE who have fully transitioned over to software, some of whom may have taken the exam at some point.
Likewise, not all "real" engineers have taken the PE or FE exam. Lots of electrical engineers don't bother unless their job requires it.
Are they being mean because they think I wasted their time or because I killed it and they want to push me?
- 44% of the time, the actual and perceived ranking was identical.
- 91% of the time, the difference between the rankings was at most one star.
- Only in 9% of cases was there was a strong disagreement about how well the interview went (2 or more stars out of four).
- There is also a bias towards engineers thinking they did slightly worse than they actually did.
Even in an alternative universe where the same interview was repeated, you would not expect rankings by the same individual to be perfect. A more meaningful benchmark (though probably unattainable) would be interviewer/interviewee consistency relative to interviewer self-consistency.
I think it's arguable that, when the question you're asking is 'how well do X and Y two things correlate with each other', the statistic you should be giving is r (which here is 0.49), not r^2. r^2 is the proportion of the variance of Y that's explained by X -- not that that isn't an interesting thing, but ITSM the correlation coefficient is more to what people expect a measure of relationship to be. (IANAStatistician)
- 25% of the time, the actual and perceived ranking would be identical.
- 62.5% of the time, the difference would be at most one.
- Only in 37.5% of cases would there be a strong difference in scores (2 or more stars out of four).
Would you say that 44% is a good score on a multiple choice test? Obviously candidates did better than random guessing, but not by enough to call it a strong correlation. Determining the strength of a correlation is what R-squared is for, which makes it the right choice for this analysis.
But on your point, the Jury's still out on what to do with Likert-type data like this. Purists do say that it's ordinal and therefore you can't do any numeric analysis on it, but practically speaking you can learn a lot from it by treating it as interval data.
In my opinion, statistical analysis like this is ALWAYS two parts speculation and one part science (if you're lucky). It's designed to give insight without declaring fact so you're allowed to bend the rules a little if it's in the spirit of the math.
My "gut" feeling agrees with the original post. If we treat the data as ordinal data, then we can only look at the accuracy, and 44% is, if anything, more compelling.
Cards on the table: I was involved in analyzing the data in the first place and have a vested interest in things-- but I am saying what I believe without any intentional bias.
Trying to guess how well you did in an interview is folly. Hell, one interview tactic I use with people is to push them until they break: I keep asking technical questions until I reach a point where you say "I don't know". This sometimes makes people think they "failed" the interview, when all I'm really trying to do is see how much you know while also seeing how much you'll bullshit me on topics you don't know as well. Nobody can be an expert on everything, and interviews are as much about behavioral profiling as they are subject matter expertise.
But I always cared about my interviewer's assessment of me.
So much so that I've always reserved the last 10 minutes of every interview just to find that out.
They almost always ask if I have any questions, but even if they don't I ask them anyway:
Based on what you've just learned:
and of course: I have never had an interviewer afraid to honestly (or appear to be honest) answer these questions. See how easy?When I'm interviewing, I always do a portion at the end to tell them at least one thing that impressed me and one thing that gave me pause.
How do you phrase or bring up your praise/concerns?
I specifically state when I perform interviews that I will not answer the question "How well would you say I did in this interview?" The interview isn't over yet, and that's impossible to answer.
You should really spend this time to see how much you would want to work at the company. Ask about their processes, the things that would absolutely turn you off to a company or the things that would absolutely turn you on to a company and make you forget about all their other bullshit. Ask about the problems they are trying to solve. Ask about how smart the other people are who you'll be working with.
Don't waste time in the interview checking if you passed. You'll know very soon anyway and there is always the chance to get this feedback afterward.
I think that would be a mistake. If the interviewer isn't able to answer those questions satisfactorily, it might be a sign that the hiree isn't a good fit. Nobody wants to change jobs to something that doesn't work out.
And "How well do my skills fill your requirement?" may not apply for some companies, where their interviewing practices don't look for a specific requirement but go for a more general set of characteristics.
Not commentating on whether this is effective or not, just that it is the case at several of the large companies you might imagine applying for these days.
Translation: The specific requirements aren't explicit.
You're better off asking questions about the business and trying to determine whether they're a good fit for you than you are deciding whether they're impressed by you or not. By assessing their business and their fit for you, you'll decide whether you want to accept any offer they may or may not decide to offer. If they don't offer, they weren't impressed, and they still shouldn't provide you much feedback because of aforementioned liability issues. If they do offer, then they were impressed, and you can ask those questions.
Do they?
Maybe resume's specifically do suck, but surely it's beneficial to have at least some way to see an overview of what a candidate has accomplished in the past. Especially when you're hiring for more senior roles where you need to place more value on experience and a proven track record than raw technical excellence. I can't really see these "anonymous" interviewing platforms working well for those kinds of roles.
I have seen hundreds of CVs that show no achievements of note - and tend to be more along the lines of describing the duties of the job and little else besides. I've also had hundreds of subsequent conversations with those people that revealed significant accomplishments.
CVs are better than nothing. But there's no set way to write them that's widely known, and people tend to be rather reticent about their own accomplishments - perhaps not even thinking of them as such. There's a lot of value being left on the table.
Comparing interviewers to other interviewers makes a "He did above average" from a person who's hard to impress as good as a "She did really great!" from someone else.
So the committees has a decent way of correcting for interviewers who use different scales.
The problem is very few things are short term tasks in isolation. So, someone that takes a while to solve 6h problems can be really efficient in the long term. Worse, short term problems need to fit the same type as what your actually doing.
PS: As soon as someone says "i don't know" you need to stop and realize they would spend a few minutes on google. Spending more time on the topic at this point is pointless.
Or, give them the answer and see what they can do with the information. IE, pretend that they DID lookup the answer and go from there. Did they grasp the concept? Could they make logical inferences from that new concept? If they can't do it when you tell them the answer, why would you think they could if Google told them it instead?
Remember the context is the Job not the interview. On the job, having not done something for 3 years is a minimal setback for most people as the retraining time is almost meaningless. In an interview that 'highly' skilled person is going to take a long time to get back up to speed.
b) Find a bug in a source code. Describe the end-user captured bug, and measure the time and the quality of the solution.
c) Simulate a distributed code review and measure the found problems. This can be extended to UX as well.
(These are just quick examples.)
P.S. It turned out that guy was just grouchy because he had interviewed so many underwhelming candidates. He admitted he was just trying to break me.
The goal of the interviewer should be to evaluate a candidate, not dig and dig until you find a weakness. If it takes that much effort it's not worth doing in the first place.
Sounds like a great guy to work with.
Be honest now, do you do this?
A lot of you do this. Cut it out. It's idiotic. It presupposes a pretty arrogant view of the relative value of your own perceptions. What you really need to be doing is listening.I had the experience of someone doing this to me with concurrency, then concluding I'm an impostor, while I'm telling him about 3 concurrent systems I've worked on. The disconnect? He was "fishing" for the current undergrad spiel about scheduling. Sorry, we covered it, but that wasn't emphasized in the same way when I was an undergrad, and the poorly thought out puzzle you came up with on the spot wasn't such a great hint.
Given the number of times I was mis-identified as an impostor programmer during interviews (a minority of instances, but still) there is clearly something going on. You guys are just like the young Ivy league grads in Mad Men. You just have different clothes and different language, but you are looking out for the signs of your own tribe just the same.
It's interesting isn't it? This was all much easier in decades past. The questions don't matter. The answer doesn't matter.
What matters are the body language cues.
[1] https://en.m.wikipedia.org/wiki/Dunning–Kruger_effect
I hate the assertion of causation here. As always, this relationship means that A causes B, B causes A, and/or A and B have a common cause (or even more possibilities, given that we're essentially conditioning on this event taking place). Aline argues that "some portion of interviewees are losing interest in joining your company just because they didn’t think they did well," but I think it's more plausible that there is something that happens in the interview that simultaneously makes the interviewer think they did poorly and makes them think poorly of the company. If you're rating me on my ability to name country capitals in a statistics interview, I'm going to think you don't want to hire me and I'll think you're an idiot I'd hate to work for. Or if you're being kind of a jerk or being impatient and checking your cell phone every 30 seconds, I'll think I'm doing poorly and that you're rude and I wouldn't like to work with you. Common cause seems much more likely to me.