Ask HN: Was any Starfighter postmortem ever published?
About seven years ago, HN veterans 'elptacek, 'patio11 and 'tptacek launched an hiring startup called Starfighter.
Their approach aimed to shift away from traditional interviews and instead employ a CTF/Microcorruption kind of test, inspired by 'tptacek experience at Matasano.
Unfortunately, Starfighter eventually wound down.
Was there ever any post-mortem written on Starfighter? If not, would 'elptacek, 'patio11 and 'tptacek be willing to provide any details?
176 comments
[ 3.1 ms ] story [ 232 ms ] threadhttps://www.kalzumeus.com/2016/12/30/kalzumeus-software-year... says the same thing.
Did that ever happen?
https://www.kalzumeus.com/2016/12/30/kalzumeus-software-year...
I think I might've been the one referred to as submitting the excel file with the explanation... as I think that was how I solved the last problem on the stockfighter thing.
Slightly OT: everyone feels hiring is broken, can you list some things that are annoying from the employee and employer perspective? Here are some points:
- the process often stretches out over weeks and often months
- job posts often get 100s of applications, a lot are low effort applications, it just muddies the water for both sides
- ATS systems/job boards are annoying with the need to create an account on many sites, some forms have more than 20 questions, often asking what's already there in the resume.
A question to everyone: What would a good application process look like? For me, it should just solve the above mentioned problems. I send an email with my resume, a few sentences about why I might be a good fit for the role/what interests me about the company. The jobs@<company>.com email address could be linked to some Saas product which makes it easy for the employer to go through the applications and further communication about video calls or take home assignments or whatever are all in this email thread. The employer can set the stage of the application such as 2/5 or whatever, they can mark it as rejected or accepted after all rounds to trigger automated emails etc. Is there any Saas like this? (I can build this in a week if it doesn't exist, but no clue how to market it/get users, any pointers in case me or someone else builds this?)
Especially because lowering the barrier usually means getting more low effort application.
People either put a lot of effort on few applications and invest significant amount of time in each one hoping to increase the chance they get picked up; or they put a lot of effort on reaching out as much as possible, knowing they have a very small chance on each application but playing on volume.
Most recruiters are only interested in the first category of candidates. Having a long initial process is a feature. The goal is that the second category considers that given their chance of getting picked up, it's not worth filling the form.
Reality is:
1. Screening tools (background check, skills analysis and testing) are really good. There's really no reason to put someone through a six-week gauntlet just to find out if they can do the job.
2. Interviews are really a tool for seeing if coworkers and managers get along with the candidate, and the candidate gets along with their future coworkers and managers. In many companies interview == where manager has discretion to hire the people they want. Lots of interviews usually means lots of managers who all want to put their stamp on the hire.
3. Being good at getting hired is a function of experience getting hired. The skills needed to be a good candidate do not align with the skills needed to be a good worker in most roles.
3. Agree, it's simplified, but for most companies, field and "hiring" managers are barely able to do an effective interview, let alone smoke out someone lacking interview skills from one lacking communication skills.
Unfortunately hundreds of other people found it just as easy.
This is why I never blindly submit my resume to an ATS. I only did it this time because I was unemployed (for two weeks) and had nothing better to do.
The five opportunities that led to interviews were still based on good old fashion networking
1. A full time offer (didn’t accept) and a side contract (did accept) based on referrals from former managers
2. Two interviews based on reaching out to the internal recruiter for positions I was objectively the best person on the market for - I was a major contributor to an open source official “AWS Solution” they were implementing and modifying. I accepted an offer.
3. Responding to a recruiter he reached out to me. There was some miscommunication about the role. The team thought the client wanted someone “local” to the US since their whole team was in India. They wanted someone local to the city - I declined.
Out of the 100 applications I randomly spammed, most didn’t get around to even looking at my application and only four actually downloaded my resume.
A complete overhaul of credentialing, so that when I have a degree in computer science from a top university and decades of industry experience, employers don't need to independently confirm that I know how to reverse a string or traverse a tree or define what XSS is.
Also, make a time machine so we can make this change 30 years in the past, if you wouldn't mind.
Similarly lots of people with significant industry experience… also can’t code.
Everyone involved in hiring should read “The Case Against Education” by Caplan.
The problem is computer science university degrees aren't enterprise training academies. "Computer Science is no more about computers than astronomy is about telescopes." Universities aren't incentivized to fix this because universities are research institutions - they don't make money from placing students in jobs, so trying to solve this is a dead end.
[1] https://blog.codinghorror.com/why-cant-programmers-program/
Government pressure for hiring quotas goes back to the 1970s.
in the networking side of these we used to have a couple of Cisco 2960s with slightly borked configs and our interviewees had to log in and tell us what's wrong. 5 minute activity, but often took some folks 30 minutes.
And yet, when scanning resumes, people often look for prestigious/"reputable" universities as a filter. Is the assumption that if they have a degree from Stanford and can reverse a string then that is a positive signal, but if they can't reverse a string that is somehow a personal failing and not a failure of the university to educate (and then evaluate) the candidate?
What's the alternative? Just give leetcode tests to people who didn't go to the same university as the hiring panel?
The fact that we have the same tests for everyone no matter where they studied at is a good thing and prevents nepotism in hiring decisions.
What I am noticing on all threads about HN hiring: Hiring knowledge workers is fundamentally a difficult, fuzzy process. Few want to admit it. And, there are some people are naturally much better at it because they can control themselves very well during an interview. People who are very strong at solving puzzles will nearly always outperform during interviews for hands-on roles. More than anything, interviewing feels like dating. Everyone hates it until you find what you are looking for... then it becomes: "Oh, it worked fine for me."
That's why the time machine is so important - to get back any CS degrees issued to people who can't code.
So I think you are actually agreeing with me?
-Dave Barry
I’ve never regretted a single person I hired and they have always been able to do the simple CRUD work that most of the 2.7 million developers are doing.
And before the gate keeping starts. I’ve been through the “Making Great Hiring Decisions” training and have conducted behavioral and system design interviews at AWS.
Sounds like a good way to filter for the right class background. (Which, given how well that's correlated with intelligence, is probably as good a hiring method as any)
Let’s just say when I walk through my neighborhood I’ve read comments about me in my own NextDoor group about someone “suspicious” and realized they were talking about me or my son…
This is why I find it so frustrating to talk to Americans.
> Let’s just say when I walk through my neighborhood I’ve read comments about me in my own NextDoor group about someone “suspicious” and realized they were talking about me or my son…
So you're a member of the class that lives in "good" neighbourhoods where it's suspicious for certain types of people to be? I think you've proved my point.
But back on topic. I base my interviews on behavioral traits that for framing, can be sussed out by asking “tell me about a time when” type questions.
You did see the part about I’ve gone through the interview training by the second largest employer in the US didn’t you?
It doesn’t matter if you can reverse a binary tree on the whiteboard if I’m looking for someone who “takes ownership”, “digs deep”, and has shown the ability to deal at the level of “scope” that I’m looking for an knows how to “deal with ambiguity”
My current opinion is that while it's effective, it means there's a significant bottleneck (sometimes a single person) for the hiring pipeline. i.e. you still need at least one earlier tier of interview to winnow the candidates down to a volume that's manageable for the bottleneck, and the earlier tier needs to be something more standardized that can be delegated to a larger group of people. i.e. "can this person explain common technical details A, B, and C of work in this department well enough to demonstrate solid understanding?"
[1] I read years ago that border guards in some countries use a similar technique, because if someone is being deceptive, it's unlikely they can handle a random walk to an arbitrary depth of detail.
It’s based on the Leadership Principles at Amazon. Not that Amazon is a great place to work in practice (I just got Anazoned two months ago).
But they talk good game.
https://managementconsulted.com/amazon-leadership-principles...
I have a sticky note of the LPs I care about. A lot of times when I just ask a generic “tell me about yourself” question, they will talk about their experience and I can just ask them more details around their project that let’s me get the information I want.
Other times, I have to specifically pull out my questions. For instance, I really care about having someone who is willing to speak up in meetings and is willing to express their opinions and even someone who is willing to tell me they think I’m wrong. I speak very authoritatively about my areas of expertise. But I always try to emphasize “strong opinions weakly held”.
In LPs terms it’s called “Have backbone disagree and commit”.
My favorite question for that is “Say I’m your manager and you are in the meeting with me , the CTO and other managers and we come up with an idea that you think is completely stupid. What would you do?”
And that starts an entire conversation. I’ve given a thumbs down to people who give me an answer that I don’t think is a fit for what I’m looking for.
I love for people to give me feedback and I hire people who help my team fill in technical gaps. I need people to speak up
Such people are not necessarily deceptive; in my experience, much more often
- the person does not have the best memory, but loves to talk
- the person has a tendency to "hallucinate" (i.e. has some mild (typically harmless) schizophrenic traits)
> It doesn’t matter if you can reverse a binary tree on the whiteboard if I’m looking for someone who “takes ownership”, “digs deep”, and has shown the ability to deal at the level of “scope” that I’m looking for an knows how to “deal with ambiguity”
That definitely sounds like something that would filter more for "went to the right schools" (or, more generically, "mingled with the right people", but schools are the easy way). I could certainly believe it's as effective as "reverse a binary tree", but I very much doubt it's fairer.
Do you realize that scope, impact and dealing with ambiguity is literally in the leveling guidelines stated in some way or the other in every tech company?
https://dropbox.github.io/dbx-career-framework/
I’m not looking for someone who just graduated from college.
The right schools give you the opportunity and mentality to do that (and before that, even more fundamentally, so does having a comfortable upbringing), and then you'll compound that with your experiences in the good jobs that going to the right schools gets you.
> Do you realize that scope, impact and dealing with ambiguity is literally in the leveling guidelines stated in some way or the other in every tech company?
Everyone does it, that's exactly the point - you limit class mobility by ensuring that the good jobs are only available to the people from the right class. Even if someone from the lower classes has the skills to excel, they never get the chance to take ownership, learn new skills, and expand the scope of their impact because they're never hired for a position in which they can take ownership, take the time to learn, and have impact beyond their narrow position, because they never have a way to show that they can take ownership and learn new skills...
When I’m looking for the behavioral traits I wouldn’t care if they could only tell me stories about being the shift lead at McDonalds.
You realize I only got a job at a company that anyone has ever heard of at 46?
It’s called “work ethic”. Right now, I am building a team where the actual coding is really simple in the grand scheme of things. Since ChatGPT is well trained on the AWS SDK, the CDK, etc. you could literally use it to do 90% of the work.
But the organizational complexity, business rules, etc are complicated and we are still figuring everything out. I need someone with the following behavioral traits. I’m going to frame them in terms of the Amazon LPs since it gives everyone a publicly available point of reference:
- Customer Obsession
- Ownership
- Learn and Be Curious
- Bias for Action
- Dive Deep
- Have Backbone; Disagree and Commit
- Deliver Results
If you have any kind of work experience at the mid or senior level you should be able to demonstrate those.
I could answer “tell me about a time when” type questions three years out of college graduating from my no name HBCU in the south after my first job.
Which is fine if you've got a job with reasonable hours that doesn't leave you drained at the end of it, and the rest of your life is stable enough for that, and...
> I ... got a job at a company that anyone has ever heard of at 46
> It’s called “work ethic”.
Maybe. Or maybe it's luck. People are particularly bad at assessing which when it comes to their own history.
> I need someone with the following behavioral traits.
Or you need someone with something else that's correlated with them. Or what you need isn't really related to them at all.
> If you have any kind of work experience at the mid or senior level you should be able to demonstrate those.
Right. Once you've made the leap into the good jobs it's easy to stay there (and if you went to the right schools then it's easy to get the first good job, and if you're born to the right family then it's easy to get into the right schools). The trouble is getting there at all.
> I could answer “tell me about a time when” type questions three years out of college graduating from my no name HBCU in the south after my first job.
Sounds like you had a good first job.
It sounds like you’re making excuses. You mean you spend every hour of your life working? Are you suggesting that we should have no hiring criteria?
> Maybe. Or maybe it's luck. People are particularly bad at assessing which when it comes to their own history
You did see the part where I would find it perfectly acceptable to show the behavioral traits by working any job? Of course you do have to have technical experience - not even in the stack we are using. I can teach the specific technology.
> Sounds like you had a good first job.
My first job was a computer operator working on mainframes.
> You did see the part where I would find it perfectly acceptable to show the behavioral traits by working any job?
I'm suggesting that hiring criteria that reflect more directly on the candidate's abilities, and less on what opportunities they've had to date, are fairer. (And, distinctly but relatedly, the more objective the assessment is, the better for those from lower-class backgrounds). For someone who's grown up in a working class household where showing initiative had negative consequences and then gone through public schools where showing initiative had negative consequences through to line-level jobs where showing initiative had negative consequences, a tech interview that focuses primarily on whether they've got the skills, and lets them show it through their fizzbuzz/leetcode/hackerrank/what-have-you, is one of the few ways to make it out.
1. People who were too busy to study to learn how to code would definitely be too busy to grind leetcode as well as adults who are already in the industry with families
2. Okay you can reverse a binary tree on the whiteboard. That tells me nothing about whether you are the kind of person that I can assign a task like “a customer has some vague idea of what they want. I need you to figure out the details and come up with a proposal and a design and present it to them and get feedback”. Then if they don’t know the technology, I need them to be able to learn it. This is what I mean by “Customer Obsession”, “Learn and be curious”, “Digging Deep” and dealing with ambiguity. If I need them to lead a project, I need to know they are able to handle items dealing with “scope”.
I want to know if they have a history of “delivering results “.
These are all traits that you were required to show to get a return offer as an intern for a full time job as an L4 consultant working in Professional Services at AWS.
I threw tasks like this at a first year employee who I had mentored as intern and they passed everything I threw at then with flying colors.
At my current job, I’m working with a green field project where everything is ambiguous and I need someone I can throw a poorly defined objective at and they can take the ball and run with it. Leetcode isn’t going to tell me anything.
I think many more people have the opportunity to grind leetcode than have the opportunity to "take the time to learn a new to you technology" that they can then apply to a work project - which requires not just time but also having the right work environment in which they can apply it. Of course you're right that many people don't have the time for either, and a handful of professional-managerial class people who are already in good jobs might be able to do the latter during work hours but not the former.
> whether you are the kind of person that I can assign a task like “a customer has some vague idea of what they want. I need you to figure out the details and come up with a proposal and a design and present it to them and get feedback”.
> I want to know if they have a history of “delivering results “.
Do you think you're measuring something objective about their ability to do the job? To the extent that you are, do you think what you're measuring is something different / better than "has decent general intelligence + has existed in professional-managerial-class spaces (or, more specifically, in jobs like this one) for a while?" Do you have evidence about the answers to either of those questions?
The difference is that you are an active programmer, you’re actually programming. I graduated in 1996. I had to learn “new to me technology* as part of my job for 25 years on the job. I’ve never had to do anything approaching leetcode on my day to day job.
I’ve learned on the job:
- infrastructure and networking
- C++
- Perl
- JavaScript
- VB6
- C#
- Python
- litterally 3 dozen AWS technologies
- databases (OLAP, OLTP, key/value, document, etc)
- how to manage large projects
And pick up soft skills
> Do you think you're measuring something objective about their ability to do the job?
The best predictor of future behavior is past behavior. That’s where behavioral question comes from.
> To the extent that you are, do you think what you're measuring is something different / better than "has decent general intelligence
None of the behavioral traits I mention have any correlation to intelligence. I didn’t wake up one morning and learn how to communicate effectively .
Right, I'm saying that's a rarer privilege than being able to do leetcode or similar (whether during work hours or not).
> None of the behavioral traits I mention have any correlation to intelligence. I didn’t wake up one morning and learn how to communicate effectively .
If you think being able to communicate effectively has nothing to do with intelligence you're crazy. (Of course, it probably has have even more to do with having a common class experience than with intelligence)
How long have you been working? I doubt very seriously that anyone who works professionally hasn’t had to learn some new technology on the job. Out of the 2.7 million working software developers, I could say that almost all of them had to learn new to them technologies on the job. Also almost none of them did anything approaching leetcode on the job.
> If you think being able to communicate effectively has nothing to do with intelligence you're crazy. (Of course, it probably has have even more to do with having a common class experience than with intelligence)
What exactly point are you trying to make? That if I need someone who can communicate affectingly I should test that by using leetcode?
Yes, you can learn how to be an effective communicator much easier than you can learn how to pass a coding interview.
Sure. It's a great way to hire people who already have professional-class jobs, mostly already software developers.
> Yes, you can learn how to be an effective communicator much easier than you can learn how to pass a coding interview.
People from the "wrong" class aren't generally ineffective communicators, they just have a communication style that's adapted to their circumstances. PMC communication styles are easy to learn if you have access to an environment where everyone else is using them, but very hard otherwise. (Also, by your own logic if the communication style is easier to learn wouldn't you want to hire people for the harder skill and teach them the communication style on the job?)
On the other hand, in my experience such fraudsters are in most cases easy to detect: just ask them some hard questions (where the candidate will either know the answer or not, thus rhetorics or charisma won't help) about the topics that the candidate claims to have expertise in (choose the question in a way that even very capable candidates will only be able to answer a fraction of them - this is fine and intended).
Typically, after 30-45 minutes you are either sure that the candidate tells the truth about his capabilities or is a liar (hardly ever anything inbetween happens).
> know how to reverse a string or traverse a tree or define what XSS is.
In my experience, it is absolutely not safe to assume a correlation between these two sets of traits.
But 'top universities' don't know how to teach CS. That's the point. Don't waste your time there.
Maybe we should have a 'bar exam' type system, but the university system is a joke at this point, and there's no point trying to put lipstick on a pig.
But 'top universities' don't know how to teach CS. That's the point. Don't waste your time there.
Maybe we should have a 'bar exam' type system, but the university system is a joke at this point, and there's no point trying to put lipstick on a pig.
Their live coding questions were
1. Either practical , or core CS stuff. No fancy algorithms, just basic stuff you'd get in a CS 101 DS&A class.
2. Collaborative. No one cared about perfect code or syntax, it was very relaxed and dare I say fun (if it wasn't for the nerves).
3. System design was practical and a good back and forth.
The interviews were challenging overall, but nothing they asked seemed unfair or about memorization either.
Certainly coding under pressure sucks, but I don't think there's a better way.
I work in a top company with infamously difficult interview. I made major contributions to several opensource projects that modern tech runs on. I can easily show a clean standalone pull request attributed to me that introduces complex multithreaded code. That code runs daily on millions instances.
Why the hell do I need to spend hours answering “what a mutex is”? Also, understand that a lot of stuff is much less obvious and clear cut (e.g. how HTTP protocol works) for me than for your engineer. They only saw the basic case. For me the answer will be “it depends”.
Please, respect the candidate time and experience.
A lot of applicants will lie or bullshit something similar. How would you suggest an employer distinguish someone like you from someone pretending to be someone like you?
Any interesting technical position posted by a company with a good reputation gets so many unqualified applicants that I wouldn't believe it if I hadn't seen it over and over myself at different organizations. Every HR department I've worked with has done a good job of filtering out the ones that didn't even look good on paper. Out of the ones that get past that stage, I'd estimate maybe 1-5% actually have the technical qualifications for the job. Of course, even if they have the technical qualifications, they may be lacking in other areas, like professionalism.
The last three companies I've worked for have all used a custom CTF as one of the interview phases, and it's been invaluable at screening out people who were good at talking about the work, but not good at actually doing the work. So, superficially, I'd love to be able to use something like Starfighter instead of having to maintain one in-house. But how likely is it that a company like that would build truly unique challenges for every customer? IMO it would cause cheating to be easier, because people doing the CTF in bad faith could just work from lists of known elements used in previous CTFs from the same company.
As a data engineer I'd like to have a chat about the pain points of the current team and contribute. I think a coding test is OK too as long as it is just one round and actually test something meaningful.
I read it as an homage to HN days of yore.
Can you reduce hiring an airplane engine designer to a set of tests? Or a custom home builder? If so, why is there not just a stack ranking of best ones, or an algorithmic way of hiring one?
Software engineering at elite levels is an art. It is conducted by master craftsmen. On top of it the best people are often quirky / weird and the social aspect when communication of needs is so essential makes it even more difficult to hire "the best".
The business models of recruiting is always either like sales (recruiting firms) or marketing (LinkedIn). I would stay away from the recruiting industry as a software startup - difficult to make any money here in a novel way.
I guess it's a hard thing to certify when the landscape itself is constantly evolving and skills are obsoleted in 3-4 years. It seems even the pace of change is itself speeding up, and every year I see more technologies come to popularity (if not quite maturity) than I can even remember, much less learn. And it's not just languages and frameworks, but entire architectures... today's best practices are tomorrow's tech debt.
There should always be room for innovators to push the bleeding edge forward, but many businesses don't really need that. Instead, if there was just some shared certification for "good enough engineering" that can deliver a reliable, maintainable product somewhat on time, reproducibly, hiring and job-hunting would be so much easier. I'm so tired of trying to charm recruiters, impress managers, placate HR, all while hoping against hope that the actual dev 2 or 3 interviews later happen to have some overlap in languages, toolsets, shared pain points, etc. It's all such a dog and pony show, with many teams having ill-fitting skillsets and cultures between members because nobody really knows what they're signing up for or getting.
Software engineering is building a factory. The inputs are data. The outputs are data. All of the stuff inside the application are the different conveyor belts, elevators, gears, rollers, pounders, etchers, etc. to move and manipulate the data to make the output valuable. The factory can be constructed infinite ways without any care about physical dimensions, and the raw material can be duplicated to infinite amounts. The situation is beyond regulation and it should not be regulated. Only the product output should be regulated.
And in general I've seen little correlation between credentials and quality, both in the software industry and elsewhere, so I don't think any such certifications will really solve anything either.
Top-end devs make a name for themselves and don't really have to worry about hiring. Companies compete for them with crazy high offers.
Bottom-end devs try to make up for their lack of experience or skill by "hacking" the application or interview process somehow.
In between, "good enough" devs power the world's bog-standard apps. They're not going to win any awards or lead any talks, but they can do the jobs well enough to make the company money. They know they aren't top-end and can't just say "I'm quitting, who's hiring?" on twitter, but they are also competing with the hordes of less-skilled devs because there's no easy way to tell them apart.
I've never heard of "credentialism", but why doesn't it work in this instance, when it does for doctors, lawyers, engineers, etc.?
There are loads of bad doctors though, and bad lawyers, and bad engineers.
"Credentialism" refers to over-reliance on credentials, rather than just looking at what someone can do. I'm not against education, but education is not the same as credentials. People that can hack the interview process can also hack the credentials.
I've seen people with college degrees in IT employed as programmers that literally cannot program. I'm not talking "bad programmers" or "brain fart moment", I'm talking "consistently being unable to grasp even basic control flow over a period of months or years". Also some good programmers can be hard to work with for reasons other than their programming ability.[1] Plus not every programmer is suitable for every position or every type of programming or "jibes" with every other programmer in their sensibilities and approach.[2] The hiring process is just as much about that as testing basic skills.
No one thinks credentials are a guarantee of quality (people will always slip through the cracks no matter what), but I'm not sure they're even a strong indicator of quality either, certainly not for programming. This is hard to quantify of course, and just based on my own observations over my career.
[1]: https://news.ycombinator.com/item?id=21585958
[2] Simple example: I'm firmly in team "short names", but I know some people really hate it, but I'm pragmatic if nothing else and don't mind doing whatever works well for the entire team. On one extreme end I've worked in a team that insisted on the longest possible name for everything: every variable had to spell out the full name of the class where it came from, often >20 chars, otherwise people considered it "unclear what anything refers to!" I once added a mkdir() function, this was considered "obscure" so it had to be "CreateDir()", then someone said it may not be clear what "dir" is so "CreateDirectory()", and then another remarked it actually makes all intermediate directories too, so we ended up with "CreateRecusiveDirectoryTree()" – I'm not saying this is "wrong", but it does not jibe well with my sensibilities. Several other things also didn't, so I didn't last too long).
I've encountered several doctors, etc. who weren't very good, including one who shouldn't have been allowed to practice, IMO. Point taken.
Anyway, in your experience, what actually WOULD be a good indicator of candidate quality?
Your link talks about "good programmers that lack social skills". In my experience, politeness is a finite resource and dealing with people on a regular basis fully unable to do their listed job is a severe tax on mine.
Those who don't have to worry about hiring are not (necessarily) top-end devs, but good self-promoters and networkers. Not every top-end dev is a good self-promoter or good at networking.
My read on the approach is that it works very well for certain types of engagements, but other types, such as where you are trying to build a solid team of peers, requires something _closer_ to the traditional methods. That being said the Starfighter approach is probably right most of the time -- teams are almost certainly never stable, management usually treats ICs as fungible cogs and moves them around, in which case individual value should trump things like team fit or culture. But we're human.
Appropaches like Google's work better (n.b. I am far from a proponent of what they do) because the interviewers are human and despite the whole blind-panel thing if the interviewer and interviewee can't get along the interview won't go well. So culture and team fit _is_ evaluated significantly more than something like Starfighter. People care a lot about that implicit bit, in my experience.
Everyone has to be on board for it to work, I'd wager.
I tailored the work sample test to the role. I did it for the compliance hires and for the project manager. The results were unexpectedly excellent.
As a result of this process, and having a solid company culture, there are around ten or twelve folks that have a new career in the security field that would likely not have had another opportunity in the classical hiring scheme.
I think the state of the industry with respect to hiring and evaluating folks is broken across the whole spectrum. It is like once someone becomes established in a field, they invent all manner of gatekeeping tests that they didn't pass to get where they are. The whole story of sabermetrics is another example.
For very small shops willing to incur legal risk or limit the application pool, you can do take home tests and/or lean on side projects, with an hour of explaining the nitty gritty details and choices. At scale it becomes untenable, because you open yourself up for discrimination lawsuits, and you can't convince a lot of people who already have jobs and families to do lengthier take home stuff.
If I had to bet, I'd bet some day it ends up like plumbing. You work as an apprentice, learn real stuff on the job, and then there's some kind of state certification which itself is a joke, but requires you to have gone through an apprenticeship program already where all the real shit happens.
Companies trying to hire "the best" or whatever are still going to have problems, so as long as there's not much regulation increasing cost of dev and uniformity by an alarming degree, we'll still have insane hiring (and insane pay too, regs will make it unprofitable eventually).
Is that because when looking to make a possibly subjective judgement on the performance in a test and especially what a side project shows, it then becomes more difficult to prove that the judgement was not instead made because of some protected characteristic of the candidate?
Instead I'm always surprised more places don't rely on references and prior experience. Yes people lie, but in my experience its relatively easy to tell the difference with a simple glance at their LinkedIn. If I look at a candidate that spent 2-3 yrs as a Software Engineer at some company that I'm relatively familiar with, and they seem to have 2nds and thirds to people I know in the industry, then pretty good chance they aren't lying. Same for people taking the time out to write recommendations for them. Even bigger signal if they want setup some calls with their prior co-workers who can vouch for them, to me that means they stand by their work and reputation.
I recently went through about seven rounds for a senior role. During that time I repeatedly offered to setup some time with my prior coworkers from those I directly managed, to peers, to those I reported to (executive team). My thinking being that they could hear from the horses mouth how I lead a team, my work ethic, etc. They did not take me up on the offer, which to me was crazy. Yes, I could be running some machiavellian scam with 2-3 people who also made a fake LinkedIn, I also could have put up fake articles in PR Newswire announcing my last position, could have spoofed all those blogs I co-authored from the company I worked at 2 jobs ago, etc. But really, wouldn't it be a better signal for a candidate to offer and have all these things?
Instead you see an industry that puts someone with ~10 yrs experience through a whiteboard interview. It makes no sense.
Social stuff is absolutely the first thing I'm checking. It's a quick test to spot total lies, e.g. you claim you worked somewhere but are connected with zero people on any social network, or claim attendance at a school with zero connection to anyone. If nothing else, it helps to build rapport for the interview.
I would 100%, absolutely trust a personal recommendation over a dumb whiteboard interview.
There exist quite a lot of people who have no account on any social network for privacy reasons.
Every time I’ve had a company go longer than 2 rounds, another company got me an offer first.
Glassdoor had the gall to just ghost me after the 6th! Never again.
The problem is: there exist a lot of companies - you can only be familiar with a very small fraction of them. Also, for many big companies, the differences between departments or groups can be a lot larger than between companies of similar size and sector.
At the end of the day, programming is just a writing job. So if you apply for non-technical writing jobs, e.g. to be a columnist at The New York Times or The New Yorker, do they not read any of your preexisting work on the grounds that it could open them up to discrimination lawsuits? Similarly, do folks who hire designers typically not look at their portfolios?
If you work in an industry where nothing is published publicly, and then demand to see people's side blogs and use that for hiring decisions, then yea that'll make lawyers at a big company a little skittish.
It's a different thing when your job is to write things that everyone can see, vs having your company own everything you do.
Fake it to you make it is actually a real thing. Though you actually have to be able to make it. You can overstate qualifications and experience but not competence. I don't lie about anything but i've definitely known people who have lied in order to get hired who have gone on to perform extremely well.
People absolutely embellish and get away with it.
At the start of a SWE career, I don't think things broken. Study up some leetcode and write code on a whiteboard to prove that you, as a candidate, are smart and motivated enough to play the game. It's after that, that things get out of sync and you get absolutely amazing people who have written world changing software who can't pass your stupid puzzle of a whiteboard interview. And then what do you do? This person failed your test but you want to hire them. Here's the part where we fail, where we are too stuck on following the rules.
They failed the test so you can't make an offer? No. Move heaven and Earth to bend the rules, and make them the offer anyway. Don't hide behind corporate handbooks and the HR or the legal department.
Obviously this power can be abused, but I don't take it as a given that it must be, given a hiring panel.
Google could have made an offer to Homebrew's Max Howell but chose not to, which speaks volumes.
Programming is divided into a multitude of various cultures with their own (often incompatible) rules. Not without reason, you tend to talk of a "Java shop", "C# shop", "Python shop", ...
So, when looking for a good programmer, you either look for a beginner who is still "malleable" into your desired culture, or has the same right pedigree as your company.
In this sense, I a priori have no doubt that Max Howell is a decent programmer, but I do believe that he comes from and is attached to a very different programming culture than Google's; thus he would likely not be a good fit.
Oh, if only the worst was not contributing. I've been at a company full of brilliant programmers that was just driven into the ground by the senior architect, who wasn't necessary. Hiring someone who can't Fizzbuzz for that position (and giving him the authority to overrule the development leads) was absolutely the single decision that sank that company.
https://news.ycombinator.com/item?id=22258113
Notably virtually no big companies will give feedback by policy, because of their own lawyers. All the engineers on the other side would freaking love to give constructive feedback. Their own lawyers prevent them from doing so.
Microsoft settled three years ago on a race discrimination in hiring thing. And two years ago for discriminating against non-citizens (and in some twist, Facebook settled the same year for discriminating against citizens, so... damned either way).
Just this year Workday got sued for discrimination in hiring, which is mildly hilarious because they're the HR-as-a-service company.
> Or is it just the amorphous possibility of suits?
It's mostly this, driven by the lawyers. They've seen a few previous settlements and just decide "well, none of that here".
I don't know if you've ever done hiring at a big company, or sat through the legal training things, but even if you don't draw a lawyer who tells it to you straight, it's pretty easy to infer what's going on.
Give feedback -> get sued due to something in the feedback
The worry is:
Give feedback -> ??? -> get sued for discrimination-> feedback is used as evidence that the hiring decision wasn't for a concrete/justifiable reason once its torn apart by a lawyer in front of a jury of 12 random people, only 7 of whom you need to convince.
I did a lot of silly things after winning the initial play through.
- Solving the final challenge using a basysian spam filter on trade directions triples.
- Live 3D market visualization in Minecraft
- Buddy of mine built a stock exchange stack, speaking the same protocol, and we had PvP contests on who could market make the best on it. I won with a PHP powered marketing making bot with live code reloading.
- Built a cardboard box out of scrap electronics from adafruit that would let me adjust a market maker bots net position by tilting the box from side to side.
- Was either #2 or #3 to solve all the challenges
On the business side, if I recall correctly, most companies wanted to just shove the winners down their existing bad hiring pipeline process, and a lot of successful contestants who showed up already had great jobs, and so weren't actually looking.
patio11 once said that the code wasn't in an open-source ready state.
There were some folks who went so far as re-implementing/duplicating the backend trade simulator in other languages. Some of those were on github . . .
I don't think very many players did all the Starfighter trading challenges. patio11 or tptacek might be able to confirm, but I vaguely recall being one of like 100 or so to do all six levels of out of 10,000 that started. But I also had free time for playful coding and was nerding out on online challenges back then.
When Starfighter came out, I was energized and hopeful that hiring would go in this direction for programmers or that it would open doors to me as a job seeker that I couldn't every push through. No such luck.
From the hiring side, I have been on two hiring committees this year. Most applicants lacked any meaningful programming projects they could use to demonstrate skills. Something like completing a series of online challenges would be a difference maker.
That said, one candidate mentioned doing HackerRank challenges to learn Python and SQL, but totally flamed out on being able to even talk through a for loop in Python or come up with a simple select against a table in the in-person interview. Maybe nerves, but it also felt like the candidate's experience was 100% as just a warm body in various cubicle farms. It was very awkward. Several non-developers on the hiring committee felt like the candidate had actively lied about previous experience . . .
At least 30% bombed writing a for loop and if statement though. And we let them pick any language they wanted, so that wasn't it.
Our screening was pretty good, so most (but not all) candidates could demonstrate one loop, but an unfortunate number couldn't do the nested loop. shrug
I gave an open-book final exam that asked students to write a for loop on one of the questions. I would say around 20% bombed the question despite the very book sitting beside them having a chapter on looping. With examples.
After the class ended, I ultimately concluded a non-trivial number of students probably didn't do any of their own coding during the course. I'm pretty sure I have worked with at least two or three people in coding positions who couldn't actually code and relied on outside (spouse? out-sourced?) support.
This tends to be especially bad in companies that use a matrix organization where your department or career manager is in a totally separate hierarchy from the project-level managers you actually receive taskings from because the latter know your performance traits but the former decide where you get assigned to work and whether you can be fired or not, and they tend to have a huge incentive to staff critical programs, even if "staff" means fill in with net-negative warm bodies, because that's at least faster than filling with new hires, which also may not even be possible when you have very specific location and security requirements.
As far as I remember, at the beginning and for quite some time the Stockfighter trading challenges had reliability and performance issues (server capacity). So I decided to wait a little bit for the product to become more stable, and then get into the challenges. Unluckily, at that time Stockfighter was already decided to become shut down.
Do you remember any rough numbers on started vs finished on the two CTF tracks y'all put up?
FYI, the six stockfighter challenges were some of the most fun I ever had hacking around. I enjoy puzzle games like Exit and happily spend money on them. I would probably have paid to play the starfighter challenges even though the embedded one wasn't particularly my cup of tea. I wonder how many other nuts like me might have happily paid a monthly subscription for a few releases per year . . .
I also remember you saying in an early thread that clients' hiring processes were a mess and I think implied most treated Starfighter as just another candidate funnel. Did any of your clients take up a challenge/CTF process after working with Starfighter?
I know Thomas says hiring with challenges/CTFs is a competitive advantage but I'm not so sure. If it was true, wouldn't we see more of it out in the real world?
IMO, the messiness of hiring processes and the lack of adoption of work sample testing have a lot to do with each other. One of the questions I regularly ask in interviews (and have for a long time) is, "What are the first 5 things you'll have me work on?" During the last round of interviews, no one I talked to could answer that. I even had a couple of interviewers who didn't even know the name of the hiring manager or the position. Then you have other artificial filters like preferred certifications where the assumption seems to be that whatever training was required to pass can be mapped onto every organisation... or that simply possessing a cert magically grants insight into how most companies do stuff. The point is that companies don't really seem to have a good idea of what they're actually hiring for, but they get by hiring more senior people who have enough experience (hopefully) that they can figure it out on their own.
One of the things we discussed early on was avoiding having Starfighter turn into a certification process and maybe that's one of the reasons why we stumbled around a bit. I am aware that certifications have their place, but they're not really a substitute for industry experience and they definitely aren't a guarantee that a candidate can quickly figure out how a company's tech is glued together. IMO that's the power of work sample testing: you're giving a candidate a challenge where they can demonstrate aptitude. But if you don't know what actually needs to be done, you can't create that challenge, much less a rubric for describing expectations.
https://gist.github.com/DanielVF/d43e079050d1cf3c05d6
I’ll almost certainly buy it, though.
Which then puts any candidates in the position of, why should I spend time and effort on this little game if it doesn't lead to any advantage at all in actually getting good positions?
And so, the only people who messed with it were developers who had the free time to play with stuff like this and didn't particularly care that it wouldn't lead to any advantage in getting jobs.
Which means, it's a cool toy that was high-effort to build and probably to maintain, but had no viable path to earn money from anybody.
> Which then puts any candidates in the position of, why should I spend time and effort on this little game if it doesn't lead to any advantage at all in actually getting good positions?
This is the same position TripleByte was in. They advertised themselves to developers as "show us you can code", but they advertised themselves to companies as "we will find candidates who pass your existing hiring process". And when pressed on which of those things they were actually trying to do, they admitted it was the second one.
If TripleByte really succeeded with this, the customer companys could save quite some money by simplifying their hiring processes. The problem in my opinion rather is that most company's hiring processes have an official and a hidden agenda, and TripleByte only attempted to find candidates that might succeed in the official agenda.
No, that's what happens if TripleByte succeeds at their fake goal of finding candidates who are good candidates. That's the goal they claimed to have without actually having.
If all TripleByte does is find candidates who pass your process, you can't simplify your hiring process without affecting the quality of the candidates you hire through references from TripleByte.
If TripleByte is capable of finding candidates who would pass a specific, described process, you theoretically can omit the whole process because by definition, you know that the candidate is able to pass it. In practise, for good reasons, you don't want to do this, but it already save quite some money to shorten the process or just do plausibility checks.
If these plausibility checks make you realize that TripleByte was overselling (or to put it much more directly: are fraudsters), this should hopefully cause sufficient reputation damage for TripleByte that they go bust; thus TripleByte hopefully has a strong incentive not to oversell.
If you (as a hiring company) can describe your hiring process accurately enough that it's possible to specify it, you don't need recruiting at all. You can just take applications directly. What you're imagining is not related to the problems TripleByte faced.
If your goal is, as TripleByte's was, to find people who can pass a particular company's hiring process, the only way for you to know what that process is like is by experimenting with it empirically.
> you theoretically can omit the whole process because by definition, you know that the candidate is able to pass it.
Once again, you (aleph_minus_one) are describing the goal of "find candidates who are good at the job", not the goal of "find candidates who will pass the hiring process".
> If these plausibility checks make you realize that TripleByte was overselling (or to put it much more directly: are fraudsters)
No, that's just what it looks like to you (the company trying to fill spots). The problem people are attempting to address is that the company's hiring process is unrelated to what the company wants in a candidate. You are the fraudster; TripleByte in this scenario is the scapegoat for the bad job you do at candidate evaluation.
We started with this dream of completely revolutionizing the way people hire in tech (with hopes of expanding to other fields).
First of all, we learned that for 99% of companies they don't have a screening problem, but they have a sourcing problem. Our original company was along the lines of HackerRank / LeetCode / etc, and created due to the founder's experience as a manager at Amazon where he'd get 1000s of resumes that needed to be screened and the process was eating up too much dev resource. A product that solves this problem is appealing to companies that get 1000s of resumes per posting, but basically useless to most companies who get very few candidates, and even less appealing to recruiters who care much more about top of the funnel than the screening process.
After that we pivoted to an event format. The idea being that we could attract 100s of applicants to do a challenge in realtime. This reduces cheating risk and allowed us to act as a means of sourcing candidates (instead of just a screening platform). The idea was 100s of candidates would enter and the companies we got to sign up would compete for the top applicants. It sorta worked. The platform was good, companies were interested, and we got some people placed.
HOWEVER, the profile of people that will do a multi-hour open ended challenge for the chance to maybe get hired by some companies largely leans toward entry-level people. We had some experienced people that did it because they were bored or just curious, but generally seasoned engineers aren't going to waste their time, and forget about "passive job seekers" which is what all recruiters want most of all.
Ultimately, the company ended due to a lot of personal issues between the founding team, but the product itself was good, and the market fit was decent, but far from perfect. I'm pretty sure if we would've kept it together we would've eventually became very similar to HackerRank just offering a platform where people can conduct the same crappy tech interviews we've been using forever.
Wasn't this assumed to be the case going it to it when you discussed the business cases? It's hard to imagine any other result would be likely.
* Our go-to-market was contingency recruiting; we'd get paid for placements. Our CTF and HN calling cards easily got us added to the recruiting funnels of big tech companies (I'm assuming any of you could have done that too; I'm just saying, we never had to cold call). But it didn't change anything about how those companies qualified candidates, so to make that business successful we'd have had to do the same dialing-for-dollars matchmaking work any recruiter does, with comparable results.
* We took things very personally. You build relationships with your earliest users (candidates, in our case). Recruiting funnels at our clients had no such connection. The impedance mismatch was demoralizing; our lived experience was that every time a candidate we presented to a client got rejected, we had ourselves failed (I mean, really, we had!). Similarly, the idea that we were just another hoop that candidates had to hop through (but only the non-name-brand candidates who didn't know how to skip us to get an interview at a client) grated on us.
* A lot of second-system-syndrome on the implementation side; we'd done successful recruiting CTFs before, and set what was in retrospect a completely bizarre goal of outdoing ourselves at that job. That mistake is all me.
* Japan/Chicago remote founding team was very difficult, most especially for Patrick. There are teams that could handle that, but their members are more punctual about routine meetings than I am. Both the US and the Japan side felt checked out to each other. This didn't kill us; other things did, and we could have pushed through this problem. But it made things miserable for everyone. Again: this is all me.
* Erin & I were self-funding the company, which was expensive (just in terms of paying everybody's personal bills). Towards the end, it had become clear that we weren't going to drive 3 FTE worth of revenue on contingency placements in the next 12 months. We could have switched up the business model, driven user growth metrics, and gone out for seed funding, but none of us at that point were convicted enough of the business to stake our reputations on that. For all 3 of us, our time was worth too much. I think we made the right call.
It's been almost a decade since we did this, and those years haven't been especially kind to recruiting startups. There are knobs we could have turned to keep things chugging, and probably chugging comfortably, for many more years. But ultimately, I don't think there were any significant rewards to win by building a tech-company-specific recruiting firm. I think the company was ultimately fated to wind down sooner or later. If you're in that situation, sooner is always the better option! Life is short!
The underlying idea behind Starfighter, about the right way (or, at least, a right-er way) to hire people, has proven out repeatedly since then. We hired resume-blind and with work-sample challenges at Latacora, the next company me and Erin started, and we hire resume-blind and with work-sample challenges at Fly.io. Recruiting this way is a competitive advantage. The degree of advantage varies with the market; when the market is weak you can hire good people with terrible processes. But the advantage is never marginal, because bad hires are expensive, and there's so much hidden talent out there.
Got a short list of companies in addition to Fly.io that are using a hiring process you like or love?
> We took things very personally. You build relationships with your earliest users (candidates, in our case). Recruiting funnels at our clients had no such connection. The impedance mismatch was demoralizing; our lived experience was that every time a candidate we presented to a client got rejected, we had ourselves failed (I mean, really, we had!). Similarly, the idea that we were just another hoop that candidates had to hop through (but only the non-name-brand candidates who didn't know how to skip us to get an interview at a client) grated on us.
That sounds rough.
If you could do it over, do you see a way you could get more buy-in from some clients, such that they commit to "advanced placement" in their process, to candidates you provide? Or are pretty much all companies going to have their entire hiring process set in concrete?
What about interfacing at the hiring manager level, at companies where the hiring manager normally has some discretion to vouch for the candidate and bypass a lot of process? Could you get sufficient trust with those? Still the rejection problem?
If I was going to do it over again --- and, again, I wouldn't, because I feel like I also learned this just isn't a great business to be in --- I would have done something much closer to "Stack Exchange, but for CTFs". Run a couple of pure investment years, build up a base of users and some public profile, and then do lightweight recruiting stuff (probably just curated job ads for awhile) on top of that.
Turns out, finding/discovering hidden tech talent is relatively easy.
Finding the "Economic Buyer", a true hiring executive with budget and authority to make timely hiring decisions gets worse every year.
Bureaucratic Red Flag, when HR Bozos start 'adding value' to the process.
As companies grow, they mindlessly get in their own way.
It's very hard to shift companies that are eigher larger or with more internal history than a startup to try something like this.
Each level had an increasing level of what you needed to do, with the first level being to buy some stock (which you could do from the UI).
Most levels had you operating some kind of marketing making system, where you had to constantly update passive positions that made money as bots traded against them, and the prices shifted.
The last level required you to "hack" the system to get info you weren't supposed to have, in order to identify the insider trader in the middle of 50 other bots. It was quite different from all the other levels.
Here was the level solving code from my second run through:
https://gist.github.com/DanielVF/d43e079050d1cf3c05d6
Edit: patio11 called it Stockfighter, but others didn’t? Here’s the original launch post: https://news.ycombinator.com/item?id=10724592
https://en.wikipedia.org/wiki/The_Last_Starfighter
And it was only purchased because Lockheed paid millions in bribes to officials for which AFAIK none of them (nor the company) were ever punished.
So in terms of post-mortem I was thinking of something official like that :)
Ps its successor the F-16 was a lot better though and is still going strong today.
Wow, I thought you must be exaggerating, but it was literally hundreds. Some air forces lost nearly half the planes!
Added to that were inexperienced crews and well, disaster ensued :P They called it the lawn dart or widowmaker in Germany.
I think this is why the F-16 was so much more popular in Europe, it was actually designed as multirole and was and is good at it.
For anyone who didn't see the '80s light sf movie, The Last Starfighter, the premise is appealing and classic: (minor SPOILERS) young adult kid stuck living and working in a trailer park, wants more, there's a Starfighter video arcade game set up in the park, one day he goes for the record on the game, and turns out the game was a recruitment tool for something much bigger than his humble trailer park, and he passed, rises to subsequent challenges, and becomes a hero.
(Co-stars Robert Preston, as a very enterprising and scrappy recruiting headhunter, a bit like his The Music Man film character, but without the musical numbers.)
Just imagine how good that could've been though
https://en.wikipedia.org/wiki/The_Last_Starfighter
https://www.imdb.com/title/tt0087597/
"Greetings, Starfighter. You have been recruited by the Star League to defend the Frontier against Xur and the Ko-dan Armada."
(It seemed to change, after the dotcom boom, when the jobs turned into a lot more money that attracted a lot more people, and class barriers were erected to many of the better jobs. There's still a chance for upward mobility, despite artificial class barriers, but people will tend to burn their energy mimicking the shibboleths and practicing for the hazing rituals, rather than on things that would attract and teach them. Which brings us back to this application domain, of helping companies to hire people who would be great software developers.)