This is something I've been working on exposing to AI labs through my startup LatentEvals[1], and found similar results in other industries from lending to insurance claims.
Happy to share some sample reports if anyone is interested!
The European Union passed The Artificial Intelligence Act, which classifies:
High-risk – AI applications that are expected to pose significant threats to health, safety, or the fundamental rights of persons. Notably, AI systems used in health, education, recruitment, critical infrastructure management, law enforcement or justice. They are subject to quality, transparency, human oversight and safety obligations
This is one of those things where the first sentence sounds completely fine and reasonable, maybe even objectively good.
Of all the things listed "recruitment" doesn't belong to me. Is the argument that it is someone's fundamental human right to get someone else to pay them to do a job? Or is it strictly about human oversight?
Some job application websites I've seen actually have a yes or no option to consent to AI review that they claim is to simply assist HR and not actually screen you. I always select no. There is no way that selecting yes would ever be in my interest. I'm sorry, I'm going to force a real human to look at my stuff if I still can.
Its fucking crazy that people are using these systems for important tasks like hiring. They have zero understanding about how these systems work. And LLMs are absolutely not designed to do those sorts of jobs, they're designed to be chatbots and to fool a human conversing them that they are responding intelligently. Of course they're gonna be useless at other tasks.
(I assume they're just using a big LLM for this, it doesnt say, it just says "AI" when they say "AI like that they usually mean LLM".. A custom trained hiring ML system would be better)
> To put this in perspective: If the AI had recommended Black and Asian candidates at the same rate as it recommended the most-favored group (typically white applicants)
Some people just can't help but put their biases on display at every opportunity, even when it comes to the most minute details.
Could the AI actually see the race of the applicants? Or was it just discriminating on the basis of some factor it found that was correlated with race, like SAT scores?
> To measure adverse impact, we apply the EEOC’s “four-fifths rule,” which flags a position when one group is recommended at less than 80% of the rate of the most-recommended group
That seems like a nonsensical way to measure racial discrimination. What could justify it?
> Using our large dataset of real hiring AI recommendations, we test our hypothesis. We find that people who submit multiple applications to positions screened by the same algorithmic hiring vendor are more likely to be rejected from every position to which they apply than would be true if the companies made decisions statistically independently from one another.
I would be surprised if the results were different.
Did I miss the part of the article where they break down how they determined race? Is the algorithm blind to race? It looks like they specifically looked at 83k people applying to ~100 companies which notably were Fortune 500 companies. Could there simply be candidate discrepancies here? Hard for me to follow the full methodology but it doesn't necessarily seem either malicious or that well structured. Don't you need to have a control group of applicants who are similar on paper? To allege DISCRIMINATION is quite bold.
Would be very interested to see how this affects post-50 workers. That's a protected class and I would imagine an ambulance chasing lawyer would be excited for a class action lawsuit.
You don’t need a complicated study to find out, do it yourself for science. Get a resume, make few different versions but keep the context the same, change the layout (one time education on top other on bottom etc etc), and use different names to signal different backgrounds, and you can extend it to schools too and gender, and send it to the same employers, you will see wonders!!
I tried it before, and discrimination is there, I would get one resume rejected quickly and few days later the same company would invite another resume for a screening call. I tried this before and after AI hype, results weren’t that different btw, and that was tested in US and Canada employers only.
I truly don't doubt it's possible for the AI to be 'racist'.
>If the AI had recommended Black and Asian candidates at the same rate as it recommended the most-favored group (typically white applicants), 40,000 more of their applications would have advanced to the next stage of hiring.
I don't think this is the right benchmark here, or at least, it would be very interesting if the actual outcome, offer or rejected, was considered at the end.
I’m sure (really sure) there are real problems with AI and bias, but this is a weird study that isn’t looking at resumes or anything, it’s looking at how candidates did in some weird psychometric tests.
We can't take blanket percentages as a reason for racial bias. Were they all equally qualified?
Too many of these studies only focus on percentages and the end result is unqualified candidates getting hired from minority groups at the expense of qualified ones.
"Cards held by African-American sellers sold for approximately 20% ($0.90) less than cards held by Caucasian sellers, and the race effect was more pronounced in sales of minority player cards."
Misleading title the paper [0] does not mention any CV screening that might suggest racial or gender bias. It is purely about assessment tool. No AI or LLMs.
I'm not saying AI is not biased, but this study does not prove that.
> Fig. 1. The pymetrics process.
> Stage 1: Applicants apply to positions.
> Stage 2: Applicants are directed to the pymetrics platform to play assessment games.
> Stage 3: pymetrics algorithms use applicant gameplay features to recommend 58.2% of applicants per position on average.
> Stage 4: Employers decide which applicants to interview or hire, typically rejecting applicants that were not recommended by pymetrics.
54 comments
[ 2.5 ms ] story [ 45.4 ms ] threadHappy to share some sample reports if anyone is interested!
1. https://www.latentevals.com/
High-risk – AI applications that are expected to pose significant threats to health, safety, or the fundamental rights of persons. Notably, AI systems used in health, education, recruitment, critical infrastructure management, law enforcement or justice. They are subject to quality, transparency, human oversight and safety obligations
That's a pretty common sense legislation to me.
Of all the things listed "recruitment" doesn't belong to me. Is the argument that it is someone's fundamental human right to get someone else to pay them to do a job? Or is it strictly about human oversight?
There's no reason to single out AI vs any other approach to the same topics.
(I assume they're just using a big LLM for this, it doesnt say, it just says "AI" when they say "AI like that they usually mean LLM".. A custom trained hiring ML system would be better)
Some people just can't help but put their biases on display at every opportunity, even when it comes to the most minute details.
That seems like a nonsensical way to measure racial discrimination. What could justify it?
I would be surprised if the results were different.
They find "disparate impact" of pymetrics across racial groups, but it doesn't seem like they controlled for anything.
Definitely open to opposing or critical views
> 30% of Black applicants apply to at least one position that demonstrates adverse impact against Black applicants.
The whole thing reads like a tautology.
I tried it before, and discrimination is there, I would get one resume rejected quickly and few days later the same company would invite another resume for a screening call. I tried this before and after AI hype, results weren’t that different btw, and that was tested in US and Canada employers only.
>If the AI had recommended Black and Asian candidates at the same rate as it recommended the most-favored group (typically white applicants), 40,000 more of their applications would have advanced to the next stage of hiring.
I don't think this is the right benchmark here, or at least, it would be very interesting if the actual outcome, offer or rejected, was considered at the end.
I guess this one just compounds.
[1] https://news.ycombinator.com/item?id=48620142
Too many of these studies only focus on percentages and the end result is unqualified candidates getting hired from minority groups at the expense of qualified ones.
"Cards held by African-American sellers sold for approximately 20% ($0.90) less than cards held by Caucasian sellers, and the race effect was more pronounced in sales of minority player cards."
I'm not saying AI is not biased, but this study does not prove that.
[0] https://arxiv.org/pdf/2605.27371
From the paper:
> Fig. 1. The pymetrics process. > Stage 1: Applicants apply to positions. > Stage 2: Applicants are directed to the pymetrics platform to play assessment games. > Stage 3: pymetrics algorithms use applicant gameplay features to recommend 58.2% of applicants per position on average. > Stage 4: Employers decide which applicants to interview or hire, typically rejecting applicants that were not recommended by pymetrics.