I assume this might be an unpopular opinion, but shouldn't these programs be evaluated on some metric of success rather than their racial selection?
If the software is accurate at picking choices that match the set goals, then it would seem that: the software is doing its job and is 'blameless', the racism is on the side of those that set the goals and/or the people claiming 'racism' because a person of a given skin colour was selected.
If it could be shown that there's low success in the software, yet it's relied on despite racial bias, then I would think few rational people could argue against its removal. (edit: s/for/against)
Edit: I suppose that I'm asking for better auditing of the results vs the desired outcomes. Race seems like a red-herring until it's shown that the metrics support poor performance of these systems.
How do you evaluate how successful an algorithm to decide if someone should get bail is?
The problem with most of these deep learning/AI systems is that they are more or less pattern detection systems. To an AI racial biases in its input data is just another pattern.
Doesn't the verdict of the trial related to the bail request have impact upon that? (legitimate question. I don't know if non-guilty parties are more/less likely to skip than guilty parties?)
That's assuming that we can agree that the justice system works impartially to begin with. I don't know of many people (including myself) that could agree with that.
Use a trial period where you grant bail to everyone (non violent offences only). You could run many different algorithms and models, recording all of their predictions. Since you granted everyone bail, you have a clear picture for which predictors accurately denied bail to the skippers.
> If the software is accurate at picking choices that match the set goals, then it would seem that: the software is doing its job and is 'blameless', the racism is on the side of those that set the goals and/or the people claiming 'racism' because a person of a given skin colour was selected.
Well, no.
It's not enough to simply decline to input racial data into the machine-learning system. The system may discover a proxy for race, and make decisions based on that proxy. Name and address, for instance, may be enough to give a good idea as to someone's race.
We expect human judges and parole-boards, for example, to discount racial information, even if there are demonstrable correlations associated with race. If the machine-learning system is not built with that discounting in mind, it isn't going to do so of its own accord, as it was of course built to detect and exploit correlations.
> If it could be shown that there's low success in the software, yet it's relied on despite racial bias, then I would think few rational people could argue for its removal.
I don't follow. If the software has been unsuccessful, that seems like a good reason to remove it. If, additionally, it has been making decisions on racial grounds, that seems like another good reason to remove it.
> Race seems like a red-herring until it's shown that the metrics support poor performance of these systems.
As I hope I've made clear above, this isn't right. It's not just about performance, it's also about ensuring the system does not make decisions on the basis of an individual's race.
I'm confused here, I read the article and it was somewhat muddy about the point trying to be made other than "machine bad!".
Racially biased could mean two things: despite not taking into account race the algorithm groups results racially at a statistically significant rate, or the algorithm is outright using race as a factor.
I assumed the former because it was not made clear in the article what was being talked about. Half the article is about things that may/may not use racial information (it's not said), and half the article is about things that do.
>> If it could be shown that there's low success in the software, yet it's relied on despite racial bias, then I would think few rational people could argue for its removal.
> I don't follow. If the software has been unsuccessful, that seems like a good reason to remove it. If, additionally, it has been making decisions on racial grounds, that seems like another good reason to remove it.
I mistyped there. I meant to say "few rational people could argue AGAINST its removal".
I had originally edited my sentence partially and mangled the meaning entirely.
A good example of this that I saw recently was AI used for underwriting health insurance. In order to make the algorithm "unbiased" in terms of race they removed race from the input dimensions. It then started heavily biasing along "frequency of doctor visits" which turns out to have a strong relationship with race. My takeaway is if equality of outcome is part of the goal that has to be part of your training as opposed to attempting to ignore the input, which seems obvious in hindsight.
If you're looking for equality of outcome, why go to the trouble of training a system in the first place? A pre-selected distribution applied arbitrarily is just as good.
Frequency of doctor visits seems a good indicator for determining how likely an insurance company will have to pay on a policy. If you want equal outcome, you end up setting a higher premium for everyone, and skip the underwriting altogether. The next part of the discussion would be subsidizing the higher premium for those that can't afford it. Just keep in mind that because there are fingers on the scale, fewer people can afford the "equalized" insurance than could afford the honestly evaluated coverage. (Prices are higher for everyone, so the pool of subscribers ends up being smaller if subscribers can opt out.)
It becomes a never-ending cycle of controls and impositions leading to collapse. Have the realistic underwriting, lower prices, and subsidize those (overall) lower premiums when needed.
By Goodhart's Law, there does not exist such a metric of success. All success metrics are arbitrary in this situation.
Software is written by people. Racist software comes from racist beliefs. A belief that a neural network will learn objectivity is a belief in objectivity, and both are wrong.
Race is not a red herring; even though race may be only a social construction with no objective science behind it, the fact that people believe in and act as if race were real is a problem. In particular it often violates the Civil Rights Acts.
Suppose racism, discrimination and bullying by the general public were the dominant cause of recidivism.. Would it make sense to allow people from the majority regularly do 3 month sentences and keep members of any minority group indefinitely on their first incident?
I think the true question is how you decide if an algorithm is actually biased or not. At a first shot I would suggest something like this: Divide some otherwise randomly sampled testing data into two sets, depending on the characteristic in question. If the ROC curve is approximately the same for both sets, I'd guess you could say the algorithm is not biased.
> If the software is accurate at picking choices that match the set goals
> auditing of the results vs the desired outcomes
The core problem is this assumption that the goal is to prioritize accuracy. When making "life-changing decisions ... in areas from job applications to immigration", many different goals need to be balanced. Who gets to define what "desired outcome" means?
We already have a hard time simply defining those goals. How would you even begin to define a cost function for e.g. "immigration" decisions when simply bringing up the topic is one of the easier ways to start a heated political argument?
> then it would seem that: the software is doing its job and is 'blameless'
The software is doing exactly what it was told to do: find a particular signal in some set of data (without explicit race. The problem is we spent several hundred years systemically embedding race into most aspects of society, which becomes a minefields of problematic local minima/maxima traps for your ML algorithm.
Let's say we have an good ML algorithm that provides some sort of evaluation about job applicants. Nothing related to race is included. The algorithm is used by a company in Chicago[1][3] or D.C.[2][3] hoping to hire the "best" people. Segregated populations means different schools/etc. Even if you didn't give the algorithm race data, most ML is practically guaranteed to find one of the many ways to infer race. Unintentionally, the ML algorithm ends up doing something similar to redlining[4], because that's what is in the data: the systemic effects of historical racism that still reverberate through society.
The issue that facial recognition is racially biased has only ever been an issue for non-industry leaders in facial recognition. And for the point, Amazon and Microsoft and Google have NEVER been industry leaders in FR - they just have enormous marketing budgets. Any police or any article you see where Amazon's FR is being used is an article about that police force being duped by marketing and expecting "best in class" from an organization that does not even rank in the industry as a serious player. The industry leaders in FR can easily be identified by going to the NIST Facial Recognition Vendor Test web site to view the annually ranked testing results from FR vendors who desire to work on Federal government contracts.
Also, the media treatment of FR is beyond pathetic. Pretty much every article I encounter is pulp crime level fiction. FWIW, I'm lead developer of one of the industry leading FR applications, typically within the first 4 on the NIST vendor rankings.
Notice how the article is very light on direct examples of actual bias - that's not a bug, it's a feature intentionally adopted to create fear, uncertainty and doubt (FUD). If anyone wants to FUD anything, just start with this article as a template, find-and-replace the subject from AI to insert_here, and change the name and quotes of the subject-matter-expert, and viola, you've got yourself a hit piece
We should be upvoting more substantive writing, making serious accusations on racism should be backed up by real evidence, not quotes and opinions of someone parroting the author's narrative
Every time this topic comes up, I'm always confused how the algorithms are biased. Which algorithms? Isn't it actually the labeled data they used to train that is biased by not including enough samples of non-white faces? What mechanisms are in place that prevent literally everyone from just re-training on better data? Why does my toy facial recognition software written in javascript detect every face I throw at it?
Many of these pieces smell phony. I'm certainly not saying there isn't institutional racism at work here, but I think we need way more detail to evaluate these claims.
This. These are articles which have a long reach being written by people who are operating far beyond their understanding. Surely the guardian can dig up a quality technical writer to dig into this?
21 comments
[ 1.8 ms ] story [ 61.1 ms ] threadIf the software is accurate at picking choices that match the set goals, then it would seem that: the software is doing its job and is 'blameless', the racism is on the side of those that set the goals and/or the people claiming 'racism' because a person of a given skin colour was selected.
If it could be shown that there's low success in the software, yet it's relied on despite racial bias, then I would think few rational people could argue against its removal. (edit: s/for/against)
Edit: I suppose that I'm asking for better auditing of the results vs the desired outcomes. Race seems like a red-herring until it's shown that the metrics support poor performance of these systems.
The problem with most of these deep learning/AI systems is that they are more or less pattern detection systems. To an AI racial biases in its input data is just another pattern.
Surely we have a baseline that can be used, such as when these systems are not in play?
That's assuming that we can agree that the justice system works impartially to begin with. I don't know of many people (including myself) that could agree with that.
Well, no.
It's not enough to simply decline to input racial data into the machine-learning system. The system may discover a proxy for race, and make decisions based on that proxy. Name and address, for instance, may be enough to give a good idea as to someone's race.
We expect human judges and parole-boards, for example, to discount racial information, even if there are demonstrable correlations associated with race. If the machine-learning system is not built with that discounting in mind, it isn't going to do so of its own accord, as it was of course built to detect and exploit correlations.
> If it could be shown that there's low success in the software, yet it's relied on despite racial bias, then I would think few rational people could argue for its removal.
I don't follow. If the software has been unsuccessful, that seems like a good reason to remove it. If, additionally, it has been making decisions on racial grounds, that seems like another good reason to remove it.
> Race seems like a red-herring until it's shown that the metrics support poor performance of these systems.
As I hope I've made clear above, this isn't right. It's not just about performance, it's also about ensuring the system does not make decisions on the basis of an individual's race.
You've been denied parol because someone with a name similar to you skipped bail is unfair regardless of race.
You've been denied parol because you come from a rough neighborhood is also unfair, regardless of race.
In fact, even if 99.9% of all people named X skipped bail, it would be unjust to deny somone bail because his name is X.
Racially biased could mean two things: despite not taking into account race the algorithm groups results racially at a statistically significant rate, or the algorithm is outright using race as a factor.
I assumed the former because it was not made clear in the article what was being talked about. Half the article is about things that may/may not use racial information (it's not said), and half the article is about things that do.
Are we talking about the same thing here?
> I don't follow. If the software has been unsuccessful, that seems like a good reason to remove it. If, additionally, it has been making decisions on racial grounds, that seems like another good reason to remove it.
I mistyped there. I meant to say "few rational people could argue AGAINST its removal".
I had originally edited my sentence partially and mangled the meaning entirely.
Frequency of doctor visits seems a good indicator for determining how likely an insurance company will have to pay on a policy. If you want equal outcome, you end up setting a higher premium for everyone, and skip the underwriting altogether. The next part of the discussion would be subsidizing the higher premium for those that can't afford it. Just keep in mind that because there are fingers on the scale, fewer people can afford the "equalized" insurance than could afford the honestly evaluated coverage. (Prices are higher for everyone, so the pool of subscribers ends up being smaller if subscribers can opt out.)
It becomes a never-ending cycle of controls and impositions leading to collapse. Have the realistic underwriting, lower prices, and subsidize those (overall) lower premiums when needed.
Software is written by people. Racist software comes from racist beliefs. A belief that a neural network will learn objectivity is a belief in objectivity, and both are wrong.
Race is not a red herring; even though race may be only a social construction with no objective science behind it, the fact that people believe in and act as if race were real is a problem. In particular it often violates the Civil Rights Acts.
> auditing of the results vs the desired outcomes
The core problem is this assumption that the goal is to prioritize accuracy. When making "life-changing decisions ... in areas from job applications to immigration", many different goals need to be balanced. Who gets to define what "desired outcome" means?
We already have a hard time simply defining those goals. How would you even begin to define a cost function for e.g. "immigration" decisions when simply bringing up the topic is one of the easier ways to start a heated political argument?
> then it would seem that: the software is doing its job and is 'blameless'
The software is doing exactly what it was told to do: find a particular signal in some set of data (without explicit race. The problem is we spent several hundred years systemically embedding race into most aspects of society, which becomes a minefields of problematic local minima/maxima traps for your ML algorithm.
Let's say we have an good ML algorithm that provides some sort of evaluation about job applicants. Nothing related to race is included. The algorithm is used by a company in Chicago[1][3] or D.C.[2][3] hoping to hire the "best" people. Segregated populations means different schools/etc. Even if you didn't give the algorithm race data, most ML is practically guaranteed to find one of the many ways to infer race. Unintentionally, the ML algorithm ends up doing something similar to redlining[4], because that's what is in the data: the systemic effects of historical racism that still reverberate through society.
[1] https://www.washingtonpost.com/graphics/2018/national/segreg...
[2] https://www.washingtonpost.com/graphics/2018/national/segreg...
[3] red-dots="White", blue-dots="Black", yellow-dots="Hispanic", etc. See the full article for many more examples: https://www.washingtonpost.com/graphics/2018/national/segreg...
[4] https://en.wikipedia.org/wiki/Redlining
We should be upvoting more substantive writing, making serious accusations on racism should be backed up by real evidence, not quotes and opinions of someone parroting the author's narrative
Many of these pieces smell phony. I'm certainly not saying there isn't institutional racism at work here, but I think we need way more detail to evaluate these claims.