> The idea of AI picking up the biases within the language texts it trained on may not sound like an earth-shattering revelation
That's an understatement.
> But the study helps put the nail in the coffin of the old argument about AI automatically being more objective than humans
Again, this isn't AI, and anyone with knowledge on the subject has always known that a traditional machine learning algorithm is only as good as its training data.
This also seems like a case where the researchers are simply unhappy with the results they received, rather than being able to show that the results are wrong.
Everything you said is true, but I think it is dangerously missing the point.
Yes, everyone "in the know" knows that there's no such thing as "AI" right now, and what we actually have are just statistical models with "bias in, bias out". To us, this news is not surprising.
But that's not how these algorithms are being marketed, hyped, and sold, or how their decisions are being justified. Right now there's a lot of people selling "AI" as an unbiased and better decision-maker than humans. Where this gets really bad is when they start justifying the biased decisions of the machines as, "it's an AI program, so this can't be bias: whatever icky things it decided must be the truth!". That's the real worry here: when the marketing- and hence the policy- doesn't match the reality, and starts amplifying and reinforcing the very problems it was supposed to solve.
This is how I like to think about it: the term Artificial Intelligence is like artificial sweetener. It isn't sugar.
Machine Learning is artificial intelligence. When / if computers (or whatever they evolve in to) actually become intelligent we'll have to drop the the word artificial.
Artificial, adjective:
1.
made by human skill; produced by humans (opposed to natural ). eg. artificial flowers.
The idea of AI picking up the biases within the language
texts it trained on may not sound like an earth-
shattering revelation. But the study helps put the nail
in the coffin of the old argument about AI automatically
being more objective than humans
Was... anyone arguing that a model trained on a natural-language corpus would be entirely unbiased? What a magnificent strawman.
The word 'bias' implies that the belief is incorrect. If the information is correct, it shouldn't be called a bias. It is simply a conclusion.
For example: "It also tended to associate "woman" and "girl" with the arts rather than with mathematics."
This is a correct and valid conclusion. In all societies, women tend to engage in artistic activity more, while men engage in mathematical/systemic study more. (This is even more true in places which are more free like Scandinavia, than it is in less-free places like Iran. Iran has more women in tech studies.)
An AI learning this is a success, not a 'bias'. It doesn't mean no women should study these things; it's not a statement about what should be at all. It's simply an observation about the physical configuration of the world.
II
What these researchers are really discovering is that AI thinks without morals, and that this reveals the barriers that their own moral convictions and ideologies have placed in their minds.
An AI has no fear, so it's not afraid of reaching contrarian or politically-incorrect conclusions. It doesn't know social pressure, so it doesn't know to manipulate its impressions to follow the socially-acceptable beliefs. It doesn't know about the Overton window. It has no concept that its conclusions might lead to some undesirable outcome. It doesn't do motivated reasoning. It doesn't understand the concept of should. It simply describes the world (through the lens of the data available to it). What they've discovered is not that the AI is becoming biased, but that they are biased since they're not willing to reach morally-forbidden facts.
Their own bias appears because they've signed up to the reprehensible idea that the only reason people should be treated equally is because people are the same. Which is absurd. The correct morality here is: people are different and we should treat them equally anyway.
III
"To understand the possible implications, one only need look at the Pulitzer Prize finalist "Machine Bias" series by ProPublica that showed how a computer program designed to predict future criminals is biased against black people."
Of course it is. Black people are more likely to commit crimes. Therefore, like being male or being young, being black is a factor that one can apply predictively to an estimate of someone's likelihood to commit crimes. This is definitely true, it's just that people 'mindkill' themselves into not seeing it because most people are willing to blind their minds to fit into a socially-accepted morality and thus achieve personal benefit. What's the point in believing the truth if it doesn't benefit you?
Of course, being male or young are both just as inherent and unchangeable as being black. But nobody is going to complain when the machine realizes that youth and maleness predict criminality. We all know which facts are permissible and which facts are immoral, and thus forbidden.
Religion never went away, it just became non-theistic. If medieval Christians invented and AI that concluded there was no God, you can be sure they'd want to 'fix' its 'bias' too.
IV
Language lesson of the day!
Fact: A piece of knowledge which is morally acceptable.
Bias: A piece of knowledge which is morally unacceptable.
And here we have an example of one of the very few people saying anything actually worth listening to.
Expect to be hanged for this.
Edited to add: I shouldn't write comments like this as they aren't in the spirit of HN's guidelines of actually contributing to the conversation. So, what I mean to say is:
I appreciate comments like this and feel the parent comment hits the nail on the head.
The parent comment is, in my mind anyway, such a great conclusion to the whole topic that it become a conversation-killer. What else is there to be said? We all have a lot of things in common and some difference. If we don't end this quest to same-arise everyone then, eventually, only one of us will be necessary. And what would be the point of that.
"Black people are more likely to commit crimes. Therefore, like being male or being young, being black is a factor that one can apply predictively to an estimate of someone's likelihood to commit crimes. This is definitely true"
Actually this is definitely false.
You can use the statistic to estimate the probability that a black person selected at random from the population will have already committed a crime.
Given an individual black person who has not committed a crime, their being black does cause them to be more likely to commit a crime in future.
Blackness is not a predictor of criminality, even though black people are more likely to have been in environments that lead to criminality.
You simply don't understand causality and prediction. Your logic is literally the logic of racism - which is to treat people as if their race was causal.
>Blackness is not a predictor of criminality, even though black people are more likely to have been in environments that lead to criminality.
And your logic is that of nonsensical political correctness. You concede that these environments are more likely to lead to criminality, you conceded that black people are more likely to exist in these environments, yet you refuse to see that if A->B and B->C then A->C.
Let's instead consider, since you do allow that race is predictive of previous criminality, if the circumstances which have created increased chance for previous criminality have not changed, why is it invalid to presume that the same predictors are then valid for future probability? Do you also pretend that credit scores are not statistically accurate predictors of repayment potential?
These kinds of mental gymnastics are detrimental for all parties involved.
Given the existing biases is the systems involved, black people are more likely to be identified by the existing biased system as having committed a crime, because they are more likely to be investigated if suspected (or even without a reasonable basis I of suspicion) of having committed a crime, more likely to be prosecuted given the same degree of evidence, more likely to be convicted given the same degree of evidence, and (as a result of all that) more likely to have a criminal history which subjects them to additional law enforcement scrutiny and systematic bias on top of that more directly due to race.
The degree, if any, to which black people are more likely to actually commit crimes is difficult to tease out since all statistics on this area are affected, directly and indirectly, by these systematic biases.
(On top of that, there's the bias from the fact that things have been made crimes, or more sever crimes, specifically, in whole or in part, because black people were, at the time, more likely to do them.)
Why are people so adamant in denying the potential that blacks on average may in fact commit more crimes? Even considering that the system is biased, is it really so difficult to accept that there really IS more crime in poor, minority neighborhoods?
It is as if those who kowtow to political correctness don't understand that bias can exist in both directions, and when it comes to making decisions that affect society in general, both types of bias cause problems.
> Why are people so adamant in denying the potential that blacks on average may in fact commit more crimes?
If you read GP again, you'll note I never sent the potential at all. I just note that all the bases for the claim—made in the post it responds to—that that is a fact and not a potential rests on data affected by known sources of bias whose magnitude cannot be independently measured and corrected for.
Did you read the article on the study you're criticizing?
> The formula was particularly likely to falsely flag black defendants as future criminals, wrongly labeling them this way at almost twice the rate as white defendants.
The algorithm did not just say black people are more likely to arrested, it said black people were more likely to be arrested than the future data revealed. The algorithm was wrong, and it was wrong even if you take the demonstrably false position that being arrested is the same thing as committing a crime, ignoring the blatant and harmful biases in police behaviors already.
Given that this algorithm is being used to inform sentences by judges today, this is absolutely appalling.
The broader problem is that algorithms have imperfect signals to work with, and given those signals, it's easier to figure out someone's race than it is to figure out your attitude toward crime, so the algorithm overfits to the data it can see. This exacerbates the problem in a positive feedback loop, and we absolutely need to be cognizant of the damaging implications of this.
this NLP might be missing perceptions on parts of different groups of listeners. Different cultures may correlate language and race / gender differently
20 comments
[ 2.9 ms ] story [ 49.5 ms ] thread> The idea of AI picking up the biases within the language texts it trained on may not sound like an earth-shattering revelation
That's an understatement.
> But the study helps put the nail in the coffin of the old argument about AI automatically being more objective than humans
Again, this isn't AI, and anyone with knowledge on the subject has always known that a traditional machine learning algorithm is only as good as its training data.
This also seems like a case where the researchers are simply unhappy with the results they received, rather than being able to show that the results are wrong.
Yes, everyone "in the know" knows that there's no such thing as "AI" right now, and what we actually have are just statistical models with "bias in, bias out". To us, this news is not surprising.
But that's not how these algorithms are being marketed, hyped, and sold, or how their decisions are being justified. Right now there's a lot of people selling "AI" as an unbiased and better decision-maker than humans. Where this gets really bad is when they start justifying the biased decisions of the machines as, "it's an AI program, so this can't be bias: whatever icky things it decided must be the truth!". That's the real worry here: when the marketing- and hence the policy- doesn't match the reality, and starts amplifying and reinforcing the very problems it was supposed to solve.
Machine Learning is artificial intelligence. When / if computers (or whatever they evolve in to) actually become intelligent we'll have to drop the the word artificial.
Artificial, adjective:
1. made by human skill; produced by humans (opposed to natural ). eg. artificial flowers.
2. imitation; simulated; sham. eg. artificial vanilla flavouring.
3. lacking naturalness or spontaneity; forced; contrived; feigned. eg. an artificial smile.
For example: "It also tended to associate "woman" and "girl" with the arts rather than with mathematics."
This is a correct and valid conclusion. In all societies, women tend to engage in artistic activity more, while men engage in mathematical/systemic study more. (This is even more true in places which are more free like Scandinavia, than it is in less-free places like Iran. Iran has more women in tech studies.)
An AI learning this is a success, not a 'bias'. It doesn't mean no women should study these things; it's not a statement about what should be at all. It's simply an observation about the physical configuration of the world.
II
What these researchers are really discovering is that AI thinks without morals, and that this reveals the barriers that their own moral convictions and ideologies have placed in their minds.
An AI has no fear, so it's not afraid of reaching contrarian or politically-incorrect conclusions. It doesn't know social pressure, so it doesn't know to manipulate its impressions to follow the socially-acceptable beliefs. It doesn't know about the Overton window. It has no concept that its conclusions might lead to some undesirable outcome. It doesn't do motivated reasoning. It doesn't understand the concept of should. It simply describes the world (through the lens of the data available to it). What they've discovered is not that the AI is becoming biased, but that they are biased since they're not willing to reach morally-forbidden facts.
Their own bias appears because they've signed up to the reprehensible idea that the only reason people should be treated equally is because people are the same. Which is absurd. The correct morality here is: people are different and we should treat them equally anyway.
III
"To understand the possible implications, one only need look at the Pulitzer Prize finalist "Machine Bias" series by ProPublica that showed how a computer program designed to predict future criminals is biased against black people."
Of course it is. Black people are more likely to commit crimes. Therefore, like being male or being young, being black is a factor that one can apply predictively to an estimate of someone's likelihood to commit crimes. This is definitely true, it's just that people 'mindkill' themselves into not seeing it because most people are willing to blind their minds to fit into a socially-accepted morality and thus achieve personal benefit. What's the point in believing the truth if it doesn't benefit you?
Of course, being male or young are both just as inherent and unchangeable as being black. But nobody is going to complain when the machine realizes that youth and maleness predict criminality. We all know which facts are permissible and which facts are immoral, and thus forbidden.
Religion never went away, it just became non-theistic. If medieval Christians invented and AI that concluded there was no God, you can be sure they'd want to 'fix' its 'bias' too.
IV
Language lesson of the day!
Fact: A piece of knowledge which is morally acceptable. Bias: A piece of knowledge which is morally unacceptable.
Expect to be hanged for this.
Edited to add: I shouldn't write comments like this as they aren't in the spirit of HN's guidelines of actually contributing to the conversation. So, what I mean to say is:
I appreciate comments like this and feel the parent comment hits the nail on the head.
The parent comment is, in my mind anyway, such a great conclusion to the whole topic that it become a conversation-killer. What else is there to be said? We all have a lot of things in common and some difference. If we don't end this quest to same-arise everyone then, eventually, only one of us will be necessary. And what would be the point of that.
Actually this is definitely false.
You can use the statistic to estimate the probability that a black person selected at random from the population will have already committed a crime.
Given an individual black person who has not committed a crime, their being black does cause them to be more likely to commit a crime in future.
Blackness is not a predictor of criminality, even though black people are more likely to have been in environments that lead to criminality.
You simply don't understand causality and prediction. Your logic is literally the logic of racism - which is to treat people as if their race was causal.
And your logic is that of nonsensical political correctness. You concede that these environments are more likely to lead to criminality, you conceded that black people are more likely to exist in these environments, yet you refuse to see that if A->B and B->C then A->C.
Let's instead consider, since you do allow that race is predictive of previous criminality, if the circumstances which have created increased chance for previous criminality have not changed, why is it invalid to presume that the same predictors are then valid for future probability? Do you also pretend that credit scores are not statistically accurate predictors of repayment potential?
These kinds of mental gymnastics are detrimental for all parties involved.
Credit scores are individual and based on actual financial events in that persons hustory.
Blackness tells you nothing about an individual persons history.
This shows for certain that your understanding is faulty, and so are your conclusions.
Given the existing biases is the systems involved, black people are more likely to be identified by the existing biased system as having committed a crime, because they are more likely to be investigated if suspected (or even without a reasonable basis I of suspicion) of having committed a crime, more likely to be prosecuted given the same degree of evidence, more likely to be convicted given the same degree of evidence, and (as a result of all that) more likely to have a criminal history which subjects them to additional law enforcement scrutiny and systematic bias on top of that more directly due to race.
The degree, if any, to which black people are more likely to actually commit crimes is difficult to tease out since all statistics on this area are affected, directly and indirectly, by these systematic biases.
(On top of that, there's the bias from the fact that things have been made crimes, or more sever crimes, specifically, in whole or in part, because black people were, at the time, more likely to do them.)
It is as if those who kowtow to political correctness don't understand that bias can exist in both directions, and when it comes to making decisions that affect society in general, both types of bias cause problems.
If you read GP again, you'll note I never sent the potential at all. I just note that all the bases for the claim—made in the post it responds to—that that is a fact and not a potential rests on data affected by known sources of bias whose magnitude cannot be independently measured and corrected for.
> The formula was particularly likely to falsely flag black defendants as future criminals, wrongly labeling them this way at almost twice the rate as white defendants.
The algorithm did not just say black people are more likely to arrested, it said black people were more likely to be arrested than the future data revealed. The algorithm was wrong, and it was wrong even if you take the demonstrably false position that being arrested is the same thing as committing a crime, ignoring the blatant and harmful biases in police behaviors already.
Given that this algorithm is being used to inform sentences by judges today, this is absolutely appalling.
The broader problem is that algorithms have imperfect signals to work with, and given those signals, it's easier to figure out someone's race than it is to figure out your attitude toward crime, so the algorithm overfits to the data it can see. This exacerbates the problem in a positive feedback loop, and we absolutely need to be cognizant of the damaging implications of this.