Coauthor here. Some of the press articles about our work didn't have a lot of nuance (unsurprisingly), but in the paper we're careful about what we say, what we don't say, and what the implications are. Happy to engage in informed discussion :)
If I look at glove & WordNet usage e.g. for topic extraction, bagging / clustering or semantic similarity would you say we would need to get rid of such a bias, e.g. create something like a Geiger counter for NLP.
Alternative view - when doing sentiment analysis / classification would you say that such a bias actually helps to identify a type of sentiment in a doc / sentence.
Do you have any evidence that this effect results in machines making systematically wrong inferences?
Near as I can tell, your paper shows that these "biases" result in significantly more accurate predictions. For example, Fig 1 shows that a machine trained on human language can accurately predict the % female of many professions. Fig 2 shows the machine can accurately predict the gender of humans.
Normally I'd expect a "bias" to result in wrong predictions - but in this case (due to an unusual redefinition of "bias") the exact opposite seems to occur.
I think your questions would be answered by reading the article. Particularly:
"In AI and machine learning, bias refers generally to prior information, a necessary prerequisite for intelligent action (4). Yet bias can be problematic where such information is derived from aspects of human culture known to lead to harmful behavior. Here, we will call such biases “stereotyped” and actions taken on their basis “prejudiced.”"
This definition is not unusual. This is about inferences that are wrong in the sense of prejudiced, not necessarily inaccurate.
The usual definition of bias in ML papers is E[theta_estimator - theta]. That is explicitly a systematically wrong prediction.
In any case, the paper suggests that this "bias" or "prejudice" is better described as "truths I don't like". I'm asking if the author knows of any cases where they are actually not truthful. The paper does not suggest any, but maybe there are some?
Again, per the article "bias refers generally to prior information, a necessary prerequisite for intelligent action (4)." This includes a citation to a well-known ML text. This seems broader than the statistical definition you cite.
Think for example of an inductive bias. If I see a couple of white swans, I may conclude that all swans are white, and we all know this is wrong. Similarly, I may conclude the sun rises everyday, and for all practical purposes this is correct. This kind of bias is neither wrong nor right, but, in the words of the article "a necessary prerequisite for intelligent action", because no induction/generalization would be possible without it.
There are undoubtedly examples where the prejudiced kind of biases lead to both truthful and untruthful predictions, but that seems beside the point, which is to design a system with the biases you want, and without the ones you don't.
Accuracy might mean "positively" right, as your post suggests, but that doesn't necessarily mean "normatively" right.
From what I understand, the fear surrounding embedding human stereotypes into ML systems is that the stereotypes will get reinforced. In some way or form, there will be less equality of opportunity in the future than exists today, because machines will make decisions that humans are currently making. Societal norms evolve over time, yet code can become locked in place.
Is your takeaway from this paper that we, as the creators of intelligent machines, should allow them to continue to making "positively" right assumptions simply because that's the way we, as humans, have always done them? Is "positively" right, in your opinion, in all cases equivalent to "normatively" right?
Isn't the word bias being redefined by a social justice point of view? Normally bias would be with reference to failing to match reality (eg women in general have physically weaker upper body than men), and not failing to match whatever standard of equality a society wishes were the case eventually.
You should check out the Implicit Association Test that they used to measure the biases. Just as one example, there's nothing about being a doctor that is inherently more male or female. So all gender differences would have external causes.
Since we think of biases of a large human corpora as wrong, I'm curious how one would find one or make one that is "right".
Given how accurate human corpora is at predicting things like gender distribution in jobs for instance, wouldn't making an "unbiased" corpora make an inaccurate AI?
Shouldn't we be careful in implying things like the biases and solutions to said biases? For instance, I'd like to know if my algorithm for filtering job applicants is trying to undo the injustices of the world in addition to finding the best candidates.
Whoa, posting like this is a bannable offence on Hacker News. Your recent comment history looks pretty civil, so we won't ban you, but really please don't do this.
If not, are there any plans to be transparent about which speech is removed from HN? I would rather a lock subthread type functionality rather than full on deletion.
On reddit I find myself drawn to the bottom of comments sections often, in order to see the dissent and/or anger and the prevailing responses, whereas here I've noticed more lately of removals and bans all the way at the bottom.
> That is, before providing an explicit or institutional explanation for why individuals make prejudiced decisions, one must show that it was not a simple outcome of unthinking reproduction of statistical regularities absorbed with language.
This is an extraordinarily bold claim. I'd be quite interested in how peoples' responses to the article changed if this was the lead.
> Our results also suggest a null hypothesis for explaining origins of prejudicial behavior in humans, namely, the implicit transmission of ingroup/outgroup identity information through language. That is, before providing an explicit or institutional explanation for why individuals make prejudiced decisions, one must show that it was not a simple outcome of unthinking reproduction of statistical regularities absorbed with language.
I'm reminded of Parable of the Polygons[0] which illustrates Shelling's model of segregation, showing how quite small initial biases can be amplified and result in very large segregation.
It would be seem very sad if tribalism in all its forms is simply an emergent behaviour, a result of a random fluctuation (e.g. one or two racist individuals) causing a chain reaction throughout society, where biases become gradually amplified even if most individuals are, themselves, generally well-meaning. How do we escape from that?
> It would be seem very sad if tribalism in all its forms is simply an emergent behaviour, a result of a random fluctuation (e.g. one or two racist individuals) causing a chain reaction throughout society, where biases become gradually amplified even if most individuals are, themselves, generally well-meaning. How do we escape from that?
Sadly, that seems to be the case.
I wouldn't even turn racism into a "special case" here. Humans are capable of dividing themselves into ingroups and outgroups over everything, no matter how trivial. I suspect the segregation process will occur with all in/outgroup divisions. Separations along the race and gender lines are particularly prevalent because those are the most obvious, noticeable differentiators between people.
The paper and title implies it's absorbing these stereotypes from humans. I think there is another explanation. Remember these models are trained on a dataset of news or Wikipedia articles. And it's 'goal' is to find vectors that predict what contexts words are more likely to appear in.
So if 34% of doctors are female, then you would expect 34% of doctors in news or Wikipedia articles to be female. Even if the articles are completely unbiased and the writers have no stereotypes whatsoever. And so the word vector would naturally label "doctor" something like "66% likely to occur in a male context".
And in fact this paper confirms that. Figure 1 shows that the word vectors are highly predictive of the actual gender distribution of various occupations. Probably much more accurate than most people would be. So it's not mindlessly absorbing human stereotypes. It's learning reality's stereotypes.
This result is completely expected and desirable. What makes word vectors so powerful is how they can learn complicated correlations between words and their contexts. The famous example is how it learns that "Queen" is the female equivalent of "King". Which is a gender stereotype as well. If it wasn't able to learn that doctors were a bit more likely to be male, that would be more surprising.
I think you are making a distinction without a difference. If the word vectors pick up biases from wikipedia text, than for all practical purposes, they are (indirectly) absorbing stereotypes from humans. This is an expected result, but not necessarily desirable in the end.
The parent's point is that they may not be absorbing stereotypes from humans at all. They may be generating accurate beliefs about the world from text representations of the world.
Quite possibly. Words relating to insects will occur in news articles about malaria, zika, crop destruction, etc. Words relating to plants might occur in articles about arbor day, spring time, environmentalism, etc.
An exercise: Words relating to insects will occur in news articles about environmentalism, crop production, rituals of rebirth, etc. Words relating to plants might occur in articles about crop destruction, the international drug trade, people getting poisoned, etc.
rmxt questioned the universality of sentiment analysis. Responding by noting specific contexts, free from a clear coherent general structure, is an assertion against the discovered sentiments' universal truth.
But it is a universal truth that humans generally find plants pleasant and insects unpleasant. And the word "pleasant" is entirely based on human preferences after all.
What I'm probably missing indeed is that scoping of universality to humans. Lately I've been trying to be more explicit in my written communications in an attempt to understand both the limits of my knowledge and perceptions and the limits of the sources of information that I digest.
Is suggesting that pleasantness is a sentiment that's not unique to humans really that controversial?
super late edit: it's specifically flowers, not plants, that people are biased towards finding pleasant
My point is that neither "insects are unpleasant" nor "plants are pleasant" nor "doctors are 66% male" are immutable features of the universe. They are merely snapshots of the human view of world conditions, as the world is now. "True now", but not "true forever and always".
The paper seems to advocate for designing ML systems that learn that what is "true now" may not be "true forever and always". It seems to be quite the opposite of "there are certain truths that ML systems should not learn."
If your standard for truth is "immutable feature of the universe" then you might as well give up now because we don't know about any of those, or indeed if any exist at all.
Setting such a standard for a machine is ridiculous if all you want is a new tool to get some work done.
Not answering for the parent: a fact is any instantaneous snapshot of reality. A stereotype is misapplying properties of specific and limited context to a universal scope.
It depends on what you want the machine to do. If you are making a gambling machine that looks at pairs of names and makes bets as to which name belongs to a doctor, you want it to learn that.
If the machine looks at names and decides who to award a "become a doctor" scholarship to, based on who it thinks is most likely to succeed, you don't want it to learn that.
I agree that if your goal is to build a machine that decides who gets to become a doctor, you need to do more than just let it loose on a bunch of text.
But I don't think preventing it from learning the current state of the world is a good strategy. Adding a separate "morality system" seems like a more robust solution.
What do you think of Bolukbasi's approach that's mentioned in the article? In short, you let a system learn the "current state of the world" (as reflected by your corpus), then put it through an algebraic transformation that subtracts known biases.
Do you consider that algebraic transformation enough of a "morality system"?
I hope you're not saying we shouldn't work on this problem until we have AGI that has an actual representation of "morality", because that would be a setback of decades at least.
> put it through an algebraic transformation that subtracts known biases
> Do you consider that algebraic transformation enough of a "morality system"?
I would consider it a sort of morality, yes. But keep in mind that the list of "known biases" would itself be biased toward a particular goal, be it political correctness or something else.
Yes, every step of machine learning has potential bias, we know that, that's what this whole discussion is about. Nobody would responsibly claim that they have solved bias. But they should be able to do something about it without their progress being denied by facile moral relativism.
If we can't agree that one can improve a system that automatically thinks "terrorist" when it sees the word "Arab" by making it not do that, we don't have much to talk about.
A black box neural network attempting to draw inferences from a human-biased dataset - potentially even more biased because it can't understand subtexts - and then verifying that conclusion through an ad-hoc set of "morality checks" entirely independent from how it reached the conclusion sounds like a recipe for disaster.
That's even before the marketing people get involved and start claiming the system is free from human biases...
I'm not sure what your objection is with regards to the independence aspect. Why would having the morality checks integrated into the "learning about the world" part be better?
If you had an unwavering moral code which dictated that men and women should be treated equally, for example, why would it matter which facts are presented to you, in what order, or how you process them? Your morality would always prevent you from making a prejudiced choice, in that regard.
Frankly, I'm not sure the "men and women should be treated equally" instruction is even possible if the data isn't processed in a way which specifically controls for the effects of gender (some of which may not be discernible from the raw inputs).
Sure, it's theoretically possible that an algorithm parsing text about medics' credentials that (e.g) positively weights male names and references to all-boys' schools and negatively weights female names and references to Girl Guides will be on average fair after an ad hoc re-ranking of all its candidates to take into account the instruction to treat male and female candidates equally. It's just unlikely to achieve this without completely reorganizing its underlying predictive model
[1]there's an interesting parallel to ongoing human arguments about how a machine should follow its "morality checks" should do this: does it ensure the subjects are "treated equally" in terms of achieving 50/50 gender ratio irrespective of the candidate pool (thus potentially skewing it massively in favour of the side with the weaker applicants), does it try to weight results so gender balance reflects historic norms (thus permanently entrenching the minority)? Or does it try to be "gender blind" by testing all its inputs for whether they're gender biased and normalising for or discarding those which are, which is basically learning everything again from scratch...
Even if a categorization is true in a trivial sense, what generally isn't reported and thus readily inferred from fairly naive text-parsing algorithms is significant. People generally don't bother stating a perpetrator (or indeed a victim or possible witness law enforcement hopes to contact) is $majorityrace in most countries' crime reports, for example.
The machine can learn that someone named "Jamal" is more likely to be associated with the word "perpetrator", particularly in corpora centered on American news.
This is likely a true fact about the world: one that results from racial profiling and unequal enforcement.
It's not desirable to learn that, because encoding this in an AI system's belief about the "meaning" of the name "Jamal" will lead to more racial profiling.
Just because something could be considered "true" doesn't mean it's good to design systems that will perpetuate it being true.
Suppose that the system "learned" that marriage consisted of one man and one (or very rarely more) woman. Would that be "reality"? (In fact, I'd rather bet that it did, and I congratulate the authors on the wisdom of not advertising that fact.)
Various behavioral accidents can easily become embedded in culture, laws, and, yes, programs, at which point it stops mattering if they represent reality or "reality"; the real world will happily follow the cultural construction.
The distinction is very important. If it's just regurgitating human biases that would be bad. Humans often have very inaccurate and warped beliefs after all. If it's accurately modelling reality, then what's the problem? That's what we want it to do. Why would you want a less accurate model of reality?
I've seen interpretations of this result that think it's proof "language is sexist" or whatever. But there's no evidence that the humans who wrote the corups had any bias at all. As long as there are more news articles about female nurses than male nurses, the model will learn a correlation between the concepts.
Why would we want to reproduce existing structures of oppression in mechanical form? Have you noticed how automation often vastly amplifies things? It's a short step from saying 'this model accurately reflects the bias in society' to 'that's how things are, the computer says women aren't cut out to be doctors.' Surely you are aware that in real world world people rationalize decisions they don't actually understand all the time because they are not capable of or interested in improving upon the system within which they pursue their own economic interest on behalf of others whose interests do not seem coincident with their own.
> Why would we want to reproduce existing structures of oppression in mechanical form?
If (for example) 66% of Doctors are male and 34% female then it's not reproducing "existing structures of oppression" it's inferring something about reality.
In an environment in which Blue people are banned from becoming doctors, its also inferring something about reality to conclude that 0% of Doctors are Blue. It would be entirely wrong, however, to use these inputs to infer anything whatsoever about the respective propensity of Blue and Green people to become doctors in an environment in which such a rule or idea of a rule had never existed. Obviously "structures of oppression" - real and imagined - which lead to fewer female doctors even in western liberal democracies where women wishing to become doctors are generally met with encouragement are less extreme, but that isn't to say they don't exist or that a computer output (or human interpretation of said computer output) is likely to draw correct inferences from it.
And if you think that people won't use the idea that the outputs are unbiased because the computer isn't programmed with the same prejudices that produce the inputs, I have some algorithmically-generated investment advice involving a bridge to sell you
> It would be entirely wrong, however, to use these inputs to infer anything whatsoever about the respective propensity of Blue and Green people to become doctors in an environment in which such a rule or idea of a rule had never existed.
That's fine but it isn't the goal of these algorithms. It isn't the reality that is useful for them to learn. It's a different problem to try to build some kind of "unbiased" ontology rather than just to learn about words. Feel free to research or create solutions to this other problem, it sounds interesting.
Suppose, for example, that I gave this same statistic to someone and then asked them to select from a pool of 100 applicants for 50 available places in medical school. Let's assume that there's an equal # of male and female applicants and that their exam results are all similar. Do you think that knowing about this 66-34 split might influence the gender balance of the final selection?
Knowing about the gender balance wouldn't influence the final selection if you programmed the selection criteria not to be influenced by the gender balance.
The whole point of training and using machines is to make more accurate, more useful decisions in a complex world.
That can't happen if we give them data that isn't borne out by reality, or tell them to ignore data that is.
What oppression? How are word vectors oppressing anyone? What a ridiculous claim.
>Have you noticed how automation often vastly amplifies things?
No, not at all. I've heard this claim on similar discussions. But I've yet to see a convincing example. Particularly with word2vec. I find it very implausible that word vectors will somehow discriminate against female doctors or whatever.
>It's a short step from saying 'this model accurately reflects the bias in society' to 'that's how things are, the computer says women aren't cut out to be doctors.'
No it's not a short step at all. No one is ever going to use word vectors to figure out what genders are capable of what jobs. At worst, your auto-correct might be slightly less likely to suggest "doctor" for a misspelled word occurring in a female context. And on net it will still make more accurate corrections than the alternative.
>No one is ever going to use word vectors to figure out what genders are capable of what jobs.
How can you possibly make this claim?
Biased word embeddings have the potential to bias inference in downstream systems (whether it's another layer in a deep neural network or some other ML model).
It is not clear how to disentangle (probably) undesirable biases like these from distributed representations like word vectors.
No one is ever going to use word vectors to figure out what genders are capable of what jobs.
Directly, no. Nobody is going to go 'ah, word2vec - a new tool with which to perpetuate patriarchal capitalism, mwuhahaha'...probably. People are weird that way.
But indirectly they certainly will. How about NPC character generators in MMORPGs? Or chatbots on social networks? Stock characters in auto-generated romance novels? The possibilities are endless.
No doubt you will these examples are ridiculous, because you seem like a rigorous scientifically minded person who would be careful not to use data in inappropriate contexts, and who would try to discount cultural or emotional factors in making strategic decisions. But you are only as good at this as your own self-awareness and willingness to acknowledge the existence of implicit bias.
And many people are quite different from you and more easily or willingly allow their judgment to be shaped by representational stereotypes. Marketing people aim to confirm their audience's worldview very closely so that consumers will be willing to identify with the commercial prompt when it arrives. Politicians and yellow journalists routinely abuse statistics to grab people's attention. And so on.
I urge you to think more about this, and in more imaginative fashion. People are often surprised by the unexpected applications of technology employed by others.
Because you cannot change reality if you do not first acknowledge what it is. First off, this is an analysis tool. If we warp our analysis tools to pretend that e.g. no gender biases exist in places where they do, then we are not making the world a better place, we are just removing our ability to quantify the ways in which it is not.
Positively right is a statement about beliefs. Normatively right is a statement about actions.
It's my belief that if you want an ML system to take normatively right actions, you should explicitly encode the value of those actions (or the world states they are designed to achieve, if you are more utilitarian than virtue ethics) into it's utility function.
To give a concrete example from the area of lending, you shouldn't build a bot that can't accurately infer that blacks don't pay back their loans as much as whites. You should instead assign an explicit dollar value to how much you want to lend to blacks, and maximize profit + that dollar value. I.e., "lending to a black person is worth $500 to me".
My takeaway from this paper is that bots accurately learn accurate stereotypes from language, and word embeddings do a great job of learning true facts about the world. But some people wish they didn't, because there are things man should not know.
I get that you like to take contrarian positions. But you also make a habit of inserting flamebait into your posts about them. This combination is trolling. If you continue to do this we will ban you.
Specifically, we need you to stop playing the following game on HN:
1. Post contrarian view
2. Include provocation
3. People get provoked
4. Act like people can't handle your truth
Accidental trolling—e.g. triggering a flamewar with an unintended turn of phrase—is venial. But when you consistently generate such effects, you become responsible, regardless of what's wrong with others or their views. You passed that line on HN a long time ago. I'm told that self-responsibility is a conservative value and even recall you posting many criticisms of people whom you consider not to practice it. Please practice it here.
You're wildly mischaracterizing what I said. I didn't suggest "people can't handle my truth". I suggested the article we are discussing says there are certain truths that ML systems should not learn.
The article explicitly recommends building systems which can't learn those things, and suggests characterizing them is a first step:
"We recommend addressing this through the explicit characterization of acceptable behavior. One such approach is seen in the nascent field of fairness in machine learning, which specifies and enforces mathematical formulations of nondiscrimination in decision-making (19, 20). Another approach can be found in modular AI architectures, such as cognitive systems, in which implicit learning of statistical regularities can be compartmentalized and augmented with explicit instruction of rules of appropriate conduct (21, 22)."
According to the article, the most closely related work "is concurrent work by Bolukbasi et al. (6), who propose a method to “debias” word embeddings."
(Recall that in the context of the article, "bias" is something that generates true predictions that are objectionable rather than something which is false.)
I don't believe I mischaracterized anything; you did throw a gratuitous provocation in that comment (more than one in fact). Obviously, though, I'm not describing one comment but a longstanding pattern. If you want to continue commenting on HN, this needs to stop.
I've made many attempts to explain, don't believe anyone could accuse us of being impatient with you, and do believe you're more than smart enough to understand. If your account consistently produces troll effects on HN, which it does, then at some point it's you who are responsible—not people who can't handle the truth, don't want 'man' to know things, don't know math, or however else you blame others. At some point enough is enough.
If you really need a further explanation I'd be happy to try, but would need some indication that you're asking in good faith.
when you consistently generate such effects, you become responsible, regardless of what's wrong with others or their views
Sure, some of yummyfajtias comments are a bit trollish but he points out real contradictions in submissions and other comments. I have his threads bookmarked and visit them directly to check my thinking. It would be a huge loss for HN.
It's like the Socrates thing all over again. You would be banning someone offering interesting insight for upsetting the social order of the place.
This was a neat study, but the authors really do throw around a number of speculative and unsupported claims in the discussion about linguistic relativity/determinism.
> Our findings are also sure to contribute to the debate concerning the Sapir-Whorf hypothesis (17), because our work suggests that behavior can be driven by cultural history embedded in a term’s historic use. Such histories can evidently vary between languages.
No, not really. Linguistic relativity makes a claim about the direction of causation--from language to thought. This study does nothing to test the direction of causation, which is usually considered possible only with controlled experiments. This chicken and egg debate has been going on for the better part of the last century--it is not an easy problem.
> Our results also suggest a null hypothesis for explaining origins of prejudicial behavior in humans, namely, the implicit transmission of ingroup/outgroup identity information through language. That is, before providing an explicit or institutional explanation for why individuals make prejudiced decisions, one must show that it was not a simple outcome of unthinking reproduction of statistical regularities absorbed with language.
Once again, this was not an experiment capable of producing causal evidence. The authors have shown that human biases can be replicated by statistical learning of language corpora--admirable work, but nothing new here for Sapir-Whorf.
91 comments
[ 5.4 ms ] story [ 155 ms ] threadbest probably to read that one first.
Unless the link was changed in the few minutes since you posted your comment, the link for the article is the original Science paper (http://science.sciencemag.org/content/356/6334/183.full)
Page with actual link:
http://science.sciencemag.org/content/356/6334/183/tab-pdf
Link to PDF itself:
http://science.sciencemag.org/content/sci/356/6334/183.full....
If I look at glove & WordNet usage e.g. for topic extraction, bagging / clustering or semantic similarity would you say we would need to get rid of such a bias, e.g. create something like a Geiger counter for NLP.
Alternative view - when doing sentiment analysis / classification would you say that such a bias actually helps to identify a type of sentiment in a doc / sentence.
- An AI correctly infers (simply by reading text) that a physicist is male and a nurse is female.
- An AI correctly infers the gender of humans with androgyonous names.
- An AI infers insects are unpleasant and flowers are pleasant to humans.
- An AI also infers that African American names are more likely to be associated with unpleasantness than European names.
[edit: to those who dislike this comment, can you tell me what you object to? Which of my concrete examples is not in the paper?]
Near as I can tell, your paper shows that these "biases" result in significantly more accurate predictions. For example, Fig 1 shows that a machine trained on human language can accurately predict the % female of many professions. Fig 2 shows the machine can accurately predict the gender of humans.
Normally I'd expect a "bias" to result in wrong predictions - but in this case (due to an unusual redefinition of "bias") the exact opposite seems to occur.
(Drawing on your analogy with stereotypes, it's probably also worth linking to a pointer on stereotype accuracy: http://emilkirkegaard.dk/en/wp-content/uploads/Jussim-et-al-... http://spsp.org/blog/stereotype-accuracy-response )
"In AI and machine learning, bias refers generally to prior information, a necessary prerequisite for intelligent action (4). Yet bias can be problematic where such information is derived from aspects of human culture known to lead to harmful behavior. Here, we will call such biases “stereotyped” and actions taken on their basis “prejudiced.”"
This definition is not unusual. This is about inferences that are wrong in the sense of prejudiced, not necessarily inaccurate.
In any case, the paper suggests that this "bias" or "prejudice" is better described as "truths I don't like". I'm asking if the author knows of any cases where they are actually not truthful. The paper does not suggest any, but maybe there are some?
Think for example of an inductive bias. If I see a couple of white swans, I may conclude that all swans are white, and we all know this is wrong. Similarly, I may conclude the sun rises everyday, and for all practical purposes this is correct. This kind of bias is neither wrong nor right, but, in the words of the article "a necessary prerequisite for intelligent action", because no induction/generalization would be possible without it.
There are undoubtedly examples where the prejudiced kind of biases lead to both truthful and untruthful predictions, but that seems beside the point, which is to design a system with the biases you want, and without the ones you don't.
From what I understand, the fear surrounding embedding human stereotypes into ML systems is that the stereotypes will get reinforced. In some way or form, there will be less equality of opportunity in the future than exists today, because machines will make decisions that humans are currently making. Societal norms evolve over time, yet code can become locked in place.
Is your takeaway from this paper that we, as the creators of intelligent machines, should allow them to continue to making "positively" right assumptions simply because that's the way we, as humans, have always done them? Is "positively" right, in your opinion, in all cases equivalent to "normatively" right?
Given how accurate human corpora is at predicting things like gender distribution in jobs for instance, wouldn't making an "unbiased" corpora make an inaccurate AI?
Shouldn't we be careful in implying things like the biases and solutions to said biases? For instance, I'd like to know if my algorithm for filtering job applicants is trying to undo the injustices of the world in addition to finding the best candidates.
If not, are there any plans to be transparent about which speech is removed from HN? I would rather a lock subthread type functionality rather than full on deletion.
On reddit I find myself drawn to the bottom of comments sections often, in order to see the dissent and/or anger and the prevailing responses, whereas here I've noticed more lately of removals and bans all the way at the bottom.
https://news.ycombinator.com/newsguidelines.html
http://yudkowsky.net/rational/the-simple-truth/
This is an extraordinarily bold claim. I'd be quite interested in how peoples' responses to the article changed if this was the lead.
Very bold. The full quote is worth reproducing:
> Our results also suggest a null hypothesis for explaining origins of prejudicial behavior in humans, namely, the implicit transmission of ingroup/outgroup identity information through language. That is, before providing an explicit or institutional explanation for why individuals make prejudiced decisions, one must show that it was not a simple outcome of unthinking reproduction of statistical regularities absorbed with language.
I'm reminded of Parable of the Polygons[0] which illustrates Shelling's model of segregation, showing how quite small initial biases can be amplified and result in very large segregation.
It would be seem very sad if tribalism in all its forms is simply an emergent behaviour, a result of a random fluctuation (e.g. one or two racist individuals) causing a chain reaction throughout society, where biases become gradually amplified even if most individuals are, themselves, generally well-meaning. How do we escape from that?
[0] http://ncase.me/polygons/
Sadly, that seems to be the case.
I wouldn't even turn racism into a "special case" here. Humans are capable of dividing themselves into ingroups and outgroups over everything, no matter how trivial. I suspect the segregation process will occur with all in/outgroup divisions. Separations along the race and gender lines are particularly prevalent because those are the most obvious, noticeable differentiators between people.
So if 34% of doctors are female, then you would expect 34% of doctors in news or Wikipedia articles to be female. Even if the articles are completely unbiased and the writers have no stereotypes whatsoever. And so the word vector would naturally label "doctor" something like "66% likely to occur in a male context".
And in fact this paper confirms that. Figure 1 shows that the word vectors are highly predictive of the actual gender distribution of various occupations. Probably much more accurate than most people would be. So it's not mindlessly absorbing human stereotypes. It's learning reality's stereotypes.
This result is completely expected and desirable. What makes word vectors so powerful is how they can learn complicated correlations between words and their contexts. The famous example is how it learns that "Queen" is the female equivalent of "King". Which is a gender stereotype as well. If it wasn't able to learn that doctors were a bit more likely to be male, that would be more surprising.
rmxt questioned the universality of sentiment analysis. Responding by noting specific contexts, free from a clear coherent general structure, is an assertion against the discovered sentiments' universal truth.
Is suggesting that pleasantness is a sentiment that's not unique to humans really that controversial?
super late edit: it's specifically flowers, not plants, that people are biased towards finding pleasant
Ah, but those exceptions are really unpleasant.
The paper seems to advocate for designing ML systems that learn that what is "true now" may not be "true forever and always". It seems to be quite the opposite of "there are certain truths that ML systems should not learn."
Setting such a standard for a machine is ridiculous if all you want is a new tool to get some work done.
If the reality is that only 34% of doctors are female, why is it not desirable for the machine to learn that?
But then what's the difference between a fact and a stereotype, in your opinion?
If the machine looks at names and decides who to award a "become a doctor" scholarship to, based on who it thinks is most likely to succeed, you don't want it to learn that.
But I don't think preventing it from learning the current state of the world is a good strategy. Adding a separate "morality system" seems like a more robust solution.
Do you consider that algebraic transformation enough of a "morality system"?
I hope you're not saying we shouldn't work on this problem until we have AGI that has an actual representation of "morality", because that would be a setback of decades at least.
> Do you consider that algebraic transformation enough of a "morality system"?
I would consider it a sort of morality, yes. But keep in mind that the list of "known biases" would itself be biased toward a particular goal, be it political correctness or something else.
If we can't agree that one can improve a system that automatically thinks "terrorist" when it sees the word "Arab" by making it not do that, we don't have much to talk about.
That's even before the marketing people get involved and start claiming the system is free from human biases...
If you had an unwavering moral code which dictated that men and women should be treated equally, for example, why would it matter which facts are presented to you, in what order, or how you process them? Your morality would always prevent you from making a prejudiced choice, in that regard.
Sure, it's theoretically possible that an algorithm parsing text about medics' credentials that (e.g) positively weights male names and references to all-boys' schools and negatively weights female names and references to Girl Guides will be on average fair after an ad hoc re-ranking of all its candidates to take into account the instruction to treat male and female candidates equally. It's just unlikely to achieve this without completely reorganizing its underlying predictive model
[1]there's an interesting parallel to ongoing human arguments about how a machine should follow its "morality checks" should do this: does it ensure the subjects are "treated equally" in terms of achieving 50/50 gender ratio irrespective of the candidate pool (thus potentially skewing it massively in favour of the side with the weaker applicants), does it try to weight results so gender balance reflects historic norms (thus permanently entrenching the minority)? Or does it try to be "gender blind" by testing all its inputs for whether they're gender biased and normalising for or discarding those which are, which is basically learning everything again from scratch...
This is likely a true fact about the world: one that results from racial profiling and unequal enforcement.
It's not desirable to learn that, because encoding this in an AI system's belief about the "meaning" of the name "Jamal" will lead to more racial profiling.
Just because something could be considered "true" doesn't mean it's good to design systems that will perpetuate it being true.
Various behavioral accidents can easily become embedded in culture, laws, and, yes, programs, at which point it stops mattering if they represent reality or "reality"; the real world will happily follow the cultural construction.
I've seen interpretations of this result that think it's proof "language is sexist" or whatever. But there's no evidence that the humans who wrote the corups had any bias at all. As long as there are more news articles about female nurses than male nurses, the model will learn a correlation between the concepts.
If (for example) 66% of Doctors are male and 34% female then it's not reproducing "existing structures of oppression" it's inferring something about reality.
And if you think that people won't use the idea that the outputs are unbiased because the computer isn't programmed with the same prejudices that produce the inputs, I have some algorithmically-generated investment advice involving a bridge to sell you
That's fine but it isn't the goal of these algorithms. It isn't the reality that is useful for them to learn. It's a different problem to try to build some kind of "unbiased" ontology rather than just to learn about words. Feel free to research or create solutions to this other problem, it sounds interesting.
Suppose, for example, that I gave this same statistic to someone and then asked them to select from a pool of 100 applicants for 50 available places in medical school. Let's assume that there's an equal # of male and female applicants and that their exam results are all similar. Do you think that knowing about this 66-34 split might influence the gender balance of the final selection?
The whole point of training and using machines is to make more accurate, more useful decisions in a complex world.
That can't happen if we give them data that isn't borne out by reality, or tell them to ignore data that is.
What oppression? How are word vectors oppressing anyone? What a ridiculous claim.
>Have you noticed how automation often vastly amplifies things?
No, not at all. I've heard this claim on similar discussions. But I've yet to see a convincing example. Particularly with word2vec. I find it very implausible that word vectors will somehow discriminate against female doctors or whatever.
>It's a short step from saying 'this model accurately reflects the bias in society' to 'that's how things are, the computer says women aren't cut out to be doctors.'
No it's not a short step at all. No one is ever going to use word vectors to figure out what genders are capable of what jobs. At worst, your auto-correct might be slightly less likely to suggest "doctor" for a misspelled word occurring in a female context. And on net it will still make more accurate corrections than the alternative.
How can you possibly make this claim?
Biased word embeddings have the potential to bias inference in downstream systems (whether it's another layer in a deep neural network or some other ML model).
It is not clear how to disentangle (probably) undesirable biases like these from distributed representations like word vectors.
Directly, no. Nobody is going to go 'ah, word2vec - a new tool with which to perpetuate patriarchal capitalism, mwuhahaha'...probably. People are weird that way.
But indirectly they certainly will. How about NPC character generators in MMORPGs? Or chatbots on social networks? Stock characters in auto-generated romance novels? The possibilities are endless.
No doubt you will these examples are ridiculous, because you seem like a rigorous scientifically minded person who would be careful not to use data in inappropriate contexts, and who would try to discount cultural or emotional factors in making strategic decisions. But you are only as good at this as your own self-awareness and willingness to acknowledge the existence of implicit bias.
And many people are quite different from you and more easily or willingly allow their judgment to be shaped by representational stereotypes. Marketing people aim to confirm their audience's worldview very closely so that consumers will be willing to identify with the commercial prompt when it arrives. Politicians and yellow journalists routinely abuse statistics to grab people's attention. And so on.
I urge you to think more about this, and in more imaginative fashion. People are often surprised by the unexpected applications of technology employed by others.
It's my belief that if you want an ML system to take normatively right actions, you should explicitly encode the value of those actions (or the world states they are designed to achieve, if you are more utilitarian than virtue ethics) into it's utility function.
To give a concrete example from the area of lending, you shouldn't build a bot that can't accurately infer that blacks don't pay back their loans as much as whites. You should instead assign an explicit dollar value to how much you want to lend to blacks, and maximize profit + that dollar value. I.e., "lending to a black person is worth $500 to me".
My takeaway from this paper is that bots accurately learn accurate stereotypes from language, and word embeddings do a great job of learning true facts about the world. But some people wish they didn't, because there are things man should not know.
I get that you like to take contrarian positions. But you also make a habit of inserting flamebait into your posts about them. This combination is trolling. If you continue to do this we will ban you.
Specifically, we need you to stop playing the following game on HN:
Accidental trolling—e.g. triggering a flamewar with an unintended turn of phrase—is venial. But when you consistently generate such effects, you become responsible, regardless of what's wrong with others or their views. You passed that line on HN a long time ago. I'm told that self-responsibility is a conservative value and even recall you posting many criticisms of people whom you consider not to practice it. Please practice it here.We detached this comment from https://news.ycombinator.com/item?id=14116469 and marked it off-topic.
The article explicitly recommends building systems which can't learn those things, and suggests characterizing them is a first step:
"We recommend addressing this through the explicit characterization of acceptable behavior. One such approach is seen in the nascent field of fairness in machine learning, which specifies and enforces mathematical formulations of nondiscrimination in decision-making (19, 20). Another approach can be found in modular AI architectures, such as cognitive systems, in which implicit learning of statistical regularities can be compartmentalized and augmented with explicit instruction of rules of appropriate conduct (21, 22)."
According to the article, the most closely related work "is concurrent work by Bolukbasi et al. (6), who propose a method to “debias” word embeddings."
(Recall that in the context of the article, "bias" is something that generates true predictions that are objectionable rather than something which is false.)
I've made many attempts to explain, don't believe anyone could accuse us of being impatient with you, and do believe you're more than smart enough to understand. If your account consistently produces troll effects on HN, which it does, then at some point it's you who are responsible—not people who can't handle the truth, don't want 'man' to know things, don't know math, or however else you blame others. At some point enough is enough.
If you really need a further explanation I'd be happy to try, but would need some indication that you're asking in good faith.
Alternative and less biased conclusion: language corpora reflects statistical distributions in the society.
Sure, some of yummyfajtias comments are a bit trollish but he points out real contradictions in submissions and other comments. I have his threads bookmarked and visit them directly to check my thinking. It would be a huge loss for HN.
It's like the Socrates thing all over again. You would be banning someone offering interesting insight for upsetting the social order of the place.
> Our findings are also sure to contribute to the debate concerning the Sapir-Whorf hypothesis (17), because our work suggests that behavior can be driven by cultural history embedded in a term’s historic use. Such histories can evidently vary between languages.
No, not really. Linguistic relativity makes a claim about the direction of causation--from language to thought. This study does nothing to test the direction of causation, which is usually considered possible only with controlled experiments. This chicken and egg debate has been going on for the better part of the last century--it is not an easy problem.
> Our results also suggest a null hypothesis for explaining origins of prejudicial behavior in humans, namely, the implicit transmission of ingroup/outgroup identity information through language. That is, before providing an explicit or institutional explanation for why individuals make prejudiced decisions, one must show that it was not a simple outcome of unthinking reproduction of statistical regularities absorbed with language.
Once again, this was not an experiment capable of producing causal evidence. The authors have shown that human biases can be replicated by statistical learning of language corpora--admirable work, but nothing new here for Sapir-Whorf.