Yes, we are, and the author of the NYT article seems weirdly unaware of that. However, the issue here, I’m sure, is that Facebook’s algorithms don’t classify videos of white humans as involving “primates” as well.
The entire point of the article is to highlight what was an unexpected, unanticipated, and most unfortunate AI error. This is a learning experience, pun intended.
It is not incorrect by a dictionary definition, but there are other ways of being incorrect. If I identified you at a party by saying "fallingknife is the ape sitting on the sofa" you would probably interpret it differently than if I had said "fallingknife is the person sitting on the sofa". Both are correct, but most people wouldn't interpret them as equally polite or friendly.
Members of the set "person" are also members of the superset "ape", but that's irrelevant because the connotations are different, and it's the connotations that are annoying people.
The offense would be entirely based on the fact that I know you know the correct label, and there is emotional processing behind your word choice, neither of which is true in the case of AI.
We don’t really know that. It’s possible at some point the AI may have been classifying white humans as primates but nobody cares because it’s white humans. You only have a story worth reading when it’s against an underprivileged minority.
It's unlikely. Search for images of primate, and you'll see fairly dark, humanoid shapes. Poor training data then seems like a more probable reason for visual confusion.
> You only have a story worth reading when it’s against an underprivileged minority.
To some extent. But I think there would also be a story if some medicine would work badly on white people, or men. The problem here is that these labels would be there for everyone to see and comment on.
If facebook would live by their own rules, they would ban the AI team from using fb accounts.
Yeah, white primates are pretty weird. It only happened because a few migrated north into temperate Europe and Asia and lost their melanin a while ago.
Runaway sexual selection effects generally don't result in beneficial traits; reduced melanin is beneficial in less sunny climates, because cholecalciferol (so-called “Vitamin D₃”, even though it's not a vitamin) synthesis requires sun exposure.
And yes, having less of something is correctly called “losing” it in the English language. That was not a correction.
It would be fine for Black people too if the context was purely taxonomy and all people would be classified as primates. But that is not what happened here.
What "outrage machine?" I don't see anyone being outraged. The article doesn't project or describe any outrage, no one in this thread seems to be outraged. (yet.)
Taking something seriously is what I’m suggesting, not being outraged. Being flippant or intentionally over pedantic is dismissing it as unimportant or a non-issue.
It would be easy as a white person (as I am) to look at this and say, ah well it’s just an honest mistake. But this is just an example of a wider problem of AIs being trained on white people.
A while back I made the first automatic face swap app Face Juggler and I would often get complaints that it didn’t work as well on black people - I was using apple’s face detection so couldn’t do anything about it which was frustrating.
It might seem trivial but it isn’t. AI’s are going to be running more and more stuff that is a lot more important.
There's a history of emphasizing a closer relationship of some racial groups to the rest of the animal kingdom as a way of othering them or implying their nature/capabilities are more animal-like and less human-like.
So while it's technically true all humans are primates, the selective emphasis and its connotations is an issue.
(Weirdly, there's some substantial overlap between people who take offense at the idea they're related to other primates and feel a little freer to emphasize the relationship of a human out-group to other primates, which possibly illuminates something about the problems.)
The angry answers to your comment, and also the fact that the AI has this problem, seems to highlight the USA centric character of the data used and the people working on Facebook.
I wonder what would an AI trained with pictures from India would show.
> The video, dated June 27, 2020, was by The Daily Mail and featured clips of Black men in altercations with white civilians and police officers. It had no connection to monkeys or primates.
Huh. All the other incidents the NY Times article mentions come with links, but this one (that the article is ostensibly about) doesn't. Odd.
> The Daily Mail and featured clips of Black men in altercations with white civilians and police officers. It had no connection to monkeys or primates.
I'm guessing it ingested the Daily Mail's comment sections.
Don't these kinds of companies have a basic checklist before releasing features like this? And if so, why doesn't this checklist include things like "doesn't identify members of racial group as animals" ? It makes me think that any woke PR coming from Facebook is lip service, if they don't have even basic checks in place.
A large and complex task is well suited for the talented people behind Facebook. But even a list of basic searches that exercised the engine would have caught this. No excuses imo, particularly since it happened to Google[0] not that long ago. Was nobody paying attention?
Another way to put this is: nobody working in this field has control over what they are building, and can't make any promises about how it will behave.
It's like launching a rocket ship that you can't test in a physical simulation first. Something is going to go wrong that can't be predicted because we lack the understanding, but this is seen as an acceptable risk.
Which is inherent to the problem. The domain is the space of _all possible natural images_, it’s so big, it’s ridiculous. Fundamentally, these techniques do not analyze images like humans do, but are rather trained to pick out any salient signal it can latch on to. That seems to be “primates are mostly like humans but darker”, which is superficially true but a pretty weak definition as it includes dark skinned humans.
It's probably because humans are primates – but the AI systems often have to treat “human” as a completely separate category as “primate”, so they have to draw weird, complex boundaries around “primate” (actually “all non-human primates”). When the “primate” classification is stronger than the “human” classification, the system says “primate” rather than “human”, and if it's predominantly been trained on “pictures of white Americans are not pictures of primates”, its “primate” definition might not be skewed to miss everyone else.
I expect you'd get better results if you allowed the system to call humans “primates”, then accept “human primate” as “human” when parsing the output. (That is, leave the “is_primate” output line floating while training on pictures of humans.) I don't know whether that would work, though.
These models typically don’t have hierarchical labels like that, and they apply a softmax to their output - which means /one/ label will be considered correct. (A softmax means taking the exp of your predicted scores, then divide by the sum.)
I know – but there's no technical reason they shouldn't have more complex relationships between labels (assuming you can train that, which I don't know). If we can't get better training data, at least trying to fix the problem at the algorithms (instead of slapping crude filters on the end of them) would be nice.
I agree in spirit but disagree in practice, I think. Like we said previously, the domain is humongous so even establishing meaningful relationships between labels and sublabels is extremely difficult. Many cases are likely ambiguous too, our human understanding isn’t actually hierarchical - it’s much more elusive. It’s a square peg round hole type problem really, humans don’t really think in terms of labels in the first place, we mostly use them for the purposes of language.
Primates are not darker than humans. Primates is the Order, they came in all colors, including brilliant blue, white with black stripes, orange, yellow or red.
We live in a world when everybody is offended by nothing. The problem is that nobody should be offended by being called a Gorilla. In the same way as nobody is offended by being called an eagle or a wolf. Is a wonderful animal, smart, strong, protective and gentle. What if some idiots used the term pejoratively five generations ago? We know better. Societies can change.
If white people is not being classified as primates, the algorithm should be corrected so they are. Not fixed excluding black people from humanity.
People should be educated also to understand that an AI algorithm is returning probability, not truth
nobody working in this field has control over what they are building, and can't make any promises about how it will behave. If you are working on machine learning as an employee of Facebook, Google, etc. shouldn't you assume the worst possible outcome from your efforts? I often wonder how people manage this cognitive dissonance.
Regardless of whether they do, the tech that underlies modern image classification approaches (i.e. deep neural networks) is pathologically hard to understand to the point where it's practically impossible to provide the guarantees you're describing. For example, adding noise to a picture of a panda can make a network think it's a gibbon: https://towardsdatascience.com/breaking-neural-networks-with...
That's not to say, of course, that there's nothing to be done. People who aren't white have historically been laughably underrepresented in computer vision datasets, and while that's been changing, I gather that representation still isn't where it needs to be.
> It makes me think that any woke PR coming from Facebook is lip service
Corporate PR about anything that isn't concrete things which misleading statements would be both easily revealed and have adverse (e.g., securities-laws-related) consequences is almost 100% lip service, so, yeah, that's a safe assumption, anyway.
The question is how would they know they are members of a racial group when it couldn't even identify them as part of the righ species? The answer is that these algorithms are not fit for serious classification and are not appropriate for this type of application.
EDIT: pardon me, what I wrote is wrong. We are all primates after all :)
Not really wanting to delve into the touchy issue of this particular ai fuckup, but it got me thinking about the issue of using ai to compartmentalize and categorize things in general.
This is a very obvious example of how bad it can be. Regular complaints from people about ai recommendation engines and most other similar ai driven tagging systems exist.
What i wonder though is on a more subtle level, how these ai categorizers are quietly altering our perceptions of things and the world?
We notice when it's something obvious like the story in the OP, but how many quiet little mistakes do we just ignore or not notice that on a subconcious level are changing the way we think about or classify things.
More and more it seems like society and all things are being neatly placed in little boxes, where they're filed, stamped, indexed, and numbered by fancy algorithms operating without much human oversight, until something like this happens.
How much of this is subconciously affecting human perception of the world?
OK, I'll try to tread carefully here, but I think it's really helpful to separate the technical issues from the societal issues.
In the Google photo case, I've seen the picture in question, and it's not difficult an all to see how a rudimentary statistical algorithm would mistake the photo for a gorilla. No, no human would make this mistake, but due to the lighting in the photo and the woman's hair it's really not hard to see how that mistake would be made, similar to how Tesla's AI can mistake the sun, low on the horizon, as a yellow traffic light.
The algorithm was not "racist", it just didn't have the social context that describing black people as apes and monkeys has a long racist history. And that is really the fundamental problem with most AI these days - it can get very good at statistical inference, but it doesn't have the background knowledge and logic to be able to "think" in the same way humans do.
In a similar vein, I recall someone lamenting a couple years ago how Google's street directions would never mispronounce "Malcolm X Boulevard" as "Malcolm 10 Boulevard" if they had any black programmers. Again, given 99% of the time when you see "X" in an address you'd pronounce it as 10, it's not hard to see how this could happen. The problem is that some errors are much more offensive than others, and AI can't really reason about that.
The problem is the training set, which apparently skews too much towards white ppl, and doesn't include checks for the primates/non-white humans differentiation. One could argue this is due to white people building things for white ppl, which does indeed point towards systemic racism in the overall development process.
But the problem is that there isn't any primates/non-white humans differentiation because humans are primates. I doubt anybody smart enough to be an AI engineer of any race would care if they were called a primate, because they are.
I feel like this is a very "engineer" response that completely ignores how society in general would view this.
Yes, humans are primates. But this wasn't a discussion about taxonomy in biology class. There are lots of cases where primates is used in everyday language where it is clearly intended to refer solely to monkeys and apes. When I go to the "primate house" at the zoo I'm not going to expect to see naked humans behind the glass.
And in cases where that engineer thing is used on people - it matters when you label a black person as a primate on a live social media network, since it's understood that would be likely to cause significant offense.
Right and that’s why we should curtail its use because thinking in an engineering box suits very few real world examples.
This algorithm is clearly broken, it doesn’t label all people as primates, it labels Black people wearing a green T-shirt possibly as such, I mean I’m sorry but that algorithm is definitely broken. The person in question should not be labeled as primate by any means not should any other humans unless the context is specifically some form of species taxonomy.
Also your reaction seems to be all daisies and roses, and I don’t blame you for your social context ignorance you displat, I guess you are just an engineer after all..
I was gonna write "monkey-blackppl-differentiation", but decided it was too offensive and aimed for more neutral language, thinking for anybody who doesn't want to be an asshole it's clear what it's about.
Interested in hearing how you obtained the proprietary training material for this algo from Apple. Can we see it?
Also which teams at apple made the algorithm? Since you claim to know they were all 'white'.
In reality, I'm sure everyone working on this team was well aware of these potentials for training set bias before most of us even knew what ML was.
Even if this was considered it'd be pretty risque to have an employee dedicate time specifically to differentiating black ppl from apes in the algorithm.
I worked closely with an AI ethicist at $100B+ tech company company and their findings were that all of our AI training data was biased and the ML teams had no plans to de-bias their training data because they had no incentive to do so. I have no reason to believe this finding doesn’t hold for the FAANG companies. AI ethics are mostly an afterthought in tech today.
And to your second point about how would we know if the team was “all white,” of course we don’t know that and it’s probably not true. What we do know is that tech in the us is disproportionately whiter than the general public, and ~10 person teams with no black people on them are the norm, not the exception.
Edit: I should say that 10 person teams without black people on them are the norm in FAANG companies and others headquartered in the Bay Area. Other tech hubs in the US are much more diverse.
Clearly my point was to highlight your prejudicial instincts around this issue.
For all you know the entire team that worked on this project could've been black. Or zero of them could've been white etc.
We can and should have a conversation about training sets and ML ethics. Resorting to unprovoked racist attacks is quite a counterproductive approach imo.
Google's street directions also fail miserably with international road names.
Words are fine, but person names or abbreviations will often use the English pronunciation which can be inexact or even incomprehensible.
For example: In Brazil, the major roadways are named with the initial code of the state, or BR if it's a federal road, and a bunch of numbers. So BR-101, for example. These are never named in full, so someone reading "BR-101" would not say "Brasil 101", they would say "B R 101".
There is a state called Santa Catarina, with initials SC, but Google decides that we somehow get transported to the United States somehow, since it reads "SC-406" as "South Carolina 406".
A human would at the very least have the context that SC definitely is not a common abbreviation for South Carolina in Brazil, even if they did not know any state names.
Separating technical issues from social ones is the source of the problem. I don't think it'll be the source of the solution.
One of the big issues with relying on AI is training data, and the responsibility for an inadequately trained AI model falls on the people who judged it ready for use, but that doesn't mean the outcome of the AI isn't racist. An AI doesn't need to know the sociopolitical history of race to be doing racist things.
Also, I've never heard of a location with Roman numerals in the name. I agree that programming in the context of knowing who Malcolm X is into a map program might be asking a bit much, but I think an "easier" implementation (reading them as letters) would have also done the right thing.
>I've never heard of a location with Roman numerals in the name.
At least in Europe, extremely common. In Paris you have Boulevard Henry IV, in Lyon the street named after the same king. There's thousands of kings and queens and many more roads named after them. There's also schools, buildings and such named after them.
If you know in advance that Primate vs. Human confusions are going to be a potential issue that upset a lot of people (say, because it already happened with a similar system at another company) there are certainly ways of trying to ensure that same confusion doesn't happen.
Ways to actually fix the problem:
1. Probe the decision boundary between these two classes in your training and test sets. I.E. look for the humans closest to being misclassified as primates. You could probably quickly get a sense if you are near/at risk of making this misclassification. You may also possibly find 1 or 2 mislabeled examples that are throwing things off and can be corrected.
2. Boost your training set by labeling more unlabeled images that are near this decision boundary.
So my point was just: this was an issue that could have been anticipated because it has already happened before. It is unfortunate that this issue came up when there are things you can do (such as the steps above), which I believe could have totally squashed this issue with enough iteration.
I can't know for sure that this is not what the Facebook team did. If they did take care to try and avoid this harmful model confusion, then I would be very surprised given my experience with deep learning and computer vision.
At a minimum, it would have been pretty trivial and a low impact to users to remove the primate class if they didn't have the time/bandwidth to really investigate this more thoroughly as I've described.
You can try to argue "hey no one should get upset about this confusion because computer vision is hard and mistakes happen", but I don't think that's a very solid argument either given the long and relatively recent history of racists misclassifying a particular subset of humans as primates.
For Asians, there is a history of cameras saying "Someone Blinked!" I think it is, or at least should be, part of the conversation. My partner is Asian, and we had a camera with this problem. We thought it was funny, but it demonstrates weaknesses in computer image processing vs. human visual processing that probably has nothing to do with racism.
To me it is just a reminder that AI based picture recognition is not a perfect science and that I should fight any and all attempts to include it in places where it could cause most harm ( policing comes to mind ).
Hahahaha! Artificial intelligence correctly labels human as member of category "primate." Human "intelligence" flags comment because it doesn't know it is a member of category "primate"
According to the article, the video in question "featured clips of Black men in altercations with white civilians and police officers.", so the primates label could just as well apply to the white civilians or police.
So? The label is correct. I have only contempt for people who are offended by the fact that they are a member of a biological group that includes apes.
People are saying that this is a very difficult problem to solve. It is not.
The problem is not classifying people as "animals;" the problem is classifying people as a known stereotype. Make a list of every animal/thing that correlates to a known stereotype in humans. Then program your AI to never classify those animals/things.
It is not a big deal if your AI isn't able to detect gorillas. That doesn't really affect UX.
I will add that just because it is an easy problem to solve does not mean they should solve it. An AI classifying someone as something else should be something to laugh at, not take offense to.
This particular AI failure mode keeps surfacing, but seems to be fixed rapidly when it generates bad press. How do they actually correct it? Teach it some more granular categories of living things? Slap a sensitivity filter on top of its output?
There’s an amazing number of “but people are primates” comments. Correctness here includes specificity. If person and primate are both candidates, then primate isn’t a correct answer to what this is trying to do. You wouldn’t react to the phrase ‘treating women as objects’ by saying people are objects, right?
You make a good point, and I'd agree that it's implicit that they mean "non-human primates". Otherwise they could solve this by labelling lots of humans as "primates" in the dataset.
On the other hand, I think there's a semantic difference between your two cases. I'd argue that treating a woman as an "object" is a different sense of the word to how women (and men and teapots) are literally "objects".
The people here saying the label is "correct" are both wrong, and missing the point. The objective of the model is not "taxonomic classification", the objective is "video recommendation". The label, in the context of the purpose of the model, is completely wrong.
Why does there tend to be a reflexive set of predictable comments on any article about racism within the tech industry that boils down to "Here is some half baked unresearched explanation that justifies why this instance of racism isn't that big of a deal?"
Because it permits people to take no action. If people admit there is a problem then they must justify their inaction. But if there is no problem then their inaction is justified be definition.
On a technical level, yes, all humans are primates, but if the algorithm were following this rule, it would tag all humans in all photos as primates, but it didn’t do that. It only tagged black people.
That takes us to a social meaning level. In this context, you have to consider the long, racists history of dehumanizing black people. They have often been called “monkey” and “ape” with the intent to degrade and to segregate them from “people”.
In that light, and the fact that the label was not applied in all cases where appropriate, it is hard to see it as anything but racist. Probably not deliberately but if the training material contained racist material, it could bias the result. Even if the training material was inadvertently unbalanced and contained very few black faces, the algorithm would have less experience with black faces and would not know how to categorize them as well as white faces.
This is analogous to people who grow up and live largely isolated from close associates with those of other races. Their experiences are filled with varied samples of interactions with people of their own race, but are poor in samples with other races. In the case where we have less experience, we tend to fall back on broad stereotypes and our reactions show strong racial bias.
I've dealt with a lot of badly labelled datasets, but I can't say I ever saw one maliciously labelled by racists.
There are other more subtle problems that may contribute. As an amateur photographer I've been told that most cameras are calibrated for light skin. If you aren't careful with your setup you'll get bad definition in shadows and relatively very dark areas. If you have badly calibrated/exposed photos then very dark faces, whether of sub-Saharan people or gorillas, may be poorly distinguished. I bet there are loads of unfair, hard-to-catch issues like that.
By the way I don't mean to say that you're wrong, but just to offer a Hanlon's Razor-type counterpoint as it occurred to me.
This is usually a good example of societal bias compounding.
It's possible that nobody directly involved in this was actively racist, but we're building on centuries of racism in every facet. So a little bit of bias in each technology and data set involved, not to mention the engineers and their testing, just adds up
>but we're building on centuries of racism in every facet.
Nooo. We're building on techniques that have worked well enough up til now, but people don't bother to dig in a d come to terms with what's actually going on.
Not everything is racist. It's more a tyranny of 'good enough' and convenience.
So you're saying there's no bias in the data sets , or in society? Do you think that someone who thinks there's no bias in their dataset would then dig in to see what's going on? Do you think people are going to recognize the negative biases in the data if they don't face those issues themselves?
Bigotry is part of society, because it's existed for so long and still does. It doesn't mean that everyone is actively racist. It just means you can't just say there's no bigotry affecting things, and in saying there isn't, you're now blinding yourself to the possibility of negative biases.
Once we know about the the systemic bias, even if accumulated unintentionally - then what do we do about it. I think we are obligated morally and technically to do more after that point.
A similar situation exists with accommodation of people with disabilities. Once you know a building architecture can shut out people by design decisions- at some point we worked through at least some basic level of agreement on principle that accommodation is required.
That was my point. If the training data contained mostly white faces and few black faces, the AI would have fewer cases to base it’s weightings when trying to analyze a photo with black faces.
I'm sorry but no. I don't believe this at all. What are the chances of say, you finding a dataset maliciously labelled by racists and how would you uncover that? Would you ask every software engineer how he felt about Black Lives Matter as he tagged images? It's more likely that there is malicious intent in a company where 2% of software engineers are African-American than not.
LIME - Local Interpretable Model-Agnostic Explanations comes to mind.
As far as I do understand the classification was the tagging 'primate' on a video of people with different color of skin and different facial features.
The interpretation, that our Lady AI meant harm to the people with dark skin color with tagging them as primates is a loading of the viewer.
By the way,´a great new opportunity for cyber lawyers and machine inquisitors.
Maybe the AI was right and referring to the police behavior as "primates".
We all know how chimps can behave towards other chimps from another population. We humans are not much different.
There was also a video of an altercation between black and white people, and the prompt still enquired if it should save the video under "Primates" category.
The title has obviously been editorialized for outrage.
I would like to know if the model was poorly trained or if this was caused by a malicious individual modifying the algorithm to generate offensive suggestions.
I think the technology is very limited but instead it’s overused in cases that shouldn’t. No matter what model is used and how much it’s trained, we have to come to our senses and realize that it will never be perfect and should’t be used everywhere. It’s shudders me to think that this technology is used in legal cases to make life affecting decision that could be absolutely wrong
172 comments
[ 4.2 ms ] story [ 226 ms ] threadThe entire point of the article is to highlight what was an unexpected, unanticipated, and most unfortunate AI error. This is a learning experience, pun intended.
Members of the set "person" are also members of the superset "ape", but that's irrelevant because the connotations are different, and it's the connotations that are annoying people.
> You only have a story worth reading when it’s against an underprivileged minority.
To some extent. But I think there would also be a story if some medicine would work badly on white people, or men. The problem here is that these labels would be there for everyone to see and comment on.
If facebook would live by their own rules, they would ban the AI team from using fb accounts.
And yes, having less of something is correctly called “losing” it in the English language. That was not a correction.
It would be easy as a white person (as I am) to look at this and say, ah well it’s just an honest mistake. But this is just an example of a wider problem of AIs being trained on white people.
A while back I made the first automatic face swap app Face Juggler and I would often get complaints that it didn’t work as well on black people - I was using apple’s face detection so couldn’t do anything about it which was frustrating.
It might seem trivial but it isn’t. AI’s are going to be running more and more stuff that is a lot more important.
So while it's technically true all humans are primates, the selective emphasis and its connotations is an issue.
(Weirdly, there's some substantial overlap between people who take offense at the idea they're related to other primates and feel a little freer to emphasize the relationship of a human out-group to other primates, which possibly illuminates something about the problems.)
Which, by the way, is related to the discussed problem.
The embarrassing thing about these phenomena is that a machine holds up a mirror to us. Naked and ugly.
To 'patch' the mirror won't change a thing. Especially when he is right under the right circumstances. In the right context.
I wonder what would an AI trained with pictures from India would show.
Huh. All the other incidents the NY Times article mentions come with links, but this one (that the article is ostensibly about) doesn't. Odd.
I'm guessing it ingested the Daily Mail's comment sections.
Inference tasks isn’t like software testing where the states are well defined.
0. https://www.theverge.com/2018/1/12/16882408/google-racist-go...
It's like launching a rocket ship that you can't test in a physical simulation first. Something is going to go wrong that can't be predicted because we lack the understanding, but this is seen as an acceptable risk.
I expect you'd get better results if you allowed the system to call humans “primates”, then accept “human primate” as “human” when parsing the output. (That is, leave the “is_primate” output line floating while training on pictures of humans.) I don't know whether that would work, though.
We live in a world when everybody is offended by nothing. The problem is that nobody should be offended by being called a Gorilla. In the same way as nobody is offended by being called an eagle or a wolf. Is a wonderful animal, smart, strong, protective and gentle. What if some idiots used the term pejoratively five generations ago? We know better. Societies can change.
If white people is not being classified as primates, the algorithm should be corrected so they are. Not fixed excluding black people from humanity.
People should be educated also to understand that an AI algorithm is returning probability, not truth
That's not to say, of course, that there's nothing to be done. People who aren't white have historically been laughably underrepresented in computer vision datasets, and while that's been changing, I gather that representation still isn't where it needs to be.
Corporate PR about anything that isn't concrete things which misleading statements would be both easily revealed and have adverse (e.g., securities-laws-related) consequences is almost 100% lip service, so, yeah, that's a safe assumption, anyway.
https://www.credera.com/insights/racial-bias-in-machine-lear...
If only it was explained in a conveniently linked article.
EDIT: pardon me, what I wrote is wrong. We are all primates after all :)
This is a very obvious example of how bad it can be. Regular complaints from people about ai recommendation engines and most other similar ai driven tagging systems exist.
What i wonder though is on a more subtle level, how these ai categorizers are quietly altering our perceptions of things and the world?
We notice when it's something obvious like the story in the OP, but how many quiet little mistakes do we just ignore or not notice that on a subconcious level are changing the way we think about or classify things.
More and more it seems like society and all things are being neatly placed in little boxes, where they're filed, stamped, indexed, and numbered by fancy algorithms operating without much human oversight, until something like this happens.
How much of this is subconciously affecting human perception of the world?
https://www.ecupatria.org/2020/11/24/the-primate-of-church-o...
https://en.wikipedia.org/wiki/Primate_(bishop)
https://www.theverge.com/2018/1/12/16882408/google-racist-go...
In the Google photo case, I've seen the picture in question, and it's not difficult an all to see how a rudimentary statistical algorithm would mistake the photo for a gorilla. No, no human would make this mistake, but due to the lighting in the photo and the woman's hair it's really not hard to see how that mistake would be made, similar to how Tesla's AI can mistake the sun, low on the horizon, as a yellow traffic light.
The algorithm was not "racist", it just didn't have the social context that describing black people as apes and monkeys has a long racist history. And that is really the fundamental problem with most AI these days - it can get very good at statistical inference, but it doesn't have the background knowledge and logic to be able to "think" in the same way humans do.
In a similar vein, I recall someone lamenting a couple years ago how Google's street directions would never mispronounce "Malcolm X Boulevard" as "Malcolm 10 Boulevard" if they had any black programmers. Again, given 99% of the time when you see "X" in an address you'd pronounce it as 10, it's not hard to see how this could happen. The problem is that some errors are much more offensive than others, and AI can't really reason about that.
Yes, humans are primates. But this wasn't a discussion about taxonomy in biology class. There are lots of cases where primates is used in everyday language where it is clearly intended to refer solely to monkeys and apes. When I go to the "primate house" at the zoo I'm not going to expect to see naked humans behind the glass.
This algorithm is clearly broken, it doesn’t label all people as primates, it labels Black people wearing a green T-shirt possibly as such, I mean I’m sorry but that algorithm is definitely broken. The person in question should not be labeled as primate by any means not should any other humans unless the context is specifically some form of species taxonomy.
Also your reaction seems to be all daisies and roses, and I don’t blame you for your social context ignorance you displat, I guess you are just an engineer after all..
Also which teams at apple made the algorithm? Since you claim to know they were all 'white'.
In reality, I'm sure everyone working on this team was well aware of these potentials for training set bias before most of us even knew what ML was.
Even if this was considered it'd be pretty risque to have an employee dedicate time specifically to differentiating black ppl from apes in the algorithm.
And to your second point about how would we know if the team was “all white,” of course we don’t know that and it’s probably not true. What we do know is that tech in the us is disproportionately whiter than the general public, and ~10 person teams with no black people on them are the norm, not the exception.
Edit: I should say that 10 person teams without black people on them are the norm in FAANG companies and others headquartered in the Bay Area. Other tech hubs in the US are much more diverse.
For all you know the entire team that worked on this project could've been black. Or zero of them could've been white etc.
We can and should have a conversation about training sets and ML ethics. Resorting to unprovoked racist attacks is quite a counterproductive approach imo.
Words are fine, but person names or abbreviations will often use the English pronunciation which can be inexact or even incomprehensible.
For example: In Brazil, the major roadways are named with the initial code of the state, or BR if it's a federal road, and a bunch of numbers. So BR-101, for example. These are never named in full, so someone reading "BR-101" would not say "Brasil 101", they would say "B R 101".
There is a state called Santa Catarina, with initials SC, but Google decides that we somehow get transported to the United States somehow, since it reads "SC-406" as "South Carolina 406".
A human would at the very least have the context that SC definitely is not a common abbreviation for South Carolina in Brazil, even if they did not know any state names.
It has gotten better, but even in 2018 it should have been in- excusable to be mispronouncing names that badly.
One of the big issues with relying on AI is training data, and the responsibility for an inadequately trained AI model falls on the people who judged it ready for use, but that doesn't mean the outcome of the AI isn't racist. An AI doesn't need to know the sociopolitical history of race to be doing racist things.
Also, I've never heard of a location with Roman numerals in the name. I agree that programming in the context of knowing who Malcolm X is into a map program might be asking a bit much, but I think an "easier" implementation (reading them as letters) would have also done the right thing.
At least in Europe, extremely common. In Paris you have Boulevard Henry IV, in Lyon the street named after the same king. There's thousands of kings and queens and many more roads named after them. There's also schools, buildings and such named after them.
Ways to actually fix the problem:
1. Probe the decision boundary between these two classes in your training and test sets. I.E. look for the humans closest to being misclassified as primates. You could probably quickly get a sense if you are near/at risk of making this misclassification. You may also possibly find 1 or 2 mislabeled examples that are throwing things off and can be corrected.
2. Boost your training set by labeling more unlabeled images that are near this decision boundary.
So my point was just: this was an issue that could have been anticipated because it has already happened before. It is unfortunate that this issue came up when there are things you can do (such as the steps above), which I believe could have totally squashed this issue with enough iteration.
I can't know for sure that this is not what the Facebook team did. If they did take care to try and avoid this harmful model confusion, then I would be very surprised given my experience with deep learning and computer vision.
At a minimum, it would have been pretty trivial and a low impact to users to remove the primate class if they didn't have the time/bandwidth to really investigate this more thoroughly as I've described.
You can try to argue "hey no one should get upset about this confusion because computer vision is hard and mistakes happen", but I don't think that's a very solid argument either given the long and relatively recent history of racists misclassifying a particular subset of humans as primates.
They were careful, otherwise they wouldn't have labeled it "primates", but e.g. "monkeys" instead.
edit:
Hahahaha! Artificial intelligence correctly labels human as member of category "primate." Human "intelligence" flags comment because it doesn't know it is a member of category "primate"
https://en.wikipedia.org/wiki/Primate#Classification_of_livi...
If you find a picture of my white face anywhere, feel free to put a "primate" caption under it.
The problem is not classifying people as "animals;" the problem is classifying people as a known stereotype. Make a list of every animal/thing that correlates to a known stereotype in humans. Then program your AI to never classify those animals/things.
It is not a big deal if your AI isn't able to detect gorillas. That doesn't really affect UX.
I will add that just because it is an easy problem to solve does not mean they should solve it. An AI classifying someone as something else should be something to laugh at, not take offense to.
On the other hand, I think there's a semantic difference between your two cases. I'd argue that treating a woman as an "object" is a different sense of the word to how women (and men and teapots) are literally "objects".
Words have meaning on multiple levels.
On a technical level, yes, all humans are primates, but if the algorithm were following this rule, it would tag all humans in all photos as primates, but it didn’t do that. It only tagged black people.
That takes us to a social meaning level. In this context, you have to consider the long, racists history of dehumanizing black people. They have often been called “monkey” and “ape” with the intent to degrade and to segregate them from “people”.
In that light, and the fact that the label was not applied in all cases where appropriate, it is hard to see it as anything but racist. Probably not deliberately but if the training material contained racist material, it could bias the result. Even if the training material was inadvertently unbalanced and contained very few black faces, the algorithm would have less experience with black faces and would not know how to categorize them as well as white faces.
This is analogous to people who grow up and live largely isolated from close associates with those of other races. Their experiences are filled with varied samples of interactions with people of their own race, but are poor in samples with other races. In the case where we have less experience, we tend to fall back on broad stereotypes and our reactions show strong racial bias.
There are other more subtle problems that may contribute. As an amateur photographer I've been told that most cameras are calibrated for light skin. If you aren't careful with your setup you'll get bad definition in shadows and relatively very dark areas. If you have badly calibrated/exposed photos then very dark faces, whether of sub-Saharan people or gorillas, may be poorly distinguished. I bet there are loads of unfair, hard-to-catch issues like that.
By the way I don't mean to say that you're wrong, but just to offer a Hanlon's Razor-type counterpoint as it occurred to me.
It's possible that nobody directly involved in this was actively racist, but we're building on centuries of racism in every facet. So a little bit of bias in each technology and data set involved, not to mention the engineers and their testing, just adds up
Nooo. We're building on techniques that have worked well enough up til now, but people don't bother to dig in a d come to terms with what's actually going on.
Not everything is racist. It's more a tyranny of 'good enough' and convenience.
Bigotry is part of society, because it's existed for so long and still does. It doesn't mean that everyone is actively racist. It just means you can't just say there's no bigotry affecting things, and in saying there isn't, you're now blinding yourself to the possibility of negative biases.
A similar situation exists with accommodation of people with disabilities. Once you know a building architecture can shut out people by design decisions- at some point we worked through at least some basic level of agreement on principle that accommodation is required.
LIME - Local Interpretable Model-Agnostic Explanations comes to mind.
As far as I do understand the classification was the tagging 'primate' on a video of people with different color of skin and different facial features.
The interpretation, that our Lady AI meant harm to the people with dark skin color with tagging them as primates is a loading of the viewer.
By the way,´a great new opportunity for cyber lawyers and machine inquisitors.
The title has obviously been editorialized for outrage.