This article was eye rollingly heavy on moralizing ultimately to only eventually hit on the point that what the algorithms are measuring is the odds somebody will be convicted and not the odds somebody does illegal things. Appearance is known to dramatically affect your chances at trial.
I remember reading about a similar result in software that judged job applicants. The system was trained on resumes of past applicants and the hired/not-hired outcome. Unfortunately, instead of providing a neutral guide, it just learned the biases of the humans who had made past hiring decisions. Of course, the designers of the system had excluded demographic information, but the system found proxies like extra-curricular activities that correlated with race, gender or social class.
> that criminals are under-evolved, subhuman beasts
This has everything to do with common attitudes. Ask any person about what kind of person does the kind of crime or moral aberration they are most passionate about and you'll get a similar response. People can't imagine that criminals might have had some understandable train of thought prior to committing a crime; or worse that they may have done exactly the same thing in their shoes.
> These algorithms reportedly make false accusations against students with disabilities who move their faces and hands in atypical ways, and Black students have indicated that they have been required to shine bright lights in their faces so as to have their features detected at all.
Warning: I will be using the term police quite broadly here, but it's anyone involved in any kind of policing or detective activity.
Any algorithm used in policing that produces false negatives needs to be retired until it can be proven they can be used provably equally on people, which I suspect is never. The problem is that most people believe that policing should be more efficient. More efficient police work means violating your privacy, breaking algorithms, and using lazy surveillance and data collection. If you want your freedoms intact then good old fashioned police and detective work needs to be just that; the police need to be smarter, not have smarter tools.
> Yet equating convictions with criminality seems to register with the authors mainly as an empirical flaw, in that using pictures of convicted criminals, but not of the ones who were cleared, introduces a statistical bias that skews the results. They said they were “deeply baffled” at the public outrage in reaction to a study that was intended “for pure academic discussions,” which also suggests an unawareness that their work’s flaws go beyond sloppy statistics.
Absolutely a fair criticism. I'm surprised anyone was baffled that the criminal justice system is highly flawed, other than some mathematicians who lead quite prestigious lives. I've long thought it's pretty suspect that there's a morality bar forming (or that has formed) in programming. A force beyond laws can make you unhireable, what a precedent! Really, I find the idea that we might deny entry to programming even for convicted felons that have served their time problematic. Their insights might be quite useful here.
> Ask any American about what kind of person does the kind of crime or moral aberration they are most passionate about and you'll get a similar response
> ....
> Especially Americans can't imagine...
Why is this targeting Americans? In my home country these same biases exist. There's a "group of people" that are deemed more likely prone to criminal behavior.
For some reason America seems to serve as a lightning rod for these types of criticisms, as if the rest of the world is better somehow.
The American context is easier to understand for most people. We're surrounded by American culture on English-speaking websites. The context of <other country> isn't as easy to understand for people that don't know <other country> well.
I'm certainly not saying your struggles don't exist. I live in America and it'd be disingenuous and probably counter-intuitive for me to create an exhaustive list of all people's who suffer these circumstances.
That's to say, I can really only speak in earnest about the devil that I know.
> People can't imagine that criminals might have had some understandable train of thought prior to committing a crime; or worse that they may have done exactly the same thing in their shoes.
People can imagine this. They understand it. They just implicitly recognize that humans are rationalization machines, that their reasons don't matter. Every single person in human history, no matter how abhorrent, had their reasons, had a whole litany of reasons that they could recite. Plenty of them were able to convince other people they were right, too. That's how you get everything from mass surveillance to genocides.
Now, it's part of being human that we recognize that in extraordinary circumstances, maybe those exigencies can justify or ameliorate your wrongs. But every single person in human history is going to be able to explain to you why that applies to their case, that they had to do it.
I had previously written a critique of this article, for a different forum. I'm reproducing it here, slightly edited for this venue:
With concerns of systemic racial bias such a hot topic today, the subject of hidden bias in machine learning systems driven by "big data" has been a frequent question. An article "The Dark Past of Algorithms That Associate Appearance and Criminality", by Catherine Stinson, appearing in the January-February 2021 issue of American Scientist, attempts to make the point that "Machine learning that links personality and physical traits warrants critical review", and largely succeeds at this task. However, most of the piece attempts to provide that critical review, and at this task it fails.
The article starts off offering phrenology as a close cousin to contemporary efforts to use facial recognition to identify criminals, or as "gaydar". A series of examples of it (mis)use in Chinese surveillance, or to monitor schoolchildren to ensure they pay attention in class, capping this off with the claim that Black students have had to endure bright lights pointed at their faces to enable such systems to work. Such applications do seem unfair, and potentially so disruptive as to be counterproductive. So in this way the author is successful in showing a need for critical review.
Stinson goes on to discuss how some of the derided foundations of phrenology - the idea that particular cognitive functions can each be seated in a particular locale within the brain - has now become "a standard assumption in mainstream neuroscience". However, she then claims that despite this, the science has suffered from sloppy methods. One example study is cited that looked for criminality based on photographic input had biased training data by using police photos to represent the convicts, versus professional photos scraped from the web for nonconvicts. It's easy to see how this would skew the results.
The author first claims, "using pictures of convicted criminals, but not of the ones who were cleared, introduces a statistical bias that skews the results", and that seems a fair point. She also says, "the researchers also admitted that taking court convictions as synonymous with criminality was a 'serious oversight'", but offers no explanation for why that should be. My best guess is that we should be able to assume that a "guilty" verdict is the best judgment we can get as to whether someone is guilty of a given crime; but that other people who are guilty of a crime but never caught will be left out of the data set, so our training would bias the system to give a pass to people who don't get caught. I suppose that's fair, too.
It's at this point that I think things start to derail. Stinson goes on to say,
their framing suggests that criminality is an innate characteristic, rather than a response to social conditions such as poverty or abuse, or a label applied to exert social control. […] Part of what makes the data set questionable on empirical grounds is that who gets labeled “criminal” is hardly value-neutral.
That's largely a philosophical question that she doesn't attempt to back up. But the fact that she does make this point, about extrinsic forces being an important contributor, actually defeats her later argument.
Along the way, she feels the need to throw in the standard anti-HBD argument that "psychologists who studied the heritability of intelligence, such as Cyril Burt and Philippe Rushton, had to play fast and loose with their data to make it look like they had found genuine connections between skull size, race, and IQ." I don't intend to debate that here, there's ample material out there showing the correlation between race and IQ (although I don't know about skull size). Either way, it doesn't seem to be relevant to her actual argument about revelations of criminality from facial characteristics.
Technologists tend to not think through the humanist problem very well. Just because you can doesn't always mean you should.
Associations between appearance (as expressed genetically) and that person's capacity for something (intellectually, morally, or otherwise) is plain old eugenics. Given it's sordid history and it's horrible effects on the 20th century, we should be banning this sort of work.
13 comments
[ 3.6 ms ] story [ 43.7 ms ] threadhttps://www.youtube.com/watch?v=ySXpRf3ggJE
This has everything to do with common attitudes. Ask any person about what kind of person does the kind of crime or moral aberration they are most passionate about and you'll get a similar response. People can't imagine that criminals might have had some understandable train of thought prior to committing a crime; or worse that they may have done exactly the same thing in their shoes.
> These algorithms reportedly make false accusations against students with disabilities who move their faces and hands in atypical ways, and Black students have indicated that they have been required to shine bright lights in their faces so as to have their features detected at all.
Warning: I will be using the term police quite broadly here, but it's anyone involved in any kind of policing or detective activity.
Any algorithm used in policing that produces false negatives needs to be retired until it can be proven they can be used provably equally on people, which I suspect is never. The problem is that most people believe that policing should be more efficient. More efficient police work means violating your privacy, breaking algorithms, and using lazy surveillance and data collection. If you want your freedoms intact then good old fashioned police and detective work needs to be just that; the police need to be smarter, not have smarter tools.
> Yet equating convictions with criminality seems to register with the authors mainly as an empirical flaw, in that using pictures of convicted criminals, but not of the ones who were cleared, introduces a statistical bias that skews the results. They said they were “deeply baffled” at the public outrage in reaction to a study that was intended “for pure academic discussions,” which also suggests an unawareness that their work’s flaws go beyond sloppy statistics.
Absolutely a fair criticism. I'm surprised anyone was baffled that the criminal justice system is highly flawed, other than some mathematicians who lead quite prestigious lives. I've long thought it's pretty suspect that there's a morality bar forming (or that has formed) in programming. A force beyond laws can make you unhireable, what a precedent! Really, I find the idea that we might deny entry to programming even for convicted felons that have served their time problematic. Their insights might be quite useful here.
Edit: Changed "Americans" to "people"
Why is this targeting Americans? In my home country these same biases exist. There's a "group of people" that are deemed more likely prone to criminal behavior.
For some reason America seems to serve as a lightning rod for these types of criticisms, as if the rest of the world is better somehow.
That's to say, I can really only speak in earnest about the devil that I know.
People can imagine this. They understand it. They just implicitly recognize that humans are rationalization machines, that their reasons don't matter. Every single person in human history, no matter how abhorrent, had their reasons, had a whole litany of reasons that they could recite. Plenty of them were able to convince other people they were right, too. That's how you get everything from mass surveillance to genocides.
Now, it's part of being human that we recognize that in extraordinary circumstances, maybe those exigencies can justify or ameliorate your wrongs. But every single person in human history is going to be able to explain to you why that applies to their case, that they had to do it.
But the fact remains nearly all of them didn't.
With concerns of systemic racial bias such a hot topic today, the subject of hidden bias in machine learning systems driven by "big data" has been a frequent question. An article "The Dark Past of Algorithms That Associate Appearance and Criminality", by Catherine Stinson, appearing in the January-February 2021 issue of American Scientist, attempts to make the point that "Machine learning that links personality and physical traits warrants critical review", and largely succeeds at this task. However, most of the piece attempts to provide that critical review, and at this task it fails.
The article starts off offering phrenology as a close cousin to contemporary efforts to use facial recognition to identify criminals, or as "gaydar". A series of examples of it (mis)use in Chinese surveillance, or to monitor schoolchildren to ensure they pay attention in class, capping this off with the claim that Black students have had to endure bright lights pointed at their faces to enable such systems to work. Such applications do seem unfair, and potentially so disruptive as to be counterproductive. So in this way the author is successful in showing a need for critical review.
Stinson goes on to discuss how some of the derided foundations of phrenology - the idea that particular cognitive functions can each be seated in a particular locale within the brain - has now become "a standard assumption in mainstream neuroscience". However, she then claims that despite this, the science has suffered from sloppy methods. One example study is cited that looked for criminality based on photographic input had biased training data by using police photos to represent the convicts, versus professional photos scraped from the web for nonconvicts. It's easy to see how this would skew the results.
The author first claims, "using pictures of convicted criminals, but not of the ones who were cleared, introduces a statistical bias that skews the results", and that seems a fair point. She also says, "the researchers also admitted that taking court convictions as synonymous with criminality was a 'serious oversight'", but offers no explanation for why that should be. My best guess is that we should be able to assume that a "guilty" verdict is the best judgment we can get as to whether someone is guilty of a given crime; but that other people who are guilty of a crime but never caught will be left out of the data set, so our training would bias the system to give a pass to people who don't get caught. I suppose that's fair, too.
It's at this point that I think things start to derail. Stinson goes on to say,
their framing suggests that criminality is an innate characteristic, rather than a response to social conditions such as poverty or abuse, or a label applied to exert social control. […] Part of what makes the data set questionable on empirical grounds is that who gets labeled “criminal” is hardly value-neutral.
That's largely a philosophical question that she doesn't attempt to back up. But the fact that she does make this point, about extrinsic forces being an important contributor, actually defeats her later argument.
Along the way, she feels the need to throw in the standard anti-HBD argument that "psychologists who studied the heritability of intelligence, such as Cyril Burt and Philippe Rushton, had to play fast and loose with their data to make it look like they had found genuine connections between skull size, race, and IQ." I don't intend to debate that here, there's ample material out there showing the correlation between race and IQ (although I don't know about skull size). Either way, it doesn't seem to be relevant to her actual argument about revelations of criminality from facial characteristics.
Her main argument starts
<...
Associations between appearance (as expressed genetically) and that person's capacity for something (intellectually, morally, or otherwise) is plain old eugenics. Given it's sordid history and it's horrible effects on the 20th century, we should be banning this sort of work.