I really hope that at some point, the world starts talking about how Mohammed and Fatima ended up there. This is a part of what happens when you spend two decades bombing villages.
Most of Europe does now. Governments aren't keeping the locals or other foreigners safe because of blatantly racist policies. But it's wrong to simply say 'anti-immigration', it's for a more screened immigration process for places from where people are seen to have trouble fitting in. Almost everybody (95%+) is totally accepting of people who (intend to, even) speak the language and accept the laws, regardless of what they look like. Moreover, the sentiment is shared by most demographics. They respect their homes and don't want abusers here simply because of some skin-color in common. They love their children and don't want them stolen and prostituted.
The UK has actual rape syndicates now. Thousands of people running human trafficking rings. Left almost unmolested (hah!) and undocumented because of the origin of the predators. This is fact. The people on the ground just want their kids to be safe but by reporting the crimes they're accused of being racist. It's really a religion problem, and only the sort that comes from a few of the worst countries, but racist is a better slander word and costs the accuser nothing (ie, burden of proof) in today's climate.
Worse, the government that fails to say 'Muslim Pakistani males' is willing to say 'South Asian rape gangs' which slanders a lot more innocent people while totally denying the problem. This lie makes Qatari men (also Muslim) look bad despite generally being as civilized as anyone, as well as non-Muslim Indians and Balinese, etc. Literally 1/3rd of the world, to avoid naming and fixing the problem.
And then there's the "criticism of islam" thing. Nobody is willing to accept more of that since the recent attacks in fall 2020.
Americans just try to see everything through a racist lens.
So you think bigotry based on someone's religion shouldn't be considered racism. You may be technically right, but that's a bit of a strange hill to die on.
And yes, it is bigotry. Islam may be awful, but so is Christianity. Fortunately, people tend to cherry-pick from their faith and interpret it in a way that is actually compatible with a free modern society.
Right, race and religion are not the same. And no, it's the hill from which to not die. How are you doing over there on rocky Mt Everyone-is-Identical?
My discrimination is your bigotry when you're trying to make me the bad guy. But yes, I do reserve the right to judge things for myself. You obviously do to, and have lashed into my views simply because they differ, not because you've shown them to be incorrect.
In an area where a vast majority of a class of criminals share a combination of home country and religion it makes perfect sense to want to screen them more thoroughly, because the only other option is every citizen having to be race-conscious when hiring men with access to children.
To make this specific situation worse, the governments refusal to acknowledge it means that the criminals aren't getting an appropriate conspiracy charge because that would have to acknowledge the gang nature which would then demonstrate the race (if not religion) and as such, they face minor charges (ex 3y for kidnapping and gang rape) that don't actually stop the problem and then release them back into their communities where they poison others. A gang charge would also prevent them associating with their past conspirators.
> Islam may be awful, but so is Christianity.
One, your moral relativism is disgusting. There are obvious differences between all religions and trying to one-size-fits-all totally craps on people whose faiths aren't violent or abhorrent, or whose churches have practiced differently for long enough to be recognized.
Two, I'm not making the claim that either is bad or good, but that islam and pakistan together are a huge red flag. Whether that means Pakistani islam is different is irrelevant in the context of protecting children.
> Fortunately, people tend to cherry-pick from their faith and interpret it in a way that is actually compatible with a free modern society.
Yes, fortunately. But what do we do about those who don't? Close our eyes and pretend they do?
I mean, the plight of refugees is certainly a good reason not to mistreat and discriminate against immigrants, but it doesn't matter how they ended up there, you still shouldn't be discriminating against them because of their nationality, their religion, their names.
No one deserves to be the target of racism, no matter how bad or good their circumstances are. If someone has done something worthy of derision, then justly revile them for that, but leave their race, ethnicity, religion, sex, gender, orientation, disability, etc out of it.
Their numbers will be higher because they've been immigrating since the 60's.[1] In the last decade, Morocco isn't even in the top 10.[2] Most of the asylum seekers are coming from Syria.[3]
The article is a lot better than the headline lets on. In particular:
> Crucially, she said, we should place blame on the human individuals behind the creation and use of the algorithm rather than reify the technology as being the main driver.
> “Systems and algorithms are human-made, and do exactly what they’ve been instructed to do,” she said. “They can act as an easy and sort of cowardly way to take the blame off yourself. What’s important to understand is that often algorithms can use historical data without any proper interpretation of the context surrounding that data. With that pattern, you can only expect social problems like institutional racism to increase. The result is a sort of feedback loop that increases social problems that already exist.”
As someone in an adjacent institution this affair has been frustrating to watch. In my opinion black-box algorithms have no place in government, and for this reason it's critical for the government to develop the expertise to develop and understand models themselves, so they aren't reliant on the trade secrets of contractors.
Interestingly though the administration resigned over this affair, it does not look like they'll suffer much over it in the upcoming election this wednesday. The responsible prime minister is likely to get a fifth term if things go as the polls are predicting.
> In my opinion black-box algorithms have no place in government, and for this reason it's critical for the government to develop the expertise to develop and understand models themselves, so they aren't reliant on the trade secrets of contractors.
We learned from 2000 years of human history - war, disease etc. - to create a transparent system of checks and balances. These days we are trying to pretty much remove all of it (digitally) for "convenience" and "efficiency".
I can not understand how any semi-decent societal system can accept a situation of opaqueness and no accountability.
> These days we are trying to pretty much remove all of it (digitally) for "convenience" and "efficiency".
Well, the same 'we' that push voting machines and other crap like locking government records up in proprietary software.
I push for algorithms specifically because they're (as a written document) transparent. If a human has to judge a historical context you're never going to get repeatable results. But I want the system to be able to diagram, with weights, all the inputs that went into something before I'll trust an automated version.
> I can not understand how any semi-decent societal system can accept a situation of opaqueness and no accountability.
Part of it comes down to an effect that's been known for a long time with human-computer interaction.
Whenever you have a machine in the decision loop of a job, many workers WILL DEFER to the machine whenever there's a question of whether something is wrong or right.
By deferring to the machine, the workers absolve themselves of accountability and shift all responsibility to "the system". I think this is often deliberately "baked in" by management to customer service of all kinds. You can see this in a small way if you've ever had to call an internet service provider in the USA to deal with a billing mistake.
I don't think "black-box" algorithms are the problem per se. In this case the system was doomed from the get-go by using biased and flawed input attributes.
What I find very problematic in the use of automated systems for fraud detection is taking the system decision as final and not offering an easy and clear way of appealing decisions.
> you can only expect social problems like institutional racism to increase. The result is a sort of feedback loop
I dislike how people just carelessly make these claims without justifying them. For instance, another (far more likely imo) possibility is that it converges into certain state.
"inherently discriminatory because they took variables such as whether someone had a second nationality into account".
Which is not necessarily wrong. If people with a second nationality are statistically more prone to cheating the system this is a valid variable to determine the probability that they actually cheat and not made a mistake.
What would be wrong though is to blindly take that probability for the fact.
But it is really hard to say what actually is going on there cause the article is full of emotions and carries close to zero information.
Even if being an immigrant is correlated with higher rates of some crime, an algorithm cannot take that into account without bigotry. Imagine being an innocent immigrant. If the algorithm tags you as guilty, no amount of explanation about averages or neural nets are going to make you feel better about your erroneous charge. This is the reason “stop and frisk” was such a huge deal: it let the cops “racial profile” for drugs, meaning that at any time all black and brown people were more likely to experience 4th amendment violations by the cops.
Our language around probability doesn’t help either. We say things like “being an immigrant makes it /more likely/ that you will take some action”. There might be a correlation in aggregate data but in any individual case we can only look at the relevant facts. Think again about being an innocent immigrant. Would you accept someone saying that /you personally/ were more likely to have committed a crime? Of course not!
The moral of the story is that stats describing a cohort cannot be projected onto individuals in that cohort. The data flows the other way; individuals comprise an average, no individual /conforms/ to an average. They may happen to coincidentally be average, but no causation has occurred.
Our justice system is supposed to be “innocent until proven guilty”, not “innocent until Bayesian inference suggests high enough probability.”
"What would be wrong though is to blindly take that probability for the fact."
And it's right: your algorithms can use all sorts of indicators to flag suspicious cases, but in the end the evaluation needs to be done by a person on a case by case basis. As long as the investigation on the single case is performed "silently" (i.e. without causing any loss of time or money or any annoyance to the person who is investigated) I don't see the issue. Otherwise it's the same as saying you cannot sort rows in an excel sheet according to some criterion and start looking at the data from where it looks more promising.
I still think racial profiling to flag suspicious cases is an example of discrimination. That's because the system more stringently checks people in one racial group for wrongdoing, which results in more of those people in the criminal justice pipeline.
As a concrete example of where this can go wrong, let's say that (hypothetically) 1% of people commit fraud, and that this prevalence is not correlated with race. But, due to past discriminatory practices, more people of one race than another are convicted of fraud. Then, this algorithm would perpetuate this discrimination, since, say, it would identify 80% of fraud in the discriminated group but only 50% of the non-discriminated group.
Correct, and algorithm shouldn't use a protected class as an input. But at the same time, simply pointing out that a inequitable outcome is produced is not evidence of bias.
Also, dual citizenship is not a protected class last time I checked.
A model that does not incorporate all the relevant information will have more false positives. The practical consequence of doing what you suggest is, cetaris-paribus, that you will incorrectly penalize more people.
> There might be a correlation in aggregate data but in any individual case we can only look at the relevant facts.
Definitely. Thats where courts and human review come in. The mistake the Dutch made here was not the variables in the model, but that they had limited human review of a model with (I'm assuming) low predictive power.
> Our justice system is supposed to be “innocent until proven guilty”, not “innocent until Bayesian inference suggests high enough probability.”
"Guilty beyond a reasonable doubt" (or whatever the Dutch equivalent is) implies exactly the Bayesian calculation you are rejecting. That said, I agree that a machine learning model with limited inputs is not sufficient to determine guilt.
> Even if being an immigrant is correlated with higher rates of some crime, an algorithm cannot take that into account without bigotry. Imagine being an innocent immigrant.
That's insane. Men commit 99% of violent rape, imagine not being able to say that. And I say that as a man, innocent of rape.
Similarly, if people with dual-citizenship smuggle more you're wasting time by treating everyone equally when looking for illegal imports.
> If the algorithm tags you as guilty, no amount of explanation about averages or neural nets are going to make you feel better about your erroneous charge.
Yes, any black-box trial is crap. But a transparent algorithm based on real data that a certain subset of the population is more likely to commit a crime, which is used for proper planning and investigation, is not.
> Our language around probability doesn’t help either. We say things like “being an immigrant makes it /more likely/ that you will take some action”. There might be a correlation in aggregate data but in any individual case we can only look at the relevant facts. Think again about being an innocent immigrant. Would you accept someone saying that /you personally/ were more likely to have committed a crime? Of course not!
You have one proper point about terminology and then an appeal to rage.
You are right that nothing my demographic does (men, raping) makes me more likely to rape. But you're wrong later where you say "[claimed] /you personally/ were more likely to have" because, yes I (by the nature of being physically capable of rape) am more likely to have committed it than other people. To me I'm not a statistic, to you I literally am a population sampling.
> Would you accept someone saying ...
If they're right, yes. Otherwise you're just saying that my outrage should trump the truth.
If a woman wants to organize a women's only bus, or hotel room, because she fears the harm I as a man could commit I shouldn't have the right to force myself on her.
> This is the reason “stop and frisk” was such a huge deal: it let the cops “racial profile” for drugs, meaning that at any time all black and brown people were more likely to experience 4th amendment violations by the cops.
No, the problem with stop and frisk is that it was (allegedly, I'm not from there) used by racists to target black people. If it was used as commanded by an algorithm then it would only happen where data showed an actual correlation.
If you look at the demographics of people convicted after being flagged by a flawed algorithm, you'll find a lot of correlelations. Pick a totally spurious indicator. Whether you live in a house with an odd or even number. If the algorithm decides odd numbers are highly indicative of fraud, and those are 90% of cases that get investigated, then you're going to have 90% of convictions happen to have odd house numbers. "See, look, the algorithm works!"
While plausible, there are a lot of 'if's that underpin this scenario. In particular, you need people to not notice that the conviction rate is 50% for both odd and even numbered households.
No, that implies the algorithm is wrong and the chance of crime is equal. If there was an actual bias, and you searched 90% of homes based on that, you'd expect closer to 100% of the arrests to be in those homes.
As far as I know the Netherlands doesn't actually allow for dual citizenship, with few exceptions - notably Moroccans, who are not allowed to give up their Moroccan citizenship.
In this sense, dual citizenship is basically code for being Moroccan.
Some of the major issues with this come in with omitted variables. For example, suppose for a second that low wage earners cheat the system more. Suppose that race is correlated with wage for historical reasons. Suppose that at each given wage level race has nothing to do with cheating the system. If you don't include wage data in your system, you'll just see race and fraud connected. Now you've created a tragic feedback loop you didn't need to create.
What if somehow we as a society magically get equality overnight and there's no more connection between race and wage? But you use historical data where they were connected, still excluding wage data, and so even now your model connects race and fraud.
In this hypothetical example, I'd argue it would be inethical to include race without wage. Hell, wage is only going to correlate with fraud via yet other variables. Maybe an ethical system needs to go out and gather those measurements too.
(side note, in an ML system with regularization, even if you do include wage in your data, the regularization might pin some fraud on race anyways.)
The solution is to acknowledge that there are always going to be omitted variables and either a) be extremely careful and rigorous in your data gathering, design, and roll out or b) don't try to automate this thing, leaving it the slow expensive way where people can gather facts as needed, case by case.
This paragraph, holy shit - I have to wonder if whoever wrote it(the algorithm) can look at themselves in the mirror. Or if they just think it's all for the greater good and it's mostly targetting unwashed masses so it's all fine.
"In one of the more egregious examples of the lack of humanity in the authorities’ approach, a report from Trouw revealed that the tax office had baselessly applied the mathematical Pareto principle to their punishments, assuming without evidence that 80 percent of the parents investigated for fraud were guilty and 20 percent were innocent. "
Isnt that also just not how the Pareto principle works? Doesn’t it say that 20% of inputs (people accused of fraud) are responsible for 80% of outputs (cases of fraud)?
So this is actually exactly backwards. On top of being frighteningly amoral, I mean.
I checked the original article, it doesn't explicitly refer to the Pareto principle, it just calls it the 80/20 rule. Normally by this it's understood that you should be referring to the Pareto principle, but it seems they've invented another 80/20 rule (that makes no sense and is not related to Pareto).
Yeah if that's really what they're doing, they've misunderstood the pareto principle pretty much on every level, from what it is to how someone might attempt to apply it.
I'm Dutch and am following this scandal for quite some time. This article strikes me as being odd.
Yes, computers were used in the decision process and no I don't believe them to be algorithms.
It was specifically coordinated and covered up from the top. As your highlight shows, it was specifically designed this way. Which means it was a business process.
Sounds like what they had was a tool that used correlation to bucket people based on statistics with increased risk of fraud, and for some reason they were using it as proof of fraud.
Yes, this is a massive scandal, but why is every article about it focused on spooky *ALGORITHMS* (I have never seen the word used as much as I have in the articles covering this story) instead of the clearly massive fault of the people running this.
Just to get you an idea of how wrongheaded this entire thing was, I have a personal anecdote relating to this shitstorm.
Back in autumn 2013 I got a letter stating that I had received too much in child care benefits, with a request to pay it back. On the back of said letter was the text 'If you are still entitled to child care benefits and want to settle this amount against upcoming entitlements, you do not have to do anything'. Given as I was still entitled to benefits, I proceeded to not do anything. A few weeks later I got a reminder, once again stating that if I wanted to settle, I wouldn't have to do a thing. The reminder was followed by a final notice with opportunity to pay, once again with the exact same text on the back of the letter. So, obviously, I once again did nothing.
Then we got a 'Dwangbevel in naam des konings', i.e. a writ of execution in the name of the king. Red envelope. No added text. So I had to pay up. Now, as you may well guess, the tax service had already also started settling against the benefits that I was still entitled to! This prompted me to write a strongly worded letter, asking the tax service in no unclear language what I should have done differently, and requesting them to not dock me for that particular money twice. Literally the only reaction to that letter from the tax service was in silently restituting to me the double payment.
Now, I'm a reasonably well paid and well educated software developer, who just happens to hold two nationalities. I am perfectly sure that I would not even have had a reminder had I just had the Dutch nationality.
I could have paid in the first place and not settled against upcoming benefits; I was just lazy. However, imagine you're scraping by on minimum wage and are put in such a situation. There is no other way you could have acted than I did, but you also would not have been able to pay up to the writ of execution. And there would have been no recourse!
Add to that the fact that in many cases, the people hadn't even actually been paid too much, as I had, but merely been tagged as possibly fraudulent and put in to the system for reclaiming paid benefits, so the government could take a stance of being hard against benefits fraud while figuring out if any fraud had taken place in the first place.
Adding this to my running list of algorithms encoding and amplifying systemic bias, like:
* A hospital AI algorithm discriminating against Black people when providing additional healthcare outreach by amplifying racism already in the system. https://www.nature.com/articles/d41586-019-03228-6
* Misdiagnosing people of African decent with genomic variants misclassified as pathogenic due to most of our reference data coming from European/white males. https://www.nejm.org/doi/full/10.1056/NEJMsa1507092
* When AI criminal risk prediction software used by judges in deciding the severity of punishment for those convicted predicts a higher chance of future offence for a young, Black first time offender than for an older white repeat felon. https://www.propublica.org/article/machine-bias-risk-assessm...
And here's some good news though:
* When police wrongfully arrested a person based on faulty facial recognition match using grainy security camera footage, without any due diligence, asking for an alibi, or any other investigation. https://www.npr.org/2020/06/24/882683463/the-computer-got-it...
Maybe you should make multiple lists. There are some overt racist things, like redlining was, and gerrymandering can be, but a lot of those things appear to be natural accidents, or just looking at people without race before finding some physical (sickle-cell anemia) reason to.
For instance:
> * Misdiagnosing people of African decent with genomic variants misclassified as pathogenic due to most of our reference data coming from European/white males.
It's obviously caused by studying the people who present, and then demographics changing. Nobody made a decision here with any ill intent. Both the hospital and the insurance company share the patient's interest in them getting better and are already looking for better data.
> * When AI criminal risk prediction software used by judges in deciding the severity of punishment for those convicted predicts a higher chance of future offence for a young, Black first time offender than for an older white repeat felon.
A higher chance of any offense, or of being a burden to society? Kids are more likely to commit smaller crimes, and those are a gateway to larger crimes that then put you away until you're a hardened repeat felon. If black kids were being enticed into crime we certainly want to know about it. If it's a subpopulation rather than an area the causes are likely different and good decisions only come from good data.
I'm not sure how you think my list would bifurcate into multiple lists.
> It's obviously caused by studying the people who present, and then demographics changing. Nobody made a decision here with any ill intent. Both the hospital and the insurance company share the patient's interest in them getting better and are already looking for better data.
Not a single example I gave was the result of ill intent. That's literally the point of my list. There's a difference between systemic bias and intentional bias. These are examples of systemic bias.
> A higher chance of any offense, or of being a burden to society?
I provided the link which answers your question. In this case, it predicted a higher likelihood of future crime for a kid who attempted to steal someone's bike and scooter, than for an adult who had shoplifted, been previously convicted of armed robbery, and served 5 years in prison already.
> Kids are more likely to commit smaller crimes, and those are a gateway to larger crimes that then put you away until you're a hardened repeat felon.
Your logic seems to acknowledge that one of the large drivers of crime is being "put away" in prison. In this case, the algorithm predicted a higher likelihood of committing future crimes for a kid than for someone who had already served 5 years in prison for armed robbery. Even worse is that the algorithm was being used to determine the severity of their respective punishments. So, the act of predicting a higher likelihood of future crime for the kid becomes a self-fulfilling prophecy, giving her a harsher sentence, which in turn is more likely to drive her toward future crime. This is systemic bias in action.
> I'm not sure how you think my list would bifurcate into multiple lists.
Because the two things I pulled from your list aren't systematic. One was local to a specific health system and about a specific condition, and the other about kids from a certain area. Neither was just about black people in general. Your list is predominantly racial and I thought that was the point. This medical thing is really about any group not being recognized as different enough from the whole, women get this a lot, but even men do in areas where women are the primary sufferers of something. (Breast-cancer for instance.)
Also, to distinguish things that are intentional (redlining) vs absolutely unforeseen but still problematic. A few absolutely undeniable things, and they exist, would make the list more meaningful and perhaps give more weight to the rest which are a bit ambiguous.
> There's a difference between systemic bias and intentional bias. These are examples of systemic bias.
No, they're an example of non-systemic bias. The aren't broadly across any system, or symptomatic of a general attitude. Systematic bias doesn't mean that the majority gets more attention, it means that the entire system is biased against something, and you aren't showing that.
> Your logic seems to acknowledge that one of the large drivers of crime is being "put away" in prison.
It is a separate topic, but yes. Being put away unjustly, and in the wrong place or severity for your crime hurts. But not serving a sentence for crime is just as bad, if not worse. I'm not against jailing criminals, just doing it where it doesn't serve us or them.
> In this case, the algorithm predicted a higher likelihood of committing future crimes for a kid than for someone who had already served 5 years in prison for armed robbery.
Could be. Without knowing the severity of the crime predicted though it's kind of meaningless.
> Even worse is that the algorithm was being used to determine the severity of their respective punishments.
Well, without knowing the prediction you can't know if a stiffer sentence would have encouraged or deterred future crime. If they were predicted to be killers, a hard but fair sentence at the first crimes could turn them around. If they're predicted to steal a few cars then even a year in prison is probably going to make them worse.
> So, the act of predicting a higher likelihood of future crime for the kid becomes a self-fulfilling prophecy, giving her a harsher sentence, which in turn is more likely to drive her toward future crime. This is systemic bias in action.
It would be, if it were system wide. It was a specific population though, and there's still no indicator that it said black kids specifically other than because in those neighborhoods they were the ones at risk. fwiw, stealing scooters from children doesn't seem like the path to good behavior.
Systematic risk is AIs for this purpose in regard to anyone, not that they targeted black people more in some cases. That's essentially random.
> Your list is predominantly racial and I thought that was the point.
Ah, no, that wasn't the point. They were just examples of systemic bias. The fact that so many are racial is just a reflection of the unfortunate state we find ourselves in.
> A few absolutely undeniable things, and they exist, would make the list more meaningful and perhaps give more weight to the rest which are a bit ambiguous.
I honestly don't know how any of the examples posted are deniable, they're all well founded with research and evidence. But you're right that adding more to the list can only help.
> Systematic bias doesn't mean that the majority gets more attention, it means that the entire system is biased against something, and you aren't showing that.
That's not what systemic bias means.
"Systemic bias, also called institutional bias, and related to structural bias, is the inherent tendency of a process to support particular outcomes."
Also, examples are evidence which support a theory (of systemic bias in this case). Any one data point is rarely intended to be absolute proof of that theory, much less a description of its scope in its entirety. All of these examples do indeed provide evidence toward the existence of systemic bias within systems without fully describing or proving the extent of the entire scope of systemic bias within those systems.
> But not serving a sentence for crime is just as bad, if not worse.
The article I linked explains that this is not what the algorithm was used for. It was not used to decide whether or not to sentence them, it was used to help decide the severity of their sentence.
> If they were predicted to be killers, a hard but fair sentence at the first crimes could turn them around. If they're predicted to steal a few cars then even a year in prison is probably going to make them worse.
You seem to be agreeing that this was in fact a badly biased algorithm, since it was recommending a stiffer punishment for a kid who attempted to steal a bike than for a repeat felon, previously convicted of armed robbery, who shoplifted.
> It would be, if it were system wide. It was a specific population though, and there's still no indicator that it said black kids specifically other than because in those neighborhoods they were the ones at risk. fwiw, stealing scooters from children doesn't seem like the path to good behavior.
If something affects a population, that is system-wide by definition, since the definition of "system" is pretty broad; see the description above. Regardless of whether or not you think stealing scooters is a good prediction, the fact here is that it rated the likelihood of future crime as higher than for someone who was already a career criminal.
> Systematic risk is AIs for this purpose in regard to anyone, not that they targeted black people more in some cases. That's essentially random.
If only. Unfortunately, this is seen over and over again, more so than would happen by randomness. That's the point of the list.
> They were just examples of systemic bias. The fact that so many are racial is just a reflection of the unfortunate state we find ourselves in.
Do you think racial cases of bias outweigh other demographic biases? And what's your opinion of the general level of intent (or lack of, to fix) relative to other issues?
The medical one made me think about the similar case of differing heart-attack symptoms in men and women. In this case actually because doctors just focused on men.
> That's not what systemic bias means.
> "Systemic bias, also called institutional bias, and related to structural bias, is the inherent tendency of a process to support particular outcomes."
That just shifts the definitional question to the scale of the institution, structure, or process.
At one end, if everyone follows the same rules it's a system, but what about if one clinic got bad data, do we write that up as a systematic failure? That's okay if it's your definition, but cases at the other end seem a lot more important and a lot more amenable to systematic fixes.
> If something affects a population, that is system-wide by definition, since the definition of "system" is pretty broad
Is the definition, "a system is broad" or (I think you mean) "broad as in flexible", such that a system can be anything? Any micro or macro population?
I guess where I was going with this is that the list seemed to be about some things that are very broadly impactful but not specifically racist, like facial-recognition not recognizing anyone but white men very well, but then a related issue of how this is encoded in the system. Is usage required, or was it random, etc.
Such that maybe top-to-bottom it'd be sorted by breadth. How many people does each impact, and how deeply-mandated or intertwined are the issues. And then different populations and problem to both give contrast, but also to indicate (when complete) the demographics of the problems.
So I was wondering if you'd focused on black issues to make the list.
> Unfortunately, this is seen over and over again, more so than would happen by randomness. That's the point of the list.
So, statistically, how much subpopulation misrepresentation would you expect in various ways? And are you saying there's more in general, or more racial, than expected?
I interpreted the one about kidneys as an error because of changing demographics and how people had caught it, made note that this sort of thing happens, and started checking other research for similar sampling bias. It read as a success story where the one about facial-recognition was more actively black mirror.
> You seem to be agreeing that this was in fact a badly biased algorithm
No. But not a good one either. More that I'm wondering if it was an attempt to model, like for predictive policing, or a tool sold to simplify the sorting of people? Because models are good, even when they're wrong, but crappy predictive tools are worse than useless and - where I was ultimately going with this - perhaps fraudulent to sell.
The 'good' case would be if someone built a criminality model and the city was trying to work with police and communities to intervene in a predicted pattern. It's not unreasonable that the societal harm from a non-criminal becoming criminal could be worse than an existing criminal remaining that way. So modeling and discussing this isn't bad, even if the data has a racial component and some of the questions are of bias.
> Do you think racial cases of bias outweigh other demographic biases? And what's your opinion of the general level of intent (or lack of, to fix) relative to other issues?
When talking about systemic bias, I think it's less about intent, but that is one of the things that makes it so dangerous in terms of the level and longevity of impact; it's much more difficult to fix when there is no single bad actor at which to point the finger. This makes it easier for people to deny, either outright or at least in extent, and makes it harder for the systems to be fixed.
I don't know that I could place one type of demographic bias over another, but I think it's better for people to understand and acknowledge systemic bias of any form, if we have any hope to rectify any of them. I also think a lot of them are woefully entangled. For example, I've heard the argument that some bias isn't racial, it's economic, as in, a bias against poor people not people of color. But when there are already so many systemic economic biases against people of color, a lot of times it ends up being a distinction without a difference. I have plenty of examples of this, but don't want to dilute the impact of that last thought by going into detail.
> That just shifts the definitional question to the scale of the institution, structure, or process.
I agree. Systemic bias is everywhere, for almost all scales of systems.
> Is the definition, "a system is broad" or (I think you mean) "broad as in flexible", such that a system can be anything? Any micro or macro population?
Yes, you're right, I meant flexible. We can debate the severity of the impact of some form of systemic bias based on the size of the system (and the population or sub-population impacted), but to the individuals who are the victims of systemic bias, the result ends up being the same. At the end of the day, I don't think I'd fault any individual for taking up the cause of identifying and rectifying bias at any scale.
> I guess where I was going with this is that the list seemed to be about some things that are very broadly impactful but not specifically racist, like facial-recognition not recognizing anyone but white men very well, but then a related issue of how this is encoded in the system. Is usage required, or was it random, etc.
Yeah, I didn't really provide much context for why I specifically keep a list. It's not intended to single out racism, and in fact, it is intended to focus on unintentional codification of bias, because I think there are more software developers out there who are at risk for unintentionally codifying bias into their algorithms than there are those who may intentionally encode it.
The reason I keep the list is as a reminder of the level of impact you can unintentionally have in the systems you build without extremely deep thought and broad context. It's a reminder that what we do can have real impact on real people in ways we never imagined.
This is especially recurring as a theme for machine learning and AI systems which are taught based on limited datasets which often systemically under-represent those to whom it will end up being applied.
> Such that maybe top-to-bottom it'd be sorted by breadth. How many people does each impact, and how deeply-mandated or intertwined are the issues. And then different populations and problem to both give contrast, but also to indicate (when complete) the demographics of the problems.
I think this is a good idea.
> So I was wondering if you'd focused on black issues to make the list.
I did not specifically focus on that. I started tracking the list I think 2 or 3 years ago, and I think issues of systemic racism have been especially visible in the mainstream in this time period for reasons I won't get into. So, it's largely a reflection of the systems I've been made aware of through research and publication.
> So, statistically, how much subpopulation misre...
> When talking about systemic bias, I think it's less about intent, but that is one of the things that makes it so dangerous in terms of the level and longevity of impact; it's much more difficult to fix when there is no single bad actor at which to point the finger.
Thanks for approaching it this way. I think we often (societally, looking for blood) want to find a bad actor and are intentionally blind to bad things that just sort of happen at the edges, but need oversight to find and fix.
> For example, I've heard the argument that some bias isn't racial, it's economic, as in, a bias against poor people not people of color. But when there are already so many systemic economic biases against people of color, a lot of times it ends up being a distinction without a difference.
It doesn't help the sufferer, at the time, to be told that someone else would be suffering equally, but it does help fix the problem I think, to realize that it's circular via poverty or whatever, not simply racial, because in some areas and at some times it has simply been racial and it hugely changes how you fix it. If it's overt you can't just offer change, you have to prevent further damage during the repair process.
> The reason I keep the list is as a reminder of the level of impact you can unintentionally have in the systems you build without extremely deep thought and broad context.
Have you ever read comp.risks? I really like it as a source of Therac-25 type stories (across all fields) that engineering types should think about when building things.
> I just know that I've not yet been able to find any real systemic bias, at least in the US, against rich, caucasian males.
Is Twitter not a system? :D
It gets a bit fuzzy with bias against the majority. Every model that isn't right disadvantages everyone and the majority is part of everyone. So bad drug laws impact white people too. But because actual race is only encoded in one direction (affirmative action, "positive" directions) then anything that impacts white people also impacts everyone, whereas there are often specific laws (such as for constructing "The Projects" in the first place) that do directly exclude whites from the harm they caused. So subgroups definitely experience more exclusive problems, even aside from the amount of problem.
> how difficult these things are to rectify at a systemic level, since the vast majority of doctors using the still-biased eGFR formula have no idea that it has this problem.
I think it's just that the story is best told from that moment. That's the OMG. From there it improves, and I'm sure they sent a copy of the report worldwide asap. But no solution is ever 100% so there's no wrap-up party and it will never look done and solved. (Even one doctor who didn't check their email...)
> Fraud generally requires mal-intent, which most of these models and products didn't have, at least if we're being generous and optimistic.
Probably, but if they're saying "our product does X" maybe there's something to grab onto and investigate. Maybe they did misrepresent it.
> I just finished reading a book called, Weapons of Math Destruction, which talks about the damage models can do. The author posits a set of tests to tell whether the model is beneficial or destructive. One of the hallmarks, the author argues, of a good predictive modelling system is one which includes feedback into the system as a result of its predictions.
Good point, and thanks for the book recommendation.
A lot of things aren't amenable to that though, because the hypothetical city/community meeting can't take years to watch the outcomes and train continue to build a model, they've got to work from historical data up to that point and make policy decisions in the meeting.
> One of the other primary tests is dependent on the context of how the model is used, so it's not really reasonable to...
> When AI criminal risk prediction software used by judges in deciding the severity of punishment for those convicted predicts a higher chance of future offence for a young, Black first time offender than for an older white repeat felon.
Younger people are more likely to re-offend than older people. Remove race from the situation entirely, and this is still the expected result. There's zero reason to think this algorithm was based with respect to race.
If you read deeper than a one line description you'll see:
1. Even after accounting for criminal history, recidivism, age and gender, black defendants were still scored as much more likely to re-offend.
2. It incorrectly predicted that black defendants would re-offend much more frequently than white defendants.
3. It incorrectly predicted that white defendants would not re-offend much more frequently than black defendants.
Considering those three things and that they have refused to give any details about how the algorithm works, I see zero reason to give them the benefit of the doubt. Fire them until they can demonstrate that the algorithm is not in fact biased like it appears to be.
The score also takes into account answers to questions like whether the defendant's parents were separated or whether their parents were ever arrested. Those things are completely out of the defendant's control and are highly correlated with race. Even including those things in their score is damning.
Phrasing it in the way you do is misleading. The defendants labeled as high risk and low risk were just as likely to re-offend. To put this in simpler numbers
* Out of 100 white people, 5 were labeled high risk.
* Out of 100 black people, 20 were labeled high risk.
* Out of the 5 white people labeled high risk, 4 re-offended.
* Out of 20 black people labeled high risk, 16 re-offended.
In either case, someone labeled high risk had the same likelihood to re-offend: 80%.
"It incorrectly predicted that black defendants would re-offend more frequently than white defendants." This is technically correct, but not because the algorithm was bad a predicting rates if re-offending. It's because there was higher rates of re-offending. The likelihood of re-offending among someone labeled high risk is the same.
This kind of objection seems like a blanket rejection of any system that produces an inequitable outcome. But the reality is that rates of re-offending is not equal. Even a perfectly accurate prediction of re-offense is going to predict higher rates of re-offending among men. Because men re-offend at higher rates. This isn't sexism.
That doesn't mean we shouldn't recognize the disparate impact of incarceration on underprivileged people. But simply concluding bias due to inequitable outcomes is simplistic.
When you are doing crystal ball voodoo based on stuff that has no connection to the defendant's choices and no direct connection to criminality like whether or not their parents were separated and whether their parents were ever arrested, then you're damn right there should be a blanket rejection of any inequitable outcome.
I'm not even convinced that you should be allowed to make a decision based on stuff like that even if it is somehow equitable.
> no direct connection to criminality like whether or not their parents were separated and whether their parents were ever arrested
Again, not right at all. These things heavily correlate with crime. A broken home is the primary indicator of someone's future success, even over a 'better' but broken home.
I'm male, and not a rapist, but having a penis heavily correlates with rape. I suggest you do not pick me, or any male, to watch your children. Sure it's rude to the innocent, but oh well, a little rudeness vs potential harm.
> I'm not even convinced that you should be allowed to make a decision based on stuff like that even if it is somehow equitable.
Then nobody will ever follow the law. If it's illegal for me to reject a potentially bad babysitter for something I know about them that could risk my child's safety I'll happily lie and say I didn't like their haircut.
You'd get further if you tried to identify these people and treat them better - grants to move out of bad neighborhoods, to get educated, to get pardons for unrelated crimes, to get counseling for abuse, etc - than with this "wrong unless it's 100% identical, equity-of-outcome" thing.
We're talking about an algorithm being used as part of the court system to determine whether or not someone rots in jail, not about how you choose your babysitter.
It needs to be held to a higher standard and punish people based on their actions, not based on what their parents did.
No, you're talking about making it illegal to use the things you know about someone or something to make a fully qualified decision. About a babysitter, a potential criminal, an immigrant, whatever.
> It needs to be held to a higher standard and punish people based on their actions, not based on what their parents did.
And that's not what actually happened. A guy got arrested on bad data and then released, we know some kids might be at risk because they're from broken homes, etc. Nobody was jailed for their parent's actions, and nobody even proposed it. Some trends are worrying, but you act like they're the intent of the entire system not just some scammy products that a company is pushing.
Voting machines, for instance, should all be burned. But I understand that most people don't know why and support them for convenience. I think they're anti-democratic but I don't think people are evil for using them. You should try to get a similar perspective.
I am not talking about that. You are misinterpreting my statements. "stuff that has no connection to the defendant's choices". Defendants only exist within the court system.
> Nobody was jailed for their parent's actions
Not jailed, but they are being kept in jail because of them. That is the same thing.
I am not claiming that anyone is evil. Just that this is unjust and that they need to stop doing this. You are putting words in my mouth.
Your analysis above is a prime example of how dangerous statistics can be when not properly considered. Even if we took your numbers above as truth, it actually does not indicate that the system isn't biased. In order to determine that you'd need to know what percentage of each group _not_ labeled as high risk never re-offended. You're only analyzing the true positive rate without taking into account the false negative rate. You'd need to know how many people in each group that weren't labeled went on to re-offend.
Thankfully, the article I linked to looked at both:
White African American
Labeled Higher Risk, But Didn’t Re-Offend 23.5% 44.9%
Labeled Lower Risk, Yet Did Re-Offend 47.7% 28.0%
> Fire them until they can demonstrate that the algorithm is not in fact biased like it appears to be.
What do the non-black box findings from the area show? If blacks are a poor demographic in the area it might be true. (Crime tracks poverty, not race.)
But yeah, certainly don't pay anyone for, or use, a black-box algorithm.
> The score also takes into account answers to questions like whether the defendant's parents were separated or whether their parents were ever arrested. Those things are completely out of the defendant's control and are highly correlated with race. Even including those things in their score is damning.
Nope. That's perfectly fair to look at. For instance, a broken family is another predictor. It's not fair, but being a victim increases your chance to offend. Similarly, the number one predictor of child sex crimes is to have suffered them yourself. If you have a child and are picking a babysitter, skip the one who was molested.
fwiw, those things don't correlate with race, they correlate with poverty which correlates with race. Broken families are more likely to be poor and abusive.
For decades the Netherlands have been helping corporations and the super rich evade taxes at a cost of several hundred billion Euros to the global economy per year.
How petty to now attack the poor in such a cowardly manner...
It sounds to me like the real problem is what appears to be a screening tool being used as evidence of guilt.
For other examples: field drug test kits--they really only say "this might be drugs", but the prosecutors use jail to try to get people to plead guilty rather than promptly doing a proper test.
Also, the ubiquitous breathalyzer--it's actually coded into law as proof in many places but it's not. There is an inherent biological flaw that makes the readings inaccurate.
60 comments
[ 4.6 ms ] story [ 126 ms ] threadI really hope that at some point, the world starts talking about how Mohammed and Fatima ended up there. This is a part of what happens when you spend two decades bombing villages.
The UK has actual rape syndicates now. Thousands of people running human trafficking rings. Left almost unmolested (hah!) and undocumented because of the origin of the predators. This is fact. The people on the ground just want their kids to be safe but by reporting the crimes they're accused of being racist. It's really a religion problem, and only the sort that comes from a few of the worst countries, but racist is a better slander word and costs the accuser nothing (ie, burden of proof) in today's climate.
Worse, the government that fails to say 'Muslim Pakistani males' is willing to say 'South Asian rape gangs' which slanders a lot more innocent people while totally denying the problem. This lie makes Qatari men (also Muslim) look bad despite generally being as civilized as anyone, as well as non-Muslim Indians and Balinese, etc. Literally 1/3rd of the world, to avoid naming and fixing the problem.
And then there's the "criticism of islam" thing. Nobody is willing to accept more of that since the recent attacks in fall 2020.
Americans just try to see everything through a racist lens.
And yes, it is bigotry. Islam may be awful, but so is Christianity. Fortunately, people tend to cherry-pick from their faith and interpret it in a way that is actually compatible with a free modern society.
My discrimination is your bigotry when you're trying to make me the bad guy. But yes, I do reserve the right to judge things for myself. You obviously do to, and have lashed into my views simply because they differ, not because you've shown them to be incorrect.
In an area where a vast majority of a class of criminals share a combination of home country and religion it makes perfect sense to want to screen them more thoroughly, because the only other option is every citizen having to be race-conscious when hiring men with access to children.
To make this specific situation worse, the governments refusal to acknowledge it means that the criminals aren't getting an appropriate conspiracy charge because that would have to acknowledge the gang nature which would then demonstrate the race (if not religion) and as such, they face minor charges (ex 3y for kidnapping and gang rape) that don't actually stop the problem and then release them back into their communities where they poison others. A gang charge would also prevent them associating with their past conspirators.
> Islam may be awful, but so is Christianity.
One, your moral relativism is disgusting. There are obvious differences between all religions and trying to one-size-fits-all totally craps on people whose faiths aren't violent or abhorrent, or whose churches have practiced differently for long enough to be recognized.
Two, I'm not making the claim that either is bad or good, but that islam and pakistan together are a huge red flag. Whether that means Pakistani islam is different is irrelevant in the context of protecting children.
> Fortunately, people tend to cherry-pick from their faith and interpret it in a way that is actually compatible with a free modern society.
Yes, fortunately. But what do we do about those who don't? Close our eyes and pretend they do?
No one deserves to be the target of racism, no matter how bad or good their circumstances are. If someone has done something worthy of derision, then justly revile them for that, but leave their race, ethnicity, religion, sex, gender, orientation, disability, etc out of it.
[1]https://www.iamexpat.nl/expat-info/dutch-expat-news/largest-... [2]https://www.statista.com/statistics/525804/netherlands-large... [3]https://www.cbs.nl/en-gb/news/2019/43/more-syrian-asylum-see...
Two centuries.
> Crucially, she said, we should place blame on the human individuals behind the creation and use of the algorithm rather than reify the technology as being the main driver.
> “Systems and algorithms are human-made, and do exactly what they’ve been instructed to do,” she said. “They can act as an easy and sort of cowardly way to take the blame off yourself. What’s important to understand is that often algorithms can use historical data without any proper interpretation of the context surrounding that data. With that pattern, you can only expect social problems like institutional racism to increase. The result is a sort of feedback loop that increases social problems that already exist.”
As someone in an adjacent institution this affair has been frustrating to watch. In my opinion black-box algorithms have no place in government, and for this reason it's critical for the government to develop the expertise to develop and understand models themselves, so they aren't reliant on the trade secrets of contractors.
Interestingly though the administration resigned over this affair, it does not look like they'll suffer much over it in the upcoming election this wednesday. The responsible prime minister is likely to get a fifth term if things go as the polls are predicting.
We learned from 2000 years of human history - war, disease etc. - to create a transparent system of checks and balances. These days we are trying to pretty much remove all of it (digitally) for "convenience" and "efficiency".
I can not understand how any semi-decent societal system can accept a situation of opaqueness and no accountability.
Well, the same 'we' that push voting machines and other crap like locking government records up in proprietary software.
I push for algorithms specifically because they're (as a written document) transparent. If a human has to judge a historical context you're never going to get repeatable results. But I want the system to be able to diagram, with weights, all the inputs that went into something before I'll trust an automated version.
And "cost." As in, "I'd rather get crap news online for free than pay for a newspaper or broadcaster to keep the government honest."
Part of it comes down to an effect that's been known for a long time with human-computer interaction.
Whenever you have a machine in the decision loop of a job, many workers WILL DEFER to the machine whenever there's a question of whether something is wrong or right.
By deferring to the machine, the workers absolve themselves of accountability and shift all responsibility to "the system". I think this is often deliberately "baked in" by management to customer service of all kinds. You can see this in a small way if you've ever had to call an internet service provider in the USA to deal with a billing mistake.
[0] https://en.wikipedia.org/wiki/Government#History
What I find very problematic in the use of automated systems for fraud detection is taking the system decision as final and not offering an easy and clear way of appealing decisions.
I dislike how people just carelessly make these claims without justifying them. For instance, another (far more likely imo) possibility is that it converges into certain state.
Even if being an immigrant is correlated with higher rates of some crime, an algorithm cannot take that into account without bigotry. Imagine being an innocent immigrant. If the algorithm tags you as guilty, no amount of explanation about averages or neural nets are going to make you feel better about your erroneous charge. This is the reason “stop and frisk” was such a huge deal: it let the cops “racial profile” for drugs, meaning that at any time all black and brown people were more likely to experience 4th amendment violations by the cops.
Our language around probability doesn’t help either. We say things like “being an immigrant makes it /more likely/ that you will take some action”. There might be a correlation in aggregate data but in any individual case we can only look at the relevant facts. Think again about being an innocent immigrant. Would you accept someone saying that /you personally/ were more likely to have committed a crime? Of course not!
The moral of the story is that stats describing a cohort cannot be projected onto individuals in that cohort. The data flows the other way; individuals comprise an average, no individual /conforms/ to an average. They may happen to coincidentally be average, but no causation has occurred.
Our justice system is supposed to be “innocent until proven guilty”, not “innocent until Bayesian inference suggests high enough probability.”
"What would be wrong though is to blindly take that probability for the fact."
And it's right: your algorithms can use all sorts of indicators to flag suspicious cases, but in the end the evaluation needs to be done by a person on a case by case basis. As long as the investigation on the single case is performed "silently" (i.e. without causing any loss of time or money or any annoyance to the person who is investigated) I don't see the issue. Otherwise it's the same as saying you cannot sort rows in an excel sheet according to some criterion and start looking at the data from where it looks more promising.
As a concrete example of where this can go wrong, let's say that (hypothetically) 1% of people commit fraud, and that this prevalence is not correlated with race. But, due to past discriminatory practices, more people of one race than another are convicted of fraud. Then, this algorithm would perpetuate this discrimination, since, say, it would identify 80% of fraud in the discriminated group but only 50% of the non-discriminated group.
Also, dual citizenship is not a protected class last time I checked.
A model that does not incorporate all the relevant information will have more false positives. The practical consequence of doing what you suggest is, cetaris-paribus, that you will incorrectly penalize more people.
> There might be a correlation in aggregate data but in any individual case we can only look at the relevant facts.
Definitely. Thats where courts and human review come in. The mistake the Dutch made here was not the variables in the model, but that they had limited human review of a model with (I'm assuming) low predictive power.
> Our justice system is supposed to be “innocent until proven guilty”, not “innocent until Bayesian inference suggests high enough probability.”
"Guilty beyond a reasonable doubt" (or whatever the Dutch equivalent is) implies exactly the Bayesian calculation you are rejecting. That said, I agree that a machine learning model with limited inputs is not sufficient to determine guilt.
A lot immigrants from Marocco and Turkey own real estate in their country of origin. Their IRS doesn't readily share information with ours.
See: https://www.uwv.nl/zakelijk/images/handreiking-inkomen-en-ve...
That's insane. Men commit 99% of violent rape, imagine not being able to say that. And I say that as a man, innocent of rape.
Similarly, if people with dual-citizenship smuggle more you're wasting time by treating everyone equally when looking for illegal imports.
> If the algorithm tags you as guilty, no amount of explanation about averages or neural nets are going to make you feel better about your erroneous charge.
Yes, any black-box trial is crap. But a transparent algorithm based on real data that a certain subset of the population is more likely to commit a crime, which is used for proper planning and investigation, is not.
> Our language around probability doesn’t help either. We say things like “being an immigrant makes it /more likely/ that you will take some action”. There might be a correlation in aggregate data but in any individual case we can only look at the relevant facts. Think again about being an innocent immigrant. Would you accept someone saying that /you personally/ were more likely to have committed a crime? Of course not!
You have one proper point about terminology and then an appeal to rage.
You are right that nothing my demographic does (men, raping) makes me more likely to rape. But you're wrong later where you say "[claimed] /you personally/ were more likely to have" because, yes I (by the nature of being physically capable of rape) am more likely to have committed it than other people. To me I'm not a statistic, to you I literally am a population sampling.
> Would you accept someone saying ...
If they're right, yes. Otherwise you're just saying that my outrage should trump the truth.
If a woman wants to organize a women's only bus, or hotel room, because she fears the harm I as a man could commit I shouldn't have the right to force myself on her.
> This is the reason “stop and frisk” was such a huge deal: it let the cops “racial profile” for drugs, meaning that at any time all black and brown people were more likely to experience 4th amendment violations by the cops.
No, the problem with stop and frisk is that it was (allegedly, I'm not from there) used by racists to target black people. If it was used as commanded by an algorithm then it would only happen where data showed an actual correlation.
In this sense, dual citizenship is basically code for being Moroccan.
What if somehow we as a society magically get equality overnight and there's no more connection between race and wage? But you use historical data where they were connected, still excluding wage data, and so even now your model connects race and fraud.
In this hypothetical example, I'd argue it would be inethical to include race without wage. Hell, wage is only going to correlate with fraud via yet other variables. Maybe an ethical system needs to go out and gather those measurements too.
(side note, in an ML system with regularization, even if you do include wage in your data, the regularization might pin some fraud on race anyways.)
The solution is to acknowledge that there are always going to be omitted variables and either a) be extremely careful and rigorous in your data gathering, design, and roll out or b) don't try to automate this thing, leaving it the slow expensive way where people can gather facts as needed, case by case.
"In one of the more egregious examples of the lack of humanity in the authorities’ approach, a report from Trouw revealed that the tax office had baselessly applied the mathematical Pareto principle to their punishments, assuming without evidence that 80 percent of the parents investigated for fraud were guilty and 20 percent were innocent. "
So this is actually exactly backwards. On top of being frighteningly amoral, I mean.
Yes, computers were used in the decision process and no I don't believe them to be algorithms.
It was specifically coordinated and covered up from the top. As your highlight shows, it was specifically designed this way. Which means it was a business process.
Yes, this is a massive scandal, but why is every article about it focused on spooky *ALGORITHMS* (I have never seen the word used as much as I have in the articles covering this story) instead of the clearly massive fault of the people running this.
Back in autumn 2013 I got a letter stating that I had received too much in child care benefits, with a request to pay it back. On the back of said letter was the text 'If you are still entitled to child care benefits and want to settle this amount against upcoming entitlements, you do not have to do anything'. Given as I was still entitled to benefits, I proceeded to not do anything. A few weeks later I got a reminder, once again stating that if I wanted to settle, I wouldn't have to do a thing. The reminder was followed by a final notice with opportunity to pay, once again with the exact same text on the back of the letter. So, obviously, I once again did nothing.
Then we got a 'Dwangbevel in naam des konings', i.e. a writ of execution in the name of the king. Red envelope. No added text. So I had to pay up. Now, as you may well guess, the tax service had already also started settling against the benefits that I was still entitled to! This prompted me to write a strongly worded letter, asking the tax service in no unclear language what I should have done differently, and requesting them to not dock me for that particular money twice. Literally the only reaction to that letter from the tax service was in silently restituting to me the double payment.
Now, I'm a reasonably well paid and well educated software developer, who just happens to hold two nationalities. I am perfectly sure that I would not even have had a reminder had I just had the Dutch nationality.
I could have paid in the first place and not settled against upcoming benefits; I was just lazy. However, imagine you're scraping by on minimum wage and are put in such a situation. There is no other way you could have acted than I did, but you also would not have been able to pay up to the writ of execution. And there would have been no recourse!
Add to that the fact that in many cases, the people hadn't even actually been paid too much, as I had, but merely been tagged as possibly fraudulent and put in to the system for reclaiming paid benefits, so the government could take a stance of being hard against benefits fraud while figuring out if any fraud had taken place in the first place.
* A hospital AI algorithm discriminating against Black people when providing additional healthcare outreach by amplifying racism already in the system. https://www.nature.com/articles/d41586-019-03228-6
* Misdiagnosing people of African decent with genomic variants misclassified as pathogenic due to most of our reference data coming from European/white males. https://www.nejm.org/doi/full/10.1056/NEJMsa1507092
* When the dangers of ML in diagnosing Melanoma exacerbating healthcare disparities for darker skinned people. https://jamanetwork.com/journals/jamadermatology/article-abs...
* When Google's hate speech detecting AI inadvertantly censored anyone who used vernacular referred to in this article as being "African American English". https://fortune.com/2019/08/16/google-jigsaw-perspective-rac...
* When Amazon's AI recruiting tool inadvertantly filtered out resumes from women. https://www.reuters.com/article/us-amazon-com-jobs-automatio...
* When AI criminal risk prediction software used by judges in deciding the severity of punishment for those convicted predicts a higher chance of future offence for a young, Black first time offender than for an older white repeat felon. https://www.propublica.org/article/machine-bias-risk-assessm... And here's some good news though:
* When police wrongfully arrested a person based on faulty facial recognition match using grainy security camera footage, without any due diligence, asking for an alibi, or any other investigation. https://www.npr.org/2020/06/24/882683463/the-computer-got-it...
* When the above is compounded for people of color according to studies which show that facial recognition systems misidentify dark-skinned women 40x more often than for light-skinned men. http://news.mit.edu/2018/study-finds-gender-skin-type-bias-a.... Another study showed false positives can be 10x to 100x more frequent for Asian and African American faces compared to Caucasian. https://www.nist.gov/news-events/news/2019/12/nist-study-eva...
* When an algorithm blocked kidney transplants for Black patients. https://www.wired.com/story/how-algorithm-blocked-kidney-tra...
* When clinical algorithms include “corrections” for race which directly raise the bar for...
For instance:
> * Misdiagnosing people of African decent with genomic variants misclassified as pathogenic due to most of our reference data coming from European/white males.
It's obviously caused by studying the people who present, and then demographics changing. Nobody made a decision here with any ill intent. Both the hospital and the insurance company share the patient's interest in them getting better and are already looking for better data.
> * When AI criminal risk prediction software used by judges in deciding the severity of punishment for those convicted predicts a higher chance of future offence for a young, Black first time offender than for an older white repeat felon.
A higher chance of any offense, or of being a burden to society? Kids are more likely to commit smaller crimes, and those are a gateway to larger crimes that then put you away until you're a hardened repeat felon. If black kids were being enticed into crime we certainly want to know about it. If it's a subpopulation rather than an area the causes are likely different and good decisions only come from good data.
> It's obviously caused by studying the people who present, and then demographics changing. Nobody made a decision here with any ill intent. Both the hospital and the insurance company share the patient's interest in them getting better and are already looking for better data.
Not a single example I gave was the result of ill intent. That's literally the point of my list. There's a difference between systemic bias and intentional bias. These are examples of systemic bias.
> A higher chance of any offense, or of being a burden to society?
I provided the link which answers your question. In this case, it predicted a higher likelihood of future crime for a kid who attempted to steal someone's bike and scooter, than for an adult who had shoplifted, been previously convicted of armed robbery, and served 5 years in prison already.
> Kids are more likely to commit smaller crimes, and those are a gateway to larger crimes that then put you away until you're a hardened repeat felon.
Your logic seems to acknowledge that one of the large drivers of crime is being "put away" in prison. In this case, the algorithm predicted a higher likelihood of committing future crimes for a kid than for someone who had already served 5 years in prison for armed robbery. Even worse is that the algorithm was being used to determine the severity of their respective punishments. So, the act of predicting a higher likelihood of future crime for the kid becomes a self-fulfilling prophecy, giving her a harsher sentence, which in turn is more likely to drive her toward future crime. This is systemic bias in action.
Because the two things I pulled from your list aren't systematic. One was local to a specific health system and about a specific condition, and the other about kids from a certain area. Neither was just about black people in general. Your list is predominantly racial and I thought that was the point. This medical thing is really about any group not being recognized as different enough from the whole, women get this a lot, but even men do in areas where women are the primary sufferers of something. (Breast-cancer for instance.)
Also, to distinguish things that are intentional (redlining) vs absolutely unforeseen but still problematic. A few absolutely undeniable things, and they exist, would make the list more meaningful and perhaps give more weight to the rest which are a bit ambiguous.
> There's a difference between systemic bias and intentional bias. These are examples of systemic bias.
No, they're an example of non-systemic bias. The aren't broadly across any system, or symptomatic of a general attitude. Systematic bias doesn't mean that the majority gets more attention, it means that the entire system is biased against something, and you aren't showing that.
> Your logic seems to acknowledge that one of the large drivers of crime is being "put away" in prison.
It is a separate topic, but yes. Being put away unjustly, and in the wrong place or severity for your crime hurts. But not serving a sentence for crime is just as bad, if not worse. I'm not against jailing criminals, just doing it where it doesn't serve us or them.
> In this case, the algorithm predicted a higher likelihood of committing future crimes for a kid than for someone who had already served 5 years in prison for armed robbery.
Could be. Without knowing the severity of the crime predicted though it's kind of meaningless.
> Even worse is that the algorithm was being used to determine the severity of their respective punishments.
Well, without knowing the prediction you can't know if a stiffer sentence would have encouraged or deterred future crime. If they were predicted to be killers, a hard but fair sentence at the first crimes could turn them around. If they're predicted to steal a few cars then even a year in prison is probably going to make them worse.
> So, the act of predicting a higher likelihood of future crime for the kid becomes a self-fulfilling prophecy, giving her a harsher sentence, which in turn is more likely to drive her toward future crime. This is systemic bias in action.
It would be, if it were system wide. It was a specific population though, and there's still no indicator that it said black kids specifically other than because in those neighborhoods they were the ones at risk. fwiw, stealing scooters from children doesn't seem like the path to good behavior.
Systematic risk is AIs for this purpose in regard to anyone, not that they targeted black people more in some cases. That's essentially random.
Ah, no, that wasn't the point. They were just examples of systemic bias. The fact that so many are racial is just a reflection of the unfortunate state we find ourselves in.
> A few absolutely undeniable things, and they exist, would make the list more meaningful and perhaps give more weight to the rest which are a bit ambiguous.
I honestly don't know how any of the examples posted are deniable, they're all well founded with research and evidence. But you're right that adding more to the list can only help.
> Systematic bias doesn't mean that the majority gets more attention, it means that the entire system is biased against something, and you aren't showing that.
That's not what systemic bias means.
"Systemic bias, also called institutional bias, and related to structural bias, is the inherent tendency of a process to support particular outcomes."
https://en.wikipedia.org/wiki/Systemic_bias
Also, examples are evidence which support a theory (of systemic bias in this case). Any one data point is rarely intended to be absolute proof of that theory, much less a description of its scope in its entirety. All of these examples do indeed provide evidence toward the existence of systemic bias within systems without fully describing or proving the extent of the entire scope of systemic bias within those systems.
> But not serving a sentence for crime is just as bad, if not worse.
The article I linked explains that this is not what the algorithm was used for. It was not used to decide whether or not to sentence them, it was used to help decide the severity of their sentence.
> If they were predicted to be killers, a hard but fair sentence at the first crimes could turn them around. If they're predicted to steal a few cars then even a year in prison is probably going to make them worse.
You seem to be agreeing that this was in fact a badly biased algorithm, since it was recommending a stiffer punishment for a kid who attempted to steal a bike than for a repeat felon, previously convicted of armed robbery, who shoplifted.
> It would be, if it were system wide. It was a specific population though, and there's still no indicator that it said black kids specifically other than because in those neighborhoods they were the ones at risk. fwiw, stealing scooters from children doesn't seem like the path to good behavior.
If something affects a population, that is system-wide by definition, since the definition of "system" is pretty broad; see the description above. Regardless of whether or not you think stealing scooters is a good prediction, the fact here is that it rated the likelihood of future crime as higher than for someone who was already a career criminal.
> Systematic risk is AIs for this purpose in regard to anyone, not that they targeted black people more in some cases. That's essentially random.
If only. Unfortunately, this is seen over and over again, more so than would happen by randomness. That's the point of the list.
Do you think racial cases of bias outweigh other demographic biases? And what's your opinion of the general level of intent (or lack of, to fix) relative to other issues?
The medical one made me think about the similar case of differing heart-attack symptoms in men and women. In this case actually because doctors just focused on men.
> That's not what systemic bias means.
> "Systemic bias, also called institutional bias, and related to structural bias, is the inherent tendency of a process to support particular outcomes."
That just shifts the definitional question to the scale of the institution, structure, or process.
At one end, if everyone follows the same rules it's a system, but what about if one clinic got bad data, do we write that up as a systematic failure? That's okay if it's your definition, but cases at the other end seem a lot more important and a lot more amenable to systematic fixes.
> If something affects a population, that is system-wide by definition, since the definition of "system" is pretty broad
Is the definition, "a system is broad" or (I think you mean) "broad as in flexible", such that a system can be anything? Any micro or macro population?
I guess where I was going with this is that the list seemed to be about some things that are very broadly impactful but not specifically racist, like facial-recognition not recognizing anyone but white men very well, but then a related issue of how this is encoded in the system. Is usage required, or was it random, etc.
Such that maybe top-to-bottom it'd be sorted by breadth. How many people does each impact, and how deeply-mandated or intertwined are the issues. And then different populations and problem to both give contrast, but also to indicate (when complete) the demographics of the problems.
So I was wondering if you'd focused on black issues to make the list.
> Unfortunately, this is seen over and over again, more so than would happen by randomness. That's the point of the list.
So, statistically, how much subpopulation misrepresentation would you expect in various ways? And are you saying there's more in general, or more racial, than expected?
I interpreted the one about kidneys as an error because of changing demographics and how people had caught it, made note that this sort of thing happens, and started checking other research for similar sampling bias. It read as a success story where the one about facial-recognition was more actively black mirror.
> You seem to be agreeing that this was in fact a badly biased algorithm
No. But not a good one either. More that I'm wondering if it was an attempt to model, like for predictive policing, or a tool sold to simplify the sorting of people? Because models are good, even when they're wrong, but crappy predictive tools are worse than useless and - where I was ultimately going with this - perhaps fraudulent to sell.
The 'good' case would be if someone built a criminality model and the city was trying to work with police and communities to intervene in a predicted pattern. It's not unreasonable that the societal harm from a non-criminal becoming criminal could be worse than an existing criminal remaining that way. So modeling and discussing this isn't bad, even if the data has a racial component and some of the questions are of bias.
But yeah, to recommend sentences, total crap.
When talking about systemic bias, I think it's less about intent, but that is one of the things that makes it so dangerous in terms of the level and longevity of impact; it's much more difficult to fix when there is no single bad actor at which to point the finger. This makes it easier for people to deny, either outright or at least in extent, and makes it harder for the systems to be fixed.
I don't know that I could place one type of demographic bias over another, but I think it's better for people to understand and acknowledge systemic bias of any form, if we have any hope to rectify any of them. I also think a lot of them are woefully entangled. For example, I've heard the argument that some bias isn't racial, it's economic, as in, a bias against poor people not people of color. But when there are already so many systemic economic biases against people of color, a lot of times it ends up being a distinction without a difference. I have plenty of examples of this, but don't want to dilute the impact of that last thought by going into detail.
> That just shifts the definitional question to the scale of the institution, structure, or process.
I agree. Systemic bias is everywhere, for almost all scales of systems.
> Is the definition, "a system is broad" or (I think you mean) "broad as in flexible", such that a system can be anything? Any micro or macro population?
Yes, you're right, I meant flexible. We can debate the severity of the impact of some form of systemic bias based on the size of the system (and the population or sub-population impacted), but to the individuals who are the victims of systemic bias, the result ends up being the same. At the end of the day, I don't think I'd fault any individual for taking up the cause of identifying and rectifying bias at any scale.
> I guess where I was going with this is that the list seemed to be about some things that are very broadly impactful but not specifically racist, like facial-recognition not recognizing anyone but white men very well, but then a related issue of how this is encoded in the system. Is usage required, or was it random, etc.
Yeah, I didn't really provide much context for why I specifically keep a list. It's not intended to single out racism, and in fact, it is intended to focus on unintentional codification of bias, because I think there are more software developers out there who are at risk for unintentionally codifying bias into their algorithms than there are those who may intentionally encode it.
The reason I keep the list is as a reminder of the level of impact you can unintentionally have in the systems you build without extremely deep thought and broad context. It's a reminder that what we do can have real impact on real people in ways we never imagined.
This is especially recurring as a theme for machine learning and AI systems which are taught based on limited datasets which often systemically under-represent those to whom it will end up being applied.
> Such that maybe top-to-bottom it'd be sorted by breadth. How many people does each impact, and how deeply-mandated or intertwined are the issues. And then different populations and problem to both give contrast, but also to indicate (when complete) the demographics of the problems.
I think this is a good idea.
> So I was wondering if you'd focused on black issues to make the list.
I did not specifically focus on that. I started tracking the list I think 2 or 3 years ago, and I think issues of systemic racism have been especially visible in the mainstream in this time period for reasons I won't get into. So, it's largely a reflection of the systems I've been made aware of through research and publication.
> So, statistically, how much subpopulation misre...
Thanks for approaching it this way. I think we often (societally, looking for blood) want to find a bad actor and are intentionally blind to bad things that just sort of happen at the edges, but need oversight to find and fix.
> For example, I've heard the argument that some bias isn't racial, it's economic, as in, a bias against poor people not people of color. But when there are already so many systemic economic biases against people of color, a lot of times it ends up being a distinction without a difference.
It doesn't help the sufferer, at the time, to be told that someone else would be suffering equally, but it does help fix the problem I think, to realize that it's circular via poverty or whatever, not simply racial, because in some areas and at some times it has simply been racial and it hugely changes how you fix it. If it's overt you can't just offer change, you have to prevent further damage during the repair process.
> The reason I keep the list is as a reminder of the level of impact you can unintentionally have in the systems you build without extremely deep thought and broad context.
Have you ever read comp.risks? I really like it as a source of Therac-25 type stories (across all fields) that engineering types should think about when building things.
> I just know that I've not yet been able to find any real systemic bias, at least in the US, against rich, caucasian males.
Is Twitter not a system? :D
It gets a bit fuzzy with bias against the majority. Every model that isn't right disadvantages everyone and the majority is part of everyone. So bad drug laws impact white people too. But because actual race is only encoded in one direction (affirmative action, "positive" directions) then anything that impacts white people also impacts everyone, whereas there are often specific laws (such as for constructing "The Projects" in the first place) that do directly exclude whites from the harm they caused. So subgroups definitely experience more exclusive problems, even aside from the amount of problem.
> how difficult these things are to rectify at a systemic level, since the vast majority of doctors using the still-biased eGFR formula have no idea that it has this problem.
I think it's just that the story is best told from that moment. That's the OMG. From there it improves, and I'm sure they sent a copy of the report worldwide asap. But no solution is ever 100% so there's no wrap-up party and it will never look done and solved. (Even one doctor who didn't check their email...)
> Fraud generally requires mal-intent, which most of these models and products didn't have, at least if we're being generous and optimistic.
Probably, but if they're saying "our product does X" maybe there's something to grab onto and investigate. Maybe they did misrepresent it.
> I just finished reading a book called, Weapons of Math Destruction, which talks about the damage models can do. The author posits a set of tests to tell whether the model is beneficial or destructive. One of the hallmarks, the author argues, of a good predictive modelling system is one which includes feedback into the system as a result of its predictions.
Good point, and thanks for the book recommendation.
A lot of things aren't amenable to that though, because the hypothetical city/community meeting can't take years to watch the outcomes and train continue to build a model, they've got to work from historical data up to that point and make policy decisions in the meeting.
> One of the other primary tests is dependent on the context of how the model is used, so it's not really reasonable to...
Younger people are more likely to re-offend than older people. Remove race from the situation entirely, and this is still the expected result. There's zero reason to think this algorithm was based with respect to race.
1. Even after accounting for criminal history, recidivism, age and gender, black defendants were still scored as much more likely to re-offend.
2. It incorrectly predicted that black defendants would re-offend much more frequently than white defendants.
3. It incorrectly predicted that white defendants would not re-offend much more frequently than black defendants.
Considering those three things and that they have refused to give any details about how the algorithm works, I see zero reason to give them the benefit of the doubt. Fire them until they can demonstrate that the algorithm is not in fact biased like it appears to be.
The score also takes into account answers to questions like whether the defendant's parents were separated or whether their parents were ever arrested. Those things are completely out of the defendant's control and are highly correlated with race. Even including those things in their score is damning.
* Out of 100 white people, 5 were labeled high risk.
* Out of 100 black people, 20 were labeled high risk.
* Out of the 5 white people labeled high risk, 4 re-offended.
* Out of 20 black people labeled high risk, 16 re-offended.
In either case, someone labeled high risk had the same likelihood to re-offend: 80%.
"It incorrectly predicted that black defendants would re-offend more frequently than white defendants." This is technically correct, but not because the algorithm was bad a predicting rates if re-offending. It's because there was higher rates of re-offending. The likelihood of re-offending among someone labeled high risk is the same.
This kind of objection seems like a blanket rejection of any system that produces an inequitable outcome. But the reality is that rates of re-offending is not equal. Even a perfectly accurate prediction of re-offense is going to predict higher rates of re-offending among men. Because men re-offend at higher rates. This isn't sexism.
That doesn't mean we shouldn't recognize the disparate impact of incarceration on underprivileged people. But simply concluding bias due to inequitable outcomes is simplistic.
I'm not even convinced that you should be allowed to make a decision based on stuff like that even if it is somehow equitable.
Again, not right at all. These things heavily correlate with crime. A broken home is the primary indicator of someone's future success, even over a 'better' but broken home.
I'm male, and not a rapist, but having a penis heavily correlates with rape. I suggest you do not pick me, or any male, to watch your children. Sure it's rude to the innocent, but oh well, a little rudeness vs potential harm.
> I'm not even convinced that you should be allowed to make a decision based on stuff like that even if it is somehow equitable.
Then nobody will ever follow the law. If it's illegal for me to reject a potentially bad babysitter for something I know about them that could risk my child's safety I'll happily lie and say I didn't like their haircut.
You'd get further if you tried to identify these people and treat them better - grants to move out of bad neighborhoods, to get educated, to get pardons for unrelated crimes, to get counseling for abuse, etc - than with this "wrong unless it's 100% identical, equity-of-outcome" thing.
It needs to be held to a higher standard and punish people based on their actions, not based on what their parents did.
> It needs to be held to a higher standard and punish people based on their actions, not based on what their parents did.
And that's not what actually happened. A guy got arrested on bad data and then released, we know some kids might be at risk because they're from broken homes, etc. Nobody was jailed for their parent's actions, and nobody even proposed it. Some trends are worrying, but you act like they're the intent of the entire system not just some scammy products that a company is pushing.
Voting machines, for instance, should all be burned. But I understand that most people don't know why and support them for convenience. I think they're anti-democratic but I don't think people are evil for using them. You should try to get a similar perspective.
> Nobody was jailed for their parent's actions
Not jailed, but they are being kept in jail because of them. That is the same thing.
I am not claiming that anyone is evil. Just that this is unjust and that they need to stop doing this. You are putting words in my mouth.
Thankfully, the article I linked to looked at both:
What do the non-black box findings from the area show? If blacks are a poor demographic in the area it might be true. (Crime tracks poverty, not race.)
But yeah, certainly don't pay anyone for, or use, a black-box algorithm.
> The score also takes into account answers to questions like whether the defendant's parents were separated or whether their parents were ever arrested. Those things are completely out of the defendant's control and are highly correlated with race. Even including those things in their score is damning.
Nope. That's perfectly fair to look at. For instance, a broken family is another predictor. It's not fair, but being a victim increases your chance to offend. Similarly, the number one predictor of child sex crimes is to have suffered them yourself. If you have a child and are picking a babysitter, skip the one who was molested.
fwiw, those things don't correlate with race, they correlate with poverty which correlates with race. Broken families are more likely to be poor and abusive.
How petty to now attack the poor in such a cowardly manner...
For other examples: field drug test kits--they really only say "this might be drugs", but the prosecutors use jail to try to get people to plead guilty rather than promptly doing a proper test.
Also, the ubiquitous breathalyzer--it's actually coded into law as proof in many places but it's not. There is an inherent biological flaw that makes the readings inaccurate.