If justice was truly blind (which is what I would consider fair) then the number of defendants incorrectly classified as dangerous would depend on the likelihood of all the groups they're part of committing dangerous crimes.
You want it fair based on race ? Ok, don't input race. Everything else stays the same. Why this complexity ?
I believe France does not collect race information from its citizens (or perh inhabitants in general). If true, it'd be interesting for an outside observer[1] to see if the omission of race makes a difference.
That said, it'd be interesting to see how other "algorithmic bias" might play out in more or less racially homogenous, if not entirely ethnically homogenous populations --for example Rwanda, Finland, S Korea. Things like class, neighborhood, education etc.
[1]Given they themselves don't officially collect race information.
France does not use any algorithmic decision making in its court system as far as I am aware, but like many people said here, in the end, the cause is not "race" itself but the correlations it comes with (e.g: in France, people that commit crimes are more likely to be from poor suburbs, which are predominantly populated by people whose parents came from North Africa/Subsaharan Africa).
I am not convinced. Including address sounds like a very bad idea as well. We don't want to punish people for being black just as much as we don't want to punish people for living in poor neighborhoods. It seems to me you should only include data you are ok to make decisions on. This will require collecting more information during court proceeding than you have in general statistics though.
But what if geography does have an influence on behavior because incidentally there is a lead-acid battery manufacturer who is a bit lax about dumping, in say a country with less regulation?
Or let's say the algorithm is deciding on insurance of some sort's premiums and someone is living next to a SuperFund site.
>>But what if geography does have an influence on behavior because incidentally there is a lead-acid battery manufacturer who is a bit lax about dumping
I don't want to punish people for living in polluted areas therefore I don't include that data in the training set.
>>Or let's say the algorithm is deciding on insurance of some sort's premiums and someone is living next to a SuperFund site.
I am not ok with doing that therefore I don't include address in the data.
Am I ok with punishing people for living in poor/dangerous neighborhood? No? Then don't include that in the data. Am I ok with people paying more for car insurance if they drive in a city with more cars/accidents/worse driving culture? Yes? Then I include it in the data there.
>"Am I ok with punishing people for living in poor/dangerous neighborhood? No? Then don't include that in the data. Am I ok with people paying more for car insurance if they drive in a city with more cars/accidents/worse driving culture? Yes? Then I include it in the data there."
I'm not sure I can reconcile that.
1. in the first option, we're not applying the consequences random people, we'd be affecting those already found to have committed a crime.
2. in the second option, if we think we'd be affecting innocent people in option 1. why do we think we would not be affecting innocent people (good drivers who happen to live among poor drivers) in options 2?
That's to say, either both are right or both are wrong. I see the same effects in both.
So this idea seems intuitive at first but turns out to be one of the worst things to treat unfairness.
There are several reasons for this from both technical and legal perspective.
It is incredibly easy to find statistically significant correlations given just a few (more than 7) different views of the data. In general these ml models are not working with less than hundreds or thousands.
If the model learned this suppose racial bias, once, you deleting this column is not going to stop it from learning it again, and I believe some research showed that it actually can make the unfairness more severe.
from a legal standpoint a company that may or may not be infringing on rights could just say, oh we can't be because we don't have these fields in our data: which makes it harder to monitor and audit wrong doing.
most of the methods that I am familiar try to ease the effects of the learned biases as a post-processing step for the model.
>>It is incredibly easy to find statistically significant correlations given just a few (more than 7) different views of the data. In general these ml models are not working with less than hundreds or thousands.
I would be interested in seeing examples. So far in this thread the arguments were along the lines of: "but then the algorithms punishes poor neighborhood instead of race" but you shouldn't have address in the data either as (I hope) nobody is ok on punishing people for living in bad neighborhood. We should only include data we would like to see in the explanation of the sentencing.
"You are not getting parole because you live in a poor neighborhood" is unfair while most people would be ok with:
"You are not getting parole because you willingly associated with people who committed crime".
The necessary result is that you are no longer judged on your crimes, but on your genes.
Why ? Because crimes aren't committed by all groups equally ... so you wouldn't expect the risk to be equal.
Even for completely practical reasons there are differences. For obvious reasons a 1.4 meter individual is limited risk for armed robbery, to give an extreme example. But there's hundreds to thousands of things like that, that all randomly affect the distribution. The result is that when all put together, things are seriously out of whack.
Also I fear the reverse effect. Communities work, to some extent, because of lack of crime. Doing this will necessarily make those neighborhoods with the "victim" genes more violent, more criminal. The result will be improvement of those neighborhoods ? I have to say, I'm VERY skeptical.
Then, the algorithm gives negative weight to locations where dangerous groups tend to live. That is, it infers race from other parameters.
What you should do is to keep the ratio of people classified as dangerous same among different groups. But this comes at a price as reduced classification accuracy.
>>Then, the algorithm gives negative weight to locations where dangerous groups tend to live.
Of course you don't include location. Why would anyone be ok with "you got longer sentence because you live in poor neighberhood"?
>>What you should do is to keep the ratio of people classified as dangerous same among different groups.
Sounds like a terrible idea. What if there is a community which just doesn't contain many dangerous people at all? If someone from this community commits a crime once in a blue moon they will be unfairly punished.
>>Sounds like a terrible idea. What if there is a community which just doesn't contain many dangerous people at all? If someone from this community commits a crime once in a blue moon they will be unfairly punished.
Now imagine if people were unfairly punished based on something they were born with and not where they lived.
The justice system is meant to treat every person using the exact same criteria, as long as it's statistically relevant. That people are born in places where crime is higher isn't a problem the justice can or should handle; it's completely outside its domain of responsibility.
Europe has also passed “right to explanation” to help with this issue which is great. unfortunately it seems they came up with it in isolation from researchers on this topic and went with the don't record it philosophy.
here is a good resource on that: https://arxiv.org/pdf/1606.08813.pdf
European Union regulations on algorithmic decision-making
and a “right to explanation”
So you are going to argue with statistical probability? It is our best determining factor. Have any better ideas?
If you don't believe in using probability to determine risk. How about you hire a pedophile to babysit your kids? How about throwing down your retirement on lotto tickets? ETC.
Do you think biased policing causes people to commit crimes? How about not committing crimes to begin with? How about not hanging with troublemakers? You think it is impossible for people to avoid being arrested and convicted of crimes because of "biased policing"? Please think harder.
So in sum: "analyzing data from Broward County, we find that optimizing for public safety yields stark racial disparities; conversely, satisfying past fairness definitions means releasing more high-risk defendants, adversely affecting public safety."
In other words, black defendants actually are more dangerous to release and there is no magic algorithm that bypasses this fact.
On a topic like this it’s important I think to address the elephant in the room. I don’t think this implies a genetic issue! I’m not an expert but from what I know this shouldn’t lead me to the conclusion that “black Americans are predisposed to violence” and, if you know approximately the same things I do, I think that’s probably fair for you too!
I don't think anyone here on HN thinks that it implies a genetic issue, but rather an issue caused by generations of discrimination and failed attempts to stop the cycle of poverty and crime that plagues many of these communities.
I am 100% certain, based on 5 years of observation, that there is a significant portion of users on this site who absolutely believe that racial disparities are genetic and immutable.
I would be amazed if most important physical, mental, or social characteristics had zero underlying biological or genetic drivers, or if they were completely driven by underlying biology. Even small underlying differences can be amplified through compounding effects over a lifetime of decisions, through culture, etc.
(The easy ones are male/female; racial differences are far less.)
But they are, according to virtually all the data that we have. However there's no indication at all that the cause is genetic but rather cultural in the US.
>>In other words, black defendants actually are more dangerous to release
Yes, but...
>>and there is no magic algorithm that bypasses this fact.
Maybe there is, it's just the method used wasn't able to find it either due to limitations of the method itself, not enough information or bias in the training set.
As a toy example: assuming you only have race and age to make your decision on then to optimize for public safety you need to include race to make good decisions. If you have race, age, number of friends who committed crime then maybe you don't need race anymore. The problem is that we are likely not getting enough data and then race is a proxy for that uncollected (and maybe uncollectable) data.
> If you have race, age, number of friends who committed crime then maybe you don't need race anymore.
I understand that your argument is a toy argument so it doesn't make sense to discuss it specifically, but I feel it's important to point out that the issue here isn't just which variables are used to make the decision, but what the decision ends up actually being. That is, maybe you find that you can make a very good decision based solely on age and number of friends who committed crimes, and don't take race into account at all -- but then if this algorithm ends up yielding "yes" to most white people and "no" to most black people (even if your algorithm doesn't use race at all), you haven't solved anything. [edit: "solved anything" was poor word choice on my part, obviously you have solved something, but you remain in the state described by the paper]
Another issue is that while you can "whitewash" variables, it's very difficult to scrub race out entirely. For example, in practice we can't use "committed crimes" as an indicator because we can never know a ground truth: we'd have to use "were convicted of crimes" instead. Unfortunately, you're far more likely to be convicted of a given crime if you're black than if you're white, so you're already mixing race into your variables even if it isn't named. With the disparity in convictions, enforcement, etc., it's very, very difficult to come up with measurable signals that are not in some way already tainted by racial decisions.
>>That is, maybe you find that you can make a very good decision based solely on age and number of friends who committed crimes, and don't take race into account at all -- but then this algorithm ends up yielding "yes" to most white people and "no" to most black people, you haven't solved anything.
It would actually solve the problem. It's ok if I give more "no's" to black people as long as black people are more dangerous in general. It's only not ok if I punish a specific non-dangerous black person just because they are black.
That's what fairness is: you get what you deserve because of your decisions and wrongdoing not because how you look or where you were born. That some groups end up with more convictions is expected and doesn't contradict fairness principle.
>> there is no magic algorithm that bypasses this fact.
> Maybe there is
No, there isn't. We actually have a mathematical proof (which is quite simple) why this is impossible.
Specifically, following conditions can't be true at the same time: 1. groups differ in base rate 2. prediction isn't perfect 3. decision is correct at the same rate for groups 4. decision is correct at the same rate for groups, restricted to positive/negative class.
1 is a brute fact. Your toy example insinuates at 2. 3 is called calibration and what is usually optimized by machine learning. When people say algorithm is unfair, it usually means 4.
If a group of people have, statistically, higher-than-average recidivism rates, should we be punishing all members of that group? That all but guarantees unfair treatment of individuals even if it makes statistical sense.
Even in your post you go from a statement that amounts to "statistically some groups have a larger number if dangerous individuals" to "black people are more dangerous". The two sentences do not mean the same thing!
You've got it backwards, though. The algorithm isn't saying "this person is black and therefore shouldn't get bail". It's saying "this person shouldn't get bail (according to a calculated flight risk based on bunch of reasonable criteria)", and a disproportionate percentage of the people who are assessed as high flight risks just happen to be black.
And the dataset that this algorithm derives its predictions from is presumably a real-world dataset, i.e. one where black people form a disproportionately large portion of convicts and recidivists.
The point stands: using statistics to meter out justice IMO amounts to collective punishment. Of course that leaves the question if whether more conventional methods are any better, but now we're opening up a new can of worms, namely what is the goal of criminal justice systems and how should those goals be achieved...
So maybe, rather than sticking our collective head in the sand and saying "no, it's impossible that black people are more likely to break bail or reoffend", we say "holy crap that's obviously caused by something" and try to address the root cause?
You cannot fix a problem by pretending it doesn't exist.
There's a difference between saying a disproportionately large number of recidivists are black and saying black people are more likely to reoffend. The latter frames the issue in terms of "what black people are like", and worse, gives the impression any given black person is more of a criminal than any given white person.
I'm not usually such a stickler for language, but in this instance it does seem to me that this kind of profiling serves to perpetuate that narrative of criminality being a feature of a group of people, and doesn't help to get at the causes.
And "public safety" conspicuously excludes the harms caused by incarceration (for both innocent and guilt suspects, and the innocent members of their families and communities).
(Not to mention that "public safety" still excludes the harms of centuries of slavery and decades of Jim Crow...)
Are you suggesting that we discard notions of justice / law due to historical sentiment? Or do you mean that we should address issues or biases which might be formed because of these events?
> In other words, black defendants actually are more dangerous to release and there is no magic algorithm that bypasses this fact.
You are right, but that's not the problem with the algorithm.
A critical assumption with large-scale data mining is that past trends continue - the problem is that the existing data fits the algorithm. It is just a conservative What-If decision maker operates on existing facts (i.e bad present day situation), just wrapped into code (or worse, encoded as opaque literal "biases" in a decision tree).
I see somewhat similar patterns in lending interest data (redline zipcode -> credit ratings) and the problem is that bigger the historical trend data, the less forgiving a "past trends" algorithm will end up being.
Since this ends up being a prisoner's dilemma, if you are a rational actor in this system and the system keeps playing a defect card on you, then the obvious move is to always defect - cut your losses.
Algorithms can't improve the job prospects of the people released. And that's not the algorithm's fault.
And therefore, the you're right - the algorithm can't change the world beyond its output result.
Which is why we shouldn't build our lives or policy around simple algorithms that do not take into account the breadth of human values, philosophy, and desire for change beyond the bad present day.
> A critical assumption with large-scale data mining is that past trends continue
Not necessarily. It would be easy to use only recent data and ignore data from decades ago. The prediction accuracy wouldn't go down much as long as there is a large enough sample size in the recent data. It may even go up if the old data really is inapplicable now.
Then if a past trend stops happening, the old data gets purged eventually and only the post-trend data is considered.
You know the phrase "round up the usual suspects"?
It's the same root idea: the goal of the system is not to find the person who committed the crime. The goal of the system is, for each crime, to find a person who can be nailed for it. And you can manufacture a class of people such that whenever you need an offender you can go grab some of them and stand a good chance at getting a guilty plea or a conviction.
You start by selectively hyper-enforcing small violations against a chosen subset of the population. Get them for minor traffic violations (which you can ensure turn into arrest warrants by setting the fines and fees and other costs high enough!), get them for "paraphernalia" offenses (where you assume everyday objects in Person A's possession are evidence of drug habits while they wouldn't be assumed evidence in the possession of Person B!), get them for all sorts of things.
Now whenever there's a crime you just go pick some random people from the demographic you've been doing this to, figure out who you can bribe to testify against whom, and then march into the courtroom with witnesses and a defendant who has a thick file of previous "encounters" with the system, and off to a cell they go for a while.
And it gets even easier each go-round: people with criminal records don't usually have the resources to move on and rebuild their lives, so once they're released they're going to be right back where you found them last time. And they've got an even longer record now, which is just proof that you've been doing a great job in figuring out who these dangerous recidivists are! The model works!
People who downvoted this (original comment is at -1): seriously, just go do some research on over-policing and corruption in police departments and prosecutors' offices.
I know it goes against some cherished beliefs, but this really is how much of the US handles "justice".
I've only had a chance to skim this, so maybe I misunderstood.
That being said, the authors' point seems to be right there in the introduction - "it's not fair to hold all individuals to the same standard, irrespective of race."
In other words, the broader community should tolerate laxer pre-trial sentencing standards applied to people of race A, precisely because they commit more crime than people of race B?
Pretty odd concept of "fairness" if you ask me.
Also interesting that explicitly adjusting for race does not affect recidivism predictions by the COMPAS model. In other words, as it stands, the model is not inadvertently discriminating on the basis of race. I wonder if the same testing has been applied to sentencing decisions.
> the broader community should tolerate laxer pre-trial sentencing standards applied to people of race A, precisely because they commit more crime than people of race B
Does "they" they mean "each individual of Race A", or the subpopulation of "Race A"? Those are two very different meanings, and the heart of the problem.
Why should I be jailed because other people with similar skin color to me committed crimes?
Should we go digging to find whether Irish or Russian caucasians have higher than average crime rates, and then refuse bail to all Irish or Russian-descendant defendants?
I don't quite understand your point. Race is explicitly not a factor in individual pre-trial detention decisions.
People are being placed in pre-trial detention because they're flight risks, as determined* by their marital status, whether or not they're unemployed, accused of a violent crime, with criminal records, etc.
As it turns out, people from race A overwhelmingly meet these criteria, leading to outcomes that the authors believe are unfair.
In fact, the authors are suggesting that race should be explicitly considered in these decisions, in order to balance intra-racial representations.
Given two people accused of the same crime, with the same job, marital and criminal history, the authors would detain one and release another, purely on the basis that one is white and one is black.
As I said, I don't think many people would agree with that definition of "fair."
* I don't know exactly what the model inputs are, I just made these up for demonstration purposes.
Your posts misunderstand both the article and the idea of fairness.
Also from the abstract: "In some cases, black defendants are substantially more likely than white defendants to be incorrectly classified as high risk."
The point is that an AI is a black box that may not explicitly use race but since it's using a variety of criteria rather opaquely, it may effectively, indirectly, use race, the neighborhood someone lives in, their social status or all sorts of things that aren't fair based on "your personal circumstances should determine whether you are considered a risk".
The authors propose a system to mitigate the problems here, though I actually don't really think that's the solution - these AIs should simply be abolished and replaced by objective criteria.
How do you figure out if criteria are unbiased without actually implementing an unbiased estimator though?
(Not an AI. A statistical Bayesian-like where we can reason on inputs and outputs.)
>Also from the abstract: "In some cases, black defendants are substantially more likely than white defendants to be incorrectly classified as high risk."
OF course they are substantially more likely to be incorrectly classified. They are in a group with far more risk. (this is simple statistics and probability)
Interestingly, if this Algorithm was a human, they would be called a racist by all on the left. Instead, the best than can be managed is "unfair".
Objective truth hurts sometimes. Perhaps some apologies are owed to the so-called racists out there. Turns out they were just being as honest as a probability algorithm (shame on them).
If AI an robots ever police our society, leftists will have to tamper with their highly accurate, objective statistical analysis of who to pull over for questioning to save them from their racism. That is already what is going to happen to this algorithm.
Machines exposing emotion initiated dishonesty (until they are beaten into social justice submission and artificial guilt subroutines added to make things "fair").
> Also from the abstract: "In some cases, black defendants are substantially more likely than white defendants to be incorrectly classified as high risk."
I did see that in the abstract, but then the only follow-up in the actual body of the paper was:
"...among defendants who ultimately
did not reoffend, blacks were more than twice as likely as whites to be labeled as risky."
Which is simply restating the premise in a roundabout fashion - i.e. that those with criminal records (etc) are more likely to reoffend, and black defendants are more likely to have criminal records. It's a bit of a long bow to draw to say the classification was "incorrect" post-hoc because "those from the risky group who did not reoffend were initially labelled as high risk".
> The point is that an AI is a black box that may not explicitly use race but since it's using a variety of criteria rather opaquely, it may effectively, indirectly, use race, the neighborhood someone lives in, their social status or all sorts of things that aren't fair based on "your personal circumstances should determine whether you are considered a risk".
I believe it's a straightforward logistic regression, which is about as objective as you can be. It's not "AI", it's not opaque and it's not indirect. It's a simple calculation of the relative odds of recidivism based on factors such as criminal history, employment, and so on.
Given that adjusting for race makes no difference to the model output, it is literally the most colour-blind approach you could take.
The authors, on the other hand, would have you treat two people differently purely on the basis of respective race.
Now obviously there are historical/legacy social issues at play, not to mention the broader question of what's "best" to eliminate disadvantage. But I suspect not many people will agree with the authors' that this would be "fair".
If I were pushing their line of thinking, I would be researching whether this type of "positive" discrimination actually reduces the race gap long-term.
That's ultimately the only thing that matters, and in my mind, it's a much easier sell to say "this isn't fair, but it's what we need to do to reduce the criminal overrepresentation and disadvantage that exist because of our history with slavery".
> In other words, the broader community should tolerate laxer pre-trial sentencing standards applied to people of race A, precisely because they commit more crime than people of race B?
Well, if you don't take the authors' approach, you don't need to. You just continue assessing pre-trial detention on the basis of each individual's background.
> As noted above, a major criticism of COMPAS is that
the rate of false positives is higher among blacks than
whites [2].
Policing in America is racist. So anything that takes arrests into account (prior arrests, recidivism, etc.) will be in error as well.
The line you quoted misses a lot of context - the 'standard' is an algorithmic risk score with a single threshold. The paper mentions why having different treatment by race might have legal problems (e.g 14th amendment). So this standard is what we're stuck with. The paper asserts that it isn't fair.
>Policing in America is racist. So anything that takes arrests into account (prior arrests, recidivism, etc.) will be in error as well.
This is really the clincher. If the algorithm is effectively aggregating multiple human decisions, it ought to correct for known biases in those decisions.
Citation needed. The fact that certain ethnic groups have problems with crime and violence can be attributed to the fact they grew up in single parent households, are less educated, and have less income than other ethnic groups, all things that contribute to a higher crime rate.
> The paper mentions why having different treatment by race might have legal problems (e.g 14th amendment).
I don't see how anti-slavery has anything to do with this. If you are convicted of breaking the law, as a citizen you will be subject to the same punishments as any other citizen.
Citation needed. Cops visit black neighborhoods more, sure. But they do that because they commit vastly more crime. Which is the cause and which the effect?
It's also a lot more complicated than that. People throw out studies that blacks are more likely to be caught for drug crimes. But they are less likely to be caught for more serious crimes. Possibly because they trust police less and are less likely to volunteer as witnesses. E.g. https://www.wsj.com/articles/the-underpolicing-of-black-amer...
When you make judgements about individuals based on their race and not their individual actions you're being racist. It's as simple as that. This is what the paper is looking at.
You can argue for being racist if you want but you can't say "it's not racist, because 'racist' is a word with bad connotations and I think what they do is good" without being called out on it.
>When you make judgements about individuals based on their race
No one is doing that. The algorithm does not take race into account. It's just an assertion that it uses other factors as a proxy for race. I've not seen this proven anywhere.
And indeed, if it were true, you would have to admit that blacks are more likely to be criminals. Even after controlling for all relevant variables. Which goes against the standard narrative that it's just socioeconomic status or whatever.
>you can't say "it's not racist, because 'racist' is a word with bad connotations and I think what they do is good" without being called out on it.
It isn't racist. And the word racist is becoming almost meaningless because how often you overuse it.
> Policing in America is racist. So anything that takes arrests into account (prior arrests, recidivism, etc.) will be in error as well.
Quite possibly - and if that were the case, it's certainly something to be addressed. As a non-American, I can't really comment.
But the paper doesn't really deal with that. It's saying that society should tolerate higher rates of recidivism amongst one racial group in the name of fairness.
I do understand the authors' thesis - I'm just saying that "fair" is a very subjective concept. I don't think many people will agree with the authors' definition.
It seems like the problem is that any data that works statistically will not be fair to individuals who are intuitively exceptions to the general trend. It sounds like algorithms relying on statistics are pretty much guaranteed to fail the fairness test. It's the wrong tool for this problem I think.
There are so many factors in morality that I think individually, case by case judgments are likely the best option. Racial / class bias seems to be just as present in these algorithms, so lets at least allow for human intervention rather than forcing ourselves to be constrained by an inherently unfair system. Yes, humans are flawed too, but perhpas we should be spending more time trying to adjust / account for those in different ways.
Note: Bias in sentencing/convictions is a huge problem that should absolutely be worked on. I'm only claiming that these approaches are inherently flawed, not that what we have now is anywhere close to good enough.
> There are so many factors in morality that I think individually, case by case judgments are likely the best option.
What are the inputs to "case by case judgments" except Bayesian priors and observations about a specific case? Your brain isn't doing anything a computer model can't in principle do. My point isn't that any particular bail pricing model is acceptable. Instead, I'm arguing that a desire to escape "statistics" by giving up on models and punting to the brain is futile, since the brain is just going to use its own statistical model whether the owner of that brain is aware of it or not --- it must, since statistics is baked into the structure of knowledge, and the brain has no private source of truth.
There is no accessible realm of knowledge about the real world somehow exempt from statistics, since we never have complete knowledge, and statistics models our uncertainty.
While I agree with your general point, artificial neural networks and biological neural networks are sufficiently different that it's not really fair to compare them just because they happen to have the same name. A biological neuron does a lot more than computing some affine transformation of its inputs.
Also, humans draw from an absurdly complex pool of inputs, usually called "common knowledge", that so far has given us a hard time when trying to replicate it in computer systems. So while in principle there is, imho, nothing stopping a statistical method to produce decisions at least as good as a human could, in practice this might not be the case.
To expand on your point to further strengthen your original argument, these models are trained on a smaller subset of observables and ideas we consider when we decide policy. Even then, the comparison between a matrix and a human brain is poor.
Another issue I have is that we don't know is whether statistical bias exists in the metrics they train on, something they discuss in the paper. We already know for example blacks are arrested and convicted harsher than whites for some of the similar crimes[0]. The paper just says that COMPAS predicts likelihood of violent crime, not whether that means they'll be arrested of crime. Moreover while Table 1 talks about a prediction regarding violent crime, Figure 2 talks about recidivism with regards to all crime, so which is it? It still seems nebulous to me what COMPAS actually is for.
I'd have to read it deeper than I have, but not sure how I feel about the paper.
3. The paper mentions "over 100 factors" the algorithm uses. Ad agencies have way more than 100 factors on me, and I still get irrelevant ads every day.
Humans aren't perfect, but they can catch things that stats won't, and vice versa.
They're both Turing machine approximators after all. Granted, ANNs aren't quite sophisticated enough to match the brain yet, but this capability gap is a difference of degree, not kind.
But I don't think the similarity of the brain to ANNs really matters to my argument, as interesting as this discussion might be --- so I've edited my original comment to make it more general and to focus on my higher-level point.
The difference between mathematical neural networks and humans is how they're trained and how they function, which is not just a difference in kind. The way that they adjust weights and perform inference is different enough that them being "theoretically isomorphic" is not a viable argument.
The algorithm also does "case by case" judgments. However its judgments are purely based on conditional probabilities observed from past events, which are computed using regression analysis (a relatively simple method). Human learning is enormously different from regression analysis.
> I'm arguing that a desire to escape "statistics" by giving up on models and punting to the brain is futile
I think we'd all agree here. The point is that both methods should always be used together, because each has huge limitations on its own.
What is the brain using except conditional probabilities derived from past inputs? Why would the use of some kind of regression (which I wouldn't characterize as "relatively simple") a problem if the overall result is nevertheless more accurate than the human equivalent? We judge models by their results, not their complexity.
The brain is applying some function over its inputs, after all, and we can apply that same function, either explicitly in code or implicitly via the training of some kind of model.
Machines are already better than humans at other tasks for which we imagined humans exercised fine and subtle judgment: https://www.sciencedaily.com/releases/2016/04/160421133831.h..., so I see no reason to believe that bail-setting is some unapproachable problem domain.
> The point is that both methods should always be usned together, because each has huge limitations on its own.
I disagree. Putting a human in the loop just invites those humans to inject unstated, socially-determined biases into the decision system, and fairness and transparency dictate that we be up-front about those biases. I see no evidence that a computer model couldn't reproduce biases we considered desirable.
> What is the brain using except conditional probabilities derived from past inputs?
Hormones, reflexes, instincts, empathy, etc. Again, human inference and learning are (as of yet) insurmountably more complex than statistical regression.
> regression (which I wouldn't characterize as "relatively simple"
I encourage you to study brain chemistry, and compare that with any stats course ever.
> Why would the use of some kind of regression (which I wouldn't characterize as "relatively simple") a problem if the overall result is nevertheless more accurate than the human equivalent? We judge models by their results, not their complexity.
Even a 99% true positive rate + 5% false positive rate can be worse than a 5% true positive rate and a 0% false positive rate, when you're talking about irreversible decisions.
> The brain is applying some function over its inputs, after all, and we can apply that same function, either explicitly in code or implicitly via the training of some kind of model.
Yes, that's very true. Executing the model is easy, and maybe even works similarly in NNs and brains. Figuring out that "some kind" part is what we formalise as "learning", and turns out is the hardest part of this equation.
Again, clearly, current regression methods are nothing like human learning. Else, they'd be able to learn a language like a human, without seeing billions of annotated data points.
> Machines are already better than humans at ...
Neither the article nor the paper claim the machine performs better. Their strong claim is that the machine is faster.
> I see no reason to believe that bail-setting is some unapproachable problem domain.
I think justice is a more important domain for many people. My guess is it's because it can set precedent.
> Putting a human in the loop just invites those humans to inject unstated, socially-determined biases into the decision system
>Hormones, reflexes, instincts, empathy, etc. Again, human inference and learning are (as of yet) insurmountably more complex than statistical regression.
I think what the parent is trying to convey is that usually when someone means when they say "this decision couldn't be done by a computer", what they mean is that "I don't really know how I'm making this decision well enough to describe it precisely enough to encode it into a set of rules."
To be able to decide if something is fair, one has to be able to understand the process by which a decision is made, no? And not just a woolly, after the fact justification that a person might well invent to explain their decision. If we can't reliably look at the inputs and determine what the outputs would be, how are we able to decide if the process is 'fair'?
As someone who is looking to understand a system, the argument of "But it couldn't make a human decision" triggers alarm bells of "What you mean is: It couldn't make a non-rational, unfounded, biased decision influenced by whether the person is hungry or not". Is that how important decisions should be made?
All these factors you mention can be modeled as constant or contingent biases applied to the inputs. There is no magical aspect of the brain here. To the extent that the brain is using unstated factors to make decisions, its decision making is unfair.
> I encourage you to study brain chemistry, and compare that with any stats course ever.
You're hand-waving. This is not a useful discussion technique. If you have a point about how some aspect of brain chemistry produces decision outputs that are difficult to encode, please make it. Otherwise, please stop this aspect of your argument.
That something is "insurmountably more complex" than some other thing doesn't make it better than that thing. Sometimes, it's just insurmountably more complex for no good reason. The blind watchmaker doesn't build elegant watches.
> Is this the sort of hormonal influence you find desirable?
Nope. I said nothing about desirability. The question was: "What is the brain using except conditional probabilities derived from past inputs?"
> All these factors you mention can be modeled as constant or contingent biases applied to the inputs.
It's possible, but we can't do it yet. If you have proof that we already can, please share it.
> To the extent that the brain is using unstated factors to make decisions, its decision making is unfair.
Fairness is human-defined, and difficult to encode in a computer. I conjecture that human + computer will always result in more fair and safe results than either one in isolation.
> You're hand-waving.
Yep. The number of variables (and their interactions) affecting a human decision far surpasses the number of variables in any regression method. In a lot of literature, number of variables characterises complexity.
> So what?
My example is empirical evidence of a human element identifying a false positive better than a computer. Europeans fighting wars is not such an example.
How would one go about supporting your conjecture (that fairness(human+computer) > fairness(computer)) if you can't define what "fairness" means, except by circularly referring to the inclusion of "human" term in the function input? Your argument doesn't make sense to me.
It's circular because the definition is circular, kind of like: "A mountain is larger than a hill, and a hill is smaller than a mountain." It's a shifting definition. Therefore I think modern statistical methods don't allow for that kind of self-reference, but humans do. I justify my conjecture by using laws as evidence. Except for the most basic of cases, they involve a human to interpret the law and the case to decide a verdict.
Philosophically speaking, I think the "key component" that humans bring, besides still-superior decision-making in many cases (not all), is creativity. So far, I haven't seen any machine learning that can outperform humans in light of limited brand-new data.
Statistics tell you something significant about a group, but little of significance about an individual. Statistical measures should not be used to make decisions about individuals. Other names for doing so are: "Sacrificing the one for the many" or "Presumption of guilt."
There is no cost of fairness here. They're playing percentages. I'd rather have an algorithm make predictions than a human judge/prosecutor. If a certain group of people are disposed towards recidivism or not showing up for trial, it doesn't matter how much they've suffered in the past. That's a policy decision to reach out to them.
I just saw another article[1] on a study using the same data set. It concluded that the COMPAS algorithm is no more accurate than random people given the task on Mechanical Turk.
Those researchers also designed a simple predictive model using only two factors as inputs: age and number of previous convictions. By comparison, COMPAS uses more than a hundred. To me, it doesn't sound like those extra factors end up contributing much.
> Those researchers also designed a simple predictive model using only two factors as inputs: age and number of previous convictions. By comparison, COMPAS uses more than a hundred. To me, it doesn't sound like those extra factors end up contributing much.
That's the nature of these things. If you use zero factors then you're basing decisions on random chance. Use one good factor and you'll do 60% better than random chance. Add another good factor and you'll do another 10% better. Every factor after that is a single digit improvement.
Because once you have something pretty good, making it even better is diminishing returns. But at the same time, "it's only 1% more accurate that way" when you're dealing with country-sized populations will change the outcome for many thousands of people.
And how does using fewer factors help anything? The underlying "problem" is that people of different races have different outcomes in actual fact, and any halfway accurate predictive algorithm will reflect that difference. It's not as if there is no racial disparity in the number of previous convictions.
If anything, using fewer factors could make the problem worse because it stops considering potential mitigating factors. A young black man with two prior convictions gets the shaft even if he has recently gotten married and bought a house in a better neighborhood.
That is a very poorly formed study precisely because systems like the Mechanical Turk actually work surprisingly well. This is a common theme in the 'wisdom of the masses.' If you ask a single person how many beans are in a jar, you're going to get an answer that's generally very wrong. Yet ask 100 people and average their answer and you tend to get an answer that's oddly extremely close to the correct answer. As the number of people independently asked approaches infinity, the error approaches 0.
So when you ask 'x' people to independently judge something, let alone something that is multiple choice with a correct answer, you're going to get answers that are far more accurate than a single individual would give you. So looking at the average answer of 'x' people, comparing it to an AI system, and arguing that they're relatively close so individual people are relatively close to the AI is completely fallacious.
---
As a tangential aside, there's another quirk to the wisdom of the masses. When you let the people communicate and try to intelligently organize and use expertise to come to answer, this effect disappears and the final answer again tends to be very wrong. Kind of an interesting perspective on the current zeitgeist of society and work.
===EDIT===
The authors were obviously aware of the wisdom of the masses. Quoting the paper itself:
To determine whether there is “wisdom in the crowd” (7) (in our case, a small crowd of 20 per subset), participant responses were pooled within each subset using a majority rules criterion. This crowd-based approach yields a prediction accuracy of 67.0%. A one-sided t test reveals that COMPAS is not significantly better than the crowd (P = 0.85).
That's a quite silly p-value and on top of that I'm not sure how they claim their system actually controls for the wisdom of the masses.
You'd think so, but that's not correct. This is not a straight forward result of probability with a filter of complexity. It works regardless of bias, though obviously if everybody was biased in the exact same way then that would cause things to break down - which is perhaps the reason that the coordination results in a worse result than independent averages.
For another example of it consider things like the television show, 'Who wants to be a millionaire?' It's a quiz show where one of the choices is for the participant to ask the audience. And the audience tends to do absurdly well on even the most esoteric questions, though independently they are certainly far from trivia experts. But very few are randomly guessing - their own experiences and biases leads them to entirely different conclusions. Yet somehow it produces the correct result time and again.
It's a strange phenomena and one that has to be constantly guarded against in anything involving sampling of people. This is a text book example of a study that gets destroyed by it.
>>> As the number of people independently asked approaches infinity, the error approaches 0.
>> Only if people are an unbiased estimator!!
> You'd think so, but that's not correct.
Either you're misinterpreting the technical term "unbiased estimator" here, or you are aware of some research that I would like to read.
In context, "unbiased" means that if you pick people at random and ask for their estimates, then on average the too-high estimates cancel out the too-low estimates (i.e. there is not a bias in one direction or another).
But people as a whole have poor understanding of many things. One common one that appears in social science research and is often replicated is that people grossly overestimate the size of the homosexual population in the US (the "wisdom of the crowds" often estimates it around 20% whereas best available polling data suggests 3-5%). Here's just one source for this phenomenon
"Wisdom of the crowds" is occasionally reliable, but it should not be assumed to be reliable for any particular problem without verification. It often fails terribly even on problems that are not very esoteric.
As mentioned, there is a difference between bias and uniform bias. In the US the media, politics, social media, and even miseducation (e.g. in my deviance class we focused on Kinsey's 10%, yet oddly enough never contrasted that against contemporary results) have heavily and uniformly biased the population on sexuality leading people to vastly overestimate the number of homo/bi/trans individuals.
Where it works phenomenally well is in areas where biases have not been directly instilled into people. This does not mean people are unbiased, however. Again the knowledge of trivia is a good example since while the crowds can generally do phenomenally well even at very esoteric questions where biases would lead them to individually come to very different conclusions, yet they will invariably fail to answer ostensibly trivial questions like 'What is the capital of Australia?' You'll get Sydney, it's not. You'd likely get a similar result for things like the capital of Pennsylvania.
In a way I view the wisdom of the masses as analogous to machine learning systems. They do an oddly good job of providing extremely precise answers to a wide array of questions even when trained with models that do not directly represent the 'questions'. Yet you can also break the systems, at times comically, with certain types of queries designed to do precisely that.
And as was the case with machine learning for quite some time, I think people remain reluctant to utilize it due to the black box nature of it. The implication of your comment is that the wisdom of the masses is little more than incorrect answers canceling out on average leaving nothing but a survey of experts. Yet I think there's no evidence for this (even if it may be a perfectly logical 'kneejerk' reaction) as it works even on things where nobody is an expert, and if this were the case then we ostensibly should be able to get comparable answers from coordination - yet coordination causes the entire system to collapse.
> As mentioned, there is a difference between bias and uniform bias.
You are simply mistaken in your interpretation of the technical term "unbiased estimator". This has a specific meaning in statistics, and is required for the convergence property you specified earlier. From wikipedia [0]
"In statistics, the bias (or bias function) of an estimator is the difference between this estimator's expected value and the true value of the parameter being estimated."
In lay terms, this means that the estimator process "ask lots of people and average the result" is unbiased only if all the too-high errors, in aggregate, cancel out the too-low errors.
I'm not sure if we're now going in circles or if you failed to read what I just wrote. Repeating it:
"The implication of your comment is that the wisdom of the masses is little more than incorrect answers canceling out on average leaving nothing but a survey of experts. Yet I think there's no evidence for this (even if it may be a perfectly logical 'kneejerk' reaction) as it works even on things where nobody is an expert, and if this were the case then we ostensibly should be able to get comparable answers from coordination - yet coordination causes the entire system to collapse. ... "
I'm not sure if you even realize all the assumptions you're making. You are assuming, for instance, that 'guesses' are regularly distributed. That assumption may be correct in some cases - I expect in many it is not. Alternatively there is the possibility is for you to claim that you're referring not to the individuals in question as the estimators, but the entire group. In that case you've spent a lot of time saying nothing as it boils down to "people are only correct if they're correct."
Sorry but you're mathematically wrong. The "wisdom of the crowds" does not work on things were everybody is wrong in the same way. For example if you ask a lot of people to estimate income inequality you get the wrong answer because everybody underestimates it.
Getting rid of algorithms means replacing them with humans. And humans are far worse.
Humans are terribly biased. Racial bias isn't even anywhere near the strongest bias we have. Unattractive people get sentences twice as long as attractive people. Judges give far harsher sentences before lunch, when they are hungriest. Socially awkward people seem to be pretty strongly discriminated against. Studies have found people discriminate by politics even more than race. Job interviews have actually been shown to degrade performance over just judging resumes. Before statistics and credit ratings, getting a good loan required being an old friend of the banker.
It's not just that humans are unfair. We are objectively terrible. Very simple statistical algorithms beat human "experts" in almost every domain they get tried on. As early as the 1920s, a statistician came up with a formula that was better at predicting recidivism than a group of 3 prison psychologists.
Simple linear regression has predicted the success of medical treatments better than doctors, diagnosed psychoticism better than trained psychiatrists, predicted academic success much better than admissions officers, predicted loan risk better than bank officers, etc, etc. To say nothing of modern machine learning methods on modern computers. It's insane we allow humans to continue doing these tasks at all.
But there has been huge resistance to algorithms in every domain. From people who stand to lose their jobs and be put to shame by them of course. But also even outsiders tend to reject algorithms. And overly trust humans. Psychologists have actually studied this. They call it a bias labelled "Algorithm Aversion". http://opim.wharton.upenn.edu/risk/library/WPAF201410-Algort... The study showed that humans were willing to forgive the mistakes of humans far more than those of algorithms, even when the algorithm made far fewer of them.
This is why they aren't everywhere already. The last thing we need is fear mongering like this. As shown, humans are far worse. If an algorithm shouldn't do it, then a human certainly should not.
A big part of the case these people make is a reference to a propublica study that found a slight bias against race in an algorithm once. Yet that study wasn't peer reviewed. The findings weren't statistically significant. https://www.chrisstucchio.com/blog/2016/propublica_is_lying....
It depends on your objective function, doesn't it? Computer models definitely do better than humans on the ostensible objective function --- predicting recidivism --- but computer models don't have decades of exposure to subtle social cues that signal to humans that the real objective function is one subtly different from the one that a naive understanding of the system would suggest.
That's really the beautiful thing about our shift toward algorithms: we'll no longer be able to rely on these subtle social pressures. If we want to optimize a specific function, we need to put that function in the open, where everyone can see it. Then, as the linked article demonstrates, we can compare the idealized and actually-desired functions and perform a real cost-benefit analysis.
Imprisoning people without trial, for predicted future crimes, is what's unfair. Letting an algorithm make the prediction instead of a judge only punctuates the injustice.
What is a "fair" criteria to base this decision on? Is it fair to throw someone in jail because they are young, or they got layed off, or they don't have friends or family? How is any of that better than jailing them for their skin color?
These are exactly the things that justice is supposed to be blind to.
I'm rooting for the algorithms here, simply because they make the inherent injustice of pre-trial detention harder to ignore. We can convince ourselves that this injustice is somehow corrected by the presumed wisdom and compassion of a human judge. But by formalizing the logic, we have to acknowledge that we are literally throwing people in jail for plainly unfair reasons.
Is there any circumstance in which you would support detention before trial? If not, would you oppose the detention of a serial killer caught in the act? If you do support pre-trial detention in some cases, how do you distinguish these cases from those cases for which pre-trial detention is unjustified? Could such a decision scheme be "fair" in principle? What would make it fair?
You could apply pre-trial detention to anyone accused of a crime, not offer bail at all. The upper classes might start to care more about speedy trials and police harassment once it became impossible for them to buy their way out of pre-trial detention.
I think detention before trial is always unfair in principle, but likely unavoidable in practice. I would like to see the issue acknowledged and taken more seriously, but it's a tricky problem and I have no easy solutions to offer. Practical mitigations may be the best we can do, which I'll grant may be expensive, non-trivial to implement, and allow more criminals to roam free. Here are some vague ideas off the cuff:
* Base decisions only on things that would be relevant in a trial, like evidence and criminal history.
* Nobody should be detained just because they haven't paid bail money. If we decide that someone can be released, it should be immediate and unconditional. The court should charge no more than they can immediately collect.
* Make detention as pleasant and convenient as possible for the accused. We should have facilities specifically for this purpose that are more like hotels than prisons, at least in principle.
* Eliminate any trial delays that aren't strictly necessary, i.e. due to congestion or beurocracy.
If a serial killer is caught in the act, there would presumably be enough evidence available at the bail hearing to justify detention.
Assume that you are insurance company with access to customers social media profiles.
You find out that if customer has a friend who posts pictures of fast motorcycles that increases the probability of customer making $100,000 claim in the future to 2% from previous 1%. Is it fair to double their insurance payment to make up for the increase in expected value even if the number of false positives is huge? General efficiency increases and customers can save money if they select their friends better. This might be how Chinese social scoring system might work in the future. Society as a whole optimizes itself.
What if instead of the insurance claim, the risk is homicide or rape. Should algorithmic judge take that into account when weighing the evidence?
Personally I think unconstrained algorithm in criminal cases is ethically wrong. Justice should be individualized or it's not justice.
Yes. That's the idea when managing insurance risk.
The question is if there are variables that should be excluded because optimizing trough them has harmful effects to the society and it's unfair. Smoking or alcohol use are noncontroversial variables. The quality of your friends, your political leanings are controversial.
This is a different problem than what people are complaining about. In fact it may be the exact opposite problem in some ways.
The problem you state is that, if you base an outcome on some measure, people will manipulate that measure. E.g. a companies use college education as a proxy for competence. And now everyone has to get a college education to get a good job. Society may be better if use of these measures is banned, even if they are predictive.
But this is not what's happening here. I don't see anyone claiming they changed their job, marital status, age, neighborhood, or any of the other measured factors. Just to manipulate this algorithm. These measures are probably safe to use in these circumstances, as far as this specific problem is concerned.
>Justice should be individualized or it's not justice.
There's no such thing as "justice". If you don't use an objective statistical algorithm to decide these things, you must use humans. And humans are far more biased and less fair. See my other comment.
Personal two cents: the actual maximally "fair" and maximally safe strategy is not to release anyone on bail. True positives are 100%, and false negatives are 0%.
Beyond that, I agree with nabla9:
> I think unconstrained algorithm in criminal cases is ethically wrong. Justice should be individualized or it's not justice.
110 comments
[ 2.8 ms ] story [ 151 ms ] threadYou want it fair based on race ? Ok, don't input race. Everything else stays the same. Why this complexity ?
That said, it'd be interesting to see how other "algorithmic bias" might play out in more or less racially homogenous, if not entirely ethnically homogenous populations --for example Rwanda, Finland, S Korea. Things like class, neighborhood, education etc.
[1]Given they themselves don't officially collect race information.
Or let's say the algorithm is deciding on insurance of some sort's premiums and someone is living next to a SuperFund site.
I don't want to punish people for living in polluted areas therefore I don't include that data in the training set.
>>Or let's say the algorithm is deciding on insurance of some sort's premiums and someone is living next to a SuperFund site.
I am not ok with doing that therefore I don't include address in the data.
Am I ok with punishing people for living in poor/dangerous neighborhood? No? Then don't include that in the data. Am I ok with people paying more for car insurance if they drive in a city with more cars/accidents/worse driving culture? Yes? Then I include it in the data there.
I'm not sure I can reconcile that.
1. in the first option, we're not applying the consequences random people, we'd be affecting those already found to have committed a crime.
2. in the second option, if we think we'd be affecting innocent people in option 1. why do we think we would not be affecting innocent people (good drivers who happen to live among poor drivers) in options 2?
That's to say, either both are right or both are wrong. I see the same effects in both.
There are several reasons for this from both technical and legal perspective.
It is incredibly easy to find statistically significant correlations given just a few (more than 7) different views of the data. In general these ml models are not working with less than hundreds or thousands.
If the model learned this suppose racial bias, once, you deleting this column is not going to stop it from learning it again, and I believe some research showed that it actually can make the unfairness more severe.
from a legal standpoint a company that may or may not be infringing on rights could just say, oh we can't be because we don't have these fields in our data: which makes it harder to monitor and audit wrong doing.
most of the methods that I am familiar try to ease the effects of the learned biases as a post-processing step for the model.
I would be interested in seeing examples. So far in this thread the arguments were along the lines of: "but then the algorithms punishes poor neighborhood instead of race" but you shouldn't have address in the data either as (I hope) nobody is ok on punishing people for living in bad neighborhood. We should only include data we would like to see in the explanation of the sentencing.
"You are not getting parole because you live in a poor neighborhood" is unfair while most people would be ok with:
"You are not getting parole because you willingly associated with people who committed crime".
Why ? Because crimes aren't committed by all groups equally ... so you wouldn't expect the risk to be equal.
Even for completely practical reasons there are differences. For obvious reasons a 1.4 meter individual is limited risk for armed robbery, to give an extreme example. But there's hundreds to thousands of things like that, that all randomly affect the distribution. The result is that when all put together, things are seriously out of whack.
Also I fear the reverse effect. Communities work, to some extent, because of lack of crime. Doing this will necessarily make those neighborhoods with the "victim" genes more violent, more criminal. The result will be improvement of those neighborhoods ? I have to say, I'm VERY skeptical.
What you should do is to keep the ratio of people classified as dangerous same among different groups. But this comes at a price as reduced classification accuracy.
Of course you don't include location. Why would anyone be ok with "you got longer sentence because you live in poor neighberhood"?
>>What you should do is to keep the ratio of people classified as dangerous same among different groups.
Sounds like a terrible idea. What if there is a community which just doesn't contain many dangerous people at all? If someone from this community commits a crime once in a blue moon they will be unfairly punished.
Now imagine if people were unfairly punished based on something they were born with and not where they lived.
No, the algorithm just gives negative weight to locations where dangerous groups tend to live. You infer race from that measure.
here is a good resource on that: https://arxiv.org/pdf/1606.08813.pdf European Union regulations on algorithmic decision-making and a “right to explanation”
Here's a very simple refutation:
Is past arrest history a valid factor in the model?
What if past arrest history is biased due to biased policing (Hint: it is, well proven)
How about zip code?
If you don't believe in using probability to determine risk. How about you hire a pedophile to babysit your kids? How about throwing down your retirement on lotto tickets? ETC.
Do you think biased policing causes people to commit crimes? How about not committing crimes to begin with? How about not hanging with troublemakers? You think it is impossible for people to avoid being arrested and convicted of crimes because of "biased policing"? Please think harder.
In other words, black defendants actually are more dangerous to release and there is no magic algorithm that bypasses this fact.
(The easy ones are male/female; racial differences are far less.)
Yes, but...
>>and there is no magic algorithm that bypasses this fact.
Maybe there is, it's just the method used wasn't able to find it either due to limitations of the method itself, not enough information or bias in the training set.
As a toy example: assuming you only have race and age to make your decision on then to optimize for public safety you need to include race to make good decisions. If you have race, age, number of friends who committed crime then maybe you don't need race anymore. The problem is that we are likely not getting enough data and then race is a proxy for that uncollected (and maybe uncollectable) data.
I understand that your argument is a toy argument so it doesn't make sense to discuss it specifically, but I feel it's important to point out that the issue here isn't just which variables are used to make the decision, but what the decision ends up actually being. That is, maybe you find that you can make a very good decision based solely on age and number of friends who committed crimes, and don't take race into account at all -- but then if this algorithm ends up yielding "yes" to most white people and "no" to most black people (even if your algorithm doesn't use race at all), you haven't solved anything. [edit: "solved anything" was poor word choice on my part, obviously you have solved something, but you remain in the state described by the paper]
Another issue is that while you can "whitewash" variables, it's very difficult to scrub race out entirely. For example, in practice we can't use "committed crimes" as an indicator because we can never know a ground truth: we'd have to use "were convicted of crimes" instead. Unfortunately, you're far more likely to be convicted of a given crime if you're black than if you're white, so you're already mixing race into your variables even if it isn't named. With the disparity in convictions, enforcement, etc., it's very, very difficult to come up with measurable signals that are not in some way already tainted by racial decisions.
It would actually solve the problem. It's ok if I give more "no's" to black people as long as black people are more dangerous in general. It's only not ok if I punish a specific non-dangerous black person just because they are black.
That's what fairness is: you get what you deserve because of your decisions and wrongdoing not because how you look or where you were born. That some groups end up with more convictions is expected and doesn't contradict fairness principle.
> Maybe there is
No, there isn't. We actually have a mathematical proof (which is quite simple) why this is impossible.
Specifically, following conditions can't be true at the same time: 1. groups differ in base rate 2. prediction isn't perfect 3. decision is correct at the same rate for groups 4. decision is correct at the same rate for groups, restricted to positive/negative class.
1 is a brute fact. Your toy example insinuates at 2. 3 is called calibration and what is usually optimized by machine learning. When people say algorithm is unfair, it usually means 4.
https://arxiv.org/abs/1609.05807
Even in your post you go from a statement that amounts to "statistically some groups have a larger number if dangerous individuals" to "black people are more dangerous". The two sentences do not mean the same thing!
The point stands: using statistics to meter out justice IMO amounts to collective punishment. Of course that leaves the question if whether more conventional methods are any better, but now we're opening up a new can of worms, namely what is the goal of criminal justice systems and how should those goals be achieved...
You cannot fix a problem by pretending it doesn't exist.
I'm not usually such a stickler for language, but in this instance it does seem to me that this kind of profiling serves to perpetuate that narrative of criminality being a feature of a group of people, and doesn't help to get at the causes.
(Not to mention that "public safety" still excludes the harms of centuries of slavery and decades of Jim Crow...)
You are right, but that's not the problem with the algorithm.
A critical assumption with large-scale data mining is that past trends continue - the problem is that the existing data fits the algorithm. It is just a conservative What-If decision maker operates on existing facts (i.e bad present day situation), just wrapped into code (or worse, encoded as opaque literal "biases" in a decision tree).
I see somewhat similar patterns in lending interest data (redline zipcode -> credit ratings) and the problem is that bigger the historical trend data, the less forgiving a "past trends" algorithm will end up being.
Since this ends up being a prisoner's dilemma, if you are a rational actor in this system and the system keeps playing a defect card on you, then the obvious move is to always defect - cut your losses.
Algorithms can't improve the job prospects of the people released. And that's not the algorithm's fault.
And therefore, the you're right - the algorithm can't change the world beyond its output result.
Not necessarily. It would be easy to use only recent data and ignore data from decades ago. The prediction accuracy wouldn't go down much as long as there is a large enough sample size in the recent data. It may even go up if the old data really is inapplicable now.
Then if a past trend stops happening, the old data gets purged eventually and only the post-trend data is considered.
It's the same root idea: the goal of the system is not to find the person who committed the crime. The goal of the system is, for each crime, to find a person who can be nailed for it. And you can manufacture a class of people such that whenever you need an offender you can go grab some of them and stand a good chance at getting a guilty plea or a conviction.
You start by selectively hyper-enforcing small violations against a chosen subset of the population. Get them for minor traffic violations (which you can ensure turn into arrest warrants by setting the fines and fees and other costs high enough!), get them for "paraphernalia" offenses (where you assume everyday objects in Person A's possession are evidence of drug habits while they wouldn't be assumed evidence in the possession of Person B!), get them for all sorts of things.
Now whenever there's a crime you just go pick some random people from the demographic you've been doing this to, figure out who you can bribe to testify against whom, and then march into the courtroom with witnesses and a defendant who has a thick file of previous "encounters" with the system, and off to a cell they go for a while.
And it gets even easier each go-round: people with criminal records don't usually have the resources to move on and rebuild their lives, so once they're released they're going to be right back where you found them last time. And they've got an even longer record now, which is just proof that you've been doing a great job in figuring out who these dangerous recidivists are! The model works!
I know it goes against some cherished beliefs, but this really is how much of the US handles "justice".
Was interesting, and worth a look.
That being said, the authors' point seems to be right there in the introduction - "it's not fair to hold all individuals to the same standard, irrespective of race."
In other words, the broader community should tolerate laxer pre-trial sentencing standards applied to people of race A, precisely because they commit more crime than people of race B?
Pretty odd concept of "fairness" if you ask me.
Also interesting that explicitly adjusting for race does not affect recidivism predictions by the COMPAS model. In other words, as it stands, the model is not inadvertently discriminating on the basis of race. I wonder if the same testing has been applied to sentencing decisions.
Does "they" they mean "each individual of Race A", or the subpopulation of "Race A"? Those are two very different meanings, and the heart of the problem.
Why should I be jailed because other people with similar skin color to me committed crimes?
Should we go digging to find whether Irish or Russian caucasians have higher than average crime rates, and then refuse bail to all Irish or Russian-descendant defendants?
People are being placed in pre-trial detention because they're flight risks, as determined* by their marital status, whether or not they're unemployed, accused of a violent crime, with criminal records, etc.
As it turns out, people from race A overwhelmingly meet these criteria, leading to outcomes that the authors believe are unfair.
In fact, the authors are suggesting that race should be explicitly considered in these decisions, in order to balance intra-racial representations.
Given two people accused of the same crime, with the same job, marital and criminal history, the authors would detain one and release another, purely on the basis that one is white and one is black.
As I said, I don't think many people would agree with that definition of "fair."
* I don't know exactly what the model inputs are, I just made these up for demonstration purposes.
Also from the abstract: "In some cases, black defendants are substantially more likely than white defendants to be incorrectly classified as high risk."
The point is that an AI is a black box that may not explicitly use race but since it's using a variety of criteria rather opaquely, it may effectively, indirectly, use race, the neighborhood someone lives in, their social status or all sorts of things that aren't fair based on "your personal circumstances should determine whether you are considered a risk".
The authors propose a system to mitigate the problems here, though I actually don't really think that's the solution - these AIs should simply be abolished and replaced by objective criteria.
But that's not really true: https://www.chrisstucchio.com/blog/2016/propublica_is_lying....
>these AIs should simply be abolished and replaced by objective criteria.
What exactly is "objective criteria"? Previously human judges made these decisions. They are far more biased than any statistical algorithm.
Interestingly, if this Algorithm was a human, they would be called a racist by all on the left. Instead, the best than can be managed is "unfair".
Objective truth hurts sometimes. Perhaps some apologies are owed to the so-called racists out there. Turns out they were just being as honest as a probability algorithm (shame on them).
If AI an robots ever police our society, leftists will have to tamper with their highly accurate, objective statistical analysis of who to pull over for questioning to save them from their racism. That is already what is going to happen to this algorithm.
Machines exposing emotion initiated dishonesty (until they are beaten into social justice submission and artificial guilt subroutines added to make things "fair").
I did see that in the abstract, but then the only follow-up in the actual body of the paper was:
"...among defendants who ultimately did not reoffend, blacks were more than twice as likely as whites to be labeled as risky."
Which is simply restating the premise in a roundabout fashion - i.e. that those with criminal records (etc) are more likely to reoffend, and black defendants are more likely to have criminal records. It's a bit of a long bow to draw to say the classification was "incorrect" post-hoc because "those from the risky group who did not reoffend were initially labelled as high risk".
> The point is that an AI is a black box that may not explicitly use race but since it's using a variety of criteria rather opaquely, it may effectively, indirectly, use race, the neighborhood someone lives in, their social status or all sorts of things that aren't fair based on "your personal circumstances should determine whether you are considered a risk".
I believe it's a straightforward logistic regression, which is about as objective as you can be. It's not "AI", it's not opaque and it's not indirect. It's a simple calculation of the relative odds of recidivism based on factors such as criminal history, employment, and so on.
Given that adjusting for race makes no difference to the model output, it is literally the most colour-blind approach you could take.
The authors, on the other hand, would have you treat two people differently purely on the basis of respective race.
Now obviously there are historical/legacy social issues at play, not to mention the broader question of what's "best" to eliminate disadvantage. But I suspect not many people will agree with the authors' that this would be "fair".
If I were pushing their line of thinking, I would be researching whether this type of "positive" discrimination actually reduces the race gap long-term.
That's ultimately the only thing that matters, and in my mind, it's a much easier sell to say "this isn't fair, but it's what we need to do to reduce the criminal overrepresentation and disadvantage that exist because of our history with slavery".
Meaning your original comment was disingenuous. Good job.
Define "race".
Well, if you don't take the authors' approach, you don't need to. You just continue assessing pre-trial detention on the basis of each individual's background.
> As noted above, a major criticism of COMPAS is that the rate of false positives is higher among blacks than whites [2].
Policing in America is racist. So anything that takes arrests into account (prior arrests, recidivism, etc.) will be in error as well.
The line you quoted misses a lot of context - the 'standard' is an algorithmic risk score with a single threshold. The paper mentions why having different treatment by race might have legal problems (e.g 14th amendment). So this standard is what we're stuck with. The paper asserts that it isn't fair.
This is really the clincher. If the algorithm is effectively aggregating multiple human decisions, it ought to correct for known biases in those decisions.
Citation needed. The fact that certain ethnic groups have problems with crime and violence can be attributed to the fact they grew up in single parent households, are less educated, and have less income than other ethnic groups, all things that contribute to a higher crime rate.
> The paper mentions why having different treatment by race might have legal problems (e.g 14th amendment).
I don't see how anti-slavery has anything to do with this. If you are convicted of breaking the law, as a citizen you will be subject to the same punishments as any other citizen.
Citation needed. Cops visit black neighborhoods more, sure. But they do that because they commit vastly more crime. Which is the cause and which the effect?
It's also a lot more complicated than that. People throw out studies that blacks are more likely to be caught for drug crimes. But they are less likely to be caught for more serious crimes. Possibly because they trust police less and are less likely to volunteer as witnesses. E.g. https://www.wsj.com/articles/the-underpolicing-of-black-amer...
Citation needed
Citation needed
It's would be difficult to have a discussion if you had to provide citations for every single well known fact.
A. Blacks commit vastly more crime.
B. The sun goes up in the morning.
You can argue for being racist if you want but you can't say "it's not racist, because 'racist' is a word with bad connotations and I think what they do is good" without being called out on it.
No one is doing that. The algorithm does not take race into account. It's just an assertion that it uses other factors as a proxy for race. I've not seen this proven anywhere.
And indeed, if it were true, you would have to admit that blacks are more likely to be criminals. Even after controlling for all relevant variables. Which goes against the standard narrative that it's just socioeconomic status or whatever.
>you can't say "it's not racist, because 'racist' is a word with bad connotations and I think what they do is good" without being called out on it.
It isn't racist. And the word racist is becoming almost meaningless because how often you overuse it.
Quite possibly - and if that were the case, it's certainly something to be addressed. As a non-American, I can't really comment.
But the paper doesn't really deal with that. It's saying that society should tolerate higher rates of recidivism amongst one racial group in the name of fairness.
I do understand the authors' thesis - I'm just saying that "fair" is a very subjective concept. I don't think many people will agree with the authors' definition.
There are so many factors in morality that I think individually, case by case judgments are likely the best option. Racial / class bias seems to be just as present in these algorithms, so lets at least allow for human intervention rather than forcing ourselves to be constrained by an inherently unfair system. Yes, humans are flawed too, but perhpas we should be spending more time trying to adjust / account for those in different ways.
Note: Bias in sentencing/convictions is a huge problem that should absolutely be worked on. I'm only claiming that these approaches are inherently flawed, not that what we have now is anywhere close to good enough.
What are the inputs to "case by case judgments" except Bayesian priors and observations about a specific case? Your brain isn't doing anything a computer model can't in principle do. My point isn't that any particular bail pricing model is acceptable. Instead, I'm arguing that a desire to escape "statistics" by giving up on models and punting to the brain is futile, since the brain is just going to use its own statistical model whether the owner of that brain is aware of it or not --- it must, since statistics is baked into the structure of knowledge, and the brain has no private source of truth.
There is no accessible realm of knowledge about the real world somehow exempt from statistics, since we never have complete knowledge, and statistics models our uncertainty.
Also, humans draw from an absurdly complex pool of inputs, usually called "common knowledge", that so far has given us a hard time when trying to replicate it in computer systems. So while in principle there is, imho, nothing stopping a statistical method to produce decisions at least as good as a human could, in practice this might not be the case.
Another issue I have is that we don't know is whether statistical bias exists in the metrics they train on, something they discuss in the paper. We already know for example blacks are arrested and convicted harsher than whites for some of the similar crimes[0]. The paper just says that COMPAS predicts likelihood of violent crime, not whether that means they'll be arrested of crime. Moreover while Table 1 talks about a prediction regarding violent crime, Figure 2 talks about recidivism with regards to all crime, so which is it? It still seems nebulous to me what COMPAS actually is for.
I'd have to read it deeper than I have, but not sure how I feel about the paper.
[0] http://www.politifact.com/truth-o-meter/statements/2016/feb/...
2. "Your brain isn't doing anything a neural network can't do." Really, now?
3. The paper mentions "over 100 factors" the algorithm uses. Ad agencies have way more than 100 factors on me, and I still get irrelevant ads every day.
Humans aren't perfect, but they can catch things that stats won't, and vice versa.
They're both Turing machine approximators after all. Granted, ANNs aren't quite sophisticated enough to match the brain yet, but this capability gap is a difference of degree, not kind.
But I don't think the similarity of the brain to ANNs really matters to my argument, as interesting as this discussion might be --- so I've edited my original comment to make it more general and to focus on my higher-level point.
> I'm arguing that a desire to escape "statistics" by giving up on models and punting to the brain is futile
I think we'd all agree here. The point is that both methods should always be used together, because each has huge limitations on its own.
The brain is applying some function over its inputs, after all, and we can apply that same function, either explicitly in code or implicitly via the training of some kind of model.
Machines are already better than humans at other tasks for which we imagined humans exercised fine and subtle judgment: https://www.sciencedaily.com/releases/2016/04/160421133831.h..., so I see no reason to believe that bail-setting is some unapproachable problem domain.
> The point is that both methods should always be usned together, because each has huge limitations on its own.
I disagree. Putting a human in the loop just invites those humans to inject unstated, socially-determined biases into the decision system, and fairness and transparency dictate that we be up-front about those biases. I see no evidence that a computer model couldn't reproduce biases we considered desirable.
Hormones, reflexes, instincts, empathy, etc. Again, human inference and learning are (as of yet) insurmountably more complex than statistical regression.
> regression (which I wouldn't characterize as "relatively simple"
I encourage you to study brain chemistry, and compare that with any stats course ever.
> Why would the use of some kind of regression (which I wouldn't characterize as "relatively simple") a problem if the overall result is nevertheless more accurate than the human equivalent? We judge models by their results, not their complexity.
Because there is no way to universally judge "model results". https://en.wikipedia.org/wiki/Confusion_matrix
Even a 99% true positive rate + 5% false positive rate can be worse than a 5% true positive rate and a 0% false positive rate, when you're talking about irreversible decisions.
> The brain is applying some function over its inputs, after all, and we can apply that same function, either explicitly in code or implicitly via the training of some kind of model.
Yes, that's very true. Executing the model is easy, and maybe even works similarly in NNs and brains. Figuring out that "some kind" part is what we formalise as "learning", and turns out is the hardest part of this equation.
Again, clearly, current regression methods are nothing like human learning. Else, they'd be able to learn a language like a human, without seeing billions of annotated data points.
> Machines are already better than humans at ...
Neither the article nor the paper claim the machine performs better. Their strong claim is that the machine is faster.
> I see no reason to believe that bail-setting is some unapproachable problem domain.
I think justice is a more important domain for many people. My guess is it's because it can set precedent.
> Putting a human in the loop just invites those humans to inject unstated, socially-determined biases into the decision system
For drastic contrast: unstated, socially-determined biases is what prevents nuclear wars. https://en.wikipedia.org/wiki/Stanislav_Petrov
More plainly: people have compassion, empathy, and can understand things about convicts that are (still) very difficult to encode into a computer.
Complexity doesn't mean better. Humans are absolutely garbage at statistical thinking. Even simple statistical regression almost always beats human experts. Let alone more modern methods. E.g. http://zaldlab.psy.vanderbilt.edu/resources/wmg00pa.pdf or http://lesswrong.com/lw/3gv/statistical_prediction_rules_out...
To be able to decide if something is fair, one has to be able to understand the process by which a decision is made, no? And not just a woolly, after the fact justification that a person might well invent to explain their decision. If we can't reliably look at the inputs and determine what the outputs would be, how are we able to decide if the process is 'fair'?
As someone who is looking to understand a system, the argument of "But it couldn't make a human decision" triggers alarm bells of "What you mean is: It couldn't make a non-rational, unfounded, biased decision influenced by whether the person is hungry or not". Is that how important decisions should be made?
Is this the sort of hormonal influence you find desirable? http://www.economist.com/node/18557594
All these factors you mention can be modeled as constant or contingent biases applied to the inputs. There is no magical aspect of the brain here. To the extent that the brain is using unstated factors to make decisions, its decision making is unfair.
> I encourage you to study brain chemistry, and compare that with any stats course ever.
You're hand-waving. This is not a useful discussion technique. If you have a point about how some aspect of brain chemistry produces decision outputs that are difficult to encode, please make it. Otherwise, please stop this aspect of your argument.
That something is "insurmountably more complex" than some other thing doesn't make it better than that thing. Sometimes, it's just insurmountably more complex for no good reason. The blind watchmaker doesn't build elegant watches.
> For drastic contrast: unstated, socially-determined biases is what prevents nuclear wars. https://en.wikipedia.org/wiki/Stanislav_Petrov
Unstated, socially-determined biases prompted Europe to slaughter itself for 200 years in absolutely pointless wars of religion. So what?
Nope. I said nothing about desirability. The question was: "What is the brain using except conditional probabilities derived from past inputs?"
> All these factors you mention can be modeled as constant or contingent biases applied to the inputs.
It's possible, but we can't do it yet. If you have proof that we already can, please share it.
> To the extent that the brain is using unstated factors to make decisions, its decision making is unfair.
Fairness is human-defined, and difficult to encode in a computer. I conjecture that human + computer will always result in more fair and safe results than either one in isolation.
> You're hand-waving.
Yep. The number of variables (and their interactions) affecting a human decision far surpasses the number of variables in any regression method. In a lot of literature, number of variables characterises complexity.
> So what?
My example is empirical evidence of a human element identifying a false positive better than a computer. Europeans fighting wars is not such an example.
Philosophically speaking, I think the "key component" that humans bring, besides still-superior decision-making in many cases (not all), is creativity. So far, I haven't seen any machine learning that can outperform humans in light of limited brand-new data.
Those researchers also designed a simple predictive model using only two factors as inputs: age and number of previous convictions. By comparison, COMPAS uses more than a hundred. To me, it doesn't sound like those extra factors end up contributing much.
[1]: https://www.economist.com/news/science-and-technology/217349...
Not paywalled.
That's the nature of these things. If you use zero factors then you're basing decisions on random chance. Use one good factor and you'll do 60% better than random chance. Add another good factor and you'll do another 10% better. Every factor after that is a single digit improvement.
Because once you have something pretty good, making it even better is diminishing returns. But at the same time, "it's only 1% more accurate that way" when you're dealing with country-sized populations will change the outcome for many thousands of people.
And how does using fewer factors help anything? The underlying "problem" is that people of different races have different outcomes in actual fact, and any halfway accurate predictive algorithm will reflect that difference. It's not as if there is no racial disparity in the number of previous convictions.
If anything, using fewer factors could make the problem worse because it stops considering potential mitigating factors. A young black man with two prior convictions gets the shaft even if he has recently gotten married and bought a house in a better neighborhood.
So when you ask 'x' people to independently judge something, let alone something that is multiple choice with a correct answer, you're going to get answers that are far more accurate than a single individual would give you. So looking at the average answer of 'x' people, comparing it to an AI system, and arguing that they're relatively close so individual people are relatively close to the AI is completely fallacious.
---
As a tangential aside, there's another quirk to the wisdom of the masses. When you let the people communicate and try to intelligently organize and use expertise to come to answer, this effect disappears and the final answer again tends to be very wrong. Kind of an interesting perspective on the current zeitgeist of society and work.
===EDIT===
The authors were obviously aware of the wisdom of the masses. Quoting the paper itself:
To determine whether there is “wisdom in the crowd” (7) (in our case, a small crowd of 20 per subset), participant responses were pooled within each subset using a majority rules criterion. This crowd-based approach yields a prediction accuracy of 67.0%. A one-sided t test reveals that COMPAS is not significantly better than the crowd (P = 0.85).
That's a quite silly p-value and on top of that I'm not sure how they claim their system actually controls for the wisdom of the masses.
Only if people are an unbiased estimator!!
For another example of it consider things like the television show, 'Who wants to be a millionaire?' It's a quiz show where one of the choices is for the participant to ask the audience. And the audience tends to do absurdly well on even the most esoteric questions, though independently they are certainly far from trivia experts. But very few are randomly guessing - their own experiences and biases leads them to entirely different conclusions. Yet somehow it produces the correct result time and again.
It's a strange phenomena and one that has to be constantly guarded against in anything involving sampling of people. This is a text book example of a study that gets destroyed by it.
>> Only if people are an unbiased estimator!!
> You'd think so, but that's not correct.
Either you're misinterpreting the technical term "unbiased estimator" here, or you are aware of some research that I would like to read.
In context, "unbiased" means that if you pick people at random and ask for their estimates, then on average the too-high estimates cancel out the too-low estimates (i.e. there is not a bias in one direction or another).
But people as a whole have poor understanding of many things. One common one that appears in social science research and is often replicated is that people grossly overestimate the size of the homosexual population in the US (the "wisdom of the crowds" often estimates it around 20% whereas best available polling data suggests 3-5%). Here's just one source for this phenomenon
http://news.gallup.com/poll/183383/americans-greatly-overest...
"Wisdom of the crowds" is occasionally reliable, but it should not be assumed to be reliable for any particular problem without verification. It often fails terribly even on problems that are not very esoteric.
Edit: changed phrasing of final paragraph
Where it works phenomenally well is in areas where biases have not been directly instilled into people. This does not mean people are unbiased, however. Again the knowledge of trivia is a good example since while the crowds can generally do phenomenally well even at very esoteric questions where biases would lead them to individually come to very different conclusions, yet they will invariably fail to answer ostensibly trivial questions like 'What is the capital of Australia?' You'll get Sydney, it's not. You'd likely get a similar result for things like the capital of Pennsylvania.
In a way I view the wisdom of the masses as analogous to machine learning systems. They do an oddly good job of providing extremely precise answers to a wide array of questions even when trained with models that do not directly represent the 'questions'. Yet you can also break the systems, at times comically, with certain types of queries designed to do precisely that.
And as was the case with machine learning for quite some time, I think people remain reluctant to utilize it due to the black box nature of it. The implication of your comment is that the wisdom of the masses is little more than incorrect answers canceling out on average leaving nothing but a survey of experts. Yet I think there's no evidence for this (even if it may be a perfectly logical 'kneejerk' reaction) as it works even on things where nobody is an expert, and if this were the case then we ostensibly should be able to get comparable answers from coordination - yet coordination causes the entire system to collapse.
You are simply mistaken in your interpretation of the technical term "unbiased estimator". This has a specific meaning in statistics, and is required for the convergence property you specified earlier. From wikipedia [0]
"In statistics, the bias (or bias function) of an estimator is the difference between this estimator's expected value and the true value of the parameter being estimated."
In lay terms, this means that the estimator process "ask lots of people and average the result" is unbiased only if all the too-high errors, in aggregate, cancel out the too-low errors.
[0] https://en.wikipedia.org/wiki/Bias_of_an_estimator
"The implication of your comment is that the wisdom of the masses is little more than incorrect answers canceling out on average leaving nothing but a survey of experts. Yet I think there's no evidence for this (even if it may be a perfectly logical 'kneejerk' reaction) as it works even on things where nobody is an expert, and if this were the case then we ostensibly should be able to get comparable answers from coordination - yet coordination causes the entire system to collapse. ... "
I'm not sure if you even realize all the assumptions you're making. You are assuming, for instance, that 'guesses' are regularly distributed. That assumption may be correct in some cases - I expect in many it is not. Alternatively there is the possibility is for you to claim that you're referring not to the individuals in question as the estimators, but the entire group. In that case you've spent a lot of time saying nothing as it boils down to "people are only correct if they're correct."
https://www.scientificamerican.com/article/economic-inequali...
The "wisdom of the crowds" is pseudo-nonsense.
Humans are terribly biased. Racial bias isn't even anywhere near the strongest bias we have. Unattractive people get sentences twice as long as attractive people. Judges give far harsher sentences before lunch, when they are hungriest. Socially awkward people seem to be pretty strongly discriminated against. Studies have found people discriminate by politics even more than race. Job interviews have actually been shown to degrade performance over just judging resumes. Before statistics and credit ratings, getting a good loan required being an old friend of the banker.
It's not just that humans are unfair. We are objectively terrible. Very simple statistical algorithms beat human "experts" in almost every domain they get tried on. As early as the 1920s, a statistician came up with a formula that was better at predicting recidivism than a group of 3 prison psychologists.
Simple linear regression has predicted the success of medical treatments better than doctors, diagnosed psychoticism better than trained psychiatrists, predicted academic success much better than admissions officers, predicted loan risk better than bank officers, etc, etc. To say nothing of modern machine learning methods on modern computers. It's insane we allow humans to continue doing these tasks at all.
But there has been huge resistance to algorithms in every domain. From people who stand to lose their jobs and be put to shame by them of course. But also even outsiders tend to reject algorithms. And overly trust humans. Psychologists have actually studied this. They call it a bias labelled "Algorithm Aversion". http://opim.wharton.upenn.edu/risk/library/WPAF201410-Algort... The study showed that humans were willing to forgive the mistakes of humans far more than those of algorithms, even when the algorithm made far fewer of them.
This is why they aren't everywhere already. The last thing we need is fear mongering like this. As shown, humans are far worse. If an algorithm shouldn't do it, then a human certainly should not.
A big part of the case these people make is a reference to a propublica study that found a slight bias against race in an algorithm once. Yet that study wasn't peer reviewed. The findings weren't statistically significant. https://www.chrisstucchio.com/blog/2016/propublica_is_lying....
Also, A.I. ‘Bias’ Doesn’t Mean What Journalists Say it Means: https://jacobitemag.com/2017/08/29/a-i-bias-doesnt-mean-what...
That's really the beautiful thing about our shift toward algorithms: we'll no longer be able to rely on these subtle social pressures. If we want to optimize a specific function, we need to put that function in the open, where everyone can see it. Then, as the linked article demonstrates, we can compare the idealized and actually-desired functions and perform a real cost-benefit analysis.
What is a "fair" criteria to base this decision on? Is it fair to throw someone in jail because they are young, or they got layed off, or they don't have friends or family? How is any of that better than jailing them for their skin color?
These are exactly the things that justice is supposed to be blind to.
I'm rooting for the algorithms here, simply because they make the inherent injustice of pre-trial detention harder to ignore. We can convince ourselves that this injustice is somehow corrected by the presumed wisdom and compassion of a human judge. But by formalizing the logic, we have to acknowledge that we are literally throwing people in jail for plainly unfair reasons.
* Base decisions only on things that would be relevant in a trial, like evidence and criminal history.
* Nobody should be detained just because they haven't paid bail money. If we decide that someone can be released, it should be immediate and unconditional. The court should charge no more than they can immediately collect.
* Make detention as pleasant and convenient as possible for the accused. We should have facilities specifically for this purpose that are more like hotels than prisons, at least in principle.
* Eliminate any trial delays that aren't strictly necessary, i.e. due to congestion or beurocracy.
If a serial killer is caught in the act, there would presumably be enough evidence available at the bail hearing to justify detention.
You find out that if customer has a friend who posts pictures of fast motorcycles that increases the probability of customer making $100,000 claim in the future to 2% from previous 1%. Is it fair to double their insurance payment to make up for the increase in expected value even if the number of false positives is huge? General efficiency increases and customers can save money if they select their friends better. This might be how Chinese social scoring system might work in the future. Society as a whole optimizes itself.
What if instead of the insurance claim, the risk is homicide or rape. Should algorithmic judge take that into account when weighing the evidence?
Personally I think unconstrained algorithm in criminal cases is ethically wrong. Justice should be individualized or it's not justice.
In some other cases it might be justifiable.
The question is if there are variables that should be excluded because optimizing trough them has harmful effects to the society and it's unfair. Smoking or alcohol use are noncontroversial variables. The quality of your friends, your political leanings are controversial.
The problem you state is that, if you base an outcome on some measure, people will manipulate that measure. E.g. a companies use college education as a proxy for competence. And now everyone has to get a college education to get a good job. Society may be better if use of these measures is banned, even if they are predictive.
But this is not what's happening here. I don't see anyone claiming they changed their job, marital status, age, neighborhood, or any of the other measured factors. Just to manipulate this algorithm. These measures are probably safe to use in these circumstances, as far as this specific problem is concerned.
>Justice should be individualized or it's not justice.
There's no such thing as "justice". If you don't use an objective statistical algorithm to decide these things, you must use humans. And humans are far more biased and less fair. See my other comment.
Beyond that, I agree with nabla9:
> I think unconstrained algorithm in criminal cases is ethically wrong. Justice should be individualized or it's not justice.