This worked really well in New Jersey. However, California's implementation of county-by-county standards as opposed to a statewide system could still leave a lot of room for error.
Specifically, even if you're allowed by Article 22 to perform automated decision-making (and decisions on bail that are mandated by law, as in the case at hand, would be allowed) must still:
* provide meaningful information about the logic involved in the decision-making process, as well as the significance and the envisaged consequences for the individual;
* use appropriate mathematical or statistical procedures;
ensure that individuals can:
* obtain human intervention;
* express their point of view; and
* obtain an explanation of the decision and challenge it;
Not OP, but they never claimed that what California is doing is unlawful because of GDPR. Their comment is still insightful because it shows an example of legislation that, if applied, could ensure more transparency and humanity to automated decision making.
The lack of such legislation becomes more glaring as more and more examples are demonstrated.
A little over a year ago I was the victim of some crimes including:
-kidnapping with a deadly weapon
-assault with a deadly weapon/firearm
-car jacking with a deadly weapon
The defendant has bonded out twice in the case. The first time there was only a charge of grand theft of my car, and bond was revoked after the investigation and the other more serious charges were filed. Thereafter, the defendant bonded out again after all the charges were filed, and posted $300,000+ bond. As the victim (and a lawyer) I knew the defendant would be granted bond (although I believed the defendant was both a flight risk and danger to the community and bond should have been denied), so I requested gps tracking. Less than a week later the defendant was rearrested for violating the terms of bond...all based on the gps tracking.
I’m not sure I trust an algorithm to determine if someone is a flight risk or danger to the community. Historically, the cash bond was like the carrot and stick to making sure a released defendant showed up to court. I think technology, like the gps tracker, can probably replace cash bond in almost all cases.
I think on the first violation/revocation of bond the defendant was arrested by police (but the defendant was in another county at the time, so they had to extradite back to the county). The 2nd violation/revocation of bond, I think the warrant was executed by police (but in the same county as the case).
But gps trackers, paid by the defendant, will then become just another bail bond: rich people buying their way out of jail. Then all the evils are back (corporations wanting more people arrested etc). If we are going to rely on gps it needs to be free, or even state-owned.
>But gps trackers, paid by the defendant, will then become just another bail bond
Why do you think a gps tracker would be that expensive? And why do you think a defendant would have to foot the entire bill? It's not like they are a one-time use piece of equipment.
You're the one saying they cost an astronomical amount that would preclude poor people from affording them, but that seems refutable by the fact that GPS trackers are commercially available for truck fleets[0] as well as tracking pets[1]. I see nothing over 200 dollars. So show some sources if you really think GPS trackers would be prohibitively expensive as currently deployed.
No. Look up how much ankle bracelets cost for people out on bail in the real world today. This is not a free market. People are not pets. There are also incidental costs for things like equipment deposits and expert opinion letters.
Ankle bracelet monitoring costs 4 to 6 thousand dollars per year. Poor people cannot afford that, makinging it little different than the previous bail system.
Or 15$/day from this company. 15 is the minimun "on a sliding scale" dependant on defendant income.
Our courts here in New Mexico got bamboozled by the use of the "Arnold Tool"which is basically the same thing. Anyways, Yada Yada Yada, we're now worst in nation for auto theft because of it and we have no recourse against it unless we want to change our state constitution because our Supreme Court mandated it as it was sold as being more fair. All it did was put repeat offenders on the street the next day. My car has been broken into three times since the change. Garbage in, garbage out.
The Arnold Foundation model was implemented in New Mexico after May 2017[0]. In 2016 Albuquerque was No.2 nationally in Auto Thefts[1], in 2017 (before the model was even in-use) you had risen to the No.1 spot[2].
So this seems quite disingenuous. Auto thefts were a HUGE problem before this, you were No.1 in the nation, and it has stayed that way for six months after the Arnold Foundation model was enacted.
But instead of looking at long term trends you're blaming something that is barely one year old?
Machine Learning Algorithms are severely racist towards defendants. We know, for example, that AI algorithms mistake Black people for primates... but they are no more racist than the criminal justice system at large. And they have a better chance of getting fixed.
Man - to hell with this forum, I ask for evidence on a claim that other posters have pointed out is a misapplication of other research 'webscam algos != cash bail algos'
And I get downvoted!?
This is tech news, something the general public can get easily manipulated into...and for demanding a degree of reasonable discourse and reference, i get downvoted?
Bleh, this place is turning into a place i cant speak relatively freely in.
> It turns out when you design the system specifically not to be racist, it isn't racist.
This is key, New Jersey went out of it's wait to avoid using anything that could be a proxy for race like zip codes. They also used big data to come up with the algorithm, the algorithm itself[1] is pretty strait forward and is not a black box.
The fear in California is that some counties will use black box ML systems that WILL have bias even if it's not intentional, just like how biases in the training data of image recognition tend to misidentify pictures of black people more then white people.
Bond is severely racist too, if you take those criteria. Black people get assigned higher bond, and often have a harder time getting it together. As such there are many people in jail that are in no way a risk but are held because they can't afford bond.
From this perspective, bond-based pretrial detention is actually better -- communities can organize bail funds to reduce its impact. You can't really fight algorithmic detention in the same way.
Our awareness that machine learning algorithms have the potential to make racist decisions gives us something to watch out for as these systems roll out. Hopefully addressing any flaws will be a matter of ordinary debugging.
Regardless, the cash bail system is terrible and I'm anticipating big improvements as we move away from it.
There's this growing trend of... "Well look, this person in a lab coat with a clipboard did some complicated things with a computer, using all the trendy words like 'algorithms', 'machine learning', and 'big data'! Their results must be very accurate and unbiased, and much better than a human looking at things."
I'm more and more convinced that most of these cases are just stuffing an excel spreadsheet into a black box, and shaking it. Hearing that we're using "algorithms" in criminal justice doesn't give me warm and fuzzy.
Up until now we were saddled with the extremely biased and unjust bond system. Calls to make the algorithms transparent are well taken -- but let's not go back.
Your thinking is very dangerous. Even if the algorithm is transparent and fair, that does not necessarily equate to a step forward because of the bias already present in the system. The algorithm will be[0] applied equally, sure, but that does not mean outcomes will be equal.
For instance, consider that owning a major asset probably decreases flight risk. That seems fair. But, this means that poor people are probably more of a flight risk than a middle- or upper-class homeowner. So if the algorithm considers this, poor people would be disproportionately affected.
Also consider that having family abroad or being from another country probably increases flight risk. Hence, immigrants would be disproportionately affected.
But now, when there is an unjust outcome, there isn't even an option to avoid jail time (even if that option is difficult to reach or undesirable); and the human responsible for the decision to incarcerate can simply wash their hands of it, since the algorithm gave the suspect a grade of "D". No need for empathy, sympathy, or guilt.
Then there are the pending discussions about what is 'fair' and the answer to those questions is politics...
Those critiques are all indeed things to watch out for, but progress is possible -- see New Jersey.
The status quo is a roaring injustice and ongoing disaster, and by insisting on the perfect over the good you are condemning millions of people to continue suffering indefinitely. How is your thinking not "very dangerous" to all those wronged day after day by the current system?
You've completely misunderstood my point if you think I'm advocating for the perfect at the cost of the good. I'm advocating caution rather than a blind faith that this solution is ipso facto forward progress, that just needs some tweaking.
The danger here is doubling down on the existing injustices, removing the limited accountability and positive exceptions that exist, and effectively sweeping conversation under the rug or arguing ideologies about the fairness of tweaks to the algorithms at the cost of real human lives.
You're making a distinction without a difference -- whether it's because of "caution" or whatever, no progress will be made if you have your way.
According to you the solution lies in "politics". Yet, this law is the product of California's government, a political body -- is that not legitimate "politics"?
So what would make for a "political" solution that would qualify for your stamp of approval? Chasing a consensus with the "tough on crime" crowd who benefit from felony disenfranchisement disproportionately suppressing minority votes? No such consensus will ever happen.
Again, you either misunderstand or misrepresent my points.
> According to you the solution lies in "politics".
My post was critical of the role of politics in determining the fairness of the algorithm. It removes the human lives factor and provides a platform for politicians and ideologues ("tough on crime" or otherwise) to either do nothing, argue about the ideology of fairness, or ignore ground truth about outcomes and pre-existing bias in the system.
The magnitude of bias being applied today is so widely distributed as to be essentially immeasurable, so I don't know if algorithm-driven policy will leave us any better or worse off.
In an open system, the bias could be identified, a patch created and approved, and a new release deployed. As biases shift, more patches are made. Ideally we find smaller and smaller "bugs" as time goes by.
Alas, we are seemingly very far from that today. There is probably great value in controlling that algorithm, and like gerrymandering district maps, the politicians have no incentive to create such an open system. I fear that algorithm-driven bail might become just another tool for politicians and power brokers to bludgeon their opponents with.
True. It is however a first step, and we can look at the results of the current implementation and adjust as needed. I imagine academics will be watching closely in particular.
That's always been the challenge in the current system, because you're leaving responsibility up to individuals it was a hard system to adjust even if it isn't working fairly (e.g. you aren't going to just randomly fire judges because minorities got a slightly unfair deal).
I’m optimistic the new sysyem will be fairer to people who can’t afford bail —but I do think all should get GPS.
I’m curious if the system does find that poor people (or rich, but one or the other) behave such that the system’s results are unexpected, in practice, what will politicians do?
This is the real danger of AI. Not that it will out think humans, but that humans will place so much faith in it that they will choose not to think for themselves.
I think there are valid reasons to not trust an algorithm to determine
whether someone is a flight risk or danger to the community.
But there was already an algorithm that came up with the cash bail amount for a defendant, which was purportedly based on the defendant's flight risk and likelihood of endangering the communoty.
I don't pretend to be an expert so I may be wrong but I think the algorithm boiled down to something like "let the judge decide".
I think transparency around the algorithm is incredibly important, so anything that explicitly makes the properties of the algorithm part of the debate I think is good.
As governments start to implement more algorithms in public policy decisions, it becomes increasingly important that ALL algorithms be open sourced for public review unless there is a national security risk.
Having a summary of how the algorithm works is necessary but not sufficient.
Several of this bill's original co-sponsors ended up opposing it due to the lack of accountability in the risk assessment system and the fact that it gives police even more power over the justice process: https://siliconvalleydebug.org/stories/silicon-valley-de-bug...
While there are some valid concerns about using algorithms I'm quite happy that they got rid of cash bail. Making presumed innocent people pay to get out of jail has never made any sense to me.
For serious crimes, the number of people who say "well I would run for it to try and most of the rest of my life in prison, but I won't because then I (or whoever put the money up for me) won't get my bail back" has to be tiny.
For non serious crimes the number of people motivated by bail and not by avoiding spending the rest of their life as a fugitive has to be tiny.
Shouldn't the "algorithm" need to be published? Surely someone being held prisoner by a computer program should at least have the right to examine and potentially challenge the source code and data being used.
This points to one of the dilemmas that societies will be hard pressed to confront. Are we capable of accepting decisions underpinned by algorithms even when the outcome is negative in a few cases? Inevitably an individual released as a result of one of these algorithms will commit a criminal act and the argument will arise that had an individual been responsible for making the evaluation and that had cash bail existed then such crime would not have occurred. I think people will have significant difficulty in accepting even the slightest of errors with algorithms (particularly with respect to violent crime) even if the overall result is lower than if humans had been the sole deciders. In many ways it's analogous with the ongoing debates about so-called driverless cars.
So we are moving toward a world where one person gets out, while the other has to languish in prison for months or years awaiting trial because an algorithm said so. And the accused aren’t allowed to know what’s happening behind the scenes of the algorithm because we need to protect the competitive advantage of the vendor who sold the software to the state. This is total and complete insanity and elected officials should pay with their jobs and their reputations for even suggesting something so unjust.
I am absolutely outraged about the way the current system treats and traps the poor. But we need to be clear that this isn’t a fix; it’s a further obfuscation of the problem. I have little doubt that said algorithms will continue to wrong the poor at similar or greater rates - the only difference is that it will be significantly harder to detect.
68 comments
[ 3.0 ms ] story [ 129 ms ] threadOptimism and expectations of spectacular failure are not mutually exclusive.
http://www.bbc.co.uk/blogs/adamcurtis/entries/78691781-c9b7-...
Specifically, even if you're allowed by Article 22 to perform automated decision-making (and decisions on bail that are mandated by law, as in the case at hand, would be allowed) must still:
* provide meaningful information about the logic involved in the decision-making process, as well as the significance and the envisaged consequences for the individual;
* use appropriate mathematical or statistical procedures;
ensure that individuals can:
* obtain human intervention;
* express their point of view; and
* obtain an explanation of the decision and challenge it;
That said, it may be a useful framework to adopt for uses such as regulating pre-trial risk assessment algorithms.
The lack of such legislation becomes more glaring as more and more examples are demonstrated.
-kidnapping with a deadly weapon
-assault with a deadly weapon/firearm
-car jacking with a deadly weapon
The defendant has bonded out twice in the case. The first time there was only a charge of grand theft of my car, and bond was revoked after the investigation and the other more serious charges were filed. Thereafter, the defendant bonded out again after all the charges were filed, and posted $300,000+ bond. As the victim (and a lawyer) I knew the defendant would be granted bond (although I believed the defendant was both a flight risk and danger to the community and bond should have been denied), so I requested gps tracking. Less than a week later the defendant was rearrested for violating the terms of bond...all based on the gps tracking.
I’m not sure I trust an algorithm to determine if someone is a flight risk or danger to the community. Historically, the cash bond was like the carrot and stick to making sure a released defendant showed up to court. I think technology, like the gps tracker, can probably replace cash bond in almost all cases.
Why do you think a gps tracker would be that expensive? And why do you think a defendant would have to foot the entire bill? It's not like they are a one-time use piece of equipment.
[0]https://www.gpsinsight.com/gps-tracking-benefits/what-is-the...
[1]https://www.pcmag.com/roundup/354245/the-best-pet-trackers-a...
http://www.recoveryscience.ca/criminal-court-programs/pricin...
Ankle bracelet monitoring costs 4 to 6 thousand dollars per year. Poor people cannot afford that, makinging it little different than the previous bail system.
Or 15$/day from this company. 15 is the minimun "on a sliding scale" dependant on defendant income.
http://www.judicialoptions.com/inmatecost.html
It says so right in the article:
a system to ascertain a suspect’s risk of flight or committing another crime during the trial process
Sounds like your state is doing it wrong?
So this seems quite disingenuous. Auto thefts were a HUGE problem before this, you were No.1 in the nation, and it has stayed that way for six months after the Arnold Foundation model was enacted.
But instead of looking at long term trends you're blaming something that is barely one year old?
[0] https://www.abqjournal.com/1011380/courts-to-implement-new-r...
[1] https://www.abqjournal.com/813721/city-grabs-no-2-spot-in-ve...
[2] https://www.abqjournal.com/1015730/report-albuquerque-area-w...
https://www.nytimes.com/2016/06/26/opinion/sunday/artificial...
"Algorithms are racist" ha!
https://www.propublica.org/article/machine-bias-risk-assessm...
https://www.technologyreview.com/s/607955/inspecting-algorit...
http://fortune.com/longform/ai-bias-problem/
https://www.pcmag.com/article/361661/artificial-intelligence...
Ethics and Bias in Machine Learning: A Technical Study of What Makes Us “Good” https://academicworks.cuny.edu/cgi/viewcontent.cgi?article=1...
And I get downvoted!?
This is tech news, something the general public can get easily manipulated into...and for demanding a degree of reasonable discourse and reference, i get downvoted?
Bleh, this place is turning into a place i cant speak relatively freely in.
Try a more polite tone and you may get better results.
1. A blatant lie
2. A willful misinterpretation of new research
3. A gross misunderstanding of research outside one's area of expertise
4. A misapplication of one thing for another, e.g. conflating webcam algorithms and all their issues with criminal justice algorithms
Also, I'm not sure how misidentification of pictures relates to the topic at hand.
This is key, New Jersey went out of it's wait to avoid using anything that could be a proxy for race like zip codes. They also used big data to come up with the algorithm, the algorithm itself[1] is pretty strait forward and is not a black box.
The fear in California is that some counties will use black box ML systems that WILL have bias even if it's not intentional, just like how biases in the training data of image recognition tend to misidentify pictures of black people more then white people.
1. https://www.arnoldfoundation.org/wp-content/uploads/PSA-Risk...
Regardless, the cash bail system is terrible and I'm anticipating big improvements as we move away from it.
I'm more and more convinced that most of these cases are just stuffing an excel spreadsheet into a black box, and shaking it. Hearing that we're using "algorithms" in criminal justice doesn't give me warm and fuzzy.
That is the assumption that scares me. The algorithm encodes the bias and applies it in every case.
This feature will only save you from a shifting bias.
For instance, consider that owning a major asset probably decreases flight risk. That seems fair. But, this means that poor people are probably more of a flight risk than a middle- or upper-class homeowner. So if the algorithm considers this, poor people would be disproportionately affected.
Also consider that having family abroad or being from another country probably increases flight risk. Hence, immigrants would be disproportionately affected.
But now, when there is an unjust outcome, there isn't even an option to avoid jail time (even if that option is difficult to reach or undesirable); and the human responsible for the decision to incarcerate can simply wash their hands of it, since the algorithm gave the suspect a grade of "D". No need for empathy, sympathy, or guilt.
Then there are the pending discussions about what is 'fair' and the answer to those questions is politics...
[0] - Edit: Could be
The status quo is a roaring injustice and ongoing disaster, and by insisting on the perfect over the good you are condemning millions of people to continue suffering indefinitely. How is your thinking not "very dangerous" to all those wronged day after day by the current system?
The danger here is doubling down on the existing injustices, removing the limited accountability and positive exceptions that exist, and effectively sweeping conversation under the rug or arguing ideologies about the fairness of tweaks to the algorithms at the cost of real human lives.
According to you the solution lies in "politics". Yet, this law is the product of California's government, a political body -- is that not legitimate "politics"?
So what would make for a "political" solution that would qualify for your stamp of approval? Chasing a consensus with the "tough on crime" crowd who benefit from felony disenfranchisement disproportionately suppressing minority votes? No such consensus will ever happen.
> According to you the solution lies in "politics".
My post was critical of the role of politics in determining the fairness of the algorithm. It removes the human lives factor and provides a platform for politicians and ideologues ("tough on crime" or otherwise) to either do nothing, argue about the ideology of fairness, or ignore ground truth about outcomes and pre-existing bias in the system.
In an open system, the bias could be identified, a patch created and approved, and a new release deployed. As biases shift, more patches are made. Ideally we find smaller and smaller "bugs" as time goes by.
Alas, we are seemingly very far from that today. There is probably great value in controlling that algorithm, and like gerrymandering district maps, the politicians have no incentive to create such an open system. I fear that algorithm-driven bail might become just another tool for politicians and power brokers to bludgeon their opponents with.
"just redistrict, lol. But after the election"
That's always been the challenge in the current system, because you're leaving responsibility up to individuals it was a hard system to adjust even if it isn't working fairly (e.g. you aren't going to just randomly fire judges because minorities got a slightly unfair deal).
I’m curious if the system does find that poor people (or rich, but one or the other) behave such that the system’s results are unexpected, in practice, what will politicians do?
But there was already an algorithm that came up with the cash bail amount for a defendant, which was purportedly based on the defendant's flight risk and likelihood of endangering the communoty.
I don't pretend to be an expert so I may be wrong but I think the algorithm boiled down to something like "let the judge decide".
I think transparency around the algorithm is incredibly important, so anything that explicitly makes the properties of the algorithm part of the debate I think is good.
Having a summary of how the algorithm works is necessary but not sufficient.
For serious crimes, the number of people who say "well I would run for it to try and most of the rest of my life in prison, but I won't because then I (or whoever put the money up for me) won't get my bail back" has to be tiny.
For non serious crimes the number of people motivated by bail and not by avoiding spending the rest of their life as a fugitive has to be tiny.