Summary: the presumption that computers are always correct had lead the UK post office to believe that hundreds of post offices had shortfalls in their accounts, leading to hundreds of postmasters getting accused of fraud, resulting in their prosecution and conviction.
Fortunately, group action has lead to the error in the system being discovered and their convictions being overturned.
It seems pretty dangerous to blindly assume that a computer system is always correct without thorough proof that it is indeed correct. But we already knew that.
A complete proof of correctness might be a bit much to ask, but slightly more burden of proof than "it's what the computer says" would be nice. Every piece of software of any meaningful size has bugs. First check if the discrepancy might be the result of a bug before you start sending people to prison.
It's not possible to prove, without billions of dollars of investigation per device, that any complex consumer electronic device is operating 100% correctly.
So wouldn't it practically invalidate any evidence that was touched by any device?
Why would it cost billions of dollars to satisfy the court of the accuracy of a program’s claims when it doesn’t cost billions of dollars to satisfy the court of the accuracy of an investigator’s claims?
Courts don't bother themselves with confidence intervals nor do they bother with 100% ("reasonable doubt" is not 0/100%).
The problem is the current situation. Evidence is being used without actual proof that said evidence was collected "truthfully". An algorithm will generate a result but if you haven't proven/shown that the algorithm generates a correct result then any result from that algorithm shouldn't be usable in court.
This problem comes up a lot with people who (un)cleverly use NULL for things (i.e. License Plates). Or if you live in the middle of America and somebody decides your address will be the default for geo-ip lookups if none is found.
To stick with the article, immediately when the system was deployed the postmasters thought something was wrong. If the state had to show that the computer was correct the whole situation would've been avoided.
No sudden unexplainable wealth, no bank transactions to personal accounts, no incriminating text messages. Even if the code was correct that alone is not proof the individuals accused took the money, but they were still convicted. That isn’t supposed to happen.
For the postmaster example, they did show it to be correct at their accepted level of confidence. Which is that it returns superficially believable results.
100% correctness for such a system would certainly cost many billions.
If you mean that they should have had a higher bar, then sure, but that's a political choice of how much effort to invest, not a technical one.
"Oh, but the computer is doing the right more often than (9.4608e+16)-1 times out of 9.4608e+16 times!"*
Yes.. but that's "less than" a single mistake per year, per computer. How many cores do your data go through per year ?
* Totally made up, rough number of cpu ticks of a 3 ghz core in a year.
Still, the point I'm trying to make is that with the mind-numbing number of actions carried out by another mind-numbing number of machines on a third absolutely mind-numbing number of bits every year, you will need a metric fuckton of zeroes after your decimal point, before it's "not going to put innocent people in jail".
I don't think it is an either or question. But it seems very reasonable to not assume computers are always correct.
In a situation in which a bug could have conceivably caused the evidence against the defendant the defandant, should be allowed to examine the computer system (read the source code). Without that I see no reason why it should be allowed as evidence against someone, when the evidence could have been the result of a flaw in the software.
Of course "could" is pretty ambigous. Things you might look at are: Have this or other, similar systems suffered from a bug which could have caused this? How much/often is the claim out of line with what the defence claims. How serious and easy to notice would a bug have to be to cause this? Is there other evidence, besides the computer? And probably many more.
Of course "the camera changed the face of the person to the defendants" is something which probabbly should be disregarded.
1. Very rarely, a computer will add 1 and 1 and end up with something other than 2. Search for 'cosmic bitflips'
Given how rare this kind of event is, it shouldn't be necessary, in every case involving a computer, to prove that that specific computer will reliably produce 2 when asked to add 1 and 1.
As a result, it makes sense to presume that a computer is operating correctly unless there is evidence to the contrary.
However, challenge 2 comes in when you start to generalise.
2. Humans often (accidentally or intentionally) tell computers to do things wrongly. Consider the below obviously contrived example:
function divide(pounds, divisor) {
return (pounds/divisor).toFixed(2);
}
function multiply(pounds, multiplier) {
return (pounds*multiplier).toFixed(2);
}
function getUnitPrice(pricePerKg, unitsPerKg) {
return divide(pricePerKg, unitsPerKg)
}
function makeSale(unitPrice, qty) {
return multiply(unitPrice, qty)
}
console.assert(totalSales === expectedSalesValue, 'Total Sales is not equal to Expected Sales Value');
This sample shows how easily a computer, operating correctly, can be mislead by a careless programmer. You could even have test suites which end up with errors, for example if the units per Kg were 4, or the price per Kg were 13 and the units per Kg were 10.
It is important to draw the line between 'operating correctly' and 'operating as intended'.
correct! if ever faced with this situation, the hair should be split. We can accept that the action of the computer machinery is, without evidence to the contrary, assumed to be correct. It's not a "computer" error.
BUT, the programming of said computer is certainly not infallible. In fact, without evidence to the contrary, I think there is ample precedent that a computer program is full of bugs and should be trusted not one bit. Where's the evidence that the program was tested? Designed to safety or reliability standards? Etc.
If I was ever required to "prove" that a computer system (including programming) was faulty, Hoo-Boy! Give me access to the source code. It's a fair bet you could find one bug per every 100 lines of code without trying.
The days of "nope - computer sez..." are over. Every judge, jury person has experienced these issues personally too. The position of infallible computer system will not stand long.
As a professional software developer, I find this absolutely fucking terrifying..
Finding out THAT there is a malfunction can already be just about practically impossible.. Add to that proving that there is.. That's probably provably impossible..
It's well-known that hardware randomly flips bits, and enough of those happen that even with ECC, it's not always detected.
It's even more well known that software is rarely provably functional.
Presuming a computer is accurate is reasonable because otherwise you couldn't prove any crime ever in a digital world. In any one of these cases they could have gone back to the underlying bank records and found where the money actually went. They didn’t. Blaming the internal accounting software is an easy excuse for the outrageously low burden of proof the UK government holds itself to.
It is a reasonable assumption indeed, but it shouldn't be mindlessly trusted, and the burden of proof of innocence should never be on the defendant.
If some idiot system places you on the scene of the crime, fine, good starting point, it'd be reasonable to start investigating whether you were there, but it should count for nothing in and of itself. There should be actual evidence that you were there, like DNA, fingerprints, actual photos, and a lack of any countering evidence.
The issue here is not the computers making mistakes or presuming computers are accurate. The problem is that the UK courts regularly imprisoned people even though no bank transactions showed any money being taken. The problem with the UK justice system is much deeper and it is fundamentally biased and heartless against ordinary non-politically connected people and always has been. Monty Python criticizes this in the fish license sketch, as does George Orwell. This is why Americans hated the crown so much. It doesn't give its own people any benefit of the doubt. And has presumed guilt until proven innocent. It is no accident this happened in Britain.
26 comments
[ 6.3 ms ] story [ 84.5 ms ] threadFortunately, group action has lead to the error in the system being discovered and their convictions being overturned.
It seems pretty dangerous to blindly assume that a computer system is always correct without thorough proof that it is indeed correct. But we already knew that.
Think of it in the other direction, why should you be allowed to use evidence that has no documented reason of being accurate.
So wouldn't it practically invalidate any evidence that was touched by any device?
If they are fine with a lower bound of confidence, then that's already the current situation.
The problem is the current situation. Evidence is being used without actual proof that said evidence was collected "truthfully". An algorithm will generate a result but if you haven't proven/shown that the algorithm generates a correct result then any result from that algorithm shouldn't be usable in court. This problem comes up a lot with people who (un)cleverly use NULL for things (i.e. License Plates). Or if you live in the middle of America and somebody decides your address will be the default for geo-ip lookups if none is found.
To stick with the article, immediately when the system was deployed the postmasters thought something was wrong. If the state had to show that the computer was correct the whole situation would've been avoided.
100% correctness for such a system would certainly cost many billions.
If you mean that they should have had a higher bar, then sure, but that's a political choice of how much effort to invest, not a technical one.
Yes.. but that's "less than" a single mistake per year, per computer. How many cores do your data go through per year ?
* Totally made up, rough number of cpu ticks of a 3 ghz core in a year.
Still, the point I'm trying to make is that with the mind-numbing number of actions carried out by another mind-numbing number of machines on a third absolutely mind-numbing number of bits every year, you will need a metric fuckton of zeroes after your decimal point, before it's "not going to put innocent people in jail".
‹ducks›
In Danish we use the word "Ton" which is 1000 Kg, didn't realized that English has two words for it, didn't realize there's an emperial ton either :D
In a situation in which a bug could have conceivably caused the evidence against the defendant the defandant, should be allowed to examine the computer system (read the source code). Without that I see no reason why it should be allowed as evidence against someone, when the evidence could have been the result of a flaw in the software.
Of course "could" is pretty ambigous. Things you might look at are: Have this or other, similar systems suffered from a bug which could have caused this? How much/often is the claim out of line with what the defence claims. How serious and easy to notice would a bug have to be to cause this? Is there other evidence, besides the computer? And probably many more. Of course "the camera changed the face of the person to the defendants" is something which probabbly should be disregarded.
1. Very rarely, a computer will add 1 and 1 and end up with something other than 2. Search for 'cosmic bitflips'
Given how rare this kind of event is, it shouldn't be necessary, in every case involving a computer, to prove that that specific computer will reliably produce 2 when asked to add 1 and 1.
As a result, it makes sense to presume that a computer is operating correctly unless there is evidence to the contrary.
However, challenge 2 comes in when you start to generalise.
2. Humans often (accidentally or intentionally) tell computers to do things wrongly. Consider the below obviously contrived example:
function divide(pounds, divisor) { return (pounds/divisor).toFixed(2); }
function multiply(pounds, multiplier) { return (pounds*multiplier).toFixed(2); }
function getUnitPrice(pricePerKg, unitsPerKg) { return divide(pricePerKg, unitsPerKg) }
function makeSale(unitPrice, qty) { return multiply(unitPrice, qty) }
const widgetPricePerKg = 10; const widgetUnitsPerKg = 3; const widgetUnitPrice = getUnitPrice(widgetPricePerKg, widgetUnitsPerKg); const widgetsSold = 3;
const totalSales = makeSale(widgetUnitPrice, widgetsSold);
const kilosOfWidgetsSold = divide(widgetsSold, widgetUnitsPerKg); const expectedSalesValue = multiply(kilosOfWidgetsSold, widgetPricePerKg);
console.assert(totalSales === expectedSalesValue, 'Total Sales is not equal to Expected Sales Value');
This sample shows how easily a computer, operating correctly, can be mislead by a careless programmer. You could even have test suites which end up with errors, for example if the units per Kg were 4, or the price per Kg were 13 and the units per Kg were 10.
It is important to draw the line between 'operating correctly' and 'operating as intended'.
BUT, the programming of said computer is certainly not infallible. In fact, without evidence to the contrary, I think there is ample precedent that a computer program is full of bugs and should be trusted not one bit. Where's the evidence that the program was tested? Designed to safety or reliability standards? Etc.
If I was ever required to "prove" that a computer system (including programming) was faulty, Hoo-Boy! Give me access to the source code. It's a fair bet you could find one bug per every 100 lines of code without trying.
The days of "nope - computer sez..." are over. Every judge, jury person has experienced these issues personally too. The position of infallible computer system will not stand long.
Finding out THAT there is a malfunction can already be just about practically impossible.. Add to that proving that there is.. That's probably provably impossible..
It's well-known that hardware randomly flips bits, and enough of those happen that even with ECC, it's not always detected.
It's even more well known that software is rarely provably functional.
If some idiot system places you on the scene of the crime, fine, good starting point, it'd be reasonable to start investigating whether you were there, but it should count for nothing in and of itself. There should be actual evidence that you were there, like DNA, fingerprints, actual photos, and a lack of any countering evidence.