For a broader discussion of just how bad, and dangerous, forensic sciecne can be there is a Scientific American article [0] from about 5 years ago. Along with many others if you take a look. Most access this issue through one specific case that is of local relevance...but at a high level, people have literally been executed based on bluntly fake science and courts opt out of doing anything.
If AAAS is telling you your field is bad at science, everyone should situp and listen [1]. For example...there are no scientific standards around fingerprinting.
There was a PBS show that looked into how wonky forensic science could be.
A common certification at the time was earned by taking an online class and answering multiple choice questions ... and blamo you had a certificate that they found on a lot of resumes. It didn't require much of any actual scientific knowledge.
This appears to be surprisingly common in both law and fire. I hold a handful of such certificates, all of which say I'm qualified to do things I'm mostly certainly not.
I'm trying to change this behavior by incorporating objective examinations into one of my specializations and it's extremely difficult.
They are both available online for free and you download a little certificate afterward.
ICS is the Incident Command System used by law, fire, and other agencies to manage "incidents" -- anything from, "my kid just disappeared from my front yard" to "the entire town of Paradise just burned to the ground, now what do we do?"
There are some principles in ICS that any technical manager should find helpful. Someday I'd like to write a long article on what technical managers can learn from ICS, but in the meantime I'm happy to suggest other people just take the first two courses.
I had a professor in my graduate education who was a mathematician focusing on criminal forensics. She was involved in the AAAS report amongst other efforts. The short version of the story that she told about two years ago is that, in all of criminal forensics, there is only one method which has been subjected to rigorous research: DNA matching, due to it being a topic of interest in many fields (e.g. genetic anthropology).
More junk science used to convict people in the courts of America? I'm going to die now of not surprise! I guess a better way to say this is, how does the proliferation of shit work like this happen so fast but the refutation take so damned long?
Incentives. If a system is designed to convict (prosecutors are measured on conviction rates, not "justice" or "the truth"), it will bias towards tools that enable meeting those metrics. You get what you measure.
We need tooling to better understand when people have been wrongfully convicted using inaccurate forensics sciences so they can more quickly get their case re-evaluated (and speed exoneration if warranted). If someone from Code For America sees this, consider such a tool (similar to what has been built to rapidly expunge marijuana convictions in states that have legalized it).
There's an asymmetry to the process of how judges allow evidence.
If one judge somewhere allows a prosecutor to present junk science as evidence, that becomes precedent that other prosecutors can point to in order to convince other judges to allow it, too.
Once precedent is established, it takes the decision of a higher court to stop use of that junk science. It takes time for cases to work their way up.
At what point are judges liable here? They play a vital role in our society yet a lot of what they do seems to be based on precedent established on a whim.
If you mean criminally liable though, we don’t hold them liable for the same reasons we don’t hold lawyers criminally liable for defending a convicted criminal. At the end of every human process, you need someone somewhere to exercise some human judgement and be able to rely on it. Humans are not infallible though, and neither is anything else we create whether it’s court systems or computer systems.
This is one of the reason’s why we have trials by Jury in cases where life, limb and liberty are at stake. It is one thing to convince one person, the judge presiding over the case to pass judgement one way or another, but a whole other thing to convince several of your peers of equal social standing to pass judgement. It’s still not perfect, but nothing ever will be.
Do judges lose their jobs after successful appeals? I'm sure it has happened, but this seems like an incredibly rare event. Yet lawyers who do their jobs poorly (even ones defending criminals) can get fired from being a lawyer and have consequences on their ability to practice law. For most jobs, if you do it poorly enough then there can be severe consequences that follow. Yet here there are matters of life and death at play and nobody seems to have any responsibility in their screw up. Even police officers are held accountable when great harm comes to the innocent due to what they did.
Federally, Article III judges can be impeached and removed by Congress. Administrative and Military (Article II) judges can be fired outright or demoted. State-level, depends on the State, but in California we have something called retention elections, which rarely result in a change in office, but changes in office have come about after particularly unpopular rulings.
An appeal doesn’t directly lead to jobs loss, but it doesn’t look good on your record when your judgements are regularly overturned, particularly if you aspire to a better position. Keeping your job isn’t always the only, or most important incentive if you have any kind of ambition or want to leave any kind of lasting legacy.
TL;DR: the "wear" pattern are those light/dark marks on both sides of pants, along the seam, that would be "unique" to each jeans isn't enough to give a "true positive" identification keeping the "false positives" low enough.
> there was no crime, everything pointing to a tragic accident
From just reading the Wiki article (key caveat), I would say the case is not "beyond reasonable doubt" thus a conviction is unwarranted. But that doesn't mean everything points to a tragic incident. This is a pretty murky case.
I've read quite a bit about this, including a re-read of the wiki page just now. The only "murky" thing is a natural human aversion to contemplating the suffering of children. There was never any credible evidence of murder.
> (1998) FBI scientists reported at the American Academy of Forensic Sciences in February that they are using a new technique to identify criminals – analyzing the unique wear patterns of denim jeans and comparing them to photographs taken from crime scenes.
Most forensic science is complete BS. Very little has changed since the pivotal National Academies of Sciences report, "Strengthening Forensic Science in the United States" came out in 2009.
A false positive rate of 0.1% and a false negative rate of 40-50%? Isn't that... actually pretty useful? You'd never want to convict someone based exclusively on this, but it could fit into the "preponderance of evidence" standard. No one piece of evidence is foolproof, e.g. the classic examples of DNA & fingerprints- did you even lift the right set of prints or collect the right sample of hair?
That was my thought. I think the main complaint is that there evidence is presented without the statistics you mentioned, as if it's near effective 100% of the time.
Without confirmation, false positive rates varied from 2 to 4 percent. That means up to 4 percent of test subjects will end up arrested because of a false positive.
What they found was a correlation between false positive rates and true positive rates, where IF you could accept a false positive rate of 0.1%, THEN you could get a little over 50% true positive rate with the longest seam length under "ideal imaging conditions".
They do not attempt to estimate the false positive rate of whatever methodology FBI agents are using in practice. I doubt the FBI has a rigorous well-defined test that could be analyzed; I suspect this is just a general technique that is applied by different agents in different ways.
Even if the FBI's methods approach this ideal false positive rate (which they can't), they are not describing it honestly in court.
> When, for example, an FBI analyst identified the plaid shirt in a surveillance video as that belonging to Wilbert McKreith, the analyst stated in court that the odds that two different shirts would match were a staggering 1-in-650 billion. The analyst arrived at this number by making eight measurements along two seams, estimating the probability that the plaid stripes at these eight locations were misaligned by a certain distance, and then multiplying all eight of these probabilities to reach the astronom-ically low odds of 1-in-650 billion.
It would be, but those numbers are for the absolutely ideal conditions present in the study. It seems unlikely that a photo like this will come up in a real court case:
>The analyst excluded any seams that did not satisfy each of the following properties: 1) the length of the seam was at least 30 cm, 2) the resolution of this 30-cm seam was at least 1,500 pixels, 3) the width of the seam was at least 30 pixels, and 4) the final extracted 1D signal clearly showed a distinct pattern of ridges and valleys.
Even with photos taken specifically to see these patterns, "[e]nforcement of these criteria reduced the original 211 left/right and inner/outer seams to 164 left/inner, 162 right/inner, 81 left/outer, and 90 right/outer seams."
Unfortunately they aren't very clear on what they mean by different orientations/lighting conditions/drapes.
1 person committed a crime. So the true positive rate is 1 out of 7 billion or so. So how many false positives would 0.1% lead to? About 7 million pants wearers?
And for false negatives, we expect half the time the true criminal's pants would prove them innocent?
I don't think you can really trust evidence if the true criminal has a coin-flip chance to show innocence with it.
36 comments
[ 3.0 ms ] story [ 81.9 ms ] threadhttps://www.pnas.org/content/pnas/early/2020/02/18/191722211...
If AAAS is telling you your field is bad at science, everyone should situp and listen [1]. For example...there are no scientific standards around fingerprinting.
[0] https://www.scientificamerican.com/article/can-we-trust-crim...
[1] https://www.aaas.org/resources/forensic-science-assessments-...
A common certification at the time was earned by taking an online class and answering multiple choice questions ... and blamo you had a certificate that they found on a lot of resumes. It didn't require much of any actual scientific knowledge.
I'm trying to change this behavior by incorporating objective examinations into one of my specializations and it's extremely difficult.
100: https://training.fema.gov/is/courseoverview.aspx?code=IS-100...
200: https://training.fema.gov/is/courseoverview.aspx?code=IS-200...
They are both available online for free and you download a little certificate afterward.
ICS is the Incident Command System used by law, fire, and other agencies to manage "incidents" -- anything from, "my kid just disappeared from my front yard" to "the entire town of Paradise just burned to the ground, now what do we do?"
There are some principles in ICS that any technical manager should find helpful. Someday I'd like to write a long article on what technical managers can learn from ICS, but in the meantime I'm happy to suggest other people just take the first two courses.
We need tooling to better understand when people have been wrongfully convicted using inaccurate forensics sciences so they can more quickly get their case re-evaluated (and speed exoneration if warranted). If someone from Code For America sees this, consider such a tool (similar to what has been built to rapidly expunge marijuana convictions in states that have legalized it).
Ultimately, when no one in the process has scientific/mathematical literacy, pseudoscience proliferates.
If one judge somewhere allows a prosecutor to present junk science as evidence, that becomes precedent that other prosecutors can point to in order to convince other judges to allow it, too.
Once precedent is established, it takes the decision of a higher court to stop use of that junk science. It takes time for cases to work their way up.
If you mean criminally liable though, we don’t hold them liable for the same reasons we don’t hold lawyers criminally liable for defending a convicted criminal. At the end of every human process, you need someone somewhere to exercise some human judgement and be able to rely on it. Humans are not infallible though, and neither is anything else we create whether it’s court systems or computer systems.
This is one of the reason’s why we have trials by Jury in cases where life, limb and liberty are at stake. It is one thing to convince one person, the judge presiding over the case to pass judgement one way or another, but a whole other thing to convince several of your peers of equal social standing to pass judgement. It’s still not perfect, but nothing ever will be.
An appeal doesn’t directly lead to jobs loss, but it doesn’t look good on your record when your judgements are regularly overturned, particularly if you aspire to a better position. Keeping your job isn’t always the only, or most important incentive if you have any kind of ambition or want to leave any kind of lasting legacy.
Interesting and on the side of caution is that false negatives are much much higher than false positives in this Jeans pattern matching.
Executed following a conviction based on the worst junk science.
(there was no crime, everything pointing to a tragic accident)
From just reading the Wiki article (key caveat), I would say the case is not "beyond reasonable doubt" thus a conviction is unwarranted. But that doesn't mean everything points to a tragic incident. This is a pretty murky case.
House fires happen a lot. According to the NFPA, in the US there were on average around 350000 house fires per year between 2013 and 2017, 92% of them accidental: https://www.nfpa.org/News-and-Research/Data-research-and-too...
> (1998) FBI scientists reported at the American Academy of Forensic Sciences in February that they are using a new technique to identify criminals – analyzing the unique wear patterns of denim jeans and comparing them to photographs taken from crime scenes.
https://www.ncbi.nlm.nih.gov/m/pubmed/12423006/
Without confirmation, false positive rates varied from 2 to 4 percent. That means up to 4 percent of test subjects will end up arrested because of a false positive.
What they found was a correlation between false positive rates and true positive rates, where IF you could accept a false positive rate of 0.1%, THEN you could get a little over 50% true positive rate with the longest seam length under "ideal imaging conditions".
They do not attempt to estimate the false positive rate of whatever methodology FBI agents are using in practice. I doubt the FBI has a rigorous well-defined test that could be analyzed; I suspect this is just a general technique that is applied by different agents in different ways.
Even if the FBI's methods approach this ideal false positive rate (which they can't), they are not describing it honestly in court.
> When, for example, an FBI analyst identified the plaid shirt in a surveillance video as that belonging to Wilbert McKreith, the analyst stated in court that the odds that two different shirts would match were a staggering 1-in-650 billion. The analyst arrived at this number by making eight measurements along two seams, estimating the probability that the plaid stripes at these eight locations were misaligned by a certain distance, and then multiplying all eight of these probabilities to reach the astronom-ically low odds of 1-in-650 billion.
>The analyst excluded any seams that did not satisfy each of the following properties: 1) the length of the seam was at least 30 cm, 2) the resolution of this 30-cm seam was at least 1,500 pixels, 3) the width of the seam was at least 30 pixels, and 4) the final extracted 1D signal clearly showed a distinct pattern of ridges and valleys.
Even with photos taken specifically to see these patterns, "[e]nforcement of these criteria reduced the original 211 left/right and inner/outer seams to 164 left/inner, 162 right/inner, 81 left/outer, and 90 right/outer seams."
Unfortunately they aren't very clear on what they mean by different orientations/lighting conditions/drapes.
1 person committed a crime. So the true positive rate is 1 out of 7 billion or so. So how many false positives would 0.1% lead to? About 7 million pants wearers?
And for false negatives, we expect half the time the true criminal's pants would prove them innocent?
I don't think you can really trust evidence if the true criminal has a coin-flip chance to show innocence with it.
If a piece of evidence is used to identify, you select for the population that matches that property.