I had a shocking experience with "expert" testimony in the case of a large lawsuit I witnessed.
This expert, paid by the litigant, had written a few papers for low-quality journals and taught at a class at a west-coast medical school on forensics. His written claims supporting the lawsuit were provided to the defense and he was questioned in a deposition.
The expert's conclusions in his report were based on bad probability. This was several years ago so I don't remember all of the details, but one error was the using 50% as a prior probability because the actual prior condition wasn't known and could have been one of two possibilities, X or not X; 50% was picked because there were two possibilities and disregarded the relative rarity of X in the general population. Another error was really quite laughable, at one point essentially arguing that P(A & B) = P(A) + P(B). All of this was hidden by somewhat confusing language.
It really wasn't a laughing matter because the lawyers, which had asked me to explain the report to them, couldn't understand the probability well and were consequently unsuccessful in exposing the expert's faulty reasoning during his deposition. Fortunately, the expert's so called report was not introduced at trial and he wasn't called upon during the trial.
Bad probability calculations show up a lot in the court room. A lot of probability theory is counterintuitive so people can get away with making an intuitive sounding argument that is fundamentally wrong. Two high profile cases that I like to bring up are OJ Simpson's murder trial and Sally Clark's trial.
I don't remember the exact numbers used in OJ Simpson's trial but the rationalization used is more important than the numbers. His lawyer argued that some very small percentage of men who beat their wives later go on to murder them. Therefore, the fact that OJ beat his wife shouldn't factor into his trial at all. The fundamental flaw in the argument is that there was no question on whether or not his wife was dead. We knew she was dead, so the correct statistic to look at would be, of women who were beaten by their husbands and later murdered, how often was the husband the murderer? And the answer was some large percentage, like more than 90%.
Sally Clark's case was even sadder. She was eventually released from prison but committed suicide not long after. She was on trial for murder after her second child died from SIDS, her first child having died the same way. The prosecution brought in an "expert witness" to give the probability of one woman having two children die from SIDS. All he did was take the incidence of SIDS at the time, 1 in 8500, and square it, coming up with 1 in 75 million. No other evidence of murder was found, she was convicted because of the statistical argument. This argument has multiple flaws. It assumes that the two incidences of SIDS in this case were independent events, which is not necessarily the case. On top of that, even if you could assume they were independent events, a 1 in 75 million event doesn't mean the event is impossible. A person should not be convicted on that probability alone with no other evidence.
I wonder if whoever is supplying evidence via tests should be forced to (blindly) apply the tests to people other than the defendant. If your test says some random sample is more likely match than I am, it's very weak evidence.
there are certainly a number of forensic fields which are garbage.
other portions are fine, and are effectively analytical chemistry applied to legal cases. are you aware of a lot of double blind studies in analytical chemistry? what would that even mean?
How much of the conclusions drawn from these investigative techniques are justified entirely by analytic chemistry, and how much goes beyond that? Using an example from a related article, there's a substantial difference between being able to say what's the chemical element composition of a particular bullet and being able to identify the particular manufacturing facility and batch it came from. The latter relies on a whole lot of assumptions about the ammunition industry.
the US legal system relies on prosecution and defense both functioning. when expert testimony comes into play, that means there should be prosecution and defense experts. unfortunately public defenders are underfunded and frequently inexperienced, and can't/don't get opposing experts to review evidence.
many of the state lab testing scandals would be discovered years earlier if defense lawyers regularly had things retested.
More to the point, there is no disincentive for convicting an innocent person unless the victim can prove that the prosecutor was acting in bad faith and has the resources to mount a lawsuit. It is an extremely hard case to win, and even if they do win the government is punished (via monetary judgement), not the prosecutor.
If prosecutors were punished for every conviction that is later overturned then they might be more reluctant to push for conviction on flimsy fabricated evidence.
Are you entirely sure? Someone's race, ability to pay a lawyer, and prior history can be huge factors in convicting them.
Convicting a poor black man who dropped a cigarette butt at the scene hours prior to the crime may ultimately be far easier than finding the actual perp.
One of the big problems is the anti scientific method.
the scientific method consists of formulating a hypothesis and a method of testing the probability of failing to reject a null version. in common speech this means make an assumption, observe data/evidence and alter the assumption to fit the evidence. in the case of anti science such as forensics or investigations of crime, the opposite occurs. the assumption or theory is made the data or evidence is observed, and discounted or re interpreted to fit the theory.
Lots of strawman examples. Anything imperfect will have errors. I didn't see much 'scientific' about the criticisms in the article. For instance, measures of how often techniques are right, how often labs are correct.
How do you propose a scientific study be done of the end result with a billion confounding factors. Further the people on trial are heavily predisposed to be actually guilty to start with supposing we aren't just picking random people off the streets. They are the opposite of a random sample. Using nonsense to "prove" that the actually guilty is still false even if the ultimate answer was known by other means to be correct.
The logical thing to validate is the actual underlying science that is vastly easier to tease out and test independently and scientifically if we haven't even tried to disprove something it has limited merit and historically if its proponents resist even trying to prove it it is almost certainly hokum.
Wherein the means to test and validate the means we are using to deprive people of their freedom and sometimes their lives represents a fraction of the budget of the criminal justice system we are guilty of malpractice as a society.
You don't need to disprove measures that nobody has proved in the first place. Much as the burden of proof lies with the prosecution the burden of proof lies with those who assert they have the wherewithal to prove the defendants guilt.
This assertion is entirely inline with proper scientific thinking.
This expert, paid by the litigant, had written a few papers for low-quality journals and taught at a class at a west-coast medical school on forensics. His written claims supporting the lawsuit were provided to the defense and he was questioned in a deposition.
The expert's conclusions in his report were based on bad probability. This was several years ago so I don't remember all of the details, but one error was the using 50% as a prior probability because the actual prior condition wasn't known and could have been one of two possibilities, X or not X; 50% was picked because there were two possibilities and disregarded the relative rarity of X in the general population. Another error was really quite laughable, at one point essentially arguing that P(A & B) = P(A) + P(B). All of this was hidden by somewhat confusing language.
It really wasn't a laughing matter because the lawyers, which had asked me to explain the report to them, couldn't understand the probability well and were consequently unsuccessful in exposing the expert's faulty reasoning during his deposition. Fortunately, the expert's so called report was not introduced at trial and he wasn't called upon during the trial.