Why Science is in Trouble, Exhibit A

Garbage In, Garbage Out

I look at scientific papers frequently. For example, one of my clients is developing a gene therapy for patients with Danon disease, a rare genetic mutation that causes life-threatening musculoskeletal, cognitive, and heart problems. In front of me is a paper describing a natural history study of patients with Danon disease.

As I read this paper, I assume that there’s no fraud, blatant incompetency, misrepresentations, gross data errors, compromising biases, inconsistences, political interests, etc. Because…well, it’s a scientific paper published in a prominent journal.

But what if this weren’t true?

Exhibit A for why this isn’t always true is ivermectin.

Ivermectin was formerly used around the world for parasitic infections but was suddenly thrown into the spotlight during the COVID-19 pandemic. Many asked: Does ivermectin work for SARS-CoV-2 infections, the virus the causes COVID-19? This simple medical question has become surprisingly divisive.

The respected science writer at Reason, Ronald Bailey, reviewed the literature and concluded that ivermectin is largely ineffective as a treatment for COVID-19 infections. He looked at five review papers to reach his conclusion. Let’s consider the first in his list: “Efficacy and safety of ivermectin for treatment of non-hospitalized COVID-19 patients: A systematic review and meta-analysis of 12 randomized controlled trials with 7,035 participants,” published in 2024 in International Journal of Antimicrobial Agents by Hernandez et al.

A few of the dozen studies covered in this paper were eliminated by the authors due to concerns over bias.

Consider the results for all-cause mortality, from which the results of nine studies with 6,353 patients were included. By far the most important study for this meta-analysis was the TOGETHER Trial (Reis et al., 2022), which comprised 80% of the recorded patient deaths (45 of 56). This is the same clinical trial with which David R. Henderson and I discovered so many problems that we wrote about it here and here. A sampling of the problems with Reis includes a severe conflict of interest, broken trial blinding, failures with treatment randomization, multiple impossible numbers, uncorrected errors, protocol violations, a broken promise to release the data, and no responses from the authors.

In summary, the TOGETHER Trial was so bad that I am left wondering whether the cause was incompetence or fraud. Further, while the TOGETHER Trial authors claimed that ivermectin didn’t help patients, the results of their own study belie that conclusion.

Reporting on the narrative presented in Reis, the New York Times quoted one infectious disease expert, Dr. David Boulware of the University of Minnesota, stating, “There’s really no sign of any benefit,” while another, Dr. Paul Sax of Brigham and Women’s Hospital in Boston, said, “At some point it will become a waste of resources to continue studying an unpromising approach.”

And yet, David and I calculated that the TOGETHER Trial results—flawed as they were—suggested that ivermectin had a 68% chance of helping reduce mortality and a 91% chance of helping reduce hospitalization among COVID-19 sufferers. While we realize that the usual statistical threshold is 95%, based on our results, it is difficult to agree with the conclusion that the TOGETHER Trial showed “no sign of any benefit” for ivermectin. It showed promise, just not enough promise to satisfy statisticians.

There were 27 authors of Reis who published the TOGETHER Trial results. Did none of them notice the problems with the study? There were 11 authors of Hernandez. Did none of them notice the problems with the TOGETHER Trial? Or were they blindly pulling numbers out of the paper to put into their statistical models?

Another study used in this meta-analysis, López-Medina et al., 2021, was condemned as fatally flawed by more than 100 physicians in an open letter to Journal of the American Medical Association.

A third study, Vallejos et al., 2021, ended up on one researcher/MD’s list of flawed and/or fraudulent ivermectin studies.

A fourth study, Buonfrate et al., 2022, also exhibited some important flaws.

A fifth study, Bramante et al., 2022, had almost as many problems as the TOGETHER Trial.

A sixth study, Naggie et al., 2022 and 2023, had so many inconsistencies it was later updated by the authors. However, the updated version still has inconsistencies.

The authors in Hernandez attempt to explain how some other studies generated positive results for ivermectin. “The dissemination of some positive results of poorly designed and powered ivermectin studies caused confusion in the general population, patients, clinicians, and decision makers, who may have manipulated the information due to individual and political interests as had happened previously with hydroxychloroquine.”

They were aware of the issues that could affect any study, including their own. But, of course, it’s the other guys who suffer these problems, right? Could this be what is called “projection” in psychology, a defensive mechanism of unconsciously projecting one’s own unacceptable behaviors onto others?

Stepping back, we see that there have been 20 randomized controlled trials of ivermectin with a total of 7,864 patients that have measured mortality. The results? Even with some of the problems highlighted above, patients on ivermectin died 26% less frequently. This result is statistically significant: relative risk = 0.74, confidence interval = 0.56-0.97, p = 0.027.

You may be asking whether there are any good studies of ivermectin. Yes, there are.

A group which has evaluated all the studies of ivermectin has removed those with “significant issues,” such as Reis, leaving 70. A meta-analysis was conducted using the most serious outcome measured by each study. So, for example, that might be mortality for one study and hospitalization for another. There were a total of 125,440 patients in these 70 studies and the net result was a 69% lower risk for those patients on ivermectin in early treatment (p < 0.0001), a 48% lower risk for patients in late treatment (p < 0.0001), and an 84% lower risk for patients with prophylaxis (p < 0.0001). Even statisticians can cheer these results.

While writing this post, I’ve swung between feelings of humor and horror. Are the defective papers referenced in Hernandez representative of what passes for science today? If so, science is in trouble. And we should remember that the lives of people are at risk. People can and almost certainly have died because of the conclusions of papers such as Hernandez.

Science is in trouble if so many bad studies make it into a supposedly objective meta-analysis, overwhelm the results, drive the authors to a conclusion that isn’t supported by a wider review of the data, and cause people like Ronald Bailey to repeat these erroneous results.

Next, I’ll review the other meta-analyses that Bailey used to reach his conclusion, but I expect to find more of the same garbage flowing in and garbage flowing out. GIGO.