By Kate Johnson – October 14, 2009
In just a few days Dr. Supachai Rerks-Ngarm, from Thailand’s Ministry of Health, and members of the U.S. Military will present their HIV vaccine study to their peers at the AIDS Vaccine conference in Paris.
It won’t be their first presentation of their findings, but they probably wish it was.
Their first presentation to the world’s media was a bit of a circus, that left many wishing the substance had matched the hype.
As a medical journalist I’ve seen my share of circus acts, and collected a whole folder of abandoned stories to show for it.
They are half-finished articles about published, or yet-to-be published medical studies that I started writing and then discarded because things just didn’t seem to add up. These are not stories that I’ve set aside easily. I have labored over data, interviewed the authors, queried the discrepancies, and torn my hair out. In the case of one study, which was already published in a medical journal, I even tried to take my doubts to the journal’s editor in chief, who had issued a glowing press release about the study. In the end, I abandoned all of these stories because I doubted the integrity of the science.
For me, such abandoned articles are time and money down the drain. More troubling though is the knowledge that other people, more scientifically savvy than me, have turned a blind eye to this dishonesty. Not only is shoddy science making it to the podium of scientific meetings, or the pages of medical journals, it is influencing future research and patient care.
I’ve called this phenomenon dishonest – a word I have chosen carefully and with some hesitation. Certainly not all shoddy science is dishonest. Some is just sloppy, clumsy, and inaccurate. But according to an unpublished analysis presented at the Sixth International Congress of Peer Review and Biomedical Publication, and reported in Nature Medicine , only 28% of medical paper retractions were for “honest error” and a whopping 43% were for unethical reasons.
In my last blog I talked about the pharmaceutical industry and how shoddy science from this arena could be considered smart marketing. Whether it’s ethical is another question.
But Big Pharma is not the only bad guy. As we’ve seen from the latest ghostwriting scandal, which I’ve blogged about here , and expanded upon here, independent researchers also play a role in dishonest scientific publishing.
In fact, they may even play a bigger role than the pharmaceutical industry, according to another unpublished analysis presented at the Peer Review conference.
“We absolutely should not let up on our scrutiny of industry,” Karen Woolley, one of the investigators, told Nature. “But why are we always pointing our finger over there? There’s an elephant in the room, and that’s the nonfinancial conflicts of interest in academia.”
“Academic data doesn’t get scrutinized at all,” commented Liz Wager, another co-investigator.
A recent study in the Journal of the American Medical Association uncovered bias or the potential for bias in almost 70% of more than 300 published drug or device trials. As one of the co-authors, David Moher, Ph.D. told me, most of the bias came from selective reporting of outcomes, in other words, telling only part of the story.
Dr. Moher and I discussed how the general public may not fully understand the independent researcher’s motive for this kind of dishonesty. Certainly the academic pressure to “publish or perish” is easy enough to understand. But the barriers to medical publishing may be less obvious.
I see these barriers first hand in my capacity as a writing and editing consultant for researchers. Medical journals reject many submissions. Fearing rejection, researchers may be tempted to “improve” their papers by putting a spin on their findings. Another analysis presented at the Peer Review congress detected inappropriate spin in more than 40% of papers with statistically insignificant findings, reported Nature Medicine.
“There’s some very, very questionable behavior that goes on by people in research,” says Dr. Moher. Examples include “trying to hide certain parts of studies, trying to make things look good, or being so fuzzy that its impossible to figure out what’s going on”.
This type of dishonesty is hard to pinpoint. As my folder of unfinished articles shows, it takes time and a lot of hair pulling to realize that things don’t add up. For the mainstream press, and even hurried medical professionals that realization may take even longer.
The recent news about the HIV vaccine is a good example. The U.S. Military and Thailand’s Ministry of Public Health conducted the study and released their results to the lay press before running them through the peer-review process of a scientific conference or medical journal. What started as headline news about a 31% reduction in HIV transmission then fizzled into a suggestion that the results may have been a statistical fluke.
Soon after, Jon Cohen’s blog in Science voiced complaints from an anonymous HIV/AIDS expert: “The press conference was not a scholarly, rigorously honest presentation…it doesn’t meet the standards that have been set for other trials, and it doesn’t fully present the borderline results. It’s wrong.”
The investigators will have a second chance in Paris and we will all be watching.
It will be interesting to see if they change their spin.