How a study about chronic fatigue syndrome was doctored, adding to pain and stigma

March 23, 2017 by Steven Lubet, The Conversation

The public relies on scientists to report their findings accurately and completely, but that does not always happen. Too often, researchers announce only their most favorable outcomes, while keeping more disappointing results well out of sight.

This phenomenon, first identified by the psychologist Robert Rosenthal in 1979, is called the "file drawer problem." Although it is widely recognized – affecting drug trials, psychology experiments and most other fields – it has seldom been documented, for obvious reasons. Suppressed results are, well, suppressed, and they are usually discovered only by chance.

It was therefore almost unprecedented when a group of patients, at the end of last year, successfully unmasked the skewed data behind an influential British study, first published in The Lancet in 2011, of the devastating disease known as chronic fatigue syndrome (sometimes called myalgic encephalomyelitis or ME/CFS).

My interest in this issue is both professional and personal. As a law professor, I have devoted much of my career to the study of judicial ethics, including the problem of implicit biases that can undermine the reliability of both court trials and clinical trials.

I have also been living with ME/CFS for over a decade, so I am acutely attuned to the need for responsible and transparent research on the illness. Unfortunately, the most extensive study of ME/CFS – called the PACE trial – was deeply flawed from its inception, in ways that the principal investigators have yet to acknowledge.

'Dysfunctional' beliefs all too real for those in pain

The story began in 2005, when a group of psychiatrists set out to test their theory that ME/CFS is primarily a psychosocial illness, characterized by patients' "unhelpful cognitions" and their "dysfunctional" beliefs that their symptoms are caused by an organic disease.

Under this assumption, they recruited over 600 ME/CFS patients for the PACE trial and randomly divided them into four categories. One group was treated with cognitive behavior therapy (CBT), a form of psychotherapy that addresses patients' "false perceptions" of their illness, and a second group received graded exercise therapy (GET), which consisted of supervised increases in their activity levels. The other two groups were essentially controls, receiving neither of the treatments under study.

In a 2013 article in Psychological Medicine, the PACE team announced its most striking results. This follow-up article claimed that the therapy arms of the study – CBT and GET – had achieved impressive 22 percent recovery rates – not just improvement rates – as opposed to only 7 or 8 percent in the control arms.

The result was enthusiastically promoted in the press, but many patients were suspicious, especially of the GET outcomes, which contradicted their experience of debilitating crashes following the simple movements of daily life.

ME/CFS patients have consistently explained that exertion exacerbates their worst symptoms. For many, even moderate exercise can result in a days-long crash, in which they are nearly immobilized by muscle weakness and joint pain. In the U.S., post-exertional relapse has been recognized as the defining characteristic of the illness by the Centers for Disease Control, the National Institutes of Health and the Institute of Medicine.

For the PACE investigators, however, the announced recovery results validated their conviction that psychotherapy and exercise provided the key to reversing ME/CFS.

There was just one problem. A subsequent investigation found that the PACE investigators had changed the standard for recovery midstream, weakening one of the key criteria to the point that a subject could actually have gotten worse in the course of the trial and yet still count as "recovered" following supervised GET.

Unraveling the mystery

Here is how it worked, as shown by the investigation: At the outset of the trial, patients were recruited who scored at 65 or lower on a measure called the physical function score, and recovery was defined as achieving a subsequent score of 85 or higher, which indicates a relatively healthy person.

Before the unblinded trial was completed, however, the definition of recovery was reduced to a score of 60, which was below the level that qualified research subjects in the first place.

It was the change in this outcome measure (and several others) that allowed the PACE researchers to declare their favorable outcome for GET. The unimpressive results under the original protocol went unpublished, as though they had been stuck in a a figurative file drawer.

When the Psychological Medicine article was published in 2013, members of the patient community immediately pointed out the discrepancy. Because the study had been publicly funded, they sought the underlying data under the U.K.'s Freedom of Information law. The PACE investigators refused to release any of the raw results.

In October 2015, David Tuller of the University of California at Berkeley published a lengthy expose of the PACE trial, pointing out the jiggered outcome measure, as detailed above, and many other flaws. His report attracted the attention of numerous American scientists who joined an open letter seeking an independent review of the PACE data.

Finally, in summer 2016, a British Freedom of Information tribunal ordered the PACE team to unlock the file drawer and disclose their raw data. A revelation followed.

Exaggerated recovery claims

A group of patients and scholars reanalyzed the PACE data according to the original determinants and, as suspected, the "recoveries" under CBT and GET all but disappeared. As they reported last December in a peer-reviewed medical journal, the recovery rate for CBT fell to seven percent and the rate for GET fell to four percent, which were statistically indistinguishable from the three percent rate for the untreated controls.

Thus, the PACE investigators proved nothing more than a familiar adage among statisticians: If you torture the data, they will confess anything.

Researchers in the U.S. and Australia have recently made great progress toward identifying biomarkers for ME/CFS, which may lead to an effective medical intervention. Over 100 prominent researchers, clinicians and organizations have called on Psychological Medicine to retract the PACE article, although the journal has not yet publicly responded.

Thanks to the original PACE announcement, however, graded exercise is still routinely prescribed throughout the U.S. and the U.K. despite reports that the treatments can cause intolerable pain and relapse. Those who question GET are often told that they must simply exercise more, no matter how badly they crash afterward.

It is bad enough to torture the data, but it is indefensible to torture patients based on manipulated results.

Explore further: Fatal flaws in PACE chronic fatigue syndrome follow-up study

Related Stories

Fatal flaws in PACE chronic fatigue syndrome follow-up study

November 2, 2015
At the outset, let me say that I'm skeptical whether we can hold the PACE investigators responsible for the outrageous headlines that have been slapped on their follow-up study and on the comments they have made in interviews.

Researchers identify promising treatments for chronic fatigue

January 31, 2013
(Medical Xpress)—Two treatments most likely to lead to recovery from chronic fatigue syndrome (CFS) have been identified by UK researchers.

Reducing fear avoidance beliefs key to improving symptoms and reducing disability in chronic fatigue syndrome

January 14, 2015
Reducing fears that exercise or activity will make symptoms worse is one of the most important factors determining the success of cognitive behaviour therapy (CBT) or graded exercise therapy (GET) in reducing fatigue and ...

New music strategy shows 70 per cent increase in exercise adherence

May 20, 2015
The use of personalized music playlists with tempo-pace synchronization increases adherence to cardiac rehab by almost 70 per cent—according to a study published in Sports Medicine -Open.

The Internet may be an effective tool for treating chronic knee pain

February 21, 2017
An online intervention combining home exercise and pain-coping skills training provided substantial clinical benefits for patients suffering from chronic knee pain. This model of care delivery could greatly improve patient ...

Recommended for you

Probing how Americans think about mental life

October 20, 2017
When Stanford researchers asked people to think about the sensations and emotions of inanimate or non-human entities, they got a glimpse into how those people think about mental life.

Itsy bitsy spider: Fear of spiders and snakes is deeply embedded in us

October 19, 2017
Snakes and spiders evoke fear and disgust in many people, even in developed countries where hardly anybody comes into contact with them. Until now, there has been debate about whether this aversion is innate or learnt. Scientists ...

Dutch courage—Alcohol improves foreign language skills

October 18, 2017
A new study published in the Journal of Psychopharmacology, conducted by researchers from the University of Liverpool, Maastricht University and King's College London, shows that bilingual speakers' ability to speak a second ...

Inflamed support cells appear to contribute to some kinds of autism

October 18, 2017
Modeling the interplay between neurons and astrocytes derived from children with Autism Spectrum Disorder (ASD), researchers at University of California San Diego School of Medicine, with colleagues in Brazil, say innate ...

Study suggests psychedelic drugs could reduce criminal behavior

October 18, 2017
Classic psychedelics such as psilocybin (often called magic mushrooms), LSD and mescaline (found in peyote) are associated with a decreased likelihood of antisocial criminal behavior, according to new research from investigators ...

Taking probiotics may reduce postnatal depression

October 18, 2017
Researchers from the University of Auckland and Otago have found evidence that a probiotic given in pregnancy can help prevent or treat symptoms of postnatal depression and anxiety.


Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.