How a study about chronic fatigue syndrome was doctored, adding to pain and stigma

March 23, 2017 by Steven Lubet, The Conversation

The public relies on scientists to report their findings accurately and completely, but that does not always happen. Too often, researchers announce only their most favorable outcomes, while keeping more disappointing results well out of sight.

This phenomenon, first identified by the psychologist Robert Rosenthal in 1979, is called the "file drawer problem." Although it is widely recognized – affecting drug trials, psychology experiments and most other fields – it has seldom been documented, for obvious reasons. Suppressed results are, well, suppressed, and they are usually discovered only by chance.

It was therefore almost unprecedented when a group of patients, at the end of last year, successfully unmasked the skewed data behind an influential British study, first published in The Lancet in 2011, of the devastating disease known as chronic fatigue syndrome (sometimes called myalgic encephalomyelitis or ME/CFS).

My interest in this issue is both professional and personal. As a law professor, I have devoted much of my career to the study of judicial ethics, including the problem of implicit biases that can undermine the reliability of both court trials and clinical trials.

I have also been living with ME/CFS for over a decade, so I am acutely attuned to the need for responsible and transparent research on the illness. Unfortunately, the most extensive study of ME/CFS – called the PACE trial – was deeply flawed from its inception, in ways that the principal investigators have yet to acknowledge.

'Dysfunctional' beliefs all too real for those in pain

The story began in 2005, when a group of psychiatrists set out to test their theory that ME/CFS is primarily a psychosocial illness, characterized by patients' "unhelpful cognitions" and their "dysfunctional" beliefs that their symptoms are caused by an organic disease.

Under this assumption, they recruited over 600 ME/CFS patients for the PACE trial and randomly divided them into four categories. One group was treated with cognitive behavior therapy (CBT), a form of psychotherapy that addresses patients' "false perceptions" of their illness, and a second group received graded exercise therapy (GET), which consisted of supervised increases in their activity levels. The other two groups were essentially controls, receiving neither of the treatments under study.

In a 2013 article in Psychological Medicine, the PACE team announced its most striking results. This follow-up article claimed that the therapy arms of the study – CBT and GET – had achieved impressive 22 percent recovery rates – not just improvement rates – as opposed to only 7 or 8 percent in the control arms.

The result was enthusiastically promoted in the press, but many patients were suspicious, especially of the GET outcomes, which contradicted their experience of debilitating crashes following the simple movements of daily life.

ME/CFS patients have consistently explained that exertion exacerbates their worst symptoms. For many, even moderate exercise can result in a days-long crash, in which they are nearly immobilized by muscle weakness and joint pain. In the U.S., post-exertional relapse has been recognized as the defining characteristic of the illness by the Centers for Disease Control, the National Institutes of Health and the Institute of Medicine.

For the PACE investigators, however, the announced recovery results validated their conviction that psychotherapy and exercise provided the key to reversing ME/CFS.

There was just one problem. A subsequent investigation found that the PACE investigators had changed the standard for recovery midstream, weakening one of the key criteria to the point that a subject could actually have gotten worse in the course of the trial and yet still count as "recovered" following supervised GET.

Unraveling the mystery

Here is how it worked, as shown by the investigation: At the outset of the trial, patients were recruited who scored at 65 or lower on a measure called the physical function score, and recovery was defined as achieving a subsequent score of 85 or higher, which indicates a relatively healthy person.

Before the unblinded trial was completed, however, the definition of recovery was reduced to a score of 60, which was below the level that qualified research subjects in the first place.

It was the change in this outcome measure (and several others) that allowed the PACE researchers to declare their favorable outcome for GET. The unimpressive results under the original protocol went unpublished, as though they had been stuck in a a figurative file drawer.

When the Psychological Medicine article was published in 2013, members of the patient community immediately pointed out the discrepancy. Because the study had been publicly funded, they sought the underlying data under the U.K.'s Freedom of Information law. The PACE investigators refused to release any of the raw results.

In October 2015, David Tuller of the University of California at Berkeley published a lengthy expose of the PACE trial, pointing out the jiggered outcome measure, as detailed above, and many other flaws. His report attracted the attention of numerous American scientists who joined an open letter seeking an independent review of the PACE data.

Finally, in summer 2016, a British Freedom of Information tribunal ordered the PACE team to unlock the file drawer and disclose their raw data. A revelation followed.

Exaggerated recovery claims

A group of patients and scholars reanalyzed the PACE data according to the original determinants and, as suspected, the "recoveries" under CBT and GET all but disappeared. As they reported last December in a peer-reviewed medical journal, the recovery rate for CBT fell to seven percent and the rate for GET fell to four percent, which were statistically indistinguishable from the three percent rate for the untreated controls.

Thus, the PACE investigators proved nothing more than a familiar adage among statisticians: If you torture the data, they will confess anything.

Researchers in the U.S. and Australia have recently made great progress toward identifying biomarkers for ME/CFS, which may lead to an effective medical intervention. Over 100 prominent researchers, clinicians and organizations have called on Psychological Medicine to retract the PACE article, although the journal has not yet publicly responded.

Thanks to the original PACE announcement, however, graded exercise is still routinely prescribed throughout the U.S. and the U.K. despite reports that the treatments can cause intolerable pain and relapse. Those who question GET are often told that they must simply exercise more, no matter how badly they crash afterward.

It is bad enough to torture the data, but it is indefensible to torture patients based on manipulated results.

Explore further: Fatal flaws in PACE chronic fatigue syndrome follow-up study

Related Stories

Fatal flaws in PACE chronic fatigue syndrome follow-up study

November 2, 2015
At the outset, let me say that I'm skeptical whether we can hold the PACE investigators responsible for the outrageous headlines that have been slapped on their follow-up study and on the comments they have made in interviews.

Researchers identify promising treatments for chronic fatigue

January 31, 2013
(Medical Xpress)—Two treatments most likely to lead to recovery from chronic fatigue syndrome (CFS) have been identified by UK researchers.

Reducing fear avoidance beliefs key to improving symptoms and reducing disability in chronic fatigue syndrome

January 14, 2015
Reducing fears that exercise or activity will make symptoms worse is one of the most important factors determining the success of cognitive behaviour therapy (CBT) or graded exercise therapy (GET) in reducing fatigue and ...

New music strategy shows 70 per cent increase in exercise adherence

May 20, 2015
The use of personalized music playlists with tempo-pace synchronization increases adherence to cardiac rehab by almost 70 per cent—according to a study published in Sports Medicine -Open.

The Internet may be an effective tool for treating chronic knee pain

February 21, 2017
An online intervention combining home exercise and pain-coping skills training provided substantial clinical benefits for patients suffering from chronic knee pain. This model of care delivery could greatly improve patient ...

Recommended for you

Suicidal thoughts rapidly reduced with ketamine, finds study

December 14, 2017
Ketamine was significantly more effective than a commonly used sedative in reducing suicidal thoughts in depressed patients, according to researchers at Columbia University Medical Center (CUMC). They also found that ketamine's ...

Do bullies have more sex?

December 14, 2017
Adolescents who are willing to exploit others for personal gain are more likely to bully and have sex than those who score higher on a measure of honesty and humility. This is according to a study in Springer's journal Evolutionary ...

Children's screen-time guidelines too restrictive, according to new research

December 14, 2017
Digital screen use is a staple of contemporary life for adults and children, whether they are browsing on laptops and smartphones, or watching TV. Paediatricians and scientists have long expressed concerns about the impact ...

Eating together as a family helps children feel better, physically and mentally

December 14, 2017
Children who routinely eat their meals together with their family are more likely to experience long-term physical and mental health benefits, a new Canadian study shows.

The iceberg model of self-harm

December 14, 2017
Researchers have created a model of self-harm that shows high levels of the problem in the community, especially in young girls, and the need for school-based prevention measures.

Anti-stress compound reduces obesity and diabetes

December 13, 2017
For the first time, scientists from the Max Planck Institute of Psychiatry in Munich could prove that a stress protein found in muscle has a diabetes promoting effect. This finding could pave the way to a completely new treatment ...

0 comments

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.