Statisticians review landmark HIV vaccine trial

Hopes ran high in 2009 when a New England Journal of Medicine article announced success in developing a vaccine against HIV, the virus that causes AIDS.

But a new Bayesian statistical analysis initiated at Duke demonstrates that the results were far from conclusive.

“The world has been waiting for an HIV vaccine for almost 30 years,” said Victor DeGruttola, a biostatistician at the Harvard School of Public Health who participated in the new analysis. "Its efficacy is an emotionally charged issue.” The HIV vaccine study, called RV 144, had been viewed as the “first evidence that suggested we were close."

New analysis of the RV144 data shows it is more likely that the vaccine actually has no effect compared to what was presented in the initial analysis. The new analysis also highlights the need to regularly use multiple statistical methods to review new medicines, said James Berger, a statistician at Duke University who spearheaded the study along with Peter Gilbert of the University of Washington and DeGruttola.

The results appear in the April 1 issue of the Journal of Infectious Diseases. The new analysis uses Bayesian statistics, which includes information from the original data and also data from previous HIV vaccine research.

The original RV144 study reported new HIV infections in 51 of the 8,197 volunteers given the vaccine versus 74 of the 8,198 participants given the placebo. These results suggested that the vaccine, a combination of two other failed vaccines, was 31.2 percent effective, with a p-value of .04. Many researchers and journalists interpreted the statistics to mean that there was only a four percent chance that the vaccine was not effective, and alternatively a 96 percent chance that it was 31-percent effective.

“But it’s not that simple,” DeGruttola said. He was dismayed with the interpretation and criticized it in the journal Science. He said the correct way to think about the numbers is to assume the vaccine does nothing. “Assume it is water. Repeating the experiment 50 times, researchers would expect that one out of 50 times they'd find equal-to or greater-than 31.2 percent vaccine efficacy,” he said, adding that these standard statistics ultimately say nothing direct about whether the vaccine works.

Berger said Bayesian analysis improves upon the standard statistical methods because it can make statements about the chance that the vaccine had some efficacy. The P-values, which were used in the original analysis, cannot. In the new analysis, the researchers explain that the p-value of 0.04 was misinterpreted and say that, using Bayesian statistics, there is no more than a 71 percent chance that the vaccine was effective at preventing HIV.

In other words, there is a 29 percent chance that the vaccine was not effective, not a 4 percent chance, he said.

Because of what is at stake in HIV vaccine research, the conversation related to RV144 has not always been as scientific as it could have been, Gilbert said. He has reviewed and devised HIV vaccine trials since 1996 and said the new analysis provides statisticians and non-statisticians with a “golden opportunity” to have rational, scientific, and careful discussions when determining “false positive flukes” versus real effects.

“The new evidence doesn’t give us any reason for great joy, or greater gloom,” said De Gruttola, but the researchers agree that the Bayesian analysis provides a new framework to add increased rigor into future vaccine trials.