New methodological approach allows more precise summary of study results

Before new drugs are launched on the market they are tested in clinical studies in which one group of study participants often receives the new treatment and another group receives the current standard treatment. The results of these studies are needed for regulatory approval and later for so-called benefit assessments, which compare the advantages and disadvantages of the new and previous treatment: Is the new treatment more effective, does it have milder or more severe side effects, how do patients do afterwards, and so on. If the new treatment is better overall, it has an "added benefit."

When there are several studies on a medical question, meta-analyses are performed for the benefit assessment. A meta-analysis is a statistical technique that combines the data from several studies in order to draw the most reliable conclusion possible about the benefit or harm of a treatment. A prerequisite for a meta-analysis is that the studies are similar enough to avoid comparing apples and oranges. For example, the patients should have comparable disease severity and the studies should have been conducted as similarly as possible.

However, the studies often differ at least slightly in these or other aspects, which means that the effectiveness of the treatment varies somewhat from study to study. This is called heterogeneity of treatment effects. Often, only a few studies are suitable for evaluation, which leads to some uncertainty in the assessment of benefits, especially in the case of heterogeneity.

In these situations, the German health technology assessment (HTA) agency, the Institute for Quality and Efficiency in Health Care (IQWiG) had previously used a complex approach—several so-called frequentist meta-analyses were calculated and then compared. Different assumptions were made about the heterogeneity of the treatment effects between the different studies.

If the comparison of the meta-analyses based on the different assumptions did not show a clear overall result, a purely qualitative summary of the study results was made. This means that the studies were not statistically pooled in an analysis, but could only be described individually. However, it was then not possible to quantify the extent of the added benefit; a lot of work for an often unsatisfactory and vague result.

A team of researchers led by Professor Dr. Ralf Bender from IQWiG and Professor Dr. Tim Friede from the University Medical Center Göttingen (UMG) has now developed a simpler approach that only requires a single meta-analysis and is more likely to produce quantifiable conclusions about benefit: So-called Bayesian random-effects meta-analyses can take into account existing information on heterogeneity from previous analyses of similar studies.

To ensure that the assumed heterogeneity of the studies is neither too large nor too small, the required information is derived from previous benefit assessments stored in an IQWiG database. "This will make the procedure much easier for everyone involved in the future," says Ralf Bender, Head of Medical Biometry at IQWiG.

"In addition, the new empirical approach leads to a standardized and transparent procedure, as the preliminary information is based on previous benefit assessments. This should help to avoid tedious discussions about the approach in the future," adds Tim Friede, Director of the Institute of Medical Statistics at the UMG.

Heterogeneity assumptions derived and successfully tested

The team had already published a corresponding model for deriving the necessary prior information (so-called prior distributions or priors) at the beginning of 2023. A new publication in Research Synthesis Methods follows on from this: Recommendations for plausible heterogeneity assumptions for different effect measures, namely hazard ratio, risk ratio, odds ratio and standardized mean difference, have now been derived from a database containing all meta-analyses from IQWiG's benefit assessments up to the end of 2021.

Compared with similar publications by other research groups, the recommended distributions tend to cover lower levels of heterogeneity. The authors assume that this is due to the more precise inclusion criteria in IQWiG's benefit assessments: The studies all meet the requirements of the so-called PICO scheme (Population, Intervention, Comparison, Outcome), i.e. they examine, for example, clearly defined patient populations and interventions. They are therefore more similar to each other than studies included in many other medical meta-analyses.

A comparison of the effect estimates and confidence intervals using the old, complex meta-analysis method and the new, simplified one shows that the results are generally in good agreement. While there is often an increase in precision, the Bayesian meta-analyses sometimes also lead to more cautious and conservative conclusions. The authors therefore argue for the continued use of qualitative summaries of study results. If the results of the qualitative summary and the Bayesian meta-analysis are in agreement about the effect, it can now be quantified using the Bayesian analyses.

This approach markedly reduces the proportion of meta-analyses with non-quantifiable results: from 23% in the old approach based on three or four studies in IQWiG's previous benefit assessments, to 6% now.

Work in progress

The team's conclusion: When there are only two studies for a research question, the Bayesian meta-analysis does not offer clear advantages, so the frequentist approach is still generally used. When there are three or four studies, a Bayesian meta-analysis should be performed with the proposed priors for heterogeneity and compared with a qualitative summary. For five or more studies, frequentist meta-analyses are still performed. In this way, meta-analyses based on fewer studies will be simpler and more precise for IQWiG.

Such an approach is likely to be suitable for other organizations with a similar remit. Particularly in light of the implementation of the EU HTA Regulation, it is important to have clear empirically based priors available that are suitable for benefit assessments when using Bayesian methods in meta-analyses.

A software solution is being developed to integrate the new approach into IQWiG reports. In addition, IQWiG's meta-analysis database will be maintained so that the proposed priors for the heterogeneity parameters can be further developed.