Manor View Surgery, Bushey Health Centre, London Road, Bushey, Watford, UK

Abstract

Background

Calculation of numbers needed to treat (NNT) is more complex from meta-analysis than from single trials. Treating the data as if it all came from one trial may lead to misleading results when the trial arms are imbalanced.

Discussion

An example is shown from a published Cochrane review in which the benefit of nursing intervention for smoking cessation is shown by formal meta-analysis of the individual trial results. However if these patients were added together as if they all came from one trial the direction of the effect appears to be reversed (due to Simpson's paradox).

Whilst NNT from meta-analysis can be calculated from pooled Risk Differences, this is unlikely to be a stable method unless the event rates in the control groups are very similar. Since in practice event rates vary considerably, the use a relative measure, such as Odds Ratio or Relative Risk is advocated. These can be applied to different levels of baseline risk to generate a risk specific NNT for the treatment.

Summary

The method used to calculate NNT from meta-analysis should be clearly stated, and adding the patients from separate trials as if they all came from one trial should be avoided.

Introduction

Calculation of summary statistics from single trials and pooled data

In a single trial that reports outcomes in a dichotomous (binary) fashion the results can be reported in a variety of ways. The relative effect of treatment may be reported as an Odds Ratio, or as a Risk Ratio. These ratios describe the effect of treatment in reducing or increasing the odds or risks of events, and may be fairly independent of the patients' baseline risk status. In contrast the reduction in risk may be described as a Risk Difference (otherwise described as Absolute Risk Reduction), and this may be turned into a Number Needed to Treat (NNT) by taking the inverse of the Risk Difference.

Discussion

Example of Simpson's paradox from a Cochrane review of nursing interventions for smoking cessation

Thus, for example, a single trial of nursing interventions

The results of all such high intensity nursing interventions are summarised in a systematic review on the Cochrane Library

Risk Difference meta-analysis.

Risk Difference meta-analysis of trials of high intensity nursing intervention for smoking cessation (showing proportion of patients in each trial arm who had ceased smoking at the longest follow-up). The trial results are combined using the Mantel-Haenszel fixed effects method.

The raw totals from each trial can be added together and these are shown at the bottom of each column, 546/3820 stopped smoking in the nursing group and 356/2301 did so without intervention from nurses. Some authors have advocated the use of these proportions to calculate NNT from the pooled results of systematic reviews.

Adding together the total number of patients from each trial who stop smoking following intensive nursing intervention gives a cessation rate 14.3% (546 out or 3820), whilst adding those who stop in the placebo groups yields 15.5% (356 out of 2301). Thus it would appear that overall 1.2% less patients stop smoking when nurses intervene, suggesting a Number Needed to Harm or NNT(H) of 100/1.2 = 83. This is in contrast to the result of calculating the Risk Difference from each trial and combining the weighted trial results, which yield a pooled Risk Difference of 0.037, that is to say that 3.7% more patients stop with help from the nurses (NNT 100/0.037 = 27).

The direction of the effect has been reversed using the raw totals due to Simpson's paradox, because the arms in the individual trials are not equal in size. For example the Hollis study has roughly three times as many patients who have a nursing intervention as the control group, because there are four active arms in the trial. Three of these arms used nursing intervention and these have been added together for the purposes of the meta-analysis. This avoids the danger of triple-counting the control group, which would occur if each arm were entered separately against the control arm in the meta-analysis.

There are further theoretical reasons why the raw totals should not be used to calculated NNT or any other pooled effect from meta-analysis of controlled trials. In each trial the patients are randomly assigned to either the treatment or control group, and conventional analysis using weighted trial effects preserves the benefit of randomisation by considering the patients in each trial only in comparison with that trial's controls. When raw totals are added together the treated patients in one trial are compared to the controls in all the trials and the benefits of randomisation are lost.

Finally adding the patients as if they all came from a single trial prevents us from considering the differences in treatment effects between the trials, and as a consequence the confidence intervals may be too narrow. In the example shown in Figure

Risk Difference meta-analysis of trials.

Risk Difference meta-analysis of trials of high intensity nursing intervention for smoking cessation (showing proportion of patients in each trial arm who had ceased smoking at the longest follow-up). The meta-analysis is performed using a random effects model and the confidence interval of the pooled result is wider than for the fixed effects method.

The point estimate is altered slightly using the random effects model because the weights given to each trial are different and more weight is given to the smaller trials. The confidence interval is considerably wider using a random effects model and now includes a risk difference of zero. The fact that the pooled result is altered when a random model is used should be included, as part of a sensitivity analysis, but this cannot be done if only the sum of the raw totals is considered.

It should be noted that Simpson's paradox will also alter the direction of effect of all the other summary statistics if the raw totals are used, and that this is independent of whether statistical significance is present. The shift is caused by imbalance in the size or the treatment and control arms (not the heterogeneity that is present in this case).

Limitations of Number Needed to Treat as a summary statistic

Thus far I have considered the disadvantages of using raw totals from meta-analysis to calculate NNT from pooled data instead of a pooled Risk Difference, which can be inverted (with its confidence interval) to form a pooled NNT

As can be seen from Figure

If we pool the results of these studies using Odds Ratios (which have better properties for calculation

Odds Ratio meta-analysis of trials.

Odds Ratio meta-analysis of trials of high intensity nursing intervention for smoking cessation (showing proportion of patients in each trial arm who had ceased smoking at the longest follow-up). The trial results are combined using the Mantel-Haenszel fixed effects method.

The pooled Odds Ratio of 1.39 can be applied to the control group odds and the NNT derived from the type of patients in the Hollis study with low rates of smoking cessation (2% in the control group) would be 125, whilst the patients of the type included in the Miller study (20% cessation rate in the control group) would yield an NNT of 17. For the "average" patient (15.5% cessation rates in the pooled control groups) the NNT is 21.

If NNT is used as the main descriptive measure of the result of trials or meta-analyses without reference to the baseline risks of the included patients there is a danger of seriously misleading the reader, who in the above case might wrongly assume that the type of nursing intervention used by Miller was far more effective than that used by Hollis. In trying to make the results easier for clinicians to apply, the use of NNT without reference to baseline risk may inadvertently give the impression of spurious differences between treatments.

An example of this problem can be seen in a recent Bandolier article on Nicotine replacement for smoking cessation

Moreover there is imbalance in the size of the trial arms in the included studies, so use of the raw totals for calculation is subject to Simpson's paradox as well. Meta-analysis with pooled risk differences for all trials would produce an NNT of 17 for gum (compared to the reported NNT of 12 from the raw totals) and an NNT of 18 for patches (compared to the reported NNT of 17 from raw totals).

A suggested way forward

Egger

It is wise to limit this conversion to patients whose expected risk lies within the control event rates of the included trials (which in the example from the Cochrane review is 2% to 50% cessation of smoking), and to use this method cautiously where, as in this example, there is heterogeneity in the Odds Ratio.

The formula for conversion of the pooled Odds ratio to an individual NNT is rather tedious

Conclusion

More clarity is required from the authors of meta-analyses that calculate NNT from pooled data; the method used to derive the NNT should be specified (as well as the control event rate), and simple addition of the patients from each trial should be avoided. The data from individual trials in a meta-analysis should always be included in published meta-analyses, as electronic publication has relieved the constraints of space in paper journals. One of the principles of systematic reviewing is that the processes should be transparent, so that others can repeat the analysis of the data so as to assess the influence of using different methodologies.

Competing Interests

CC devised Visual Rx, which is a free calculator available on the Internet for use with Java-enabled browsers to allow clinicians to convert pooled Odds Ratios into a Number Needed to Treat.

Editorial note

Two commentaries on this article are published alongside: from Andrew Moore and Henry McQuay

Acknowledgements

I am grateful to Jon Deeks and Steven Senn for helpful comments on the assessment of binary outcomes in Meta-analysis.

Pre-publication history

The pre-publication history for this paper can be accessed here: