The use of evidence-based approaches to therapeutic decision making can frequently raise the problem of how to make decisions when evidence is in limited supply. Often systematic reviews limit trial inclusion in an attempt to generate clinical homogeneity and allow sensible meta-analysis. The problem, though, is that other, useful, information is omitted. An example of this is the popular combination of paracetamol with codeine for treatment of acute and, more frequently, chronic pain. For the combination of 1000 mg paracetamol plus 60 mg codeine, for instance, there was information on only 127 patients in two placebo-controlled acute pain studies [1, 2].

Systematic reviews should seek unbiased evidence, which may limit the number of studies available for analysis. One approach to resolving the problem of apparently insufficient evidence may be to assess evidence of lower methodological quality. The amount of bias that could result from this approach would be a concern.

Schultz et al [3] demonstrated that lack of randomisation is the major source of bias in trials; studies which are not randomised can lead to overestimation of treatment effects by up to 40%. Restricting systematic reviews to include only randomised studies therefore makes sense for reviews of effectiveness. A classic example is a review of transcutaneous nerve stimulation for post-operative pain relief. Randomised studies overwhelmingly showed no benefit over placebo, while non-randomised studies did show benefit [4].

Non-blinded studies over-estimate treatment effects by about 17% [3]. In a review of acupuncture for back pain [5], the inclusion of both blinded and non-blinded studies changed the overall conclusion. The blinded studies showed 57% of patients improved with acupuncture and 50% with control, a non-significant relative benefit of 1.2 (95% confidence interval 0.9 to 1.5). Five non-blinded studies showed a difference from control, with 67% improved with acupuncture and 38% with control. Here the relative benefit was significant at 1.8 (1.3 to 2.4).

Trials of poor reporting quality consistently overestimate the effect of treatment. Using a validated scoring system for methodological quality [6], studies of lower quality are likely to overestimate treatment effects [7, 8]. Other sources of bias may include small trials [9,10,11], covert duplication [12], and geography [13]. Vickers and colleagues [13] showed that trials of acupuncture conducted in east Asia were universally positive, while those conducted in Australasia, north America or western Europe were positive only about half the time. Randomised trials of therapies other than acupuncture conducted in China, Taiwan, Japan or Russia/USSR were also overwhelmingly positive.

There is also the issue of the overall validity of a randomised trial. In some areas, like acute pain, valid methods for the conduct of clinical studies have been set out for many years, and are well understood [14]. There are therefore many trials that are randomised and double blind, and conducted on patients with the same initial severity of pain under similar conditions and assessing identical or similar outcomes over the same time periods. Trials with low validity are more likely to have a positive result than those with higher validity [15], seen in acupuncture for head and neck pain.

It is obviously sensible to avoid bias where it is likely to occur. That means avoiding studies with designs or features where bias is possible. What other strategies remain when faced with apparently inadequate information from the literature? There are four that could be applied to the particular combination of paracetamol 1000 mg plus codeine 60 mg. These are:

1. The number of patients required for the number needed to treat (NNT) to be within ± 0.5 of the true value varies with the efficacy of an analgesic [11]. We can therefore calculate how confident we can be in an NNT given the number of patients in the analysis, or calculate the number of patients needed to reach a required level of certainty.

2. We can assess evidence from other dose combinations of paracetamol and codeine and see whether there is a dose response relationship in placebo-controlled trials.

3. We can assess results for paracetamol 1000 mg plus codeine 60 mg against results for active comparators in high quality active controlled and placebo controlled studies. This can then be compared with systematic reviews done in similar clinical settings.

4. We can assess paracetamol 1000 mg plus codeine 60 mg in studies which, though randomised and double blind, had designs that did not allow their inclusion in a meta-analysis.

We applied these approaches to paracetamol 1000 mg plus codeine 60 mg through systematic review of the published literature.

Full reports of randomised controlled trials of oral paracetamol combined with codeine for acute postoperative pain were sought. Different search strategies were used to identify eligible reports in MEDLINE (1966 - March 2000), EMBASE (1980 - March 2000), the Cochrane Library (Issue 3, 2000) and the Oxford Pain Relief Database (1950 - 1994) [16]. Reference lists of retrieved reports and reviews [1,2] were searched for additional trials. Abstracts, review articles and unpublished reports were not considered.

Criteria for inclusion for postoperative pain were: full journal publication, randomised controlled trials which included single dose treatment groups of oral paracetamol combined with codeine, double blind design, baseline postoperative pain of moderate to severe intensity, patients over 15 years of age, at least 10 patients per group, and the pain outcome measures of total pain relief (TOTPAR) or summed pain intensity difference (SPID) over 4-6 hours or sufficient data provided to allow their calculation. Pain measures allowed for the calculation of TOTPAR or SPID were a standard five point pain relief scale (none, slight, moderate, good, complete), a standard four point pain intensity scale (none, mild, moderate, severe) or a standard visual analogue scale (VAS) for pain relief or pain intensity. Each report was scored for quality using a three item, 1-5 score, quality scale [6].

For each trial, mean TOTPAR, SPID, VASTOTPAR or VASSPID values for each drug group were converted to %maxTOTPAR by division into the calculated maximum value [17]. The proportion of patients in each treatment group who achieved at least 50%maxTOTPAR was calculated using valid equations [18,19,20]. The number of patients with >50%maxTOTPAR was then used to calculate relative benefit and NNT for paracetamol plus codeine versus placebo. Relative benefit and relative risk estimates were calculated with 95% confidence intervals (CI) using a fixed effects model [21]. NNT with 95% confidence intervals was calculated by the method of Cook and Sackett [22]. A statistically significant difference from control was assumed when the 95% confidence interval of the relative benefit did not include 1. Confidence intervals of proportions were calculated according to Morris and Gardner [23]. Calculations were performed using Excel v 5.0 on a Power Macintosh G3. Simulations and calculations of probability were conducted as described in Moore et al, 1998 [11].

There were three placebo controlled [24,25,26] and three active controlled studies [27,28,29] involving paracetamol 1000 mg plus codeine 60 mg (Additional material: Table 3). Quality scores were 3 or better for all six. Information on other dose combinations is in Additional material: Table 4

In placebo controlled studies of paracetamol 1000 mg plus codeine 60 mg 65/114 patients given paracetamol plus codeine had at least 50% pain relief compared with 9/83 for placebo (Table 1). For a single dose of paracetamol/codeine the proportion of patients with at least 50% pain relief was 57% (48% to 66%). The NNT for at least 50% pain relief over four to six hours was 2.2 (1.7 to 2.9).

Additional analyses undertaken to assess the reliability of this assessment of paracetamol 1000 mg plus codeine 60 mg included:

It can be the case that attempts to practice evidence based medicine come up against the problem that there is no evidence, or relatively little evidence. Reviews of paracetamol and codeine [1, 2] revealed that the combination of paracetamol 1000 mg plus codeine 60 mg was apparently highly effective, with an NNT of 1.9, but this was derived from only 127 patients in two trials. Using evidence from sources other than direct comparisons of paracetamol/codeine with placebo buttresses the conclusion of these earlier reviews.

Firstly we uncovered information about another trials that increased the number of patients to 197. The inclusion of the third trial barely changed the NNT from 1.9 to 2.2. Unpublished information from a fourth study is presently being sought.

We know that the amount of information required to be sure that an NNT is within ± 0.5 of the true value varies depending on the efficacy of treatment [11, 53]. Where efficacy is high, the requirement is for small numbers of patients. Where efficacy is low, the requirement is for large numbers of patients. For paracetamol 1000 mg plus codeine 60 mg, calculations show that with 197 patients we can be about 92% confident that the NNT obtained is within ± 0.5 of the true value, close to the conventionally accepted value of 95%. To be within the 95% limits we would need information from a further 60 to 80 patients.

Biological plausibility derives also from a substantial dose response relationship with the placebo controlled studies as the doses of paracetamol and of paracetamol and codeine increase. Seventeen additional studies of high quality (randomised, double blind, with clear entry criteria and outcomes measures) with 1195 additional patients demonstrated this (Table 1; Figure 1). Dose response was also shown when information from placebo and active controlled trials was pooled to examine the overall percentage of patients with at least 50% pain relief (Table 1).

This type of analysis additionally demonstrated that in trials examining paracetamol/codeine combinations, the paracetamol 1000 mg plus codeine 60 mg combination was much better than either paracetamol 1000 mg or codeine 60 mg alone (Table 2; Figure 2). There was clear separation between the lower confidence interval for paracetamol 1000 mg plus codeine 60 mg (59%) and the higher confidence interval for paracetamol 1000 mg (50%) and codeine 60 mg (44%). This confirms the added benefit seen with this combining codeine with paracetamol in a previous review [1].

Moreover, paracetamol 1000 mg plus codeine 60 mg demonstrated efficacy similar to effective doses of NSAIDs drugs tested in the selfsame trials (Table 2; Figure 2). There is now considerable and growing evidence on the effectiveness of NSAIDs as analgesics in acute pain [54, 55]. While ibuprofen 400 mg performed relatively poorly in two trials here, pooled information from large numbers of studies show it to be an effective analgesic with an NNT of 2.7 (2.5 to 3.0) [54]. Ketorolac has also been shown to be effective orally; the 10 mg dose had an NNT of 2.6 (2.3 to 3.1) [55]. In trials involving NSAIDs these effective analgesics are no better than paracetamol/codeine combinations.

Only six randomised comparisons of paracetamol 1000 mg plus codeine 60 mg with placebo or other analgesics without placebo were available for analysis using methods that have become traditional for analysis in acute pain [14]. This number was doubled by examining studies that were randomised and double blind, but which used different pain intensity or relief outcomes, or other study designs. These additional six studies confirmed that paracetamol 1000 mg plus codeine 60 mg was an effective analgesic in acute pain.

Taken together, this approach shows two things. There is considerable supportive evidence for the conclusion that paracetamol 1000 mg plus codeine 60 mg is an effective analgesic with a low NNT. This support could be produced without resource to low quality studies, or those with known propensity for bias.

It will not always be possible to extend knowledge though high quality and essentially bias free trials in all circumstances: acute pain is particularly rich in trials, and especially those of high quality. Strategies like this will always be preferable. When placebo controlled trials are not available conclusions may have to be drawn from other trials. The techniques described here may be useful in circumstances where large numbers of high quality placebo controlled randomised trials are not available to us. Observational studies of good quality need not give results that are different from those of randomised trials, as testified by three recent studies [56,57,58]. But the potential for bias through lack of quality needs always to be addressed.