Importance of different sources of bias in systematic reviews of controlled trials: systematic review of empirical studies

Objective

To examine the importance of different sources of bias in controlled trial research.

Methods

Systematic review and meta-analysis of empirical studies. We performed citation searches, searched the Cochrane Review Methodology Database and performed searches by hand of the proceedings of conferences. We included empirical studies which (i) compared estimates of treatment effects between trials that differed with respect to trial quality or publication type, (ii) were based on several meta-analyses and (iii) avoided confounding by disease area or intervention. We identified 13 articles or abstracts; seven studies met our inclusion criteria. The results are presented as ratios of odds ratios (RORs). RORs were, for example, used to compare the results of trials of inadequate quality with those of trials of adequate quality and published trials with unpublished trials. A ROR of 1 indicates that there is no difference in estimated treatment effects between these groups of trials whereas a ratio below 1 would indicate, for example, that the published trials showed a more beneficial treatment effect than unpublished trials. A ratio above 1 would indicate the opposite. We combined RORs using a random-effects model.

Results

Based on a total of 103 meta-analyses and 960 trials, 4 empirical studies examined the importance of concealment of treatment allocation and blinding. Three studies examined the generation of the allocation sequence (58 meta-analyses, 567 trials), 2 studies each examined publication bias (101 meta-analyses, 1,250 trials) and language bias (68 meta-analyses, 811 trials) and 1 study was concerned with indexing in MEDLINE (66 meta-analyses, 741 trials).

Conclusions

To our knowledge this is the first attempt to gauge the relative importance of different sources of bias that may distort the body of evidence from randomised clinical trials. Our results indicate that inadequate design and conduct of trials is, on average, more important than biased reporting and dissemination of trials. We may have underestimated the impact of reporting bias if hidden trials were missed by meta-analysts. The studies included were based on meta-analyses that complied with recommendations.

Have something to say? Post a comment on this article!