Research article

The interpretation of systematic reviews with meta-analyses: an objective or subjective process?

Ian Shrier¹ , Jean-François Boivin^1,2 , Robert W Platt² , Russell J Steele³ , James M Brophy^2,4 , Franco Carnevale⁵ , Mark J Eisenberg¹ , Andrea Furlan⁶ , Ritsuko Kakuma² , Mary Ellen Macdonald⁷ , Louise Pilote^2,4 and Michel Rossignol^2,1

¹Centre for Clinical Epidemiology and Community Studies, Lady Davis Institute for Medical Research, SMBD-Jewish General Hospital, McGill University, Montreal, Canada

²Department of Epidemiology and Biostatistics, McGill University, Montreal, Canada

³Department of Mathematics and Statistics, McGill University, Montreal, Canada

⁴McGill University Health Centre, McGill University, Montreal, Canada

⁵School of Nursing, McGill University, Montreal, Canada

⁶Institute for Work and Health, Toronto, Canada

⁷Department of Oncology, McGill University, Montreal, Canada

author email corresponding author email

BMC Medical Informatics and Decision Making 2008, 8:19doi:10.1186/1472-6947-8-19

Published:	21 May 2008

Abstract

Background

Discrepancies between the conclusions of different meta-analyses (quantitative syntheses of systematic reviews) are often ascribed to methodological differences. The objective of this study was to determine the discordance in interpretations when meta-analysts are presented with identical data.

Methods

We searched the literature for all randomized clinical trials (RCT) and review articles on the efficacy of intravenous magnesium in the early post-myocardial infarction period. We organized the articles chronologically and grouped them in packages. The first package included the first RCT, and a summary of the review articles published prior to first RCT. The second package contained the second and third RCT, a meta-analysis based on the data, and a summary of all review articles published prior to the third RCT. Similar packages were created for the 5^th RCT, 10^th RCT, 20^th RCT and 23^rd RCT (all articles). We presented the packages one at a time to eight different reviewers and asked them to answer three clinical questions after each package based solely on the information provided. The clinical questions included whether 1) they believed magnesium is now proven beneficial, 2) they believed magnesium will eventually be proven to be beneficial, and 3) they would recommend its use at this time.

Results

There was considerable disagreement among the reviewers for each package, and for each question. The discrepancies increased when the heterogeneity of the data increased. In addition, some reviewers became more sceptical of the effectiveness of magnesium over time, and some reviewers became less sceptical.

Conclusion

The interpretation of the results of systematic reviews with meta-analyses includes a subjective component that can lead to discordant conclusions that are independent of the methodology used to obtain or analyse the data.