Missing the forest (plot) for the trees? A critique of the systematic review in tobacco control
1 Dept. of Health Promotion, School of Public Health, Sackler Faculty of Medicine, Tel Aviv University, POB 39040, Ramat Aviv 69978, Israel
2 Dept. of Statistics, Faculty of Exact Sciences, Tel Aviv University, POB 39040, Ramat Aviv 69978, Israel
3 Dept. of Occupational Medicine, Israel Ministry of Health, Ben Tabai 2, Jerusalem 93591, Israel
BMC Medical Research Methodology 2010, 10:34 doi:10.1186/1471-2288-10-34Published: 25 April 2010
The systematic review (SR) lies at the core of evidence-based medicine. While it may appear that the SR provides a reliable summary of existing evidence, standards of SR conduct differ. The objective of this research was to examine systematic review (SR) methods used by the Cochrane Collaboration ("Cochrane") and the Task Force on Community Preventive Services ("the Guide") for evaluation of effectiveness of tobacco control interventions.
We searched for all reviews of tobacco control interventions published by Cochrane (4th quarter 2008) and the Guide. We recorded design rigor of included studies, data synthesis method, and setting.
About a third of the Cochrane reviews and two thirds of the Guide reviews of interventions in the community setting included uncontrolled trials. Most (74%) Cochrane reviews in the clinical setting, but few (15%) in the community setting, provided pooled estimates from RCTs. Cochrane often presented the community results narratively. The Guide did not use inferential statistical approaches to assessment of effectiveness.
Policy makers should be aware that SR methods differ, even among leading producers of SRs and among settings studied. The traditional SR approach of using pooled estimates from RCTs is employed frequently for clinical but infrequently for community-based interventions. The common lack of effect size estimates and formal tests of significance limit the contribution of some reviews to evidence-based decision making. Careful exploration of data by subgroup, and appropriate use of random effects models, may assist researchers in overcoming obstacles to pooling data.