Email updates

Keep up to date with the latest news and content from BMC Medical Research Methodology and BioMed Central.

Open Access Highly Accessed Research article

Do health technology assessments comply with QUOROM diagram guidance? An empirical study

Daniel Hind* and Andrew Booth

Author Affiliations

School of Health and Related Research (ScHARR), University of Sheffield, Regent Court, 30 Regent Street, Sheffield, S1 4DA, UK

For all author emails, please log on.

BMC Medical Research Methodology 2007, 7:49  doi:10.1186/1471-2288-7-49

Published: 20 November 2007

Abstract

Background

The Quality of Reporting of Meta-analyses (QUOROM) statement provides guidance for improving the quality of reporting of systematic reviews and meta-analyses. To make the process of study selection transparent it recommends "a flow diagram providing information about the number of RCTs identified, included, and excluded and the reasons for excluding them". We undertook an empirical study to identify the extent of compliance in the UK Health Technology Assessment (HTA) programme.

Methods

We searched Medline to retrieve all systematic reviews of therapeutic interventions in the HTA monograph series published from 2001 to 2005. Two researchers recorded whether each study contained a meta-analysis of controlled trials, whether a QUOROM flow diagram was presented and, if so, whether it expressed the relationship between the number of citations and the number of studies. We used Cohen's kappa to test inter-rater reliability.

Results

87 systematic reviews were retrieved. There was good and excellent inter-rater reliability for, respectively, whether a review contained a meta-analysis and whether each diagram contained a citation-to-study relationship. 49% of systematic reviews used a study selection flow diagram. When only systematic reviews containing a meta-analysis were analysed, compliance was only 32%. Only 20 studies (23% of all systematic reviews; 43% of those having a study selection diagram) had a diagram which expressed the relationship between citations and studies.

Conclusion

Compliance with the recommendations of the QUOROM statement is not universal in systematic reviews or meta-analyses. Flow diagrams make the conduct of study selection transparent only if the relationship between citations and studies is clearly expressed. Reviewers should understand what they are counting: citations, papers, studies and trials are fundamentally different concepts which should not be confused in a diagram.