Email updates

Keep up to date with the latest news and content from BMC Medical Research Methodology and BioMed Central.

Open Access Research article

Does updating improve the methodological and reporting quality of systematic reviews?

Beverley Shea1*, Maarten Boers2, Jeremy M Grimshaw3, Candyce Hamel4 and Lex M Bouter5

Author Affiliations

1 Community Information and Epidemiological Technologies (CIET), Ottawa, Ontario, Canada and VU University Medical Center, Amsterdam, The Netherlands

2 Professor of Clinical Epidemiology and Director, Department of Clinical Epidemiology and Biostatistics, VU University Medical Center, Amsterdam, The Netherlands

3 Professor and Director, Clinical Epidemiology Program, Ottawa Health Research Institute, Department of Medicine, University of Ottawa, Ottawa, Ontario, Canada

4 Community Information and Epidemiological Technologies (CIET), Ottawa, Ontario, Canada

5 Professor of Epidemiology and Director, Institute for Research in Extramural Medicine (EMGO Institute), VU University Medical Center, Amsterdam, The Netherlands

For all author emails, please log on.

BMC Medical Research Methodology 2006, 6:27  doi:10.1186/1471-2288-6-27


The electronic version of this article is the complete one and can be found online at: http://www.biomedcentral.com/1471-2288/6/27


Received:21 October 2005
Accepted:13 June 2006
Published:13 June 2006

© 2006 Shea et al; licensee BioMed Central Ltd.

This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

Background

Systematic reviews (SRs) must be of high quality. The purpose of our research was to compare the methodological and reporting quality of original versus updated Cochrane SRs to determine whether updating had improved these two quality dimensions.

Methods

We identifed updated Cochrane SRs published in issue 4, 2002 of the Cochrane Library. We assessed the updated and original versions of the SRs using two instruments: the 10 item enhanced Overview Quality Assessment Questionnaire (OQAQ), and an 18-item reporting quality checklist and flow chart based upon the Quality of Reporting of Meta-analyses (QUOROM) statement. At least two reviewers extracted data and assessed quality. We calculated the percentage (with a 95% confidence interval) of 'yes' answers to each question. We calculated mean differences in percentage, 95% confidence intervals and p-values for each of the individual items and the overall methodological quality score of the updated and pre-updated versions using OQAQ.

Results

We assessed 53 SRs. There was no significant improvement in the global quality score of the OQAQ (mean difference 0.11 (-0.28; 0.70 p = 0.52)). Updated reviews showed a significant improvement of 18.9 (7.2; 30.6 p < .01) on the OQAQ item assessing whether the conclusions drawn by the author(s) were supported by the data and/or analysis presented in the SR. The QUOROM statement showed that the quality of reporting of Cochrane reviews improved in some areas with updating. Improvements were seen on the items relating to data sources reported in the abstract, with a significant difference of 17.0 (9.8; 28.7 p = 0.01), review methods, reported in the abstract 35 (24.1; 49.1 p = 0.00), searching methods 18.9 (9.7; 31.6 p = 0.01), and data abstraction 18.9 (11.7; 30.9 p = 0.00).

Conclusion

The overall quality of Cochrane SRs is fair-to-good. Although reporting quality improved on certain individual items there was no overall improvement seen with updating and methodological quality remained unchanged. Further improvement of quality of reporting is possible. There is room for improvement of methodological quality as well. Authors updating reviews should address identified methodological or reporting weaknesses. We recommend to give full attention to both quality domains when updating SRs.

Background

A number of papers have been published on the methodological and reporting quality of reviews. A review of 86 English language meta-analyses published between 1950 and 1986 by Sacks [1] assessed every report on fourteen items covering six content areas believed to be critical in the reporting of meta-analysis. Only 28% of these meta-analyses were found to address all six content areas. An updated survey in 1996 showed little change [2]. Shea [3] compared the methodological quality of paper-based and, electronic systematic reviews and found little difference and a lot of room for improvement. Assendelft [4] reviewed 51 reviews and noted that reviews that favoured a given intervention tended to have higher methodological quality scores. In contrast, Jadad and McQuay [5] reviewed 80 systematic reviews published between 1980 and 1992 and found a disconcerting link between reviews whose results favoured an intervention and poor methodological quality. Jadad found that Cochrane reviews had greater methodological rigor, more frequent updates and higher overall quality scores than those published in peer-reviewed paper journals, though both types were found to contain extensive and serious flaws.

The Cochrane Collaboration [6] is an international not-for-profit organization that conducts and updates systematic reviews of healthcare studies. With the exception of a few studies completed by Jadad [7,8] and Shea [9], little is known about the quality of Cochrane reviews and whether their quality is improving over time and with updating. To our knowledge, the impact of updating on their quality has not been evaluated nor studied.

Studies of the quality of systematic reviews can focus on methodological or reporting quality. Methodological quality is concerned with how well a systematic review was designed and conducted (e.g. literature searching, pooling of data, etc.). Reporting quality considers how well systematic reviewers have reported their methodology and findings.

The purpose of our study was to compare the methodological and reporting quality of Cochrane systematic reviews with that of their updated versions in order to determine whether updating contributed significantly to the improvement of their quality in these two dimensions.

Methods

We selected all updated Cochrane systematic reviews from the Cochrane Database of Systematic Reviews 2002 [10], and the same reviews prior to their update. Updated reviews were chosen following the definition of 'updating' included in the Cochrane handbook [11].

Based on a previously published study [12], two instruments chosen to assess the quality of Cochrane systematic reviews for this study were the enhanced Overview Quality Assessment Questionnaire (OQAQ) scale [12-14] (1) and the Quality of Reporting of Meta-analsyses (QUOROM) checklist [15] (2).

Additional File 1. The enhanced version of the overview quality assessment questionnaire (OQAQ).

Format: DOC Size: 41KB Download file

This file can be viewed with: Microsoft Word ViewerOpen Data

Additional File 2. Quality of reporting of meta-analyses (QUOROM) for randomized controlled trials.

Format: DOC Size: 61KB Download file

This file can be viewed with: Microsoft Word ViewerOpen Data

The OQAQ was selected because it has strong face validity, provided data on several essential elements of its development, and had a published assessment of its construct validity available [13]. In addition, its validity has been thoroughly tested and established using a number of independent measures [14]. However, we noted difficulty applying the questions so we developed an enhanced version of the OQAQ which incorporated guidelines for its' use [12]. The OQAQ scale measures across a continuum using nine questions (items 1–9) designed to assess various aspects of the methodological quality of systematic reviews and one overall assessment question (item 10). When the scale is applied to a systematic review, the first nine items are scored by selecting either yes, no, partial/can't tell. The tenth item requires assessors to assign an overall quality score on a 7-point scale [13].

The QUOROM statement was chosen for assessing reporting quality. Although, this checklist has not yet been fully validated, extensive work has been conducted and reported [15]. The QUOROM statement is comprised of a checklist and flow diagram and was developed using a consensus process designed to strengthen the reliability of the estimates it yields when applied by different assessors. It estimates the overall reporting quality of systematic reviews. The checklist asks whether authors have provided readers with information on 18 items, including searches, selection, validity assessment, data abstraction, study characteristics, quantitative data syntheses and trial flow. It also asks whether authors have included a flow diagram with information about the number of randomized controlled trials identified, included and excluded, and the reasons for any exclusion. Individual checklist items included in this instrument are answered either yes, no and partial/can't tell [15].

For each included Cochrane systematic review, we calculated the percentage (with a 95% confidence interval) of 'yes' answers to each item of OQAQ and QUOROM. Individual Cochrane systematic reviews were then compared to their updated versions with respect to the percentage of 'yes' ratings received. A percentage difference (with a 95% confidence interval and p-values) of 'yes' answers was calculated for each individual question. In addition, a mean difference (with a 95% confidence interval and p-value) was calculated for the overall quality score (OQAQ) of the updated and pre-updated versions.

Assessments of all individual reviews were conducted independently by at least two assessors. One of these assessors (CH) had been involved with the Cochrane Collaboration for six years. The other assessor (BS) had been involved with the Cochrane Collaboration for ten years, and had carried out a number of systematic reviews. A third assessor (DF) was available to assist in the resolution of assessment discrepancies.

Results

In total, 53 reviews were included. The mean period between the publication of the original and updated versions of the reviews was 2.7 years (range four months to five years). Complete results for both instruments applied to the reviews are provided in Tables 1 and 2, but highlights are summarized below.

Table 1. Comparisons of percent 'yes' and percent 'differences' using the enhanced Overview Quality Assessment Questionnaire (OQAQ).

Table 2. Comparisons of percent 'yes' and percent 'differences' using the Quality of Reporting of Meta-analyses (QUOROM).

OQAQ

Table 1 presents the methodological quality assessments obtained using this scale. There was no significant difference in the global assessment (item 10 – How would you rate the scientific quality of the overview?) (Mean score original review 4.70, mean score updated review 4.81, difference in means 0.11 (95% CI -0.28; 0.70 p = 0.52)).

There were improvements on seven individual items, although only one item showed a significant improvement (item 9 – Were the conclusions made by the author(s) supported by the data and/or analysis reported in the overview?) (percent original reviews complying with item 76% (95% CI 4.47; 4.94) percentage updated reviews complying with item 94% (95% 88.1; 100.0) difference 18.9% (95% CI 7.2%; 30.6% p < 0.01)).

QUOROM

Table 2 presents the assessments of reporting quality obtained using the Quality of Reviews of Meta-analyses (QUOROM) checklist. Scores (awarded for 'yes' response) on the 18 items for original Cochrane reviews ranged from 0% (item 1) to 100% (items 2 and 18). Four of the 18 items revealed improvements relating to data sources reported in the abstract (item 4) with a significant difference of 17.0% (9.8; 28.7 p = 0.01), review methods reported in the abstract (item 5) with a difference of 35% (24.1; 49.1 p = 0.00), searching methods (item 9) with a difference of 18.9% (9.7; 31.6 p = 0.01), and data abstraction (item 12) with a difference of 18.9% (11.7; 30.9 p = 0.00).

Three questions, item (10) -3.7% (95% CI: -9.0; 3.8 p = 0.15), item (14) -3.7% (-14.8; 6.4 p = 0.62) and item (16), -5.7% (-15.8; 3.0 p = 0.40) had lower mean scores on updated reviews than on original reviews, but these differences were not statistically significant.

Discussion

In assessing the quality of the sample of 53 systematic reviews of the Cochrane Collaboration, two assessment tools were used. The larger improvement on the QUOROM checklist than on the OQAQ suggests that although the quality of reporting has improved slightly, the quality of design and conduct has not changed. For example, the items in reporting of selection criteria and searching are improved on QUOROM, but the equivalent items on OQAQ, which relate to how well these were carried out have not changed.

The significant improvements for the OQAQ item assessing whether the conclusions drawn by the author(s) were supported by the data and/or analysis reported in the overview is also worthy of note. This could suggest that authors are paying more attention to state their conclusion in relation to the data provided. This item has been reported by other methodologists in the context of asessing quality [16,17].

Improvement in the quality of reporting of the abstract, as assessed by the QUOROM instrument, might also suggest that authors are paying more attention to the details. For example, authors might be more aware of better access to the Cochrane abstracts through electronic systems such as Medline. Also, reporting of literature searches improved. However, the literature searches themselves did not improve. This should be further explored.

Assessors expressed that they found the two instruments used in the study to have associated practical weaknesses. Their combined length (28 items) made their use somewhat cumbersome and inefficient. Another problem encountered was the lack of adequate published guidance on the application of the OQAQ question, assessing the overall methodological quality. The QUOROM checklist had the advantage of coming with fairly detailed user instructions. However, it proved very time-consuming to apply [12]. Another problem noted was that several of the questions asked by these two instruments appeared to cover the same subject matter.

The methodological and reporting quality of Cochrane reviews was fair-to-good, but further improvement is obviously needed in both areas. This was also concluded in a recent study by Moja [16]. In addition, the quality-improvement impact of updating was found to be relatively minor. On some assessed factors, particularly with respect to reporting quality, updated reviews actually scored lower than original reviews. Currently, Cochrane review updates are carried out primarily to incorporate new findings, rather than to improve quality. It would be beneficial for updates to also address reporting, such as the omission of methodological descriptions. And of course, updates should also try to improve methodological weaknesses of the reviews.

The Cochrane Collaboration endeavours to improve the quality of its systematic reviews through the application of a continuous peer review process during their development. The effectiveness of the peer-review process could probably be improved by increased attention to areas of reporting and/or methodological weaknesses. Reviewers should adhere more faithfully to the guidelines provided in the Cochrane handbook [17] in order to improve the design and conduct of reviews and to the QUOROM statement [15] to improve the quality of reporting of systematic reviews.

Conclusion

The overall quality of Cochrane reviews is fair-to-good. Although quality of reporting improved on certain individual items there was no overall improvement seen with updating and methodological quality remained unchanged. Further improvement of quality of reporting is possible. There is room for improvement of methodological quality as well. Authors updating reviews should address identified methodological or reporting weaknesses. We recommend to give full attention to both quality domains when updating SRs.

Competing interests

The author(s) declare that they have no competing interests.

Authors' contributions

BS designed the protocol and the study, abstracted the data, conducted the analysis, and wrote the paper; MB designed the protocol and the study, and assisted with writing the paper; JMG provided intellectual input into the design and conduct of the study, and assisted with writing the paper; CH carried out the data abstraction, quality assessment and assisted with the analysis; LMB provided intellectual input into the design and conduct of the study, and assisted with writing the paper.

Acknowledgements

The authors would like to acknowledge Daniel Francis for his assistance with the quality assessment, Ron Habinski, Ashley Porter and Crystal Huntly-Ball for their comments on earlier drafts of this paper; David Moher for his methodological recommendations; Tim Ramsey for his statistical advice, and our external reviewer Penny Whiting for her helpful suggestions.

References

  1. Sacks HS, Berrier J, Reitman D, Ancona-Berk VA, Chalmers TC: Meta-analyses of randomized controlled trials.

    N Engl J Med 1987, 316:450-5. PubMed Abstract OpenURL

  2. Sacks HS, Reitman D, Pagano D, Kupelnick B: Meta-analysis: an update.

    Mount Sinai Journal of Medicine 1996, 63:216-24. PubMed Abstract OpenURL

  3. Shea B: Assessing the quality of reporting meta-analyses of randomized controlled trials. In MSc thesis. University of Ottawa, Department of Epidemiology and Community Medicine; 1999. OpenURL

  4. Assendelft WJJ, Koes BW, Knipschild PG, Bouter LM: The relationship between methodological quality and conclusions in reviews of spinal manipulation.

    JAMA 1995, 274:1942-1948. PubMed Abstract | Publisher Full Text OpenURL

  5. Jadad AR, McQuay HJ: Meta-analyses to evaluate analogies interventions: a systematic qualitative review of their methodology.

    J Clin Epidemiol 1996, 49:235-43. PubMed Abstract | Publisher Full Text OpenURL

  6. The Cochrane Collaboration [http://www.cochrane.org/] webcite

  7. Jadad AR, Cook DJ, Jones A, Klassen TP, Tugwell P, Moher M, Moher D: Methodology and reports of systematic reviews and meta-analyses: a comparison of Cochrane reviews with articles published in paper-based journals.

    JAMA 1998, 280:278-280. PubMed Abstract | Publisher Full Text OpenURL

  8. Jadad AR, Moher M, Brownman GP, Booker L, Sigouin C, Fuentes M, Stevens R: Systematic reviews and meta-analyses on treatment of asthma: critical evaluation.

    BMJ 2000, 320:537-540. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  9. Shea B, Moher D, Graham I, Pham B, Tugwell P: A comparison of the quality of reporting of Cochrane review and systematic reviews published in paper-based journals.

    Evaluations and the Health Professions 2002, 25:116-129. Publisher Full Text OpenURL

  10. The Cochrane Library. Chichester, UK: John Wiley & Sons, Ltd; 2002. OpenURL

  11. Clarke M, Oxman AD, editors: Cochrane Reviewers' Handbook. In The Cochrane Library. UK: Update Softeware; 2002. OpenURL

  12. Shea B, Dube C, Moher D: Assessing the quality of reports of systematic reviews: the QUOROM statement compared to other tools. In Systematic reviews in health care: meta-analysis in context. Edited by Egger M, Smith GD, Altman DG. London, UK: BMJ Publishing Group; 2001:122-39. OpenURL

  13. Oxman AD, Guyatt GH: Validation of an index of the quality of review articles.

    J Clin Epidemiol 1991, 44:1271-8. PubMed Abstract | Publisher Full Text OpenURL

  14. Oxman AD: Checklists for review articles.

    BMJ 1994, 309:648-651. PubMed Abstract OpenURL

  15. Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, Stroup DF: Improving the quality of reports of meta-analyses of randomized controlled trials: the QUOROM statement. Quality of reporting of meta-analyses.

    Lancet 1999, 354:1896-900. PubMed Abstract | Publisher Full Text OpenURL

  16. Moja LP, Telaro E, D'Amico R, Moschetti I, Coe L, Liberati A, on behalf of the Metaquality Study Group: Assessment of methodological quality of primary studies by systematic review: results of the metaquality cross sectional study.

    BMJ 2005, 330:1053-1058. PubMed Abstract | Publisher Full Text | PubMed Central Full Text OpenURL

  17. The Cochrane Library. Chichester, UK: John Wiley & Sons, Ltd; 2005. OpenURL

Pre-publication history

The pre-publication history for this paper can be accessed here:

http://www.biomedcentral.com/1471-2288/6/27/prepub