Debate

A system for rating the stability and strength of medical evidence

Jonathan R Treadwell¹ , Stephen J Tregear¹ , James T Reston¹ and Charles M Turkelson²

¹  ECRI Evidence-Based Practice Center and Health Technology Assessment Group, 5200 Butler Pike, Plymouth Meeting, Pennsylvania 19462, USA

²  American Academy of Orthopaedic Surgeons, 6300 North River Road, Rosemont, Illinois 60018, USA

author email corresponding author email

BMC Medical Research Methodology 2006, 6:52doi:10.1186/1471-2288-6-52

Published:	19 October 2006

Abstract

Background

Methods for describing one's confidence in the available evidence are useful for end-users of evidence reviews. Analysts inevitably make judgments about the quality, quantity consistency, robustness, and magnitude of effects observed in the studies identified. The subjectivity of these judgments in several areas underscores the need for transparency in judgments.

Discussion

This paper introduces a new system for rating medical evidence. The system requires explicit judgments and provides explicit rules for balancing these judgments. Unlike other systems for rating the strength of evidence, our system draws a distinction between two types of conclusions: quantitative and qualitative. A quantitative conclusion addresses the question, "How well does it work?", whereas a qualitative conclusion addresses the question, "Does it work?" In our system, quantitative conclusions are tied to stability ratings, and qualitative conclusions are tied to strength ratings. Our system emphasizes extensive a priori criteria for judgments to reduce the potential for bias. Further, the system makes explicit the impact of heterogeneity testing, meta-analysis, and sensitivity analyses on evidence ratings. This article provides details of our system, including graphical depictions of how the numerous judgments that an analyst makes can be combined. We also describe two worked examples of how the system can be applied to both interventional and diagnostic technologies.

Summary

Although explicit judgments and formal combination rules are two important steps on the path to a comprehensive system for rating medical evidence, many additional steps must also be taken. Foremost among these are the distinction between quantitative and qualitative conclusions, an extensive set of a priori criteria for making judgments, and the direct impact of analytic results on evidence ratings. These attributes form the basis for a logically consistent system that can improve the usefulness of evidence reviews.