Skip to main content

A framework for organizing and selecting quantitative approaches for benefit-harm assessment

Abstract

Background

Several quantitative approaches for benefit-harm assessment of health care interventions exist but it is unclear how the approaches differ. Our aim was to review existing quantitative approaches for benefit-harm assessment and to develop an organizing framework that clarifies differences and aids selection of quantitative approaches for a particular benefit-harm assessment.

Methods

We performed a review of the literature to identify quantitative approaches for benefit-harm assessment. Our team, consisting of clinicians, epidemiologists, and statisticians, discussed the approaches and identified their key characteristics. We developed a framework that helps investigators select quantitative approaches for benefit-harm assessment that are appropriate for a particular decisionmaking context.

Results

Our framework for selecting quantitative approaches requires a concise definition of the treatment comparison and population of interest, identification of key benefit and harm outcomes, and determination of the need for a measure that puts all outcomes on a single scale (which we call a benefit and harm comparison metric). We identified 16 quantitative approaches for benefit-harm assessment. These approaches can be categorized into those that consider single or multiple key benefit and harm outcomes, and those that use a benefit-harm comparison metric or not. Most approaches use aggregate data and can be used in the context of single studies or systematic reviews. Although the majority of approaches provides a benefit and harm comparison metric, only four approaches provide measures of uncertainty around the benefit and harm comparison metric (such as a 95 percent confidence interval). None of the approaches considers the actual joint distribution of benefit and harm outcomes, but one approach considers competing risks when calculating profile-specific event rates. Nine approaches explicitly allow incorporating patient preferences.

Conclusion

The choice of quantitative approaches depends on the specific question and goal of the benefit-harm assessment as well as on the nature and availability of data. In some situations, investigators may identify only one appropriate approach. In situations where the question and available data justify more than one approach, investigators may want to use multiple approaches and compare the consistency of results. When more evidence on relative advantages of approaches accumulates from such comparisons, it will be possible to make more specific recommendations on the choice of approaches.

Peer Review reports

Background

Some decisions on health care interventions are straightforward because the benefits clearly outweigh the harms or vice versa. Many decisions, however, require careful balancing of the benefits and harms. For example, in order to decide on the use of aspirin for the prevention of myocardial infarction, one would typically consider the risk reduction for myocardial infarction over a certain period of time (e.g. 10 years) as well as the increased risks for hemorrhagic stroke and gastrointestinal bleeding [1]. One would also need to consider that the benefit-harm comparison varies across patients since the risks for these outcomes and absolute treatment effects depend much on an individual patient’s profile including her or his preferences. Sometimes, the decisionmaking context is even more complex than for aspirin. Tamoxifen for the prevention of breast cancer, for example, modifies the risk not only for breast cancer, but also for endometrial carcinoma, bone fractures, pulmonary embolism, stroke and cataracts [2]. As with aspirin, the benefit-harm comparison of tamoxifen depends on a woman’s profile, which in this case includes age, the risk for invasive breast cancer, race and whether the uterus is intact or has been removed [3].

In situations where multiple outcomes, patient profiles and also patient preferences need to be considered, it is challenging to compare benefits and harms without a quantitative approach. Without a quantitative approach, it is not verifiable if and how different outcomes, patient profiles, patient preferences, and sources of evidence were considered and weighted. Non-quantitative assessments of benefits and harms may lead to inappropriate decisions for or against treatments. This lack of transparency may be less problematic in an individual decisionmaking context, but it seems unacceptable when major regulatory decisions or clinical guideline recommendations are at stake. A number of quantitative approaches have been developed and applied to handle the multidimensionality of a benefit-harm assessment [3–8]. But there is little guidance on the selection of an appropriate quantitative approach for a particular clinical question in a benefit and harm assessment [5, 7, 9]. Past reviews of quantitative approaches for benefit and harm assessment have not organized methods according to important characteristics [5, 7, 9]. A framework that recognizes their important characteristics and organizes them accordingly could be a step forward to understanding the common and different elements of existing approaches, to guide their further development, and support investigators, who plan to conduct a quantitative benefit and harm assessment, in their choice of approach.

Such an organizing framework could be particularly attractive for organizations or investigators who conduct or use systematic reviews because the literature is mostly silent about the use of quantitative benefit and harm assessment approaches in the context of a systematic review. Familiarity with available approaches and their key characteristics will help systematic reviewers to develop protocols that specify all sources and types of data needed for benefit-harm assessments. For example, additional database searches may be needed to identify evidence on baseline risks, harms or patient preferences that would be missed by standard searches that commonly focus on randomized trials. Therefore, our aim was to review existing quantitative approaches for benefit-harm assessment and to develop an organizing framework that helps understanding of differences among methods and their selection for a specific clinical question.

Methods

Scope of the literature review

Our purpose was to review quantitative approaches for benefit and harm assessments that use formulas or graphical displays to compare the benefits and harms. We evaluated methods feasible for use in systematic reviews or to use the data synthesized in systematic reviews. We did not consider theoretical frameworks and qualitative approaches for benefit and risk assessment, nor approaches that have not been used in the medical field. It is important to note that our review focused on quantitative assessment and not on the entire process of a benefit-harm assessment (e.g. in the context of a new drug approval) that includes quantitative and qualitative processes [10, 11]. Also, we did not review approaches for making treatment recommendations for populations or individual patients because this requires consideration of specific health care contexts, costs of treatment, and other contextual factors, which is beyond the scope of most systematic reviews.

Review of the literature

We began our search for quantitative benefit and harm approaches with key articles culled from the investigators’ reference libraries, including prior work on approaches for assessing benefits and harms. We looked for articles that were written for quantitative benefit and harm assessment, which included consideration of at least one outcome for both benefit and harm of a medical or public intervention. We included approaches that analyzed benefit and harm outcomes entirely separately as well as approaches that provided a benefit and harm comparison metric where single or multiple benefit and harm outcomes are put on the same scale, which we will call the benefit-harm comparison metric (e.g. Quality-adjusted Life Years [QALYs] or probability scale). We screened the reference lists of all included articles for more relevant articles. The group then discussed each article that seemed potentially relevant, as described below. We also reviewed the manuals of the Evidence Based Practice Center program of the Agency for Healthcare Research and Quality (AHRQ) (http://www.ahrq.gov/clinic/epc/) and of the Cochrane Collaboration.

We did not perform a formal systematic review of the literature because a review on the topic of quantitative benefit and harm assessment already existed [5] and because our focus was on organizing available approaches. We capitalized on the work done to create a list of relevant approaches [5], which allowed us to devote adequate resources to the main focus of developing an organizing framework that helps understanding the differences among methods and their selection for a specific clinical question.

Defining the decisionmaking context of a benefit and harm assessment and identification of key characteristics of existing quantitative approaches for benefit and harm assessment

Our team, consisting of clinicians, epidemiologists and statisticians, discussed the identified quantitative approaches for benefit and harm assessment and the context in which they might be used in 12 one-hour sessions. The discussion served to define properties that characterize quantitative approaches for benefit and harm assessments. Based on these characteristics we developed a simple algorithm that may guide investigators, systematic reviewers, guideline developers or policymakers in their selection of a quantitative approach for benefit-harm assessment. We iteratively defined key characteristics with which existing quantitative approaches for benefit and harm assessment can be described and that allow comparisons across quantitative approaches. We also recorded limitations inherent to each of them that may threaten their usefulness or limit their applicability for certain questions of benefit and harm. In addition, we prepared examples for quantitative approaches for benefit-harm assessment that would highlight differences between them.

Results

Decisionmaking context of benefit and harm assessment

We identified three characteristics of a decisionmaking context that need to be defined. First, the treatment comparison and population for which the benefit-harm assessment is made should be characterized. The comparison of interest can be an intervention versus no intervention or an intervention A versus an intervention B. A population can be broadly defined, for example as a screening or primary prevention population, or be restricted to a particular setting (e.g. primary care) or a particular clinical population (e.g. patients with manifest coronary heart disease). Secondly, the key benefit and harm outcomes of interest for which the evidence and the benefit-harm comparison is sought should be identified. The figure shows that there might be a single benefit and a single harm outcome that are of interest, or there could be multiple outcomes. As is noted in literature on the more general processes of assessing the benefits and harms of interventions in guidelines and systematic reviews, patient-important outcomes might be preferred as key outcomes but, sometimes, a surrogate outcome may be considered if there is a strong correlation with patient-important outcomes or if no evidence on patient-important outcomes is available. Finally, one should decide whether a benefit-harm comparison metric is desired that puts all outcomes on a common scale so that a single number will inform about the comparison of benefits and harms. The decisionmaking context is likely to be important to decide on the need for a benefit-harm comparison metric because the output of the available quantitative approaches differ substantially and may not be appropriate for all decisionmakers such as patients and their health care providers, clinical or public health guideline developers, regulatory agencies, policymakers, or payers.

Overview of existing quantitative approaches for benefit-harm assessment

We identified 16 approaches, which can be grouped into two broad categories (Figure 1): One category comprises simpler approaches that typically deal with a single outcome for benefit (e.g. prevention of myocardial infarction) and one outcome for harm (e.g. gastrointestinal bleeding). The single benefit and harm can, however, also be composite outcomes (e.g. cardiovascular events) that summarize several outcomes (acute coronary syndrome, stroke, and cardiovascular death). The more complex approaches consider multiple outcomes for either benefit or harm, or both. Of note, although some approaches like the Number needed to treat (NNT) [12, 13] and Number needed to harm (NNH) and (Quality-adjusted) Time without Symptoms and Toxicity (Q- TWiST) [14, 15] are mostly used when there is a single benefit and a single harm outcome. Researchers can use these approaches separately for different outcomes in situations where multiple outcomes are important (e.g. multiple NNTs and NNHs). In the figure, we categorized approaches according to how they are typically used in the medical literature (e.g. NNT and NNH are typically used for single outcomes), but listed a few of them in two categories if we could not clearly categorize them (e.g. Minimum Target Event Risk for Treatment [MERT], (Quality-adjusted) Time without Symptoms and Toxicity).

Figure 1
figure 1

Key characteristics of the benefit-harm question that may guide selection of quantitative assessment for benefit-harm assessment. Abbreviations: INHB, Incremental net health benefit; MCE, Minimum clinical efficacy; NCB, Net Clinical Benefit; NNT, Number needed to treat; NNH, Number needed to treat for harm; Q-Twist, (Quality-adjusted) Time without Symptoms and Toxicity: RBC, Risk–benefit contour; RV-NNT, Relative value adjusted number needed to treat; QFRBA, Quantitative Framework for Risk and Benefit Assessment; TURBO, Transparent Uniform Risk Benefit; BLRA, Benefit-less-risk analysis; PSM, Probabilistic simulation methods; MERT, Minimum Target Event Risk for Treatment; MCDA, Multicriteria decision analysis: RBP, Risk–benefit plane; SPM, Stated preference method; MAR, Maximum acceptable risk.

The figure shows that the use of a benefit and harm comparison metric further distinguishes various approaches. The number of outcomes, the need for a benefit and harm comparison metric, and the quality and quantitative of available data will likely drive the selection of approaches.

To facilitate the understanding of and between the approaches, we selected three approaches for a more detailed description (Table 1). NNT and NNH are examples of an approach that can be used if a single benefit and a single harm outcome is of interest or when multiple outcomes are treated separately. Multicriteria decision analysis (MCDA) [16, 17] and the Gail/National Cancer Institute [3] approach provide examples for approaches where multiple outcomes are considered and put on a benefit-harm comparison metric.

Table 1 Examples for quantitative approaches for benefit-harm assessment

Description of key characteristics of 16 quantitative approaches for benefit-harm assessment

We focus here on additional key characteristics that enable researchers to understand the similarities and differences and choose the appropriate approaches for their benefit-harm question and context. Papers focusing on each individual approach have described in detail the 16 approaches although not in the context of systematic reviews [5].

We identified the following additional key characteristics:

  1. 1.

    The type of data needed: Individual patient data have advantages over aggregate data typically available for evidence synthesis. In an individual study, information on the co-occurrence of benefit and harm outcomes is available for each patient. An important consequence of the availability of such individual data is that researchers can consider the joint probability of benefit and harm outcomes.

  2. 2.

    The type of analyses: Analyses for benefit-harm assessment could include any type of statistical analysis. There is a major distinction, however, between approaches that are data driven (deterministic) and approaches that use modeling (stochastic) where data are not described by unique values, but rather by probability distributions.

  3. 3.

    The type of benefit-harm comparison metric: Researchers may use absolute and relative metrics as well as QALYs.

  4. 4.

    Assumptions: In benefit-harm analyses researchers need to make a number of assumptions. For example, if researchers use a benefit-harm comparison metric, the assumption is that it is justifiable that outcomes are combined on a single scale. Other assumptions relate to the joint occurrence of separate outcomes. Some approaches assume that separate outcomes occur independently, which may be justifiable in some instances.

  5. 5.

    Consideration and incorporation of patient preferences: Some quantitative approaches explicitly consider patient preferences for different outcomes in order to weight the benefits and harms.

  6. 6.

    Types of presenting benefit risk comparisons: Researchers can use various formats to present the results of a quantitative benefit-harm assessment. Researchers may present the benefit-harm comparison as a difference in the number of events between a treatment or no treatment, or as a ratio. They may also express the comparison by the time gained or lost without symptoms through a treatment. Or, researchers can use graphics that depict the benefit-harm comparison for patients at different outcome risks or based on other patient characteristics.

In Table 2, we compare each of the 16 approaches against these key characteristics. Table 3 provides a more detailed discussion of each approach. Three approaches require individual patient data (Table 2) whereas 13 approaches do not require individual patient data. This means that, although most quantitative approaches were developed for use in randomized trials and observational studies, researchers can also use most of these approaches at the synthesis stage (systematic reviews). Fourteen of the approaches are data driven but four of them may also use simulation. One approach (Probabilistic Simulation Methods [PSM]) is entirely based on simulation. Twelve of the 16 approaches put benefit and harm outcomes on the same scale to provide a benefit and harm comparison metric. Only four approaches provide measures of uncertainty around the benefit and harm comparison metric. Four approaches could consider the joint distribution of benefit and harm outcomes for the estimation of uncertainty. But many examples using these four approaches do not consider the dependence between benefit and harm, but only consider their marginal distributions (i.e. consider them to be independent,). Five approaches use composites outcome for benefit and composite outcomes for harm while 12 approaches use multiple outcomes. Researchers have adapted or potentially could adapt nine of the 16 approaches to incorporate patient preferences.

Table 2 Key characteristics of quantitative approaches for benefits and harm assessment
Table 3 Brief description of the 16 approaches for quantitative benefit-harm assessment

Discussion

The main finding of our review of quantitative approaches for benefit and harm assessment used in the medical literature is a simple algorithm that categorizes existing quantitative approaches broadly into approaches that consider single or multiple benefit and harm outcomes and into approaches that use a benefit-harm comparison metric or present outcomes side by side. We also found that for most approaches, researchers use aggregate data so as to make the approaches suitable for systematic reviews even if that is not their intended purpose. Interestingly, only few approaches provide measures of uncertainty and none of the approaches considers a potential correlation between benefit and harm outcomes (joint distribution).

We identified a number of assumptions that researchers make when applying some of the quantitative approaches: First, for some approaches researchers assume that one or more benefit and harm outcomes can be put on the same scale to calculate a benefit and harm comparison metric. Challenges for putting different outcomes on the same scale include their relative importance to decisionmakers, simplification of the outcomes (e.g. dichotomizing continuous outcomes, which may lead to substantial loss of information), or different methods and timing in the ascertainment of different outcomes.

However, the advantages of a benefit-harm comparison metric may be substantial, for example, in the context of complex situations where multiple outcomes are important and where patient, provider, and policymaker preferences vary [7]. It is a great cognitive challenge to process such a multidimensional task without a benefit-harm comparison metric. The major advantage of using a benefit-harm comparison metric (over using an approach without such a common metric) is that it can make explicit assumptions about the relative importance of outcomes or the arbitrary selection of the evidence on benefits and harms or on baseline risks, and that sensitivity analyses can provide evidence as to how the benefit-harm comparison changes if different assumptions are made. Also, a single number may provide some advantages for the communication of benefit-harm comparison to patients because it avoids overwhelming the patients with data on multiple different outcomes.

Second, we were surprised to see that there were no quantitative approaches that considered or even discussed the joint distribution of benefit and harm outcomes, even when individual patient data were available. The joint distribution describes the correlation between benefit and harm outcomes. Trial reports commonly describe standard errors and confidence intervals for the benefit and harm outcomes separately, but rarely describe the joint distribution of the effects of the treatment on the benefit and harm outcomes. Without the joint distribution of all the effects, we have to assume independence of the benefit and harm effects. This may not yield a valid estimate of the uncertainty of the benefit-harm balance metric. Changes in reporting practices, such as online journal appendix materials or online repositories of covariance data for later data synthesis, could address this limitation. Systematic reviewers should keep in mind the limitation of not considering the joint distribution when interpreting results from a quantitative benefit and harm assessment.

The figure and table show a number of characteristics that help distinguish various existing quantitative approaches. While these characteristics are important for the selection of an appropriate quantitative approach, there are a number of additional considerations that researchers need to make because they have implications regarding the type of evidence included in the benefit-harm assessment. For example, clinical trials are commonly designed to provide high-quality evidence and sufficient power for benefit outcomes. Harms often receive much less attention in terms of accurate and valid methods of measurement [40]. Such asymmetry in the quality of outcome ascertainment affects the validity of a quantitative benefit and harm assessment, but it is yet unclear how to downgrade the quality of evidence for this reason.

In contrast to the framework developed here that focused entirely on the quantitative assessments, Lynd and others developed criteria that apply to the entire process of a benefit-harm assessment. This usually requires that researchers consider both quantitative and qualitative approaches to make conclusions regarding benefit-harm comparisons of health care interventions [10, 11, 41, 42]. Lynd and others proposed 10 criteria for benefit-harm assessments--be universal, inclusive, comprehensive, patient-sensitive, easily interpreted consider preferences, define when benefits outweigh harms, incorporate uncertainty, be flexible and integrate economic evaluations) [41]. We agree with these guiding principles but also think that researchers cannot readily use them to judge the adequacy of specific quantitative approaches. Whether or not a specific approach is adequate depends much on the type and quality of available data. Regulatory decisionmakers, guideline developers, or users of the evidence are likely to perceive the ease of use and ease of interpretation of quantitative approaches very differently because of different levels of methodological expertise or different perspectives. Therefore, we believe that our framework for organizing quantitative approaches is complementary to, rather than competing with, what Lynd and others have proposed. The frameworks proposed by our team, Lynd, and others support a systematic, well-structured, and transparent process for reducing the multidimensionality of a benefit-harm assessment.

Our review showed that current quantitative approaches for benefit-harm assessment might need some further development. Firstly, many quantitative approaches identified here focus on binary outcomes that occur just once, with or without consideration of time to event. Current methods need extensions that also consider different types of data. Some patient-important outcomes, such as quality of life or symptoms, cannot be expressed appropriately as binary outcomes without substantial loss of information. Some benefit and harm events can occur several times so that the number of events per person-time needs to be considered rather than the proportion of persons with at least one event. Secondly, uncertainty estimates for the benefit and harm comparison metric (e.g. 95 percent confidence or credible intervals) are likely to be of key importance for decisionmakers and organizations making treatment recommendations. Researchers do not commonly report estimates of uncertainty that arises from sampling variability. In addition, none of the methods considers the joint distribution of benefit and harm outcomes. Researchers should develop statistical methods for considering joint distributions when estimating standard errors for benefit-harm comparison metrics. For systematic reviews it would be valuable to develop approaches for making assumptions about joint distributions because covariance matrices are rarely available from reports of primary studies and it may be challenging to request them from authors of primary studies. Thirdly, researchers should develop systematic approaches for sensitivity analyses that assess the influence of the various assumptions commonly made. One approach would be to agree on a list of standard sensitivity analyses for key aspects of a benefit-harm assessment. For example, data for estimating baseline risks (e.g. probability of outcome without treatment) can come from different sources (e.g. surveillance data, observational studies, and placebo arms of randomized trials). The best available evidence on treatment effects may sometimes come from single randomized trial or observational study rather than from meta-analyses. Researchers may be able to derive patient preferences by different eliciting techniques. A systematic outline of these options (the choices for the primary analysis and for sensitivity analyses), would make benefit-harm assessments transparent and give users of the evidence a sense for how sensitive the results are to different assumptions.

A strength of our review is the collaborative effort of clinicians, epidemiologists, and statisticians that helped us to develop a comprehensive framework for characterizing quantitative approaches for benefit-harm assessment. Some may perceive it as a limitation that we did not conduct a separate formal systematic review but we capitalized on an existing, recent review [5]. Also, we used an iterative approach of developing a framework rather than following a more standardized approach, such as Delphi-like procedures, to identify important characteristics of quantitative approaches for benefit-harm assessment. However, a more standardized approach also has its limitations because it does not allow discussing intertwined issues or considering different perspectives of an interdisciplinary research group in great depth.

We developed a framework for the use of quantitative approaches for benefit-harm assessment that can help researchers select specific approaches. We do not make recommendations for or against specific approaches. It is too early to make such recommendations because of the lack of evidence from studies that directly compare quantitative approaches applied to a specific question. The adequacy of approaches depends on the specific benefit-harm question and on the amount and quality of data that determine how justifiable certain assumptions are. In some situations, there may be a single approach that appears to be most appropriate. But commonly, there will be several approaches that are reasonable options given the question, the goal of the benefit-harm assessment, and the available data. In such situations, we suggest that investigators use several approaches, as commonly used in other areas [43, 44], which acknowledges that none of them is perfect and based on some assumptions. The confidence in the results of benefit-harm assessments then depends on the extent to which different approaches arrive at similar results, and how useful they are to end-users. Evidence from studies applying multiple approaches to the same benefit-harm question, together with recognition of their advantages and disadvantages, would make it possible to identify approaches that are consistently superior over others, and to develop recommendations for specific approaches.

Funding

This work was funded by a contract to AHRQ, Rockville, MD (Contract No. HHSA 290-2007-10061-I). The findings and conclusions in this document are those of the author(s), who are responsible for its content, and do not necessarily represent the views of AHRQ. No statement in this report should be construed as an official position of AHRQ or of the U.S. Department of Health and Human Services.

Dr. Boyd was supported the Paul Beeson Career Development Award Program (NIA K23 AG032910, AFAR, The John A. Hartford Foundation, The Atlantic Philanthropies, The Starr Foundation and an anonymous donor).

References

  1. Aspirin for the prevention of cardiovascular disease: U.S. Preventive Services Task Force recommendation statement. Ann Intern Med. 2009, 150 (6): 396-404.

    Article  Google Scholar 

  2. Fisher B, Costantino JP, Wickerham DL, Redmond CK, Kavanah M, Cronin WM, Vogel V, Robidoux A, Dimitrov N, Atkins J, et al: Tamoxifen for prevention of breast cancer: report of the National Surgical Adjuvant Breast and Bowel Project P-1 Study. J Natl Cancer Inst. 1998, 90 (18): 1371-1388. 10.1093/jnci/90.18.1371.

    Article  CAS  PubMed  Google Scholar 

  3. Gail MH, Costantino JP, Bryant J, Croyle R, Freedman L, Helzlsouer K, Vogel V: Weighing the risks and benefits of tamoxifen treatment for preventing breast cancer. J Natl Cancer Inst. 1999, 91 (21): 1829-1846. 10.1093/jnci/91.21.1829.

    Article  CAS  PubMed  Google Scholar 

  4. Dolan JG: Multi-criteria clinical decision support: A primer on the use of multiple criteria decision making methods to promote evidence-based, patient-centered healthcare. Patient. 2010, 3 (4): 229-248. 10.2165/11539470-000000000-00000.

    Article  PubMed  PubMed Central  Google Scholar 

  5. Guo JJ, Pandey S, Doyle J, Bian B, Lis Y, Raisch DW: A review of quantitative risk-benefit methodologies for assessing drug safety and efficacy-report of the ISPOR risk-benefit management working group. Value Health. 2010, 13 (5): 657-666. 10.1111/j.1524-4733.2010.00725.x.

    Article  PubMed  Google Scholar 

  6. Holden WL, Juhaeri J, Dai W: Benefit-risk analysis: a proposal using quantitative methods. Pharmacoepidemiol Drug Saf. 2003, 12 (7): 611-616. 10.1002/pds.887.

    Article  PubMed  Google Scholar 

  7. Lynd LD, Najafzadeh M, Colley L, Byrne MF, Willan AR, Sculpher MJ, Johnson FR, Hauber AB: Using the incremental net benefit framework for quantitative benefit-risk analysis in regulatory decision-making–a case study of alosetron in irritable bowel syndrome. Value Health. 2010, 13 (4): 411-417. 10.1111/j.1524-4733.2009.00595.x.

    Article  PubMed  Google Scholar 

  8. Walter SD, Sinclair JC: Uncertainty in the minimum event risk to justify treatment was evaluated. J Clin Epidemiol. 2009, 62 (8): 816-824. 10.1016/j.jclinepi.2008.09.017.

    Article  PubMed  Google Scholar 

  9. O'Neill RT: A Perspective on Characterizing Benefits and Risks Derived From Clinical Trials: Can We Do More?. Drug Inf J. 2008, 42 (3): 235-245. 10.1177/009286150804200305.

    Google Scholar 

  10. European Medicine Agency: Reflection paper on benefit-risk assessment methods in the context of the evaluation of marketing authorisation applications of medicinal products for human use. 2008, Doc. Ref. EMEA/CHMP/15404/2007 http://www.ema.europa.eu/docs/en_GB/document_library/Regulatory_and_procedural_guideline/2010/01/WC500069634.pdf, last accessed February 16 2012

    Google Scholar 

  11. European Medicine Agency: Report of the Committee for Medicinal Products Working Group on Benefit-Risk Assessment Models and Methods. 2007, Doc. Ref. EMEA/CHMP/15404/2007 http://www.ema.europa.eu/docs/en_GB/document_library/Regulatory_and_procedural_guideline/2010/01/WC500069668.pdf, last accessed February 16 2012

    Google Scholar 

  12. Laupacis A, Sackett DL, Roberts RS: An assessment of clinically useful measures of the consequences of treatment. N Engl J Med. 1988, 318 (26): 1728-1733. 10.1056/NEJM198806303182605.

    Article  CAS  PubMed  Google Scholar 

  13. Cook RJ, Sackett DL: The number needed to treat: a clinically useful measure of treatment effect. BMJ. 1995, 310 (6977): 452-454. 10.1136/bmj.310.6977.452.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  14. Gelber RD, Goldhirsch A, Cole BF: Evaluation of effectiveness: Q-TWiST. The International Breast Cancer Study Group. Cancer Treat Rev. 1993, 19 (Suppl A): 73-84.

    Article  PubMed  Google Scholar 

  15. Gelber RD, Goldhirsch A, Cole BF, Wieand HS, Schroeder G, Krook JE: A quality-adjusted time without symptoms or toxicity (Q-TWiST) analysis of adjuvant radiation therapy and chemotherapy for resectable rectal cancer. J Natl Cancer Inst. 1996, 88 (15): 1039-1045. 10.1093/jnci/88.15.1039.

    Article  CAS  PubMed  Google Scholar 

  16. Mussen F, Salek S, Walker S: A quantitative approach to benefit-risk assessment of medicines - part 1: the development of a new model using multi-criteria decision analysis. Pharmacoepidemiol Drug Saf. 2007, 16 (Suppl 1): S2-S15.

    Article  PubMed  Google Scholar 

  17. Mussen F, Salek S, Walker S, Phillips L: A quantitative approach to benefit-risk assessment of medicines - part 2: the practical application of a new model. Pharmacoepidemiol Drug Saf. 2007, 16 (Suppl 1): S16-S41.

    Article  PubMed  Google Scholar 

  18. Singer DE, Albers GW, Dalen JE, Fang MC, Go AS, Halperin JL, Lip GY, Manning WJ: Antithrombotic therapy in atrial fibrillation: American College of Chest Physicians Evidence-Based Clinical Practice Guidelines (8th Edition). Chest. 2008, 133 (6 Suppl): 546S-592S.

    Article  CAS  PubMed  Google Scholar 

  19. Guyatt GH, Sinclair J, Cook DJ, Glasziou P: Users' guides to the medical literature: XVI. How to use a treatment recommendation. Evidence-Based Medicine Working Group and the Cochrane Applicability Methods Working Group. JAMA. 1999, 281 (19): 1836-1843. 10.1001/jama.281.19.1836.

    Article  CAS  PubMed  Google Scholar 

  20. March JS, Klee BJ, Kremer CM: Treatment benefit and the risk of suicidality in multicenter, randomized, controlled trials of sertraline in children and adolescents. J Child Adolesc Psychopharmacol. 2006, 16 (1–2): 91-102.

    Article  PubMed  Google Scholar 

  21. Singh S, Dolan JG, Centor RM: Optimal management of adults with pharyngitis–a multi-criteria decision analysis. BMC Med Inform Decis Mak. 2006, 6: 14-10.1186/1472-6947-6-14.

    Article  PubMed  PubMed Central  Google Scholar 

  22. Elwyn G, O'Connor A, Stacey D, Volk R, Edwards A, Coulter A, Thomson R, Barratt A, Barry M, Bernstein S, et al: Developing a quality criteria framework for patient decision aids: online international Delphi consensus process. BMJ. 2006, 333 (7565): 417-10.1136/bmj.38926.629329.AE.

    Article  PubMed  PubMed Central  Google Scholar 

  23. Boers M, Brooks P, Fries JF, Simon LS, Strand V, Tugwell P: A first step to assess harm and benefit in clinical trials in one scale. J Clin Epidemiol. 2010, 63 (6): 627-632. 10.1016/j.jclinepi.2009.07.002.

    Article  PubMed  Google Scholar 

  24. Garrison LP, Towse A, Bresnahan BW: Assessing a structured, quantitative health outcomes approach to drug risk-benefit analysis. Health Aff (Millwood). 2007, 26 (3): 684-695. 10.1377/hlthaff.26.3.684.

    Article  Google Scholar 

  25. Ponce RA, Bartell SM, Wong EY, LaFlamme D, Carrington C, Lee RC, Patrick DL, Faustman EM, Bolger M: Use of quality-adjusted life year weights with dose–response models for public health decisions: a case study of the risks and benefits of fish consumption. Risk Anal. 2000, 20 (4): 529-542. 10.1111/0272-4332.204050.

    Article  CAS  PubMed  Google Scholar 

  26. Dolan JG: Shared decision-making–transferring research into practice: the Analytic Hierarchy Process (AHP). Patient Educ Couns. 2008, 73 (3): 418-425. 10.1016/j.pec.2008.07.032.

    Article  PubMed  PubMed Central  Google Scholar 

  27. Holden WL, Juhaeri J, Dai W: Benefit-risk analysis: examples using quantitative methods. Pharmacoepidemiol Drug Saf. 2003, 12 (8): 693-697. 10.1002/pds.794.

    Article  PubMed  Google Scholar 

  28. Sutton AJ, Cooper NJ, Abrams KR, Lambert PC, Jones DR: A Bayesian approach to evaluating net clinical benefit allowed for parameter uncertainty. J Clin Epidemiol. 2005, 58 (1): 26-40. 10.1016/j.jclinepi.2004.03.015.

    Article  PubMed  Google Scholar 

  29. Lynd LD, O'Brien BJ: Advances in risk-benefit evaluation using probabilistic simulation methods: an application to the prophylaxis of deep vein thrombosis. J Clin Epidemiol. 2004, 57 (8): 795-803. 10.1016/j.jclinepi.2003.12.012.

    Article  PubMed  Google Scholar 

  30. Sherrill B, Amonkar MM, Stein S, Walker M, Geyer C, Cameron D: Q-TWiST analysis of lapatinib combined with capecitabine for the treatment of metastatic breast cancer. Br J Cancer. 2008, 99 (5): 711-715. 10.1038/sj.bjc.6604501.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  31. Shakespeare TP, Gebski VJ, Veness MJ, Simes J: Improving interpretation of clinical studies by use of confidence levels, clinical significance curves, and risk-benefit contours. Lancet. 2001, 357 (9265): 1349-1353. 10.1016/S0140-6736(00)04522-0.

    Article  CAS  PubMed  Google Scholar 

  32. Shaffer ML, Watterberg KL: Joint distribution approaches to simultaneously quantifying benefit and risk. BMC Med Res Methodol. 2006, 6: 48-10.1186/1471-2288-6-48.

    Article  PubMed  PubMed Central  Google Scholar 

  33. Hauber AB, Mohamed AF, Johnson FR, Falvey H: Treatment preferences and medication adherence of people with Type 2 diabetes using oral glucose-lowering agents. Diabet Med. 2009, 26 (4): 416-424. 10.1111/j.1464-5491.2009.02696.x.

    Article  CAS  PubMed  Google Scholar 

  34. Johnson FR, Ozdemir S, Mansfield C, Hass S, Miller DW, Siegel CA, Sands BE: Crohn's disease patients' risk-benefit preferences: serious adverse event risks versus treatment efficacy. Gastroenterology. 2007, 133 (3): 769-779. 10.1053/j.gastro.2007.04.075.

    Article  PubMed  Google Scholar 

  35. Johnson FR, Ozdemir S, Mansfield C, Hass S, Siegel CA, Sands BE: Are adult patients more tolerant of treatment risks than parents of juvenile patients?. Risk Anal. 2009, 29 (1): 121-136. 10.1111/j.1539-6924.2008.01135.x.

    Article  PubMed  Google Scholar 

  36. Chuang-Stein C: A new proposal for benefit-less-risk analysis in clinical trials. Control Clin Trials. 1994, 15 (1): 30-43. 10.1016/0197-2456(94)90026-4.

    Article  CAS  PubMed  Google Scholar 

  37. Osiri M, Suarez-Almazor ME, Wells GA, Robinson V, Tugwell P: Number needed to treat (NNT): implication in rheumatology clinical practice. Ann Rheum Dis. 2003, 62 (4): 316-321. 10.1136/ard.62.4.316.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  38. Johnson FR, Van Houtven G, Ozdemir S, Hass S, White J, Francis G, Miller DW, Phillips JT: Multiple sclerosis patients' benefit-risk preferences: serious adverse event risks versus treatment efficacy. J Neurol. 2009, 256 (4): 554-562. 10.1007/s00415-009-0084-2.

    Article  PubMed  Google Scholar 

  39. Ryan M, McIntosh E, Shackley P: Methodological issues in the application of conjoint analysis in health care. Health Econ. 1998, 7 (4): 373-378. 10.1002/(SICI)1099-1050(199806)7:4<373::AID-HEC348>3.0.CO;2-J.

    Article  CAS  PubMed  Google Scholar 

  40. Edwards JE, McQuay HJ, Moore RA, Collins SL: Reporting of adverse effects in clinical trials should be improved: lessons from acute postoperative pain. J Pain Symptom Manage. 1999, 18 (6): 427-437. 10.1016/S0885-3924(99)00093-7.

    Article  CAS  PubMed  Google Scholar 

  41. Lynd L, Coombes M, Rishidi A, Sculpher M, Willan A: First steps in developing a risk-benefit analysis framework: a systematic review and critical evaluation of risk-benefit analysis methods. 2004, York (UK): University of York, http://www.york.ac.uk/media/che/documents/papers/presentations/04.pdf,

    Google Scholar 

  42. Stang PE, Pham SV, Kinchen K, Raff SB, Mussen F, Gondek K: The identification of benefit in medical intervention: an overview and suggestions for process. Am J Ther. 2008, 15 (5): 495-503. 10.1097/MJT.0b013e31816b8fff.

    Article  PubMed  Google Scholar 

  43. Puhan MA, Chandra D, Mosenifar Z, Ries A, Make B, Hansel NN, Wise RA, Sciurba F: The minimal important difference of exercise tests in severe COPD. Eur Respir J. 2011, 37 (4): 784-790. 10.1183/09031936.00063810.

    Article  CAS  PubMed  Google Scholar 

  44. Puhan MA, Frey M, Buchi S, Schunemann HJ: The minimal important difference of the hospital anxiety and depression scale in patients with chronic obstructive pulmonary disease. Health Qual Life Outcomes. 2008, 6: 46-10.1186/1477-7525-6-46.

    Article  PubMed  PubMed Central  Google Scholar 

Pre-publication history

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Milo A Puhan.

Additional information

Competing interests

The authors declare that they have no competing interests.

Authors’ contribution

All authors read and approved the final manuscript.

Authors’ original submitted files for images

Below are the links to the authors’ original submitted files for images.

Authors’ original file for figure 1

Authors’ original file for figure 2

Rights and permissions

This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Reprints and permissions

About this article

Cite this article

Puhan, M.A., Singh, S., Weiss, C.O. et al. A framework for organizing and selecting quantitative approaches for benefit-harm assessment. BMC Med Res Methodol 12, 173 (2012). https://doi.org/10.1186/1471-2288-12-173

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/1471-2288-12-173

Keywords