Center for Evidence-based Medicine and Health Outcomes Research, University of South Florida, Tampa, FL, USA

Department of Mathematics, Indiana University Northwest, Gary, IN, USA

Department of Epidemiology and Biostatistics, Memorial Sloan-Kettering Cancer Center, NY, NY, USA

H. Lee Moffitt Cancer Center& Research Institute, Tampa, FL, USA

Abstract

Background

Decision curve analysis (DCA) has been proposed as an alternative method for evaluation of diagnostic tests, prediction models, and molecular markers. However, DCA is based on expected utility theory, which has been routinely violated by decision makers. Decision-making is governed by intuition (system 1), and analytical, deliberative process (system 2), thus, rational decision-making should reflect both formal principles of rationality and intuition about good decisions. We use the cognitive emotion of regret to serve as a link between systems 1 and 2 and to reformulate DCA.

Methods

First, we analysed a classic decision tree describing three decision alternatives: treat, do not treat, and treat or no treat based on a predictive model. We then computed the expected regret for each of these alternatives as the difference between the utility of the action taken and the utility of the action that, in retrospect, should have been taken. For any pair of strategies, we measure the difference in net expected regret. Finally, we employ the concept of acceptable regret to identify the circumstances under which a potentially wrong strategy is tolerable to a decision-maker.

Results

We developed a novel dual visual analog scale to describe the relationship between regret associated with "omissions" (e.g. failure to treat) vs. "commissions" (e.g. treating unnecessary) and decision maker's preferences as expressed in terms of threshold probability. We then proved that the Net Expected Regret Difference, first presented in this paper, is equivalent to net benefits as described in the original DCA. Based on the concept of acceptable regret we identified the circumstances under which a decision maker tolerates a potentially wrong decision and expressed it in terms of probability of disease.

Conclusions

We present a novel method for eliciting decision maker's preferences and an alternative derivation of DCA based on regret theory. Our approach may be intuitively more appealing to a decision-maker, particularly in those clinical situations when the best management option is the one associated with the least amount of regret (e.g. diagnosis and treatment of advanced cancer, etc).

Background

Decision making is often governed by uncertainty that inevitably affects the overall decision process. In their efforts to model uncertainty, decision theorists have proposed many methodologies with the majority of them having been based on statistics and probability

In clinical medical research, much effort has been invested in developing decision support systems for diagnosis and treatment of various clinical conditions such as management of infectious diseases in an intensive care unit, chronic prostatitis, or liver surgery

The goal of this paper is to develop a novel decision-making approach that incorporates the decision maker's attitudes towards multiple treatment strategies. Our goal is addressed through the following three specific aims. First, we deviate from the traditional expected utility theory in an attempt to satisfy both formal criteria of rationality and human intuition about good decisions

To implement our approach, we first compute the threshold probability at which the decision maker is indifferent between alternative actions, based on the level of regret one might feel when he/she makes a wrong decision. We then employ the regret based DCA to identify the optimal strategy for a particular decision maker. The optimal strategy is the one that brings the least regret in the case that it is, in retrospect, wrong. We also show how to employ a prediction model to estimate the probability of disease for a patient and contrast it with the decision maker's threshold probability. Finally, we incorporate the concept of acceptable regret in the decision process to identify the conditions under which the decision maker tolerates a potentially wrong decision.

Methods

Decision analysis based on regret theory

Figure

Decision tree for administration of treatment

**Decision tree for administration of treatment**. In this figure, _{i}, i ∈ [1,4], are the utilities corresponding to each outcome. Note that we use the term "treatment" in the generic sense of health care intervention, which may indicate therapy, procedure, or a diagnostic test.

In Figure _{
i
}, _{1 }is the utility of administering treatment to a patient who has the disease (e.g. treat when necessary), and _{2 }is the utility of administering treatment to a patient who does not have the disease (e.g. administering unnecessary treatment). Note that we use the term "treatment" in the generic sense of health care intervention, which may indicate therapy, procedure, or a diagnostic test.

The probabilistic nature of prognostication models complicates significantly the decision process. For example, if a prediction model estimates the probability of a patient having a disease equal to 40%, it is unclear whether this patient should receive treatment or not. A solution from the point of view of the classical decision theory is to employ the concept of threshold probability _{
t
}, which is defined as the probability at which the decision maker is indifferent between two strategies (e.g. administer treatment or not)_{
t
}and should not be treated otherwise.

However, since in most cases decisions are made under uncertainty and can never be 100% accurate

Formally, regret can be expressed as the difference between the utility of the outcome of the action taken and the utility of the outcome of the action that, in retrospect, should have been taken

We first employ regret theory to estimate the threshold probability, _{
t
}, at which the physician is indifferent between alternative management strategies (e.g. administer treatment or not). In order to accomplish this, we describe regret in terms of the errors of (1) not treating the patient who has the disease, and (2) treating the patient who does not have the disease.

Figure _{
t
}),

Regret model of the decision tree for administration of treatment

**Regret model of the decision tree for administration of treatment**. In this figure, _{t}: threshold probability for treatment;

Similarly, the regret associated with treating the patient who should not have received treatment (the probability of disease is _{
t
}), _{
t
}is equal to

Equation 1 effectively captures the preferences of the decision maker towards administering or not administering treatment. At the individual level, equation 1 shows how the threshold probability relates to the way the decision maker weighs false negative (i.e. failing to provide necessary treatment) vs. false positive (i.e. administering unnecessary treatment) results

Note that the fraction _{4 }- _{2 }= 0, which means that in this situation there is no regret associated with administering unnecessary treatment. Under these circumstances, _{
t
}= 100%, indicating that treatment is justified only in case of absolute certainty of disease (p = 100%), a realistically unachievable goal

Elicitation of threshold probability

There are numerous techniques for eliciting the decision maker's preferences regarding treatment administration

There are few commonly used methods to assess the value of this indifference for a decision maker such as the standard gamble, and the time trade-off

The proposed method retains the simplicity of VAS but it takes into account the consequences of possible mistakes in decision-making by utilizing two visual analog scales. The first scale aims to assess the regret associated with potential error of failing to administer beneficial treatment ("regret of omission"). The second scale measures the regret of administration of unnecessary treatment ("regret of commission"). Using these two scales we can capture trade-offs and compute the threshold probability at which a decision maker is indifferent between two alternative management strategies.

We employed the two visual analog scales with typical 100 points

For example, to elicit the physician's threshold probability, we may ask the following questions:

For example, suppose that the physician answers 60 and 30 to the questions 1 and 2, respectively. This means that the physician considers 60/30 = 2 times worse to fail to administer treatment that should have been given than to continue unnecessary treatment. Then, the threshold probability for this physician is:

Thus, the physician would be unsure as to whether to treat or not the patient if the patient's probability of disease as computed by the prediction model was 33%. Thus, the recommended action, which is based on elicitation of the decision-maker preferences, is directly derived from the underlying theoretical model.

Regret based decision curve analysis (DCA)

Decision-makers may be presented with many alternative strategies that can be difficult to model. A simple, yet powerful approach that is based on experience of a typical practicing physician is to compare the strategy based on modeling with those scenarios when all or no patient is treated. That is, the clinical alternatives to the prediction model strategy is to assume that all patients have the disease and thus treat them all, or to assume that no patient has the disease and thus treat none_{
t
}("model").

The optimal decision depends on the preferences of the decision maker as captured by the threshold probability. We use Decision Curve Analysis (DCA)

One view about decision curves is that they should not be used in clinical practice: the researcher determines whether the decision curve justifies the use of the model in practice and then makes a simple recommendation yes or no as to whether clinicians should base their decisions on the model

Generalized decision tree for administration of treatment

**Generalized decision tree for administration of treatment**. In this figure,

Figure

Here, _{
t
}|

_{
t
}|

Similarly,

_{t}|

_{t}|

After re-scaling the utilities by dividing each utility with the expression _{1 }- _{3}, and replacing

For the strategies of administering treatment and not administering treatment, the expected regret is derived as:

Subtracting each of these expected regrets from the expected regret of the "Treat none" (baseline) strategy we obtain the "**Net Expected Regret Difference (NERD)**":

Note that these are **exactly **the same formulas as those derived by Vickers and Elkin

In addition to equations 6-8, we are interested in the NERD between the strategies "Treat all" and "Model":

The NERD equations associated with each strategy, 6-8, can be further reformulated as follows

Similarly, equation 9 can be re-written as:

Equations 10 and 11 above are useful when calculating _{
t
}. The probabilities _{
t
}∩ _{
t
}∩ _{
t
}∩ _{
t
}∩

• _{
t
}∩ _{
t
}(with #TP = number of patients with true positive results,

• _{
t
}∩ _{
t
}(with #FP = number of patients with false positive results,

• _{
t
}∩ _{
t
}(with #TN = number of patients with true negative results,

• _{
t
}∩ _{
t
}(with #FN=number of patients with false negative results,

When computing

NERDs of each of the strategies described are plotted against different values of threshold probability. The NERD values provide information relative to **decrease in regret **when two strategies are compared against each other for a given threshold probability. If NERD = 0, this means that there is no difference in the regret between two strategies:

If NERD > 0, this means that the second strategy will inflict less regret than the first strategy, and hence it is preferable:

Similarly, if NERD < 0, the first strategy represents the optimal decision among the two strategies:

The algorithm for the Regret DCA is implemented as follows:

1. Select a value for threshold probability.

2. Assuming that patients should be treated if _{
t
}and should not be treated otherwise, compute #TP and #FP for the prediction model.

3. Calculate the

4. Calculate

5. Compute the

6. Repeat steps 1 - 6 for a range of threshold probabilities.

7. Graph each NERD calculated in steps 3-5 against each threshold probability.

Based on the Regret DCA methodology, the optimal decision at each threshold probability is derived by comparing each pair of strategies through their corresponding NERDs according to the transitivity principle (i.e., if A > B, B > C then A > C). Thus, if

Acceptable Regret

No decision model can guarantee that the recommended strategy will be the correct one. Therefore, we can always make a mistake and recommend treatment we should not have, or fail to recommend treatment we should have administered _{0}, is defined as the portion of utility a decision maker is willing to lose/sacrifice when he/she adheres to a decision that may prove wrong

We assume that there is a linear relationship between the value of acceptable regret and the benefits of receiving treatment as well as the harms of receiving unnecessary treatment. This is a reasonable assumption because acceptable regret is expected to operate within a narrow range, at the lower or the upper end, of the probability scale. We define acceptable regret in terms of benefits of treatment,_{b}
_{b}
_{1 }- _{3})the decision maker is willing to forgo if his/her decision NOT to treat was wrong:

Alternatively, we define acceptable regret in terms of harms of unnecessary treatment, _{
h
}, as_{
h
}) of harms (U_{4 }- U_{2}) the decision maker is willing to incur if his/her decision of treating was wrong:

We use the concept of acceptable regret to further refine the conditions under which the decision maker is indifferent between two strategies. Recall that these conditions have been initially captured in terms of threshold probability, which does not incorporate the sense of tolerable losses. Thus, we proceed with the following definition: Two strategies are considered _{0}. In other words, there is no difference between choosing the strategy "treat all" or "treat none" in terms of regret if:

Similarly, the strategies "model" and "treat none" are equivalent in regret if:

and the strategies "model" and "treat all":

The acceptable regret,_{0}, can be computed using any of the two definitions described in equations 15 and 16.

We can also use equations 15 and 16 to identify the prognostic probabilities at which the decision maker would not regret the decision to which he/she is committed even if that decision may prove wrong. For instance, we are typically interested in the prognostic probability above which a physician would commit to the decision to treat a patient, and the probability below which he/she would not to treat a patient without feeling undue consequences of these decisions_{
h
}, and _{
b
}. Solving the inequalities using equations 4, 5, 15, and 16 and after scaling _{
0
}by (_{1 }- _{3}), we obtain

Where _{treat all }is

represents the prognostic probability below which the physician would comfortably withhold treatment that may prove beneficial, in retrospect.

Note that equations 20 and 21 express acceptable regret in terms of probabilities while equations 17-19 define it in terms of NERD. Hence, the outputs of these equations are not the same; rather, they complement each other.

Elicitation of acceptable regret

In most cases the decision maker does not have a complete understanding of benefits lost or harms inflicted and cannot assign a precise number to them. For this reason, we do not suggest inquiring directly about the value of

**100 patients with the same probability of disease as the patient you are currently treating**. You need to decide whether each of these patients should receive treatment or not. Since no prediction model is 100% accurate, it is expected that you will make some mistakes in your treatment recommendations (e.g. you may recommend treatment to a patient who does not need it, or fail to recommend treatment to a patient who needs it)

**unnecessary **treatment i.e. we want to learn what the magnitude of the **unavoidable error **you can live with is by inflicting potentially harmful treatment on a patient. Note that if you say that your acceptable regret is zero, this means that you can only make decision if you **absolutely certain **that your recommendation is correct

_{
h
}).

**failing **to provide necessary treatment i.e. we want to learn what the magnitude of **unavoidable error **you can live with is by forgoing potentially beneficial treatment. Note that if you say that your acceptable regret is zero, this means that you can only make decision if you **absolutely certain **that your recommendation is correct

_{
b
}).

It is unnecessary to ask the decision maker to answer both questions. We suggest asking only the question related to the recommendation the physician is about to make e.g. if the recommendation is about administering treatment, then the decision maker should be asked the second question, while if it is about not giving treatment, then he/she can ask the first question.

The value of acceptable regret is plotted in the regret DCA graph to visually facilitate the decision making process. At a specific threshold probability all strategies for which |_{0 }are considered equivalent in regret, according to the definition in the previous section.

Example

We will employ a prostate cancer biopsy example to demonstrate the applicability of our approach. Prostate cancer biopsy is an invasive and uncomfortable procedure, which can be painful and is associated with a risk of infection. However, it is often necessary for diagnosis of prostate cancer, one of the leading causes of cancer death in men.

Men are typically biopsied for prostate cancer if they have an elevated level of prostate-specific antigen (PSA). However, most men with a high PSA do not have prostate cancer. This has led to the idea that statistical models based on multiple predictors (PSA, age, family history, other markers) might be used to predict biopsy outcomes and hence aid biopsy decisions for individual patients. A physician seeing a patient with an elevated PSA has three possible options: go for biopsy, refuse biopsy or look up his probability in a statistical model and then make a decision.

We utilize an unpublished statistical model that computes probability of cancer based on the dataset described in

1.

2.

3.

Consequently, "model" corresponds to the optimal strategy.

Regret DCA regarding biopsy to detect prostate cancer

**Regret DCA regarding biopsy to detect prostate cancer**. Thin line: biopsy all patients; solid line: biopsy no patients; dashed line: prediction model. The optimal strategy is derived by the comparison of each pair of strategies from all NERDs as per equations 12-14. The statistical model is the optimal strategy for threshold probabilities between 8% and 42%. For threshold probabilities between 43% and 95%, the optimal strategy is to biopsy no patients, while for 0% to 8% both "model" and "biopsy all" strategies are optimal. The lines of acceptable regret denote the regret area in which different strategies are equivalent.For example, at threshold probability equal to 20%, the optimal strategy is acting based on the prognostic model. However, NERD(biopsy none, model) is below the acceptable regret line which indicates that the strategies "biopsy none" and "model" are equivalent in regret. Therefore the optimal strategy is to biopsy no patients as the use of model is deemed to be superfluous. Similarly,at threshold probability equal to 15%, the optimal strategy is to act based on the model and the strategies "biopsy all" and "biopsy none" are equivalent in regret. Finally, at threshold probability equal to 9%, the optimal strategies are both "model" and "biopsy all". However, since NERD(biopsy all, model) is below the acceptable regret, the strategies "biopsy all" and "model" are equivalent in regret. Therefore the optimal strategy is to biopsy all patients.

Repeating the same procedure for all threshold probabilities, we can see that deciding based on the statistical model is the optimal strategy (i.e. results in the minimum expected regret) for threshold probabilities between 8% and 43%. For threshold probabilities between 42% and 95%, the optimal strategy is to biopsy no patients, while for 0% to 8% both model and biopsy all strategies are optimal.

To interpret these results, we have to consider how a typical physician values the harms of a false negative (missing a cancer) and a false positive (an unnecessary biopsy) result. If regret associated with unnecessary biopsy is felt to be worse than missing cancer, then according to equation 1, the threshold probability is greater than 50%. However, it is unlikely that a physician would consider an unnecessary biopsy to be worse than missing a cancer, so the threshold probability for biopsy must be less than 50%. Thus, a reasonable range of threshold probabilities might indeed be between 8% - 43% as suggested by our model. As the model is superior across this entire range, we can conclude that, **
irrespective of the physician's exact preferences
**, making a biopsy decision based on the statistical model will lead to lower expected regret than an alternative such as biopsying all or no men. Based on discussions with clinicians, we believe that a reasonable range of threshold probability is 10% - 40%. As the regret associated with the model strategy is lowest across this entire range, we can recommend use of the model. Nonetheless, we do not have a complete sample of all physician preferences and it is possible that a physician may have a probability outside of this range.

To illustrate the applicability of the acceptable regret model, assume that the value of acceptable regret for forgoing the benefits of biopsy (equation 15) is equal to ± 0.01. Consider the case that the decision maker's threshold probability is equal to 20%. According to Figure

which means that the strategies "biopsy none" (biopsy no patients) and "model" are equivalent in regret. Therefore, the prediction model does not offer any better information and thus, it can be disregarded.

Case Study

This section describes the overall decision process regarding prostate cancer biopsy. The process begins with elicitation of the threshold probability from the treating physician and continues with evaluation of the available strategies based on regret DCA (Figure **bold **and

- underlined

The overall decision process is described as follows:

1. Interview with the physician to elicit his/her threshold probability.

a. **On the scale 0 to 100, where 0 indicates no regret and 100 indicates the maximum regret you could feel, how would you rate your level of regret if you failed to provide necessary treatment?**

Physician #1 answer:

- 50

- 70

b. **On the scale 0 to 100, where 0 indicates no regret and 100 indicates the maximum regret you could feel, how would you rate your level of regret if you administered unnecessary treatment?**

Physician #1:

- 10

- 60

2. Using the graph in Figure

Physician #1:

1. NERD(biopy all, model) > 0,

2. NERD(biopsy none, model) > 0,

3. NERD(biopsy none, biopsy all) > 0,

Physician #2:

3. Compute the cancer probability for the specific patient based on the statistical model.

4. Elicitation of the level of acceptable regret.

**Assume that you have 100 patients, all with probability of cancer equal to 20% (the same as your patient). This means that out of 100 patients, 20 patients will have cancer while 80 will not have cancer. You need to decide whether each of these patients should undergo biopsy or not. Since no prediction model is 100% accurate, it is expected that you will make some mistakes in your recommendations (e.g. you may recommend biopsy to a patient who does not need it, or fail to recommend biopsy to a patient who may need it)**.

**Out of the 20 patients who should be biopsied, for how many patients would you tolerate not recommending a necessary biopsy? **

- 1

_{b }= _{b}
_{1 }- _{3})_{* }0.5 = 0.025. _{
t
}= 16% _{
t
}= 16% _{
b
}= 0.025

**Out of the 80 patients who should not undergo biopsy, for how many patients would you tolerate recommending an unnecessary biopsy? **

- 40

_{
h
}= _{
h
}(_{4 }- _{2}) = 0.5 _{* }0.6 = 0.3.

_{
t
}= 46% _{
t
}= 46% _{
h
}= 0.3, _{
h
}, |_{h }and _{
h
}.

5. Based on equations 20 and 21, we can determine the prognostic probabilities above and under which the physician would tolerate performing an unnecessary biopsy, or not to do so when he should have done it.

_{treat none }
_{b}

_{treat all }
_{h }

Discussion

Currently, there is no agreed upon method for how preferences regarding multiple objectives that typically go in opposite directions (i.e. most medical interventions are associated both with benefits and harms) should be elicited. We have presented and demonstrated an approach to decision making based on regret theory and decision curve analysis. The approach presented in this paper relies on the concept of the threshold probability at which a decision maker is indifferent between strategies, to suggest the optimal decision

We believe that the model described here has a direct practical application in overcoming many difficulties related to linking evidence with patient's preferences to arrive at the optimal decision- the issues that plagued the field of decision-making. The problem of eliciting preferences and integrating them in a coherent decision is not a simple one. We argue that the approach we are advocating here represents a contribution to the field of decision making, be should not be seen as the panacea to medical decision making. However, we anticipate our methodology to be suitable for medical decision primarily associated with trade-offs between quality and quantity of life.

Over that last couple of decades, many attempts have been made to develop the best method to take these considerations in real-life settings. Unfortunately, as explained, no approach has succeeded

Specifically, we argue that eliciting people's preferences using regret theory may be superior to using traditional utility theory because regret forces decision-makers to explicitly consider consequences of decisions. We have previously shown that we can always make errors in decision-making: recommend treatment that does not work, or fail to recommend treatment that does

Moreover, while descriptive, normative, and prescriptive theories

In general, since our method relies on the elicitation of threshold probability we recommend using our methodology for every patient. As every patient's values are different the threshold probability should indeed be patient-specific. For example, a physician may act "aggressively" for a young patient who is the father of two underage kids and less aggressively for an older patient. However, in the cancer biopsy example, it is expected that most of the patients should present with similar characteristics and therefore most physicians would settle in a small area of threshold probabilities. In this case repeating the elicitation process for every patient would be impractical. Nevertheless, this is an empirical question worthy of further investigation as alluded above.

Our approach may help reconcile formal principles of rationality and human intuitions about good decisions that may better reflect "rationality" in medical decision-making

Conclusions

We have presented a decision making methodology that relies on regret theory and decision curve analysis to assist physicians in choosing between appropriate health care interventions. Our methodology utilizes the cognitive emotion of regret to determine the decision maker's preferences towards available strategies and DCA to suggest the optimal decision for the specific decision maker. We believe that our approach is suitable for those clinical situations when the best management option is the one associated with the least amount of regret (e.g. diagnosis and treatment of advanced cancer, etc).

As with any other novel theoretical work, our approach has its limitations. First, it has not been empirically tested in a clinical setting. However, we are in the process of developing the appropriate decision support tools to bring our model into clinical practice and evaluate its usefulness with actual physicians and patients. Second, the methodology presented is appropriate for single point decision making. Further investigation is required to determine the application of regret theory to decisions that re-occur over time. Finally, we assume that there is only one decision maker involved in the decision process. Nevertheless, our plan for future work includes extending our methodology to shared decision-making that will include both physician and patient in the decision process and investigate whether in practice there is a difference between preferences and choices made by physicians and their patients.

We summarize the contribution presented in this paper as follows:

1. We propose a novel method for eliciting decision makers' preferences towards treatment administration. Contrary to traditional methodologies on eliciting preferences, our method considers the consequences of potential mistakes in decisions. We propose a dual visual analog scale to capture errors of omission and errors of commission and, therefore, evaluate the trade-offs associated with each of the available strategies.

2. We have reformulated DCA from the regret theory point of view. Our approach is intuitively more appealing to a decision maker and should facilitate decision making particularly in those clinical situations when the best management option is the one associated with the least amount of regret.

3. Finally, we utilize the concept of acceptable regret to identify the circumstances under which a decision maker tolerates a wrong decision.

We envision facilitation of the decision process in clinical settings through a computerized decision support system available at the point of care. In fact, we are in the process of developing such a system and hope to report about it soon.

Abbreviations

DCA: Decision Curve Analysis; NERD: Net Expected Regret Difference; VAS: Visual Analog Scale, _{t}
_{i}
_{1 }- _{3}: Consequences of not administering treatment where indicated; _{4 }- _{2: }Consequences of unnecessarily administering treatment; _{0}: Acceptable regret; _{b}
_{h}
_{b}
_{h}
_{treat all }
_{treat none}

Competing interests

The authors declare that they have no competing interests.

Authors' contributions

AT prepared the first draft, formalized the proposed methodology, and applied it into treatment administration examples; IH developed the mathematical formulation of the model; AV is the author of DCA; BD proposed the regret theory extension to DCA. All authors contributed equally in reviewing multiple versions of the paper and provided important feedback to the final version of the paper. BD is a guarantor. All authors read and approved the final draft.

Acknowledgements

This work is supported by the Department of Army grant #W81 XWH 09-2-0175.

Pre-publication history

The pre-publication history for this paper can be accessed here: