From algorithms to action: improving patient care requires causality

Abstract

In cancer research there is much interest in building and validating outcome prediction models to support treatment decisions. However, because most outcome prediction models are developed and validated without regard to the causal aspects of treatment decision making, many published outcome prediction models may cause harm when used for decision making, despite being found accurate in validation studies. Guidelines on prediction model validation and the checklist for risk model endorsement by the American Joint Committee on Cancer do not protect against prediction models that are accurate during development and validation but harmful when used for decision making. We explain why this is the case and how to build and validate models that are useful for decision making.

Introduction

Treatment decisions in cancer care are guided by treatment effect estimates from randomized controlled trials (RCTs). RCTs estimate the average effect of one treatment versus another in a certain population. However, treatments may not be equally effective for every patient in a population. Knowing the effectiveness of treatments tailored to specific patient and tumor characteristics would enable individualized treatment decisions. Getting tailored treatment effects by averaging outcomes in different patient subgroups in RCTs requires an infeasible number of patients to have sufficient statistical power in all relevant subgroups for all possible treatments. Instead, we must rely on statistical modeling, potentially using observational data from non-randomized studies to further the individualization of treatment decisions.

The American Joint Committee on Cancer (AJCC) recommends that researchers develop outcome prediction models in an effort to individualize treatment decisions [1, 2]. Outcome prediction models, sometimes called risk models or prognosis models, use patient and tumor characteristics to predict a patient outcome such as cancer recurrence or overall survival. The assumption is that the predictions are useful for treatment decisions using rules such as “prescribe chemotherapy only if the outcome prediction model predicts the patient has a high risk of recurrence”. Many outcome prediction models are published every year. Recognizing the importance of reliable predictions, the AJCC published a checklist for outcome prediction models to ensure dependable prediction accuracy in the patient population for which the outcome prediction model was designed [1]. However, accurate outcome predictions do not imply that these predictions yield good treatment decisions. In this comment, we show that outcome prediction models rely on a fixed treatment policy which implies that outcome prediction models that were found to accurately predict outcomes in validation studies can still lead to patient harm when used to inform treatment decisions. We then give guidance on how to evaluate whether a model has value for decision-making and how to develop models that are useful for individualized treatment decisions.

Accurate predictions have unknown value for decision-making

Individualizing treatment decisions means changing the treatment policy. For example, if for a specific cancer type and stage the current treatment policy is to give the same treatment to all patients, then individualizing treatment decisions means recommending treatments tailored to a patient’s characteristics. The value of an outcome prediction model is not in how well it predicts under a certain historic treatment policy, but rather what is the effect of deploying this model on treatment decisions and patient outcomes?

Consider an outcome prediction model that uses pre-treatment tumor characteristics to predict an outcome but ignores whatever treatment the patients may have had, i.e. treatment-naive models (e.g. [3,4,5]). Interestingly, the decision to ignore treatments in the outcome prediction model is in line with the AJCC checklist for outcome prediction models (item 12 [1]). However, these outcome prediction models can cause more harm than good when used to support treatment decisions, even when they are accurate under the historic treatment policy. Consider for example an outcome prediction model that predicts overall survival for stage IV lung cancer patients based on the pre-treatment growth-rate of the tumor. An accurate model would predict shorter survival for patients with faster growing tumors. Applying this outcome prediction model, a clinician could decide to refrain from palliative radiotherapy in patients with faster growing tumors under the assumption that their life expectancy is too short to benefit from radiotherapy. This decision based on the outcome prediction model would be unjustified and harmful, as faster growing tumors are more susceptible to radiotherapy [6]. See Fig. 1 for an illustration of introducing an outcome prediction model for treatment decisions.

Fig. 1

Illustration of the use of outcome prediction models that ignore treatment allocations in the historical data (i.e. are treatment naive) for treatment decision making. These models change the treatment decisions and thus patient outcomes but whether this change improves patient outcomes is not determined by the prediction accuracy of the outcome prediction model

Prospective validation does not test value for decision-making

The gold standard for evaluating the accuracy of an outcome prediction model is prospective validation [1, 7]. In a prospective validation, patient characteristics and outcomes are recorded for a new patient cohort according to a predefined protocol. Comparing the outcome prediction model’s predictions with the observed outcomes provides an estimate of how accurate the outcome prediction model is outside the cohort in which the model was developed. The outcome prediction model from the lung cancer example above, if well-estimated, would be found accurate in a prospective validation that uses the historic treatment policy because the outcome prediction model was developed under the same historic policy. It would then fulfill all the AJCC checklist items but still lead to patient harm when used for treatment decisions because the differential effect of radiotherapy depending on tumor growth-rate is not accounted for in the outcome prediction model.

As an additional validation step, one may conduct a prospective validation study where the outcome prediction model is used for treatment decisions in new patients, thus changing the treatment policy. If such a validation were carried out for the lung cancer survival outcome prediction model, the patients with fast-growing tumors would be given radiotherapy less often due to the predictions of the outcome prediction model, leading to even worse survival for these patients than before introduction of the outcome prediction model. Introducing the outcome prediction model for decision making caused harm because under the new policy treatments are withheld from those who would have benefited most (the patients with fast-growing tumors). However, in this validation study with model deployment the prediction model is still accurate as the model already predicted that patients with fast-growing tumors have a poor prognosis.

Models should improve decisions

The crux of the issue with outcome prediction models is that they answer the question “What is the chance of the outcome given these patient and tumor characteristics, with the assumption that we will keep making the same treatment decisions as we always did?”. Similar issues exist with other kinds of outcome prediction models which make predictions using the historical treatments but without regards to the policy for how those treatments were assigned (i.e. post-decision models such as [8,9,10,11,12]). Post-decision outcome prediction models are also in line with the AJCC checklist (item 12 [1]). To improve treatment decisions however, we need models with a foreseeable positive effect on outcomes when used in decision-making.

Outcome prediction models assume treatment decisions follow the historical policy and thereby cannot inform us on the effect of a new policy derived from the outcome prediction model. This reliance on the historical treatment policy leads to a fundamental gap between a prediction model’s accuracy and its value for treatment decision-making in clinical practice (Fig. 2). Bridging the gap from prediction accuracy to value for decision making is only possible with causality. Evaluating the effect of a prediction model-based treatment policy on patient outcomes requires a causal study design or causal assumptions.

Fig. 2

Illustration of the difference between outcome prediction model accuracy and its value for treatment decision making. Validation of an outcome prediction model following the AJCC checklist leads to a reliable estimate of the outcome prediction model’s accuracy if the treatment policy does not change. However, because the outcome prediction model relies on a fixed historic treatment policy, prediction accuracy does not imply value for decision making, as visualized with the gap. This gap can only be bridged with causality

How to validate models used for treatment decisions?

The ultimate test of the effect of introducing a new treatment policy for example based on an outcome prediction model is a cluster randomized controlled trial [7, 13]. In a cluster RCT with outcome prediction models, some groups of clinicians are randomly selected to get access to the model while others are not. This allows for the estimation of the effect of introducing the model on treatment decisions and patient outcomes. For example, the cluster RCT could demonstrate that using the model leads to fewer treatment side effects and better overall survival. However, in the context of shared decision-making, patients may weigh the value of overall survival versus treatment discomfort differently [14]. These individual preferences need to be taken into account in the cluster RCT when calculating the value of introducing a model for decision-making.

As an alternative to cluster RCTs, the expected outcomes under a treatment policy (e.g. based on a prediction model) can be evaluated in data from a standard RCT. This can be done by calculating the average outcome in the subgroup of patients for whom the randomized treatment assignment was concordant with the policy [15]. Multiple policies can be compared this way, for example comparing a policy based on a new prediction model with current clinical practice. The policy with the best outcomes is preferable. However, such an analysis does not take into account that in practice the compliance with the new treatment policy might not be perfect. Notably, the validation steps recommended in the AJCC checklist [1] provide no information on what the effect is of deploying an outcome prediction model on treatment decisions and patient outcomes.

Building models to individualize treatment decisions

Cluster RCTs are costly and time consuming. With tools from causal inference we can improve the chance of success of models for decision making. One way to construct a good individualized treatment policy is with models that predict the outcome under hypothetical interventions, where the intervention is the decision to give a certain treatment. The optimal treatment policy selects the treatment that leads to the most beneficial expected outcome.

Estimating such models for prediction under intervention requires unconfoundedness, which holds when there are no unknown variables that influence both the treatment assignment and the outcome (i.e. confounders). RCTs are ideal for this as unconfoundedness holds by design because the treatment assignment is random. However, individual RCTs are generally too small to include many important patient and tumor characteristics in the modeling. Observational data from regular clinical practice on the other hand are often more readily available. If all variables that influence the treatment policy are available in a particular dataset, meaning that unconfoundedness holds, there are many approaches to prediction under intervention. These include ‘conventional’ statistical approaches such as regression, or machine learning approaches, for example using neural networks [16].

To express available background knowledge and judge whether unconfoundedness holds in observational data, researchers can use directed acyclic graphs (DAGs) [17]. DAGs depict variables (such as treatment, outcome and confounders) and causal dependencies between the variables as arrows that point from a cause variable to an effect variable, see Fig. 3 for an example. Using tools from causal inference, the DAG determines whether a prediction under intervention model can be estimated and if so what confounders need to be accounted for [17].

Fig. 3

Simplified Directed Acyclic Graph for the decision between surgery and radiotherapy for overall survival in lung cancer patients. As an example, consider a hypothetical study in early-stage lung cancer where researchers investigate whether the relative effectiveness of surgery versus radiotherapy for overall survival depends on a certain single-nucleotide polymorphism (SNP). The SNP assay was performed for the study only so this information did not affect the treatment decision. A DAG with four variables for this study is presented in this Figure. In this DAG, the variables age and SNP both have arrows to overall survival, but only age influences the treatment decision as older patients are less likely to get surgery. This DAG indicates that unconfoundedness holds when age is conditioned on in the analysis, as age is the only confounder between the treatment and the outcome [17]

In some cases not all confounders are available. In this setting with unobserved confounding standard methods based on confounder adjustment cannot be used, but sometimes prediction under intervention models may be estimated using specialized methods. Two examples are methods based on proxy-variables of unmeasured confounders [18, 19] and instrumental variable methods [20] and their machine learning variants [21, 22]. These methods rely on assumptions that may not hold perfectly in reality, so figuratively speaking they might reduce the gap between model accuracy and treatment policy value, but not close the gap entirely. DAGs encode assumptions about the data which may not hold perfectly in practice. The effects of potential violations of these assumptions may be estimated using sensitivity analyses [23].

A special case for prediction under intervention is the untreated risk, which is the hypothetical outcome under no treatment (or some baseline treatment) and would be observed in the control group of an RCT. For instance, when deciding to give adjuvant therapy after breast cancer surgery, the untreated risk of recurrence is the risk of recurrence when no adjuvant therapy would be given [24]. Knowing the untreated risk is valuable when considering giving no further treatment, and as a baseline to compare other potential treatments against. Although estimating the untreated risk requires unconfoundedness, in some cases it may be estimated quite accurately even from confounded data using specialized methods [25].

Because RCTs randomly assign patients to interventions, models for prediction under intervention can be validated in RCTs with standard prediction validation approaches [7]. For shared decision-making, prediction under intervention of different treatment options allows the patient to make their own judgment on how to weigh e.g. expected overall survival with expected treatment discomfort. Whereas individual RCTs randomize a patient to a certain treatment, cluster RCTs randomize clinicians’ access to a model for decision support. Thereby individual treatment decisions may still be confounded in cluster RCTs meaning that cluster RCTs cannot validate predictions from prediction under intervention-models directly. Both policy evaluation with cluster RCTs and prediction-under-intervention validation in standard RCTs are also possible in observational data but require unconfoundedness and thereby sensitivity analyses for potentially omitted confounders [23, 26].

Discussion

In line with American Joint Committee on Cancer recommendations [1, 2] many researchers develop outcome prediction models to individualize treatment decisions. The AJCC checklist provides important guidelines for outcome prediction model development and validation, such as clearly defining the patient population, predictor variables and prediction time-point, in addition to validation in external datasets. These items improve the dependability of outcome prediction models for predicting outcomes in the intended patient population if there are no changes in the treatment policy [1]. However, not changing the treatment policy directly contradicts the intended purpose of these models. Outcome prediction models that satisfy all the criteria in the checklist still have unknown clinical utility because high prediction accuracy in prospective validation studies does not imply value for treatment decision-making in clinical practice [27]. Because the gap between outcome prediction model accuracy and value for decision-making is due to causal issues, it is not resolved by larger datasets, more flexible prediction algorithms (e.g. machine learning) or even by prospective validation with model deployment. In contrast, we explained how models for prediction under intervention are useful for decision-making and how to validate any model used in decision-making.

The gap between outcome prediction model accuracy and value for decision-making is due to causal issues, but it is different from the standard “correlation does not imply causation”. In the standard “correlation is not causation” setting, all variables (treatment, outcome, patient/tumor characteristics) are already present in the historical data, whereas in this case, the output of the outcome prediction model cannot be a cause of the outcome. This is because the outcome prediction model is not a variable in historical data, but a shift in policy that changes the distribution of the treatment.

It was noted before that cluster RCTs are the ultimate test for the impact of a new prediction model on clinical practice due to issues related to compliance with treatment recommendations [13]. We show that because of the gap between prediction accuracy and value for treatment decision-making, many accurate outcome prediction models will fail to demonstrate value in cluster RCTs. Also, cluster RCTs measure the effect of a new policy on average outcomes but do not directly measure whether a model accurately predicts the outcome under intervening to give a certain treatment, for this individually randomized data are most valuable. For shared decision-making accurate predictions-under-intervention may be most important. Two patients with the same predicted outcomes may make a different treatment decision because each patient has their own values and preferences. In a cluster RCT, these individual values need to be accounted for when evaluating a treatment policy, for example by eliciting the values and incorporating them in the analysis when weighing for example overall survival and treatment discomfort.

Previous work underlined the value of prediction-under-intervention models (sometimes referred to as counterfactual prediction) for supporting treatment decisions [28, 29]. Our comment highlights the potential harm of current common practice where outcome prediction models are deployed for decision making based on prediction accuracy alone, further emphasizing the relevance of prediction-under-intervention. In addition, we note how models may be validated for decision support for example with cluster RCTs.

Building models for prediction under intervention is harder than developing outcome prediction models due to the extra requirement of unconfoundedness, which involves formalizing assumptions about confounders for example with DAGs, gathering data on all confounders, often more complex statistical estimation, and sensitivity analyses. When the cost to do a cluster RCT is low, it may suffice to build outcome prediction models in line with the AJCC checklist and test them in cluster RCTs before model deployment. As illustrated in Fig. 4, when cluster RCTs are costly, impractical or unethical, models that predict under interventions are preferable as they have foreseeable effects when used for treatment decision-making.

Fig. 4

Flowchart of what to do depending on the costliness of cluster randomized controlled trials. Costliness of cluster RCTs should be taken broadly, including time, money and ethical considerations

There is a classical distinction between treatment effect estimation and prediction that amounts to “treatment effect estimation is causal (and thus requires RCTs)” but “prediction is not causal”. When it comes to individualizing treatment decisions with prediction models, this distinction is unhelpful and confusing as the goal is to predict what would happen under different interventions. Selecting the best treatment for a patient is a causal question and requires causal answers.