Biometry Research Group, Division of Cancer Prevention, National Cancer Institute, USA

Office of Disease Prevention, National Institutes of Health, USA

Abstract

Background

Randomized trials stochastically answer the question. "What would be the effect of treatment on outcome if one turned back the clock and switched treatments in the given population?" Generalizations to other subjects are reliable only if the particular trial is performed on a random sample of the target population. By considering an unobserved binary variable, we graphically investigate how randomized trials can also stochastically answer the question, "What would be the effect of treatment on outcome in a population with a possibly different distribution of an unobserved binary baseline variable that does not interact with treatment in its effect on outcome?"

Method

For three different outcome measures, absolute difference (

Results

For

Conclusion

The BK-Plot provides a simple method to understand generalizability in randomized trials. Meta-analyses of randomized trials with a binary outcome that are based on

Background

Consider a randomized trial in which subjects are randomized to either a control or experimental intervention. The approach to statistical inference depends on the question one would like to answer.

One question is "What would be the effect of an intervention on outcome if we turned the clock backwards so that subjects randomized to the experimental treatment received the control treatment and vice versa?" Of course this question cannot be answered empirically by direct observation because one cannot go back in time. In a landmark paper on causal inference, Rubin

A broader question is "What is the effect of intervention in a different population that is not a random sample from the target population?" This question cannot be answered empirically. (In fact, if it were required for valid generalization of results, it would present a serious limitation of the scientific method in medical decision making.) In the most general situation in which the treatment effect varies by population, the question is also unanswerable stochastically. However a restricted version of this question can be answered stochastistically. Our starting point is to postulate an unobserved baseline binary random variable. Unobserved baseline variables have often been considered in discussing randomization. According to Meier

Using the above framework, we address the following question, "What is the effect of intervention in a population in which a different fraction have an unobserved binary variable that does not interact with treatment in its effect on outcome?" We investigate this question for three common outcome measures, absolute difference (

In related work, Gail et al

Methods

We start with a standard BK-Plot (Figure

The left side represents a standard BK-Plot, where the diagonal lines correspond to the probabilities of outcome in two randomization groups as a function of the fraction of subjects with the unobserved binary variable

The left side represents a standard BK-Plot, where the diagonal lines correspond to the probabilities of outcome in two randomization groups as a function of the fraction of subjects with the unobserved binary variable. The right side depicts a modified BK-Plot, where the outcome measure is plotted as a function of the fraction of subjects with the unobserved binary variable. We assume no interaction between the unobserved binary variable and treatment effect on the probability of outcome. Graphically, this means that we created BK-Plots so that the outcome measure has the same value at the leftmost and rightmost points.

We consider three common outcomes measures: the absolute difference in probability of outcome (

For each outcome measure we present a BK-Plot under the assumption of no-interaction between treatment and the two levels of the unobserved binary variable in their effect on the outcome measure. In other words, to fulfill the condition of no interaction between the treatment and the unobserved binary variable, the outcome measure comparing treatment groups, whether

To investigate how the outcome measure changes as the proportion of subjects with a given level of the unobserved binary variable varies from 0 to 1, we present a modified BK Plot (Figure

Results

Based on Figure _{z}(_{xz }denote the probability of outcome in randomization group

_{A}(_{0A}(1 - _{1A }

_{B}(_{0B}(1 - _{1B }

For an additive model, the outcome measure is the absolute difference, _{xA }- _{xB}. Under the assumption of no interaction between treatment effect and the unobserved binary variable, _{xA }- _{xB }= _{A}(_{B }(

For a multiplicative model, the outcome measure is the relative risk, _{xA}/_{xB}. Under the assumption of no interaction between treatment effect and the unobserved binary variable, _{xA}/_{xB }= _{A}(_{B}(

The results differ when the outcome measure is the odds ratio, _{xA }(1 - _{xB})/(_{xB }(1 - _{xA})). Under the assumption of no interaction between treatment effect and the unobserved binary variable, _{xA }(1 - _{xB})/(_{xB }(1 - _{xA})) - _{A}(_{B}(_{B}(_{A}(

Discussion

There is a large literature discussing the relative merits of using

Because the analyst must weight all the issues, we think it is helpful to present our perspective on some of the other factors that affect the choice of outcome measure. We believe the outcome measure should reflect the underlying model if it is known. Also we agree that one should consider how well the model of constant

It is sometimes argued that _{avg }= .47. The estimated probability of outcome in the last trial would then be .65 + _{avg }= 1.12, which violates the constraint on

Sometimes it is argued that

We agree with much of the literature that, in terms of interpretation,

Besides the choice of outcome measure, other factors affect the appropriateness of combining results from randomized trials and should be considered by the analyst. One factor is the variation in all-or-none compliance among trials. To reduce the variation from this factor, one can fit a model based on inherent compliance (i.e., with baseline subgroups "always-takers", "compliers", and "never-takers")

Another factor affecting the combination of results from randomized trials is the variation in treatment (e.g. variation in doses or levels of ancillary care). Despite the theoretical results in this paper, a large empirical study comparing the use of

Conclusion

The issue of generalizability of randomized trials is important in meta-analyses of randomized trials. To avoid bias from an unobserved binary variable that does not interact with treatment in its effect on outcome (and hence increase generalizability of results), one should use

Authors' Contributions

SGB wrote the initial draft. BSK made substantial improvements to the manuscript.

Appendix

If one has data from a randomized trial, the following calculation shows the possible bias from using _{A }= _{A}(.5) and _{B }= _{B}(.5). With _{0z }will be the same distance above _{z }as _{1z }is below _{z}. Therefore we can write _{0A }= _{A}(1 - _{1A }= _{A }(1 + _{0B }= _{B }(1 - _{1B }= _{B }(1 + _{B }- 1, 1) and _{A }- 1, 1). Let _{A }(1 - _{B})/(_{B }(1 - _{A})) denote the apparent odds ratio. Let OR^{*}_{x} = _{xA }(1 - _{xB})/(_{xB }(1 - _{xA})) denote the true odds ratio when all or none of the subjects have the unobserved covariate. Under the assumption of no interaction between the unobserved covariate and treatment effect, OR*_{0} = OR*_{1}. Solving this equation for

Substituting the above formula for _{0} gives a function of _{0} (_{A }= .2 and _{B }= .4, the apparent odds ratio is _{0}(.3) = .36, OR*_{0}(.5) = .32, or OR*_{0}(.9) = .20.

Pre-publication history

The pre-publication history for this paper can be accessed here: