Biometry Research Group, Division of Cancer Prevention, National Cancer Institute, Bethesda, Maryland, USA

Office of Disease Prevention, National Institutes of Health, Bethesda, Maryland, USA

Abstract

Background

There is a common belief that most cancer prevention trials should be restricted to high-risk subjects in order to increase statistical power. This strategy is appropriate if the ultimate target population is subjects at the same high-risk. However if the target population is the general population, three assumptions may underlie the decision to enroll high-risk subject instead of average-risk subjects from the general population: higher statistical power for the same sample size, lower costs for the same power and type I error, and a correct ratio of benefits to harms. We critically investigate the plausibility of these assumptions.

Methods

We considered each assumption in the context of a simple example. We investigated statistical power for fixed sample size when the investigators assume that relative risk is invariant over risk group, but when, in reality, risk difference is invariant over risk groups. We investigated possible costs when a trial of high-risk subjects has the same power and type I error as a larger trial of average-risk subjects from the general population. We investigated the ratios of benefit to harms when extrapolating from high-risk to average-risk subjects.

Results

Appearances here are misleading. First, the increase in statistical power with a trial of high-risk subjects rather than the same number of average-risk subjects from the general population assumes that the relative risk is the same for high-risk and average-risk subjects. However, if the absolute risk difference rather than the relative risk were the same, the power can be less with the high-risk subjects. In the analysis of data from a cancer prevention trial, we found that invariance of absolute risk difference over risk groups was nearly as plausible as invariance of relative risk over risk groups. Therefore

Conclusion

Unless the intervention is targeted to only high-risk subjects, cancer prevention trials should be implemented in the general population.

Background

Some prevention trials are restricted to high-risk subjects. If the investigators are only interested in the effects of the intervention on subjects at increased risk

However some investigators who are interested in studying the effect of the intervention in the general population may be tempted to design a "definitive" study to estimate the effect of the intervention in a high-risk group. Some investigators may believe that a trial of high-risk subjects would have greater power than a trial of the same size among average-risk subjects. Some examples of this type of thinking can be found in papers on risk prediction models

Methods and results

Possibly lower statistical power

To crystallize our thinking about statistical power, we consider the following simple hypothetical and realistic example. Investigators want to estimate the effect of intervention in the general population, so they first consider designing a randomized trial among the general at-risk population. Suppose they anticipate that the cumulative probability of incident cancer over the course of the study is _{C }= .02 in the control arm and _{I }= .01 in the study arm, and they believe that the difference in probabilities is clinically significant. Also suppose that due to the limited availability of the intervention, they can enroll at most

where NormalCDF is the cumulative distribution function for a normal distribution with mean 0 and variance 1, Δ is the anticipated difference one wants to detect, _{Null }is the standard error under the null hypothesis, and _{Alt }is the standard error under the alternative hypothesis. Let _{C }+ _{I})/2. As discussed in

For a study designed to estimate the relative risk, the statistic of interest is

Applying these formulas to the above example and substituting either (2) or (3) into (1), the investigators obtain a power of .74 based on the absolute risk difference statistic and a power .76 based on a relative risk statistic [see

Appendix A, worked-out calculations of power.

Click here for file

Suppose the investigators think this power is too low. To increase power they propose to restrict the study to a high-risk group in which the probability of cancer is .04. Also suppose the investigators make the typical assumption that if the intervention yields a relative risk of .5 in the general population, it would also yield a relative risk of .5 in the high-risk group. Applying (1–3) with high risk subjects for whom _{C }= .04 and _{I }= .02 with

Is there a free lunch? An underlying assumption in this example is that the relative risk is invariant between the general population and the high-risk group. There is no free lunch because the impact of violating this assumption could be substantial. For example, suppose instead that the absolute risk difference is invariant between the general population and the high risk group. Under this scenario the absolute risk difference in the general population is .01, so the absolute risk difference in the high-risk group is also .01. In this case for _{C }= .04, _{I }= .03, and _{C }- _{I }is the same, but _{I }is increasing, the variances, _{C}(1 - _{C})/_{I}(1 - _{I})/_{C }increases up to .5, which will reduce the power.

A crucial issue is whether or not the absolute risk difference or the relative risk is likely invariant between average-risk subjects in the general population and high-risk subjects. The answer depends on the cancer, the interventions, and the biology. To gain some appreciation of this issue, we analyzed published data (summarized in Table

Data from a cancer prevention trial for investigating assumptions of constant risk difference and relative risk when risk groups change.

Placebo group

Tamoxifen group

Variable

risk group

cancer

at risk

cancer

at risk

age at entry

1

≤ 49

68

10149

38

10045

2

50–59

50

7912

25

8040

3

>60

57

7719

26

7782

predicted risk

1

≤ 2.00%

35

6318

13

6311

2

2.01–3.01%

42

8108

29

8262

3

3.01–5.00%

43

7313

27

6959

4

≤ 5.01%

55

4142

20

4425

family risk

1

0

38

5891

17

5724

2

1

90

15000

46

15182

3

2

37

4263

20

4211

4

3

10

729

6

855

Cancer is invasive breast cancer. Predicted risk is the 5-year predicted risk. Family risk is number of first degree relatives with breast cancer. Data are from Table 5 of [5] with number at risk computed by dividing number of breast cancers by reported breast cancer rate.

constant risk difference,

where

varying risk difference,

where _{i }is the risk difference that varies over groups;

constant relative risk,

where

varying relative risk,

where

We obtained maximum likelihood estimates of _{i}, _{i }using a Newton-Raphson procedure [see

Appendix B, likelihood formulations

Click here for file

To investigate the plausibility of the constant relative risk and constant risk difference models in this example, we plotted the estimates of _{i}, _{i }along with confidence intervals (Figure

Data from the tamoxifen prevention trial

Data from the tamoxifen prevention trial. See text for a description of groups. Horizontal lines are estimates and 95% confidence intervals for model for constant absolute risk difference per 1000 (RD) or relative risk (RR). P-values correspond to likelihood ratio tests comparing the models with varying and constant risk difference or relative risks.

The trial designer does not know the true state of nature. If

Possibly increased costs

Even if the model is correct (namely _{C }and _{I }are correctly chosen), the smaller trial of high-risk subjects may be more expensive than the larger trial of average-risk subjects from the general population. Consider the following two trials with a power of .90 and a one-sided type I error of .05. In the trial of high-risk subjects _{C }= .04 and _{I }= .02, and in the trial of average-risk subjects, _{C }= .02 and _{I }= .01. Suppose the statistic of interest is the absolute risk difference. To obtain sample size for each randomization group we use the standard sample size formula

where _{C }+ _{I})/2, 1.644485 is the z-statistics corresponding to the 95th percentile of the normal distribution (for a one-sided type I error of .05) and 1.28155 is the z-statistics corresponding to the 90th percentile (for a power of .90). Based on (4), the sample size for a trial using average-risk subjects from the general population study is 2529 per group and the sample size for a trial of high-risk subjects is 1244 per group. Let _{R }denote the cost of recruitment per subject and _{I }denote the cost of intervention and follow-up per subject

_{general }= 2(_{R }2529 + _{I }2529), (5)

and the total cost of the trial for high-risk subjects is

_{high-risk }= 2(_{R }1244/_{I }1244). (6)

where the factor of 2 is for the two randomization groups. The condition for the trial of high-risk subjects to cost more than the trial of average-risk subjects (namely _{high-risk }>_{general}) is

when 1244/_{R}/_{I }> .34. If _{R}/_{I }> .13.

In many cancer prevention trials the above values of _{R}/_{I }are likely. For example, diagnostic testing to identify high-risk smokers can include expensive airway pulmonary function tests or bronchoscopy. In the future, more trials will likely involve expensive genetic testing of subjects _{R}/_{I}.

Even without diagnostic testing, the costs of obtaining high-risk subjects can be substantial. If

One additional consideration is how noncompliance and contamination affect the intent-to-treat analysis. If noncompliance and contamination can be anticipated, the investigator can correspondingly adjust the sample size and costs. Mathematically the effect of noncompliance and contamination is to change the values of _{C }and _{I }in (4), which would then affect (5) and (6). In some settings, investigators may anticipate that high-risk subjects are more likely to comply with the intervention than average-risk subjects. To compensate for the anticipated increased compliance, study designers could reduce the sample size which would lower costs. However, in other situations, investigators may anticipate that subjects found to be at high-risk on a diagnostic test would likely seek the best therapy outside of the trial rather than chance randomization to standard or experimental therapy. To compensate for the anticipated dilution in treatment effect, investigators would need to increase the sample size which would increase the costs.

For the above reasons even if the probabilities under the alternative hypothesis are correctly specified, some trials of high-risk subjects may be more expensive than larger trials of average-risk subjects with the same power and type I error.

Possibly misleading ratio of benefits to harms

When there is strong evidence prior to the trial of a high probability of harmful side effects due to the intervention, one would want to restrict the intervention to high-risk subjects. Otherwise, some investigators may be tempted to estimate the ratio of benefit to harms in the trial of high-risk subjects and extrapolate the ratio to average risk subjects. Unfortunately, even if the assumption of constant relative risk over risk categories were true, extrapolating the benefit-harm ratio from a high risk group to the general population could be misleading.

Suppose that in a randomized trial involving average-risk subjects from the general population the probability of cancer is .02 in the control arm and .01 in the study arm. Also suppose that relative risk is same in the general population as in the high-risk group, so that in a randomized trial involving a high-risk group, the probability of cancer is .04 in the control arm and .02 in the study arm. Furthermore, suppose that the probability of harmful side effects is the same for high-risk subjects as for average-risk subjects in the general population, namely .015 in the control arm and .025 in the study arm. Based on these results, for every 1000 high-risk persons who receive the intervention, (.04 - .02) 1000 = 20 will benefit from the intervention and (.025 - .015) 1000 = 10 will be harmed by side effects, yielding a benefit-harm ratio of 20:10 = 2:1. Similarly for every 1000 average-risk person who receive the intervention, (.02 - .01) 1000 = 10 will benefit from the intervention and (.025 - .015) 1000 = 10 will be harmed by side effects yielding a benefit-harm ratio of 10:10 = 1:1. In this example it would be incorrect to extrapolate the high benefit-harm ratio estimated from the high-risk group to the general population for whom the benefit-harm ratio is much lower. For many cancer prevention interventions, the ratio of life-threatening disease avoided to life threatening harms would be favorable in the high-risk group but not favorable when extrapolated to the general population.

Conclusion

There is no "free lunch" when using high-risk subjects in prevention trials design to make inference about the general population. Using high risk subjects instead of average-risk subjects from the general population may lower statistical power, increase costs, and yield a misleading ratio of benefit to harms than actually the case.

Given the substantial costs of definitive randomized trials in cancer prevention, and the importance of accurately assessing the balance of benefit and harm when treating healthy and asymptomatic people, it is therefore important to conduct trials in the actual target population rather than try to conduct them in high-risk populations with the plan to extrapolate results to the general population.

Competing Interests

The authors declare that they have no competing interests.

Authors' contributions

SGB wrote the initial draft, and BSK and DC made valuable improvements. All authors read and approved the final manuscript.

Pre-publication history

The pre-publication history for this paper can be accessed here: