Abstract
Background
Simon’s twostage designs are widely used for cancer phase II trials. These methods rely on statistical testing and thus allow controlling the type I and II error rates, while accounting for the interim analysis. Estimation after such trials is however not straightforward, and several different approaches have been proposed.
Methods
Different approaches for point and confidence intervals estimation, as well as computation of pvalues are reviewed and compared for a range of plausible trials. Cases where the actual number of patients recruited in the trial differs from the preplanned sample size are also considered.
Results
For point estimation, the uniformly minimum variance unbiased estimator (UMVUE) and the bias corrected estimator had better performance than the others when the actual sample size was as planned. For confidence intervals, using a midp approach yielded coverage probabilities closer to the nominal level as compared to socalled ’exact’ confidence intervals. When the actual sample size differed from the preplanned sample size the UMVUE did not perform worse than an estimator specifically developed for such a situation. Analysis conditional on having proceeded to the second stage required adapted analysis methods, and a uniformly minimum variance conditional estimator (UMVCUE) can be used, which also performs well when the second stage sample size is slightly different from planned.
Conclusions
The use of the UMVUE may be recommended as it exhibited good properties both when the actual number of patients recruited was equal to or differed from the preplanned value. Restricting the analysis in cases where the trial did not stop early for futility may be valuable, and the UMVCUE may be recommended in that case.
Background
Phase II trials primarily aim at evaluating the activity of a new therapeutic regimen to decide if it warrants further evaluation in a largerscale phase III trial, where it is usually compared to a standard treatment. The screening purpose of phase II trials implies that they are designed to reject a new therapeutic regimen showing low therapeutic activity. In cancer phase II trials, therapeutic activity is typically defined in terms of tumor shrinkage [1,2], and a patient with tumor shrinkage is referred as a responder. The endpoint of such phase II trials is thus a binary endpoint (responder / nonresponder), and a new anticancer agent with too low a response rate should be excluded from further consideration.
Cancer phase II trials are often designed as multistage trials (two stages being most common) allowing early trial termination in case of a low response rate, in order to avoid giving patients an ineffective treatment and wasting resources. The original idea of such a strategy with early termination was suggested by Gehan [3], and many designs were then proposed ([46], among others). Among all available multistage designs, Simon’s design [6] is probably the most commonly used in practice. Conversely, early termination for high efficacy is not as important in the phase II setting. Actually, there are less ethical needs to stop the trial early for an effective agent, and accumulating data on both therapeutic activity and safety is important before setting up a largescale randomized phase III trial.
As phase II trials primarily lead to the decision to proceed to a next step in the evaluation of the therapeutic regimen or not, their design essentially relies on statistical testing. Cancer phase II trials are therefore designed to control the probabilities to continue with an ineffective regimen or to abandon an effective one (type I and II error rates, respectively). Further analysis, and in particular estimation, is nevertheless useful and usually conducted, especially if the new regimen is selected for further consideration [7,8]. A point estimate of the response rate, a confidence interval and sometimes a pvalue are then computed at the termination of the trial. In particular, the point and confidence interval estimates are useful to design the future phase III trial, as well as other phase II trials. Owing to the possibility of early termination, the sample response rate, i.e. the maximum likelhood estimator (MLE), is typically biased, which is known as the optional sampling effect. Many approaches have thus been proposed to reduce the bias or the mean squared error (MSE) of estimators in such a setting [7,914].
One important point concerning inference in twostage phase II trials has been somewhat overlooked in the literature. As estimation is most important when the therapeutic regimen has been considered as effective, inference may be more common when the phase II trial proceeded to the second stage as compared to cases where it was stopped for futility at the first stage. Inference may thus be conditional on proceeding to the second stage (as e.g. in [12,13]), or unconditional, over all possible paths as implicitely considered in most other works.
Another issue is the actual total sample size of the trial. Cancer phase II trials are generally of limited sample size, and methods are derived from the ’exact’ binomial distribution of data. However, the actual number of patients recruited in the trial may be different from the planned sample size [11,15]. Inference in a Simon’s design where the sample size has been modified is however not straightforward, even in terms of hypothesis testing. A method has thus been proposed in the case where dropouts are noninformative so that the interim analysis can always be performed after inclusion of the planned number of patients and the actual second stage sample size does not depend on results observed during the first stage [11]. Although designs where the second stage sample size can be adapted according to the first stage result exist [16,17], this was not considered here.
In this paper, we compare the performance of the different approaches proposed in the literature for inference in a twostage Simon’s phase II trial. In the next section, we present the different point estimators, confidence intervals and pvalues proposed in the case where the actual sample size is as planned and in the case where the actual stage 2 sample size of the trial is different from the planned one. Then, results of a numerical study comparing the properties of the different methods in various settings are presented. We conclude with some discussion.
Methods
Simon’s design and notations
Let us denote Πas the true response rate when given some anticancer agent. Usual methodology of cancer phase II trials consists in testing the null hypothesis Π≤Π_{0}versus Π≥Π_{1} = Π_{0} + δ, where Π_{0} is the highest probability of response which would indicate that the agent is of no further interest, and Π_{1}the smallest probability of response indicating that the agent may be promising. Simon [6] considered twostage designs where no stopping for efficacy is possible after the first stage. Briefly, n_{1}subjects are accrued during the first stage. If the number of responses observed in the first stage X_{1} is lower or equal to a critical value r_{1}, the trial is stopped for futility. If X_{1}>r_{1}, the trial proceeds to a second stage where n_{2} additional patients are accrued. Let us denote X_{2} the number of responses observed in the n_{2}second stage patients, X_{t} = X_{1} + X_{2}and r_{t} the final critical value. Then if X_{t}≤r_{t}futility is concluded at the end of the trial, whereas efficacy is concluded if X_{t}>r_{t}. Given (Π_{0}Π_{1}) many such twostage designs may satisfy the prespecified type I and II error rates (αβ). Simon proposed two criteria to choose an appropriate design among such acceptable designs. The first one minimises the expected sample size under the null hypothesis and is referred to as the ’optimal’ design. The second one minimizes the maximal sample size n_{t} = n_{1} + n_{2}and is referred to as the ’minimax’ design. Jung et al.[18] further proposed a graphical method to search for alternative compromises between Simon’s optimal and minimax designs. For simplicity, we will however here concentrate on the two original Simon’s designs, although all following results may apply to any twostage design where no early stopping for efficacy is possible.
We suppose here that the sample size of the trial corresponds to the planned n_{1} and n_{2}, and that the stopping rules have been respected at the end of the first stage. Then, as X_{1} and X_{2}are both sums of independent Bernoulli trials, they follow a Binomial distribution of parameters (n_{1}, Π) and (n_{2}, Π), respectively. Let us denote M the stopping stage, S the total number of response observed at the end of the trial (S = X_{1} if M = 1 and S = X_{t} if M = 2), and N the total sample size of the trial (N = n_{1} if M = 1 and N = n_{t} if M = 2). Jung et al.[10] showed that (MS) is a complete and sufficient statistic for Π, and that the probability mass function of (MS) was given by
for s = 1,…,r_{1} if m = 1 and s = r_{1} + 1,…,n_{t} if m = 2, and where and .
Inference following a twostage design
Point estimate
Although the primary goal of phase II trials is decision making rather than inference, obtaining an estimate of the true response rate is often of interest, particularly when the trial was deemed successful and the new drug accepted for further evaluation in phase III trials [7].
The maximum likelihood estimator (MLE) is simply the sample proportion
Due to the sequential nature of the trial, the MLE is biased. Actually, in Simon’s design, when extreme small values of X_{1} are observed at the first stage, the trial is terminated without a chance to correct the downward bias, leading to a negatively biased MLE. More precisely, the bias is given by
Building on prior work of Whitehead [19], Chang et al.[9] proposed a biasadjusted estimator as the numerical solution of
Guo and Liu [7] proposed a simplified estimator motivated by the same bias substraction idea, but much simpler to obtain numerically by evaluating the bias at the MLE:
Noting that X_{1}/n_{1} is unbiased for Π, an unbiased estimator of Π can be obtained by the Rao–Blackwell theorem as the conditional expectation of X_{1}/n_{1} given (m,s), where (m,s) is the value of (M,S) observed in the trial. This estimator was first considered by Chang et al.[9] and further studied by Jung et al.[10] who showed this estimator was the uniformly minimum variance unbiased estimator (UMVUE). In the case of Simon’s twostage design, it is given by
A median unbiased estimator may be considered as the value of Π such that the corresponding pvalue would be 0.5 (see next section). It was used by Koyama and Chen [11] when n_{2} is different from its prespecified value, and will thus be denoted by , although they used in their article when n_{2} was as planned.
Another approach was used by Tsai et al.[12], who restricted their analysis to cases where the trial proceeded to the second stage. In these cases, they derived a (conditional) maximum likelihood estimator of Π accounting for the truncated distribution of X_{1} (which must be at least r_{1} + 1). This conditional estimator will be denoted by . To compare all estimators on a fair basis, we assumed that when the trial stopped at the first stage, an unconditional MLE was used. A conditional distribution given X_{1}≤r_{1} may also be derived, but it makes little sense in cases where r_{1} is small, in particular when r_{1} is 0 or 1, which is the case for optimal and minimax designs for Π_{0} = 0.05 and Π_{1} = 0.2 or Π_{1} = 0.25 with α=0.05 and β=0.1, for instance. We thus preferred not to consider conditional inference for early trial termination.
Relating to the work of Tsai et al.[12], Li recently proposed an MSEreduced estimator of Π as a weighted mean of the naive estimator and [14]. This estimator showed slightly higher bias than , with a slightly lower MSE, but no clear advantage. It was thus not further considered here.
For inference conditional on proceeding to the second stage, the uniformly minimum variance conditionally unbiased estimator (UMVCUE) can also be obtained, as proposed by Pepe et al. who proposed it and studied its properties [13]. Noting that X_{2}/n_{2}is unaffected by the early stopping option and thus conditionally unbiased for Π, the UMVCUE is obtained similarly to the UMVUE as the conditional expectation of X_{2}/n_{2} given (m = 2,s). It will be denoted by . To provide an estimate when the trial stops at the first stage, several choices are possible, and we decided to use the first stage sample proportion X_{1}/n_{1}, which is equal to the UMVUE in this case. For Simon’s design, the UMVCUE can thus be obtained by
Numerical studies in various settings showed that the biasedcorrected estimators and had often similar performance in terms of bias and mean squared error (MSE), with much smaller bias and slightly higher MSE than the MLE. As compared to the UMVUE, the MLE and the biascorrected estimators have been shown to have smaller MSE in many situations, but not always [7,10]. Other estimators were not extensively compared to each other or to the previous ones, in particular in the setting of conditional inference or when the actual sample size differes from the preplanned one. Determining in which situation one estimator would be preferable thus remains unclear.
Pvalue
Once (ms) is observed, the decision rules using critical thresholds r_{1}and r_{t} are sufficient to conclude at the rejection of the null hypothesis or not. It remains however common practice to compute a pvalue at the end of the trial [11]. The first idea that can still be found in many applications is to compute the pvalue as if the number of responders followed a binomial distribution of parameters (nΠ_{0}). This yields the naive pvalue p_{n}
The assumption on the distribution of S is true if m = 1, but obviously wrong if m = 2. This is exemplified on equation (7) by the summation on impossible sample paths where X_{1}<r_{1} and X_{2} = s−X_{1}.
It is therefore necessary to use the proper distribution of observed data to compute a pvalue. The pvalue is the probability under the null hypothesis to obtain a result at least as extreme as the one observed. Owing to the multistage procedure, several orderings, i.e. several definitions of ”at least as extreme”, may however be considered even if the proper distribution is used [20]. For instance, assume a design with n_{1}=24, n_{2}=39, r_{1}=8 and r_{t}=24 (optimal design for Π_{0}=0.30, Π_{1}=0.50, α=0.05 and β=0.10). One may consider that obtaining 18 responders out of 63 patients after proceeding to the second stage is less extreme than obtaining 7 responders out of 24 patients and stopping at the first stage, because 18/63=0.286 is less than 7/24=0.292. This corresponds to MLE ordering [20,21]. Conversely, one may also use stagewise ordering, and consider that 18/63 is a more extreme result than 7/24 because it was observed after proceeding to the second stage instead of stopping at the first stage. Indeed, to proceed to the second stage the number of responders in the first stage was at least 9. This is the ordering recommended in Jennison and Turnbull in the general case of sequential clinical trials [20, chapter 18.4, p 180], and the one they use to compute exact confidence bounds for Π[22].
The pvalue based on MLE ordering is
The biascorrected estimators have the same ordering as the MLE [23]. They thus result in exactly the same pvalue.
Jung et al.[10] showed that UMVUE ordering is equivalent to stagewise ordering and later defined a pvalue based on this ordering as [23]
It can be rewritten as
which is equivalent to the pvalue given by KoyamaChen for designs where attained n_{2} is as planned [11].
When estimation is performed conditional on proceeding to the second stage, a conditional pvalue can also be proposed. Let us denote f_{Π}(sm = 2) the probability mass function of S conditional on m = 2,
where f_{Π}(m,s) is given in (1). When the trial proceeds to the second stage, the conditional pvalue p_{c} is computed by
If the trial is stopped at the first stage, p_{c} can simply be computed by and is thus equal to p_{s}.
Confidence interval
Beside point estimates, confidence intervals are often reported in phase II trials. Despite the onesided nature of Simon’s design, it is not uncommon to report twosided (1−2α) confidence intervals rather than left (1−α) onesided confidence intervals. We will thus make this choice although both approaches are consistent with the onesided test performed at level α. Note however that in many applications, twosided 95% confidence intervals are reported, whatever the choice on the (onesided) αlevel.
The first basic idea is to use Clopper–Pearson [24] exact confidence interval ignoring the group sequential nature of the trial. We refer to this approach as the naive exact confidence interval in the sequel. Another solution is to use the Clopper–Pearson definition of an exact confidence interval using the appropriate distribution of (MS) [20]. This defines the exact equal tail (1−2α) confidence interval as (Π_{1}Π_{2}), where Π_{1} and Π_{2} are the numerical solutions of
and
The existence of this interval relies on the stochastic ordering of the distribution of (M,S) with respect to Π[10]. It is the same as the confidence interval used in several other works [11,22]. As it uses the UMVUE or stagewise ordering, we refer to it as the exact stagewise confidence interval. Using MLE ordering instead of stagewise ordering does not result in the same property of stochastic ordering [10]. It was therefore not further considered.
In the simple setting of a single binomial proportion, the Clopper–Pearson confidence interval is known to be conservative [25]. Actually, the actual confidence level is bounded below by (1−2α) [26]. To correct for this conservative nature, it has been suggested to use socalled midp confidence intervals [27]. We thus extended the stagewise ordering confidence intervals with a midp approach as (), where and are the numerical solutions of
and
Tsai et al.[12] considered several other intervals, both asymptotic and exact, but focusing on cases were the trial proceeds to the second stage, and using conditional inference as stated earlier. Asymptotic confidence intervals considered were the Wald and score intervals, both with or without continuity correction, and based on the conditional MLE given the trial proceeds to a second stage (referred as MLE in their article). Exact confidence intervals were Clopper–Pearson as explained above, but based on the conditional distribution of (M,S) given m = 2 (equation 10), and Sterne exact interval, modified to obtain an interval when the original method produces disjoint intervals as a confidence region. They concluded upon recommendation of score confidence intervals with continuity correction. Only the latter and Clopper–Pearson intervals will thus be considered here, and referred as the conditional score and conditional exact confidence intervals. Moreover, we proposed a midp confidence interval using the conditional distribution of (M,S) given m = 2. It is referred as the conditional midp confidence interval. Pepe et al. used parametric and nonparametric bootstrap confidence intervals for the UMVCUE in their article [13]. They showed that both methods yielded coverage probabilities reasonably close to the nominal level, but lower for the parametric bootstrap than for the nonparametric bootstrap. However, these methods do not provide correct confidence intervals in some situations, for instance when X_{2}=0 or s = n_{t}. They were thus not considered here.
Extended or shortened trial
It is not uncommon that the actual sample size of a phase II trial would be different from the planned sample size [11,15]. This may be due to differences between anticipated and actual accrual and drop out rates, for instance. For a two stage design, current practice often relies on ignoring the over or underaccrual or in recomputing the decision boundaries as if the attained sample size had been planned in a singlestage design, which leads to bias and possible inflation of the type I error rate. Koyama and Chen [11] recently proposed a method to calculate a new critical value for the second stage analysis assuming dropouts and overrun would be totally noninformative. In this case, the interim analysis can always be performed on the preplanned n_{1}subjects, and the difference in sample size only concerns the second stage sample size. They also proposed a method for inference at the end of the trial, thus providing a point estimate, a confidence interval and a pvalue.
Assume =n_{2} + Δn_{2} patients are accrued at the second stage instead of the preplanned n_{2}, and that success are then observed, where follows a binomial distribution of parameters (). Briefly, the method proposed consists in defining a new critical value for the second stage as the one leading to the same decision as when comparing the conditional pvalue of the second stage Pr to the conditional type I error rate given X_{1} = X_{1} in the original design with n_{2} patients at the second stage. The new conditional type I error rate is thus lower or equal to the original conditional type I error rate, allowing to control the unconditional type I error rate.
They also proposed to compute the unconditional pvalue as
where is the conditional power function at the second stage, and ^{Π∗}is the solution of . Solving for ^{Π∗} allows to extend the conditional power to all potential values of X_{1}, whereas only one particular value (X_{1}) was observed. The use of the conditional power function allows ordering different sample paths with different X_{1} and the actual sample size for stage 2 by comparing the ^{Π∗}, smaller ^{Π∗} indicating stronger evidence against the null hypothesis. This ordering is coherent with the hypothesis testing strategy they proposed, based on a new critical value to control the conditional type I error. In that respect, the pvalue _{pk} is lower than α if and only if the null hypothesis is rejected.
Koyama and Chen proposed the estimator as the value of Π_{0} yielding a pvalue p_{k}=0.5, and a twosided Clopper–Pearsonlike confidence interval based on p_{k}. The definition of p_{k} by equation 11 should allow to control the overall type I error rate, but the properties of the test, estimator and confidence interval have not been thoroughly studied.
Although Koyama and Chen used a biasedcorrected estimator when the second stage sample size was as planned, we denoted the median estimator presented above also in the case where n_{2} patients are accrued at the second stage.
Numerical study
To examine the properties of the different methods, numerical studies were conducted. Several design scenarios were considered, that covered a range of possible phase II trials in oncology. To help determining these scenarios, a limited literature search of phase II cancer trials using Simon’s design over the last years was performed. As this study was informal and arbitrarily limited to some journals, no results are reported. Twelve design scenarios where thus considered, with response rates under the null hypothesis of 0.05, 0.1, 0.2, 0.3, 0.4 and 0.5. Trials with higher values of Π_{0}were considered as pretty rare, and therefore not considered. For each value of Π_{0}, two differences in response rate between the null and alternative hypotheses were considered, namely 0.15 and 0.2. In all cases, the type I error rate α was set to 0.05 and the type II error rate β to 0.10 (90% power). Then, for each combination of design parameters, a choice between Simon’s optimal and minimax design was made on a case by case basis, according to the expected total sample size of the trial and the probability of early termination under H_{0} and H_{1}.
For each design scenario considered, the probability of all possible outcomes (M,S) was computed using equation (1) for a range of values of the response rate Π varying from Π_{0} to Π_{0} + 0.20 (thus Π_{1} when δ was 0.20 and slightly more than Π_{1}when δ was 0.15). For each possible outcome, the resulting estimators, pvalues and confidence intervals were also computed. As the probability of each outcome was the probability distribution of the estimators, pvalues and confidence intervals, the bias and root mean square error (RMSE) of estimators, the probability of rejection of the tests based on the pvalues and coverage probability of the confidence intervals could be derived.
To investigate the impact of accrual of some more or some fewer patients at the second stage as compared to the planned n_{2} value, trials where the second stage sample size was decreased by 1 or 2 or increased by 1, 2 or 5 were considered. These settings were not symmetrical because it was felt that overaccrual would be more frequent, because of the time delay to close a trial and because investigators would more likely want to protect the trial from patients exclusion and thus easily accrue more patients. Main analysis was unconditional: i.e. performance of the different methods was averaged over all possible outcomes. As some methods were more specifically developed to correct the analysis of the second stage results only, analysis restricted to cases where the trial proceeded to a second stage was also performed, and referred as conditional analysis.
To keep results simple and because the main findings were close to one scenario or another, only the results of six of the twelve scenarios are presented in detail. Additionally, these detailed results are only presented for situations where the second stage sample size was as planned. For situations where the second stage sample size was different from planned, the tables present results averaged over the different scenarios and the different values of Δn_{2} (simple arithmetic average without any weighting). However, the description of results encompassed the whole range of data obtained and not only the results presented in the tables. Particular cases where results were representative or different from the overall message were then isolated.
All computations were performed using R 2.13.2 statistical software [28].
Results
Trial accrual as planned
Results displayed in Figure 1 show that the UMVUE has no marginal bias as expected, the bias corrected estimator is almost unbiased, and if the median unbiased estimator and the MLE are biased, the bias remains limited, lower that 2% for the considered scenarios. In terms of RMSE, perfoms better than for values of Π closer to Π_{0} than to Π_{1}, while the RMSE of both estimators become similar when Πapproaches Π_{1}. As already noted in the illustrative examples of Guo and Liu [7], the MLE has the smallest RMSE under H_{0}. The median estimator also perfoms well in terms of RMSE, and even exhibits the smallest one for values of Π near Π_{0}. The conditional estimators have similar properties to each other, with much higher negative bias than the MLE, especially for values of Π close to Π_{0}. They had also higher or equal RMSE than the MLE.
Figure 1. Performance of the estimators: bias and root mean squared error (RMSE).
In terms of statistical testing, the test sizes represented on Figure 2 when Π = Π_{0} show that the naive binomial test and the test based on the conditional distribution are not adequate, these tests being too conservative in several settings. The test based on stagewise ordering leads to the correct decision, with the same probability of rejection as given by design. In our numerical settings, the test based on MLE ordering had similar characteristics as the test based on stagewise ordering. Actually, both only differ for a limited range of possible (M,S) outcomes, which has no impact in terms of test conclusion in the situations covered by the numerical study, although the nominal pvalues may be different.
Figure 2. Performance of the tests based on p values and the twosided 90% confidence intervals: probability of rejection and coverage probability. The line denoted by ’Design’ presents the probability of rejection according to the trial’s design i.e. when X_{t}>r_{t}.
Coverage probabilities of the 90% confidence intervals are presented in the right subpanel of Figure 2 for each design scenario. Overall, the properties of all methods but the midp approach where disappointing, in particular for small values of Π_{0} such as 0.05 for instance. The midp confidence interval had coverage probabilities closer to the nominal level than the other approaches in almost all situations. It was conservative under H_{0} for smaller values of Π_{0}, but the coverage probability fluctuated around 90% when Π_{0} was 0.20 or more, within a margin of −1%to + 2% only. On the contrary, the exact (stagewise ordering) confidence intervals had always a coverage probability above 90%, but often 2 to 3% above, and even between 7 and 8% above for smaller sample size trials. The conservative nature of Clopper–Pearson approach has already been reported, and the performance observed here for such intervals was however not clearly worse as that reported for socalled exact confidence intervals in a one sample (onestage) setting [25]. Note that the phenomenon of oscillations in coverage probability according to Π appearing on the graphs is known, and caused by the lattice structure of the binomial distribution [29]. The confidence intervals based on the conditional score with continuity correction which exhibited better conditional performance in the work by Tsai et al.[12] and the conditional midp confidence interval had close performance, but for Π departing from Π_{0}, their coverage probabilities were lower than the nominal level in this unconditional setting. This occurred less frequently and less dramatically for the conditional exact confidence interval, which however had a coverage probability clearly above its nominal level for Π close to Π_{0}, especially for small values of Π_{0}.
Extended or shortened trial
Results obtained when the second stage sample size was modified are presented in Tables 1 (average over all scenarios) and 2 for some of the situations. When the actual number of patients accrued was a little smaller or larger than planned, the UMVUE still yielded an unbiased estimator of the response rate. This was expected as the UMVUE is obtained as the conditional expectation of the first stage proportion given (M,S), without using any information on the decision boundaries at the second stage. If more or less patients are accrued in stage 2, this implies modifying this boundary to control for the type I error rate, but it has no impact on estimation. All other estimators were biased. In particular, Koyama–Chen method, aiming at correcting for increased or decreased sample size at the second stage also yielded an unconditionnally biased estimator, with bias and RMSE even superior to Guo’s corrected estimator. Both had however smaller RMSE than the UMVUE in most cases. The UMVCUE estimator and the conditional estimator had larger bias than the others under H_{0}, but their bias under H_{1} was similar to the one of Koyama–Chen estimator, with even lower RMSE for the UMVCUE.
Table 1. Performance of the different methods when second stage sample size was different from planned: average over the different design scenarios and differences between the planned and attained second stage sample size
Table 2. Performance of the estimators when second stage sample size is modified by Δn_{2}: bias and root mean squared error in selected situations
In terms of hypothesis testing and pvalues, all methods except the conditional test yielded very close results, with no increase of the type I error rate in the situations studied. Actually, the possible values of (M,S) where these methods disagreed in terms of rejection of the null hypothesis had very small probabilities in general, thus almost no impact on test size or power. In several situations, there were even no values of (M,S) for which the methods disagreed. On the contrary, the test based on the conditional pvalue had a probability of rejection markedly smaller than other methods, with both a type I error rate and a power clearly under their nominal value.
The midp confidence intervals had again coverage probabilities closer to the nominal 90% level than the other methods, in particular than the Koyama–Chen method which was corrected for sample size modifications. Over all 120 situations covered, the Koyama–Chen confidence intervals were rather conservative but always preserved the nominal confidence level, with coverage probabilities ranging from 90.0% to 98.5%, with an average of 93.4%. On the contrary, coverage probabilities ranged from 85.7% to 96.5% for the midp confidence intervals, with an average of 90.1%. Coverage probabilities under the nominal level were more frequent under H_{1} than under H_{0} and for higher values of the probability of response Π.
Analysis conditional on proceeding to stage 2
When analysis was restricted to the trials proceeding to the second stage, the performance of the estimators was different from previously (Figure 3). The UMVCUE of Pepe et al. was unbiased, whereas the conditional estimator of Tsai et al. had very small negative bias. All other estimators were positively biased, with marked bias under the null hypothesis that decreased when the true response rate increased towards the alternative hypothesis. Overall, the MLE estimator had less bias than Guo’s corrected estimator and the UMVUE. Interestingly, the Koyama–Chen estimator was even slightly negatively biased for Π close to Π_{1} or above, with a bias of the same magnitude than the bias of the conditional estimator under H_{1}.
Figure 3. Performance of the estimators for conditional inference: bias and root mean squared error (RMSE).
In terms of RMSE, the conditional estimators and had close performance, with negligible differences in favor of under H_{0} and of under H_{1}. Despite their bias, all unconditional estimators except the UMVUE had generally lower RMSE than the conditional estimators. With biases as high as 4% for response rate of 5% or as 8% for a response rate of 20%, these estimators cannot be recommended for conditional inference, however.
Conditional inference was also the only one preserving the conditional type I error, but the test could be rather conservative in some situations (Figure 4). As a consequence, the power conditional on proceeding to the second stage could be lower than 90% in some cases. As described in Tsai et al.[12], the conditional score performed better than the conditional exact confidence interval. The conditional midp confidence interval had coverage probabilities very close to the conditional score interval.
Figure 4. Performance of the tests based on p values and the twosided 90% confidence intervals for conditional inference: probability of rejection and coverage probability.
When the sample size at the second stage n_{2} was different from its planned value, the conditional estimators achieved similar bias reduction as when n_{2} was as planned (Table 3). In particular, the UMVCUE was virtually unbiased, at least in all designs scenarios considered here. The test based on the conditional pvalue p_{c}also allowed to control the conditional type I error. The coverage probabilities of conditional score and conditional midp confidence intervals tended to be higher under H_{0} than under H_{1}, and closer to their nominal value under H_{1}, whereas the reverse was observed for other methods. As compared to the conditional estimator, Koyama–Chen estimator had similar bias and lower RMSE under H_{1}, but much higher bias under H_{0}. It should however be noted that this estimator is constructed as a median and not a mean estimator, so that some degree of bias can be expected when estimating the response rate. In terms of hypothesis testing, this method however failed to adequately control the conditional type I error rate and confidence intervals had too high coverage probability in most cases.
Table 3. Performance of the different methods for conditional inference when second stage sample size was different from planned: average over the different scenarios
Discussion
In terms of estimation, and should be recommended as they perform better than the other estimators, in particular when the true response rate is higher than the one under H_{0}, i.e. in cases when estimation is the most important. Although our simulations did not encompass all possible ranges of response rates and treatment effects, they cover a wide range of plausible situations, in which no clear advantage of the bias corrected estimator over the UMVUE could be found.
The choice of a conditional or unconditional inference is clearly overlooked in practical applications. Conditional inference — and conditional bias in particular — has attracted some interest in the setting of group sequential phase III trials, with concerns rather directed at the conditional bias of the estimator of the treatment effect when trials were stopped early for efficacy [30,31]. In the setting of Simon’s twostage phase II trials, conditional inference would rather be favored when the trial did not stop at the first stage, especially if the trial was deemed succesful at the end [13]. Such aspects of conditional inference have however been rarely discussed to our knowledge [13,32]. Results show that unbiased or almost unbiased estimation can be performed using the UMVCUE [13] or the proper conditional distribution [12], respectively, both with very similar RMSE. In addition, both performed well even when the sample size at the second stage was slightly different from its planned value. To construct an estimator that would be both conditionally and unconditionally unbiased, one could also derive an estimator for trials stopping at the first stage that would use the conditional distribution given X_{1}≤r_{1}. In such a case, the estimator would be conditionally unbiased whether the trial was stopped at the first or the second stage, and thus would be unconditionally unbiased. Using a distribution of outcomes conditional on early stopping makes however little sense — if any — when r_{1} is small. For instance, if r_{1}=0, then the only potential outcome in case of early stopping is X_{1}=0, thus leading to a single possible value for the estimator of Π. It is therfore not possible to construct an unbiased estimator of any value of Π in this case. We therefore did not further develop this point in the paper. Another solution, however, would be to use a biasedcorrected estimator such as Whitehead’s [19] or Guo’s [7] when the trial was stopped early. This has already been evoked by Pepe et al.[13], without further investigations.
In this study, we have concentrated on Simon’s design for phase II cancer trials. Other designs or adaptations however exist. In particular, Jovic and Whitehead have recently proposed point estimates, confidence intervals and pvalues for a modified Simon’s design with early stopping for efficacy [33]. Other extensions of Simon’s design could also have been considered [5,34]. In cases where early stopping for efficacy is possible, the results of the methods proposed by Jovic and Whitehead could have been used. Tsai et al. also applied their conditional method to Shuster’s design [34]. Nevertheless, a short look at cancer literature shows that a majority of cancer phase II trials still use Simon’s design.
In practical applications, it may occurr that the actual number of patients recruited would be slightly different from the preplanned value. For instance some patients may be unevaluable for response or they may withdraw their consent during study. On the contrary, some patients may be included in the study before recruitment is formally closed. For these cases, where the decrease or increase of second stage sample size may be considered as non informative, Koyama and Chen proposed inference procedures based on conditional power [11]. They clearly state in their article that the properties of their estimators, pvalues and confidence intervals need to be further studied. In our numerical settings, it turned out that the UMVUE, which can still be used because it only makes use of boundary decisions at the second stage, performed better than the Koyama–Chen method. The behaviour of both estimators with modified sample size however deserve further investigations. Concerning confidence intervals, the midp intervals performed better than the socalled exact confidence intervals in most settings for both unconditional and conditional inference. Koyama and Chen however did not consider such an approach, and their confidence intervals rely on Clopper–Pearson method. Using a midp approach with their modifed pvalue (equation 11) may also have improved the coverage probabilities of the confidence intervals.
Another interesting field of further research concerns inference in adaptive phase II trials, where the second stage sample size can be adapted according to the first stage results [16,17]. In such cases, the decrease or increase in sample size cannot be considered as non informative anymore, and the method of Koyama and Chen does not apply. New developments are thus needed here.
Conclusions
For point estimation, the UMVUE was unbiased both when the actual number of patients recruited was equal to or differed from the preplanned value. The bias corrected estimator had negligible bias and slightly lower RMSE than the UMVUE only when the true response rate Π was close to its value under the null hypothesis. Both estimators perfomed better than the others and can thus be recommended. In terms of confidence intervals, midp confidence intervals performed best, as compared to the other exact confidence intervals, whether they ignore the group sequential nature of the trial or not.
When one is more particularly interested on inference conditional on having proceeded to the second stage, the UMVCUE which is unbiased may be recommended. Conditional score or conditional midp confidence intervals should then be used.
Competing interests
The authors declare that they have no competing interests.
Authors’ contributions
RP and KD designed the study, performed all statistical analyses and participated to article writing. Both authors read and approved the final manuscript.
References

Miller AB, Hoogstraten B, Staquet M, Winkler A: Reporting results of cancer treatment.
Cancer 1981, 47:207214. PubMed Abstract  Publisher Full Text

Eisenhauer EA, Therasse P, Bogaerts J, Schwartz LH, Sargent D, Ford R, Dancey J, Arbuck S, Gwyther S, Mooney M, Rubinstein L, Shankar L, Dodd L, Kaplan R, Lacombe D, Verweij J: New response evaluation criteria in solid tumours: revised RECIST guideline (version 1.1).
Eur J Cancer 2009, 45:228247. PubMed Abstract  Publisher Full Text

Gehan EA: The determination of the number of patients required in a preliminary and a followup trial of a new chemotherapeutic agent.
J Chron Dis 1961, 13(4):346353. PubMed Abstract  Publisher Full Text

Fleming TR: Onesample multiple testing procedure for phase II clinical trials.
Biometrics 1982, 38:143151. PubMed Abstract  Publisher Full Text

Chang MN, Therneau TM, Wieand HS, Cha SS: Designs for group sequential phase II clinical trials.
Biometrics 1987, 43:865874. PubMed Abstract  Publisher Full Text

Simon R: Optimal twostage designs for phase II clinical trials.
Control Clin Trials 1989, 10:110. PubMed Abstract  Publisher Full Text

Guo HY, Liu A: A simple and efficient biasreduced estimator of response probability following a group sequential phase II trial.
J Biopharm Stat 2005, 15(5):773781. PubMed Abstract  Publisher Full Text

Liu A, Wu C, Yu KF, Gehan E: Supplementary analysis of probabilities at the termination of a group sequential phase II trial.
Stat Med 2005, 24(7):10091027. PubMed Abstract  Publisher Full Text

Chang M, Wieand H, Chang V: The bias of the sample proportion following a group sequential phase II clinical trial.
Stat Med 1989, 8(5):563570. PubMed Abstract  Publisher Full Text

Jung SH, Kim KM: On the estimation of the binomial probability in multistage clinical trials.
Stat Med 2004, 23(6):881896. PubMed Abstract  Publisher Full Text

Koyama T, Chen H: Proper inference from Simon’s twostage designs.
Stat Med 2008, 27(16):31453154. PubMed Abstract  Publisher Full Text

Tsai W, Chi Y, Chen C: Interval estimation of binomial proportion in clinical trials with a twostage design.
Stat Med 2008, 27:1535. PubMed Abstract  Publisher Full Text

Pepe MS, Feng Z, Longton G, Koopmeiners J: Conditional estimation of sensitivity and specificity from a phase 2 biomarker study allowing early termination for futility.
Stat Med 2009, 28(5):762779. PubMed Abstract  Publisher Full Text  PubMed Central Full Text

Li Q: An MSEreduced estimator for the response proportion in a twostage clinical trial.
Pharm Stat 2011, 10:277279. PubMed Abstract  Publisher Full Text

Green SJ, Dahlberg S: Planned versus attained design in phase II clinical trials.
Stat Med 1992, 11(7):853862. PubMed Abstract  Publisher Full Text

Banerjee A, Tsiatis AA: Adaptive twostage designs in phase II clinical trials.
Stat Med 2006, 25(19):33823395. PubMed Abstract  Publisher Full Text

Englert S, Kieser M: Adaptive designs for singlearm phase II trials in oncology.
Pharm Stat 2012, 11(3):241249. Publisher Full Text

Jung SH, Lee T, Kim KM, George SL: Admissible twostage designs for phase II cancer clinical trials.
Stat Med 2004, 23(4):561569. PubMed Abstract  Publisher Full Text

Whitehead J: On the bias of maximum likelihood estimation following a sequential test.
Biometrika 1986, 73(3):573581. Publisher Full Text

Jennison C, Turnbull BW: Group Sequential Methods with Applications to Clinical Trials. CRC Press, Boca Raton; 2000.

Armitage P: Numerical studies in the sequential estimation of a binomial parameter.
Biometrika 1958, 45(12):115. Publisher Full Text

Jennison C, Turnbull BW: Confidence intervals for a binomial parameter following a multistage test with application to MILSTD 105D and medical trials.
Technometrics 1983, 25:4958. Publisher Full Text

Jung SH, Owzar K, George SL, Lee T: Pvalue calculation for multistage phase II cancer clinical trials.
J Biopharm Stat 2006, 16(6):765775. PubMed Abstract  Publisher Full Text

Clopper CJ, Pearson ES: The use of confidence or fiducial limits illustrated in the case of the binomial.
Biometrika 1934, 26(4):404413. Publisher Full Text

Newcombe RG: Twosided confidence intervals for the single proportion: comparison of seven methods.
Stat Med 1998, 17:857872. PubMed Abstract  Publisher Full Text

Neyman J: On the problem of confidence intervals.
Ann Math Statist 1935, 6(3):111116. Publisher Full Text

Mehta CR, Walsh SJ: Comparison of exact, midp, and MantelHaenszel confidence intervals for the common odds ratio across several 2×2 contingency tables.

R Development Core Team: R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria; 2011.
[ISBN 3900051070]

Brown L, Cai T, DasGupta A: Interval estimation for a binomial proportion.

Pocock SJ, Hughes MD: Practical problems in interim analyses, with particular regard to estimation.
Control Clin Trials 1989, 10(4):209221. Publisher Full Text

Freidlin B, Korn EL: Stopping clinical trials early for benefit: impact on estimation.
Clin Trials 2009, 6(2):119125. PubMed Abstract  Publisher Full Text

Ohman Strickland PA, Casella G: Conditional Inference Following Group Sequential Testing.
Biom J 2003, 45(5):515526. Publisher Full Text

Jovic G, Whitehead J: An exact method for analysis following a twostage phase II cancer clinical trial.
Stat Med 2010, 29(30):31183125. PubMed Abstract  Publisher Full Text

Shuster J: Optimal twostage designs for single arm phase II cancer trials.
J Biopharm Stat 2002, 12:3951. PubMed Abstract  Publisher Full Text
Prepublication history
The prepublication history for this paper can be accessed here: