Abstract
Background
Mixed effects logistic models have become a popular method for analyzing multicenter clinical trials with binomial data. However, the statistical properties of these models for testing homogeneity of odds ratios under various conditions, such as withincenter and amongcenters inequality, are still unknown and not yet compared with those of commonly used tests of homogeneity.
Methods
We evaluated the effect of withincenter and amongcenters inequality on the empirical power and type I error rate of the three homogeneity tests of odds ratios including likelihood ratio (LR) test of a mixed logistic model, DerSimonianLaird (DL) statistic and BreslowDay (BD) test by simulation study. Moreover, the impacts of number of centers (K), number of observations in each center and amount of heterogeneity were investigated by simulation.
Results
As compared with the equal sample size design, the power of the three tests of homogeneity will decrease if the same total sample size, which can be allocated equally within one center or among centers, is allocated unequally. The average reduction in the power of these tests was up to 11% and 16% for withincenter and amongcenters inequality, respectively. Moreover, in this situation, the ranking of the power of the homogeneity tests was BD≥DL≥LR and the power of these tests increased with increasing K.
Conclusions
This study shows that the adverse effect of amongcenters inequality on the power of the homogeneity tests was stronger than that of withincenter inequality. However, the financial limitations make the use of unequal sample size designs inevitable in multicenter trials. Moreover, although the power of the BD is higher than that of the LR when K≤6, the proposed mixed logistic model is recommended when K≥8 due to its practical advantages.
Background
The results from multicenter clinical trials or metaanalysis studies with binomial data are often summarized in K 2 × 2 contingency tables, where K denotes the total number of centers or studies. Combining data in such tables and proposition a summary measure is the primary objective of such studies. However, before computing the overall odds ratio, we often need to assess whether the specific odds ratios are homogeneous across tables [14].
Nowadays, investigators have a wide range of methods available for this purpose, including modelbased and testbased approaches. The excellent simulation studies conducted by pioneer researchers in this field assist us in choosing the most appropriate test for the assessment of homogeneity among K 2 × 2 tables [16]. Nevertheless, the results of these simulation studies indicate that homogeneity tests show different behaviors under combinations of parameters such as the number of centers, center sizes and amount of heterogeneity [3,5,7].
In recent years, a class of models called mixed logistic models has been used for analysis of multicenter clinical trials with binomial data. Although Agresti has discussed a likelihood ratio (LR) test based on a mixed logistic model for testing homogeneity of odds ratios in K 2 × 2 contingency tables [8], the statistical properties of this test and the other traditional homogeneity tests such as BreslowDay (BD) [9] and DerSimonianLaird (DL) [10] are still unknown. A situation which occurs frequently in multicenter trials and has not been evaluated in previous studies is the effect of unequal sample size designs on the statistical properties of these homogeneity tests. For example, in some multicenter clinical trials, when one center is larger, it may seem reasonable to select a larger sample from it, but this leads to amongcenters inequality. Moreover, withincenter inequality could occur when the costs of two treatment groups are very different. In this situation, due to financial restrictions, it is reasonable to allocate more patients to the cheaper treatment in each center. This simulation study hence compares the empirical power and type I error rate of the three tests of homogeneity of odds ratios, including LR, DL and BD tests when the sample size is unequal within one center or among centers.
Methods
Consider a series of K independent 2 × 2 contingency tables, with the data in the kth table denoted as shown in Table 1. In this paper, K is the number of centers in multicenter clinical trials.
Table 1. Summary of data from the kth 2 × 2 contingency table
Suppose that each y_{ik }follows a binomial distribution with parameters n_{ik }and π_{ik }(i = 1,2; k = 1,2,...,K). Let n_{ik }denote the total number of observations in the ith treatment arm and kth center, and let π_{ik }denote the success probability at treatment level x_{ik }in the kth center, where x_{ik }is the treatment indicator with x_{ik }= 1 representing treatment 1 and x_{ik }= 0 treatment 2.
In this paper, we focus on the following logisticnormal probability model in order to assess the homogeneity of odds ratios:
In this model, the segment α + βx_{ik }is the fixed effect part in which β is the common treatment effect. Here, u_{k }and b_{ik }are independent random components of the model, where u_{k }is the center effect and and the parameter summarizes center heterogeneity. In addition, b_{ik }is the centerbytreatment interaction random effect, and the parameter describes variability in the logodds ratios [8]. The two advantages of working with this model are: first, a test of homogeneity of odds ratios can be performed by testing the null hypothesis: against ; and, second, the common treatment effect and also centerspecific odds ratios can be obtained by estimating β and b_{ik }[8].
It should be noted that the homogeneity test can be performed using likelihood ratio (LR) test Δ = 2(l_{0 } l_{1}), where l_{0 }is the loglikelihood under the assumption of homogeneity of odds ratios and when there is just one random effect, u_{k}, in the Model 1 and l_{1 }is the loglikelihood when both u_{k }and b_{ik }are in the model. Under the null hypothesis, the asymptotic distribution of the LR test is a mixture of a chisquared distribution with zero and one degrees of freedom, respectively, both with weight of 1/2 [11,12].
Since the properties of the LR test in order to assess homogeneity of odds ratios had not been evaluated in the previous studies, we investigated the behavior of this test under various conditions, particularly under withincenter and amongcenters inequality in multicenter trials and also compared it with the other two most common test statistics, including DL and BD tests. Brief calculation details of these statistics are given in the appendix [912].
Simulation study
We studied the statistical properties of the three mentionedabove homogeneity tests, in terms of empirical power and type I error rate, by simulation in SAS statistical package v 9.1. Our simulation design was based on the mixed logistic model 1; we set the fixed effect parameters for all simulation scenarios arbitrarily as: α = 1 and β = 1 . The random components u_{k }and b_{ik }are generated independently from normal distribution with mean zero and variance and . In order to generate the response variable, the binomial distribution with parameters n_{ik }and was used. The influence of withincenter and amongcenters inequality, different values of K, , and also number of observations in each centers, n_{k}, on the statistical properties of all the homogeneity tests were evaluated. We generated 1000 random data sets for each simulation scenario. Namely, we chose three values for K (4, 6 and 8), two values for (0.1 and 0.5) and five values for (0, 0.2, 0.4, 0.6 and 0.8). In the equal sample size design, in which equal numbers of patients were allocated between two treatment arms and also among all centers, the number of individuals per center, n_{k}, was set at 40, 100 and 200. In addition, in this study two forms of unequal sample size designs were considered. In the first one, ie withincenter inequality, the same sample size, which can be allocated equally between two treatment arms within a center, is allocated unequally in the ratio of 3:1. In this case, the sample sizes were considered equal in all centers. In the latter one, ie amongcenters inequality, the same total sample size, which can be allocated equally among centers, is allocated unequally. In this situation, one center has a much larger sample size in comparison with the other ones. The exact details of different sample size configurations are described in Table 2. In addition, SAS simulation code for performing the three homogeneity tests is presented in the additional file 1.
Table 2. Description of different configurations of sample size in equal and unequal sample size designs.
Additional file 1. SAS simulation code for performing the three homogeneity tests of odds ratios in a certain case, in which K = 6, n_{ik }= 100, and .
Format: TXT Size: 16KB Download file
Results
Equal sample size design
Tables 3 shows empirical type I error rate (when ) and power (when ) of the three homogeneity tests under the equal sample size design and various combinations of n_{k}, K, , and . In terms of power, the tests are ordered BD≥DL≥LR in all cases and they perform similarly when the total sample size in each center is as large as 200 and . In addition, our findings reveal that substantial gain in the power of all homogeneity tests occurred with increasing number of centers, sample size per center and degree of heterogeneity of odds ratios, . However, increasing degree of heterogeneity of center, , from 0.1 to 0.5 resulted in a reduction of approximately 4.93% and 3.76% in the power of DL and BD, respectively, while no regular trend was detected for the change in LR.
Table 3. Type I error rate (when ) and the statistical power (when ) of homogeneity tests under equal sample size design.
Unequal sample size designs
Table 4 shows type I error rate and the power of the homogeneity tests under withincenter inequality for different values of n_{k}, K, , and . Indeed, when = 0.1, as compared with the equal sample size design, there were decreases of approximately 16.35%, 9.93% and 11.62% in the power of LR test for withincenter inequality, when K was equal to 4, 6 and 8, respectively. In addition, the power of the BD statistic decreased approximately by 10.67%, 7.79% and 7.15% and the power of DL was reduced approximately by 14.46%, 11.24% and 9.76% for K = 4, 6 and 8, respectively. It should be noted that, for a given value of K, the reported amount of reduction was the average reduction for all values of .
Table 4. Type I error rate (when ) and the statistical power (when ) of homogeneity tests under withincenter inequality.
Type I error rate and the power of the homogeneity tests under amongcenters inequality are illustrated in Table 5. In fact, when = 0.1 as compared with the equal sample size design, there were decreases of approximately 17.15%, 11.78% and 11.89% in the power of LR test and 16.63%, 13.62% and 12.57% in the power of the BD statistic for amongcenters inequality, when K was equal to 4, 6 and 8, respectively. In addition, this amount of reduction in the power of DL test was more critical and approximately equal to 21.56%, 18.4% and 16.61% for the same sequence of K.
Table 5. Type I error rate (when ) and the statistical power (when ) of homogeneity tests under amongcenters inequality.
It should be pointed out that the amount of reduction in the power of the three homogeneity tests for = 0.5 under two forms of unequal sample size designs was approximately similar to those of = 0.1.
Type I error rate
Tables 3, 4 and 5, when = 0, represent empirical type I error rate of the three homogeneity tests at the nominal significance level of 0.05. As indicated, LR test showed conservative behavior when K = 4, otherwise, the type I error rate was close to the nominal level. In addition, BD performed adequately in terms of type I error rate in almost all cases. On the other hand, type I error rate of the DL statistic was close or below the nominal level.
Discussion and conclusions
In a simple randomized clinical trial, the use of unequal allocation ratios, particularly the allocation ratio of 3:1, will significantly reduce the power of study for detecting significance difference between two treatments [1315]. To our knowledge, few published studies investigated the impact of withincenter and amongcenters inequality on the statistical properties of the tests of homogeneity of odds ratios in multicenter clinical trials [1,3,4]. As illustrated in Tables 3, 4 and 5, the type I error rate of the three homogeneity tests is approximately close to the nominal level of 0.05 except for LR when K = 4. Since the results show that these tests have almost the same type I error rate, power comparisons are possible. As compared with the equal sample size design, the power of the LR, BD and DL tests will decrease if the same total sample size, which can be allocated equally within one center or among centers, is allocated unequally. In this case, the power ranking of the tests was BD≥DL≥LR. It is worth mentioning that, as compared with withincenter inequality, amongcenters inequality has stronger adverse effect on the power of the homogeneity tests. Despite the use of different tests, these findings are inconsistent with those of Paul, who reported the adverse effect of withincenter inequality to be stronger [3].
Also, this paper shows how to use a mixed logistic model to test homogeneity of odds ratios in multicenter trials. In Model 1, there are two types of homogeneity: homogeneity of odds ratios among centers and homogeneity of centers. However, removing the centerbytreatment interaction from Model 1 leads to a model which can only be used to test homogeneity of centers. This model, which has been previously discussed by Gao, assumes that the odds ratios are constant over centers [16]. Therefore, it should not be used to generate data for comparing the tests of heterogeneity of odds ratios. Furthermore, the power of the three tests of homogeneity increases more when we increase the number of K and n_{k }compared to when we increase the number of K and . This result is in agreement with previous studies which have evaluated the influence of K and n_{k }on the power of the homogeneity tests [5,1719]. Nevertheless, our simulation study shows that the degree of amongcenters heterogeneity, , has little or no effect on the power of the three tests of homogeneity, except for DL when and sample size is small.
In addition, it is noteworthy that we used the DL statistic calculated from the oneway random effects model, which has approximately a chisquare distribution. However, Biggerstaff and Jackson [20] have calculated the exact distribution and power of the wellknown Q statistic based on the same random effects model, which can be used for testing homogeneity of odds ratios and be compared with the tests used in the present study.
In conclusion, of the three tests of homogeneity, the BD seems to be the most appealing with regard to its statistical properties: its type I error rate is close to the nominal level and its power is greater than that of DL and LR. Moreover, it has the advantage of simplicity of calculation and is recommended by a number of authors [1,46]. However, one limitation of BD test is that it has low power when the sample size within each center is small, even if the number of centers is large [1,2]. Nevertheless, despite having low power under small number of centers and its complexity, Model 1 has its own advantages. Firstly, when the centers are a random sample themselves, the LR test from the Model 1 enables inferences to extend to the population of centers. Secondly, a further consideration is that common odds ratio can be estimated from the fixed part of the Model 1, even when the odds ratios are not homogeneous. Thirdly, in each center, Model 1 provides a predicted logodds ratio that shrinks the sample value toward the mean. This is especially useful when the sample size in a center is small and the ordinary sample odds ratio has a large standard error [8]. In addition, the mixed logistic model described in this study will potentially be applicable to metaanalysis studies.
It is clear that, based on Model 1, the odds ratio in the kth center, as given in the appendix 4, is exp(β + b_{1k } b_{2k}), which can be written as C × exp(b_{1k } b_{2k}) where C = exp(β) .This indicates that the odds ratio in each center is absolutely independent of α and u_{k}. Indeed, the odds ratios are affected by b_{1k }and b_{2k}, and β has the same effect on odds ratio in all centers. Hence, to generate heterogeneous odds ratios among centers, the fixed simulation parameters, ie α and β, can be chosen arbitrarily.
It should be noted that, although using unequal sample size designs in multicenter clinical trials reduces both the power of the study and the power of the homogeneity tests, a substantial reduction in the total cost of the trial will compensate for the reduction in the power of the statistical tests [14,15]. Finally, further research is warranted to investigate the influence of the number of centers, unequal sample size design, sparseness and also deviation from normal assumption of the random effects on the robustness and accuracy of the estimates of the fixed and random parameters of the Model 1.
Appendix
1. BreslowDay statistic
Breslow and Day [9] proposed the test statistic:
Where OR_{MH }is the MantelHaenszel estimator of common odds ratio. E(y_{1k}OR_{MH}) and V(y_{1k}OR_{MH}) are the expected value and variance of y_{1k }under the null hypothesis of homogeneity of odds ratios. Under the assumption of large sample size in each 2 × 2 table, BD has approximately chisquare distribution with k  1 degree of freedom [9].
2. DerSimonianLaird statistic
The Dersimonian and laird statistic is based on random effects model, which obtained by:
When logodds ratio is used as a summary measure ie, common treatment effect, y_{k }= ln(OR_{k}) where ln(OR_{k}) is the logodds ratio in kth center and where . DL statistic has approximately chisquare distribution with k  1 degree of freedom [10].
3. Likelihood ratio test based on mixed logistic models
To test homogeneity of odds ratios based on mixed logistic models, we compare the model
to the simpler model not containing the random effect term of the centerbytreatment interaction, namely:
which is equivalent to test the null hypothesis of versus [8]. As the null hypothesis is on the boundary of the parameter space, the asymptotic null distribution for the likelihood ratio test is a mixture of a and a with equal probability 1/2, rather than classical single chisquare distribution [11,12].
4. Calculation of odds ratio based on the mixed logistic model with two random effects
Based on Model * which was presented above, the odds of success in treatment 1 in kth center is derived as:
and in a similar manner the odds of success in treatment 2 is derived as: therefore, the odds ratio for treatment 1 versus 2 in kth center is exp(β+b_{1k } b_{2k}).
5. SAS code for performing likelihood ratio test in a 2 × 2 × 2 contingency tables
data binomial;
input center treat y n; * y successes out of n trials;
cards;
1 1 30 100
1 2 50 100
2 1 45 100
2 2 75 100
;
run;
proc nlmixed data = binomial qpoints = 15; *Mixed logistic model with one random effect, no interaction;
parms alpha = 1 beta = 1 su = 0.2; *Initial values for parameters estimates;
bounds su>=0;
z=alpha+beta*treat+u; *Logistic formula;
expz=exp(z);
pi=expz/(1+expz);
model y~binomial(n,pi);
random u~ normal(0,su*su) subject=center;
ods output FitStatistics=test1;
ods listing select test1;
run;
proc nlmixed data=binomial qpoints = 15; *Mixed logistic model with two random effects, interaction;
parms alpha=1 beta = 1 su = 0.2 sb = 0.8; *Initial values;
bounds su>0; bounds sb>0;
z=alpha+beta*treat+a+b*treat; *Logistic formula;
expz=exp(z);
pi=expz/(1+expz);
model y~binomial(n,pi);
random a b~ normal([0,0],[su*su,0,sb*sb]) subject=center;
ods output FitStatistics=test2;
ods listing select test2;
run;
data mixed; *Calculating likelihood ratio statistic;
merge test1(rename=(value=d1)) test2(rename=(value=d2));
if descr='2 Log Likelihood';
run;
data combmix; set mixed;
delta=d1d2;
run;
data lr; set combmix;
x=probchi(delta,1); *Testing hypothesis of homogeneity of odds ratios, based on mixture chisquare;
run;
data rejectmixed;
set lr;
rej1 = 0.5*(1x);
a=(rej1<0.05);
run;
Competing interests
The authors declare that they have no competing interests.
Authors' contributions
ZB and PJ were responsible for the design, simulation, analysis and interpretation. SMTA supervised the study and interpreted the results. All authors read and approved the final manuscript.
Acknowledgements
The authors are grateful to Keivan Shalileh for his kind editing of this paper. We are also thankful to the referees for their invaluable comments.
References

Jones MP, O'Gorman TW, Lemke JH, Woolson RF: A Monte Carlo investigation of homogeneity tests of the odds ratio under various sample size configurations.
Biometrics 1989, 45:17181. PubMed Abstract  Publisher Full Text

O'Gorman TW, Woolson RF, Jones MP, Lemke JH: Statistical analysis of K 2 × 2 tables: A comparative study of estimators/test statistics for association and homogeneity.
Environ Health Perspect 1990, 87:103107. PubMed Abstract  PubMed Central Full Text

Paul SR, Donner A: A comparison of tests of homogeneity of odds ratios in K 2 × 2 tables.
Stat Med 1989, 8:14551468. PubMed Abstract  Publisher Full Text

Reis IM, Hirji KF, Afifi AA: Exact and asymptotic tests for homogeneity in several 2 × 2 tables.
Stat Med 1999, 18:893906. PubMed Abstract  Publisher Full Text

Gavaghana DJ, Moore RA, McQuay HJ: An evaluation of homogeneity tests in metaanalyses in pain using simulations of individual patient data.
Pain 2000, 85:415424. PubMed Abstract  Publisher Full Text

Paul SR, Donner A: Small sample performance of tests of homogeneity of odds ratios in K 2 × 2 tables.
Stat Med 1992, 11:159165. PubMed Abstract  Publisher Full Text

Mittlböck M, Heinzl H: A simulation study comparing properties of heterogeneity measures in metaanalyses.
Stat Med 2006, 25:43214333. PubMed Abstract  Publisher Full Text

Agresti A, Hartzel J: Strategies for comparing treatments on a binary response with multicentre data.
Stat Med 2000, 19:11151139. PubMed Abstract  Publisher Full Text

Breslow N, Day NE: Calssical methods of analysis of grouped data. In Statistical Methods in Cancer Research. The Analysis of CaseControl Studies. Volume 1. Lyon: International Agency for Research on Cancer; 1980::122159.

DerSimonian R, Laird N: Metaanalysis in clinical trials.
Control Clin Trials 1986, 7:177188. PubMed Abstract  Publisher Full Text

Verbeke G, Molenberghs G: The use of score tests for inference on variance components.
Biometrics 2003, 59:254262. PubMed Abstract  Publisher Full Text

Molenberghs G, Verbeke G: Likelihood ratio, score, and Wald tests in a constrained parameter space.
Am Statistician 2007, 61:16. Publisher Full Text

Jafari P, Ayatollahi SMT, Behboodian J: Sequential boundaries approach in clinical trials with unequal allocation ratios.
BMC Med Res Methodol 2006, 6:1. PubMed Abstract  BioMed Central Full Text  PubMed Central Full Text

Torgerson DJ, Campbell MK: Use of unequal randomisation to aid the economic efficiency of clinical trials.
BMJ 2000, 321:759. PubMed Abstract  Publisher Full Text  PubMed Central Full Text

Torgerson DJ, Campbell MK: Unequal randomisation can improve economic efficiency of clinical trials.
J Health Serv Res Policy 1997, 2:8185. PubMed Abstract

Gao S: Combining binomial data using the logistic normal model.
J Stat Comput 2004, 74:293306. Publisher Full Text

Hardy RJ, Thompson SG: Detecting and describing heterogeneity in metaanalysis.
Stat Med 1998, 17:841856. PubMed Abstract  Publisher Full Text

HuedoMedina TB, SanchezMeca J, MarinMartinez F: Assessing heterogeneity in metaanalysis: Q statistic or I^{2 }index?
Psychol Methods 2006, 11:193206. PubMed Abstract  Publisher Full Text

Viechtbauer W: Hypothesis tests for population heterogeneity in metaanalysis.
Br J Math Stat Psychol 2007, 60:2960. PubMed Abstract  Publisher Full Text

Biggerstaff BJ, Jackson D: The exact distribution of Cochran's heterogeneity statistic in oneway random effects metaanalysis.
Stat Med 2008, 27:60936110. PubMed Abstract  Publisher Full Text
Prepublication history
The prepublication history for this paper can be accessed here: