Abstract
Background
Estimates of relative efficacy between alternative treatments are crucial for decision making in health care. When sufficient head to head evidence is not available Bayesian mixed treatment comparison models provide a powerful methodology to obtain such estimates. While models can be fit to a broad range of efficacy measures, this paper illustrates the advantages of using continuous outcome measures compared to binary outcome measures.
Methods
Using a case study in rheumatoid arthritis a Bayesian mixed treatment comparison model is fit to estimate the relative efficacy of five antiTNF agents currently licensed in Europe. The model is fit for the continuous HAQ improvement outcome measure and a binary version thereof as well as for the binary ACR response measure and the underlying continuous effect. Results are compared regarding their power to detect differences between treatments.
Results
Sixteen randomized controlled trials were included for the analysis. For both analyses, based on the HAQ improvement as well as based on the ACR response, differences between treatments detected by the binary outcome measures are subsets of the differences detected by the underlying continuous effects.
Conclusions
The information lost when transforming continuous data into a binary response measure translates into a loss of power to detect differences between treatments in mixed treatment comparison models. Binary outcome measures are therefore less sensitive to change than continuous measures. Furthermore the choice of cutoff point to construct the binary measure also impacts the relative efficacy estimates.
Keywords:
Bayesian mixed treatment comparison models; Rheumatoid arthritis; AntiTNF agentsBackground
Metaanalysis has developed to be a widely used tool to combine trials evaluating the same intervention. This allows more powerful conclusions when a single study group is too small, and it fits naturally in the Bayesian framework where the inclusion of all available evidence into an analysis is anticipated [1]. A natural extension of a metaanalysis is a mixed treatment comparison (MTC) where more than two treatments are compared in a network of evidence. While direct evidence is preferable it is often not available due to ethical or financial reasons and MTC models provide a powerful tool to estimate relative efficacy among treatments which are not directly compared in a trial. A Bayesian approach to indirect comparison allows the inclusion of a wide range of evidence and is flexible to deal with increasingly complex evidence structures [2].
The variance of an indirect comparison is typically larger than the variance of a comparable direct comparison; the variance increases with every indirect link in the chain. It is therefore crucial to make the greatest use of available data.
Using a case study in rheumatoid arthritis this paper presents a MTC model on a range of efficacy measures illustrating the increased power of continuous outcome measures in such models compared to using binary measures. For the presentation of the analysis we follow the checklist suggested by Spiegelhalter et al. [3].
Dichotomized outcome measures
Dichotomized outcome measures are widespread in medical research. The perceived advantage of simplicity in the interpretation comes at a cost, however [4]. The loss of information results in a loss of power to detect relationships, furthermore the type I error rate may be inflated [5] and there is a risk of underestimating the variance parameter. These issues have been discussed in the literature relating to a number statistical analyses [69].
In this work we demonstrate the consequences of dichotomized outcome measures in the context of MTC modelling. It has been pointed out that dichotomization can be used when designing trials to quantify a treatment effect [10,11]. This is the primary aim of many clinical trials aiming for license approval. The anticipated efficacy level is taken as a cutoff point and results allow a straightforward interpretation. Pharmacoeconomic assessments of healthcare interventions are now a formal component of decision making in many countries [12]. Evidence syntheses typically rely on published clinical trials to estimate the relative efficacy among alternative agents to inform decision making. This has created an additional use of clinical trials demonstrating treatment efficacy, for which dichotomized measures suffer from disadvantages compared to the underlying continuous effect measure. The loss of power due to dichotomization adds to the increased variance in indirect comparisons. This is especially problematic for MTCs (vs. pairwise metaanalysis), since they typically have large standard errors due to the indirect nature of the comparison.
The intervention
Rheumatoid arthritis (RA) is a chronic, progressive and disabling autoimmune disease, causing swelling and damaging cartilage and bone around the joints. Any joint may be affected but it is commonly the hands, feet and wrists. Common symptoms are joint swelling, pain, morning joint stiffness, poor sleep, fatigue and weight loss [13].
Over the past decade, enhanced understanding of the molecular pathogenesis has led to the development of biologic agents that target specific parts of the immune system. These innovative treatments have altered the path and face of RA and outcomes for patients and society. Tumour necrosis factor alpha antagonists (antiTNFα) are the first of the biologic treatment groups used in RA. There are currently five antiTNF agents licensed for RA in Europe; adalimumab, certolizumab, etanercept, golimumab and infliximab. All of these agents have demonstrated considerable efficacy in placebo controlled randomized controlled trials (RCTs) in patients who have had an inadequate response to conventional Disease Modifying AntiRheumatic Drugs (DMARDs) such as methotrexate (MTX) or sulphasalazine.
While there is a wealth of RCT evidence available for these agents compared to either placebo or conventional DMARDs, there are currently very limited headtohead RCTs of antiTNF agents. Despite this, some estimate of relative efficacy in order to inform choice of agent is needed. In the absence of headtohead trials of relevant comparators, it is necessary to combine evidence from placebo controlled trials of different treatments and thereby derive an estimate of effect of one treatment against another. This can be broadly termed as mixed treatment comparison (MTC), an extension of metaanalysis. Different methodologies have been described for MTC; one such method uses Bayesian hierarchical models. Such models provide more flexibility than classical methods to include more data and handle more complex modelling structures [14].
Aim of the analysis
The aim of this analysis is to demonstrate the advantage of using continuous measures in MTC models compared to using binary measures. In the present analysis a MTC model is fitted to estimate all pairwise comparisons among the five TNFαinhibitors and placebo for a range of outcome measures.
Nixon et al. [15] have developed a MTC model to fit the binomial American College of Rheumatology (ACR) outcome measure; it is possible to include trials with multiple treatment arms and to adjust for study level covariates. Jansen et al. [16] have presented a MTC model on the outcome of the Health Assessment Questionnaire (HAQ), which is measured on a continuous scale. However, they did not allow for multiple treatment arms or the inclusion of baseline characteristics. The present analysis extends Jansen’s methods to allow for these.
While binary measures are useful for demonstrating a certain level of efficacy in clinical trials, we will show that the loss of information when changing from the underlying continuous scale to the binary outcome measure results in a loss of power to detect differences between treatments in MTC analyses.
Fitting models for a continuous improvement measure and a discretised version thereof as well as for binary outcome measures and a continuous version of these illustrates the enhanced power to detect differences of continuous measures compared to binary measures.
The chosen outcome measures are based on the ACR criteria and the HAQ score. Details are described in the next section.
Efficacy Measures
In order to estimate the relative efficacy between treatments one has to decide on a measure of disease activity and improvement. Commonly used measures in RA are the ACR criteria, the Disease Activity Score (DAS) and the HAQ score. Table 1 summarises the different measures of improvement. While other measures exist, they are outside the scope of this paper and are discussed elsewhere [17].
Table 1. Measuring improvement in RA: ACR (American College of Rheumatology); HAQ (Health Assessment Questionnaire); DAS28 (Disease Activity Score)
The ACR response criteria is a binary combination measure including the number of tender and swollen joints, patient’s global assessment, physician’s global assessment, pain, degree of disability and level of acutephase reactant. In order to achieve an ACR 20, ACR 50 or ACR 70 result, an improvement of 20%, 50% or 70% respectively is required in the swollen and tender joint counts as well as in 3 of the 5 remaining dimensions [18,19].
Continuous measures based on the ACR criteria have been introduced including the meanACR and the ACR hybrid measure [20]. The meanACR measures the mean % improvement in the seven ACR core set measures. The ACR hybrid measure combines the ACR 20, ACR 50 and ACR 70 with the ACRmean. A patient’s ACR hybrid outcome is the same as the ACRmean, but restricted by his binary ACR response. For example, the outcome for a patient who is an ACR 20 responder, but not an ACR 50 responder is restricted to the interval [0.2, 0.5).
The HAQ score represents the result of a selfreport questionnaire in which patients rate their ability to perform daily life activities such as washing one’s hair or getting in and out of a car. Values range from 0 to 3 in steps of 0.125, where high values indicate a more severe disease status. The improvement in HAQ score is measured on a continuous scale.
For the purpose of demonstrating the enhanced power of continuous measures in MTC models we have defined a discretised version of the HAQ score: HAQ 20 and HAQ 50. They are defined analogously to ACR 20 and ACR 50. A patient achieves a HAQ 20 outcome, if his HAQ score has improved by at least 20%; a 50% improvement is required for HAQ 50.
The DAS28 score is a combination measure on a continuous scale from 0 to 10 incorporating swollen joint count (swollen28), tender joint count (tender28) (out of 28 defined joints), an evaluation of the patients general health (GH) and the erythrocyte sedimentation rate (ESR). The score is obtained using the formula [21]:
Methods
Data
A systematic literature review following the PRISMA method [22] was performed to identify trials meeting our inclusion criteria. The search included published studies up to and including October 2010 in PubMed, Embase and the Cochrane Database. Rheumatological inflammatory diseases other than RA, such as ankylosing spondylitis, psoriatic arthritis and connective tissue diseases were excluded from the search. The inclusion criteria were randomised controlled trials (RCTs), patients with established RA who have had an inadequate response to methotrexate (MTX) and who have been treated for at least 24 weeks (where 24 week data were not available, data within 6 weeks either before or after 24 weeks were used). Both monotherapy and combination therapy were included with an explicit term in the statistical model allowing for the additional effect of MTX. More details on the selection process can be found elsewhere [23].
The outcome measures chosen were those described in the previous section based on the ACR criteria, the HAQ score and the DAS28. Unfortunately the DAS28 was reported in too few trials to fit a MTC model (DAS28 was only reported in 6 trials, not representing each of the treatments). The total number of responders achieving ACR 20, ACR 50 and ACR 70 response and the mean improvement and standard deviations (SDs) for the continuous HAQ measure were extracted. Authors were contacted in cases where the required data were not reported. Where no access to the missing data was provided, the following methodology was applied: in cases where the mean was not reported, the median was used; in the absence of SDs, interquartile ranges (IQRs) were used to estimate SDs using a normal approximation, and, in the remaining cases, the maximum of clinical trial SDs was used. The doses of biological agents included are those included in the RCTs. Demographic data including age, gender, mean disease duration, baseline HAQ score and number of previous DMARDs were recorded.
Statistical Model
A Bayesian MTC model is fitted to the data for each of the outcome measures of interest (HAQ, HAQ 20, HAQ 50, ACR 20, ACR 50, ACR 70 and ACRcont). Such models simultaneously perform indirect comparisons between treatments that are not directly compared and allow estimation of all pairwise comparisons. A network diagram represents the underlying evidence structure of such models. Figure 1 shows the generalisation of metaanalysis to the simplest case of an indirect comparison. Network diagrams consist of nodes representing the interventions included in the analysis and edges between nodes identify interventions which are directly compared in one or more trials. Dotted lines are used to indicate the indirect comparison of interest. For simplicity, these are often omitted when all comparisons for which no direct evidence is available is of interest.
Figure 1. Generalisation from metaanalysis to mixed treatment comparison.
In a simple metaanalysis evidence from a range of trials comparing the same two interventions A and B is combined yielding an overall A vs. B estimate (see evidence network in Figure 1(a)). The underlying methodology is well explored and used extensively in practice. The simplest case of a MTC occurs when this situation is extended to include a third intervention, drug C, which has also been compared to drug A in clinical trials (see Figure 1(b)). MTC modelling assumes that the combined evidence from A vs. B and A vs. C trials contains some information about the relative efficacy of B vs. C; in particular B vs. C is assumed to be the difference between A vs. B and A vs. C (see Figure 1(c)). MTC models can be fitted for all underlying evidence structures as long as the network diagram is connected. Combining evidence this way evokes the assumption of treatment exchangeability, a similarity assumption among the trials. The effect of B in the A vs. B trials is assumed to occur if C was replaced by B in the A vs. C trials. Unfortunately this assumption can only be tested in networks of closed loop design, where consistency measures can be obtained; methods are described elsewhere [24]. When this is not the case, great care has to be taken in selecting the trials for the analysis.
All models can be fitted in a Bayesian or in a classical framework. The Bayesian approach to MTC modelling is the most flexible, allowing the inclusion of a wide range of data and borrowing strength across the network ensures the optimal use of the data. Details on classical methods can be found elsewhere [25,26]. This analysis focuses on the Bayesian approach only.
In the remainder of this section the mathematical models for binary and continuous outcome measures are described beginning with simple metaanalytic models which are then extended to MTC models including baseline characteristics and other assumptions necessary for the RA model.
Simple Meta Analysis
In a simple meta analysis only two interventions are compared; evidence of a number of trials is combined to get an overall estimate for the difference in effect. The continuous mathematical model is the following [16]:
Δ_{i} is the observed relative treatment effect in study i; σ_{i}the associated sampling error; i=1,…N, where N is the number of trials included in the analysis. The model assumes random effects,
meaning the study specific true effects δ_{i}are drawn from a normal distribution with mean d and between trial variance parameter
The binary counterpart to calculate odds ratios can be formulated as follows:
A binomial likelihood is assumed for the number of patients and number of responders in control arm and treatment arm of each study; n.c_{i}, n.t_{i}, r.c_{i}, r.t_{i}respectively. The model calculates log odds ratios (LORs) for each study, δ_{i}, for which random effects are assumed yielding an overall log odds ratio estimate d and a between trial standard deviation σ_{δ}. Prior distributions need to be assigned to d, the logits in the control group μ_{i} and σ_{δ}.
Figure 2 shows the directed acyclic graph (DAG) for (a) the continuous and (b) the binary metaanalysis model.
Figure 2. DAG for (a) continuous and (b) binary meta analysis model.
Extension to MTC
The models described in (2) and (3) for a simple meta analysis can be extended to a MTC model, which allows the estimation of relative efficacy among more than two interventions [14,16]. Mathematically this means the continuous model in equation (2) changes to:
As before Δ_{i}refers to the observed relative treatment effects of the comparison in study i with respective measure of variability σ_{i}. k and l indicate the antiTNF agent evaluated in the trials; 1 indicates adalimumab, 2 infliximab, 3 etanercept, 4 golimumab and 5 certolizumab. The baseline treatment is placebo, indicated by P. Assuming random effects, the study specific true effects are drawn from a normal distribution with a mean specific to the comparison; i.e. the effects of all trials comparing the same two interventions are drawn from the same normal distribution. The between trial variance parameter is assumed to be the same for all comparisons. Each comparison of drug k versus drug l, IC_{k,l}, can be written in terms of basic parameters d_{k,P}. Prior distributions are required for basic parameters and between trial standard deviation σ_{δ}.
For the binary outcome measures the model in equation (3) extends to:
Again n.c_{i}, n.t_{i}, r.c_{i} and r.t_{i} refer to the number of patients and number of responders in the control and treatment arm of each study. The model calculates LORs δ_{i,k,P}for each study, which are combined assuming random effects with a common mean for each baseline comparison d_{k,P}. The between trial SD is assumed to be the same for all drugs. All other comparisons IC_{k,l}can be estimated from the baseline parameters. Prior distributions need to be defined for d_{k,P}, μ_{i} and σ_{δ}.
Further extensions
For the use of the model for RA data three more adjustments have to be made. We want to allow for more than one treatment arm per study; adjustments have to be made for the concurrent treatment with MTX. Furthermore, we would like to model the improvement relative to baseline disease activity, since disease activity at baseline influences the effectiveness of the intervention [27]. Also, the results of this analysis provide the basis for an economic analysis, which requires the percentage improvement in HAQ score as input parameters.
A step by step approach is taken to include these assumptions into the model described in (4) for the continuous case and (5) for the binomial case.
As a first step the allowance for multiple treatment arms is included. Let index j refer to treatment arm j (j=1,…,M, where M is the total number of treatment arms in the analysis) and s(j) refers to the study of treatment arm j. The continuous mathematical model then takes the following form:
For each treatment arm the observed effect relative to the comparator arm is calculated as Δ_{j}, which as before has a normal distribution with measure of variability σ_{j}. The model makes the assumptions that effects in treatment arms of the same study are constant, we therefore assume fixed effects for within study effects. This assumption is described in line 3 of equation (6). Between trials we assume random effects, as before and each comparison IC_{k,l} can be written in terms of basic parameters d_{k,P}. Prior distribution need to be defined for basic parameters and between trial standard deviation σ_{δ}.
This model takes the following form for binary outcome measures:
Changes from equation (5) to (7) are analogous to changes in the continuous case. A LOR δ_{j,k,P} is obtained for each treatment arm; treatment arms of the same study are assumed to estimate the same treatment effect, hence line 4 in equation (7). Everything else remains the same.
In a next step we will extend the model to allow for the concurrent treatment with MTX. The effect due to MTX is assumed to be additive, meaning the relative effect between two arms where no MTX is given is the same as the relative effect between two arms where MTX is given in both arms:
The continuous model takes the following form:
The difference to the previous model described in equation (6) is the splitting of treatment effect δ_{j,k,P}into a part representing the effect due to the drug of interest (in the RA case the antiTNF effect) α_{s(j),k,P}and a part explaining the effect which is due to the concurrent treatment with MTX, β_{s(j)}. I.t_{i} and I.c_{s(i)} are indicator variables indicating whether MTX was given in treatment and comparator group. By multiplying the β_{s(i)} with the difference of the indicator the assumptions of an additive effect as explained above is implemented. In this way, only treatment arms, where MTX is given in either the treatment arm or in the respective control arm inform parameter β_{s(j)}. As previously the effect in arms of the same study is assumed constant, hence β_{j}=β_{s(j)} and α_{j,k,P}=α_{s(j),k,P} (this line has been omitted in the equation for simplicity; the effect is incorporated in line 3 of equation 9). Between trials random effects are assumed for α_{s(j),k,P} and fixed effects for β_{s(j)}. One could just as well assume random effects for β_{s(j)}; but the RA data does not provide sufficient evidence to inform a between trial variability for b. All comparisons IC_{k,l} can be expressed in terms of basic parameters a_{k,P}. Prior distributions need to be specified for the basic parameters, for b and for the between trial standard deviation σ_{α}. The binary version of this model is mathematically described as follows:
Again, changes follow analogously to changes made in the continuous case. In a last step we want to extend the model such that the drug effect α_{s(j),k,P}depends on baseline disease activity. To model relative percentage improvement α is modelled as a multiplier to the HAQ score at baseline λ_{i}. All other aspects of model (9) remain unchanged.
The binomial measures selected for this study are by definition relative to baseline disease activity, since the cutoff between case and no case is a % improvement relative to baseline. It is therefore not necessary to adjust the model defined in equation 10 further. For completeness, we will introduce the concept of metaregression, which allows for the adjustment for baseline characteristics in the analysis. The resulting model is:
γ refers to the regression parameter for the baseline HAQ score.
Models described in equations 10 and 11 enable the estimation of relative efficacy of a number of treatments while allowing for multiple treatment arms, concurrent treatment with additional medication and dependence of treatment effect on baseline disease activity for both continuous and binary outcome measures. In the continuous case differences in efficacy are estimated; odds ratios are estimated in the binary case. Using methods by Warn et al. [28] the model can be modified to calculate other measures, such as risk ratios.
This model has been developed by Nixon et al. [15] for the binary case. Figure 3 shows the DAG for the final MTC models described by equations (11).
Figure 3. DAG for continuous MTC model in equation (11). Square shaped nodes represent constants, oval shaped nodes are either random or logical. A random dependency between two nodes can be found where a single arrow connects both nodes, a double arrow shows that there is a logical relationship. Loops are represented by boxes.
Prior distributions
The models described above require prior distributions for the baseline treatment effect, for the effect due to MTX, for the between trial standard deviation and in the binomial models for the log odds of response in the control group and the metaregression parameter. For the analysis we have chosen vague priors for all of these as described below. The prior distribution in a Bayesian analysis represents the knowledge about the parameter prior to the analysis; in the absence of prior knowledge vague prior distributions intend to cover a wide area of plausible values, such that little information is entered into the analysis. For the treatment effect parameters d in model (2) and (3), d_{k,P} in models (4)(7), for a_{k,P}and b in (9)  (12), for the log odds in the control group of the binomial models μ_{i} as well as for γwe have chosen a normal distribution centred at no treatment effect 0 with a very large variance:
The same prior can be used for both continuous and binary models, since the risk difference and the log odds ratios take values on the real line where 0 represents no difference. The priors do not favour any of the drugs and allows for a wide range of likely values in favour for each. This is a widely used vague prior on treatment effects.
A uniform distribution was chosen for the between trial standard deviation parameter σ_{δ/α}, as proposed by Gelman [29].
To test whether the range of the uniform prior was chosen appropriately the analysis has been rerun using wider ranges (dunif(0,5) and dunif(0,7)) and estimates did not change.
Computation/ Software
The MTC models were fitted in WinBUGs, a MCMC software using Gibbs sampling [30]. The DAG in Figure 3 is drawn in WinBUGs. Square shaped nodes represent constants; oval shaped nodes are either random or logical. A random dependency between two nodes can be found where a single arrow connects both nodes, a double arrow shows that there is a logical relationship. Loops are represented by boxes. The complete code and input data is provided in Additional file 1. Computational feasability allowed for a large number of iterations. Each model discarded 50,000 burnin iterations and was run with 100,000 iterations and two chains. Convergence was analysed using CODA; the effective sample size was checked, and visual inspection of the autocorrelation and the chains confirmed convergence.
Additional file 1. Bugs code. Additional file providing the WinBUGs code and input data for the MTC model.
Format: DOCX Size: 18KB Download file
Results and discussion
Data
The systematic literature review identified sixteen RCTs meeting our inclusion criteria. Figure 4 shows the network of available evidence. Table 2 summarises the data extracted from the trials, baseline demographics can be found in Additional file 2.
Additional file 2. Baseline Demographics. Mean Baseline Demographics of randomized controlled trials for antiTNF agents.
Format: DOCX Size: 13KB Download file
Figure 4. Network diagram for RA analysis. Edges are labelled with the number of studies and the total number of patients included in these studies. Numbers in square brackets refer to HAQ evidences where this differs from ACR evidence.
Table 2. Trial Data: Number of patients N; improvement in HAQ score Δ HAQ; number of ACR20, ACR50, ACR70 responders; HAQ score at baseline HAQ_{base}; + indicates additional treatment with MTX
The MTC models described previously are fit to the data; the continuous model for the improvement in HAQ score and the binary model for the ACR 20, ACR 50 and ACR 70 outcome measures.
The aim of this paper is to demonstrate the increased power of continuous outcome measures to detect differences in MTC models compared to binary measures. To explore this hypothesis the HAQ score was discretised to a binary outcome measure and the ACR criteria were transformed into a continuous measure; the resulting measures were analysed in addition to the trial reported outcome measures and results compared. The rationale for doing this was that comparing HAQ and ACR outcomes is not comparing like with like; they measure quite different aspects of disease and additional significant findings using the HAQ score could be because of this rather than there being any effect of a continuous vs. a binary outcome measure. Estimating a discrete version of the HAQ and a continuous measure of the ACR allows us to compare binary and continuous outcomes while the measure is kept fixed.
Discretised HAQ: HAQ 20 and HAQ 50
For the purpose of comparing continuous and binary outcome measures based on the same data, the HAQ 20 and HAQ 50 measures defined previously are estimated based on the continuous HAQ improvement. These discretised HAQ outcomes are not validated outcome measures for RA and are therefore not reported in the trials.
The trials report mean and standard deviation of HAQ improvement. The number of HAQ 20 and HAQ 50 responders in each trial arm was estimated as follows. For each trial arm, the HAQ improvement of 1000 patients was generated by reference to normal distribution with mean and SD given from the data in that group. From these patients the proportion of HAQ 20 and HAQ 50 responders was calculated and applied to the number of patients in each trial arm. Data is summarised in Table 3.
Table 3. Input Data for discretised HAQ score and continuous ACR: Number of patients N; number of HAQ20 and HAQ50 responders; continuous ACR response ACRcont; + indicates additional treatment with MTX
Continuous ACR: ACRcont
While the continuous ACR measures were proposed by the Americal College of Rheumatology [20], they were not assessed in any of the trials. Therefore a continuous ACR measure ACRcont is generated based on the ACR 20, ACR 50 and ACR 70 outcomes reported in the trials. This enables us to compare the outcomes of the binary measures with those resulting from the continuous scale. ACRcont takes a value between 0 and 1 representing the percentage improvement in the dimensions which are combined for the binary ACR criteria.
The trial data allows the categorisation of patients into nonresponders (group 1) (patients not achieving an ACR 20 response), patients achieving ACR 20 but not ACR 50 response (group 2), patients achieving ACR 50 but not ACR 70 response (group 3) and ACR 70 responders (group 4). Assuming a mean ACRcont response m_{i} for each group of patients these numbers are used to generate the mean and SD of ACRcont for each trial arm:
where N_{i} refers to the number of patients in each group and
Patient level data from a RA cohort allowed the estimation of m_{i} for the four groups [47]. Mean ACRcont response can be based on either continuous ACR measure described previously, depending upon which underlying measure we are trying to recreate. For the main analysis we are using the meanACR measure; results based on the ACRhybrid are discussed in the sensitivity analysis. Table 4 summarizes the mean ACRcont responses for the four groups based on both outcome measures. Improvement in the core sets is measured by a drop in score; therefore overall improvement is restricted by an upper bound of 100%. To achieve a symmetric measuring scale overall worsening has also been restricted to 100% in each of the criteria, as proposed elsewhere [20].
Table 4. Mean improvement estimated from patient level data
The patient level estimates are based on patients receiving biologic treatment; estimates can therefore be applied to the treatment arms of the studies. Unfortunately no patient level data representing the control groups is available to us. Given the lack of further information the analysis assumes interval midpoints as mean ACRcont estimates for the control groups. Group 1 is assumed to have a response in [1, 0.2), group 2 in [0.2, 0.5), group 3 in [0.5, 0.7) and the response of group 4 lies in [0.7, 1]. The interval for non responders is very broad covering a wide range of extreme values. The baseline analysis therefore assumes a conservative mean value of 0.00 for this group. This yields the control group estimates in Table 4. Alternative scenarios assuming the same mean response for the control group as estimated for the treatment group are analysed in the sensitivity analysis. The estimated input data for the continuous ACR model based on meanACR is summarised in Table 3.
Efficacy Results
The models calculate the relative efficacy among all antiTNF agents and placebo. Relative efficacy for the continuous measures (HAQ improvement and ACRcont) was measured as difference in improvement. Relative improvement was modelled as a multiplier to baseline HAQ score. Odds ratios were calculated for the binomial measures ACR 20, ACR 50, ACR 70 and HAQ 20 and HAQ50. Thirteen trials were included for the HAQ analysis and sixteen for the ACR analysis.
The MTC results are summarised in forest plots; in Figure 5 for the HAQ measures and in Figure 6 for the ACR measures. Binomial results are plotted on the log scale. Plots show mean and 80% credible intervals to summarize the posterior distribution for the relative efficay between antiTNF agents; significant differences on the 80% level are marked in red. The 80% level was chosen to shift away from the misunderstood acceptancerejection dichotomy of random signifiance levels [23]. For additional information, Tables 5 and 6 provide the exact means and interval endpoints; significant differences on the 80% level are marked with a “∗”.
Figure 5. Forest Plot of HAQ results. Plots pairwise LORs for HAQ 20 and HAQ 50 outcome and improvement differences for % improvement in HAQ score of antiTNF agents against placebo and one another. Red colour indicates significant differences at the 80% level.
Figure 6. Forest Plot of ACR results. Plots pairwise LORs for ACR 20, ACR 50 and ACR 70 outcome and improvement differences for ACRcont of antiTNF agents against placebo and one another. Red colour indicates significant differences at the 80% level.
Table 5. Results from HAQ analysis: Mean estimate with 80% credible intervals for each pairwise comparison
Table 6. Results from ACR analysis: Mean estimate with 80% credible intervals for each pairwise comparison
The key point is that all differences between agents detected by the binary HAQ measures are also detected by the continuous HAQ improvement measure, while not all differences detected in the continuous HAQ model are detected using the binary HAQ data; the differences detected by the binary HAQ measures are subsets of the differences detected by the continuous HAQ improvement measure. This is also demonstrated in the ACR analysis. It illustrates the point made that a continuous measure of effect has a greater power to detect a difference between treatments in an evidence synthesis.
Conclusion
This analysis illustrates the enhanced sensitivity to change of continuous measures in MTC models compared to using binary outcome measures, which was recently highlighted for epidemiological studies [48]. While binary measures work well for demonstrating a certain level of response, which is the primary aim of many clinical trials, the information lost when categorizing the underlying continuous response can have significant impact on the results of a mixed treatment comparison.
MTC models are often utilized when faced with a choice of agents rather than for demonstrating efficacy. In the above application, for example, it is already known that all of the antiTNF agents provide improvement in the treatment of RA. What remains is to determine whether some of these agents work better than others. Therefore it would be of interest to know whether an agent provides a 30% improvement or a 60% improvement. When looking at the ACR 20 outcome, no difference would be seen between the two response rates.
The above analyses have shown two things. Firstly, the enhanced power to detect differences of a continuous measure as opposed to a binary measure was shown for both, the HAQ and the ACR measures. Secondly, the choice of cutoff level (e.g. 20%, 50% or 70%) has shown to have a strong impact on the results. Different significant results were detected when choosing different cutoffs; even the sequence of treatments when ordering from best to worse changes.
It may be argued that binary outcomes are clear and easy to interpret. However, where binary outcome measures are required, results based on continuous measures can be transformed subsequently using cutoff points, [49].
From a clinical point of view these results show that a MTC model based on continuous outcome measures provides greater precision of estimates of efficacy. This is of great benefit when carrying out an economic evaluation.
Sensitivity Analysis
The trials included for the analysis were conducted over a time period of 10 years. The model was extended to a metaregression to include potential confounding parameters such as duration of disease, number of previous DMARDs and year of publication. None of these were found to have a significant impact and therefore were not included in the main analysis.
Different prior distributions on the between trial variance parameter allowing for a wider range of values were tested, but did not alter the conclusions. Varying the precision parameter of the normal prior distributions for the remaining parameters between 1,000 and 100,000 did not alter the results either.
The continuous ACR response underlying the binary ACR 20, ACR 50 and ACR 70 measures is estimated using the mean ACRcont response for each of the four responder groups. For the baseline analysis meanACR was used as the underlying continuous effect. In a sensitivity analysis a model was fit using the ACR hybrid as the underlying effect. This did not alter the results much, however, adalimumab was not found to be superior to infliximab in this scenario. There was substantial uncertainty regarding the mean ACRcont response in the control arms, as no patient level data was available. An additional model has been fit assuming the mean response in each group to be the same for treatment arms and control arms. While this did influence the biologic treatment effect versus control, the relative efficacy between biologic treatments was not affected much. Again, these models did not detect a difference between adalimumab and infliximab. Outcomes of the various MTCs can be accessed in Additional file 3.
Additional file 3. Sensitivity Analysis (1). Additional file providing outcomes of the sensitivity analysis conducted on the continuous ACR measure.
Format: DOCX Size: 17KB Download file
The scale chosen for dichotomized outcome measures has been shown to influence outcomes of MTC models [50]. In addition to the OR scale we therefore also fitted an analysis based on risk ratios (RRs). The OR model can be adapted to estimate RRs making a few changes, for further details see Schmitz et al. [23]. The results can be accessed in Additional file 4; the findings of increased power of continuous outcomes are confirmed.
Additional file 4. Sensitivity Analysis (2). Additional file providing outcomes of the sensitivity analysis conducted on the risk ratio scale.
Format: DOCX Size: 19KB Download file
Limitation
Doses across treatments and treatment arms varied, which has not been accounted for in the analysis. Generalising the model to include a metaregression for dose would raise difficulties of comparability; doses across treatments are hard to compare as well as within one treatment when the same dose is given, but with different frequency.
The analysis only includes RCT evidence. However, there exists a large body of observational data collected via registries and open label studies. The inclusion of such evidence is possible within this framework, helping to reduce uncertainty further [51].
Patient level data allowed the estimation of mean ACR response within each of the responder groups. While this provided some additional information towards estimating the continuous ACR response measure, it did not replace the full information lost. The estimated ACRcont measure is still subject to high uncertainty. Patient level data of the RCTs included in the analysis would allow the exact calculation of the continuous ACR measure.
The evidence network has a star design, which does not allow testing of the assumption of treatment exchangeability. Baseline characteristics across the trials were compared and and the model was extended to a metaregression to explain the potential influence of differing demographics.
This paper confirms the effectiveness of antiTNF agents in the treatment of RA. The study illustrates the enhanced ability to detect differences between treatments when using continuous measures and shows the high dependency of MTC outcomes on the cutoff level when using binary data.
Rather than steering away from binary measures, we would like to advocate the additional reporting of underlying continuous effect measures in clinical trials to facilitate further analyses. The importance of the involvement of statisticians in the choice of clinical measures has been recently highlighted [10].
Competing interests
The authors declare that they have no competing interests.
Authors’ contributions
SS is responsible for the study concept and design, designed and implemented the MTC models and participated in the systematic review and data extraction. RR is responsible for the study concept and design and carried out the systematic review and data extraction. CW oversaw the design and implementation of the MTC models. All authors contributed to the interpretation of findings. All authors read and approved the final manuscript.
References

Sutton A, Higgins JPT: Recent developments in metaanalysis.
Stat Med 2008, 27:625650. PubMed Abstract  Publisher Full Text

Lu G, Ades AE: Combination of direct and indirect evidence in mixed treatment comparisons.
Stat Med 2004, 23:31053124. PubMed Abstract  Publisher Full Text

Spiegelhalter DJ, Abrams KR, Jonathan PM: Bayesian Approaches to Clinical Trials and HealthCare Evaluation. Wiley: NewYork; 2004.

Royston P, Altman D, Sauerbrei W: Dichotomizing continuous predictors in multiple regression: a bad idea.
Stat Med 2006, 25:127141. PubMed Abstract  Publisher Full Text

Austin P, Brunner L: Inflation of the type I error rate when a continuous confounding variable is categorized in logistic regression analyses.
Stat Med 2004, 23:11591178. PubMed Abstract  Publisher Full Text

Chen H, Cohen P, Chen S: Biased odds ratios from dichotomization of age.

Sauerbrei W, Royston P, Zapien K: Detecting an interaction between treatment and a continuous covariate: A comparison of two approaches.
Comput Stat Data Anal 2007, 51:40544063. Publisher Full Text

Breitling L, Brenner H: Odd odds interactions introduced through dichotomisation of continuous outcomes.
J Epidemiol Community Health 2010, 64:300303. PubMed Abstract  Publisher Full Text

Dawson N, Weiss R: Dichotomizing continuous variables in statistical analysis: a practice to avoid.
Med Decision Making 2012, 32:225226. Publisher Full Text

Senn S, Julious S: Measurement in clinical trials: A neglected issue for statisticians?
Stat Med 2009, 28:31893209. PubMed Abstract  Publisher Full Text

Julious S, George S, Machin D, Stephens R: Sample sizes for randomized trials measuring quality of life in cancer patients.
Qual Life Res 1997, 6:109117. PubMed Abstract

Hutton J, McGrath C, Frybourg J, Tremblay M, BramleyHarker E, Henshall C: Framework for describing and classifying decisionmaking systems using technology asesessment to determine the reimbursement of health technologies (fourth hurdle systems).
Int J Technol Assess Health Care 2006, 22:1018. PubMed Abstract

Sutton A, Abrams KR: Bayesian methods in metaanalysis and evidence synthesis.
Stat Methods Med Res 2001, 10:277303. PubMed Abstract  Publisher Full Text

Nixon R, Bansback N, Brennan A: Using mixed treatment comparisons and metaregression to perform indirect comparisons to estimate the efficacy of biologic treatments in rheumatoid arthritis.
Stat Med 2007, 26:12371254. PubMed Abstract  Publisher Full Text

Jansen J, Crawford B, Bergman G, Stam W: Bayesian metaanalysis of multiple treatment comparisons: an introduction to mixed treatment comparisons.
Value in Health 2008, 11:956964. PubMed Abstract  Publisher Full Text

Saag K, Teng G, Patkar N, Anuntiyo J, Finney C, Curtis J, Paulus H, Mudano A, Piso M, Outman R, Allison J, Suarez Almazor M, Bridges A, Chatham W, Hochberg M, Maclea C, Mikuls T, Moreland L, O’Dell J, Turkiewicz A, Furst D, ElkinsMelton M: American College of Rheumatology 2008 Recommendations for the use of nonbiologic and biologic diseasemodifying antirheumatic drugs in rheumatoid arthritis.
Arthritis & Rheumatism 2008, 59:762784. PubMed Abstract  Publisher Full Text

Felson D, Anderson J, Boers M, Combarier C, Furst D, Goldsmith C, Katz L, Lightfoot R, Paulus H, Strand V, Tugwell P, Weinblatt M, Williams H, Wolfe F, Kieszak S: American College of Rheumatology preliminary definition of improvement in rheumatoid arthritis.
Arthritis & Rheumatism 1995, 38:727735. PubMed Abstract  Publisher Full Text

Van Riel P, van Gelstel A: Clinical outcome measures in rheumatoid arthritis.

American College of Rheumatology: A proposed revision to the ACR20: The hybrid measure of american college of rheumatology response.
Arthritis & Rheumatism 2007, 57:193202. PubMed Abstract  Publisher Full Text

Prevoo M, van’t Hof M, Kuper H, van Leeuwen M, van de Putte L, van Riel P: Modified disease activity scores that include twentyeightjoint counts. Development and validation in a prospective longitudinal study of patients with rheumatoid arthritis.
Arthritis & Rheumatism 1995, 38:4448. PubMed Abstract  Publisher Full Text

Moher D, Liberati A, Tetzlaff J, Altman D, PRISMA Group: Preferred reporting items for systematic reviews and metaanalyses: the PRISMA statement.
PLoS Medicine 2009, 6:e1000097. PubMed Abstract  Publisher Full Text  PubMed Central Full Text

Schmitz S, Adams R, Walsh C, Barry M, FitzGerald O: A mixed treatment comparison of the efficacy of antiTNF agents in rheumatoid arthritis for methotrexate nonresponders demonstrates differences between treatments: a Bayesian approach.
Ann rheumatic diseases 2012, 71:225230. Publisher Full Text

Dias S, Welton N, Caldwell D, Ades A: Checking consistency in mixed treatment comparison metaanalysis.
Stat Med 2010, 29:932944. PubMed Abstract  Publisher Full Text

Bucher HC, Guyatt GH, Griffith LE, Walter SD: The results of direct and indirect treatment comparisons in metaanalysis of randomized controlled trials.
J Clin Epidemiol 1997, 50:683691. PubMed Abstract  Publisher Full Text

Lumley T: Network metaanalyis for indirect treatment comparisons.
Stat Med 2002, 21:23132324. PubMed Abstract  Publisher Full Text

Hyrich K, Watson K, Silman A, Symmons D The BSR Biologics Register: Predictors of response to antiTNFα therapy among patients with rheumatoid arthritis: results from the British Society for Rheumatology Biologics Register.
Rheumatology 2006, 45:15581565. PubMed Abstract  Publisher Full Text

Warn D, Thompson D, Spiegelhalter D: Bayesian random effects metaanalysis of trials with binary outcomes: methods forthe absolute risk difference and relative risk scales.
Stat Med 2002, 21:16011623. PubMed Abstract  Publisher Full Text

Gelman A: Prior distributions for variance parameters in hierarchical models.

Lunn D, Thomas A, Best N, Spiegelhalter D: WinBUGS – a Bayesian modelling framework: concepts, structure, and extensibility.
Stat Comput 2000, 10:325337. Publisher Full Text

Weinblatt M, Keystone E, Furst D, Moreland L, Weisman M, Birbara C, Teoh L, Fischkoff S, Chartash E: Adalimumab, a fully human antitumour necrosis factor alpha monoclonal antibody, for the treatment of rheumatoid arthritis in patients taking concomitant methotrexate: the ARMADA trial.
Arthitis & Rheumatism 2003, 48:3545. PubMed Abstract  Publisher Full Text

Keystone E, Kavanaugh A, Sharp J, Tannenbaum H, Hua Y, Teoh L, Fischkoff S, Chartash E: Radiographic, clinical, and functional outcomes of treatment with adalimumab (a human antitumor necrosis factor monoclonal antibody) in patients with active rheumatoid arthritis receiving concomitant methotrexate therapy: A randomized, placebocontrolled, 52week trial.
Arthritis & Rheumatism 2004, 50:14001411. PubMed Abstract  Publisher Full Text

Van De Putte L, Atkins C, Malaise M, Sany J, Russell A, van Riel P, Settas L, Bijlsma J, Todesco A, Dougados M, Nash P, Emery P, Walter N, Kaul M, Fischkoff A, Kupper H: Efficacy and safety of adalimumab as monotherapy in patients with rheumatoid arthritis for whom previous disease modifying antirheumatic drug treatment has failed.
Ann Rheumatic Diseases 2004, 63:508516. Publisher Full Text

Miyasaka N: Clinical investigation in highly diseaseaffected rheumatoid arthritis patients in Japan with adalimumab applying standard and general evaluation: the CHANGE study.
Modern Rheumatology 2008, 18:252262. PubMed Abstract  Publisher Full Text  PubMed Central Full Text

Kim H, Lee K, Song Y, DaeHyun Y, Koh E, Yoo B, Luo A: A randomized, doubleblind, placebocontrolled, phase III study of the human antitumor necrosis factor antibody adalimumab administered as subcutaneous injections in Korean rheumatoid arthritis patients treated with methotrexate.
APLAR J Rheumatology 2007, 10:916. Publisher Full Text

Maini R, St Clair E, Breedveld F, Furst D, Kalden J, Weisman M, Smolen J, Emery P, Harriman G, Feldmann M, Lipsky P: Infliximab (chimeric antitumour necrosis factor [alpha] monoclonal antibody) versus placebo in rheumatoid arthritis patients receiving concomitant methotrexate: a randomised phase III trial.

Westhovens R, Yocum D, Han J, Berman A, Strusberg I, Geusens P, Rahman M: The safety of infl iximab, combined with background treatments, among patients with rheumatoid arthritis and various comorbidities: a large, randomized, placebocontrolled trial.
Arthritis & Rheumatism 2006, 54:10751086. PubMed Abstract  Publisher Full Text

Zhang F, Hou Y, Huang F, Wu D, Bao C, Ni L, Yao C: Infliximab versus placebo in rheumatoid arthritis patients receiving concomitant methotrexate: A preliminary study from China.
APLAR J Rheumatology 2006, 9:127130. Publisher Full Text

Schiff M, Keiserman M, Codding C, Songcharoen A, Berman A, Nayiager S, Saldate C, Li T, Aranda R, Becker J, Lin C, Cornet P, Dougados M: Efficacy and safety of abatacept or infliximab vs placebo in ATTEST: a phase III, multicentre, randomised, doubleblind, placebocontrolled study in patients with rheumatoid arthritis and an inadequate response to methotrexate.
Ann rheumatic diseases 2008, 67:10961103. Publisher Full Text

Moreland L, Schiff M, Baumgartner A, Tindall E, Fleischmann R, Bulpitt K, Weaver A, Keystone E, Furst D, Mease P, Ruderman E, Horwitz D, Arkfeld D, Garrison L, Burge D, Blosch C, Lange M, McDonnell N, Weinblatt M: Etanercept therapy in rheumatoid arthritis: A randomized, controlled trial.

Weinblatt M, Kremer J, Bankhurst A, Bulpitt K, Fleischmann R, Fox R, Jackson C, Lange M, Burge D: A trial of etanercept, a recombinant tumor necrosis factor receptor:Fc fusion protein, in patients with rheumatoid arthritis receiving methotrexate.
New England J Med 1999, 340:253259. Publisher Full Text

Keystone E, Genovese M, Klareskog L, Hsia E, Hall A, Miranda P, Pazdur J, Bae S, Palmer W, Zrubek J, Wiekowski M, Visvanathan S, Wu Z, Rahman M: Golimumab, a human antibody to tumour necrosis factor (alpha) given by monthly subcutaneous injections, in active rheumatoid arthritis despite methotrexate therapy: The GOFORWARD Study.

Kay J, Matteson E, Dasgupta B, Nash P, Durez P, Hall A, Hsia E, Han J, Wagner C, Xu Z, Visvanathan S, Rahman M: Golimumab in patients with active rheumatoid arthritis despite treatment with methotrexate: a randomized, doubleblind, placebocontrolled, doseranging study.
Arthritis & Rheumatism 2008, 58:964975. PubMed Abstract  Publisher Full Text

Keystone E, Van Der Heijde D, Mason D, Landewe R, van Vollenhoven R, Combe B, Emery P, Strand V, Mease P, Desai C, Pavelka K: Certolizumab pegol plus methotrexate is significantly more effective than placebo plus methotrexate in active rheumatoid arthritis: Findings of a fiftytwoweek, phase III, multicenter, randomized, doubleblind, placebocontrolled, parallelgroup study.
Arthritis & Rheumatism 2009, 58:33193329. PubMed Abstract  Publisher Full Text

Smolen J, Landewe R, Mease P, Brzezicki J, Mason D, Luijtens K, van Vollenhoven R, Kavanaugh A, Schiff M, Burmester G, Strand V, Vencovsky J, van der Heijde D: Efficacy and safety of certolizumab pegol plus methotrexate in active rheumatoid arthritis: the RAPID 2 study. A randomised controlled trial.

Fleischmann R, Vencovsky J, Van Vollenhoven R, Borenstein D, Box J, Coteur G, Goel N, Brezinschek H, Innes A, Strand V: Efficacy and safety of certolizumab pegol monotherapy every 4 weeks in patients with rheumatoid arthritis failing previous diseasemodifying antirheumatic therapy: the FAST4WARD study.
Ann rheumatic diseases 2009, 68:8058011. Publisher Full Text

Adams R, Walsh C, Veale D, Bresnihan B, FitzGerald O, Barry M: Understanding the relationship between the EQ5D, SF6D, HAQ and disease activity in inflammatory arthritis.
Pharmacoeconomics 2010, 28:477487. PubMed Abstract  Publisher Full Text

Bennette C, Vickers A: Against quantiles: categorization of continuous variables in epidemiologic research, and its discontents.
BMC Med Res Methodology 2012, 12:21. BioMed Central Full Text

Hasselblad V, Hedges L: Metaanalyses of screening and diagnostic tests.

Caldwell D, Welton N, Dias S, Ades A: Selecting the best scale for measuring treatment effect in a network metaanalysis: a case study in childhood nocturnal enuresis.
Res Synth Methods 2012, 3:126141. Publisher Full Text

O’Rourke K, Walsh C, Hutchinson M: Outcome of betainterferon treatment in relapsingremitting multiple sclerosis: a Bayesian analysis.
J Neurology 2007, 254:15471554. Publisher Full Text
Prepublication history
The prepublication history for this paper can be accessed here: