Department of Clinical Epidemiology and Biostatistics, McMaster University, Hamilton, ON, Canada

Biostatistics Unit, Father Sean O’Sullivan Research Centre, St Joseph’s Healthcare, Hamilton, ON, Canada

Faculty of Health Sciences, University of Ottawa, Ottawa, ON, Canada

Abstract

Background

Multiple treatment comparison (MTC) meta-analyses are commonly modeled in a Bayesian framework, and weakly informative priors are typically preferred to mirror familiar data driven frequentist approaches. Random-effects MTCs have commonly modeled heterogeneity under the assumption that the between-trial variance for all involved treatment comparisons are equal (i.e., the ‘common variance’ assumption). This approach ‘borrows strength’ for heterogeneity estimation across treatment comparisons, and thus, ads valuable precision when data is sparse. The homogeneous variance assumption, however, is unrealistic and can severely bias variance estimates. Consequently 95% credible intervals may not retain nominal coverage, and treatment rank probabilities may become distorted. Relaxing the homogeneous variance assumption may be equally problematic due to reduced precision. To regain good precision, moderately informative variance priors or additional mathematical assumptions may be necessary.

Methods

In this paper we describe four novel approaches to modeling heterogeneity variance - two novel model structures, and two approaches for use of moderately informative variance priors. We examine the relative performance of all approaches in two illustrative MTC data sets. We particularly compare between-study heterogeneity estimates and model fits, treatment effect estimates and 95% credible intervals, and treatment rank probabilities.

Results

In both data sets, use of moderately informative variance priors constructed from the pair wise meta-analysis data yielded the best model fit and narrower credible intervals. Imposing consistency equations on variance estimates, assuming variances to be exchangeable, or using empirically informed variance priors also yielded good model fits and narrow credible intervals. The homogeneous variance model yielded high precision at all times, but overall inadequate estimates of between-trial variances. Lastly, treatment rankings were similar among the novel approaches, but considerably different when compared with the homogenous variance approach.

Conclusions

MTC models using a homogenous variance structure appear to perform sub-optimally when between-trial variances vary between comparisons. Using informative variance priors, assuming exchangeability or imposing consistency between heterogeneity variances can all ensure sufficiently reliable and realistic heterogeneity estimation, and thus more reliable MTC inferences. All four approaches should be viable candidates for replacing or supplementing the conventional homogeneous variance MTC model, which is currently the most widely used in practice.

Background

Multiple treatment comparison (MTC) meta-analysis is an extension of conventional pair wise meta-analysis where only two interventions are being compared at the time. In contrast to pair wise meta-analysis, MTCs allow for simultaneous inferences about the comparative effectiveness and safety of multiple (3 or more) interventions. The statistical models used to analyze meta-analytic data on multiple interventions are commonly employed in the Bayesian frameworks

Variance parameter estimates play an important role in the overall inferences of an MTC since they impact the width of 95% credible intervals and treatment rank probabilities. A largely under-recognized issue in random-effects MTCs (as well as Bayesian pair wise random-effects meta-analysis) is that apparently weakly informative heterogeneity variance priors may often be moderately informative

Another under-recognized issue in random-effects MTCs is the importance of the assumptions made about the similarity and correlation between the degrees of heterogeneity across treatment comparisons (i.e., assumed heterogeneity variance structures)

There are a number of approaches for eliciting or constructing informative variance priors in random-effects MTCs. Further, there are a number of possible heterogeneity variance structures under which weakly informative variance priors can be employed. To date, no comparison of the available informative and weakly informative approaches is available of their relative performance. In this article we review and compare six random-effects MTC models – four under which weakly informative variance priors are elicited, and two under which moderately informative variance priors are elicited. The four weakly informative models include the conventional homogeneous variance model, the unrestricted heterogeneous variance model, the exchangeable variances model, and the consistency variances model. The two moderately informative models are structurally based on the unrestricted heterogeneous variance model and the variance priors are either frequentistic distribution approximations from within the MTC data or distributions previously derived from a large external empirical data set. We place comparative emphasis on the homogeneous variance model since this approach is conventionally used in MTC practice. We discuss how inferences from the informative approaches as well as the other weakly informative approaches theoretically line up against inferences from the conventional homogeneous variance MTC model. We compare treatment effect estimates 95% credible intervals, heterogeneity variance estimates and posterior distributions, and treatment rank probabilities from the discussed models in two illustrative examples. MTC treatment effect and variance estimates are also compared with those from pair wise meta-analyses.

Methods

In this section we first describe, distinguish and discuss what is meant by different degrees of information contained in the prior distributions in Bayesian MTCs. We then describe the general MTC model setup, as well as the setup for the commonly applied homogeneous variance MTC model. Lastly, we describe six approaches to modelling between-trial variances that make use of different combinations of heterogeneity variance parameterizations and priors.

Prior information terminology

In the introduction we mentioned use of ‘non-informative’, ‘weakly informative’, and ‘moderately informative’ priors. These terms are often used vaguely or interchangeably in the literature. Below, we define, distinguish and discuss what exactly is meant in this article when priors are ‘non-informative’, ‘weakly informative’, or ‘moderately informative’.

Non-informative priors

In this article, we define ‘non-informative priors’ as prior distributions, that carry virtually no information about the likely true value of a parameter. For example, for treatment effects measured as log odds ratios in a logistic regression model (which is the typical set up for MTCs of binary data), a normal distribution with mean zero and variance 10000 carries virtually no information about the likely true log odds ratio, and thus, constitutes a non-informative prior distribution. For a between-trial variance parameter, an example of a non-informative prior could be a gamma distribution with shape and scale parameters of both 10^{-10}. It should be noted that because Bayesian analysis is typically realized by Markov Chain Monte Carlo (MCMC) sampling, which relies on prior distributions and initial sampling values being sufficiently reasonable to allow for convergence of the posterior distribution, there is a limit to how non-informative a prior can feasibly be. For example, running the MCMC sampling for a Bayesian MTC may not be feasible if a gamma distribution with shape and scale parameters of 10^{-10} is used for the between-trial variance parameter.

Weakly informative priors

In this article, we define a ‘weakly informative’ prior as a prior distribution that carries more information than a non-informative prior, but deliberately carries smaller degree of information than is actually available. The purpose of using weakly informative priors rather than non-informative priors is typically to achieve some stabilization in the MCMC sampling and/or estimation procedure. In the context of MTCs, a typical example of a weakly informative prior for the between-trial standard deviation parameter is the conventionally used uniform distribution between 0 and 2 when data is dichotomous and treatment effects are modelled as log odds ratios (ie, modelled in a logistic regression framework). This prior carries more information than a typical non-informative variance prior (e.g., the above mentioned gamma distribution). It is well known that between-trial variances on the log odds ratio scale generally do no exceed a value of 4, and so, this knowledge is used by truncating the between-trial standard deviation to 2. It is also known that between-trial variances on the log odds ratio scale are typically smaller than 1 and closer to 0. However, this knowledge is only used partially for this prior since the probability of observing larger between-trial variance values only decreases slightly for larger values

Moderately informative priors

In this article, we define a ‘moderately informative’ prior as a prior distribution that carries a distinguishable and larger degree of information than a weakly informative prior. The purpose of using a moderately informative prior is to either fully or partially mix prior (external) knowledge about one or more parameters with the data. To this end, the data still plays an important role. One example of a moderately informative prior is use of observational data about the magnitude of one or more comparative treatment effects. For example, if observational studies have suggested that one novel treatment exhibits a 25% reduction of symptoms over another novel treatment, one can use this evidence to produce a mean parameter value (treatment effect) in the prior distribution and subsequently elicit a variance that corresponds to the weight and confidence one is willing to put in this value. Another example of moderately informative priors in the MTC framework is the use of empirical evidence on the distribution of between-study variance estimates across published meta-analyses. This is also the last of the six heterogeneous variance approaches considered in this article, and will be illustrated below.

General MTC model set up

For this manuscript we describe MTC models of binary data. However, the modelling concepts are easily extended for other types of data such as count data and continuous data

In the binary data setting, a commonly used effect measure in MTCs is the odds ratio (OR). For each treatment comparison, odds ratios are typically estimates with a logistic regression model that simultaneously links the trial-arm odds and the treatment comparison odds ratios. Letting

Where _{
jt
} is the probability of an event in trial _{
jt
} and _{
jt
} are the number of events and the number of patients in the corresponding treatment arm; _{
jb
} is the log odds of having an event in the control arm (i.e., with ‘baseline treatment’ _{
jtb
} is the log odds ratio of treatment _{
tb
} is the ‘true’ overall treatment effect of _{
tb
}
^{
2
} is the corresponding between-trial variance. The last equation represents the ‘consistency’ assumption, which is necessary for all MTC models, and dictates that any expected relative treatment effect of a direct (head-to-head) evidence source is equal to the corresponding expected relative treatment effect of an indirect evidence source. In other words, the consistency assumptions dictates that the results from direct and indirect sources of evidence should not differ beyond the play of chance.

In the above, the control arm (baseline) log odds parameters _{
jb
} are treated as nuisance parameters, and assigned non-informative normal distribution priors with mean 0 and very large variances, typically of 1000 or 10000. For _{
tb
} (i.e., the treatment effect of

MTC models with weakly informative variance priors

The homogeneous variance model

Under the homogenous variance MTC model the assumption is made that all between-trial variances are equal. That is, strictly speaking we assume _{
tb
}
^{2} = ^{2} for all treatment comparisons ^{
2
}.

Typically a weakly informative prior is assigned to

The unrestricted heterogeneous variances model

Under the heterogeneous variance MTC models, all between-trial variances are allowed to take on different values. The _{
tb
}
^{
2
}. Conventionally, one would make use of the uniform distribution from 0 to 2 or from 0 to 10 as prior distributions for the between-trial standard deviations. The heterogeneous variance model with such priors is typically referred to as the unrestricted heterogeneous variance model.

Theoretically, this model is advantageous due to its high flexibility in modelling heterogeneity variances. In practice, however, this model is often sub-optimal because many comparisons are typically only informed by a few trials, and thus, the estimation of between-trial variances (i.e., their posterior distributions) is very imprecise. The below four Bayesian modelling approaches are modifications of the unrestricted heterogeneous variance model that apply different parameter value constraints or moderately informative prior distributions to optimize the estimation of the between-trial variance parameters.

The exchangeable variances model

One approach to gaining precision for the between-trial variance estimation is to ‘meet in the middle’ between the homogeneous and heterogeneous variances models by assuming that the between-trial variances are exchangeable. That is, one can assume that the between-trial variances are random samples from a common between-trial distribution, thus allowing them to borrow strength from each other

Here we assign a weakly informative prior distribution to the ‘common’ between-trial variance corresponding to the ‘common precision’, (1/_{
tb
}
^{
2
}, can be thought of as weakly informative due to the reliance on the ‘common variance’ parameter and the degrees of freedom. We refer to this approach as the

Theoretically, the exchangeable variances MTC model gains the best of two worlds. It gains precision by borrowing strength from the common variance assumption, but it retains flexibility in allowing for differing between-trial variances. In practice, however, this model may not perform optimally when the between-trial variances differ considerably across comparisons. This is because the assumption of a common variance ties all individual between-trial variances probalistically to some central tendency, in which case heterogeneity parameters that are truly not close to the central tendency will be inaccurately estimated. Arguably, the exchangeable variance approach may work best in situations where 1) the interventions being investigated in the MTC are all similar (e.g., of the same drug class or solely pharmacotherapies); and 2) the study designs and patient eligibility criteria are fairly comparable.

The heterogeneous variances model using second order consistency inequalities

Another approach to gaining precision but retaining flexibility in modelling of heterogeneous variances is to re-parameterize the variance structure in order to ensure that the property of _{
yx
}, _{
yb
}, and _{
xb
}, we assume that

This equation is also sometimes referred to as the

Where _{
yx
}
^{
2
}
_{
yb
}
^{
2
} and _{
xb
}
^{
2
}, are the variances of _{
yx
}, _{
yb
}, and _{
xb
}, respectively, and _{
yx
}
^{
b
} is the correlation between _{
yb
}, and _{
xb
}. The above equation implies a

Where |

Theoretically, the consistency variances model is optimal in that it largely retains the flexibility of the unrestricted variances model, and additionally restricts variances in alignment with and borrows strength from the seminal assumption of consistency. In practice, the consistency triangular inequality may not hold within the available data since between-trial variance estimates (and posterior distributions) may fluctuate and differ due to the play of chance

MTC models with moderately informative variance priors

Considering the limitation of the above models, one could argue that random-effects MTCs incorporating sensible moderately informative variance priors constitute a viable alternative. Below we propose two sensible approaches for obtaining and eliciting informative variance priors in random-effects MTCs.

Using frequentist within-data approximate distribution as priors

Informative variance priors should aid in ensuring that the estimation of between-trial variances is directed with appropriate probability mass to plausible intervals of possible values. It therefore seems reasonable to require that variance estimates and their posterior probability distributions should be directed towards the values one would have obtained in separate pair wise meta-analysis, and vice versa _{
DL
}
^{
2
}, based on the relationship between this estimator and Cochran’s _{
DL
}
^{
2
} = (_{
1
} – (_{
2
}
_{
1
})), where _{
1
} is the sum of trial weights (ie, inverse variances) and _{
2
} is the sum of squared trial weights

Where _{
yx
}
_{
yx
}
_{yx}. We refer to the paper by Biggerstaff and Tweedie for the approximate deterministic expressions of _{
yx
}
_{
yx
}

While the approximate distribution for _{
DL
}
^{
2
} for any comparison is a candidate as an informative variance prior, it does have some undesirable limitations in the context of Bayesian analysis. First, _{
DL
}
^{
2
} can yield negative estimates and will in this case be truncated to 0 _{
DL
}
^{
2
} to underestimate the between-trial variance ^{st} order method of moments estimator, the HM estimator is a 2^{nd} order method of moments based estimator and has the following expression

The HM estimator is consistently positive and has been shown to yield accurate and precise estimates of the between-trial variance _{
HM
}
^{
2
} via its original expression, the shortcomings of the DL approach are circumvented.

The above proposed approach for obtaining and eliciting informed variance priors is either optimal or sub-optimal depending on the assumptions one is willing to make. By informing variance estimation with prior distributions corresponding to the expected likelihood in a frequentist analysis, one imposes a ‘2-stage’ estimation process that lets the Bayesian MCMC sampling ‘concentrate’ on the estimation of treatment effects. An analogous process was recently proposed in the purely frequentist framework

Heterogeneous variances using empirically derived informative priors

A simpler and more general approach to incorporating informed variance priors is to borrow strength from external empirical evidence. Turner et al. reviewed 14886 Cochrane Database meta-analyses including a total 77237 trials and approximated the empirical distribution of the between-trial variance categorized by type of outcome (mortality, semi-objective and subjective), type of intervention, and field of medicine

This informative variance approach is relatively straightforward to apply. The already empirically approximated priors have general applicability due to the sample size of the empirical study from which they originated. However, to the extent other factors than the ones explored by Turner et al. determine the likely degree and distribution of heterogeneity variance, the approach may not produce optimal variance estimation.

Results

We applied the above considered models and priors to two MTC data sets of differing size and complexity to illustrate the performance. The treatment networks for our two examples are presented in Figure

Presents the treatment networks with the number of trials informing each treatment comparison in our two illustrative examples.

**Presents the treatment networks with the number of trials informing each treatment comparison in our two illustrative examples.** The treatment network on the left is the network for our first illustrative example. The treatment network on the right side is the network for our second illustrative example. The circles represent the treatments in the network, the lines represent the comparisons where head-to-head (direct) evidence is available, and the numbers in the lines present the number of randomized clinical trials available per comparison. Abbreviations: PEG-2A (Peginterferon-2a); PEG-2B (Peginterferon-2b); INF (Interferon), RBV (Ribavirin); Trt (Treatment).

The DIC is a measure of model fit computed from the likelihood function with a penalty for complexity

We compared the heterogeneity variances from all MTC models with the DerSimonian-Laird and Hartung-Makambi estimates from pair wise meta-analyses, as well as with the Bayesian pair wise meta-analysis estimates. Considering the pair wise heterogeneity variance estimates as the bench mark, we then assessed the extent to which observed differences in inferences between MTC models could be explained by poor estimation of between-study heterogeneity variances and their posterior distributions.

All Bayesian MTC models were carried out in WinBUGS v.1.4.3

Illustrative example 1

In our first example, we use data from two Cochrane Database systematic reviews on interventions for treating hepatitis C

In this data set, each of the three treatment comparisons is informed by a comparable amount of evidence. In particular, the comparison of PEG-2a+RBV and INF+RBV includes 4 trials and 1197 patients, the comparison of PEG-2b+RBV and INF+RBV includes 12 trials and 2750 patients, and the comparison of PEG-2a+RBV and PEG-2b+RBV includes 6 trials and 2994 patients. The trials in the three comparisons (pairwise meta-analyses) each incurred different degrees of heterogeneity (e.g., DerSimonian-Laird between-trial variance estimates of 0.64, 0.00, and 0.04). This suggests a need for modelling the between-trial variances as heterogeneous in the MTC model, which makes this data set a good candidate for how well the heterogeneous variance MTC models perform in this context and how they measure up against the conventional homogeneous variance model. For the ‘empirically informed variances’ model we used a log-normal distribution with mean −2.34 and variance 1.62

As expected, the homogeneous variance MTC models yielded a worse model fit than the heterogeneous variance MTC models according to the DIC (Table

**Between-trial variance estimates**

**Model fit statistics**

**Models**

**PEG-2A+RBV vs INF+RBV**

**PEG-2B+RBV vs INF+RBV**

**PEG-2B+RBV vs PEG-2A+RBV**

**pD**

**DIC**

Abbreviations: PEG-2A (Peginterferon-2a); PEG-2B (Peginterferon-2b); INF (Interferon), RBV (Ribavirin).

Frequentist (DerSimonian-Laird)

0.642

0.000

0.036

--

--

Frequentist (Hartung-Makimbi)

0.580

0.017

0.021

--

--

Bayesian (weakly informative)

0.700

0.018

0.077

--

--

Bayesian (frequentist informed)

0.422

0.001

0.038

--

--

Bayesian (empirically informed)

0.091

0.024

0.052

--

--

0.097

0.097

0.097

33.3

283.2

Unrestricted variances

1.046

0.017

0.103

32.8

277.7

Exchangeable variances

0.510

0.016

0.083

32.6

278.6

Consistency variances structure

0.225

0.024

0.164

32.9

280.2

Frequentist informed priors

0.677

0.011

0.047

30.9

275.6

Empirically informed priors

0.368

0.026

0.076

32.4

278.4

Presents the posterior distributions of the between-trial variance parameters in the first illustrative example under the six employed MTC models.

**Presents the posterior distributions of the between-trial variance parameters in the first illustrative example under the six employed MTC models: the homogeneous variance model (row 1); the unrestricted variances model (row 2); the exchangeable variances model (row 3); the consistency variances model (row 4); the frequentistically informed variances model (row 5); and the empirically informed variances model (row 6).** The two presented comparisons are: peginterferon-2a (PEG-2A) vs Interferon (INF) (column 1), and Peginterferon-2a (PEG-2A) vs Peginterferon-2b (PEG-2B) (column 2). The comparison of PEG-2B vs INF was selective excluded due to the posterior variance distributions being more similar across the five heterogeneous variance approaches.

For the comparison between peginterferon-2a and interferon and the comparison between the two peginterferons, the homogeneous variance model has narrower 95% credible intervals that all other heterogeneous variance models, except for the informed variance model based on frequentist approximate distributions (see Table

**Model**

**PEG-2A+RBV vs INF+RBV**

**PEG-2B+RBV vs INF+RBV**

**PEG-2B+RBV vs PEG-2A+RBV**

* The MCMC simulation did not converge for the log odds ratio parameter (within the first 1.000.000 runs), and thus did not produce meaningful results.

Abbreviations: PEG-2A (Peginterferon-2a); PEG-2B (Peginterferon-2b); INF (Interferon), RBV (Ribavirin).

Frequentist (DerSimonian-Laird)

3.63(1.51-8.73)

1.30(1.11-1.52)

1.38(1.07-1.79)

Frequentist (Hartung-Makimbi)

3.60(1.56-8.34)

1.35(1.11-1.64)

1.38(0.36-3.24)

Bayesian (weakly informative)

NA*

1.36(1.10-1.73)

1.38(0.94-2.22)

Bayesian (frequentis informed)

NA*

1.30(1.11-1.55)

1.40(0.86-2.47)

Bayesian (empirically informed)

NA*

1.36(1.10-1.74)

1.38(1.02-1.99)

2.42(1.75-3.60)

1.53(1.19-2.03)

1.58(1.18-2.26)

Unrestricted variances

2.11(1.40-3.57)

1.38(1.13-1.79)

1.50(1.06-2.53)

Exchangeable variances*

2.17(1.48-3.43)

1.40(1.15-1.79)

1.53(1.11-2.36)

Consistency variances structure

2.39(1.63-3.80)

1.42(1.16-1.86)

1.67(1.17-2.68)

Frequentist informed priors

2.04(1.45-2.93)

1.38(1.15-1.69)

1.46(1.12-2.05)

Empirically informed priors

2.23(1.54-3.40)

1.44(1.16-1.83)

1.54(1.13-2.29)

Illustrative example 2

Our second example data set is a larger, more diverse treatment network including four pharmacological interventions (Trt1, Trt2, Trt3, and Trt4) and a control for cessation of a harmful behaviour (See Figure

**Between-trial variance estimates**

**Model fit statistics**

**Models**

**Trt1 vs Placebo**

**Trt2 vs Placebo**

**Trt3 vs Placebo**

**Trt4 vs Placebo**

**Trt2 vs Trt1**

**pD**

**DIC**

* The’average’ variance was 0.173.

Abbreviations: DIC (Deviance information criterion); pD (effective number of model parameters); Trt (Treatment).

Frequentist (DerSimonian-Laird)

0.086

0.110

0.016

0.075

0.106

--

--

Frequentist (Hartung-Makimbi)

0.083

0.103

0.040

0.072

0.112

--

--

Bayesian (weakly informative)

0.100

0.371

0.023

0.103

0.334

--

--

Bayesian (frequentist informed)

0.087

0.121

0.036

0.067

0.093

--

--

Bayesian (empirically informed)

0.088

0.110

0.021

0.054

0.059

--

--

0.078

0.078

0.078

0.078

0.078

138.8

1229.1

Unrestricted variances

0.100

0.469

0.023

0.104

0.214

138.3

1229.9

Exchangeable variances*

0.092

0.226

0.009

0.066

0.047

133.7

1232.2

Consistency variances structure

0.091

0.172

0.033

0.075

0.133

136.6

1230.0

Frequentist informed priors

0.087

0.172

0.036

0.069

0.064

135.6

1226.4

Empirically informed priors

0.087

0.199

0.020

0.054

0.042

133.8

1229.5

According to the DIC, the informed variances model based on the frequentist approximate variance distributions yielded the best model fit (Table

Presents the posterior distributions of the between-trial variance parameters in the second illustrative example under the six employed MTC models.

**Presents the posterior distributions of the between-trial variance parameters in the second illustrative example under the six employed MTC models: the homogeneous variance model (row 1); the unrestricted variances model (row 2); the exchangeable variances model (row 3); the consistency variances model (row 4); the frequentistically informed variances model (row 5); and the empirically informed variances model (row 6).** The three presented comparisons are: Treatment 2 (Trt2) versus control (column 1); treatment 4 (Trt2) versus Control; and Trt4 versus Trt1. The remaining comparisons were selective excluded due to the posterior variance distributions being more similar across the five heterogeneous variance approaches.

The treatment effect estimate and 95% credible interval for Trt2 were considerably affected by the variance assumption, and thus, so were indirect comparisons between Trt2 versus other interventions (Table

**Model**

**Trt1 vs Placebo**

**Trt2 vs Placebo**

**Trt3 vs Placebo**

**Trt4 vs Placebo**

**Trt2 vs Trt1**

**Trt4 vs Trt2**

Abbreviations: DIC (Deviance information criterion); pD (effective number of model parameters); Trt (Treatment).

1.94(1.67-2.24)

2.11(1.42-3.13)

1.78(1.60-1.97)

2.86(2.21-3.71)

1.90 (1.17-3.09)

--

1.94(1.68-2.23)

2.09(1.42-3.09)

1.77(1.57-2.00)

2.86(2.21-3.70)

1.91 (1.16-3.13)

--

1.98(1.70-2.31)

2.38(1.30-5.82)

1.80(1.61-2.02)

2.89(2.08-4.06)

1.97 (0.66-8.88)

--

1.97(1.71-2.28)

2.16(1.48-3.68)

1.80(1.60-2.02)

2.89(2.23-3.77)

1.79 (1.17-3.55)

--

1.97(1.71-2.28)

2.13(1.47-3.98)

1.80(1.61-2.03)

2.89(2.20-3.80)

1.84 (1.17-3.53)

--

1.91(1.67-2.19)

2.59(1.97-3.50)

1.80(1.57-2.08)

2.90(2.23-3.79)

1.36 (1.02-1.84)

1.11(0.74-1.63)

Unrestricted variances

1.95(1.69-2.28)

3.05(1.84-5.50)

1.81(1.62-2.01)

2.90(2.07-4.07)

1.56 (0.94-2.75)

0.94(0.49-1.74)

Exchangeable variances*

1.93(1.67-2.26)

2.89(1.98-4.43)

1.78(1.56-1.99)

2.89(2.17-3.86)

1.50 (1.03-2.26)

1.01(0.59-1.58)

Consistency variances structure

1.93(1.66-2.23)

2.80(1.92-4.69)

1.80(1.60-2.02)

2.90(2.18-3.86)

1.45 (0.99-2.40)

1.03(0.58-1.66)

Frequentist informed priors

1.93(1.68-2.22)

2.80(1.98-4.10)

1.80(1.60-2.03)

2.90(2.22-3.78)

1.46 (1.02-2.11)

1.05(0.66-1.61)

Empirically informed priors

1.93(1.67-2.23)

2.84(2.00-4.26)

1.80(1.61-2.02)

2.91(2.22-3.81)

1.48 (1.00-2.20)

1.02(0.63-1.59)

**Model and prior**

**Trt1**

**Trt2**

**Trt3**

**Trt4**

MTC models for the second illustrative example.

0.00%

28.2%

0.00%

71.8%

Unrestricted variances

0.01%

56.8%

0.00%

43.2%

Exchangeable variances

0.01%

49.3%

0.02%

50.4%

Consistency variances structure

0.04%

45.0%

0.01%

55.9%

Frequentist informed priors

0.00%

42.5%

0.00%

57.5%

Empirically informed priors

0.00%

46.5%

0.00%

53.5%

In this example, a number of reasons suggest the informed variances model based on frequentist approximate variance distributions is the more optimal choice. First, this model clearly yields the best model fit according to the DIC. Second, it produces the variance estimates closest to those of the frequentist pair wise meta-analyses. Lastly, the full MTC from which this example is borrowed, the efficacy of the considered interventions was also investigated for 1 month, 3 months, and 12 months follow-up. For these outcomes, many of the comparisons were non-significant (i.e., the 95% credible intervals included 1.00) with the homogeneous variance model despite clear statistical significance in the pair wise meta-analyses. When we used variance priors informed by frequentist approximate variance distributions, this statistical significance was recovered.

Discussion

The variance structure in an MTC is challenging to estimate because it rests on the amount of evidence and the linkage between comparisons. A number of approaches are available, but their performance is tied with the appropriateness of the assumed linkage between comparisons, and in the Bayesian framework, the elicited variance priors. Conventional MTC models have made use of the unrealistic assumption that the between trial variances for the included comparisons are all equal

Our examples suggest that these four approaches all allow for reliable estimation of differing between-study heterogeneity variances across comparisons, whereas the unrestricted approach often does not. To this end, these four approaches seem superior to the homogeneous variance structure model as well as the unrestricted heterogeneous variances approach. The frequentist informed approach yielded the best model fits in both example, and although further research is needed at this point, one could argue for this approach as a primary supplement to the conventional homogeneous model.

Our study offers several strengths, but also has some limitations. Our chosen illustrative examples are of different size and complexity and yield heterogeneity estimates for which the homogeneous variance assumption was violated to an extend that impacted the findings of the MTCs. Our study is also the first to compare multiple weakly and moderately informed approaches to modelling heterogeneity in MTCs. Our study, however, is by no means generalizable to all MTCs. Several treatment networks may exist or emerge in which, for example, the homogeneous variance model and some heterogeneous variance model will yield close to equal inferences about all comparative treatment effects. In this vein, it is important that authors and readers of MTCs continually pay careful consideration to the fragility of variance estimation, credible intervals and treatment rank probabilities. Another limitation is the empirical nature of this study. With empirical data we can only observe differences, but never infer definitively about the truth. In this context, simulation studies would be needed to investigate the performance of the models based on bias, precision, MSE, etc., under different scenarios and types of networks. However, we believe additional empirical studies are necessary to inform which scenarios are truly important to explore under simulation.

Appropriate modelling of heterogeneity variances in MTCs will become increasingly important over the next years. First, ‘statistical significance’ and treatment rank probabilities can be sensitive to the employed variance structure and variance priors

Further, we will likely see an increase in MTCs incorporating meta-regression or subgroup analysis to explain the observed heterogeneity by effect modification caused by some clinical covariate(s). In this vein, appropriately estimating the unexplained degree of heterogeneity for each treatment comparison is seminal to reliable estimation of the effect modification caused by some clinical covariate(s). In other words, without unbiased quantification of heterogeneity it becomes increasingly challenging to explain heterogeneity.

Conclusions

In conclusion, MTC models using either a homogenous variance structure or weakly informative variance priors in connection with an unrestricted heterogeneous variance structure both have serious methodological shortcomings. Using informative variance priors in connection with an unrestricted variance structure or borrowing strength by assuming exchangeability or imposing consistency between heterogeneity variances, can all ensure sufficiently reliable and realistic heterogeneity estimation, and thus reliable MTC inferences. All four approaches should be viable candidates for replacing or supplementing the conventional homogeneous variance MTC model, which is currently used widely in practice.

Competing interests

The authors declare that they have no competing interests.

Authors’ contributions

KT drafted the first version of the manuscript, conceived the idea of the study, contributed to the design of the study, and performed all statistical analysis. LT contributed to the design of the study and writing of the manuscript. EM co-conceived the idea of the study, contributed to the design of the study, and contributed to the writing of the manuscript. All authors read and approved the final manuscript.

Pre-publication history

The pre-publication history for this paper can be accessed here: