Abstract
Background
A proportional hazards measure is suggested in the context of analyzing SROC curves that arise in the meta–analysis of diagnostic studies. The measure can be motivated as a special model: the Lehmann model for ROC curves. The Lehmann model involves study–specific sensitivities and specificities and a diagnostic accuracy parameter which connects the two.
Methods
A study–specific model is estimated for each study, and the resulting studyspecific estimate of diagnostic accuracy is taken as an outcome measure for a mixed model with a random study effect and other studylevel covariates as fixed effects. The variance component model becomes estimable by deriving withinstudy variances, depending on the outcome measure of choice. In contrast to existing approaches – usually of bivariate nature for the outcome measures – the suggested approach is univariate and, hence, allows easily the application of conventional mixed modelling.
Results
Some simple modifications in the SAS procedure proc mixed allow the fitting of mixed models for metaanalytic data from diagnostic studies. The methodology is illustrated with several meta–analytic diagnostic data sets, including a meta–analysis of the Mini–Mental State Examination as a diagnostic device for dementia and mild cognitive impairment.
Conclusions
The proposed methodology allows us to embed the metaanalysis of diagnostic studies into the well–developed area of mixed modelling. Different outcome measures, specifically from the perspective of whether a local or a global measure of diagnostic accuracy should be applied, are discussed as well. In particular, variation in cutoff value is discussed together with recommendations on choosing the best cutoff value. We also show how this problem can be addressed with the proposed methodology.
Keywords:
Diagnostic accuracy; Mixed modelling; Random effects modelling; Cutoff value modelling; SROC modellingBackground
We are interested in the following setting occurring in the field of metaanalysis of diagnostic studies (Hasselblad and Hedges [1]; Sutton et al.[2]; Deeks [3]; Schulze et al.[4]): a variety of diagnostic studies are available providing estimates of the diagnostic measures of specificity q=P(T=0D=0) as and of sensitivity p=P(T=1D=1) as , where D=1 and D=0 denote presence or absence of disease, respectively, and T=1 or T=0 denote positivity or negativity of the diagnostic test, respectively, x_{i} are the number of observed truenegatives out of n_{i} healthy individuals, and y_{i} are the number of observed truepositives out of m_{i} diseased individuals, for i=1,…,k, k being the number of studies. For more details on the statistical modelling of the diagnostic data from a single study, see Pepe [5,6]. For a more detailed introduction to meta–analysis of diagnostic studies, see Holling et al.[7]. In the following, we will look at several examples – mainly from medicine and psychology – for this special metaanalytic situation. In principle, however, applications could occur in all areas in which metaanalytic data is encountered; Swets [8] considers mainly psychological applications, but also mentions cases from engineering (quality control), manufacturing (failing parts in planes), metereology (correctness of weather predictions), information science (correctness of information retrieval), or criminology (correctness of lie detection test). We illustrate the special metaanalytic situation mentioned above with a metaanalysis on a diagnostic test on heart failure (see also Holling et al.[7]).
Example 1: MetaAnalysis of diagnostic accuracy of Brain Natriuretic Peptides (BNP) for heart failure. Doust et al.[9] provide a metaanalysis on the diagnostic accuracy of the brain natriuretic peptides (BNP) procedure as a diagnostic test for heart failure. According to the authors, diagnosis of heart failure is difficult, with both overdiagnosis and underdiagnosis occurring. The metaanalysis considers a range of diagnostic studies that use different reference standards (where a reference standard defines the presence or absence of disease). Here we only consider the eight studies (see Table 1) using the left ventricular ejection fraction of 40% or less as reference standard.
Table 1. Metaanalysis of of diagnostic accuracy of brain natriuretic peptides (BNP) for heart failure using the left ventricular ejection fraction of 40% or less as reference standard
The cut–off value problem. A separate meta–analysis of sensitivity and specificity using the meta–analytic tools for independent binomial samples is problematic when the underlying diagnostic test utilizes a continuous or ordered categorical scale and different cut–off values have been used in different diagnostic studies. A simple variation of the cut–off value from study to study might lead to quite different values of sensitivity and specificity without any actual change in the diagnostic accuracy of the underlying test.
SROC curve. Due to this comparability problem for sensitivity and specificity, interest is usually focussed on the summary receiver operating characteristic (SROC) curve consisting of the pairs (1−q(t),p(t)) where q(t)= P(T<tD=0) and p(t)=P(T≥tD=1) for a continuous test T with potential value t. For a given study i, i=1,⋯,k, with potentially unknown cut–off value t_{i}, the pairs (1−q(t_{i}),p(t_{i})) can be estimated by for i=1,…,k. The SROC curve accommodates the cut–off value problem. Different pairs could have quite different values of specificity and sensitivity, but still reflect identical diagnostic accuracy. The SROC diagram for the meta–analysis on BNP and heart failure is given in Figure 1.
Figure 1. SROC diagram for BNP and heart failure: circles are the observed pairs of false positive rate and sensitivity, dashed curve is lowess smoother.
Clearly, there is a wide range of values for specificity and sensitivity. Nevertheless, as Figure 1 shows, the possibility that the pairs might stem from a common SROC curve (as given by the dashed curve in Figure 1) cannot be discarded. Since the SROC approach accommodates the cutoff value problem, it is commonly preferred to summary measures like the Youden index [10] or the diagnostic odds ratio [11]. In the following, we focus our analysis on the SROC curve.
Background of SROC modelling. SROC modelling has received considerable attention in the field and experienced several developments. An early model was suggested by Littenberg and Moses [12], [13] and has been used in practice frequently; Deeks [3] discusses its prominent role in modeling metaanalytic diagnostic study accuracy. Littenberg and Moses [13] suggest fitting D=α+βS, where is the logdiagnostic odds ratio and is a measure for a potential threshold effect. After α and β have been estimated from the data, the SROCcurve (p vs. 1−q) is reconstructed from the estimated values of α and β. The parameter α is interpreted as the summary logDOR, which is adjusted by means of S for potential cutoff value effect.
A two–level approach has been suggested by Rutter and Gatsonis [14], which is typically given in the following notational form (Walter and Macaskill [15]): let Z_{ij}∼Bi(n_{ij},π_{ij}), where Z_{ij} is the number of testpositives in study i for arm j (j=1 is diseased, j=2 is nondiseased), n_{ij} is the size of arm j in study i and π_{i1} is the sensitivity, π_{i2} is the false positive rate; the model is where θ_{i} is an implicit threshold parameter for study i, α_{i} is the diagnostic accuracy parameter in study i, and DS_{ij} represents a binary variable for the disease status. The parameter β allows for an association between test accuracy and test threshold. When β=0, α_{i} is estimated by D_{i} and θ_{i} is estimated by S_{i}/2, where D_{i} and S_{i} are as for the Littenberg–Moses model. Furthermore, to account for betweenstudy variation, a random effect is assumed for and , with θ_{i} and α_{i} being independent. As an alternative, a bivariate normal randomeffects meta–analysis has been suggested by van Houwelingen et al.[16]; see also Reitsma et al.[17] and Arends et al.[18]. Harbord et al.[19] show that these models are closely related.
Paper overview. In the following, we propose a specific model, called the Lehmann model, which we believe is very attractive for the analysis of SROC curves. The model involves study–specific sensitivities and specificities and a diagnostic accuracy parameter which connects the two. The Lehmann model achieves flexibility by allowing the diagnostic accuracy parameter to become a random effect. In this it is similar to the RutterGatsonis model, but differs in that it retains univariate dimensionality in its outcome measure and, hence, allows a mixed model approach in a more conventional way. In section “The proportional hazards measure”, the proportional hazards measure is motivated as a specific form of SROC curve modelling and is compared to other approaches. Section “A mixed model approach” introduces the specific mixed model in which the log proportional hazards measure forms the outcome measure, the study factor is a normally distributed random effect (to cope with unobserved heterogeneity), and other observed covariates (such as gold standard or diagnostic test variation) are considered as fixed effects in the mixed model. Section “Results” considers various applications including a metaanalysis of the MiniMental State Examination to diagnose dementia or mild cognitive impairment. It also provides SAScode for a simple execution of the suggested approach. In section “Discussion”, the choice of outcome is discussed and the difference between global and local diagnostic accuracy measures highlighted. This is particularly of interest if observed cutoff value variation occurs in the metaanalysis and needs to be assessed. Here a local criterion of diagnostic accuracy appears more appropriate. The paper ends with some brief conclusions and discussion in section “Conclusions”.
Methods
The proportional hazards measure
Numerous summary measures for a pair of specificity and sensitivity have been suggested: we mention here the Youden index, J_{i}=p_{i}+q_{i}−1 [10], and the squared Euclidean distance to the upper left corner in the SROC diagram, E_{i}=(1−p_{i})^{2}+(1−q_{i})^{2}. [A review of summary measures is given in Liu [20].] Using an average over any of these measures might be problematic: not only might sensitivities and specificities be heterogeneous, this might also be true for the associated summary measures such as the Youden index or the Euclidean distance (as demonstrated by Figure 2 using the data of the metaanalysis of BNP and heart failure).
Figure 2. Index plots for sensitivity, specificity, Youden index, and Euclidean distance showing the wide variability of these measures for the data of the metaanalysis of BNP and heart failure.
We suggest using the measure , which relates the logsensitivity to the logfalse positive rate; we call it the proportional hazards (PH) measure. In Figure 3 we see that this measure shows a reduced variability for the metaanalysis of BNP and heart failure, making it more suitable as an overall measure in the metaanalysis of diagnostic studies or diagnostic problems. While the measure appears to be like any other summary measure of the pair sensitivity and specificity, it has a specific SROCmodelling background and motivation. We have mentioned previously the cutoff value problem: observed heterogeneity might be induced by cutoff value variation which could lead to different sensitivities and specificities – despite the accuracy of the diagnostic test itself not having changed – and might also lead to an induced heterogeneity in the summary measure. Hence, it is unclear whether the observed heterogeneity is due to heterogeneity in the diagnostic accuracy (authentic heterogeneity) or whether it has occurred due to cutoff value variation (artificial heterogeneity). This second form of heterogeneity can also occur when the background population changes with the study.
Figure 3. Index plots for the PH measures for the data of the metaanalysis of BNP and heart failure.
One of the features of the SROC approach is that it incorporates the cutoff value variation in a natural way; hence a measure modelling an ROC curve is favorable. We suggest the PH measure based upon the Lehman family in the following way:
This model was suggested by Le [21] for the ROC curve. It is an appropriate model since, for feasible q, (1−q)^{θ} is also feasible as long as θ is positive. Note that (1) is defined for all values of p∈[0,1] and q∈ [ 0,1] whereas is only defined for p∈(0,1) and q∈(0,1). Population values of sensitivity and specificity of 1 are rarely realistic, although observed values of 1 for sensitivity and specificity do occur in samples. This can be coped with by using an appropriate smoothing constant such as estimating specificity as (n_{i}−1)/n_{i} when x_{i}=n_{i} and sensitivity as (m_{i}−1)/m_{i} if y_{i}=m_{i}.
In Figure 4 we see a number of examples of the proportional hazards family. It becomes clear now why θ is called the proportional hazards measure. By taking logarithms on both sides of (1) we achieve
Figure 4. Some examples of the proportional hazards model for various values ofθ.
meaning if model (1) holds, the ratio of logsensitivity to logfalse positive rate is constant across the range of possible cutoff value choices t. Hence the name proportional hazards model, which was suggested in a paper by Le [21] and used again in Gönen and Heller [22]. The idea of representing an entire ROC curve in a single measure is illustrated in Figure 5. While sensitivity and specificity vary over the entire interval (0,1), the value of θ remains constant. Hence, logsensitivity is proportional to the logfalse positive rate. This assumption is similar to an assumption used for a model in survival analysis, where it is assumed that the hazard rate of interest is proportional to the baseline hazard rate; this might have motivated the choice of name used by Le [21] and Gönen and Heller [22] in this context.
Figure 5. Proportional hazards model and associated PH measure.
However, it is not our intention to make the assumption that an entire SROC curve can be represented by model (1); the explanations above are instead meant as a motivation that the PHmeasure is not just another summary measure, but can be derived from a ROC modelling perspective. We envisage that each study, with associated pair of sensitivity and specificity, can be represented by a specific PHmodel, as illustrated in Figure 6.
Figure 6. Metaanalysis of BNP and heart failure: each study is represented by its own PH model (1) – illustrated for 3 studies.
We see indeed that each pair of sensitivity and specificity can be associated with its own ROC curve provided by
where , so that the curve (3) passes exactly through the point .
Comparison to other approaches. It remains to be seen how appropriate the suggested proportional hazards model is and how it compares to other existing approaches. We emphasize that in our situation we have assumed that there is only one pair of sensitivity and false positive rate per study i. Situations where several pairs per study are observed (such as in Aertgeerts et al.[23]) are rare. Hence, on the logscale for sensitivity and falsepositive rate, we are not able to identify any straight line model within a study with more than one parameter, since this would require at least two pairs of sensitivity and specificity per study; see also Rücker and Schumacher [24,25]. However, any oneparameter straight line model, such as the proposed proportional hazards model, is estimable within each study, although withinmodel diagnostics is limited since we are fitting the full within study model. Given that sample sizes within each diagnostic study are typically at least moderately large it seems reasonable to assume a bivariate normal distribution for and with means logp and log(1−q) as well as variances and , respectively, and covariance σ with correlation ρ=σ/(σ_{p}σ_{q}). This is very similar to the assumptions in the approach taken by Reitsma et al.[17] (see also Harbord et al.[19]), with the difference that we are using the logtransformation whereas in Reitsma et al.[17] logittransformations are applied. Then, it is a wellknown result that the mean of the random variable (having unconditional mean logp) conditional upon the value of the random variable (having unconditional mean log(1−q)) is provided as
which can be written as where α= log(p)−θ log(1−q) and . This is an important result since it means that, in the logspace, sensitivity and false–positive rate are linearly related. Furthermore, if α is zero, the proportional hazards model arises.
The question then arises why not work with a straight line model
The answer is that such a model is not identifiable since we have only one pair of sensitivity and specificity observed in each study and it is not possible to uniquely determine a straight line by just one pair of observations since there are infinitely many possible lines passing through a given point in the logp – log(1−q) space. However, the proportional hazards model as a slopeonly model is identifiable and it is more plausible than other identifiable models such as the intercept–only model. Clearly, a logistictransformation would be more consistent with the existing literature [14,15] than the logtransformation. However, both models would give a perfect fit (within each study) since there are no degrees of freedom left for testing the model fit. The situation changes when there are repeated observations of sensitivity and specificity per study available. However, these metaanalyses with repeated observations of sensitivity and specificity according to cutoff value variation are extremely rare.
A mixed model approach
With the motivation of the previous sections in mind, we assume that k diagnostic studies are available with diagnostic accuracies where
We assume the following linear mixed model for :
where x_{i} is a known covariate vector in study i, δ_{i} is a normally distributed random effect δ_{i}∼N(0,τ^{2}) with τ^{2} being an unknown variance parameter, and is a normally distributed random error with variance known from the i−th study.
There are several noteworthy points about the mixed model (7). The response is measured on the logscale, where the transformation improves the normal approximation and also brings the diagnostic accuracy into a wellknown link function family: the complementary loglog function. The difference of the probability for a positive test in the groups with and without the condition is measured on the complementary loglog scale. The fixed effect part involves a covariate vector x which could contain information on study level such as gold standard variation, diagnostic test variation, or sample size information. It should be noted that there are two variance components, τ^{2} and . It is important to have information on the second variance component. If the second component is unknown, even under the assumption of homogeneity , the variance component model would not be identifiable. Hence, we need to devote some effort to derive expressions for the within study variances; this can be accomplished using the δ−method as discussed in the next section.
Within study variance. Let us consider (ignoring the study index i for the sake of simplicity)
and apply the δ−method. Recall that the variance VarT(X) of a transformed random variable T(X) can be approximated as [ T^{′}(E(X))]^{2}Var(X) assuming that the variance Var(X) of X is known. Applying this δ−method twice gives
and
so that the within study variance for the ith study is provided as
We acknowledge that the above are estimates of the variances of the diagnostic accuracy estimates, but are used as if they were the true variances.
Some important cases. If there are no further covariates, two important models are easily identified as special cases of (7). One is the fixed effects model
and the other is the random effects model
which have gained some popularity in the metaanalytic literature.
Results
Case study on MMSE and dementia
We illustrate the approach with an example and revisit a meta–analysis by Mitchell [26] on the diagnostic accuracy of the minimental state examination (MMSE) as a diagnostic test for the detection of dementia and, more recently, mild cognitive impairment (MCI). In this meta–analysis 38 studies were included and the entire data are reproduced in Table 2. We are interested in the question: is there a difference in diagnostic accuracy of the MMSE in the detection of dementia and MCI, as Figure 7 suggests.
Table 2. Metaanalysis of the diagnostic accuracy of the minimental state examination (MMSE) and dementia or mild cognitive impairment (MCI) as reference standard; TP = true positives, FN = false negatives, FP = false positives, TN = true negatives
Figure 7. SROC diagram for the meta–analysis of MMSE and dementia or MCI as reference standard.
We use proc mixed from the SAS software, version 9.2 for Windows [27], for the analysis (see also Table 3). The values of the dependent variable are easily constructed from Table 2. We are interested to see if there are differences in accuracy for diagnosing MCI compared to diagnosing dementia. Hence we have constructed a covariate condition which takes the value 1 if the study concerns MCI as condition and 0 if the study is on dementia. Since we have fixed withinstudy variances, we need to tell proc mixed to incorporate this appropriately; this can be accomplished by using a weight, . The random option induces a random effect (here study) with associated variance component τ^{2}, which is estimated. However, SAS proc mixed will automatically fit a withinstudy variance component (on top of the provided variances). To circumvent this mechanism, the option parms (1) (1) /hold=2 is used where the term hold=2 fixes the second variance component, corresponding to the withinstudy variance multiplier, to one. Note that the random effect modelling betweenstudy variation is described by a free variance parameter, τ^{2}. For this a starting value needs to be given: we have τ^{2}=1, although other choices are possible, e.g. τ^{2}=0, corresponding to the case of no heterogeneity between studies.
Table 3. SAS proc mixed adapted for meta–analysis of diagnostic accuracy study data
The results of the analysis are provided in Table 4. It can be seen that there is a significant effect of condition (dementia/MCI) on the diagnostic accuracy, with diagnostic accuracy being significantly higher in studies with patients having dementia in comparison to the diagnostic accuracy in studies with patients having mild cognitive impairment. Nevertheless, not all heterogeneity is explained by this covariate as the random effect (study effect) still remains significant, as the bottom part of Table 4 shows.
Table 4. Analysis of effects for the metaanalysis of the diagnostic accuracy of the minimental state examination (MMSE) and dementia or mild cognitive impairment (MCI) as reference standard
The inference is based here on a procedure called the Wald test. The estimated parameter value is divided by its estimated standard error, and the result is given in column four in Table 4. The likelihood ratio test may be considered as an alternative. It is defined as two times the difference of the loglikelihood including the effect of interest and the loglikelihood not including the effect of interest. For the effect of condition in Table 4, we find a value of 6.8 for the likelihoodratio test. The Wald test is asymptotically standard normal under the nullhypothesis of absence of effect, whereas the likelihood ratio test statistic is asymptotically chisquared distributed with degrees of freedom equal to the number of parameters associated with the effect considered (in this case one). It is wellknown that the likelihood ratio test is more powerful. Here, both tests provide similar pvalues, with 0.0091 for the likelihood ratio test and 0.0069 for the Wald test; this confirms the significance of the effect (dementia/MCI) on the diagnostic accuracy.
It is trivial to construct the associated SROC curves from Table 4. We find
Note that the likelihood ratio test as well as the Wald test need modification in situations where the null hypothesis is part of the boundary of the alternative such as when testing H _{0}:τ^{2}=0. In this case, the asymptotic null distribution of the likelihood ratio test statistic is no longer χ^{2} with 1 df but rather a mixture of a twomass distribution giving equal weights 0.5 to the onepoint mass distribution at 0 and a χ^{2} with 1 df [28]. Practically, this means that standard 2sided pvalues have to be divided by 2.
Case study on MOOD and depressive disorders
The MOOD module of the Patient Health Questionnaire (PHQ9) has been developed to screen and to diagnose patients in primary care with depressive disorders. The instrument consists of 9 questions, each scored from 0 to 3 points with a total score ranging from 0 to 27. In a meta–analysis of the diagnostic accuracy of MOOD, Wittkampf et al.[29] included 12 studies. These studies used either a cutoff of 10 (referred to here as “summary score”) or a more complex evaluation algorithm (“algorithm”). The complete data are listed in Table 5 and the associated SROC diagram is given in Figure 8. The impression from the graph is that the cutoff of 10 used by the summary score has a higher diagnostic accuracy than the alternative.
Table 5. MetaAnalysis of the diagnostic accuracy of the MOOD module and depression in patients in primary care as reference standard; TP = true positives, FN = false negatives, FP = false positives, TN = true negatives
Figure 8. SROC diagram for the meta–analysis of MOOD and depression in patients in primary care.
The presence or absence of a cutoff value effect is now more formally investigated using a covariate cutoff, which is zero when the summary score with a cutoff value of 10 is used and one otherwise. The results are presented in Table 6. It can be seen that the covariate cutoff level “summary score” is associated with a higher diagnostic accuracy, although, as seen from the Wald statistics provided in column four of Table 6, the effect is not significant. We see a significant random effect (study; adjusted pvalue 0.0274; see comment at the end of section “Case study on MMSE and dementia”), which indicates that the random study effect is needed in the analysis. It is not really surprising that the covariate cutoff is not significant, since the concept of the SROC is designed to accommodate the cutoff value variation. We will take up this point in the next section.
Table 6. Analysis of the cutoff effect for the metaanalysis of the MOOD module and depression in patients in primary care
Discussion
Global versus local criteria
We have focussed on the PH measure so far, as it provides an appropriate measure for comparing SROC curves globally, in the sense that cutoff value variation will not necessarily effect the estimate of the SROC curve. The situation is illustrated in Figure 9.
Figure 9. Different cutoff values with associated sensitivities and specificities on the same SROC curve with different Euclidean distances; the point on the circle has shortest Euclidean distance to the upper left vertex of the SROC diagram as indicated by the circle.
Evidently, different cutoff values are associated with the same value of logθ, hence, the PH measure logθ is not the best measure to discriminate different cutoff values. This is not surprising, since the SROC curve is a concept designed for assessing the diagnostic accuracy of a diagnostic test globally, in the sense that it adjusts for different cutoff values. Hence, a measure that assesses local performance of the diagnostic is needed. Assuming that every cutoff value used in the meta–analysis is clinically meaningful, we suggest use of the (squared) Euclidean distance to the upper left corner (0,1) of the ROC diagram as a more meaningful measure to compare cutoff values:
where and . Each point in the SROC diagram has a unique circle with center (0,1) that passes through this point. In Figure 9, one such circle is illustrated which also has the smallest Euclidean distance among the three available points (since it has smallest radius among the three possible points with associated circles). In the following, we will consider the criterion (14) as an alternative criterion to choose the cutoff value.
Since we have changed the criterion, we need to determine the associated withinstudy variances. This can be accomplished easily, using the δmethod once more, to obtain
where we have ignored study indexes for the the sake of simplicity. Using this criterion, we see in Figure 9 that cutoff values can vary considerably in their diagnostic accuracy, despite having identical diagnostic accuracy at a global level. We reanalyze the meta–analysis of MOOD and depression with respect to the (squared) Euclidean distance and provide the results in Table 7.
Table 7. Analysis of the cutoff effect for the metaanalysis of the MOOD module and depression in patients in primary care
Evidently, both criteria lead to the same conclusion, namely that using the summary score with a cutoff value of 10 leads to the higher diagnostic accuracy (although the effect is not significant). It might also be worthwhile looking at the results of the likelihood ratio test: for the PHmeasure as the outcome variable, the likelihood ratio test provides a value of 1.5; for the Euclidean distance, the value of the likelihood ratio test is 1.7, confirming the nonsignificance of the effect. Nevertheless, the analysis shows that the cutoff value of 10 provides the higher diagnsotic accuracy.
Meta–analysis of magnetic resonance spectroscopy and prostate cancer
This case study provides an example where the use of a global or local criterion leads to a different conclusion. Magnetic resonance spectroscopy has the ability to discriminate between prostate cancer and benign prostatic hyperplasia, based on reduced citrate and elevated choline in the cancerous region. The diagnostic test works on a voxel of signal intensity ratios of (choline+creatine)/citrate. Two cutoff points are in use: <0.75 and <0.86. The results collected in a meta–analysis by Wang et al.[30] include 12 studies, as presented in Table 8; the associated SROC diagram is presented in Figure 10. From the graph, there is no obvious choice for the “best” cutoff value.
Table 8. Metaanalysis of the magnetic resonance spectroscopy and prostate cancer; TP = true positives, FN = false negatives, FP = false positives, TN = true negatives
Figure 10. SROC diagram for the metaanalysis of the magnetic resonance spectroscopy and prostate cancer.
The fixed effects parts of the mixed model analysis, using the global PH measure and the local Euclidean measure as criteria, are presented in Table 9. It is interesting to note that the focus of the analysis, global or local, is an important part of the analysis. Globally, the better diagnostic accuracy is given by the cutoff value of 0.75, whereas better local performance is achieved with a cutoff value of 0.86, although neither analysis is significant.
Table 9. Analysis of the cutoff effect for the metaanalysis of the magnetic resonance spectroscopy and prostate cancer
PH measure and positive likelihood ratio
Another frequently used diagnostic measure is the positive likelihood ratio, defined as the ratio of sensitivity to false positive rate P(T=1D=1)/P(T=1D=0) or p/(1−q). It is different to the PH measure in that the ratio is taken on the logscale: θ= logp/ log(1−q). Furthermore, if reexpressed as models, the positivelikelihood ratio corresponds to p=θ^{′}(1−q), a straight line with no intercept, whereas the the PH measure corresponds to p=(1−q)^{θ}, a straight line on the logscale with no intercept. The positive likelihood ratio is a natural measure since it transfers the concept of relative risk (risk of a positive test in the diseased group to the risk of a positive test in the nondiseased group) to the diagnostic setting. However, it is less suitable as an (S)ROC model since it does not provide a function which connects the lower left vertex with the upper right vertex in the ROC diagram (which, in contrast, the PHmodel does provide).
Conclusions
The approach presented here is attractive since it is based on a simple measure of diagnostic accuracy per study, namely the ratio of logsensitivity to logfalsepositive rate. It also embeds the diagnostic metaanalysis problem into the wellknown and much used mixed model setting. In particular, the analysis of effects of observed covariates on the diagnostic accuracy can easily be incorporated.
Controversies in the meta–analysis of diagnostic studies usually focus on comparability of studies. Study types might be case–control, cohort, cross–sectional or other. Studies might differ in the gold standard, severity of disease, or in the application of the diagnostic test. Patient populations might differ across studies, as might the cutoff value (defining positivity of the diagnostic test). All these different aspects, if observed, can be easily incorporated and analyzed as fixed effects in the special mixed model suggested here.
The occurrence of heterogeneity in the metaanalysis of diagnostic studies is more the rule than the exception; it is thus important to quantify the heterogeneity across studies due to the different sources. The approach provided here offers a more detailed investigation of heterogeneity according to the various observed sources and a residual heterogeneity (measured by τ^{2}). This might allow us to construct a measure of relative residual heterogeneity, which might help to assess how trustworthy the results of a given metaanalysis may be. This will be investigated in future research.
In a recent study on the metaanalytical evaluation of coronary CT angiography studies, Schuetz et al.[31] investigated the problem of nonevaluable results that occur in the individual studies. They conclude that diagnostic accuracy measures change considerably depending on how nonevaluable results are treated. In fact, they conclude that
parameters for diagnostic performance significantly decrease if nonevaluable results are included by a 3×2 table for analysis (intention to diagnose approach).
Twentysix studies were included in their metaanalysis with a wide range of nonevaluable results from 0 to 43. Using the approach suggested here, it would be very easy to analyze the effect of nonevaluable results on the diagnostic accuracy by including the amount of nonevaluable results per study as a fixed effect in the proposed mixed model.
Competing interests
The authors declare that they have no competing interests.
Authors’ contributions
SC carried out all statistical and computing analysis. WB collected and prepared all metaanalytic data sets. HH conceived the theoretical modelling work and DB drafted the idea of the approach and finalized the manuscript. All authors read and approved the final manuscript.
Acknowledgements
This work was funded by the German Research Foundation (GZ: Ho1286/72). We are very grateful to two referees for their insightful, inspiring and helpful comments. Thanks go also to Dr Sean Ewings for a careful and critically reading of the manuscript.
References

Hasselblad V, Hedges LV: Metaanalysis of screening and diagnostic tests.
Psychol Bull 1995, 117:167178. PubMed Abstract  Publisher Full Text

Sutton AJ, Abrams KR, Jones DR, Sheldon TA, Song F: Methods for MetaAnalysis in Medical Research. New York: Wiley; 2000.

Deeks JJ: Systematic reviews of evaluations of diagnostic and screening tests. In Systematic Reviews in Health Care: MetaAnalysis in context, vol.14, pp.248282. London: BMJ Books; 2007.

MetaAnalysis. New Developments and Applications in Medical and Social Sciences. Hogrefe & Huber: Göttingen; 2003.

Pepe MS: Receiver operating characteristic methodology.
J Am Stat Assoc 2000, 95:308311. Publisher Full Text

Pepe MS: The Statistical Evaluation of Medical Tests for Classification and Prediction. Oxford: Oxford University Press; 2003.

Holling H, Böhning W, Böhning D: Metaanalysis of diagnostic studies based upon SROCcurves: a mixed model approach using the Lehmann family.
Stat Modelling – Int J 2012, 12:347375. Publisher Full Text

Swets JA: Signal Detection Theory and ROC Analysis in Psychology and Diagnostics Collected Papers. New York, London: Psychology Press Taylor & Francis Group; 1996.

Doust JA, Glasziou PP, Pietrzak E, Dobson AJ: A system reviews of diagnostic accuracy of natriyretic peptides for heart failure.

Youden D: Index for rating diagnostic tests.
Cancer 1950, 3:3235. PubMed Abstract  Publisher Full Text

Glas AS, Lijmer JG, Prins MH, Bonsel GJ, Bossuyt PMM: The diagnostic odds ratio: a single indicator of test performance.
J Clin Epidemiol 2003, 56:11291135. PubMed Abstract  Publisher Full Text

Moses LE, Shapiro D, Littenberg B: Combining independent studies of a diagnostic test into a summary ROC curve: dataanalytic approaches and some additional considerations.
Stat Med 1993, 12:1293316. PubMed Abstract  Publisher Full Text

Littenberg B, Moses LE: Estimating diagnostic accuracy from multiple conflicting reports: A new metaanalytic method.
Med Decis Making 1993, 13:313321. PubMed Abstract  Publisher Full Text

Rutter CM, Gatsonis CA: A hierarchical regression approach to metaanalysis of diagnostic test accuracy evaluations.
Stat Med 2001, 20:28652884. PubMed Abstract  Publisher Full Text

Walter SD, Macaskill P: SROC curve. In Encyclopedia of Biopharmaceutical statistics. New York: Marcel Dekker; 2004.

Van Houwelingen HC, Zwinderman KH, Stijnen T: A bivariate approach to metaanalysis.
Stat Med 1993, 12:22732284. PubMed Abstract  Publisher Full Text

Reitsma JB, Glas AS, Rutjes AWS, Scholten RJPM, Bossuyt PM, Zwinderman AH: Bivariate analysis of sensitivity and specificity produces informative measures in diagnostic reviews.
J Clin Epidemiol 2005, 58:982990. PubMed Abstract  Publisher Full Text

Arends LR, Hamza TH, van Houwelingen JC, HeijenbrokKal MH, Hunink MGM, Stijnen T: Bivariate random effects metaanalysis of ROC curves.
Med Decis Making 2008, 28:621638. PubMed Abstract  Publisher Full Text

Harbord RM, Deeks JJ, Egger M, Whiting P, Sterne JAC: A unification of models for metaanalysis of diagnostic accuracy studies.

Liu X: Classification accuracy and cut point.
Stat Med 2012, 31:26762686. PubMed Abstract  Publisher Full Text

Le CT: A solution for the most basic optimization problem associated with an ROC curve.
Stat Methods Med Res 2006, 15:571584. PubMed Abstract  Publisher Full Text

Gönen M, Heller G: Lehmann family of ROC curves.
Med Decis Making 2010, 30:509517. PubMed Abstract  Publisher Full Text

Aertgeerts FB, Kester A: The value of the CAGE in screening for alcohol abuse and alcohol dependence in general clinical populations: a diagnostic metaanalysis.

Rücker G, Schumacher M: Letter to the editor.
Biostatistics 2009, 10:806807. PubMed Abstract  Publisher Full Text

Rücker G, Schumacher M: Summary ROC curve based on a weighted Youden index for selecting an optimal cutpoint in metaanalysis of diagnostic accuracy.
Stat Med 2010, 29:30693078. Publisher Full Text

Mitchell AJ: A MetaAnalysis of the accuracy of the minimental state examination in the detection of dementia and mild cognitive impairment.
J Psychiatr Res 2009, 43:411431. PubMed Abstract  Publisher Full Text

SAS Institute Inc: SAS/STAT(R) 9.2 User’s Guide, Second Edition. Cary, NC, USA: SAS Insitute Inc.; 2008.

Self SG, Liang KY: Asymptotic properties of maximum likelihood estimators and likelihood ratio tests under nonstandard conditions.
J Am Stat Assoc 1987, 82:605610. Publisher Full Text

Wittkampf KA, Naeije L, Schene AH, Huyser J, van Weet HC: Diagnostic accuracy of the mood module of the patient health questionnaire: a systematic review.
Gen Hosp Psychiatry 2007, 29:388395. PubMed Abstract  Publisher Full Text

Wang P, Guo YM, Qiang YQ, Duan XY, Zhang QJ, Liang W: A metaanalysis of the accuracy of prostate cancer studies which use magnetic resonance spectroscopy (MRS) as a diagnostic tool.
Korean J Radiol 2008, 9:432438. PubMed Abstract  Publisher Full Text  PubMed Central Full Text

Schuetz GM, Schlattmann P, Dewey M: Use of 3Œ2 tables with an intention to diagnose approach to assess clinical performance of diagnostic tests: metaanalytical evaluation of coronary CT angiography studies.
BMJ 2012, 345:6717. Publisher Full Text
Prepublication history
The prepublication history for this paper can be accessed here: