Center for Clinical Evidence Synthesis, Institute for Clinical Research and Health Policy Studies, Tufts Medical Center, Boston, MA, 02111, USA
Department of Computer Science, Tufts University, Medford, MA, 02155, USA
Biostatistics Research Center, Institute for Clinical Research and Health Policy Studies, Tufts Medical Center, Boston, MA, 02111, USA
Abstract
Background
Metaanalysis is increasingly used as a key source of evidence synthesis to inform clinical practice. The theory and statistical foundations of metaanalysis continually evolve, providing solutions to many new and challenging problems. In practice, most metaanalyses are performed in general statistical packages or dedicated metaanalysis programs.
Results
Herein, we introduce MetaAnalyst, a novel, powerful, intuitive, and free metaanalysis program for the metaanalysis of a variety of problems. MetaAnalyst is implemented in C# atop of the Microsoft .NET framework, and features a graphical user interface. The software performs several metaanalysis and metaregression models for binary and continuous outcomes, as well as analyses for diagnostic and prognostic test studies in the frequentist and Bayesian frameworks. Moreover, MetaAnalyst includes a flexible tool to edit and customize generated metaanalysis graphs (e.g., forest plots) and provides output in many formats (images, Adobe PDF, Microsoft Wordready RTF). The software architecture employed allows for rapid changes to be made to either the Graphical User Interface (GUI) or to the analytic modules.
We verified the numerical precision of MetaAnalyst by comparing its output with that from standard metaanalysis routines in Stata over a large database of 11,803 metaanalyses of binary outcome data, and 6,881 metaanalyses of continuous outcome data from the Cochrane Library of Systematic Reviews. Results from analyses of diagnostic and prognostic test studies have been verified in a limited number of metaanalyses versus MetaDisc and MetaTest. Bayesian statistical analyses use the OpenBUGS calculation engine (and are thus as accurate as the standalone OpenBUGS software).
Conclusion
We have developed and validated a new program for conducting metaanalyses that combines the advantages of existing software for this task.
Background
Systematic reviews of randomized controlled trials or epidemiological studies have emerged as a key source of evidence across medical disciplines
Metaanalysis is usually performed using computer programs. Herein we present a new program for the Microsoft Windows operating system, MetaAnalyst, and report on its testing versus other widely used and accepted software. MetaAnalyst features an easy and intuitive graphical user interface and has a spreadsheetbased layout. The program was developed by the Tufts Evidencebased Practice Center under contract with the US Agency for Healthcare Research and Quality (AHRQ). It is available for use by the AHRQdesignated Evidencebased Practice Centers for performing metaanalyses in their evidence reports. Additionally, the software is now being made available to all interested investigators worldwide at no cost. The latest version can be obtained from
Existing software
Metaanalysis can be performed in various general statistical and numerical analysis environments (e.g., Stata, R/Splus, Octave/MATLAB), or in dedicated programs (e.g., the Microsoft DOS version of MetaAnalyst, Comprehensive MetaAnalysis, RevMan, MIX
Two of the most popular dedicated metaanalysis packages are Comprehensive MetaAnalysis and MIX. The former is a commercial product, costing $1295 for a licence, while the latter is a free plugin for the commercial Microsoft Excel package. Both feature intuitive, spreadsheet interfaces for data entry, and provide numerical and graphical output in standard formats. However, both implement only basic methods for the metaanalysis of binary and continuous data (Table
Comparison of MetaAnalysis Software
Stata/WinBUGS
R/OpenBUGS
MIX
CMA
RevMan
MetaAnalyst
Operating system
Windows, Mac, Linux
Windows, Mac, Linux
Windows
Windows
Windows, Mac, Linux
Windows
Version
10
2.6
1.7
2
5.0.18
Beta 1.0
Price
$785*
FREE
FREE
$1,295
FREE
FREE
Import data
✓
✓
✓
✓
✓
Metaanalysis interface/routines
Macros
Macros
Dedicated
Dedicated
Dedicated
Dedicated
Metaregression
✓
✓
∅
✓
∅
✓
Single group
✓
✓
✓^{†}
✓
✓^{†}
✓
Fixed effects
✓
✓
✓
✓
✓
✓
Random effects
✓
✓
✓
✓
✓
✓
Multilevel models
✓
✓
∅
∅
∅
✓
Random effects metaregression
✓
✓
∅
∅
∅
✓
Bayesian models
✓
✓
∅
∅
∅
✓
Cumulative metaanalysis
✓
✓
✓
✓
∅
✓
Subgroup analysis
✓
✓
∅
✓
✓
✓
Small study effects (Publication bias tests)**
✓
✓
✓
✓
∅
∅
Binary data
✓
✓
✓
✓
✓
✓
Continuous data
✓
✓
✓
✓
✓
✓
Diagnostic test data
✓
✓
∅
∅
✓
✓
Multivariate
✓
✓
∅
∅
∅
✓
Documentation of methods
✓
✓
✓
✓
✓
✓
Forest plot
✓
✓
✓
✓
✓
✓
Funnel plot
✓
✓
✓
✓
✓
✓
SROC
✓
✓
∅
∅
✓
✓
HSROC  bivariate metaanalysis
✓
✓
∅
∅
∅
✓
Point and click plot editing
✓
∅
∅
∅
∅
✓
Programming capabilities
✓
✓
∅
∅
∅
✓
Leave one out sensitivity
✓
✓
∅
✓
∅
✓
Results format
RTF
RTF
MS Excel
RTF, PowerPoint
RevMan
PDF, RTF, image files
* Intercooled Stata for a single processor, educational licence. Other versions are more expensive.
^{† }One group analysis available for continuous outcomes only.
**Recent empirical research challenges the routine use of the socalled "publication bias diagnostics"
RTF: Rich text format (can be opened in MS Word); SROC: summary receiver operating characteristic curve
As shown in Table
Implementation
MetaAnalyst is written primarily in C#, and runs atop Microsoft's .NET framework. The .NET framework allowed rapid development of an intuitive Windowsbased user interface. Data entry and management follows the familiar Microsoft Excel^{©}like spreadsheet layout. We use specialized opensource software libraries to create plots (Zedgraph library for graphs and charts
The design of MetaAnalyst is based on the ModelViewControl design pattern
Example call to the backend from scripting environment
Example call to the backend from scripting environment.
For Bayesian analyses, we invoke OpenBUGS
Schematic depiction of MetaAnalyst/BUGS interaction
Schematic depiction of MetaAnalyst/BUGS interaction.
Results
Methods Implemented
Statistical methods
Unlike other dedicated metaanalysis packages, MetaAnalyst integrates the capabilities to perform metaanalyses of binary or continuous outcomes and diagnostic or prognostic tests, combining the functionality of software such as MIX and MetaDiSc. For each of these types of outcomes, we have implemented standard metaanalysis routines, as well as some more advanced ones. Table
Methods available in MetaAnalyst
Fixed
Random
Bayes
IV*
MH
Peto
DL
EM ^{†}
Odds ratio (OR)
√
√
√
√
√
√^{‡}
Risk ratio (RR)
√
√

√
√
√^{‡}
Risk difference (RD)
√
√

√
√
√^{‡}
Proportion**
√


√
√
√^{‡}


WMD
√


√
√
√^{‡}
Hedge's
√


√
√
√^{‡}
Cohen's
√


√
√
√^{‡}
Glass' δ
√


√
√
√^{‡}
Mean**
√


√
√
√^{‡}


Specificity
√


√
√
√^{‡}
Sensitivity
√


√
√
√^{‡}
Accuracy
√


√
√
√^{‡}
Positive predictive value (PPV)
√


√
√
√^{‡}
Negative predictive value (NPV)
√


√
√
√^{‡}
Positive likelihood ratio
√
√

√
√
√^{‡}
Positive likelihood ratio
√
√

√
√
√^{‡}
Diagnostic odds ratio
√
√

√
√
√^{‡}
Summary ROC curve
[weighted, unweighted]
[weighted]
Bivariate


√
Hierarchical SROC


√
*Fixed effects metaregression using weighted least squares is available here if there is at least one numerical covariate.
^{†}Random effects metaregression using an expectation maximization approach is available here if there is at least one numerical covariate.
^{‡}Control rate metaregression (linear or quadratic) is possible here (with or without adjusting for additional covariates, as deemed appropriate).
** e.g., for the metaanalysis of data from single arm studies.
 = not applicable, √ = available, DL: DerSimonian and Laird model, EM = Expectationmaximization, IV = inverse variance, MH: MantelHaenszel method, ROC = Receiver operating characteristic curve
Currently (as of version Beta 3.1) MetaAnalyst implements only one Bayesian model for each type of data (binary, continuous and diagnostic; for model details see
For detailed explanation of the statistical routines used, including handling of zerocells, please see our methods document at:
Exploratory and sensitivity analyses
MetaAnalyst automates
MetaAnalyst generates different graphical output suitable to the data at hand. Figures
Plots available in MetaAnalyst
Plots available in MetaAnalyst.
Plots available in MetaAnalyst
Plots available in MetaAnalyst.
User Interface
The GUI comprises two tabs; one for data entry and editing and the other for displaying the results of analyses (see Figure
Screenshot of the Data Entry screen in MetaAnalyst
Screenshot of the Data Entry screen in MetaAnalyst.
While editing data, the outcome and corresponding confidence intervals are updated dynamically. The outcome metric can be changed via a rightclick menu (e.g., from odds ratio to risk difference) and the outcomes will be recomputed automatically. While Figure
The user can also provide additional numeric and string variables that describe characteristics of the analyzed studies. By convention, useradded numeric variables are termed
Useradded string variables are termed
Studies can be included and excluded from a particular analysis by selecting/deselecting the corresponding checkbox in the first column. Once the data is entered, the outcome metric set and the studies and covariates desired to be excluded from the analysis (if any) are deselected, users can perform an analysis via the drop down 'Analysis' menu, at which point they will be prompted with the dialogue shown in Figure
Binary analysis specifications
Binary analysis specifications.
After the analysis runs to completion, the results will be displayed in the results tab (Figure
Results tab
Results tab.
The lefthand side of the results tab shows a tree populated with collapsible parent nodes for each analysis that has been run ("Analysis 1" and "Analysis 2", in the figure). Each of these parent nodes have child nodes corresponding to the various tabular and graphical outputs associated with the analysis. Clicking on one of these child nodes, e.g., "Forest Plot", scrolls the corresponding graphic into view.
All of the tables and graphics can be copied to the user's clipboard via a rightclick menu (and subsequently pasted into other programs). Additionally, the tables and any text therein can be edited and formatted via an embedded editor. Moreover, the user can edit forest plots using the
Forest Plot Editing
Forest Plot Editing.
In addition to the interactive output shown in Figures
Validation
To validate the computational results, we systematically tested MetaAnalyst vs. the results from
Over the 11,803 analyses (over all methods and all metrics) for binary data and 6,881 analyses over continuous data, we recorded the minimum of the absolute and normalized differences between the outputs from Stata and MetaAnalyst, where the normalized difference is defined as Δ_{rel }=  Θ_{Stata } Θ_{MA}/Θ_{Stata}. Θ is any of the numerical output of the program such as a summary effect size for each metaanalysis metric and method, its variance, and the
Over the binary set of metaanalyses, the maximum discrepancy was 2.9 × 10^{6}. For continuous data analyses, the maximum discrepancy was 7.4 × 10^{5}. These maximum discrepancies appeared in metaanalyses with extreme between study heterogeneity, and are ascribed to rounding errors (version 1.86 of
As previously discussed, Bayesian analyses are run through OpenBUGS, and so the output is as thoroughly tested as OpenBUGS.
Testing for diagnostic test accuracy analyses is not as extensive, because we have not found a suitable reference scripting environment to test MetaAnalyst output against. However, the simple diagnostic test methods are based on weighted proportions (sensitivity, specificity), relative risks (likelihood ratios), odds ratios and regression (SROC, metaregression). These methods use the same computational algorithms as those for binary data and so have been tested. Two diagnostic methods remain to be sufficiently tested; bivariate metaanalysis of sensitivity and specificity for diagnostic tests and random effects SROC. These analyses are flagged as not thoroughly checked when they are requested from the user. However, these will soon be reconfigured to run in OpenBUGS so that they will be validated as well.
Discussion
In order to attain widespread use, metaanalysis software must be easy to use. In particular, requiring that users learn an entire language to run their analyses will prohibit general adaptation of a program. Dedicated metaanalysis programs such as MIX, Comprehensive Metaanalysis, and MetaDiSc are appealing due to their small learning curve. On the other hand, by their very nature, such programs are less flexible than general statistical packages. For example, they have no scripting functionality, which precludes their use for largescale empirical research or simulation studies. Further, they are not able to perform advanced analyses, such as bivariate diagnostic test metaanalyses, because they cannot maximize difficult likelihood functions, and they cannot be readily extended to include additional analytic options.
Conclusion
MetaAnalyst mitigates several of the weaknesses inherent to dedicated metaanalysis packages. It incorporates their easeofuse, while providing advanced analytic methods that can be implemented in packages such as Stata, R and SAS by a statistical programmer.
The current version of MetaAnalyst is made available free of charge to interested researchers. It runs on any version of Windows that is compatible with the .NET platform (comprising Windows 98, ME, NT 4.0, 2000, XP and Vista). We have already started development of a crossplatform completely opensource version of the software that uses the R statistical language, and will be readily modifiable and extendable by any interested party
Availability and requirements
An installer file for the latest version of MetaAnalyst has been provided as an additional/supplemental file for the peerreviewers [Additional File
MetaAnalyst installer. This is a Windows installer for the MetaAnalyst software described in this manuscript.
Click here for file
Competing interests
The authors declare that they have no competing interests.
Authors' contributions
BW developed MetaAnalyst, porting some code to the program that was originally written by CS. TT and BW designed the testing of MetaAnalyst, and BW performed all analyses. All authors interpreted the results. BW wrote the first draft of the manuscript which was critically commented on by all other authors. All authors have read and approved the final manuscript.
Acknowledgements
Prepublication history
The prepublication history for this paper can be accessed here: