Abstract
Multiple reaction monitoring mass spectrometry (MRMMS) with stable isotope dilution (SID) is increasingly becoming a widely accepted assay for the quantification of proteins and peptides. These assays have shown great promise in relatively high throughput verification of candidate biomarkers. While the use of MRMMS assays is well established in the small molecule realm, their introduction and use in proteomics is relatively recent. As such, statistical and computational methods for the analysis of MRMMS data from proteins and peptides are still being developed. Based on our extensive experience with analyzing a wide range of SIDMRMMS data, we set forth a methodology for analysis that encompasses significant aspects ranging from data quality assessment, assay characterization including calibration curves, limits of detection (LOD) and quantification (LOQ), and measurement of intra and interlaboratory precision. We draw upon publicly available seminal datasets to illustrate our methods and algorithms.
Keywords:
Multiple reaction monitoring mass spectrometry (MRMMS); stable isotope dilution (SID); quantification; interference detection; limits of detection and quantification; intra and interlaboratory precisionIntroduction
In the past decade, the scientific community has seen an uptick in the use of mass spectrometry (MS) for the quantification of proteins and peptides in complex biological matrices. However, the technique that is most frequently used in quantitative assays, selected reaction monitoring (SRM, plural form: multiple reaction monitoring, MRM) MS was first reported in 1979 during the introduction of the triple quadrupole (QqQ) mass spectrometer [1]. Initially used for the detection, identification and quantification of small molecules [28], the QqQ has become prolific in proteomics laboratories and a necessary tool for the quantification of peptides and proteins, especially for biomarker verification. Biomarker verification is a step in the proteomics pipeline in which candidate biomarkers that have been identified from unbiased discovery experiments are targeted by quantitative assays utilizing stable isotopedilution and MRMMS [9]. This manuscript will focus on the statistical characterization and evaluation of MRMMS assays arising from quantitative biomarker verification studies.
The power of the QqQ mass spectrometer comes from the inherent selectivity of its staged mass selection and detection. In the majority of quantitative MS experiments, the QqQ operates in SRM mode (plural form: multiple reaction monitoring, MRM). In this mode, as samples are ionized by electrospray ionization [10] and enter the instrument, the first quadrupole (Q1) is set to only allow the predefined m/z value of the precursor ion to pass into the second quadrupole, or the collision cell. In the collision cell, the selected ions enter a higher pressure region with argon or nitrogen gas, resulting in low energy collisions and fragmentation of the selected precursor ion into many product ions. Finally, only the preselected product ions with specific m/z values are allowed to pass through the third quadrupole (Q3) and on to the detector [1]. The result is a very selective means for separating the target ions away from everything that is being introduced into the mass spectrometer (i.e., through liquid chromatography or other sample introduction), and further detecting fragment ions of the target and reducing chemical noise from the sample. One of the benefits of MRMMS on a QqQ MS platform is the speed at which it is able to detect multiple transitions (Q1/Q3 pairs), which is on the order of 10 msec per transition or less, allowing high multiplexing capabilities. This ability can be harnessed for both the analysis of many peptides (10's100's) per assay, and the monitoring of many transitions per peptide. This ability is important because the identity of the peptide is reliant on the sparse few transitions that are detected and that discriminate it from other peptides or molecules in the sample. Therefore, a highly selective assay for a particular peptide would target several transitions, minimally three product ions. This results in three or more independent measures for a particular peptide target, which can make statistical analysis more complicated.
Due to the inherent instability of electrospray ionization, accurate and precise quantification is best achieved through the addition of a stable isotopelabeled standard (SIS) into the sample, an approach called isotope dilution [1119]. The most common internal standards have been 13C and/or 15Nlabeled peptide analogs, which introduce little chromatographic shift in reversedphase chromatography so that they coelute with the target peptide and are chemically identical to the target peptide, except for the mass difference. The isotopically labeled standard is spiked into the sample as far upstream in the sample handling process as possible. If isotopically labeled proteins are unavailable (such as uniformly 15Nincorporated proteins, or proteins with 13C and/or 15N modified amino acids such as arginine or lysine), then peptide analogs can be synthesized with isotopically labeled amino acids and spiked in pre or post enzymatic digestion of the sample. These peptide standards behave similarly to the target peptide with regards to chromatographic separation, ionization, and fragmentation. The intensity of the signals detected for the SIS peptide is then compared to the signals for the analyte peptide, and their peak areas (determined from the area under the curve of the extracted ion chromatogram, XIC, for each transition) are compared to generate a peak area ratio (PAR). When the SIS peptide is spiked into the sample in a known quantity, the PAR is multiplied by the SIS peptide amount and the analyte peptide concentration is determined. While using only 3 transitions for the detection and identification of a target analyte seems sparse, the chromatographic retention times of the analyte peptide and SIS are also paired to ensure the proper peptide is detected. Finally, another important criterion to ensure peptide identity is the fragment ion ratio for a given peptide. This concept was first described in the context of small molecules as the "branching ratio", where each time a small molecule was fragmented and multiple product ions were detected, the ratio of these ions to one another was consistent: the largest fragment was always the largest, the smallest was always the smallest, and so on, as long as no interferences were present and the concentration was within the linear range of detection [6]. In the context of peptides, this effect is also seen from the fragmentation along the peptide backbone, and ensuring this ratio is consistent between the peptide target and the IS provides another level of selectivity and can indicate the presence of interfering signal [20]. This topic is further discussed below in Section 5.
While quantitative MRMMS assays have been in practice for decades [38,1119,2126], this manuscript will focus on some more recent publications that use SIDMRMMS for the quantification of peptides in plasma or similar complex matrices [19,2126]. The first few examples describe the use of SIDMRMMS for the quantification of peptides from samples with complex biological matrices [1719]. In all cases, the work describes the use of SIS peptides as internal standards added to the sample matrix, sample analysis by MRMMS and the calculation of peptide amount present in the sample. These papers created a turning point in the use of SIDMRMMS in proteomics labs because they demonstrated the feasibility of simple assay development, throughput and precision in the quantification of target peptides present in complex samples.
The earlier publications on peptide quantification using SIDMS did not, in fact, have detailed sections on the statistical analysis of the quantitative data. Barr et al [16] report variances between MS run to MS run, or between digestion replicates, but did not discuss assay characteristics such as linear range or limits of detection and quantification. Gerber et al [17] briefly described the linearity of the assay between the concentration points assessed, but did not discuss reproducibility, the slope of the response curve or other metrics. Barnidge et al [18] showed the effect of equal weighting versus 1/x weighting when plotting the linear regression of the standard curve area versus concentration. Barnidge and Barr also discussed percent recovery of the peptide target from the proteolytic digest and sample handling, a topic that is further explored by Agger et al, for the quantification of apolipoproteins A1 and B [27]. The more recent publications have more detailed sections on these calculations [2426], but may still not be exhaustive enough to describe all aspects of calculations required to define an analytical assay for the newcomer. Therefore, this manuscript will consolidate many of the statistical and analytical approaches used to describe the quantitative aspects of SIDMRMMS assays.
One example from a recent studypublished in Nature Biotechnology, and hence referred to as the NBT studyevaluated the repeatability and reproducibility of SIDMRMMS across multiple labs for the quantification of 10 peptides from 7 proteins spiked into human plasma [28]. The overall NBT study was constituted by three (sub) studies. Study I is the peptidelevel spike, where synthetic peptides were spiked into a background sample matrix of digested plasma to generate a calibration curve between 1 fmol and 500 fmol per μL in 1 μg of digested plasma. Study II is the proteinlevel spike, in which an equimolar mixture of the 7 target proteins were digested together, and then spiked into the background of digested plasma. This phase was designed to determine the effect of protein digestion on peptide recovery and its contribution to assay variability. The third phase, Study III, was also a proteinlevel spike, but mimicked a "real world" biomarker assay in which an equimolar protein solution was spiked into neat plasma and all subsequent sample handling steps (denaturation, reduction, alkylation, digestion, desalt and addition of stable isotope labeled peptide standards) were conducted at the individual laboratories. In all three cases, target peptides (or proteins) were spiked in at 9 different concentrations (1 fmol500 fmol/μL in 1 μg/μL plasma) to generate a response curve, and the SIS peptides were spiked in at a fixed concentration of 50 fmol/μL in all samples including the blank, which consisted of digested plasma only. Eight laboratories participated in this study, seven of which used the same MRMMS platform (4000 QTRAP, ABSciex) and the remaining lab used a TSQ Quantum Ultra QqQ (Thermofisher). All labs used nanoflow chromatography and adhered to an SOP that was distributed to dictate sample handling and data acquisition. All data acquired from this study is available online (http://www.proteomecommons.org/tranche/ webcite, Tranche hash: bCKpfN0bl2ULLwCaIovXn/spuw4rYfJF6H/L+/6sHAKGzCsj4fzTD0RauJjAwf9baB8tI36HQ0izji2tupYAPM29P2cAAAAAAAT0iw==), and will be used in example calculations.
Additional studies have been reported that aim to target clinically relevant analyte concentrations of proteins in plasma [21,23,24,26]. Of importance in these assays is measurement precision, inter and intraassay reproducibility or coefficient of variation (CV), as well as accuracy, and limits of detection and quantification. The following sections will discuss calculations of these parameters and metrics and discuss the necessary experimental design, as well as several methods for calculation and statistical analysis of the data. Many of these algorithms and calculations will be illustrated using the NBT study.
Concepts and terminology for MRMMS assay characterization
MRMMS assays are characterized and evaluated based on several performance metrics and characteristics. Definitions of these metrics and associated terminology are laid out in this section, and will be used in the rest of the manuscript.
Data
Peak areas from each of the monitored transitions (usually 3 or more per peptide form) are determined based on the extracted ion chromatograms (measured ion intensity or count per chromatographic time).
Peak area ratio
In the context of SIDMRMMS, the peak area of each peptide analyte transition is divided by the peak area of the corresponding transition from the stable isotope labeled peptide form to obtain the peak area ratio.
Calibration curve
Generally, a calibration curve is represented by the analytical response versus the concentration of a given analyte. For SIDMRMMS experiments, a series of samples are analyzed that contain the sample matrix, a fixed concentration of SIS peptide, and varied concentration of the analyte peptide. The data are often plotted as "determined concentration" (or "measured concentration") versus "theoretical concentration" (see Figure 1), which will be used in the following examples. If the spike level of the internal standard is unknown, the data can also be plotted as peak area ratio versus theoretical concentration of the analyte. From the calibration curve, the slope is representative of the analytical sensitivity of the method for the analyte. Calibration curves are usually constructed so that the concentration spans at least two orders of magnitude and bracket the limit of detection and the upper limit of quantification.
Figure 1. (a) A set of calibration curves for 3 transitions of a wellbehaved peptide, with a relatively low LOD and a linear response region spanning three orders of magnitude (n = 4 for each transition at all concentration points). The left panel shows the data points on the linear scale along with the calibration curves. The panel on the right shows the data on a logarithmic scale so that all points are clearly visible, along with the calculated LOD. (b) A set of calibration curves for 3 transitions of a poorly behaved peptide with significantly inconsistent measurements, resulting in a high LOD, and a very restricted linear response region. (c) Regression lines fitted using ordinary least squares (OLS), weighted least squares (WLS) where each point is weighted by the inverse square of its theoretical concentration, robust regression (using the MMestimator) and weighted robust regression (MMestimator with inverse square weighting). Weighted regression lines for least squares regression and robust regression are almost identical, with the robust regression line coming close. OLS is most affected by a few outliers. (d) Example calibration curves (i) site 19 transition 37tr1_A in blue on the top, and (ii) site 56 transition 167tr3_A in green (bottom), that have ideal slopes (i.e., slope = 1, see Table 1 and Section 3) when the regression line is fit using logtransformed data, but clearly have slope > 1 in linear space. The black diagonal line represents slope = 1 in the panels above.
Precision
The precision of the data is determined by measuring replicates (3 or more) of one sample in the same manner. Precision is usually represented by standard deviation and coefficient of variation (CV).
Accuracy
Accuracy of the data is calculated (when the true concentration is known) as percent error.
Reproducibility
Synonymous with precision.
Limit of Detection
The lowest analyte concentration at which the signal is discernable from the noise (chemical noise, white noise, etc), or detected with confidence [29]. This can be calculated in a variety of ways, several of which will be described below.
Limit of Quantification
The lower limit of quantification refers to the lowest concentration of the analyte at which quantitative measurements can be made. The upper limit of quantification describes the highest concentration of analyte above which the signal departs from linearity. These two limits of quantification define the linear range of the assay.
Overview
MRMMS assays are used when the detection and quantification of specific analyte targets are required from a complex mixture. Stable isotopelabeled standard (SIS) peptides are used for a variety of reasons, but primarily act as an internal standard for the measurement of the peptide analyte and minimize the contributions of measurement variations due to chromatography, ionization, fragmentation and detection by MS. Assays can be designed to determine the Figures of Merit (limits of detection and quantification, precision and accuracy) by incorporating a calibration curve. The Figures of Merit can change due to differences in sample matrix (both nature of matrix and concentration) and factors affecting instrument sensitivity (chromatographic resolution, ionization, MS detection, etc). It is recommended to determine Figures of Merit if any of these factors are changed, and periodically on the same instrument, especially when analyzing samples that will be detected near the lower LOQ of the assay or when high precision is required.
In typical quantitative SIDMRMMS assays, the determined Figures of Merit are strongly influenced by system performance, both in terms of sensitivity and reproducibility from sample to sample. The noise contributed by the sample matrix also plays a major role in the magnitude of the calculated LOD and LOQ, and this is determined usually by several (at least three, preferably more) repeat measurements of matrix blanks (sample including everything except the target analyte) run throughout the course of the assay. With current technologies and on normally functioning nanoflow LCMRMMS systems, typical peptide LODs can be attained in the 100's amol per 1 ug equivalent protein digest load [21,28].
Calibration curve and regression analysis
The starting point of most quantitative assays is the calibration curve (Figure 1a and 1b). A range of analyte concentrations is analyzed in sample matrix to define the linear range of detection, limits of detection and quantification, and reproducibility of the assay. The calibration curve is designed to explore the possibility of endogenous signal in the matrix by multiple measurements of a blank sample (sample matrix and internal standard), and to also determine if there is interference in the analyte signal. In the case of SIDMRMMS, the SIS peptide is always present in the sample to normalize for any instrumentrelated issues that may affect analyte detection. It may be spiked in upstream in the workflow to also account for losses due to sample handling. The SIS peptide is spiked in at a known concentration to determine the concentration of the target analyte. With the target analyte spiked in at specific concentration and the SIS peptide spiked in at a fixed, known amount, the peak area ratio (PAR)the ratio of peak intensities of the analyte to standardis proportional to the concentration of the analyte. The measured concentration is then calculated as the product of the PAR and the concentration of the heavy standard:
When the target analyte is spiked in at various concentrations spanning a range of values, we obtain a set of measured concentrations corresponding to the spikedin theoretical concentration. A linear calibration curve relating the theoretical and measured concentration can be fitted:
An ideal calibration curve has a slope of 1 and an intercept of 0, indicating that the measured concentrations are in excellent agreement with the theoretical concentrations. An example of a wellbehaved peptide is shown in Figure 1a. Deviation of the slope from 1 indicates less than ideal response, and a significant nonzero intercept is indicative of the presence of endogenous analyte (Section 4.1 and 4.2).
A standard way to fit such calibration curves is ordinary least squares (OLS) regression [29]. While nonlinear calibration curves could also be fitted, such curves may tend to overfit the data, given the relatively small number of points used for the fit. Furthermore, the slope and yintercept of a linear regression fit have additional relevance from a quantification perspective.
MRMMS assays usually have a linear operating region where the intensity response linearly varies as the spikein concentration of the target analyte is varied. When a concentration curve is run, these limits of the linear region are not knownin fact determining this region is one of the goals of running the response curve. As such, we expect some analyte concentration values at the high and/or low end of the spectrum to lie outside the linear operating region. Therefore, when a linear OLS regression curve is fit, these points in nonlinear regions of the MRMMS response can unduly affect the regression fitting, resulting in skewed slope and yintercept values. Robust regression [30,31] is one approach used to address this problem. Robust regression fitting algorithms are resistant to outliers, and downweight points that deviate from the main bulk of data points, resulting in more reliable estimates. Some common methods for robust regression includes least median of squares (LMS) regression, least trimmed squares (LTS) regression [32] and the use of the MMestimator [33].
Furthermore, the variance of concentration measurements tends to increase at higher concentrations. In order to account for this trend data points are weighted according to the inverse square of the measurement or variance at that measurement level. This weighting can be used either with least squares regression (resulting in weighted least squares, or WLS regression) or with robust regression.
A comparison of OLS, WLS and robust regression with and without weighting for representative peptides in the NBT Study data are shown in Figure 1c. As is evident from the Figure, OLS is significantly influenced by the few points at the higher concentration level. Robust regression is more resistant to such outlier and tends to fit the regression line to follow the trend captured by a majority of points. The weighted regression lines for WLS and robust regression are much closer and are significantly less influenced by outliers and the high variance at the upper end of the calibration curve.
An alternative to WLS or weighted robust regression is to fit the regression line on logtransformed data [34], where the logarithms of both the measured and theoretical concentrations are used as the data points for the regression. The log transform converts heteroscedastic data into a homoscedastic set [29] thereby eliminating the dependence of variance on the concentration values. But regression lines fit in log space tend to downplay the deviation of the observed data from ideal, resulting in slopes that are closer to 1, and hence could provide an incorrect impression that the assay performance is better than it actually is (see Table 1, and Figure 1d). In addition, with log transformation, the intuitive interpretation of regression slope as sensitivity (see below) and intercept as endogenous level (see Section 4.1 and 4.2) are no longer valid, making it harder to interpret the results. Plotting the data on a loglog scale, on the other hand, enables more effective visualization since such a plot can naturally accommodate the concentration variation range normally used in such experiments (although the nonlogtransformed linear regression line cannot be conveniently plotted, and may appear as a nonlinear curve).
Table 1. Comparison of the regression analysis in linear and log space.
Traditionally in analytical chemistry, the slope of a linear regression is related to the sensitivity of an assay, which describes the ability of the assay to differentiate between small changes in analyte concentration [35]. Calibration sensitivity is equal to the slope of the calibration curve and is independent of concentration. This definition is the quantitative definition of sensitivity that is recognized by the International Union of Pure and Applied Chemists (IUPAC). Calibration sensitivity, however, does not take into account measurement precision. Analytical sensitivity (γ), described by Mandel and Stiehler [36], takes into account the precision of the measurements as well as the slope of the calibration curve: γ = m/s_{s}, where m is the slope of the calibration curve and s_{s }is the standard deviation of the measurement. In the context of peptide quantification, the slope of the calibration curve or the analytical sensitivity would easily aid in the selection of the best peptide targets, if there were several to choose from, and is also a good measure of whether or not similar instruments are measuring the target peptides with equal sensitivity. However, in addition to sensitivity, other figures of merit can be calculated from these values, including limit of detection, limit of quantification, and the amount of endogenous signal present in the blank [35].
Given the importance of the slope and intercept of the regression line for the calibration curve, an additional approach to evaluating the robustness and quality of the regression fit is to inspect the 95% confidence intervals for the slope and intercept. While many regression fitting algorithms provide an estimate of the standard error, the 95% confidence intervals can be easily calculated [33]. Bootstrap resampling is an alternative method for determining these limits [32] (also see Section 4.2 below).
Currently, less attention has been given to slope and yintercept, and are often not reported in publications, in lieu of R^{2 }[37]. R^{2 }is a measure of "explained variance", and does not provide an indication of the robustness of the regression fit. In addition to R^{2}, other factors of the regression fit including confidence intervals of the slope and intercept, residuals and a graph of the data should be examined before judging the quality of the regression line [38].
Limits of detection and quantification
Limits of detection (LOD) and quantification (LOQ) are important characteristics of any quantitative method, and in the MRMMS assay can be determined using the calibration curve. The intuition and definitions related to LOD and LOQ determination are described in Currie, 1968 [39]. There are a variety of methods to calculate LOD and LOQ, based on different aspects of the assay, and its intended application. A brief summary of the different classes of methods to determine detection and quantification limits is given below.
Blank Sample
In this approach, replicates of a blank samplei.e., a sample with the target analyte absentare used to determine the LOD and LOQ of the analyte [39]. Assuming that random measurement errors are normally distributed, and with 5% risk of incorrectly claiming detection in the absence of analyte (α) or missing the detection of analyte (β), LOD = 3.29 σ_{B }and LOQ = 3 × LOD = 10 σ_{B }where σ_{B }is the standard deviation of the blank sample.
Blank and Low Concentration Sample
The above method uses only the blank sample. In practice, the standard deviation of the blank sample could be significantly different from the standard deviation with the analyte present at a low level. To account for this possibility, LOD and LOQ calculation explicitly takes both the blank and the low concentration samples into account. A variation of the partly nonparametric method in [40] is to use a parametric approximation to account for a small number of replicates. This approach used in [28] is to calculate LOD as: LOD = μ_{B }+t_{(1β) }(σ_{B }+ σ_{S})/√n, where μ_{B }is the estimated mean of the blank samples, σ_{B }is the standard deviation of the blank samples and σ_{S }is the standard deviation of the low concentration samples. The equation assumes that analyte concentration is estimated using the mean of n replicates. Given the LOD, LOQ is estimated as 3 × LOD.
Calibration Curve
Instead of using just the blank or a low concentration point, this method uses the entire calibration curve to determine LOD. Also termed the calibration plot method, the standard error s_{yx }of the measured concentration (yestimate in the regression equation) is used in place of the standard deviation of the blank sample [41]. The LOD is then calculated as LOD = 3 s_{yx}/slope, and LOQ = 3 LOD.
RSD Limit
This approach [42] determines LOQ based on an accepted target value for relative standard deviation (RSD). RSD is the absolute value of the coefficient of variation (CV, the ratio of standard deviation to mean), and is expected to be small at the LOQ (typically less than 10% or 20%). The calibration curve is used to determine the RSD at each spikein level, and the RSD variation is modeled as a function of the analyte concentration using RSD = level × p_{1}^{(1  p}_{2}^{1og(level))}. The parameters p_{1 }and p_{2 }are determined using a fitting process, and the LOQ is that analyte concentration where the target RSD is achieved. The LOD is then reported as LOD = LOQ/3.
Figure 2 shows a comparison of the four methods for determining LOD and LOQ for representative peptides from the NBT Study. The calibration curve method generally tends to overestimate the LOD. The RSD Limit method tends to significantly underestimate LOD, resulting in extremely low LOD and LOQ value for many peptides (e.g, MYOLFT, CRP_ESD, LEPIND). The blank sample approach and the blank+low concentration sample method result in approximately similar values, with the blank sample method usually resulting in lower limits. Based on this evaluation, we have chosen the blank and low concentration sample algorithm as the preferred method for determining LOD and LOQ. This method is simple to implement and does not require the generation of the entire calibration curve.
Figure 2. A comparison of the various methods for calculating limit of detection (LOD, lower line in a pair) and limit of quantification (LOQ, upper line in a pair). The four methods compared are described in Section 4. The method using blank + low concentration sample is the most reliable, and consistently produces acceptable LOD and LOQ values for most practical purposes. The blank only method is a close second, but can underestimate the LOD and LOQ. The calibration curve method results in very conservative estimates, while the RSD limit method is inconsistent with some extremely low LOD and LOQ values.
Endogenous presence of analyte signal in the sample matrix is a difficult problem to deal with because it can complicate the calculation of LOD and LOQ. In addition, any signal derived from a spikedin analyte (as in a calibration curve experiment) is added to the endogenous signal. One experimental approach to circumvent this issue is to use a surrogate matrix, one that is very similar to the sample matrix, but does not contain the endogenous analyte. This can be difficult to find, especially in a sample matrix as complex as plasma with thousands of proteins ranging ten orders of magnitude of concentration [43]. Using plasma from a different species may even introduce new problems, such as interfering signal. An experimental alternative is to use the internal standard as a surrogate analyte and vary its concentration in the sample matrix to generate a standard curve [44]. While a reasonable alternative, this can cause questions to arise about the difference in the chemical noise that may be present at the m/z values for the surrogate analyte versus the real analyte. A stable isotopelabeled version of a peptide with a mass shift of 6 amu may have an entirely different level of chemical noise contributed by the sample matrix and electronic noise. Therefore, it is beneficial to consider a statistical means of estimating the endogenous level of analyte present in a sample matrix.
Endogenous levels of an analyte present in the LCMS matrix can be estimated using the linear range of the calibration curve resulting from a dilution or standard addition experiment [24]. Using the input data consisting of measured concentration values for corresponding calibration curve (theoretical, or true) concentrations, a robust linear fit using least median squares regression [32] is performed to determine the regression line yintercept for:
with the yaxis representing measured concentration and the xaxis representing theoretical concentration. The 99% confidence interval of the regression line yintercept is calculated using bootstrap estimation with repeated (1000 or more) resampling iterations [45]. The bootstrap estimation involves resampling with replacement from the data, in order to assess expected variation. For each resampled data set, the regression above is refit to recalculate the slope. The basic nonparametric confidence interval [45] for the slope is estimated as the range (m_{α}, m_{1α}), where m_{p }is the pth percentile of the slope in the resampled estimation, and (12α) is the confidence level. Usually, α = 0.025 or 0.005, for a confidence level of 95% or 99% respectively.
If the lower limit of the confidence interval is positive, then the analyte is deemed to have an endogenous level equal to the regression yintercept. If the lower 99% confidence interval is zero or negative, there is no expected endogenous level for that analyte. Once endogenous levels (if present) are calculated, the estimated LOD (and hence LOQ) in the absence of endogenous analyte is the difference of the calculated LOD (in the matrix) and the estimated endogenous level.
The method has been applied to selected transitions (the "best" transition that provides the lowest LOD) of the peptides for which MRM assays have been configured in [24]. Of the 28 peptide transitions analyzed, 3 are reported to have endogenous levels (see Table 2). The CRP peptides (bi0090 and bi0092) are expected to have endogenous levels. Thus, the only false positive is peptide bi0119 for MRP14, with an estimated endogenous level of 0.28 fmol. When all the 84 transitions monitored for these 28 peptides are considered, 16 transitions have reported endogenous levels. Of the 16, six of the transitions are for CRP peptides where an endogenous level is expected. Another 5 transitions have some form of interference (as is evident from the unusually large LOD/LOQ values of these transitions compared to other transitions for the respective peptide), and the interference is interpreted as an endogenous signalthese are situations that can be avoided by using AuDIT [20] (see Section 5). The remaining 5 transitions listed as having endogenous levels appear to be false positives.
Table 2. Summary of endogenous calculations for 28 peptides from 8 proteins.
There have been no observed instances of false negatives where an endogenous level was expected, and the method returned with a 0 endogenous level. If such instances are encountered, the confidence interval can be relaxed to 95% (from the currently used 99%) to enable overcoming false negatives (at the expense of more false positives).
Effective application of the method is dependent on having enough points on the concentration curve that are in the linear operating range. If there are too few points in the concentration curve, or if the endogenous level is so high that most of the concentration curve is nonlinear and affected by endogenous analyte, the method will fail. Theoretically, the method is likely to succeed if at least 50% of points on the concentration curve fall in the linear operating range (since least median squares regression has a breakdown point of 0.5).
Imprecision and interference in MRM MS
Inaccurate quantification in peptide MRMMS can result from many factors including incorrect peptide identification, matrix suppression, interference in one or more of the product ion transitions monitored, poor chromatography, MSinstrument related signal attenuation and saturation, and errors introduced during peak detection and integration (Table 3). Interferences in MRMMS from such sources are usually detected by painstaking and subjective manual examination of raw data [46]. Protein quantification for candidate biomarker verification in clinical proteomics [19,21,23,24,28,47] and other relatively high throughput applications increasingly require the ability to assay for many 10's to hundreds of proteins. Clearly, manual inspection of such data is no longer possible nor desired. The quality assessment of MRMMS data can be automated using AuDIT, an algorithm for Automated Detection of Inaccurate and imprecise Transitions in MRMMS for quantitative peptide analyses in any biological matrix, and can be used both in method development as well as for routine testing of patient samples [20]. AuDIT greatly increases the speed, reliability and accuracy of peptide identification and quantification from MRMMS data analysis. Figure 3 shows the analysis workflow for using AuDIT.
Table 3. Summary of potential problems encountered during analysis of SIDMRMMS data that often require manual identification or reintegration and their effect on the precision and accuracy of quantification.
Figure 3. Analysis work flow for isotope dilution MRMMS data with and without the use of AuDIT. After LCMRMMS analysis of samples, transition peaks are identified and integrated with software from either the mass spectrometer vendor or another supplier. (A), Flow of data with use of the automated algorithm. The statistical test identifies problem transitions from the variation in the relative ratios for the analyte and the SIS. The CV of the PARs is used as a filter to flag transitions with unacceptably large variation. (B), The current standard practice of careful manual inspection of all transitions by an expert. Adapted from Abbatiello, Mani, et. al., Clinical Chemistry, 56, 291305 [20].
AuDIT was designed to extensively use the concept of "relative ratio" or "branching ratio" [6] defined as the ratio of the peak areas for any 2 transitions of the same precursor. All analyte (or all SIS) transition peak areas are used in pairs to calculate the ratio. The relative ratio is unlike the PAR, which is calculated as the ratio of analyte to SIS peak areas for a given transition of a specified precursor. The AuDIT algorithm operates on preprocessed data and executes the following steps:
1. Use all transitions of a peptide (peak area from XICs) to calculate relative ratios by either the minimalpairs or allpairs method. The minimalpairs method calculates the relative ratio of a given transition by dividing its peak area by the peak area of one other transition from the same precursor. The allpairs method calculates ratios for all possible transition pairs generated from one precursor. This process is performed for each peptide analyte and its corresponding SIS so that the relative ratios of the analyte can be compared with the relative ratios of the SIS.
2. Apply the ttest to determine a pvalue for the hypothesis that the relative ratios for the analyte are different from the relative ratios of the SIS.
3. Use the BenjaminiHochberg falsediscovery rate method to correct the nominal ttest pvalues to account for multiple hypothesis testing [48].
4. Disaggregate the corrected pvalues for the relative ratios into combined pvalues for each transition. Each transition is used to calculate either 2 ratios for the minimalpairs method or n1 ratios for the allpairs method (where n is the total number of observed transitions for each peptide). Calculation of the pvalue for determining if a transition is problematic requires combining the pvalues for the respective relative ratios. Because the same peak areas from a given transition were used in calculating all its ratios, the resulting pvalues are not independent. These dependent pvalues are combined by means of a previously outlined methodology [49,50].
5. Calculate the CV for the PAR (analyte/SIS) from the results for all replicates in a transition for a given sample.
6. A transition is marked as "bad" if either the corrected combined pvalue for the transition is less than the pvalue threshold of 10^{5 }or if the CV is greater than the CV threshold of 0.2 (20%). Transitions not satisfying either of these conditions are classified as "good." Although the chosen thresholds work well for many data sets, they can be changed to finetune the algorithm as needed.
There are currently no automated methods for identifying transitions with interferences (or other problems, see Table 3) that can render them unsuitable for quantification. As such, the final decision on the quality of a transition is subjective and has relied entirely on expert review of the data [46]. In order to evaluate our algorithm for inaccurate and imprecise transition detection, we compared the results of AuDIT with that of an expert using a twostep process. In the preliminary phase, the expert looks at all the integrated extracted ion chromatograms, and creates an unbiased "prealgorithm" annotation which records any potential problems the expert observes like poor chromatography, inaccurate peak integration, etc., and is recorded at the level of the MRM transition. The data is then run through AuDIT, and the 'good' or 'bad' classification of the algorithm is compared with the expert's annotation ("global", in Table 4). In cases where AuDIT's decision and the expert's annotation disagree, the expert reevaluates those transitions to see if AuDIT's assessment (good or bad) is justifiablei.e., the actual observations of questionable data quality or interferences are such that the relative ratio may not be affected (and hence the transition may be used for quantification), or vice versa. This final phase of expert review creates a "postalgorithm" annotation, and is performed with the same criteria and rigor as the first review, but primed for issues that might have been overlooked or wrongly assessed. This focused annotation is compared with AuDIT decisions to evaluate its efficacy in identifying inaccurate transitions, and the overall performance is summarized in Table 4.
Table 4. Validation of AuDIT.
A 2 × 2 contingency matrix is created to evaluate the performance of AuDIT on each dataset. Defining a 'positive' as a 'good' transition call, the Table shows the various elements of the contingency matrix. Algorithm performance is estimated using i) overall accuracy, ii) sensitivity and iii) specificity, as described in Table 4. A receiver operating curve (ROC) is used to evaluate the combined effect of incorporating pvalue and CV in the algorithm calculations for flagging inaccurate and imprecise peptides (Figure 4). The area under the ROC curve (AUC) is an indication of the quality of the classifier [51]. The ROC and AUC also show that the ttest and the CV jointly achieve significantly better performance than either measure alone. The AUC of the ROC curve for transition quality prediction using only the CV is less than 0.5, indicating that this modality is worse than a random predictor. The CV is affected only under specific circumstances, most of which are orthogonal to situations where a significant ttest score will be obtained. It is therefore imperative that both the ttest and the CV be used in order to derive an accurate predictor of imprecise or inaccurate transitions. The pvalue for these experiments was set to 10^{5 }and the CV value set to 0.2. Both the pvalue and CV thresholds are adjustable. While the CV was set to an arbitrary value of 0.2, a sensitivityspecificity curve (Figure 4b) was used to assess the effect of changing the pvalue threshold for comparison of the relative ratios of the fragment ions between the analyte and SIS peptides. As the pvalue threshold increases above 10^{5}, a concomitant decrease in sensitivity is observed. At pvalues lower than 10^{5}, the specificity of the algorithm decreases. Thus, a pvalue of 10^{5 }was selected as the optimum threshold for sensitivity and specificity of the algorithm for identification of inaccurate or imprecise transitions.
Figure 4. ROC curve and sensitivityspecificity plots summarizing performance of AuDIT in identifying inaccurate and imprecise transitions, as evaluated by an expert. AuDIT uses the ttest pvalue and the CV of the PAR (ratio of analyte peak area to SIS peak area) to detect problem transitions. (A), Both the pvalue and the CV are required to achieve acceptable performance (i.e., as indicated by AUC values in parentheses). (B), Specificity and sensitivity values achieved as the pvalue threshold is varied from 0 to 1 (with a fixed CV threshold of 20%). The chosen pvalue threshold of 10^{5 }used for all of the analyzed data is indicated by the red circle (sensitivity, 98%; specificity, 97%). The rainbow color bar (right y axis) keys the location of the pvalue threshold on the sensitivityspecificity curve. Adapted from Abbatiello, Mani, et. al., Clinical Chemistry, 56, 291305 [20].
AuDIT can be applied to data exported from most MRMMS analysis software, and can potentially be embedded into such applications to greatly reduce manual inspection and alert the researcher of potentially errant data at an early point in the data analysis, potentially allowing problematic samples that exhibited large CV values (for example, maybe caused by column degradation and poor peak shape) to be reacquired. In addition, incorporation of AuDIT into the MRMMS workflow would streamline the processing, likely resulting in more efficient generation of accurate and precise quantitative data from SIDMRMMS analyses.
The AuDIT software is available at http://www.broadinstitute.org/cancer/software/genepattern/modules/AuDIT.html webcite.
AuDIT provides a mechanism to evaluate SIDMRMMS data quality from the perspective of minimizing interferences to enable robust quantification. A complementary approach involves assigning quality scores to the MRMMS spectra in order to statistically define error rates for peptide identities, as implemented in mProphet [52]. mProphet uses characteristics of the transition peaks and the concept of "decoy peaks" (measured where no real peaks are present) to derive a composite discriminant score that statistically captures the quality and reliability of the MRMMS data for each peptide.
In addition to AuDIT and mProphet, other data analysis software packages possess features that help to evaluate the composite signal of all transitions measured for a peptide and its SIS to monitor for differences. Such features are available in Skyline [53], a vendor neutral data analysis program, by monitoring the signal contribution from each transition and enabling the user to compare it to that of the SIS peptide with the output in visual plots. PinPoint software (Thermo Fisher Scientific) also compares the fragment ion ratio of the light and heavy peptides to look for agreement and reports the fragment ion ratios for the light and heavy peptides, also with visual plots. These software features work well for detection of interfering signal in a transition from a given peptide, and through the use of visual plots enable rapid screening of large data sets with a variety of peptide targets.
Intra and interlaboratory variation
In order for MRMMS combined with stable isotope dilution to be used as an assay for quantitative measurement of proteins and peptides, the precision and variability of the assay needs to be characterized not only in a given laboratory, but also across multiple laboratories. Assessment of the intra and interlab variation of MRMMS assays was the primary goal of the NBT Study described in Section 2.
In the NBT study, intralaboratory variability and reproducibility in studies IIII were evaluated by comparing the measured concentrations to the theoretical concentrations across the range of spikedin analytes and determining the coefficient of variation (CV = standard deviation/mean) for these quantitative measurements. In addition to calculating CV, graphical visualization [54] can assist in analysis and provide insight on variation across concentration levels, study phases and different laboratories. Figure 5a shows measured log concentration (y axis) versus theoretical (spikedin) concentration (x axis) for the SSDLVALSGGHTFGK peptide derived from horseradish peroxidase (HRPSSD). Data for each site are color coded, and organized by study phase and concentration. As expected, a linear trend is observed in the measured concentrations for studies IIII as spikedin analytes increase across the concentration range. However, measured concentrations decrease as laboratories progress from study I to II to III. This trend is a result of apparent peptide loss from incomplete digestion of HRP protein and variability in sample handling at each site, as study complexity was increased. Study I represents the optimum assay performance, as synthetic peptides (not proteins) were used as analytes. Protein digestion in study II (at a central location in the absence of plasma) and study III (at individual sites and in the presence of plasma) introduces potential sources of sample loss that decrease analyte recovery and reduce measured concentrations for studies II and III. Intralaboratory CVs for studies I and II constitute a measure of the technical variation due to instrument and data acquisition, as all sample preparation was performed centrally. The intralaboratory CVs at each analyte concentration point are shown in Figure 5b for the HRPSSD peptide with colorcoded markers representing individual laboratories. Similar calculations and plots are derived for the other 9 peptides.
Figure 5. Box plots of variation in MRM quantitative measurements, interlaboratory CV, and intralaboratory CV. All CV calculations are performed on the original data, while logscaled axes are used to enchance visualization in the plots. (a) Intralaboratory assay CV. Box plots showing measured log concentration (y axis) versus theoretical (spikedin) concentration (x axis) for HRPSSD across the entire concentration range in diluted plasma. Protein concentration in mg/ml is mg protein equivalent in 1 ml undiluted plasma. The box plots for studies I and II are based on four replicate measurements, whereas those for study III summarize 12 measurements (four from each of 3 process replicates). Each of the eight sites was assigned a random numerical code (19, 52, 54, 56, 65, 73, 86, 95) for anonymization. (b) Interlaboratory assay CV. Values are shown for studies IIII for the entire range of HRPSSD final analyte concentrations in plasma. Within each box plot, actual intralaboratory CV values for individual laboratories are shown with colorcoded markers. The CV values are calculated based on the single best performing transition (lowest combined CV) across studies I and II. This same transition is also used for study III. Adapted from Addona, et. al., Nature Biotechnology, 27(7):63341 [28].
The results are summarized in Table 5. Intralaboratory precision is represented by the median CV calculated from all concentration points for a particular peptide (based on quadruplicate measurements for a single chosen transition) for each site, and for each study. The interlaboratory precision is represented by the median CV calculated at each concentration point for a particular peptide across all sites and for each study. In Table 5 the interlaboratory precision at a concentration close the LOQ is shown. The CV calculations at each concentration point for a peptide at a given laboratory is based on four replicates for studies I and II and on 12 data points (four technical replicates for each of the three process replicates) for study III.
Table 5. Summary of Results for Studies I, II, and III (combined results for process replicates a, b, c) for each peptide across sites for intersite CV, intrasite CV, linear slope and % recovery.
In this analysis, the interlaboratory precision is calculated as the median intralaboratory CV. While this measure summarizes the precision obtained across multiple laboratories, it does not account for the accuracy of the measurement across different laboratoriesall the laboratories may have repeated measurements that are very close (high precision, and hence low CV), but the actual measurements may differ significantly from laboratory to laboratory (poor accuracy). Hence, in clinical domains, the interlaboratory precision is calculated as the CV of all the measurements of a peptide (at a concentration) across all the laboratories [55]. An additional study investigated the use of more sophisticated mixed effect models to evaluate the sources of variation in the NBT study [56].
Discussion
For researchers new to SIDMRMMS assays, this section outlines important aspects of the experimental design and data analysis, along with practical tips.
When constructing a calibration curve, attempt to use a concentration range that extends past the estimated LOD and upper LOQ so that these Figures of Merit can be calculated from the data. Prepare the calibration curve in a matrix that is identical to that of the actual sample in order to accurately reproduce the chemical noise contributions from the matrix. If this is not possible, use a matrix that is very similar in composition. Analyze matrix blank samples periodically throughout the assay. This will provide the best determination of the signaltonoise of the sample matrix and internal standards, and detect any potential for analyte carryover that would be encountered in a quantitative assay of unknown samples.
To determine the technical variability of an assay, analyzing a minimum of 3 technical replicates (repeat injections from the same sample) is suitable. The use of process replicates (preparations of the samples made at different times) can be used to calculate the analytical variability of an assay. Usually, technical variability is smaller than analytical variability. A minimum of 3 replicates should be prepared for each concentration point in calibration curves. The precision of the calculations improves with increased sample size, so if time and resources permit, more replicates are favorable.
Most methods of calculating the LOD or LOQ use the calibration curve data points to interpolate the determined value. To make sure the calculated LOD seems reasonable, it is recommended to visually inspect the individual concentration points to make sure the calculated values make sense and the concentration point above the calculated LOD is easily discernable. The main factors affecting the calculated LOD of an assay are the noise present in the matrix blank, and the reproducibility of that noise. Matrices that have a lot of noise and/or where the measurement of that noise is very variable will result in higher LODs.
Often in practice, the largest influence on the sensitivity of an assay is not the instrument itself, but how well the instrument is performing. Variability can have a profound impact on sensitivity. Evaluating the reproducibility of an LCMRMMS system is highly recommended before evaluating its sensitivity. This can be accomplished by making repeat measurements of the same sample using the same method, to achieve CV values less than 20%.
Last but not least, automated data processing tools and algorithms should be applied with care, continually assessing data quality, consistently accounting for outliers, and monitoring results.
Conclusion
MRMMS assays are increasingly being deployed to measure and quantify peptides (and hence, proteins) in a variety of matrices and backgrounds. This manuscript provides a complete toolkit for the analysis and interpretation of MRMMS experiments.
Sound statistical analysis of MRMMS data starts with high quality data. Using algorithms like AuDIT and mProphet (Section 5), the data quality assessment can be automated resulting in a more reliable high throughput analysis pipeline which quickly weeds out poor quality transitions or transitions with interferences.
Calibration and characterization of detection limits and variability are important aspects of any quantitative assay. We present a comparative set of methods and approaches for MRMMS assay calibration, regression analysis, determination of confidence intervals, dealing with endogenous signal, assessment of detection limits and multilaboratory characterization of assay performance and precision.
While systematic and principled analysis of data is essential for achieving the full potential of quantitative MRMMS assays, care has to be exercised in experiment design and data generation to maximize reproducibility and data quality. There are many experimental and other variables beyond the scope of this manuscript that need to be addressed for successful deployment and use. Several new multilaboratory studies aim to circumscribe and control these aspects. Two such factors worth mentioning are (i) digestion and (ii) system suitability assessment. Reproducible digestion of proteins is a prerequisite for reliable quantification using MRMMS. Several ongoing studies attempt to not only determine standard operating procedure to ensure proper digestion, but also use specially chosen marker peptides to detect improper or incomplete digestion. Furthermore, given the complexity of chromatography and MS instrumentation, constant assessment of optimal system performance is necessary to guarantee data quality[43]. Studies for defining, assessing and maintaining system suitability are also under way. Most of these large multilaboratory studies are being carried out under the auspices of the Clinical Proteomics Technology Assessment for Cancer (CPTAC) program sponsored by the National Cancer Institute (http://proteomics.cancer.gov webcite), with the overarching goal of advancing biomarker discovery and enabling the advancement of promising new technologies like MRMMS towards clinically deployed assays.
Competing interests
The authors declare that they have no competing interests.
Acknowledgements
Support for this work was provided in part by the Broad Institute of MIT and Harvard and by grants from the National Cancer Institute (U24CA126476) and National Heart Lung and Blood Institute (HHSN268201000033C) to SAC, and in part by a grant from the National Institutes of Health (Grant NCI R01 CA126219 to D.R.M, as part of NCI's Clinical Proteomic Technologies for Cancer Program).
This article has been published as part of BMC Bioinformatics Volume 13 Supplement 16, 2012: Statistical mass spectrometrybased proteomics. The full contents of the supplement are available online at http://www.biomedcentral.com/14712105/13/S16.
References

Yost RA, Enke CG: Triple quadrupole mass spectrometry for direct mixtxure analysis and structure elucidation.
Analytical Chemistry 1979, 51:12511264. PubMed Abstract  Publisher Full Text

Brumley WC, Sphon JA: Regulatory Mass Spectrometry.
Biomed Mass Spectrom 1981, 8:390396. PubMed Abstract  Publisher Full Text

Sphon JA: Use of mass spectrometry for confirmation of animal drug residues.
J Assoc Off Anal Chem 1978, 61:12471252. PubMed Abstract

Vargo JD: Determination of sulfonic acid degradates of chloroacetanilide and chloroacetamide herbicides in groundwater by LC/MS/MS.
Analytical Chemistry 1998, 70:26992703. PubMed Abstract  Publisher Full Text

Draisci R, Palleschi L, Ferretti E, Lucentini L, Cammarata P: Quantitation of anabolic hormones and their metabolites in bovine serum and urine by liquid chromatographytandem mass spectrometry.

Kushnir MM, Rockwood AL, Nelson GJ, Yue B, Urry FM: Assessing analytical specificity in quantitative analysis using tandem mass spectrometry.
Clinical Biochemistry 2005, 38(4):319327. PubMed Abstract  Publisher Full Text

Kuhara T: Noninvasive human metabolome analysis for differential diagnosis of inborn errors of metabolism.
J Chromatogr B Analyt Technol Biomed Life Sci 2007, 855(1):4250. PubMed Abstract  Publisher Full Text

Pitt JJ, Eggington M, Kahler SG: Comprehensive screening of urine samples from inborn errors of metabolism by electrospray tandem mass spectrometry.
Clinical Chemistry 2002, 48:19701980. PubMed Abstract  Publisher Full Text

Rifai N, Gillette MA, Carr SA: Protein biomarker discovery and validation: the long and uncertain path to clinical utility.
Nature Biotechnology 2006, 24:971983. PubMed Abstract  Publisher Full Text

Fenn JB, Mann M, Meng CK, Wong SF, Whitehouse CM: ) Electrospray ionization for mass spectrometry of large biomolecules.
Science 1989, 246(4926):6471. PubMed Abstract  Publisher Full Text

Browne TR: Stable isotopes in pharmacology studies: present and future.
J Clin Pharmacol 1986, 26:485489. PubMed Abstract  Publisher Full Text

Moore LJ, Machlan LA: High accuracy determination of calcium in blood serum by isotope dilution mass spectrometry.
Anal Chem 1972, 44:22912296. PubMed Abstract  Publisher Full Text

Cohen A, Hertz HS, Mandel J, Paule RC, Schaffer R, Sniegoski LT, Sun T, Welch MJ, White ET: Total serum cholesterol by isotope dilution/mass spectrometry: a candidate definitive method.
Clin Chem 1980, 26:854860. PubMed Abstract  Publisher Full Text

Lisek CA, Bailey JE, Benson LM, Yaksh TL, Jardine I: Quantitation of endogenouse substance P by online microcolumn liquid chromatography/continuousflow fast atom bombardment mass spectrometry.
Rapid Commun Mass Spectrom 1989, 3(2):434614. PubMed Abstract  Publisher Full Text

Parsons HG: Stable isotopes in the management and diagnosis of inborn errors of metabolism.
Can J Physiol Pharmacol 1990, 68:950954. PubMed Abstract  Publisher Full Text

Barr JR, Maggio VL, Stemman O, Jr DGP, Cooper GR, Henderson LO, Turner WE, Smith SJ, Hannon WH, Needham LL, Sampson EJ: Isotopedilution mass spectrometric quantification of specific proteins: model application with apolipoprotein A1.
Clin Chem 1996, 42:16761682. PubMed Abstract  Publisher Full Text  PubMed Central Full Text

Gerber SA, Rush J, Stemman OK, Kirschner MW, Gygi SP: Absolute quantification of proteins and phosphoproteins from cell lysates by tandem MS.
Proc Natl Acad Sci USA 2003, 100(12):69406945. PubMed Abstract  Publisher Full Text  PubMed Central Full Text

Barnidge DR, Dratz EA, Martin T, Bonilla LE, Moran LB, Lindall A: Absolute quantification of the G proteincoupled receptor rhodopsin by LC/MS/MS using proteolysis product peptides and synthetic peptide standards.
Anal Chem 2003, 75(3):445451. PubMed Abstract  Publisher Full Text

Kuhn E, Wu J, Karl J, Liao H, Zolg W, Guild B: Quantification of Creactive protein in the serum of patients with rheumatoid arthritis using multiple reaction monitoring mass spectrometry and 13Clabeled peptide standards.
Proteomics 2004, 4(4):11751186. PubMed Abstract  Publisher Full Text  PubMed Central Full Text

Abbatiello SE, Mani DR, Keshishian H, Carr SA: Automated detection of inaccurate and imprecise transitions in peptide quantification by multiple reaction monitoring mass spectrometry.
Clinical Chemistry 2010, 56:291305. PubMed Abstract  Publisher Full Text

Keshishian H, Addona TA, Burgess M, Kuhn E, Carr SA: Quantitative, multiplexed assays for low abundance proteins in plasma by targeted mass spectrometry and stable isotope dilution.
Mol Cell Proteomics 2007, 6:22122219. PubMed Abstract  Publisher Full Text  PubMed Central Full Text

Bondar OP, Barnidge DR, Klee EW, Davis BJ, Klee GG: LCMS/MS quantification of Znalpha2 glycoprotein: a potential serum biomarker for prostate cancer.
Clinical Chemistry 2007, 53:673678. PubMed Abstract  Publisher Full Text  PubMed Central Full Text

Hoofnagle AN, Becker JO, Wener MH, Heinecke JW, et al.: Quantification of thyroglobulin, a lowabundance serum protein, by immunoaffinity peptide enrichment and tandem mass spectrometry.
Clinical chemistry 2008, 54(11):17961804. PubMed Abstract  Publisher Full Text  PubMed Central Full Text

Mani DR, Addona T, Keshishian H, Burgess M, Shi X, Kuhn E, Sabatine MS, Gerszten RE, Carr SA: Quantification of cardiovascular biomarkers in patient plasma by targeted mass spectrometry and stable isotope dilution.
Molecular & cellular proteomics 2009, 8(10):23392349. PubMed Abstract  Publisher Full Text  PubMed Central Full Text

Abbatiello SE, Pan YX, Zhou M, Wayne AS, Veenstra TD, Hunger SP, Kilberg MS, Eyler JR, Richards NG, Conrads TP: Mass spectrometric quantification of asparagine synthetase in circulating leukemia cells from acute lymphoblastic leukemia patients.
Journal of Proteomics 2008, 71:6170. PubMed Abstract  Publisher Full Text

Kuhn E, Addona T, Keshishian H, Burgess M, Mani DR, Lee RT, Sabatine MS, Gerszten RE, Carr SA: Developing multiplexed assays for troponin I and interleukin33 in plasma by peptide.
2009.

Agger SA, Marney LC, Hoofnagle AN: Simultaneous quantification of apolipoprotein A1 and apolipoprotein B by liquidchromatographmultiplereactionmonitoring mass spectrometry.
Clin Chem 2010, 56(12):18041813. PubMed Abstract  Publisher Full Text  PubMed Central Full Text

Addona TA, Abbatiello SE, Schilling B, Skates SJ, Mani DR, Bunk DM, Spiegelman CH, Zimmerman LJ, Ham AJL, Keshishian H, Hall SC, Allen S, Blackman RK, Borchers CH, Buck C, Cardasis HL, Cusak MP, Dodder NG, Gibson BW, Held JM, Hiltke T, Jackson A, Johansen EB, Kinsinger CR, Li J, Mesri M, Neubert TA, Niles RK, Pulsipher TC, Ransohoff D, Rodriguez H, Rudnick PA, Smith D, Tabb DL, Tegeler TJ, Variyath AM, VegaMontoto LJ, Wahlander A, Waldemarson S, Wang M, Whiteaker JR, Zhao L, Anderson NL, Fisher SJ, Liebler DC, Paulovich AG, Regnier FE, Tempst P, Carr SA: Multisite assessment of the precision and reproducibility of multiple reaction monitoringbased measurements of proteins in plasma.
Nature biotechnology 2009, 27(7):633641. PubMed Abstract  Publisher Full Text  PubMed Central Full Text

Lavagnini I, Magno F: A statistical overview on univariate calibration, inverse regression, and detection limits: Application to gas chromatography/mass spectrometry technique.
Mass spectrometry reviews 2007, 26(1):118. PubMed Abstract  Publisher Full Text

Rousseeuw PJ, Leroy AM: Robust Regression and Outlier Detection.
WileyInterscience 2003. PubMed Abstract

Wilcox RR, Keselman HJ: Robust Regression Methods: Achieving Small Standard Errors When There Is Heteroscedasticity.
Understanding Statistics 2004, 3(4):349364. Publisher Full Text

Venables WN, Ripley BD: Modern Applied Statistics with S.
Springer 2002. PubMed Abstract  Publisher Full Text

Yohai VJ, Stahel WA, Zamar RH: A procedure for robust estimation and inference in linear regression. In Directions in Robust Statistics and Diagnosis, Part II. Edited by Stahel WA, Weisberg SW. SpringerVerlag; 1991.

Schoeller DA: A review of the statistical considerations involved in the treatment of isotope dilution calibration data.
Biological Mass Spectrometry 1976, 3(6):265271. PubMed Abstract  Publisher Full Text

Skoog DA, Holler FG, Niemann LH: Princlples of Instrumental Analysis.
Saunders College Publishing 1998. PubMed Abstract  Publisher Full Text

J Res Natl Bur Std. 1964, 53:155159. PubMed Abstract  Publisher Full Text

Yergey AL: The presentation of calibration curves and quantitative data.
Biomed Environ Mass Spectrom 1998, 15(8):465465. PubMed Abstract  Publisher Full Text  PubMed Central Full Text

Feinstein AR: Principles of Medical Statistics. In Journal of the Royal Statistical Society Series B (Methodological). Chapman and Hall; 2001.

Currie LA: Limits for qualitative detection and quantitative determination. Application to radiochemistry.
Analytical Chemistry 1968, 40(3):586593. Publisher Full Text

Linnet K, Kondratovich M: Partly nonparametric approach for determining the limit of detection.
Clinical chemistry 2004, 50(4):732740. PubMed Abstract  Publisher Full Text

Anderson DJ: DetermInatIon of the Lower Limit of Detection.
Clinical chemistry 1989, 35(10):21522153. PubMed Abstract  Publisher Full Text

Vial J, Mapiphan KL, Jardy A: What is the Best Means of Estimating the Detection and Quantification Limits of a Chromatographic Method?
Chromatographia 2003, 57:S303S306. PubMed Abstract  Publisher Full Text  PubMed Central Full Text

Anderson NL, Anderson NG: The human plasma proteome: history, character, and diagnostic prospects.
Mol Cell Proteomics 2002, 1(11):845867. PubMed Abstract  Publisher Full Text

Li W, Cohen LH: Quantitation of endogenous analytes in biofluid without a true blank matrix.
Anal Chem 2003, 75(21):58545859. PubMed Abstract  Publisher Full Text

Davison AC, Hinkley DC: CBootstrap Methods and Their Application. In Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press; 1997.

Yan Z, Maher N, Torres R, Cotto C, Hastings B, Dasgupta M, Hyman R, Huebert N, Caldwell GW: Isobaric metabolite interferences and the requirement for close examination of raw data in addition to stringent chromatographic separations in liquid chromatography/tandem mass spectrometric analysis of drug in biological matrix.
Rapid Commun Mass Spectrom 2008, 22:20212028. PubMed Abstract  Publisher Full Text

Whiteaker JR, Zhao L, Zhang HY, Feng LC, Piening BD, Anderson L, Paulovich AG: Antibodybased enrichment of peptides on magnetic beads for massspectrometrybased quantification of serum biomarkers.
Anal Biochem 2007, 362(1):4454. PubMed Abstract  Publisher Full Text  PubMed Central Full Text

Benjamini Y, Hochberg Y: Controlling the false discovery rate: a practical and powerful approach to multiple testing.
Journal of the Royal Statistical Society Series B (Methodological) 1995, 57(1):289300.

Kost JT, McDermott MP: Combining dependent Pvalues.
Statistics & Probability Letters 2002, 60(2):183190. PubMed Abstract  Publisher Full Text

Brown MB: A Method for Combining NonIndependent, OneSided Tests of Significance.
Biometrics 1975, 31(4):987992. Publisher Full Text

Sing T, Sander O , Beerenwinkel N, Lengauer T: ROCR: visualizing classifier performance in R.
Bioinformatics 2005, 21:39403941. PubMed Abstract  Publisher Full Text

Reiter L, Rinner O, Picotti P, Hüttenhain R, Beck M, Brusniak MY, Hengartner MO, Aebersold R: mProphet: automated data processing and statistical validation for largescale SRM experiments.

MacLean B, Tomazela DM, Shulman N, Chambers M, Finney GL, Frewen B, Kern R, Tabb DL, Liebler DC, MacCoss MJ: Skyline: an open source document editor for creating and analyzing targeted proteomics experiments.
Bioinformatics 2010, 26:966968. PubMed Abstract  Publisher Full Text  PubMed Central Full Text

Tufte ER: The Visual Display of Quantitative Information. 2nd edition edition. Graphics Press; 2001.

Hoofnagle AN: Quantitative clinical proteomics by liquid chromatographytandem mass spectrometry: assessing the platform.
Clinical Chemistry 2010, 56(2):161164. PubMed Abstract  Publisher Full Text

Xia JQ, Sedransk N, Feng X: Variance Component Analysis of a MultiSite Study for the Reproducibility of Multiple Reaction Monitoring Measurements of Peptides in Human Plasma.
PLoS ONE 2011, 6:e14590. PubMed Abstract  Publisher Full Text  PubMed Central Full Text

Briscoe CJ, Hage DS, Stiles MR: System Suitability in Bioanalytical LC/MS/MS.
Journal of Pharmaceutical and Biomedical Analysis 2007, 44:484491. PubMed Abstract  Publisher Full Text