Before discussing algorithmic details, it is important to introduce notation that will be used throughout this paper. Let us consider a set of tandem mass spectra S from a proteomics experiment and let be a database of proteins that the spectra are searched against. Let also be the set of all peptides in the database and, similarly, be the set of peptides that belong to protein P i . We now define two sets of indicator variables as follows

t j = 1 0 if peptide p j is confidently identified otherwise

and

x j = 1 0 if peptide p j is present in the sample otherwise

Confident peptide identifications can be determined in several ways, typically by using strict FDR thresholds on the top-scoring PSMs (per peptide) and are estimated using a decoy database 22 or tools such as PeptideProphet 38 , which calculate the posterior probability of a correct peptide identification. When posterior probabilities are available, stringent thresholds (e.g. 0.90) can be applied directly to those probabilities. Alternatively, sufficiently high scores from various search engines 9 39 40 41 42 are sometimes used to select confident identifications.

It is important to mention that variables t j and x j are different. For example, a peptide p_j that is confidently identified, e.g. using an FDR threshold of 0.01, will result in setting t j = 1 . On the other hand, x j can be seen as a hidden variable that is to be inferred. Accordingly, refers to the probability that peptide  j is present in the sample given all the data from the mass spectrometer. A set of confidently identified peptides, using any of the above-mentioned approaches will be denoted as .

In some situations it will be necessary to consider peptides with explicit designations of their parent proteins. In those cases, the  j -th peptide derived from protein P i will be denoted as p i j . Two or more such peptides will be allowed to have identical amino acid sequences. For example, peptides p i j and p k l (  i ≠  k ) with identical amino acid sequences will be referred to as degenerate peptides. In the context of protein inference, peptides that occur multiple times only within a single protein will not be considered degenerate. Finally, we define

y i = 1 0 if protein P i is present in the sample otherwise

Variable y i can be seen as an equivalent of x j at the protein level. Thus, is the posterior probability that protein P i is present in the sample. The summary of notation and abbreviations is shown in Table 1.

With a typical LC-MS/MS experiment resulting in a potentially large number of protein identifications, concerns were raised regarding the impact of misidentified proteins on biomedical science 45 . In response to this, several guidelines were proposed regarding the standards for publishing proteomics results 46 47 48 49 . The so-called "two-peptide rule" or two-hit rule, requiring two or more confidently identified peptides to define a confident protein identification, was advocated 46 48 . The same guidelines also recommended the parsimony principle (see next Section) as an explanation for the confident peptide identifications, and suggested that "protein family" - proteins with similar sequences due to single amino acid variants, homologs, splicing variants, or annotation mistakes - should be reported as one group if the proteins share the same identified peptides.

There is a good rationale for using the two-peptide rule. In principle, one correct unique peptide should be sufficient to correctly identify a protein. However, even for the low FDR associated with a set of peptides, many individual peptides in a large data set are incorrectly identified. Furthermore, proteins identified by single peptide hits are more likely to be incorrectly identified than proteins with higher peptide coverage 45 . It was reported that FDRs for single-hit proteins can be over 10 times higher than FDRs at the PSM level 50 , likely due to the clustering of correct peptide identifications to the correct proteins and the lack of clustering behavior for the incorrect peptides 50 51 .

However, the two-peptide rule has been challenged 51 52 . First, while including single-hit proteins without stringent quality control can compromise specificity, ignoring such proteins will certainly compromise sensitivity 52 . Second, controlling the confidence (FDR) at the peptide level and then deducing the proteins using heuristic rules leads to undefined FDRs at the protein level 27 50 51 52 . On the other hand, controlling FDR directly at the protein level may rescue some of the confident single-hit proteins. Indeed, Gupta and Pevnzer demonstrated that using the "single-peptide rule" results in 10-40% more protein identifications compared with the two-peptide rule at a fixed FDR level 52 . The single-peptide rule simply uses the highest scoring peptide from a protein as a score for that protein, and then directly estimates FDR at the protein level (rather than at the peptide level) using decoy databases. Thus, any protein that has one or more peptides with a score above a certain threshold is deemed confident. This statement seems problematic because proteins hit by single peptides should not be reliable. However, two mediocre peptides are not necessarily better than one good peptide; thus, many proteins hit by a single peptide can be rescued with more stringent score thresholds. Since a significant portion of such proteins are correct 53 , it is not surprising that the single-peptide rule leads to more protein identifications.

With the help of protein-level FDR estimation (using a decoy database), better and more complex rules may be devised to achieve even higher sensitivity. For example, Weatherly et al. proposed setting separate score thresholds for proteins with different number of confident peptide identifications 51 . They reported that gradually lower score thresholds were needed for proteins with increasingly higher coverage. For the coverage of 1 (i.e. proteins hit by single peptides), a MASCOT score of 44 was required, while for coverage of 6, a MASCOT score as low as 11 was necessary for the same FDR 51 .

Despite the relative simplicity of rule-based approaches, the performance of heuristic rules is fundamentally limited by the lack of rigorous treatment and proper combination of the peptide identification scores and prior knowledge.

The input to this class of algorithms typically consists of a set of confidently identified peptides = { p j | t j = 1 } and a protein database . The objective is to provide a list of proteins that optimizes certain criteria. In one way or another, all such formulations result in NP-hard problems and are usually solved using approximation algorithms.

Similarly to the previous classes of algorithms, probabilistic approaches to protein inference generally consist of two steps. First, PSM scores are converted to PSM probabilities using algorithms such as PeptideProphet 38 . After this pre-processing step, protein inference is performed based on an assumed probabilistic model. In probabilistic terms, protein inference involves computing protein posterior probabilities for every protein in .

Several classes of probabilistic algorithms have been proposed so far 21 24 60 61 63 64 65 66 67 68 69 70 71 , with different strategies and levels of rigor in addressing protein groups and different run-time performance. Some probabilistic algorithms do not address degenerate peptides 63 65 68 70 , while some such as ProteinProphet 21 combine probabilistic inference with the parsimony principle (for degenerate peptides) and protein grouping (for indistinguishable proteins). In the following subsections, we provide an in-depth discussion of the three major probabilistic methods: ProteinProphet 21 , MSBayesPro 61 , and Fido 71 , and briefly mention several other methods. We use the same notation for all models and, when possible, provide new interpretations of the algorithms. We aim to reveal inherent connections and principal differences among the methods. For original derivations and interpretations, readers are referred to the original publications.

Despite the development of computational protein identification methods, objectively evaluating the performance of the methods remains a problem. Two strategies are currently available: the use of standard samples (mixtures of known proteins) and the use of decoy protein sequences to estimate FDR at the protein level. Both approaches have limitations.

To date, only a limited number of standard samples 78 79 80 containing 10-50 proteins have been used to facilitate evaluation of peptide/protein identification. The advantage of using standard samples is that the truth is known; thus, the accuracy measures, e.g. precision and recall, of protein identification can be directly computed. However, standard samples are frequently plagued by contaminant proteins and the boundary between true and false protein identification is blurred. Another limitation of standard samples is their small number of proteins, which leads to difficulties in assessing statistical significance in method comparisons.

The second approach estimates protein-level false discovery rates with the help of decoy databases. Although the approach has been used in several studies 51 52 , two serious problems of the approach are generally ignored. We suggest that using decoy databases for evaluation of protein identification algorithms should be approached with these limitations in mind. First, unlike the decoy (e.g. reversed, randomized) database approach for peptides, the decoy database for proteins does not produce the correct estimation of the number of incorrect protein identifications when the correct proteins comprise a significant portion of the database. In an extreme scenario, when all proteins in the database are present in the sample, all the identified proteins from the forward database are correct despite many peptides being in-correct identifications. On the other hand, all identified proteins from a decoy database are incorrect. Thus, using a decoy directly will produce a non-zero FDR, while FDR = 0 is the correct answer.

This problem can be addressed by correcting for the bias due to the number of true proteins in the forward database. Let the number of identified forward and decoy proteins be n_F and n_D , and the total number of forward and decoy proteins in the databases be N_F and N_D , respectively. Let the protein level FDR in forward database be FDR_P and the rate of incorrect protein identifications from the forward and decoy database be

γ F = F D R P ⋅ n F N F - ( 1 - F D R P ) ⋅ n F ,

and

γ D = n D N D ,

respectively. An assumption regarding a decoy database is that the rates of the false protein identifications are identical; hence, γ F = γ D . By solving this equation we find

F D R P = n D ⋅ ( N F - n F ) n F ⋅ ( N D - n D ) .

Note that there is a correction factor ( N F − n F ) ( N D − n D ) in this equation compared to the FDR formula used for peptides. Also, when N F = n F ,   F D R P = 0 as expected. A related correction is implemented in the MAYU approach 50 developed for FDR estimation from large proteomics data sets, i.e. the case when n F / N F ≫ 0 . Further corrections may be needed if the average lengths of the identified vs. non-identified proteins are different.

We would like to point out that, for probabilistic protein inference algorithms, theoretical protein FDR values can be computed based on the protein posterior probabilities. However, such theoretical FDR values are only accurate when the reported protein posterior probabilities are accurate. Hence, they need to be evaluated themselves, e.g. against the target/decoy-based empirical FDRs.

The second and more serious issue for applying the decoy approach is related to the existence of protein families. In fact, to our knowledge, no solution has yet been proposed. Simply speaking, a randomized database cannot serve as a good decoy for evaluating methods on data sets that contain many degenerate peptide identifications. The reason is that such peptides are typically shared among forward proteins, which could be similar to each other due to biological/annotation reasons, but not with decoy proteins. As a result, a randomized protein database cannot provide indications whether the identifications made among homologous proteins are correct or not. For this reason, a randomized decoy database is expected to underestimate FDRs for eukaryotic samples, which have large number of shared peptides (Figure 1). The problem might be addressed using well-constructed non-random sequence database or using a closely related proteome database as decoy. Evaluating protein inference algorithms using such non-random decoys, however, remains a research problem.

We emphasize that both standard mixtures and the target/decoy approach for complex samples have their pros and cons in evaluating protein inference algorithms, and they are not mutually exclusive approaches. In fact, standard mixtures can be used to validate the target/decoy approach for protein FDR estimation. It is generally a good idea to use both strategies for a more complete and objective evaluation.

Despite the amount of published work, the protein inference problem is far from solved. We believe two aspects are crucial to the future approaches. First, the model should be probabilistic and with degenerate peptides treated in principled ways. Second, unidentified peptides should be exploited with peptide detectability incorporated into the model, perhaps adjusted to allow modeling peptide competition at the elution stage in a given sample. Despite the current limitations of peptide detectability predictions, especially for non-tryptic and modified peptides, it is believed that including detectability 24 35 69 71 or peptide-specific information for peptide probability adjustment 21 would improve the current methods for protein inference.

Furthermore, we believe that better estimation of peptide/protein quantity might also help protein inference by, for example, improving the quantity adjustment of peptide detectability 60 61 , and provide additional input information for protein inference. As mentioned in the Introduction, protein inference can be viewed as a special case of protein label-free quantification. In fact, an ideal inference algorithm should automatically be a quantification algorithm, and vice versa. We believe much better performance can be achieved by combining the protein inference and quantification tasks into one statistical framework.

Algorithmic development is equally important for rigorous and yet practical probabilistic inference. Serang et al. 76 proposed an approximate solution by setting low peptide probabilities to zero and then applying the graph pruning procedure. In this way the complexity of the problem can be controlled at arbitrarily low levels with the price of potentially high error (i.e. the computed probability may greatly deviate from the exact values). The Gibbs sampling approach implemented in MSBayesPro can achieve arbitrarily high accuracy in probability estimation; however, the time required for the inference can be prohibitively long. A fast algorithm with controllable error bound is desirable. Applying well-established exact or approximate graph inference algorithms, e.g. the junction tree algorithm 76 , is an important direction for further investigation.

Peptide detectability has been defined as the probability that a peptide will be identified in a proteomics experiment given the presence of its parent protein in a sample 19 24 . There are multiple factors, spanning all phases of a proteomics experiment, that influence peptide identification. For example, during sample storage and preparation, some peptides may be truncated at their termini resulting in semi-tryptic or non-tryptic peptides (in the case of trypsin digestion) which usually remain unidentified in a database search 25 . Peptides with different hydrophobicity patterns may not be retained in the LC stationary phase (hydrophilic peptides) or may be insoluble in the LC mobile phase (hydrophobic peptides). Peptides that eluted will be ionized with different efficiencies based on the presence and distribution of charged residues in their sequence. Furthermore, in complex biological samples, peptides are likely to co-elute with many other peptides and thus compete for ionizing protons during the electrospray ionization. Many peptides may elute and ionize well, but poorly fragment, producing MS/MS spectra with few peaks. Such peptides are difficult to interpret by computational methods. In addition, peptides whose m/z values are outside of the range of the mass spectrometer (200-2000 Da) cannot be identified. Apart from physicochemical aspects, there are several biological factors influencing peptide identification. For example, the three-dimensional structure of a protein could lead to the existence of sites with different sensitivities to proteolytic digestion. Other sites may be post-translationally modified by one of more than 200 different post-translational modifications (PTMs) observed in eukaryotes 26 . Many such peptides typi-cally remain unidentified unless a database search explicitly specifies the PTM type or the sites of interest (which, in turn, leads to a decrease in the number of identifications for regular peptides). Finally, different peptide identification software packages are based on different assumptions and are known to result in differences among identified peptides 27 .

It has been shown that the detectability of a peptide at standard quantity, i.e. standard detectability, is a property of the peptide sequence and thus can be predicted from peptide/protein sequence for a given experimental platform 17 19 24 28 29 30 31 32 33 . On the other hand, the quantity of a protein also determines the fate of a peptide with respect to its identification. For example, peptides with high standard detectability that are present in a sample in low quantity may not be identified, while peptides with relatively low detectability present at high quantity may in fact be observed. Therefore, protein quantity and standard detectability collectively determine the effective detectability of each peptide in a protein. Effective peptide detectability cannot be predicted from amino acid sequence alone (unless protein quantity can be shown to depend on protein sequence) and has to be estimated from a set of peptide identifications and their standard detectabilities 24 .

A detectable peptide is related to a proteotypic peptide, which is "an experimentally observable peptide that uniquely identifies a specific protein or protein isoform" 18 . In practice, proteotypic peptides were required to be observed in more than 50% of experiments in which their parent protein was identified 18 30 . The relationship between two definitions can be best understood from an interesting property of peptides in MS/MS experiments to group at either high or low end of the detectability scale in a standard sample 24 34

In summary, the identification of a peptide in a proteomics experiment is a stochastic event that depends on multiple factors. It is therefore convenient to summarize all these factors using a probabilistic framework. The standard detectability of peptide p_j from protein P_i (at quantity q⁰ ) will be denoted as d_ij ⁰, while the effective detectability at an arbitrary quantity q will be denoted as d_ij (q).