Abstract
Background
Recent studies have revealed the importance of considering the entire distribution of possible secondary structures in RNA secondary structure predictions; therefore, a new type of estimator is proposed including the maximum expected accuracy (MEA) estimator. The MEAbased estimators have been designed to maximize the expected accuracy of the basepairs and have achieved the highest level of accuracy. Those methods, however, do not give the single best prediction of the structure, but employ parameters to control the tradeoff between the sensitivity and the positive predictive value (PPV). It is unclear what parameter value we should use, and even the welltrained default parameter value does not, in general, give the best result in popular accuracy measures to each RNA sequence.
Results
Instead of using the expected values of the popular accuracy measures for RNA secondary structure prediction, which is difficult to be calculated, the pseudoexpected accuracy, which can easily be computed from basepairing probabilities, is introduced. It is shown that the pseudoexpected accuracy is a good approximation in terms of sensitivity, PPV, MCC, or Fscore. The pseudoexpected accuracy can be approximately maximized for each RNA sequence by stochastic sampling. It is also shown that wellbalanced secondary structures between sensitivity and PPV can be predicted with a small computational overhead by combining the pseudoexpected accuracy of MCC or Fscore with the γcentroid estimator.
Conclusions
This study gives not only a method for predicting the secondary structure that balances between sensitivity and PPV, but also a general method for approximately maximizing the (pseudo)expected accuracy with respect to various evaluation measures including MCC and Fscore.
Background
To predict the secondary structure of an RNA sequence is a classic problem of sequence analysis in bioinformatics. The importance of accurate predictions of secondary structures has increased due to the recent finding of functional noncoding RNAs whose functions are closely related to their secondary structures [13]. Secondary structure prediction also plays an important role in research on viral RNAs [4].
There are many tools and algorithms for secondary structure prediction [511]. The most popular approach is to predict the minimum free energy (MFE) structure by using the Zuker algorithm [12]. Wellknown software (Mfold [13], RNAfold [14] and RNAstructure [15]) employs this approach. From a probabilistic viewpoint, the MFE structure is equivalent to the secondary structure of the maximum likelihood (ML) estimation for the probability distribution of secondary structures given by the McCaskill model [16]. It is, however, known that the MFE/ML structure has drawbacks: there are a huge number of suboptimal structures whose free energies are similar to the minimum free energy and the probability of the MFE structure is extremely small [17]. Moreover, the MLestimator is not optimized for ac curacy measures in the target problem [10].
Therefore, another approach that considers the entire distribution of possible secondary structures of a given sequence has been introduced. Ding et al. [18] proposed the centroid estimator, which minimizes the expected Hamming loss. On the other hand, Do et al. [7] proposed the maximum expected accuracy (MEA) estimator, which gives a prediction based on maximizing the expected value of an accuracy function under a probabilistic distribution of secondary structures. The MEAbased estimators have been applied to many problems in bioinformatics, including sequence analyses for RNA sequences [6,7,10,1922], alignment of bio logical sequences [2325] and other estimation problems [2628].
For RNA secondary structure predictions, two MEAbased estimators have been proposed: (i) the estimator proposed by [7] and (ii) the γcentroid estimator proposed by [10]. Both estimators do not employ the accuracy measures that are used in actual evaluation of RNA secondary structure, namely, sensitivity (SEN), positive predictive value (PPV), Matthews correlation coefficient (MCC) and Fscore, with respect to predicted basepairs. From a view point of MEA, it is useful to consider the estimators that maximize expectation of those accuracy measures. Because the computation of those estimators generally demands huge computational time, the previous studies could not use those accuracy measures directly.
Moreover, the previous MEAbased estimators contain a parameter that controls the balance between SEN and PPV of basepairs in a predicted secondary structure. It is, however, unclear how to select the parameter in order to obtain one reasonable secondary structure (e.g., a wellbalanced secondary structure between SEN and PPV), although there are situations that only one predicted secondary structure is required. There is also a possibility that the optimal parameter might depend on the length of sequence and/or the type of RNA family, although the γcentroid estimator (and the estimator proposed by [7]) employs a default parameter, determined by a benchmark dataset, which is identical for all sequences.
In this study, to address the drawbacks of the current MEAbased methods described above, We introduce the pseudoexpected accuracy of a secondary structure with respect to a given accuracy measure, which is a function of the number of true positive basepairs (TP), truenegative base pairs (TN), falsepositive basepairs (FP) and false negative basepairs (FN). The pseudoexpected accuracy is then defined by using expected TP, TN, FP and FN. As the accuracy measures, we utilize SEN, PPV, MCC and Fscore with respect to basepairs, which are commonly used in the evaluations of RNA secondary structure predictions, because the base pairs are essential for forming secondary/tertiary structures, which are known to be biologically important.
The pseudoexpected accuracy is easily calculated using the basepairing probability matrix, and can be computed much more efficiently than the expected accuracy. Although the pseudoexpected accuracy is not equal to the expected accuracy of a predicted secondary structure, we found that the pseudoexpected accuracy gives a good approximation of the expected accuracy in our situation. Accordingly, we also introduce the approximated estimators that maximize the expected accuracy of a given accuracy measure. Moreover, by combining the pseudoexpected MCC/Fscore with the γ centroid estimator, it is possible to predict the balanced secondary structure between SEN and PPV (which seems to be a reasonable secondary structure in many situations when only one predicted secondary structure is required), although there is a small computational overhead.
The techniques described in this paper will be extended to design the maximum expected accuracy estimator for various evaluation measures (cf. [29]).
Methods
In the following, we represent a secondary structure of an RNA sequence x as a triangular binary matrix:θ = {θ_{ij}}_{i < j }where θ_{ij }= 1 means that ith base x_{i }and jth base x_{j }form a basepair, and θ_{ij }= 0 means that ith base x_{i }and jth base x_{j }do not form a basepair. In this study, pseudoknotted basepairs are not allowed in a secondary structure. For an RNA sequence denotes the space of all the possible secondary structures of x, where x is the length of x. A probability distribution on S(x) (denoted by p(·x)) is given by the McCaskill [16], CONTRAfold [7] or Simfold [11] models. The basepairing probability matrix of x, {p_{i,j}}_{i < j}, has entries called basepairing probabilities, where I(·) is the indicator function that returns 1 if the condition is true and 0 otherwise. The basepairing probability matrix of a given RNA sequence x can be computed using the McCaskill (InsideOutside) algorithm, whose complexities are O(x^{3}) and O(x^{2}) for time and space, respectively (e.g., see [16,30]).
Expected accuracy and pseudoexpected accuracy of RNA secondary structure
Accuracy measures for RNA secondary structure prediction
For two secondary structures θ = {θ_{ij}}_{i < j }∈ S(x) and σ = {σ_{ij}}_{i < j }∈ S(x) of an RNA sequence x, we define
When θ is a reference (correct) secondary structure and is a predicted secondary structure of x, Eqs. (1), (2), (3), (4), (5), (6), (7), and (8) are the number of true positive basepairs, the number of true negative basepairs, the number of false positive basepairs, the number of false negative basepairs, the SEN, the PPV, the MCC and the Fscore, respectively. Because the basepairs in a secondary structure are biologically important, accuracy measures based on basepairs are useful and SEN, PPV, MCC and Fscore are widelyused accuracy measures for secondary structure predictions. Note that MCC and Fscore are balanced measures between SEN and PPV. (Fscore is equal to a harmonic mean of SEN and PPV.) In the following, Acc means one of the SEN, PPV, MCC and Fscore.
Expected accuracy of secondary structure
Given a probability distribution p(θx) on S(x), we calculate the expected values of Eq. (1) to Eq. (4).
Where {p_{ij}} indicates the basepairing probability matrix. Moreover, we calculate the expected accuracy of an accuracy measure Acc (Acc is equal to SEN, PPV, MCC or Fscore) of σ as follows:
In order to compute the expected Acc for a given secondary structure σ (i.e., ), it is necessary to sum over all the secondary structures of the RNA sequence x because no efficient algorithm (such as a dynamic programming algorithm) has been reported. The number of candidate secondary structures increases exponentially with the length of the RNA sequence (more precisely, there are roughly 1.8^{L }possible structures for a sequence of length L), so to compute the expected Acc is an intractable problem. Therefore, we approximate it using stochastic sampling: For N secondary structures given by stochastic sampling [30,31] of secondary structures, we define
for converges to when N is sufficiently large by the properties of stochastic sampling. It should be noted that the sample size N grows exponentially with the sequence length to be a reliable approximation to the expected Acc of σ.
Pseudoexpected accuracy of secondary structure
In our situation, Acc is generally written as a function of TP, TN, FP and FN:
Acc = f(TP, TN, FP, FN)
Then, for a secondary structure σ, the pseudoexpected Acc of is defined by
For example, the pseudoexpected MCC is given by
If we have the basepairing probability matrix of x, the pseudoexpected Acc of σ can be easily computed by using Eqs. (9), (10), (11) and (12) without employing sampling/enumerating algorithms. Although the pseudoexpected Acc is not equal to the expected Acc, we shall see later that the pseudoexpected Acc is a good approximation of the expected Acc when Acc is equal to SEN, PPV, MCC or Fscore.
Prediction of secondary structure by maximizing pseudoexpected accuracy
The γcentroid estimator [10] for RNA secondary structure prediction is defined by
where γ > 0 controls SEN and PPV of a predicted secondary structure. This estimator is suitable when we would like to predict more TP and TN and fewer FP and FN because Eq. (17) is equivalent to
with γ = (α_{1 }+ α_{4})/(α_{2 }+ α_{3}) and α_{k }≥ 0. Hamada et al. [10] show that the secondary structure of the γcentroid estimator can be calculated by Nussinovstyle dynamic programming.
Eq. (18) indicates that the γcentroid estimator is not optimized for the actual evaluation measures (cf. SEN, PPV, MCC and Fscore). It is useful to introduce the estimator that maximizes expected SEN, PPV, MCC or Fscore directly:
It is, however, difficult to compute the expected Acc efficiently for given σ and p(θx). Because Acc contains products or divisions of TP, TN, FP and FN, no efficient method to compute the estimator Eq. (19) has been found, in contrast to the γcentroid estimator of Eq. (17). Instead, we consider estimators that maximize pseudoexpected Acc as follows.
Prediction of secondary structure by maximizing pseudoexpected SEN/PPV
The pseudoexpected SEN and PPV of a secondary structure σ can be computed by
Therefore, the secondary structure given by maximizing pseudoexpected SEN (Eq. (20) with SEN)) is equivalent to the secondary structure that maximizes the sum of baseparing probabilities of the predicted basepairs. The secondary structure is, therefore, equivalent to the one given by the γcentroid estimator with a sufficiently large γ [10]. On the other hand, the secondary structure given by maximizing pseudoexpected PPV (Eq. (20) with PPV)) is equivalent to the secondary structure that consists of (only) one basepair that has the highest basepairing probability. (The structure does not seem to be useful.) It should be noted that both structures can be easily computed by using the baseparing probability matrix of the target RNA sequence.
Prediction of secondary structure by maximizing pseudoexpected MCC/Fscore with stochastic sampling (Method M1)
Because of the computational difficulty of computing "argmax" in Eq. (20) with MCC and Fscore (see "Discussion" section for more details), we need to evaluate all the secondary structures in S(x). The number of secondary structures of a given RNA sequence, however, is so large that it is not practical to enumerate all of them. Therefore, we again employ the stochastic sampling of secondary structures and approximate the estimator of Eq. (20) by
where S is a set of secondary structures given by stochastic sampling. Note that the computational cost of this estimator is much smaller than that of predictions based on maximizing the expected MCC/Fscore. If the pseudoexpected MCC/Fscore gives a good approximation of the expected MCC/Fscore and we use a sufficiently large sample size, then the estimator in Eq. (23) should be a reliable approximation to the estimator in Eq. (19) that maximizes the expected MCC/Fscore.
Prediction of secondary structure with γcentroid estimator and pseudoexpected MCC/Fscore (Method M2)
In the γcentroid estimator [10] of Eq. (17) implemented in the software CentroidFold [32], there is a parameter γ that adjusts the balance between SEN and PPV. It is, however, unclear how to select the γ parameter that achieves a reasonable structure although there are several situations that only one predicted secondary structure is required. As described in the previous section, we can predict the secondary structures that maximize (pseudo)expected SEN or PPV, but the wellbalanced secondary structure between SEN and PPV will be more useful in many cases than those structures.
Eq. (18), which is equivalent in form to the γ centroid estimator, indicates that the γcentroid estimator can arbitrarily control the number/ratio of the true predictions and the false predictions by using the parameter. By combining the pseudoexpected MCC/Fscore with the γcentroid estimator, it is possible to predict the balanced secondary structure between SEN and PPV, as follows. First, we compute the basepairing probability matrix of the given RNA sequence, and then predict a set of secondary structures S^{g }of x by using the γcentroid estimators with 17 γ parameters: γ ∈ {2^{k}: 5 ≤ k ≤ 10, k ∈ ℤ} ∪ {6} that were used in our previous paper to obtain the SENPPV curve [7,10]. Here, the secondary structure of the γcentroid estimator with γ ∈ {2^{k}: 0 < k ≤ 10, k ∈ ℤ} ∪ {6} is computed by using Nussinovstyle dynamic programming, but the secondary structure of the γcentroid estimator with γ ∈ {2^{k}: 5 ≤ k ≤ 0, k ∈ ℤ} can be predicted without dynamic programming by selecting all the basepairs whose probability is larger than 1/(γ+ 1) [10]. Second, we select the secondary structure in S^{g }that has the best pseudoexpected MCC/Fscore:
where Acc is equal to MCC or Fscore. The pseudoexpected MCC/Fscore of each secondary structure σ ∈ S^{g }is easily computed because the basepairing probability matrix has already been computed.
In this case, using the γcentroid estimator is more suitable than using the MEAbased estimator proposed by [7], which also has a parameter that controls the balance between SEN and PPV, because the latter has a bias to MCC and Fscore (see [10] for details).
Results
We conducted all experiments using a Linux OS machine, which has a 2 GHz AMD Opteron Model 246 processor and 4 GB of memory.
Experimental settings
For the dataset, we utilized the S151Rfam dataset [7] that contains 151 RNA sequences with reference structures, each of which was taken from a different family in the Rfam database [1] This dataset has been widely used in previous studies of RNA secondary structure prediction, for example, [7,10,11]. For the probability distribution p(θx) of the secondary structures of RNA sequence x, we used the CONTRAfold model (version 2.02) [7] and the McCaskill model [16] (in the Vienna RNA package version 1.8.3 [14]). For evaluation measures, we employed SEN, PPV, MCC and Fscore with respect to the basepairs, which are defined by Eqs. (5), (6), (7) and (8), respectively, where σ is a predicted structure and θ is a reference structure.
Comparison between pseudoexpected accuracy and expected accuracy
In this experiment, we compared the pseudoexpected Acc (Eq. (15)) with the expected Acc (Eq. (13)) where Acc is SEN, PPV, MCC or Fscore. First, we obtained a set of secondary structures from the S151Rfam dataset in the following way. For each RNA sequence in the S151Rfam dataset, we predicted the secondary structures using the γcentroid estimator [10] (implemented in CentroidFold) with 17 γ parameters, γ ∈ {2^{k}: 5 ≤ k ≤ 10} ∪ {6} and two models (the McCaskill [16] and CONTRAfold [7] models). Then, duplicate secondary structures were removed from the set. The set of the secondary structures contains various secondary structures, because the γcentroid estimator with small γ predicts a small number of basepairs and the one with large γ predicts a large number of basepairs [10]. As described in the previous section, it is not feasible to compute the expected Acc (Eq. (13)) of a given secondary structure, because the number of possible secondary structures is immense. Therefore, we plotted (i.e., pseudoexpected Acc of; Eq. (15)) and (i.e., expected Acc of σ approximated by 1 M (1,000,000) samples; Eq. (14)) for each secondary structure σ in the set of secondary structures.
The result is shown in Figure 1, which indicates the pseudoexpected SEN, PPV, MCC and Fscore of the predicted secondary structure is a reliable approximation to the expected SEN, PPV, MCC and Fscore, respectively. The averaged squared errors of the pseudoexpected SEN, PPV, MCC and Fscore with respect to the CONTRAfold model and the McCaskill model are shown in Additional file 1, Table S1.
Figure 1. Comparison between pseudoexpected accuracy and expected accuracy. Comparison between the pseudoexpected SEN, PPV, MCC and Fscore (the horizontal axes) and the expected SEN, PPV, MCC and Fscore that are computed by stochastic sampling with a sample size of n = 1 M (the vertical axes). We used the McCaskill model (top row) and the CONTRAfold model (bottom row). The 1st, 2nd, 3rd and 4th columns indicate SEN, PPV, Fscore and MCC, respectively. See Additional file 1, Figure S1 and Figure S2 for other sample sizes.
Additional file 1. Supplementary Information for the main text. This file includes supplementary information for the main text.
Format: PDF Size: 948KB Download file
This file can be viewed with: Adobe Acrobat Reader
Results of secondary structure prediction by maximizing pseudoexpected accuracy
We conducted the experiments on RNA secondary structure prediction by maximizing the pseudoexpected MCC/Fscore of the predicted secondary structure with stochastic sampling (the estimator in Eq. (23)). Note that the results in the previous section suggest that the estimator of Eq. (23) with a sufficiently large sample size is a good approximation to the estimator of Eq. (19) that maximizes the expected MCC/Fscore.
The results are shown in Figure 2 (MCC) and Additional file 1, Figure S1 (Fscore). As the sample size increased, the performance of the prediction of the estimator in Eq. (23) converged to the points on the SENPPV curves of the γcentroid estimator [10], and favorable MCC/Fscores were achieved (Table 1). On the other hand, we need to sample a large number of secondary structures (more than 1 million) in order to obtain the secondary structure that has a good MCC/Fscore. The computational time of the estimator of Eq. (23) increased linearly with the sample size (Table 2). The result also suggests that it is difficult to improve the performance of the γcentroid estimator even if we employ the estimator of Eq. (19), that is, maximizing expected MCC/Fscore.
Figure 2. Performance of RNA secondary structure prediction by maximizing the pseudoexpected MCC with the stochastic sampling (Method M1). Performance of RNA secondary structure prediction by "XMaxpMCC (N)" means the estimator of Eq. (23) with model X and number of samples N with respect to MCC. In the figure, we have also plotted the SENPPV curves of the γcentroid estimator [10] with the CONTRAfold model ("CONTRAfoldgCentroid"; the black line) and with the McCaskill model ("McCaskillgCentroid"; the gray line). The points and curve in gray and those in black indicate the McCaskill [16] and CONTRAfold [7] models, respectively. See Additional file 1, Figure S3 in the supplementary paper for the results of Fscore.
Table 1. SEN, PPV, MCC and Fscore for each prediction algorithm
Table 2. Computational time in seconds
It should be noted that the performance of the estimator that maximizes the pseudoexpected SEN (PPV) corresponds to the leftmost (rightmost) point in the SENPPV curve of the γcentroid estimators.
Results of secondary structure prediction with γ centroid estimator and pseudoexpected accuracy
Figure 3 shows the performance of RNA secondary structure prediction with the γcentroid estimator and the pseudoexpected MCC/Fscore (Method M2). When the McCaskill model is used, Method M2 is slightly worse than the γcentroid estimator. However, the performance of Method M2 with the CONTRAfold model is slightly better than the performance of the γcentroid estimator with the CONTRAfold model. (An example of both predictions is shown in Additional file 1, Figure S5.)
Figure 3. Performance of RNA secondary structure prediction with the γcentroid estimator and the pseudoexpected MCC/Fscore (Method M2). Performance of RNA secondary structure prediction with the γcentroid estimator and the pseudoexpected MCC (Fscore) (the estimator Eq. (24) with MCC (Fscore); Method M2); "XgCentroidpMCC" ("XgCentroidpF") where X is the McCaskill or CONTRAfold model. The curves (XgCentroid) indicate the performance of the γcentroid estimator [10] with the McCaskill model and the CONTRAfold model. For comparison, we have also plotted the performance of RNAfold [14], Sfold [5] and Simfold [11] (red points). See Additional file 1, Figure S4 for performances of MEA estimators used in Do et al. [7].
It is also much better than the performance of RNAfold, Sfold and Simfold, all of which return a single prediction. Note that Method M2 with a fixed probabilistic model (e.g., the McCaskill model or the CONTRAfold model) generally achieves performance that differs from that of the γcentroid estimator with the same model for any γ value. This is because Method M2 automatically selects the secondary structure with the best pseudoexpected MCC/Fscore from a set of secondary structures given by the γcentroid estimator for 17 γ values, while each point in a SENPPV curve of the γ centroid estimator comes from a fixed γvalue.
Table 2 shows that the computational time of Method M2 is much shorter than for Method M1. This is because we do not need to perform any stochastic sampling in Method M2. In Figure 3, we also plotted the performance of Sfold [5], Simfold [11] and RNAfold [14] (the points in red). The results indicate that the secondary structure predicted by Method M2 achieved better accuracy than those methods.
The comparison between the 2nd and 3rd rows in Table 2 indicates that there is only small overhead for the computation of the estimator of Method M2, compared with the γcentroid estimator with a fixed γ parameter [10]. The reasons can be summarized as follows. The CYKtype algorithm of the Nussinovstyle dynamic programming for computing a consistent RNA secondary structure is faster than the InsideOutsidetype algorithm for computing the basepairing probability matrix in the γcentroid estimator, even though both algorithms have the same computational complexity. Moreover, we do not need to employ the CYKtype algorithm for the γ centroid estimator with γ ≤ 1 because we only select the basepairs whose basepairing probability is larger than 1/(γ + 1) [10]. Also, the computation of the pseudoexpected MCC/Fscore of a given secondary structure is fast enough when the basepairing probability matrix is computed beforehand.
In summary, by combining the pseudoexpected accuracy with the γcentroid estimator, we successfully predict the wellbalanced secondary structure between SEN and PPV (with small overhead compared to CentroidFold) and the performance (with CONTRAfold model) is better than that of RNAfold, Simfold, Sfold and CentroidFold.
Discussion and Conclusion
In this study, we introduced the pseudoexpected accuracy, (with respect to commonly used accuracy measures in RNA secondary structure prediction: sensitivity, PPV, MCC or Fscore) of a given RNA secondary structure under a probability distribution of possible secondary structures. The pseudoexpected accuracy can be computed much more easily than the expected accuracy, because it is computed using the basepairing probability matrix of the RNA sequence. Although the pseudoexpected accuracy of a given secondary structure is not equal to the expected accuracy of the structure, our computational experiments have indicated that the pseudoexpected accuracy of a given secondary structure is a good approximation of the expected accuracy of the structure when SEN, PPV, MCC and Fscore were used as the accuracy measure. This finding is one of the contributions of this study, which has not been reported in previous research.
Based on this finding, we introduced the approximate estimator that maximizes the pseudoexpected accuracy of a prediction by stochastic sampling, which achieved favorable accuracy in our computational experiments. Although the computational cost of this estimator is much smaller than the estimator that maximizes the expected accuracy, it is still unacceptably slow. Therefore, we then proposed the combination of the pseudoexpected MCC/Fscore and the γcentroid estimator, which produces one wellbalanced secondary structure with small computational overhead. The computational experiments indicate that this approach achieved the best accuracy among stateoftheart tools. To employ the γcentroid estimator in Method M2 is suitable because the γcentroid estimator is able to represent a secondary structures with an arbitrary balance between the expected TP, TN, FP and FN by adjusting the parameter γ (see Eq. (18)). This, however, does not prove that there always exists a γ such that the γcentroid estimator achieves the best pseudoexpected MCC or Fscore. Note that the combination of the pseudoexpected MCC/Fscore with the MEAbased estimator proposed by [7] is not suitable because the estimator has a bias to MCC and Fscore, compared to the γcentroid estimator [10].
Although the tradeff between SEN and PPV is inherent, and MCC or Fscore is not always the best choice of quality measure for predicted secondary structures, the proposed method (Method M2) can be applicable when only a single structure is required. The pseudoexpected MCC/Fscore is also employed as a ranking measure of several predicted secondary structures.
Remarks about terminology: "maximum expected accuracy"
As we described in the Introduction section, the term "maximum (maximizing) expected accuracy" (MEA) has been used in a number of previous studies [6,7,10,26] as well as this study. From a mathematical viewpoint, the MEA (estimator) is a (point) estimator described as follows. Given a predictive space Y that contains all the possible candidate solutions of the target problem, a function Acc(θ, y) for θ ∈ Y and y ∈ Y , and a probability distribution p(θD) on Y given data D, then the estimator
is introduced. When this estimator is called a "maximum expected accuracy" (MEA) estimator, Acc(θ, y) is equal to an accuracy measure (or is designed according to an accuracy measure) for a reference θ and a prediction y. This also implies that p(θD) is considered to be a probability distribution of references, which is misleading because p(θD) does not usually represent the distribution. In RNA secondary structure prediction, for example, The McCaskill model provides not a probability distribution of reference secondary structures but rather a full ensemble of possible secondary structures [16].
The estimator of Eq. (25) with a welldesigned function Acc(θ, y) according to accuracy measures for a target problem and a probability distribution p(θD) of solutions empirically achieves better performance than other estimators such as the maximum likelihood estimator and the centroid estimator (i.e., the estimators that minimize the expected hamming difference) in RNA secondary structure predictions [7,10] and in alignments for biological sequences [25].
Difficulty of computing Eq. (20) with MCC and Fscore
Eq. (20) with MCC and Fscore can be rewritten as
respectively. Note that Eq. (26) is an approximation of Eq. (20) with MCC since TN (i.e., the number of truenegative basepairs) is much larger than the others in RNA secondary structure predictions.
The denominators in both equations prevent division of this optimization problem into subproblems, which is required to design a dynamic programming algorithm, and hence no efficient algorithms to compute Eqs. (26) and (27) have yet been devised. Note that the "argmax" operation for only the numerator can be efficiently solved by dynamic programming [33]. (This observation does not prove that there exists no efficient (polynomial time) algorithm for computing Eq. (20) with MCC and Fscore.)
The proposed methods are extendable to other situations
We are able to introduce the pseudoexpected accuracy for common secondary structure prediction of multiple alignments of RNA sequences, because there are several probability distributions for the common secondary structures, for example, the RNAalifold model [34,35] and the Pfold model [36]. Also, the γcentroid estimator can be extended to common secondary structure prediction [10], and the pseudoexpected MCC/Fscore combined with the estimator is useful to predict the common secondary structure that balances between SEN and PPV (See [37]).
Recently, Lu et al. [6] proposed the relaxed SEN, PPV and MCC, where slippage of basepair is allowed in computing those measures. It is possible to design the γcentroidtype estimator that fits with those measures and also to introduce pseudoexpected accuracy of those measures. Moreover, the methods used in this paper can be extended to more general types of estimation problems (cf. [17]) with various accuracy measures that are defined by using TP, TN, FP and FN (cf. [29]).
The method presented in this paper can be applied to local alignments for biological sequences because the γcentroid estimator was also introduced in the problem [25]. In contrast to global alignment problems, the balance between SEN and PPV with respect to aligned bases is important in local alignment problems.
Authors' contributions
MH and KA conceived the study. MH developed the algorithms, performed the experiments and wrote the manuscript. KS implemented the algorithm in the CentroidFold software. All authors have read and approved the final manuscript.
Acknowledgements
This work was supported in part by the "Functional RNA Project" funded by the New Energy and Industrial Technology Development Organization (NEDO) of Japan and in part by a GrantinAid for Scientific Research on Innovative Areas. The authors thank Drs/Profs H. Kiryu, M. C. Frith and T. Mituyama for valuable comments.
References

GriffithsJones S, Moxon S, Marshall M, Khanna A, Eddy SR, Bateman A: Rfam: annotating noncoding RNAs in complete genomes.

Andronescu M, Bereg V, Hoos H, Condon A: RNA STRAND: the RNA secondary structure and statistical analysis database.
BMC Bioinformatics 2008, 9:340. PubMed Abstract  BioMed Central Full Text  PubMed Central Full Text

Gardner PP, Daub J, Tate JG, Nawrocki EP, Kolbe DL, Lindgreen S, Wilkinson AC, Finn RD, GriffithsJones S, Eddy SR, Bateman A: Rfam: updates to the RNA families database.
Nucleic Acids Res 2009, 37:D136140. PubMed Abstract  Publisher Full Text  PubMed Central Full Text

Schroeder SJ: Advances in RNA structure prediction from sequence: new tools for generating hypotheses about viral RNA structurefunction relationships.
J Virol 2009, 83:63266334. PubMed Abstract  Publisher Full Text  PubMed Central Full Text

Ding Y, Chan CY, Lawrence CE: Sfold web server for statistical folding and rational design of nucleic acids.
Nucleic Acids Res 2004, (32 Web Server):135141. PubMed Abstract  Publisher Full Text  PubMed Central Full Text

Lu ZJ, Gloor JW, Mathews DH: Improved RNA secondary structure prediction by maximizing expected pair accuracy.
RNA 2009, 15:18051813. PubMed Abstract  Publisher Full Text  PubMed Central Full Text

Do C, Woods D, Batzoglou S: CONTRAfold: RNA secondary structure prediction without physicsbased models.
Bioinformatics 2006, 22:e9098. PubMed Abstract  Publisher Full Text

Engelen S, Tahi F: Tfold: efficient in silico prediction of noncoding RNA secondary structures.
Nucleic Acids Res 2010, 38:24532466. PubMed Abstract  Publisher Full Text  PubMed Central Full Text

Parisien M, Major F: The MCFold and MCSym pipeline infers RNA structure from sequence data.
Nature 2008, 452:5155. PubMed Abstract  Publisher Full Text

Hamada M, Kiryu H, Sato K, Mituyama T, Asai K: Prediction of RNA secondary structure using generalized centroid estimators.
Bioinformatics 2009, 25:465473. PubMed Abstract  Publisher Full Text

Andronescu M, Condon A, Hoos H, Mathews D, Murphy K: Efficient parameter estimation for RNA secondary structure prediction.
Bioinformatics 2007, 23:1928. Publisher Full Text

Zuker M, Stiegler P: Optimal computer folding of large RNA sequences using thermodynamics and auxiliary information.
Nucleic Acids Res 1981, 9:133148. PubMed Abstract  Publisher Full Text  PubMed Central Full Text

Zuker M: Mfold web server for nucleic acid folding and hybridization prediction.
Nucleic Acids Res 2003, 31(13):34063415. PubMed Abstract  Publisher Full Text  PubMed Central Full Text

Hofacker I, Fontana W, Stadler P, Bonhoeffer S, Tacker M, Schuster P: Fast folding and comparison of RNA secondary structures.
Monatsh Chem 1994, 125:167188. Publisher Full Text

Mathews D, Disney M, Childs J, Schroeder S, Zuker M, Turner D: Incorporating chemical modification constraints into a dynamic programming algorithm for prediction of RNA secondary structure.
Proc Natl Acad Sci USA 2004, 101:72877292. PubMed Abstract  Publisher Full Text  PubMed Central Full Text

McCaskill JS: The equilibrium partition function and base pair binding probabilities for RNA secondary structure.
Biopolymers 1990, 29(67):11051119. PubMed Abstract  Publisher Full Text

Carvalho L, Lawrence C: Centroid estimation in discrete highdimensional spaces with applications in biology.
Proc Natl Acad Sci USA 2008, 105:32093214. PubMed Abstract  Publisher Full Text  PubMed Central Full Text

Ding Y, Chan C, Lawrence C: RNA secondary structure prediction by centroids in a Boltzmann weighted ensemble.
RNA 2005, 11:11571166. PubMed Abstract  Publisher Full Text  PubMed Central Full Text

Hamada M, Sato K, Kiryu H, Mituyama T, Asai K: Predictions of RNA secondary structure by combining homologous sequence information.
Bioinformatics 2009, 25:i330338. PubMed Abstract  Publisher Full Text  PubMed Central Full Text

Kiryu H, Kin T, Asai K: Robust prediction of consensus secondary structures using averaged base pairing probability matrices.
Bioinformatics 2007, 23:434441. PubMed Abstract  Publisher Full Text

Seemann S, Gorodkin J, Backofen R: Unifying evolutionary and thermodynamic information for RNA folding of multiple alignments.
Nucleic Acids Res 2008, 36:63556362. PubMed Abstract  Publisher Full Text  PubMed Central Full Text

Hamada M, Sato K, Kiryu H, Mituyama T, Asai K: CentroidAlign: fast and accurate aligner for structured RNAs by maximizing expected sumofpairs score.
Bioinformatics 2009, 25:32363243. PubMed Abstract  Publisher Full Text

Bradley RK, Roberts A, Smoot M, Juvekar S, Do J, Dewey C, Holmes I, Pachter L: Fast statistical alignment.
PLoS Comput Biol 2009, 5:e1000392. PubMed Abstract  Publisher Full Text  PubMed Central Full Text

Bradley RK, Pachter L, Holmes I: Specific alignment of structured RNA: stochastic grammars and sequence annealing.
Bioinformatics 2008, 24:26772683. PubMed Abstract  Publisher Full Text  PubMed Central Full Text

Frith MC, Hamada M, Horton P: Parameters for Accurate Genome Alignment.
BMC Bioinformatics 2010, 11:80. PubMed Abstract  PubMed Central Full Text

Kall L, Krogh A, Sonnhammer EL: An HMM posterior decoder for sequence feature prediction that includes homology information.
Bioinformatics 2005, 21(Suppl 1):i251257. PubMed Abstract  Publisher Full Text

Michal N, Tomas V, Brona B: The Highest Expected Reward Decoding for HMMs with Application to Recombination Detection. [http://arxiv.org/abs/1001.4499] webcite

Gross S, Do C, Sirota M, Batzoglou S: CONTRAST: a discriminative, phylogenyfree approach to multiple informant de novo gene prediction.
Genome Biol 2007, 8:R269. PubMed Abstract  BioMed Central Full Text  PubMed Central Full Text

Baldi P, Brunak S, Chauvin Y, Andersen CA, Nielsen H: Assessing the accuracy of prediction algorithms for classification: an overview.
Bioinformatics 2000, 16:412424. PubMed Abstract  Publisher Full Text

Durbin R, Eddy S, Krogh A, Mitchison G: Biological sequence analysis. Cambridge, UK: Cambridge University press; 1998.

Ding Y, Lawrence CE: A statistical sampling algorithm for RNA secondary structure prediction.
Nucleic Acids Res 2003, 31:72807301. PubMed Abstract  Publisher Full Text  PubMed Central Full Text

Sato K, Hamada M, Asai K, Mituyama T: CENTROID FOLD: a web server for RNA secondary structure prediction.
Nucleic Acids Res 2009, 37:W277280. PubMed Abstract  Publisher Full Text  PubMed Central Full Text

Holmes I, Durbin R: Dynamic programming alignment accuracy.
J Comput Biol 1998, 5:493504. PubMed Abstract  Publisher Full Text

Bernhart S, Hofacker I, Will S, Gruber A, Stadler P: RNAalifold: improved consensus structure prediction for RNA alignments.
BMC Bioinformatics 2008, 9:474. PubMed Abstract  BioMed Central Full Text  PubMed Central Full Text

Hofacker IL, Fekete M, Stadler PF: Secondary structure prediction for aligned RNA sequences.
J Mol Biol 2002, 319(5):10591066. PubMed Abstract  Publisher Full Text

Knudsen B, Hein J: Pfold: RNA secondary structure prediction using stochastic contextfree grammars.
Nucleic Acids Res 2003, 31(13):34233428. PubMed Abstract  Publisher Full Text  PubMed Central Full Text

Hamada M, Sato K, Asai K: Improving the accuracy of predicting secondary structure for aligned RNA sequences.
Nucleic Acids Res 2010. PubMed Abstract  Publisher Full Text