Background
De novo prediction of protein threedimensional (3D) structure from the primary sequence remains a fundamental and extraordinarily challenging problem 1. Contact maps, or similar distance restraints have been proposed as intermediate steps between the primary sequence and the 3D structure (e.g. in 234), for various reasons: unlike 3D coordinates, they are invariant to rotations and translations, hence less challenging to predict by machine learning systems 4; quick, effective algorithms exist to derive 3D structures from them, for instance stochastic optimisation methods 56, distance geometry 78, or algorithms derived from the NMR literature and elsewhere 91011. Numerous methods have been developed for protein residue contact map prediction 23412 and coarse (secondary structure element level) contact map prediction 13, and some improvements are slowly occurring (e.g. in 12, as shown by the CASP6 experiment 14).
Still, accurate prediction of residue contact maps is far from being achieved and limitations of existing prediction methods have again emerged at CASP6 and from automatic evaluation of structure prediction servers such as EVA 15. There are various reasons for this: the number of positive and negative examples (contacts vs. non contacts) is strongly unbalanced; the number of examples grows with the squared length of the protein making this a tough computational challenge; capturing long ranged interactions in the primary sequence is difficult, hence grasping an adequate global picture of the map is a formidable problem. For this reason simpler, alternative representations of protein topologies are particularly appealing, provided that they are informative and, especially, predictable (e.g. see 16).
In this paper we focus on one such representation: the principal eigenvector (PE) of residue contact maps. The PE is a sequence of the same length as a protein's primary sequence. A vast machinery of tools for sequence processing is available (see e.g. 17 for a review). Moreover, recently 18 a branchandbound algorithm was described that is capable of reconstructing the contact map from the exact PE, at least for single domain proteins of up to 120 amino acids. This means that the PE contains most of the information encoded in the contact map. Predicting the PE is thus interesting: it leads to a drastic reduction in the size of the problem compared to twodimensional contact maps, i.e. considerable data compression, and also a reduction in the scale length of interactions that need to be learned; contact maps may be derived from the PE by modifying the reconstruction algorithm in 18 to deal with noise in the PE; alternatively the PE may be adopted as an additional input feature to systems for the direct prediction of contact maps (such as 4); information contained in the PE may be used, in combination with other constraints, to guide the search for optimal 3D configurations; predicted PE may prove useful to identify domains, as in 19, and discussed in 20.
In this paper, we model the problem of inferring the PE as a classification task with multiple classes. We use machine learning methods to map amino acids into their corresponding component of the principal eigenvector. Similarly to 21, we adopt bidirectional recurrent neural networks (BRNNs) 22 with shortcut connections, accurate coding of input profiles obtained from multiple sequence alignments, secondary structure predictions, second stage filtering by recurrent neural networks, and finally largescale ensembles of predictors. Our models classify correctly up to 72.6% residues, 16% above a baseline statistical predictor always assigning a residue to the most numerous PE class.
To prove that these levels can lead to improved contact maps, we incorporate the predicted PE into a stateoftheart system for contact map prediction 413. Our tests show that the PE yields sizeable gains, and that these gains are especially significant for longranged contacts, which are known to be both harder to predict and critical for accurate 3D reconstruction.
Results and discussion
Principal eigenvector prediction
We evaluate model performances using different prediction indices. If the task is the prediction of the eigenvector components λ¯x¯
MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacuaH7oaBgaqeaiqbdIha4zaaraaaaa@3009@_{i }(see the methods section for definitions) in m classes, we measure: Q_{m}, or overall percentage of correctly predicted amino acids; the set Q_{0}, ..., Q_{m1}, where each Q_{j }is the percentage of correctly classified amino acids whose eigenvector component belongs to interval I_{j+1}; an analogous of the SOV measure 23 adapted for the case of m classes. The intent in this case is to measure the quality of prediction over contiguous segments of amino acids belonging to the same class. Finally, we compare our methodology with a baseline predictor that assigns each amino acid to its most frequently occurring class (as for instance in 242526).
We train different ensembles of BRNNs. Differences depend on whether or not we use output filtering by second stage networks (Eq.6) and whether or not the input encoding includes predicted secondary structure from Porter 21 and the hydrophobicity interaction scale in 20 (Table I, column 2). Tables 1, 2 and 3 show respectively estimated performance indices for classification in two, three and four classes. The first three columns indicate whether secondary structure, hydrophobicity profile and second stage filtering are employed in the network ensemble (see table legends for details).
<p>Table 1</p>
PE prediction: twoclass problem. Accuracy estimates with 95% confidence intervals and SOV. A * in the first three columns of a row indicates whether the results are obtained augmenting the network input with secondary structure predicted by Porter (P), hydrophobicity profile using the interactivity scale of [20] (H) and using second stage filtering network (F).
P
H
F
Q
_{2}
Q
_{0}
Q
_{1}
SOV



72.0 ± .6
73.1 ± .6
70.8 ± .6
44.4


*
72.1 ± .6
73.4 ± .6
70.7 ± .6
46.0
*


72.3 ± .6
73.1 ± .6
71.4 ± .6
47.6
*

*
72.3 ± .6
72.8 ± .6
71.9 ± .6
49.6
*
*

72.5 ± .6
74.0 ± .6
71.0 ± .6
47.2
*
*
*
72.6 ± .6
73.8 ± .6
71.2 ± .6
49.8
baseline
56.8 ± .5
58.4 ± .5
55.1 ± .5

<p>Table 2</p>
PE prediction: threeclass problem. Accuracy estimates with 95% confidence intervals and SOV. A * in the first three columns of a row indicates whether the results are obtained augmenting the network input with secondary structure predicted by Porter (P), hydrophobicity profile using the interactivity scale of [20] (H) and using second stage filtering network (F).
P
H
F
Q
_{3}
Q
_{0}
Q
_{1}
Q
_{2}
SOV



55.8 ± .5
63.7 ± .5
35.5 ± .4
67.6 ± .6
38.3


*
56.2 ± .5
61.6 ± .6
40.6 ± .5
65.8 ± .6
40.3
*


56.3 ± .6
65.8 ± .6
36.9 ± .5
65.5 ± .6
42.1
*

*
56.6 ± .6
64.1 ± .6
41.1 ± .4
63.9 ± .6
43.6
*
*

56.4 ± .5
65.2 ± .6
36.8 ± .4
66.4 ± .6
42.6
*
*
*
56.7 ± .5
63.3 ± .6
41.4 ± .4
64.6 ± .6
44.0
baseline
39.8 ± .4
50.6 ± .5
8.5 ± .2
59.3 ± .6

<p>Table 3</p>
PE prediction: fourclass problem. Accuracy estimates with 95% confidence intervals and SOV. A * in the first three columns of a row indicates whether the results are obtained augmenting the network input with secondary structure predicted by Porter (P), hydrophobicity profile using the interactivity scale of [20] (H) and using second stage filtering network (F).
P
H
F
Q
_{4}
Q
_{0}
Q
_{1}
Q
_{2}
Q
_{3}
SOV



45.6 ± .5
59.9 ± .6
29.4 ± .4
26.8 ± .4
65.0 ± .6
33.2


*
46.0 ± .5
58.5 ± .5
30.6 ± .4
30.5 ± .4
63.3 ± .6
34.6
*


46.2 ± .5
62.1 ± .6
30.5 ± .4
27.3 ± .4
63.1 ± .6
37.3
*

*
46.5 ± .5
60.7 ± .5
32.3 ± .4
29.6 ± .4
61.8 ± .5
37.4
*
*

45.9 ± .5
61.5 ± .5
30.0 ± .4
27.3 ± .3
63.3 ± .6
37.1
*
*
*
46.5 ± .5
60.0 ± .5
32.2 ± .4
30.4 ± .4
61.9 ± .6
37.8
baseline
30.7 ± .4
48.1 ± .5
5.4 ± .2
8.5 ± .2
59.0 ± .6

In all multiclass prediction cases, the best network ensemble shows an increment of global predictive accuracy of ≈ 16% with respect to the baseline predictor. The SOV and the overall accuracy increase using filters and augmenting the number of input features with hydrophobicity scales (marginally) and secondary structure (significantly). Interestingly, predicted secondary structure is a valuable feature: in all cases, using true secondary structures results in only moderate improvements with respect to the performance obtained with secondary structure predicted by Porter 21 (data not shown).
In the 2class problem, Q_{2 }exceeds 72% with the two classes almost equally well predicted. In this case, the network with full features finds a nearly optimal (Bayesian) decision threshold. This is not surprising because the threshold on the eigenvector component was chosen so as to divide the training set values in two equally distributed halves. The 3 and 4class prediction problems are more difficult to solve, but the observed improvements over the baseline predictor are roughly the same as in the 2class case. Strong improvements over the baseline predictor are observed especially for the intermediate classes (Q_{1 }in Table 2, Q_{1 }and Q_{2 }in Table 3). Interestingly, these classes are more difficult to predict even if all classes are nearly equally distributed. This is possibly because boundary classes (Q_{0 }and Q_{m1}) correspond to welldefined situations, i.e. isolated residues or residues with high connectivity, for which clear signal exists in the data. A typical example of 4class PE prediction is shown in figure 2.
<p>Figure 1</p>
Distribution of λ¯x¯
MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacuaH7oaBgaqeaiqbdIha4zaaraaaaa@3009@_{i }values in the training set
Distribution of λ¯x¯
MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacuaH7oaBgaqeaiqbdIha4zaaraaaaa@3009@_{i }values in the training set. See text for details.
<p>Figure 2</p>
4class principal eigenvector prediction for protein 1A2P (108 amino acids)
4class principal eigenvector prediction for protein 1A2P (108 amino acids). Solid line: exact eigenvector class. Dashed line: predicted eigenvector class. The class value is averaged over a moving window of 5 residues.
Contact map prediction from predicted PE
As a final step, we test the possibility of directly using the information encoded in the PE to improve stateoftheart residue contact map predictors. We choose the model based on DAGRNNs described in 4 and 13. This model was among the most successful contact map predictors at the CASP5 competition 27. The architecture we adopt is identical to the one described in 13 and used at CASP5, except for the presence of shortcut connections and for the ensembling technique (see methods section). These differences allow a substantially (roughly 5fold) faster training, and yield marginally improved results compared to 13 when the same input features and same training/test sets are adopted (not shown).
To ensure fairness, here we retrain DAGRNNs from scratch using the same training and testing sets used to predict the PE. The sets are first processed to remove sequences longer than 200 amino acids (for computational reasons, as in 13), leaving 1275 proteins in the training set and 327 proteins in the test set. Two amino acids are defined as being in contact if the distance between their C_{α }is below a contact threshold. We consider two different contact thresholds: 8 and 12 Å. For comparison purposes, we encode each pair (i, j) of amino acids in the input by four different features: a 20 × 20 matrix representing the probability distribution of pairs of amino acids observed in the two corresponding columns of the alignment (MA); MA plus the actual discretised 4class PE component for both residue i and j (MA_PE); MA plus the actual secondary structure (3 classes) and binary thresholded (at 25%) relative solvent accessibility (MA_SS_ACC); and finally, the previous feature plus the actual 4class PE components (MA_SS_ACC_PE). We train 8 predictors, with the same architecture, one for each input feature and contact threshold.
Differently from the training phase, testing takes place by encoding each pair (i, j) on input with the predicted 4class PE component as given by the filtered ensemble of BRNNs using profiles, predicted secondary structure and hydrophobicity profiles (table 3, row PHF). Secondary structure and solvent accessibility information input into the DAGRNN is also predicted during testing. These predictions are obtained from an architecture identical to the one adopted to predict the PE, and trained on the same training set. Hence the protocol we adopt leads to fully realistic results, since no protein in the the test set shows significant sequence similarity to any of the structures used to train the contact map predictor and all the underlying feature predictors.
Tables 4, 5, 6 and 7 show performance indices for all the 8 networks. Indices considered are: accuracy P = TP/(TP + FP), with TP = true positives and FP = false positives; coverage R = TP/(TP + FN), with FN = false negatives; F_{1}, defined as the harmonic mean of accuracy and coverage (F_{1 }= 2PR/(P + R)); P_{nc}, the percentage of correctly predicted noncontacts. Performances are computed for three different sets of contacts, based on the separation of two residues in the linear sequence: i  j ≥ {6,12, 24}. In tables 4 and 6 we report P, R and F_{1 }when the threshold between contacts and noncontacts is set to 0.5. In tables 5 and 7, for consistency with CASP assessment rules 14, we report P and R when only the top N/5 and N/2 contacts are considered, N being the length of the protein. In this case contacts are ranked, and top contacts are selected, based on their expected probability as estimated by the predictor.
<p>Table 4</p>
Performance results for contact map prediction. Contact threshold: 8 Å. Accuracy, coverage and F1 (as %) for 8 Å contact map predictor for distance separations greater than 5, 11 and 23 amino acids.
i  j ≥ 6
i  j ≥ 12
i  j ≥ 24
P
R
F_{1}
P_{nc}
P
R
F_{1}
P_{nc}
P
R
F_{1}
P_{nc}
MA
0
0
0
97.2
0
0
0
97.6
0
0
0
97.9
MA_PE
39.4
12.2
18.6
97.5
36.2
8.4
13.5
97.7
27.8
2.0
3.7
97.9
MA_SS_ACC
50.5
7.4
12.9
97.4
48.8
4.0
7.4
97.6
25.7
.2
.3
97.9
MA_SS_ACC_PE
43.3
11.3
17.9
97.5
38.9
7.2
12.1
97.7
25.5
2.2
4.1
97.9
<p>Table 5</p>
Performance results for contact map prediction. Contact threshold: 8 Å. Accuracy and coverage (as %) for 8 Å contact map predictor for distance separations greater than 5, 11 and 23 amino acids, when the top N/5 and top N/2 contacts are considered (where N is the length of the protein).
i  j ≥ 6
i  j ≥ 12
i  j ≥ 24
N/5
N/2
N/5
N/2
N/5
N/2
P
R
P
R
P
R
P
R
P
R
P
R
MA
30.8
3.9
26.2
8.5
23.9
3.8
19.2
7.8
14.1
3.3
11.1
6.6
MA_PE
43.0
5.5
34.6
11.2
34.2
5.5
26.6
10.8
19.6
4.6
15.0
8.9
MA_SS_ACC
44.4
5.7
36.0
11.6
34.2
5.5
26.6
10.8
17.8
4.2
14.7
8.7
MA_SS_ACC_PE
46.4
5.9
36.6
11.8
35.4
5.7
27.0
11.0
19.8
4.6
15.7
9.3
<p>Table 6</p>
Performance results for contact map prediction. Contact threshold: 12 Å. Accuracy, coverage and F1 (as %) for 12 Å contact map predictor for distance separations greater than 5, 11 and 23 amino acids.
i  j ≥ 6
i  j ≥ 12
i  j ≥ 24
P
R
F_{1}
P_{nc}
P
R
F_{1}
P_{nc}
P
R
F_{1}
P_{nc}
MA
60.4
10.6
18.1
87.2
55.8
0.1
0.1
87.8
38.9
0.03
0.06
88.8
MA_PE
49.5
24.5
32.8
88.6
39.4
16.8
23.6
89.3
34.5
13.6
19.5
89.9
MA_SS_ACC
61.6
19.6
29.7
88.2
48.9
7.5
13.1
88.5
40.2
2.8
5.3
89.0
MA_SS_ACC_PE
54.2
23.5
32.8
88.6
42.2
14.6
21.7
89.2
36.7
10.9
16.8
89.7
<p>Table 7</p>
Performance results for contact map prediction. Contact threshold: 12 Å. Accuracy and coverage (as %) for 12 Å contact map predictor for distance separations greater than 5, 11 and 23 amino acids, when the top N/5 and top N/2 contacts are considered (where N is the length of the protein).
i  j ≥ 6
i  j ≥ 12
i  j ≥ 24
N/5
N/2
N/5
N/2
N/5
N/2
P
R
P
R
P
R
P
R
P
R
P
R
MA
79.6
2.0
71.9
4.6
50.1
1.6
46.2
3.8
43.3
1.9
38.1
4.2
MA_PE
87.5
2.2
81.6
5.2
59.8
1.9
54.1
4.4
47.9
2.1
42.4
4.7
MA_SS_ACC
89.7
2.3
85.3
5.5
61.3
2.0
54.9
4.5
43.7
1.9
39.4
4.4
MA_SS_ACC_PE
89.9
2.3
85.5
5.5
62.5
2.0
55.6
4.6
49.9
2.2
43.8
4.9
As evident from the tables, the introduction of PE predictions increases the F_{1 }measure in all cases. This is true for both 8 and 12 Å maps, and for all separation thresholds. An improvent is observed both in the MA_PE vs. MA case and in the MA_SS_ACC_PE vs. MA_SS_ACC case. In all cases the introduction of the predicted PE yields larger performance gains than secondary structure and solvent accessibility combined. Interestingly, the gains become more significant for longer range contacts. For instance for i  j ≥ 24 F_{1 }grows from 0.3% to 4.1% at 8Å and from 5.3% to 16.8% at 12Å (MA_SS_ACC_PE vs. MA_SS_ACC). PEbased networks are more confident away from the main diagonal (a typical example is shown in figure 3), with a better balance between false positives and false negatives.
<p>Figure 3</p>
Examples of contact map predictions at 12 Å for protein 1A2P (108 amino acids)
Examples of contact map predictions at 12 Å for protein 1A2P (108 amino acids). Exact map in the topright half, predicted map in the bottomleft half. Prediction by MA_SS_ACC on the left, MA_SS_ACC_PE on the right (see text for details).
When we take into account only small numbers (N/5 and N/2) of contacts considered most likely by the predictor, the gains become less marked, but remain significant, especially for longer range contacts: for i  j ≥ 24 at 8Å, when considering the top N/5 contacts, P grows from 14.1% to 19.6% in the MA vs. MA_PE case and from 17.8% to 19.8% in the in the MA_SS_ACC vs. MA_SS_ACC_PE case. Similar gains (from 43.3% to 47.9% and from 43.7% to 49.9%, respectively) are observed for the 12Å predictors.
Residue contact map predictors at CASP6 14 were evaluated on a small set (11) of Novel Fold targets. The performances of the best system (group RR301) on the top N/5 contacts were 24% and 22% (accuracy) and 5.6% and 5% (coverage) for minimum residue separations of 12 and 24, respectively. Although the statistical relevance of a set of only 11 targets is limited, our predictor's accuracy on it compares favourably with the best CASP6 predictors, achieving 36.5% accuracy and 9.8% coverage for separation of at least 12 and 19.6% accuracy and 5.4% coverage for separation of at least 24.
Conclusion
We developed sophisticated predictors of a novel sequential feature of protein structure: the principal eigenvector of residue contact maps. Our predictors classify correctly up to 72.6% of residues, and show large gains over simple baseline statistical predictors.
We showed that predicted principal eigenvectors can be effectively used as an additional input feature to a stateoftheart method for contact map prediction, yielding sizeable gains especially for longrange contacts which are particularly critical for accurate protein 3D reconstruction.
These results suggest a number of futher points to investigate:
• The algorithm in 18 may be directly tested in noisy contexts, and extended to increase its robustness. This may give rise to an alternative pipeline for the prediction of contact maps.
• The PE could be used directly to improve protein domain predictors 1920.
• PEbased maps may be adopted to guide the ab initio reconstruction of quick, draft C_{α }traces, for instance using a stochastic search algorithm similar to 5.
• Residue coordination number correlates well with the PE – as such, predicted coordination number 26 may yield similar gains to contact map prediction, while providing a more intuitive structural representation of a protein.
Ultimately, the third point is the most crucial test of the validity of our approach. Even if the 3D models produced were fairly coarse, they might be provide a valuable source of information, for instance to identify protein functions more accurately than it would be possible by sequence alone 28. Although training a contact map prediction system is computationally expensive, once training is over, generating predictions is fast. Even on a small cluster of machines, this may allow multigenomic scale structural prediction efforts in manageable times.
Methods
The contact map of a protein with N amino acids is a symmetric N × N matrix C, with elements C_{ij }defined as:
C
i
j
=
{
1
if amino acid i and j are in contact
0
otherwise
(
1
)
MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacqWGdbWqdaWgaaWcbaGaemyAaKMaemOAaOgabeaakiabg2da9maaceaabaqbaeaabiGaaaqaaiabigdaXaqaaiabbMgaPjabbAgaMjabbccaGiabbggaHjabb2gaTjabbMgaPjabb6gaUjabb+gaVjabbccaGiabbggaHjabbogaJjabbMgaPjabbsgaKjabbccaGiabbMgaPjabbccaGiabbggaHjabb6gaUjabbsgaKjabbccaGiabbQgaQjabbccaGiabbggaHjabbkhaYjabbwgaLjabbccaGiabbMgaPjabb6gaUjabbccaGiabbogaJjabb+gaVjabb6gaUjabbsha0jabbggaHjabbogaJjabbsha0bqaaiabicdaWaqaaiabb+gaVjabbsha0jabbIgaOjabbwgaLjabbkhaYjabbEha3jabbMgaPjabbohaZjabbwgaLbaacaWLjaGaaCzcamaabmaabaGaeGymaedacaGLOaGaayzkaaaacaGL7baaaaa@70CC@
We define two amino acids as being in contact if the distance between their C_{α }is less than a given threshold. For the definition of the PE we adopt a fixed 8 Å threshold, while in the contact map prediction stage we test 8 Å and 12 Å thresholds. Alternative definitions are possible, for instance based on different mutual C_{α }distances (normally in the 7–12 Å range), or on C_{β}C_{β }atom distances (normally 6.5–8 Å), or on the minimal distance between two atoms belonging to the sidechain or backbone of the two residues (commonly 4.5 Å).
Let λ(C) = {λ : Cx = λx} be the spectrum of C, S
MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBamrtHrhAL1wy0L2yHvtyaeHbnfgDOvwBHrxAJfwnaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaWaaeGaeaaakeaaimaacqWFse=uaaa@3845@_{λ }= {x : Cx = λx} the corresponding eigenspace and λ¯
MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacuaH7oaBgaqeaaaa@2E78@ = max{λ ∈ λ(C)} the largest eigenvalue of C. The principal eigenvector of C, x¯
MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacuWG4baEgaqeaaaa@2E3D@, is the eigenvector corresponding to λ¯
MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacuaH7oaBgaqeaaaa@2E78@. x¯
MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacuWG4baEgaqeaaaa@2E3D@ can also be expressed as the argument which maximises the Rayleigh quotient:
∀
x
∈
S
λ
:
x
T
C
x
x
T
x
≤
x
¯
T
C
x
¯
x
¯
T
x
¯
(
2
)
MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBamrtHrhAL1wy0L2yHvtyaeHbnfgDOvwBHrxAJfwnaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaWaaeGaeaaakeaacqGHaiIicqWG4baEcqGHiiIZimaacqWFse=udaWgaaWcbaGaeq4UdWgabeaakiabcQda6maalaaabaGaemiEaG3aaWbaaSqabeaacqWGubavaaGccqWGdbWqcqWG4baEaeaacqWG4baEdaahaaWcbeqaaiabdsfaubaakiabdIha4baacqGHKjYOdaWcaaqaaiqbdIha4zaaraWaaWbaaSqabeaacqWGubavaaGccqWGdbWqcuWG4baEgaqeaaqaaiqbdIha4zaaraWaaWbaaSqabeaacqWGubavaaGccuWG4baEgaqeaaaacaWLjaGaaCzcamaabmaabaGaeGOmaidacaGLOaGaayzkaaaaaa@5872@
Eigenvectors are usually normalised by requiring their norm to be 1, e.g. x_{2 }= 1 ∀x ∈ S
MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBamrtHrhAL1wy0L2yHvtyaeHbnfgDOvwBHrxAJfwnaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaWaaeGaeaaakeaaimaacqWFse=uaaa@3845@_{λ}. Since C is an adjacency (real, symmetric) matrix, its eigenvalues are real. Since it is a normal matrix (A^{H}A = AA^{H}), its eigenvectors are orthogonal. Other basic properties can also be proven: the principal eigenvalue is positive; nonzero components of x¯
MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacuWG4baEgaqeaaaa@2E3D@ have all the same sign 29. Without loss of generality, we can assume they are positive, as in 18.
Ideally, prediction of the PE should be formulated as a sequential regression task in which each amino acid is mapped into its corresponding component of the PE. Here we consider two variations to the original problem. First, we model it as a classification task with multiple classes. Second, we predict the magnified eigenvector, i.e. λ¯x¯
MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacuaH7oaBgaqeaiqbdIha4zaaraaaaa@3009@ instead of x¯
MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacuWG4baEgaqeaaaa@2E3D@. Modelling regression problems as multiclass classifications is common practice, for instance in closely related tasks such as the prediction of protein solvent accessibility 263031. Predicting magnified eigenvector components has some advantages: by doing so we are simultaneously estimating the eigenvector components and the corresponding eigenvalue (as the norm of x¯
MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacuWG4baEgaqeaaaa@2E3D@ is equal to 1); the main eigenvalue correlates well with the protein length (correlation of 0.62 on our training set), hence it is likely predictable. An estimate of the eigenvalue will in general be needed when attempting to predict contact maps from the PE, either by using an algorithm similar to the one in 18, or more in general by attempting to satisfy the constraint:

∑
j
C
i
j
x
¯
j
−
λ
¯
x
¯
i

=
0
(
3
)
MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacqGG8baFdaaeqbqaaiabdoeadnaaBaaaleaacqWGPbqAcqWGQbGAaeqaaOGafmiEaGNbaebadaWgaaWcbaGaemOAaOgabeaakiabgkHiTiqbeU7aSzaaraGafmiEaGNbaebadaWgaaWcbaGaemyAaKgabeaakiabcYha8jabg2da9iabicdaWaWcbaGaemOAaOgabeqdcqGHris5aOGaaCzcaiaaxMaadaqadaqaaiabiodaZaGaayjkaiaawMcaaaaa@45E7@
Formally, the PE prediction task consists in learning a mapping f(·) : ℐ
MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBamrtHrhAL1wy0L2yHvtyaeHbnfgDOvwBHrxAJfwnaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaWaaeGaeaaakeaaimaacqWFqessaaa@3769@ → O
MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBamrtHrhAL1wy0L2yHvtyaeHbnfgDOvwBHrxAJfwnaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaWaaeGaeaaakeaaimaacqWFoe=taaa@383D@ from the space ℐ
MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBamrtHrhAL1wy0L2yHvtyaeHbnfgDOvwBHrxAJfwnaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaWaaeGaeaaakeaaimaacqWFqessaaa@3769@ of labelled input sequences to the space O
MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBamrtHrhAL1wy0L2yHvtyaeHbnfgDOvwBHrxAJfwnaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaWaaeGaeaaakeaaimaacqWFoe=taaa@383D@ of labelled output sequences. In practice, we want to predict a sequence of labels O = (o_{1}, ..., o_{N}), for a given sequence of inputs I = (i_{1}, ..., i_{N}), where each i_{j }∈ I is the input coding of the amino acid in position j. For PE prediction, we assume that there is a range R including all magnified eigenvector components, i.e. ∀j, λ¯x¯
MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacuaH7oaBgaqeaiqbdIha4zaaraaaaa@3009@_{j }∈ R, and we divide the range R into a series of m disjoint intervals, i.e. R=∪k=1mRk
MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacqWGsbGucqGH9aqpdaWeWaqaaiabdkfasnaaBaaaleaacqWGRbWAaeqaaaqaaiabdUgaRjabg2da9iabigdaXaqaaiabd2gaTbqdcqWIQisvaaaa@37D7@. We can represent each output label o_{j }as belonging to an alphabet of m symbols, i.e. o_{j }∈ Σ = {1, ..., m}, and o_{j }corresponds to the class or interval R_{k }in which the value of the jth magnified eigenvector component falls: o_{j }= k ⇔ λ¯x¯
MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacuaH7oaBgaqeaiqbdIha4zaaraaaaa@3009@_{j }∈ R_{k}.
Predictive architecture for the PE
To learn the mapping between our inputs ℐ
MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBamrtHrhAL1wy0L2yHvtyaeHbnfgDOvwBHrxAJfwnaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaWaaeGaeaaakeaaimaacqWFqessaaa@3769@ and outputs O
MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBamrtHrhAL1wy0L2yHvtyaeHbnfgDOvwBHrxAJfwnaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaWaaeGaeaaakeaaimaacqWFoe=taaa@383D@ we use a twolayered architecture composed of Bidirectional Recurrent Neural Networks (BRNN) 22 (also known as 1DRNN, e.g. in 13) of the same length as the amino acid sequence. Similarly to 21 we use BRNNs with shortcut connections. In these BRNNs, connections along the forward and backward hidden chains span more than 1residue intervals, creating shorter paths between inputs and outputs. These networks take the form:
o
j
=
N
(
O
)
(
i
j
,
h
j
(
F
)
,
h
j
(
B
)
)
h
j
(
F
)
=
N
(
F
)
(
i
j
,
h
j
−
1
(
F
)
,
…
,
h
j
−
S
(
F
)
)
h
j
(
B
)
=
N
(
B
)
(
i
j
,
h
j
+
1
(
B
)
,
…
,
h
j
+
S
(
B
)
)
j
=
1
,
…
,
N
MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBamrtHrhAL1wy0L2yHvtyaeHbnfgDOvwBHrxAJfwnaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaWaaeGaeaaakeaafaqabeabbaaaaeaacqWGVbWBdaWgaaWcbaGaemOAaOgabeaakiabg2da9GWaaiab=1q8onaaCaaaleqabaWaaeWaaeaacqWGpbWtaiaawIcacaGLPaaaaaGcdaqadaqaaiabdMgaPnaaBaaaleaacqWGQbGAaeqaaOGaeiilaWIaemiAaG2aa0baaSqaaiabdQgaQbqaamaabmaabaGaemOrayeacaGLOaGaayzkaaaaaOGaeiilaWIaemiAaG2aa0baaSqaaiabdQgaQbqaamaabmaabaGaemOqaieacaGLOaGaayzkaaaaaaGccaGLOaGaayzkaaaabaGaemiAaG2aa0baaSqaaiabdQgaQbqaamaabmaabaGaemOrayeacaGLOaGaayzkaaaaaOGaeyypa0Jae8xdX70aaWbaaSqabeaadaqadaqaaiabdAeagbGaayjkaiaawMcaaaaakmaabmaabaGaemyAaK2aaSbaaSqaaiabdQgaQbqabaGccqGGSaalcqWGObaAdaqhaaWcbaGaemOAaOMaeyOeI0IaeGymaedabaWaaeWaaeaacqWGgbGraiaawIcacaGLPaaaaaGccqGGSaalcqWIMaYscqGGSaalcqWGObaAdaqhaaWcbaGaemOAaOMaeyOeI0Iaem4uamfabaWaaeWaaeaacqWGgbGraiaawIcacaGLPaaaaaaakiaawIcacaGLPaaaaeaacqWGObaAdaqhaaWcbaGaemOAaOgabaWaaeWaaeaacqWGcbGqaiaawIcacaGLPaaaaaGccqGH9aqpcqWFneVtdaahaaWcbeqaamaabmaabaGaemOqaieacaGLOaGaayzkaaaaaOWaaeWaaeaacqWGPbqAdaWgaaWcbaGaemOAaOgabeaakiabcYcaSiabdIgaOnaaDaaaleaacqWGQbGAcqGHRaWkcqaIXaqmaeaadaqadaqaaiabdkeacbGaayjkaiaawMcaaaaakiabcYcaSiablAciljabcYcaSiabdIgaOnaaDaaaleaacqWGQbGAcqGHRaWkcqWGtbWuaeaadaqadaqaaiabdkeacbGaayjkaiaawMcaaaaaaOGaayjkaiaawMcaaaqaaiabdQgaQjabg2da9iabigdaXiabcYcaSiablAciljabcYcaSiabd6eaobaaaaa@9C9A@
where hj(F)
MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacqWGObaAdaqhaaWcbaGaemOAaOgabaGaeiikaGIaemOrayKaeiykaKcaaaaa@3256@ and hj(B)
MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacqWGObaAdaqhaaWcbaGaemOAaOgabaGaeiikaGIaemOqaiKaeiykaKcaaaaa@324E@ are forward and backward chains of hidden vectors with h0(F)=hN+1(B)=0
MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacqWGObaAdaqhaaWcbaGaeGimaadabaGaeiikaGIaemOrayKaeiykaKcaaOGaeyypa0JaemiAaG2aa0baaSqaaiabd6eaojabgUcaRiabigdaXaqaaiabcIcaOiabdkeacjabcMcaPaaakiabg2da9iabicdaWaaa@3C31@. We parametrise the output update, forward update and backward update functions (respectively N
MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBamrtHrhAL1wy0L2yHvtyaeHbnfgDOvwBHrxAJfwnaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaWaaeGaeaaakeaaimaacqWFneVtaaa@383B@^{(O)}, N
MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBamrtHrhAL1wy0L2yHvtyaeHbnfgDOvwBHrxAJfwnaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaWaaeGaeaaakeaaimaacqWFneVtaaa@383B@^{(F) }and N
MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBamrtHrhAL1wy0L2yHvtyaeHbnfgDOvwBHrxAJfwnaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaWaaeGaeaaakeaaimaacqWFneVtaaa@383B@^{(B) }using three twolayered feedforward neural networks. In our tests the input associated with the jth residue i_{j }contains amino acid information, secondary structure information, and hydrophobicity interaction values described in 20. Amino acid information is obtained from multiple sequence alignments of the protein sequence to its homologues to leverage evolutionary information. Amino acids are coded as letters out of an alphabet of 25. Beside the 20 standard amino acids, B (aspartic acid or asparagine), U (selenocysteine), X (unknown), Z (glutamic acid or glutamine) and (gap) are considered. The input presented to the networks is the frequency of each of the 24 nongap symbols, plus the overall frequency of gaps in each column of the alignment. I.e., if n_{jk }is the total number of occurrences of symbol j in column k, and g_{k }the number of gaps in the same column, the j^{th }input to the networks in position k is:
n
j
k
∑
v
=
1
24
n
v
k
(
4
)
MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaadaWcaaqaaiabd6gaUnaaBaaaleaacqWGQbGAcqWGRbWAaeqaaaGcbaWaaabmaeaacqWGUbGBdaWgaaWcbaGaemODayNaem4AaSgabeaaaeaacqWG2bGDcqGH9aqpcqaIXaqmaeaacqaIYaGmcqaI0aana0GaeyyeIuoaaaGccaWLjaGaaCzcamaabmaabaGaeGinaqdacaGLOaGaayzkaaaaaa@408F@
for j = 1...24, while the 25^{th }input is:
g
k
g
k
+
∑
v
=
1
24
n
v
k
(
5
)
MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaadaWcaaqaaiabdEgaNnaaBaaaleaacqWGRbWAaeqaaaGcbaGaem4zaC2aaSbaaSqaaiabdUgaRbqabaGccqGHRaWkdaaeWaqaaiabd6gaUnaaBaaaleaacqWG2bGDcqWGRbWAaeqaaaqaaiabdAha2jabg2da9iabigdaXaqaaiabikdaYiabisda0aqdcqGHris5aaaakiaaxMaacaWLjaWaaeWaaeaacqaI1aqnaiaawIcacaGLPaaaaaa@42F4@
This input coding scheme is richer than simple 20letter schemes and has proven effective in 21. The secondary structure part of the input is encoded using a threeletter scheme (helix, strand, coil). We adopt both true secondary structures, and secondary structures predicted by Porter 21. When using predicted secondary structure, we carefully design our tests so that no sequence used for testing the PE prediction system is similar to sequences in Porter's training set. A single realvalued input is used to encode hydrophobicity interaction values. In 20 an optimised version of the scale is shown to be highly correlated to the PE. A further unit is used to encode the protein length (normalised by a factor 0.001).
Based on this encoding, a total of 30 units are used to represent each residue.
We adopt a second filtering BRNN, similarly to 21. The network is trained to predict the PE given the firstlayer PE predictions. The ith input to this second network includes the firstlayer predictions in position i augmented by first stage predictions averaged over multiple contiguous windows. I.e., if c_{j1}, ... c_{jm }are the outputs in position j of the first stage network corresponding to estimated probability of eigenvector component j being in class m, the input to the second stage network in position j is the array I_{j}:
I
j
=
(
c
j
1
,
…
,
c
j
m
,
(
6
)
∑
h
=
k
−
p
−
w
k
−
p
+
w
c
h
1
,
…
,
∑
h
=
k
−
p
−
w
k
−
p
+
w
c
h
m
,
…
∑
h
=
k
p
−
w
k
p
+
w
c
h
1
,
…
,
∑
h
=
k
p
−
w
k
p
+
w
c
h
m
)
MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaafaqaceabcaaaaeaacqWGjbqsdaWgaaWcbaGaemOAaOgabeaakiabg2da9iabcIcaOiabdogaJnaaBaaaleaacqWGQbGAcqaIXaqmaeqaaOGaeiilaWIaeSOjGSKaeiilaWIaem4yam2aaSbaaSqaaiabdQgaQjabd2gaTbqabaGccqGGSaalaeaacaWLjaGaaCzcamaabmaabaGaeGOnaydacaGLOaGaayzkaaaabaWaaabCaeaacqWGJbWydaWgaaWcbaGaemiAaGMaeGymaeJaeiilaWIaeSOjGSKaeiilaWcabeaaaeaacqWGObaAcqGH9aqpcqWGRbWAdaWgaaadbaGaeyOeI0IaemiCaahabeaaliabgkHiTiabdEha3bqaaiabdUgaRnaaBaaameaacqGHsislcqWGWbaCaeqaaSGaey4kaSIaem4DaChaniabggHiLdGcdaaeWbqaaiabdogaJnaaBaaaleaacqWGObaAcqWGTbqBcqGGSaalaeqaaaqaaiabdIgaOjabg2da9iabdUgaRnaaBaaameaacqGHsislcqWGWbaCaeqaaSGaeyOeI0Iaem4DaChabaGaem4AaS2aaSbaaWqaaiabgkHiTiabdchaWbqabaWccqGHRaWkcqWG3bWDa0GaeyyeIuoaaOqaaaqaaiablAcilbqaaaqaamaaqahabaGaem4yam2aaSbaaSqaaiabdIgaOjabigdaXiabcYcaSiablAciljabcYcaSaqabaaabaGaemiAaGMaeyypa0Jaem4AaS2aaSbaaWqaaiabdchaWbqabaWccqGHsislcqWG3bWDaeaacqWGRbWAdaWgaaadbaGaemiCaahabeaaliabgUcaRiabdEha3bqdcqGHris5aOWaaabCaeaacqWGJbWydaWgaaWcbaGaemiAaGMaemyBa0gabeaakiabcMcaPaWcbaGaemiAaGMaeyypa0Jaem4AaS2aaSbaaWqaaiabdchaWbqabaWccqGHsislcqWG3bWDaeaacqWGRbWAdaWgaaadbaGaemiCaahabeaaliabgUcaRiabdEha3bqdcqGHris5aaGcbaaaaaaa@9AC2@
where k_{f }= j + f(2w + 1), 2w + 1 is the size of the window over which firststage predictions are averaged and 2p + 1 is the number of windows considered. In the tests we use w = 7 and p = 7. This means that 15 contiguous, nonoverlapping windows of 15 residues each are considered, i.e. firststage outputs between position j112 and j+112, for a total of 225 contiguous residues, are taken into account to generate the input to the filtering network in position j. This input contains a total of 16m real numbers: m representing the mclass output of the first stage in position j; 15m representing the mclass outputs of the firststage averaged over each of the 15 windows.
Five twostage BRNN models are trained independently and ensemble averaged to build the final predictor. Differences among models are introduced by two factors: stochastic elements in the trainingprotocol, such as different initial weights of the networks and different shuffling of the examples; different architecture and number of free parameters of the models. Averaging the 5 models' outputs leads to classification performance improvements between 1% and 1.5% over single models. In 32 a slight improvement in secondary structure prediction accuracy was obtained by "brute ensembling" of several tens of different models trained independently. Here we adopt a less expensive technique: a copy of each of the 5 models is saved at regular intervals (100 epochs) during training. Stochastic elements in the training protocol (similar to that described in 33) guarantee that differences during training are nontrivial. When an ensemble of 9 such copies for all the 5 models is used (45 models in total) we obtain a further slight improvement over the ensemble of 5 models.
Predictive architecture for contact maps
We build a system for the prediction of contact maps based on 2DRNN, described in 4 and 13. This is a family of adaptive models for mapping twodimensional matrices of variable size into matrices of the same size. 2DRNNbased models were among the most successful contact map predictors at the CASP5 competition 27.
As in the PE prediction case, we use 2DRNNs with shortcut connections, i.e. where lateral memory connections span Nresidue intervals, where N > 1. If o_{j,k }is the entry in the jth row and kth column of the output matrix (in our case, it will represent the estimated probability of residues j and k being in contact), and i_{j,k }is the input in the same position, the inputoutput mapping is modelled as:
o
j
,
k
=
N
(
O
)
(
i
j
,
k
,
h
j
,
k
(
1
)
,
h
j
,
k
(
2
)
,
h
j
,
k
(
3
)
,
h
j
,
k
(
4
)
)
h
j
,
k
(
1
)
=
N
(
1
)
(
i
j
,
k
,
h
j
−
1
,
k
(
1
)
,
..
,
h
j
−
S
,
k
(
1
)
,
h
j
,
k
−
1
(
1
)
,
..
,
h
j
,
k
−
S
(
1
)
)
h
j
,
k
(
2
)
=
N
(
2
)
(
i
j
,
k
,
h
j
+
1
,
k
(
2
)
,
..
,
h
j
+
S
,
k
(
2
)
,
h
j
,
k
−
1
(
2
)
,
..
,
h
j
,
k
−
S
(
2
)
)
h
j
,
k
(
3
)
=
N
(
3
)
(
i
j
,
k
,
h
j
+
1
,
k
(
3
)
,
..
,
h
j
+
S
,
k
(
3
)
,
h
j
,
k
+
1
(
3
)
,
..
,
h
j
,
k
+
S
(
3
)
)
h
j
,
k
(
4
)
=
N
(
4
)
(
i
j
,
k
,
h
j
−
1
,
k
(
4
)
,
..
,
h
j
−
S
,
k
(
4
)
,
h
j
,
k
+
1
(
4
)
,
..
,
h
j
,
k
+
S
(
4
)
)
j
,
k
=
1
,
…
,
N
MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBamrtHrhAL1wy0L2yHvtyaeHbnfgDOvwBHrxAJfwnaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaWaaeGaeaaakqGabeqaaG3abaGaem4Ba82aaSbaaSqaaiabdQgaQjabcYcaSiabdUgaRbqabaGccqGH9aqpimaacqWFneVtdaahaaWcbeqaamaabmaabaGaem4ta8eacaGLOaGaayzkaaaaaOWaaeWaaeaacqWGPbqAdaWgaaWcbaGaemOAaOMaeiilaWIaem4AaSgabeaakiabcYcaSiabdIgaOnaaDaaaleaacqWGQbGAcqGGSaalcqWGRbWAaeaadaqadaqaaiabigdaXaGaayjkaiaawMcaaaaakiabcYcaSiabdIgaOnaaDaaaleaacqWGQbGAcqGGSaalcqWGRbWAaeaadaqadaqaaiabikdaYaGaayjkaiaawMcaaaaakiabcYcaSiabdIgaOnaaDaaaleaacqWGQbGAcqGGSaalcqWGRbWAaeaadaqadaqaaiabiodaZaGaayjkaiaawMcaaaaakiabcYcaSiabdIgaOnaaDaaaleaacqWGQbGAcqGGSaalcqWGRbWAaeaadaqadaqaaiabisda0aGaayjkaiaawMcaaaaaaOGaayjkaiaawMcaaaqaaiabdIgaOnaaDaaaleaacqWGQbGAcqGGSaalcqWGRbWAaeaadaqadaqaaiabigdaXaGaayjkaiaawMcaaaaakiabg2da9iab=1q8onaaCaaaleqabaWaaeWaaeaacqaIXaqmaiaawIcacaGLPaaaaaGcdaqadaqaaiabdMgaPnaaBaaaleaacqWGQbGAcqGGSaalcqWGRbWAaeqaaOGaeiilaWIaemiAaG2aa0baaSqaaiabdQgaQjabgkHiTiabigdaXiabcYcaSiabdUgaRbqaamaabmaabaGaeGymaedacaGLOaGaayzkaaaaaOGaeiilaWIaeiOla4IaeiOla4IaeiilaWIaemiAaG2aa0baaSqaaiabdQgaQjabgkHiTiabdofatjabcYcaSiabdUgaRbqaamaabmaabaGaeGymaedacaGLOaGaayzkaaaaaOGaeiilaWIaemiAaG2aa0baaSqaaiabdQgaQjabcYcaSiabdUgaRjabgkHiTiabigdaXaqaamaabmaabaGaeGymaedacaGLOaGaayzkaaaaaOGaeiilaWIaeiOla4IaeiOla4IaeiilaWIaemiAaG2aa0baaSqaaiabdQgaQjabcYcaSiabdUgaRjabgkHiTiabdofatbqaamaabmaabaGaeGymaedacaGLOaGaayzkaaaaaaGccaGLOaGaayzkaaaabaGaemiAaG2aa0baaSqaaiabdQgaQjabcYcaSiabdUgaRbqaamaabmaabaGaeGOmaidacaGLOaGaayzkaaaaaOGaeyypa0Jae8xdX70aaWbaaSqabeaadaqadaqaaiabikdaYaGaayjkaiaawMcaaaaakmaabmaabaGaemyAaK2aaSbaaSqaaiabdQgaQjabcYcaSiabdUgaRbqabaGccqGGSaalcqWGObaAdaqhaaWcbaGaemOAaOMaey4kaSIaeGymaeJaeiilaWIaem4AaSgabaWaaeWaaeaacqaIYaGmaiaawIcacaGLPaaaaaGccqGGSaalcqGGUaGlcqGGUaGlcqGGSaalcqWGObaAdaqhaaWcbaGaemOAaOMaey4kaSIaem4uamLaeiilaWIaem4AaSgabaWaaeWaaeaacqaIYaGmaiaawIcacaGLPaaaaaGccqGGSaalcqWGObaAdaqhaaWcbaGaemOAaOMaeiilaWIaem4AaSMaeyOeI0IaeGymaedabaWaaeWaaeaacqaIYaGmaiaawIcacaGLPaaaaaGccqGGSaalcqGGUaGlcqGGUaGlcqGGSaalcqWGObaAdaqhaaWcbaGaemOAaOMaeiilaWIaem4AaSMaeyOeI0Iaem4uamfabaWaaeWaaeaacqaIYaGmaiaawIcacaGLPaaaaaaakiaawIcacaGLPaaaaeaacqWGObaAdaqhaaWcbaGaemOAaOMaeiilaWIaem4AaSgabaWaaeWaaeaacqaIZaWmaiaawIcacaGLPaaaaaGccqGH9aqpcqWFneVtdaahaaWcbeqaamaabmaabaGaeG4mamdacaGLOaGaayzkaaaaaOWaaeWaaeaacqWGPbqAdaWgaaWcbaGaemOAaOMaeiilaWIaem4AaSgabeaakiabcYcaSiabdIgaOnaaDaaaleaacqWGQbGAcqGHRaWkcqaIXaqmcqGGSaalcqWGRbWAaeaadaqadaqaaiabiodaZaGaayjkaiaawMcaaaaakiabcYcaSiabc6caUiabc6caUiabcYcaSiabdIgaOnaaDaaaleaacqWGQbGAcqGHRaWkcqWGtbWucqGGSaalcqWGRbWAaeaadaqadaqaaiabiodaZaGaayjkaiaawMcaaaaakiabcYcaSiabdIgaOnaaDaaaleaacqWGQbGAcqGGSaalcqWGRbWAcqGHRaWkcqaIXaqmaeaadaqadaqaaiabiodaZaGaayjkaiaawMcaaaaakiabcYcaSiabc6caUiabc6caUiabcYcaSiabdIgaOnaaDaaaleaacqWGQbGAcqGGSaalcqWGRbWAcqGHRaWkcqWGtbWuaeaadaqadaqaaiabiodaZaGaayjkaiaawMcaaaaaaOGaayjkaiaawMcaaaqaaiabdIgaOnaaDaaaleaacqWGQbGAcqGGSaalcqWGRbWAaeaadaqadaqaaiabisda0aGaayjkaiaawMcaaaaakiabg2da9iab=1q8onaaCaaaleqabaWaaeWaaeaacqaI0aanaiaawIcacaGLPaaaaaGcdaqadaqaaiabdMgaPnaaBaaaleaacqWGQbGAcqGGSaalcqWGRbWAaeqaaOGaeiilaWIaemiAaG2aa0baaSqaaiabdQgaQjabgkHiTiabigdaXiabcYcaSiabdUgaRbqaamaabmaabaGaeGinaqdacaGLOaGaayzkaaaaaOGaeiilaWIaeiOla4IaeiOla4IaeiilaWIaemiAaG2aa0baaSqaaiabdQgaQjabgkHiTiabdofatjabcYcaSiabdUgaRbqaamaabmaabaGaeGinaqdacaGLOaGaayzkaaaaaOGaeiilaWIaemiAaG2aa0baaSqaaiabdQgaQjabcYcaSiabdUgaRjabgUcaRiabigdaXaqaamaabmaabaGaeGinaqdacaGLOaGaayzkaaaaaOGaeiilaWIaeiOla4IaeiOla4IaeiilaWIaemiAaG2aa0baaSqaaiabdQgaQjabcYcaSiabdUgaRjabgUcaRiabdofatbqaamaabmaabaGaeGinaqdacaGLOaGaayzkaaaaaaGccaGLOaGaayzkaaaabaGaaCzcaiabdQgaQjabcYcaSiabdUgaRjabg2da9iabigdaXiabcYcaSiablAciljabcYcaSiabd6eaobaaaa@81CB@
where hj,k(n)
MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaqabeGadaaakeaacqWGObaAdaqhaaWcbaGaemOAaOMaeiilaWIaem4AaSgabaWaaeWaaeaacqWGUbGBaiaawIcacaGLPaaaaaaaaa@34BC@ for n = 1, ..., 4 are planes of hidden vectors transmitting contextual information from each corner of the matrix to the opposite corner. We parametrise the output update, and the four lateral update functions (respectively N
MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBamrtHrhAL1wy0L2yHvtyaeHbnfgDOvwBHrxAJfwnaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaWaaeGaeaaakeaaimaacqWFneVtaaa@383B@^{(O) }and N
MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBamrtHrhAL1wy0L2yHvtyaeHbnfgDOvwBHrxAJfwnaebbnrfifHhDYfgasaacH8akY=wiFfYdH8Gipec8Eeeu0xXdbba9frFj0=OqFfea0dXdd9vqai=hGuQ8kuc9pgc9s8qqaq=dirpe0xb9q8qiLsFr0=vr0=vr0dc8meaabaqaciaacaGaaeqabaWaaeGaeaaakeaaimaacqWFneVtaaa@383B@^{(n) }for n = 1, ..., 4) using five twolayered feedforward neural networks, as in 13.
In our tests the input i_{j,k }contains amino acid information, secondary structure and solvent accessibility information, and PE information for the amino acids in positions j and k in the sequence. Amino acid information is again obtained from multiple sequence alignments.