The traditional TF*IDF schema 11 just takes into consideration the occurrences of words in the whole corpus, while discarding the distribution of words in different categories. Differently, we propose a novel probabilistic feature value schema TF*ML (production of Term Frequency by Maximum Likelihood) to substitute the traditional TF*IDF as follows:

T F   ( t )   ×   M L ( t )   =   T F   ( t )   ×   ( log ⁡   P ( t | c + )   −   log ⁡   P ( t | c − ) ) MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXafv3ySLgzGmvETj2BSbqeeuuDJXwAKbsr4rNCHbGeaGqipu0Je9sqqrpepC0xbbL8F4rqqrFfpeea0xe9Lq=Jc9vqaqpepm0xbba9pwe9Q8fs0=yqaqpepae9pg0FirpepeKkFr0xfr=xfr=xb9adbaqaaeGaciGaaiaabeqaaeaabaWaaaGcbaqcLbxacqWGubavcqWGgbGrcaaMe8UcdaqadaqaaKqzWfGaemiDaqhakiaawIcacaGLPaaajugCbiaaysW7cqGHxdaTcaaMe8Uaemyta0KaemitaWKcdaqadaqaaKqzWfGaemiDaqhakiaawIcacaGLPaaajugCbiaaysW7cqGH9aqpcaaMe8UaemivaqLaemOrayKaaGjbVRWaaeWaaeaajugCbiabdsha0bGccaGLOaGaayzkaaqcLbxacaaMe8Uaey41aqRaaGjbVRWaaeWaaeaajugCbiGbcYgaSjabc+gaVjabcEgaNjaaysW7cqWGqbaukmaabmaabaqcLbxacqWG0baDcqGG8baFcqWGJbWykmaaCaaaleqabaGaey4kaScaaaGccaGLOaGaayzkaaqcLbxacaaMe8UaeyOeI0IaaGjbVlGbcYgaSjabc+gaVjabcEgaNjaaysW7cqWGqbaukmaabmaabaqcLbxacqWG0baDcqGG8baFcqWGJbWykmaaCaaaleqabaGaeyOeI0caaaGccaGLOaGaayzkaaaacaGLOaGaayzkaaaaaa@7C62@

where t means the selected feature word, c⁺ and c⁻ mean the relevant and irrelevant category, P(t|c⁺) and P(t|c⁻) mean the probability that t occurs in category c⁺andc⁻ respectively.

The sign of ML indicates the category relevance of the feature and the magnitude reflects the classification confidence. Following the same idea as TF*IDF to express the specificities of features in different documents, we also multiply TF by ML.

Here we do not depend on the posterior distribution of features to implement the prediction. To explain the reason we could rewrite the formula (2) as follows:

M L ( t )   =   log ⁡   P ( t | c + )   −   log ⁡   P ( t | c − )   =   log ⁡   P ( t , c + )   −   log ⁡   P ( t , c − )   +   log ⁡   P ( c + ) P ( c − ) MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXafv3ySLgzGmvETj2BSbqeeuuDJXwAKbsr4rNCHbGeaGqipu0Je9sqqrpepC0xbbL8F4rqqrFfpeea0xe9Lq=Jc9vqaqpepm0xbba9pwe9Q8fs0=yqaqpepae9pg0FirpepeKkFr0xfr=xfr=xb9adbaqaaeGaciGaaiaabeqaaeaabaWaaaGcbaqcLbxacqWGnbqtcqWGmbatkmaabmaabaqcLbxacqWG0baDaOGaayjkaiaawMcaaKqzWfGaaGjbVlabg2da9iaaysW7cyGGSbaBcqGGVbWBcqGGNbWzcaaMe8UaemiuaaLcdaqadaqaaKqzWfGaemiDaqNaeiiFaWNaem4yamMcdaahaaWcbeqaaiabgUcaRaaaaOGaayjkaiaawMcaaKqzWfGaaGjbVlabgkHiTiaaysW7cyGGSbaBcqGGVbWBcqGGNbWzcaaMe8UaemiuaaLcdaqadaqaaKqzWfGaemiDaqNaeiiFaWNaem4yamMcdaahaaWcbeqaaiabgkHiTaaaaOGaayjkaiaawMcaaKqzWfGaaGjbVlabg2da9iaaysW7cyGGSbaBcqGGVbWBcqGGNbWzcaaMe8UaemiuaaLcdaqadaqaaKqzWfGaemiDaqNaeiilaWIaem4yamMcdaahaaWcbeqaaiabgUcaRaaaaOGaayjkaiaawMcaaKqzWfGaaGjbVlabgkHiTiaaysW7cyGGSbaBcqGGVbWBcqGGNbWzcaaMe8UaemiuaaLcdaqadaqaaKqzWfGaemiDaqNaeiilaWIaem4yamMcdaahaaWcbeqaaiabgkHiTaaaaOGaayjkaiaawMcaaKqzWfGaaGjbVlabgUcaRiaaysW7cyGGSbaBcqGGVbWBcqGGNbWzcaaMe8UcdaWcaaqaaKqzWfGaemiuaaLcdaqadaqaaKqzWfGaem4yamMcdaahaaWcbeqaaiabgUcaRaaaaOGaayjkaiaawMcaaaqaaKqzWfGaemiuaaLcdaqadaqaaKqzWfGaem4yamMcdaahaaWcbeqaaiabgkHiTaaaaOGaayjkaiaawMcaaaaaaaa@9CD5@

And the posterior distribution is:

Posterior ( t )   =   log ⁡   P ( c + | t )   −   log ⁡   P ( c − | t )   =   log ⁡   P ( t , c + )   −   log ⁡   P ( t , c − ) MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXafv3ySLgzGmvETj2BSbqeeuuDJXwAKbsr4rNCHbGeaGqipu0Je9sqqrpepC0xbbL8F4rqqrFfpeea0xe9Lq=Jc9vqaqpepm0xbba9pwe9Q8fs0=yqaqpepae9pg0FirpepeKkFr0xfr=xfr=xb9adbaqaaeGaciGaaiaabeqaaeaabaWaaaGcbaqcLbxacqWGqbaucqWGVbWBcqWGZbWCcqWG0baDcqWGLbqzcqWGYbGCcqWGPbqAcqWGVbWBcqWGYbGCkmaabmaabaqcLbxacqWG0baDaOGaayjkaiaawMcaaKqzWfGaaGjbVlabg2da9iaaysW7cyGGSbaBcqGGVbWBcqGGNbWzcaaMe8UaemiuaaLcdaqadaqaaKqzWfGaem4yamMcdaahaaWcbeqaaiabgUcaRaaajugCbiabcYha8jabdsha0bGccaGLOaGaayzkaaqcLbxacaaMe8UaeyOeI0IaaGjbVlGbcYgaSjabc+gaVjabcEgaNjaaysW7cqWGqbaukmaabmaabaqcLbxacqWGJbWykmaaCaaaleqabaGaeyOeI0caaKqzWfGaeiiFaWNaemiDaqhakiaawIcacaGLPaaajugCbiaaysW7cqGH9aqpcaaMe8UagiiBaWMaei4Ba8Maei4zaCMaaGjbVlabdcfaqPWaaeWaaeaajugCbiabdsha0jabcYcaSiabdogaJPWaaWbaaSqabeaacqGHRaWkaaaakiaawIcacaGLPaaajugCbiaaysW7cqGHsislcaaMe8UagiiBaWMaei4Ba8Maei4zaCMaaGjbVlabdcfaqPWaaeWaaeaajugCbiabdsha0jabcYcaSiabdogaJPWaaWbaaSqabeaacqGHsislaaaakiaawIcacaGLPaaaaaa@902B@

In the ML schema, the relevant/irrelevant document rate P(c+)P(c−) MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXafv3ySLgzGmvETj2BSbqeeuuDJXwAKbsr4rNCHbGeaGqipu0Je9sqqrpepC0xbbL8F4rqqrFfpeea0xe9Lq=Jc9vqaqpepm0xbba9pwe9Q8fs0=yqaqpepae9pg0FirpepeKkFr0xfr=xfr=xb9adbaqaaeGaciGaaiaabeqaaeaabaWaaaGcbaWaaSaaaeaajugCbiabdcfaqPWaaeWaaeaajugCbiabdogaJPWaaWbaaSqabeaacqGHRaWkaaaakiaawIcacaGLPaaaaeaajugCbiabdcfaqPWaaeWaaeaajugCbiabdogaJPWaaWbaaSqabeaacqGHsislaaaakiaawIcacaGLPaaaaaaaaa@3DF1@ has been taken into consideration as a compensate factor. But in the posterior probability schema, the impact of relevant/irrelevant document rate is eliminated, according to the independent and identical distribution hypothesis, while it is not tenable in our situation (since the different relevant/irrelevant rate between training and testing set).

The essence of TF*ML schema is to fully utilize the category relevant information from the annotated samples, which cannot be inferred from the TF*IDF schema. Experiment results (see Table 2) demonstrate that ML steadily improves the discriminative capability of features.

In many text classification applications, it is appealing to take every document as a string of characters rather than a bag of words 12. Especially in bioscience, the tokenizing and stemming procedure would incur undesired loss of the informative attributions, since many of the semantically related biomedical terms that share the same stem or morpheme are often not reducible to the same stems 6. Therefore, we propose to directly utilize the length-fixed strings as feature to exploit most of the informative segments.

To the best of our knowledge, no one explicitly takes length-fixed strings as feature because of the explosion and sparse of feature space. However, the statistical analysis based on formula (1) demonstrates that distributional divergence between the training and testing set becomes much smaller under the length-fixed string feature (see Table 3). So we turn to take the fixed-length strings as feature: the length-fixed strings are extracted from the whole sequential text without considering the sentence boundaries and strictly consist of 26 lowercase English letters (all the letters are converted to the lowercase first), 10 numbers (0-9) and a white space. Chi-Square statistics 13 is employed to select out the significant features and TF*IDF is computed to build the feature vector (we substitute TF*ML for TF*IDF for further improvement).

Table 4 lists the top 10 distinct features from the selected unigram features and string features respectively. It is apparent that the length-fixed string feature has at least the following potential advantages. First, inter-word features (e.g. phrasal effects) can be exploited automatically. The segmentation process spans the boundary of adjacent words, which could exploit information from the adjacent words. Second, intra-word features (e.g. morphological variants) could be captured. For example, string “interac” would occur in the word “interact” and “interaction”, both of which are important indicator of PPI relations. Third, the special meaning of length-fixed string feature in bio-literatures is that it exploits the slight but informative commonality from the structure of the words. For example, different terminologies in bio-literatures often share the same conflation (e.g. ‘phosph’ indicates the protein phosphorylation) and most of the suffix is informative (e.g. ‘ase’ is a common suffix to proteins that function as enzyme). The specific information is not recoverable when the general tokenizing and stemming procedure is applied.

Both of the above proposed methods are domain independent, which are endowed with well generalization capacity and are not necessarily limited to the bioscience domain. But introducing domain dependent features could greatly filter out the false positive samples and further improve the performance 14. In biological literatures, named entities (words and phrases belonging to certain predefined classes, e.g. protein and gene), such as CDC42 (protein), and semantic templates (co-occurrences of a pre-specified type of relationship between entities of given types), such as “ProteinA interact with ProteinB”, are the most meaningful concepts in PPI documents and well conserve the syntactic and semantic structures in describing the protein interactions. So we introduce the named entities and semantic templates as feature to exploit the domain dependent information.

With the help of ABNER 15, a named entity recognition tool, 5 types of named entities in a given document could be identified: protein, DNA, RNA, cell types and cell line. Since the recognized entity space is large and sparse, we only utilize their types as feature to decrease the dimension of feature space without losing the universality.

After recognizing the named entities, semantic templates are ready to be extracted from the documents. We propose a novel template extraction algorithm named KeyBT, i.e. Keyword Based Template extraction algorithm, to extract the semantic templates describing the interaction patterns among all of the recognized entities.

Compared to the traditional local alignment algorithm, KeyBT operates differently: first locate statistical significant words as seeds, and expand the seeds in the contextual environment iteratively, finally preserve the most “powerful” templates as the result.

The flow chart of the KeyBT algorithm is as follows:

1) Locate the occurrences of predefined candidate keywords in each sentence; discard the sentences without any keywords; get the initial candidate sentence set S₀;

2) Locate each entity type in S₀; discard the sentences without any entities; get the initial candidate template set T₀;

3) Iteratively normalize each template in T₀: removing the redundant templates by syntax parsing; get the raw templates set T₁;

4) Evaluating the templates in T₁, filter out the templates of low quality, get the final template set T_f.

KeyBT not only depends on Chi-Square statistics to select the most distinct keywords but also utilizes ML to determine the category relevance of the keywords, because Chi-Square does not distinguish the association between features and different categories: a few high quality features of irrelevant category might be overwhelmed in the large amount of features of relevant category. Chi-Square is employed to select a raw candidate keyword list (with low threshold), and then top 50 features from both categories are preserved according to ML respectively.

We use the following formula 16 to evaluate the relevance of templates based on the balance between generality and specificity of the templates.

S ( t )   =   ( β   +   log ⁡ 2   t . pos   +   0.5 t . neg   +   0.5 )   ×   ln ⁡ ( t . pos   +   t . neg   +   1 ) MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXafv3ySLgzGmvETj2BSbqeeuuDJXwAKbsr4rNCHbGeaGqipu0Je9sqqrpepC0xbbL8F4rqqrFfpeea0xe9Lq=Jc9vqaqpepm0xbba9pwe9Q8fs0=yqaqpepae9pg0FirpepeKkFr0xfr=xfr=xb9adbaqaaeGaciGaaiaabeqaaeaabaWaaaGcbaqcLbxacqWGtbWukmaabmaabaqcLbxacqWG0baDaOGaayjkaiaawMcaaKqzWfGaaGjbVlabg2da9iaaysW7kmaabmaabaqcLbxacqaHYoGycaaMe8Uaey4kaSIaaGjbVlGbcYgaSjabc+gaVjabcEgaNPWaaSbaaSqaaiabikdaYaqabaqcLbxacaaMe8UcdaWcaaqaaKqzWfGaemiDaqNaeiOla4IaemiCaaNaem4Ba8Maem4CamNaaGjbVlabgUcaRiaaysW7cqaIWaamcqGGUaGlcqaI1aqnaOqaaKqzWfGaemiDaqNaeiOla4IaemOBa4MaemyzauMaem4zaCMaaGjbVlabgUcaRiaaysW7cqaIWaamcqGGUaGlcqaI1aqnaaaakiaawIcacaGLPaaajugCbiaaysW7cqGHxdaTcaaMe8UagiiBaWMaeiOBa4McdaqadaqaaKqzWfGaemiDaqNaeiOla4IaemiCaaNaem4Ba8Maem4CamNaaGjbVlabgUcaRiabbccaGiabdsha0jabc6caUiabd6gaUjabdwgaLjabdEgaNjaaysW7cqGHRaWkcaaMe8UaeGymaedakiaawIcacaGLPaaaaaa@8795@

where t.pos and t.neg are the positive/negative matching count of template t in the training set, and β is the parameter tuning the positive/negative matching rate.

When we get the final templates set T_f, we do not simply depend on the positive/negative matching rate of each template to make the prediction. Instead, we use them to build feature vectors and train a classifier.

Top 5 KeyBT-extracted templates are illustrated in Table 5.

Compared with the local alignment algorithm that depends on the post evaluation to remove meaningless and noisy templates, the potential advantages of KeyBT algorithm are as follows: 1) KeyBT utilizes the statistical characteristic of the candidate keywords to largely remove noise before extraction; 2) KeyBT templates need not to fix the entities' type beforehand, so that it could catch the distribution of templates in both categories to discriminate both of the relevant and irrelevant categories; 3) the heuristic rules applied on the relation of named entities and candidate words (such as their sequence, the average template length and type of distinct entities) would guarantee the biological meaning of the extracted templates.

Experiment results of the overlap among the misclassified samples by different features show that there is great complement among different features: in many cases, the false prediction caused by one feature would be treated correctly by another one. And a single type of feature is easy to lead the classifier over-fitting on the data distribution (see Table 1 and Figure 1). Thus, the integration among different features would be beneficial. In this sense, we propose two kinds of integration from different levels: feature-level and classifier-level to integrate all of above proposed features.

We perform the feature-level integration in a typical way: normalizing each part of features and unifying them into a new feature vector. We do the normalization as follows 17:

norm _ value   =   unnorm _ value − min _ value  max _ value − min _ value MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXafv3ySLgzGmvETj2BSbqeeuuDJXwAKbsr4rNCHbGeaGqipu0Je9sqqrpepC0xbbL8F4rqqrFfpeea0xe9Lq=Jc9vqaqpepm0xbba9pwe9Q8fs0=yqaqpepae9pg0FirpepeKkFr0xfr=xfr=xb9adbaqaaeGaciGaaiaabeqaaeaabaWaaaGcbaqcLbxacqWGUbGBcqWGVbWBcqWGYbGCcqWGTbqBcqGGFbWxcqWG2bGDcqWGHbqycqWGSbaBcqWG1bqDcqWGLbqzcaaMe8Uaeyypa0JaaGjbVRWaaSaaaeaajugCbiabdwha1jabd6gaUjabd6gaUjabd+gaVjabdkhaYjabd2gaTjabc+faFjabdAha2jabdggaHjabdYgaSjabdwha1jabdwgaLjaaysW7cqGHsislieGacqWFTbqBcqWFPbqAcqWFUbGBcqGGFbWxcqWG2bGDcqWGHbqycqWGSbaBcqWG1bqDcqWGLbqzcqqGGaaiaOqaaKqzWfGae8xBa0Mae8xyaeMae8hEaGNaei4xa8LaemODayNaemyyaeMaemiBaWMaemyDauNaemyzauMaeyOeI0Iae8xBa0Mae8xAaKMae8NBa4Maei4xa8LaemODayNaemyyaeMaemiBaWMaemyDauNaemyzaugaaaaa@7E52@

where max_value and min_value are the maximum and minimum values that are actually seen in the input feature set.

But there is an obvious defection in the above method: some lower dimensional features might be overwhelmed by the higher dimensional features (e.g. named entity feature has only 5 dimensions while length-fixed string feature has more than 10 thousand dimensions). Based on this consideration, we turn to perform the integration on the classifier level and propose two different ways to implement the integration. The first one is to integrate the output of each classifier: after training classifiers on different types of features respectively, we normalize and unify the output of each classifier into feature vectors and train a classifier. The other one is Adaboost18, a general classifier integration method, which has two major advantages: firstly, Adaboost tunes the weight of each classifier according to its performance in each kind of training samples, which could fully utilize the discriminative capability of features; secondly, soft margin of Adaboost avoids the risk of over-fitting in the training process. These approaches well overcome the defection mentioned above.

In Table 2, TF*ML schema improves recall performance by 6.9% without losing precision compared to the traditional TF*IDF schema. The improvement validates the effectivity of exploiting the category relevance information of features and testifies ML to be a more effective and general feature value schema in general text classification applications.

In our experiment, the best performance is achieved when the string length p is set to 7. In Table 6, the length-fixed string feature (p=7) gains encouraging recall improvement by more than 12.0% compared to unigram and bigram feature. But the precision has dropped about 7.2% as the expense, which can be further compensated by employing TF*ML as feature value. The practical efficiency confirms our statistical analysis of the distribution of features and gives us insight in the selection of lower level features.

In Table 7, only depending on a simple criterion that if a document contains at least one protein entity, the document should be judged relevant otherwise irrelevant, we could achieve a very high recall (0.96) with an acceptable precision (0.58). Our proposed template extraction algorithm KeyBT well captures the complex association between the keywords and named entities and achieves promising performance in term of precision (by 11.8%) and a better improvement comparing to our former approach ONBIRES templates 19, which is based on local alignment algorithm.

In Table 8, feature-level integration contributes the improvement in terms of F-score by 5.2% and AUC by 5.9%; in table 9, integration based on the output of classifier achieves better improvement in terms of F-score by 5.3% and AUC by 6.3%. The best performance is reached by AdaBoost: in F-score by 11.5% and in AUC by 8.8%. Advantage of the feature integration is obvious: different types of features are independently selected from the corpus, which focus on the different aspects of feature space and reinforce each other. From the result, it is apparent that different integration methods well leverage the capability of different types of features and achieve promising improvement.

Since the size of the evaluation corpus is not large enough, it is necessary to perform the statistical significance test to validate the reliability of our proposed features and integration methods. Here we employ s-test to evaluate the performance of systems on the pooled decisions on the individual documents/category pairs 20.

In Table 10, we can find that the proposed feature value schema TF*ML, length-fixed string feature and semantic template feature are much better than their counterpart (p value lower then 0.05), and two different level of feature integrations significantly improve the classification performance (p value lower then 0.005).

In Table 11, the mean, standard deviation and best performance from BioCreAtIvE 2006 are selected from 51 runs of 19 teams. Under our feature selection and integration procedure, the performance outperforms the previous best results (F-score improved by 8.2% and AUC improved by 2.2%).