Santos et al. 10, Natarajan et al. 11, Fundel et al. 12 and Rinaldi et al. 13 used full parsing to verify matches to predefined rules about descriptions of relations. Miyao et al. 14 used different natural language parsing tools to extract interactions and compared the results. Giles and Wren 15 applied full parsing in conjunction with a support vector machine (SVM) to extract the directions of interactions, since in a pair of interacting entities one tends to be the cause of an effect on the other. However full parsing is computationally expensive and relatively slow, subject to ambiguous parse results, and will only be a partial solution to the natural language processing (NLP) problem which includes semantic and other issues.

Yakushiji et al. 16 built a term recognizer to identify multi-word terms and a shallow parser to reduce lexical ambiguity. Then, they applied full parses over the preprocessed sentences. From the full parses, domain-specific knowledge including a set of target verbs and mapping rules provided by domain specialists was used to construct frame representations of interactions. Another example, GENIES 17, extracted semantic patterns by observing typical semantic and syntactic co-occurrence patterns in a sample corpus using semantic relationship categories and biological objects. It fully parsed sentences and outputted a frame structure when pattern matching was successful. GIS 18 and GIFT 19 also matched sentences to predefined interaction description patterns to identify the interactions.

There are different degrees of NLP, of course, and one way to make NLP more practical with large amounts of text is to use shallower analyses. Chilibot 20 takes this approach, using POS tagging followed by shallow parsing to extract interactions from MEDLINE and support a search engine for interactions in MEDLINE.

Template matching approaches form another and typically computationally more tractable strategy. A sentence, abstract or parsed result is matched against predefined patterns associated with interactions 21. A pattern is a partial specification of words and locations in a passage, such as <biomolecule1 verb "the" verb "of" biomolecule2 "into">. The template term 'biomolecule' in such a pattern might match, for example, any molecule synthesized by living organisms. The matching process can involve a simple match using shallow parsing to identify terms meeting category or other constraints, or a complicated full parsing that analyzes the syntactic structure of the passage before matching against parse result templates.

Although pattern-matching can yield relatively high precision because patterns may be derived from existing sentences describing interactions, recall may be limited because it is not possible to manually describe all possible patterns of biomolecular interaction descriptions 22. Therefore some interaction descriptions will not match the manually derived patterns, so some interactions will not be extracted by the template approach. For example, MedScan 23 obtained a recall of 21% with relatively restrictive templates, while Koike et al. 24 achieved 54% with more unconstraining, inclusive templates that assumed some syntactic analysis.

Term occurrence based approaches can avoid the recall issue just noted. Marcotte et al. 25 identified discriminating words based on a training set of 260 MEDLINE abstracts describing yeast protein interactions, based on differences in frequencies of occurrence of those discriminating words. They used the probabilities of each word's appearance in documents describing interactions to train Naïve Bayesian classifiers to score a document and judge whether the document describes an interaction.

A direct approach to identifying an interaction is to find co-occurrences of two biomolecules in the literature. Dragon Plant Biology Explorer (DPBE) 26 parses documents provided by users using this type of co-occurrence criterion and displays the results in, among other forms, a network of interactions. Albert et al. 27 applied co-occurrence extraction to create a protein interaction database for nuclear receptors, then post-processed this database by manual curation to delete false interactions. PDQ Wizard 28 and Hofmann and Schomburg 29 also used co-occurrences and a subsequent filtering stage to extract interactions between biomolecules.

The iHOP system (e.g. 30) converts MEDLINE into a navigable hyperlinked resource by extracting sentences from it that contain biomolecules and annotating them with hyperlinks from the biomolecular and interaction terms to related sentences. A Web-based interface provides flexible access to this resource. This and similar systems extract sentences that appear to provide evidence for biomolecular interactions from the literature, but do not analyze this evidence further for probabilities of interaction based on empirical investigations of sets of related sentences. This motivates the current work, which fills that gap.

Wren and Garner 31 assigned a weight 1 - rⁿ to the potential relationship between co-occurring terms, where n is the number of times they co-occur and r is one value when the co-occurrence is in a sentence and another value, 0.58, when the co-occurrence is in an abstract but not the same sentence. Ding et al. 8 also reported 0.58 for abstracts, but found a value for sentences different from Wren and Garner's.

Because co-occurrence based methods are relatively simple they cannot, in theory, match the potential performance of methods that incorporate information obtained by additional computation such as sentence parsing. However, they are computationally simpler and faster. NLP can get more out of text than co-occurrence based methods, while empirical facts derived from empirical analyses can provide heuristic guidance to NLP-based methods to enhance computational speed and help resolve ambiguities that arise. Thus, automated text analysis using a hybrid of both empirical facts about texts and deeper NLP-based analyses is expected to do better than either method alone. As an example, Zhou and He 32 used a machine learning method to estimate probabilities that help parse a document.

A sentence can be given a likelihood of describing an interaction based on its containing a co-occurrence, whether in a phrase, or in the sentence but across phrases. Multiple sentences containing the same co-occurrence often exist in MEDLINE, so to extract interactions from MEDLINE we would like to combine the multiple sources of evidence constituted by the multiple sentences. This can be done probabilistically as follows. Let p be the probability that a sentence describes an interaction. Then q = 1-p is the probability that it does not. Given n such independent sentences, and assuming for a moment that probability p is the same for all sentences, then qⁿ would be the probability that none of them describe an interaction, thus 1-qⁿ the probability that at least one does. Since q = 1-p, the formula for the probability of an interaction between a pair of biomolecules being described within n relevant sentences is 1-(1-p)ⁿ.

In the more typical case of n sentences each with its own value p_i, i = 1,..., n for the probability that it describes an interaction, the formula generalizes to:

assuming the sentences provide independent evidence, an assumption commonly made and found to lead to useful results though in general incorrect.

It is reasonable to ask if the value of n should be constrained. Some new interactions may be mentioned only in the most recent publications, limiting the number of publications describing these interactions. Thus, particularly for a recent discovery, the fact that only a few sentences exist containing two given biomolecules might not suggest lack of interaction. Therefore, we also assessed two variant methods for estimating the probability of an interaction between two biomolecules. These are as follows:

• Best 5: use the average of the scores of the top 5 sentences, those having the highest probability of describing an interaction between the two biomolecules: p(interaction) = (p₁+p₂+p₃+p₄+p₅)/5.

• Best 2: use the average of the scores of the top 2 sentences: p(interaction) = (p₁+p₂)/2.

Formula (1) we will call the All method. For the Best 2 and Best 5 methods, if a biomolecule pair co-occurs in fewer than 2 or 5 sentences, 0 was used for the missing probabilities to reach 2 or 5 terms in their formulas.

With these 3 evidence combination methods, given a list of biomolecule pairs we can process MEDLINE to extract biomolecular interactions and construct an interaction network. The biomolecules are the vertices in this network, and if two biomolecules are found to interact, there is an edge between their vertices. We obtained the biomolecule name list from an existing database about genome-wide plant mRNA, protein, and metabolite data, MetNetDB (http://metnet.vrac.iastate.edu/MetNet_db.htm, 3). This database focuses especially on Arabidopsis and soy. We created an interaction network from this database to demonstrate our system.

Any two biomolecules in the database can be checked to see if they interact. For each such pair any sentences where the biomolecule pair co-occurs can be collected and analyzed to estimate the probability that the corpus describes them as interacting. However, checking all pairs is computationally inefficient because there are about 2*10⁶ biomolecule records in the database, hence about 4*10¹² pairs. Instead, we scanned sentences in MEDLINE one by one, identified biomolecule pairs in the sentences, recorded the probability score that each sentence gives to its pairs and finally generate the network using the All, Best 5 and Best 2 combination methods on the sentences for each pair. The overall structure of the system is shown in Figure 1.

There are two main parts.

1. Interaction Extractor.

a. The system examines each sentence in MEDLINE for keywords (biomolecules, IITs, & cellular locations) stored in MetNetDB, tags them and stores the tagged sentences into the PathBinder system database, PathBinderDB.

b. When scanning each sentence, the system determines the interaction likelihood for the biomolecule co-occurrence of interest inside the sentence and combines the scores of multiple sentences containing the pair using All, Best 5, and Best 2. The database has two tables for biomolecules, one for their appearances in MEDLINE records and one for entity names recognized by biologists but which might not appear in MEDLINE records under those names. These tables were imported from the MetNet system (Wurtele et al. 2007 3). When combining the scores, we first calculated the score for the actual co-occurring pair, then found the entity names in the database corresponding to the co-occurring terms appearing in the text, and finally calculated the composite score for the pair of entity names based on the set of sentences containing co-occurrences of other terms associated with those entity names.

2. User Gateway. PathBinder is the user portal to PathBinderDB. PathBinder serves as a query gateway to interaction descriptions stored in PathBinderDB. Users can provide two biomolecules to PathBinder, which will access PathBinderDB and return all sentences in which the two biomolecules appear. It calculates a probability score for each returned sentence, ranks sentences based on their scores, and then shows them to the user. On the other hand, if a user provides just one biomolecule, PathBinder returns a list of other biomolecules potentially interacting with it.

We began with the test corpus of 320 sentences described earlier, for which we computed 320 probability estimates for the likelihood that they described an interaction between a given biomolecule pair. We also manually judged whether each sentence actually does describe an interaction between the queried biomolecule pair, recording 1 if so, or 0 if not, in order to facilitate doing a linear regression to fit the 320 computed likelihoods to the 320 corresponding manual data. If the probability that a sentence describes an interaction is computed accurately, then for a set of sentences with the same computed probability of describing the interaction (e.g., 0.75), that probability is also the expected fraction of those sentences manually found to actually describe the interaction. For example, given a set of sentences each computed to describe an interaction with probability p = 0.75, the statistically expected fraction of them to, in fact, describe an interaction would also be 0.75 (75%), if the computed probability was accurate. Therefore, we can test the accuracy of the computed probabilities by checking how close the linear regression result is to the line y = x (or for axes labeled as in Figure 2, p_manual = p_computed). We consider the actual regression result next.

The regression line shown in Figure 2 is not precisely p_manual = p_computed, but is fairly close:

To make our computed probabilities more accurately reflect manually determined reality (i.e., give a regression line of y = x), we can adjust them by defining a p_adjusted:

It is no accident that Eqs. (2) and (3) are so similar: showing p_adjusted on the x axis will then give a regression line of y = x or, in the present case, p_manual = p_adjusted, as desired.

We applied (3) in PathBinder, so that for each sentence s, a computed probability score p_computed(s), is calculated and then adjusted to give a probability score p_adjusted(s) for the probability that it describes an interaction between two given biomolecules.

The discrepancy between p_computed and p_manual has two possible causes. First, it can simply be a statistical artifact of noisy data. Second, the computational model underlying p_computed might represent reality imperfectly, as models in general often do, and as probabilistic models in particular often do due to implicit independence assumptions that only approximately hold.

To help determine the cause here, and thus test the validity of the p_adjusted calculation, we collected a test set of 600 sentences. Of these, 123 contained the 10 biomolecule pairs from among the 10 we used to create the training corpus, but were not already in the 320 sentence experimental set. To get the remaining 477, we collected sentences with p_adjusted values of 0, 0.1 ± 0.01, 0.2 ± 0.02, 0.3 ± 0.03, 0.4 ± 0.04, 0.5 ± 0.05, 0.6 ± 0.06, 0.7 ± 0.07 and 0.739 ± 0.07 (the p_adjusted computation gives results up to about 0.739). About 50 sentences for each of those values were collected from search results using the new pairs: ethanol & acetaldehyde, acetyl-CoA & NADH, dynamin & GTP, adenylate cyclase & ATP, and ATP & creatine.

For each of the 600 test sentences, whether it really described the interaction was judged manually and recorded as 0 (no) or 1 (yes). Then we did a linear regression on the test set (Figure 3) as was done earlier in Figure 2.

The regression line we get is

which is very close to the ideal of y = x. Thus PathBinder's calculation of p_adjusted is justified by test set B.

Equation (3) is used to evaluate the likelihood that each sentence describes an interaction. As mentioned earlier, we combine evidence from multiple sentences to evaluate the likelihood that a pair of biomolecules interacts using Equation (1) or the All method, and the Best 2 and Best 5 methods. The result is an interaction network of thousands of biomolecules and the interaction relationships among them. The key information retrieval measures of precision and recall were used to compare All, Best 5, and Best 2. Some key results are shown in Figure 4.

To determine the precisions in Figure 4, we randomly sampled a set of 400 pairs of biomolecules co-occurring in MEDLINE from the previously generated interaction network. The sentences for each pair were each evaluated by the three methods (All, Best 5 & Best 2), and the resulting computationally estimated probabilities of interaction were recorded for each pair. The 400 pairs were also manually analyzed to see whether they do in fact interact. One hundred eight of them did interact. The overall precision was thus 108/400 = 0.27 for this random set. More importantly, we calculated the precisions analogously for 7 subsets of the 400 pairs meeting 7 different thresholds for interaction probability. This was done separately for All, Best 5, and Best 2 (making 7*3 = 21 subsets). Thus each subset was associated with a threshold, a calculation method, a precision, and a recall which was the fraction of the 108 interacting pairs meeting the threshold using the calculation method. The overall recall for the whole set is necessarily 1.

Some aspects of Figure 4 are worth considering further. For the All method, the leftmost data point refers to co-occurrences with a calculated interaction probability of 1. Such a high value happens when there are a lot of sentences providing evidence. Combining that evidence using equation (1) leads to score values that are effectively 1 (for example, co-occurrences of "bilirubin" and "cytochrome P450" and their synonyms was computed to have a score of 1–10^-11). We counted any score over 1–10^-6 as 1. This was therefore the most selective threshold for the All method and it occurred for 342,492 biomolecule pairs (for MEDLINE as of October 2008).

Unlike the All method, the Best 5 and Best 2 methods only look at average scores of sentences, so calculated probability scores tend to be lower for these methods than for the All method. Thus Best 5 and Best 2 permit score thresholds met by fewer than 342,492 pairs.

The curves in Fig. 4 are not always monotonic. For example, the first part of the Best 5 curve is not monotonic. The leftmost point on that curve, (0.16, 0.61) is based on the 28 pairs meeting or exceeding a threshold score value of 0.58, computed by the Best 5 method. This was the most selective threshold used to generate the curve. Yet the 62 pairs that met a lower threshold of 0.53 actually had a higher precision, giving point (0.35, 0.63) in Figure 4. One possible reason is noise from the limited data. Another possibility is that Best 5 actually does produce this effect for some reason.

Recall and precision are often combined to get a single, composite measure of information retrieval quality called the effectiveness, or F-measure, of an information retrieval method: F = 2(recall*precision)/(recall+precision). Figure 5 shows the effectiveness for the three methods as a function of the size of the subset meeting a given threshold, with size expressed as a percentage of the full 400-member set.

For the F measure, the Best 2 method gave the highest peak value, for a threshold met by 137 pairs. For the full result interaction network, there are 1,646,337 pairs that meet that threshold.

Our technique has been applied in the PathBinder System, which provides a query gateway to users. If a user provides a biomolecule, PathBinder can find other biomolecules potentially interacting with it. Users can choose a biomolecule pair as a query for sentences describing interactions, as illustrated in Figure 6. Users can also specify more query conditions, like cellular locations (e.g., nucleus, mitochondrion, etc.), categories of IITs appearing with the co-occurring biomolecule names (e.g. association, modification, etc.), specific IITs appearing with a co-occurrence (e.g. bind, increase, etc.) and Linnaean taxonomic categories. All these data are obtained when processing MEDLINE and were pre-recorded in the database. Once Pathbinder gets a query, it will search for all sentences satisfying the query and display them in a new window, as in Figure 7. It can order the result sentences by PMID or (as in Fig. 7) by their estimated probability of describing an interaction between the biomolecules. Users can click the PMID to read the PubMed record containing the sentence directly on the PubMed Web site.