To generate the training dataset for the SVM experiment, we used 10,000 randomly selected abstracts from articles published between 2001 and 2006 in PubMed as a background dataset. The positive dataset consisted of 10,000 randomly selected gene-disease association articles from the HuGE Navigator 5 (formerly known as the HuGE Pub Lit database 6), a continuously updated database of studies relevant to human genome epidemiology sponsored by the National Office of Public Health Genomics. Inclusion and exclusion criteria for positive dataset from the HuGE Pub Lit database has been reported 6.

We developed a PubMed text extraction tool using the NCBI E-utility 20 to retrieve text content based on PubMed identification numbers (PMIDs). The text used for processing consisted of the title and the abstract, or the title alone if the abstract was not available. The text data were stored in a data structure for processing.

The abstract and title of each article were then processed with the text-processing tool we developed. A stemming technique was used to deal with morphologic word changes, for example, polymorph(isms) and polymorph(ic) were considered the same word. A stop word list was generated for some common English words, such as pronouns and articles, to reduce the number of words extracted.

We selected keywords by identifying statistically significant differences between the probability of their occurrence in the text (title and abstract) of human genetic association articles, compared with their frequency in all other articles. The sample sizes of both groups were large enough that the distribution of differences in probabilities was approximated by a normal distribution. Thus words with a z score greater than 1.96 or less than – 1.96 (significance level of α = .05) were chosen as feature keywords.

The statistical formula 21 used for calculating the z score is given by:

Z = p 1 − p 2 pq ( 1 n 1 + 1 n 2 )  if  ( n 1 pq > 5 & ( n 2 pq > 5 ) p = n 1 p 1 + n 2 p 2 n 1 + n 2 q = 1 − p MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacPC6xNi=xI8qiVKYPFjYdHaVhbbf9v8qqaqFr0xc9vqFj0dXdbba91qpepeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaqbaeqabiqaaaqaaiabbQfaAjabg2da9KqbaoaalaaabaGaeeiCaa3aaSbaaeaacqaIXaqmaeqaaiabgkHiTiabbchaWnaaBaaabaGaeGOmaidabeaaaeaadaGcaaqaaiabbchaWjabbghaXjabcIcaOmaalaaabaGaeGymaedabaGaeeOBa42aaSbaaeaacqaIXaqmaeqaaaaacqGHRaWkdaWcaaqaaiabigdaXaqaaiabb6gaUnaaBaaabaGaeGOmaidabeaaaaGaeiykaKcabeaaaaGaeeiiaaIccqqGPbqAcqqGMbGzcqqGGaaicqGGOaakcqqGUbGBdaWgaaWcbaGaeGymaedabeaakiabbchaWjabbghaXjabg6da+iabiwda1iabcAcaMiabcIcaOiabb6gaUnaaBaaaleaacqaIYaGmaeqaaOGaeeiCaaNaeeyCaeNaeyOpa4JaeGynauJaeiykaKcabaqbaeqabeGaaaqaaiabbchaWjabg2da9KqbaoaalaaabaGaeeOBa42aaSbaaeaacqaIXaqmaeqaaiabbchaWnaaBaaabaGaeGymaedabeaacqGHRaWkcqqGUbGBdaWgaaqaaiabikdaYaqabaGaeeiCaa3aaSbaaeaacqaIYaGmaeqaaaqaaiabb6gaUnaaBaaabaGaeGymaedabeaacqGHRaWkcqqGUbGBdaWgaaqaaiabikdaYaqabaaaaaGcbaGaeeyCaeNaeyypa0JaeGymaeJaeyOeI0IaeeiCaahaaaaaaaa@7101@

where:

p₁ = probability of occurrence of word in genetic association abstracts.

p₂ = probability of occurrence of word in non-genetic association abstracts.

n₁ = total occurrences of word in genetic association abstracts.

n₂ = total occurrences of word in non-genetic association abstracts.

The statistically significant keywords are called feature keywords and were used to construct the SVM features. Each feature keyword was weighted according to its z score, normalized to values from -1 to +1. For the training and testing data sets, the script generated the SVM input based on sparse format 22. The presence of each keyword was represented by its position on the feature keyword list, followed by a colon and the normalized z score; the absence of keywords was ignored and each feature was separated by a space, for example, 1:0.003589 30:- 0.81189. In the training data set, the first column of the input data was set to the known outcome, i.e., 1 for positive, -1 for negative. In the test set, the first column of the input dataset was set to 0.

Two sets of significant keywords were generated. One set contained those with positive z scores above the threshold (1.96) (called one-way weighted scheme); the other contained key words with both positive (greater than 1.96) and negative z scores (less than -1.96) (called two-way weighted scheme).

We used LibSVM 22, a freely available SVM software library, to train the SVM model. The accompanying utility, grid.py, was used to find optimum parameters for penalty parameter C and gamma in the radial basis function (RBF) kernel. The RBF kernel was chosen based on its potential in terms of performance 23.

To evaluate the performance of the screening tool, we used a series of new test data (not included in the training set). The first test data set (92253 negatives, 773 positives) consisted of selections from PubMed during five consecutive weeks (February 22, 2007 to March 22, 2007) according to the routine, traditional screening process used to build the HuGE Navigator 5. Positive or negative status assigned by the routine process was considered the gold standard. We used this data set to evaluate two keyword weighting schemes. A second data set (68255 negatives, 597 positives), selected from PubMed during four subsequent weeks (April 5, 2007 to April 26, 2007), was used to evaluate false-positive results generated by the GAPscreener using the selected weighting scheme.

Recall, specificity and precision were calculated from the test data to evaluate the performance of the application. The formulas for calculating these parameters are as follows:

Re ⁡ c a l l = T P T P + F N Pr ⁡ e c i s o n = T P T P + F P S p e c i f i c i t y = T N T N + F P MathType@MTEF@5@5@+=feaafiart1ev1aaatCvAUfKttLearuWrP9MDH5MBPbIqV92AaeXatLxBI9gBaebbnrfifHhDYfgasaacPC6xNi=xI8qiVKYPFjYdHaVhbbf9v8qqaqFr0xc9vqFj0dXdbba91qpepeI8k8fiI+fsY=rqGqVepae9pg0db9vqaiVgFr0xfr=xfr=xc9adbaqaaeGaciGaaiaabeqaaeqabiWaaaGcbaqbaeqabmqaaaqaaiGbckfasjabcwgaLjabdogaJjabdggaHjabdYgaSjabdYgaSjabg2da9KqbaoaalaaabaGaemivaqLaemiuaafabaGaemivaqLaemiuaaLaey4kaSIaemOrayKaemOta4eaaaGcbaGagiiuaaLaeiOCaiNaemyzauMaem4yamMaemyAaKMaem4CamNaem4Ba8MaemOBa4Maeyypa0tcfa4aaSaaaeaacqWGubavcqWGqbauaeaacqWGubavcqWGqbaucqGHRaWkcqWGgbGrcqWGqbauaaaakeaacqWGtbWucqWGWbaCcqWGLbqzcqWGJbWycqWGPbqAcqWGMbGzcqWGPbqAcqWGJbWycqWGPbqAcqWG0baDcqWG5bqEcqGH9aqpjuaGdaWcaaqaaiabdsfaujabd6eaobqaaiabdsfaujabd6eaojabgUcaRiabdAeagjabdcfaqbaaaaaaaa@6A27@

where TP, TN, FP and FN represent the number of true positive, true negative, false positive and false negative results, respectively.

To compare the results of classification by the SVM tool with the gold standard, we used logistic regression (SAS Version 9.13, SAS Institute, Cary, NC). We produced separate logistic regression models for results of the one-way and two-way SVM schemes during the 5-week experiment (February 22, 2007 to March 28, 2007). Results from each model were used to generate receiver-operating characteristics (ROC) and calculate the area under the curve (AUC) with 95% confidence intervals. The AUC of ROC curves for the two models were compared using nonparametric methods 2627.

A list of articles compiled independently by domain experts was used as the gold standard to evaluate the predictive accuracy of the application. A network of eight experts in the analysis of genetic associations with preterm birth performed a comprehensive literature search to build a knowledge base for systematic review and meta-analysis. The search was limited to articles published from January 1, 1990, to April 12, 2007. Complex queries compiled by a librarian were used to query PubMed and EMBASE 28. The complex queries consisted of sophisticated PubMed and EMBASE syntax filling more than four single-spaced pages. The results were manually reviewed by the domain experts.

For comparison, we used the GAPscreener to screen all PubMed abstracts published during the same period of time in a two-step process. First, we compiled a broad PubMed query based on common terms related to preterm birth. The 42,585 PubMed abstracts returned by this query were then classified by the SVM tool.

Query: Prematurity OR infant, premature OR infant, low birth weight OR labor, premature OR preterm labour OR premature birth OR preterm birth OR preterm infant OR preterm premature rupture OR preterm pregnancy outcome OR preterm delivery OR adverse outcomes of pregnancy OR obstetric labor, premature.