The underlying model for linear regression on a vector X of p independent variables is reflected in Eq. 1, wherein Y is the response variable of interest, μ _Y is the population mean of Y, β is a vector representing the sensitivities of Y to changes in X, and x is a vector of deviations in X from the population centroid μ _X.

As indicated in Eq. 1, the error ε is assumed to be normally distributed with mean 0 and a standard deviation σ _X. Best linear unbiased estimators (BLUEs) for the various parameters in Eq. 1 can be calculated from a sample T ₀ of n observations (in QSAR, compounds) drawn from the full population, provided several preconditions are met 19 :

1. the strict linear dependence of Y on X set out in Eq. 1 applies across the population;

2. the sample is random and unbiased;

3. the descriptors contributing to X are mutually independent in a statistical sense; and

4. the error distribution ε is homoskedastic and independent of X and Y – i.e., its standard deviation is the same everywhere in the descriptor space, so σ _X = σ for all X.

The corresponding regression estimators for each individual observation i and the overall standard error of regression s _FITare then given as shown in Eqs. 2 and 3.

where is the mean value of Y for the sample; x _i = X _i - X ₀, with X ₀ being the sample centroid for X; and is the predicted value of Y at X _i 19 . Note that s _FIT is greater than the root mean square error (RMSE); this is because the means and X ₀ and the calculated coefficient vector b are themselves estimates that are subject to sampling error, with 1 and p degrees of freedom, respectively.

Under these assumptions, the potential error in estimating Y increases as one moves away from the centroid X ₀. As a result, the uncertainty s _u in predicting the value of Y at some new ("unknown") value X _u is generally greater than s _FIT. In fact, under the assumptions given above 19 :

where s _u is the expected standard error of prediction (uncertainty) for the new observation u and n is the number of training set observations t used to build the model. The Mahalanobis distances d _{0, u} and d _{0, t} are measured in the model space defined by b, i.e., they are weighted Euclidean distances between the centroid X ₀ of the descriptor matrix for the training set and the vectors X _u and X _t, respectively.

The rationale behind the "extra" terms in Eq. 4 is straightforward. For any random sample, the error involved in using as an estimate of μ _Y is inversely proportional to n – hence the 1/n term in Eq. 4. In addition, the accuracy with which β is estimated by b is inversely proportional to how thoroughly X is sampled by the training set, but how much difference that makes to the error is directly proportional to the distance d _{0, u} between X _u and X ₀ in the model space. Together these countervailing effects of variation in X account for the second term within the outer brackets.

The value of s _u produced by Eq. 4 is a best linear unbiased estimator of σ _u – provided the assumptions underlying its derivation hold. Unfortunately, one or more of those assumptions are violated in most QSAR applications. In particular:

1. the dependence of Y on X rarely fits the prescribed function perfectly, linear or otherwise;

2. the training set used is usually a non-random sample, its selection biased by matters of historical accident and convenience that reflect the historical trajectory of the synthesis program that motivated the analysis;

3. the descriptors contributing to X are often correlated to a greater or lesser degree and hence are not independent variables in the statistical sense (correlation implies lack of independence, but the inverse is not true: lack of correlation does not imply statistical independence); and

4. ε is usually heteroskedastic – its standard deviation σ _X is often different in different regions of the descriptor space.

Most or all of the assumptions are, in fact, explicitly violated when ANNs, PLS, variable selection, quadratic regression, or bilinear regression techniques are applied, with the result that s _FIT and the estimator given by Eq. 4 underestimate the actual uncertainty of prediction, often drastically.

Several groups have derived theoretical variations of Eq. 4 for use with PLS and principal component analysis (PCA) that seek to address departures from ideality 20 21 22 . Unfortunately, subsequent work has demonstrated that these methods are often not robust when applied in realistic situations 23 .

An alternative, completely empirical approach to assessing aggregate predictive uncertainty is cross-validation, in which each compound in the training set is held back in turn 24 . The value of Y for the held-back compound is then predicted using a model built from the other n - 1 compounds in the training subset T _u = T ₀ - {u}. In parallel to Eq. 3, the standard error of cross-validation s _CV is calculated from the predictive error sum of squares (PRESS) according to equation 5:

where is the value of Y predicted by applying the reduced model built from the n - 1 compounds in training subset T _u to X _u and is the corresponding predictive error. The summation is indexed across u to emphasize that prediction is external to the training subset used in each case. Here p represents the number of PLS components included in the model rather than the number of descriptors.

Cross-validation statistics were originally employed in PLS solely as a way to determine an optimal model complexity, a role for which the classical goodness-of-fit measure r ² used in ordinary least squares is unsuited 24 . It has since come to widely used to assess predictivity, however. This use is unfortunate, in that a poorly predictive model will have a high s _CV and a low q ², but the converse may or may not be true: good cross-validation statistics may be due to redundancies in the training set rather than truly robust predictive performance 25 26 27 28 . Some workers prefer to use "leave-some-out" cross-validation – in which several compounds are held back together – to address this problem. Nonetheless, the LOO standard error is the best estimate of the full model's predictivity for each individual compound in the training set 29 , which makes it is a reasonable starting point for estimating a model's predictive reliability for structures occupying nearby points in the descriptor space.

Violation 4 – that error is not identically and independently distributed across compounds – is especially problematic for QSAR analyses. In one recently described case in point, the variation in predictive error was clearly correlated with one of the two descriptors being used 7 . If that is true when many descriptors are involved (as is the case for PLS), the overall variability in predictive error should be similar across the full range of Y. Such a distribution of error is, in fact, often seen in place of the quadratically increasing spread implied by Eq. 4 30 . This makes it all too easy to make the unjustified leap to the unjustified conclusion that the aggregate predictive uncertainty – typically s _CV or the root mean square error of prediction for an external test set (RMSEP or s _PRED) – is a reliable indicator of the level of uncertainty associated with individual predictions: independence from Y does not imply independence from X.

The increasing reliance of drug developers on tabulations of predicted properties makes getting accurate estimates of the uncertainty σ _u for individual predictions critically important. Unfortunately, it is rarely if ever possible to construct a unified global model for the dependence of σ _u on X. It is neither necessary nor even desirable to do so, however. A better approach is to shift from the classical, descriptor-based view of regression to a sample-based formalism such as that used in the SAMPLS algorithm 31 . This algorithm exploits the fact that Eq. 2 can be recast as Eq. 6 without loss of generality:

where c _{t, i} = [x _t1 x _i1 x _t2 x _i2 ... x _tp x _ip] is the covariance between x _t and x _i and v _t is a weight vector that is specific to compound t. Basically, Eq. 6 says that activity can be expressed as a linear function of the similarities of each compound to each of the other compounds in the training set. This suggests that the observed predictive error e _u can be cast as a sum of contributions from each compound in the training set that increases with similarity to those compounds, which is consistent with the observation that predictive error tends to increase with distance from – i.e., tends to decrease with increasing similarity to – compounds in the training set 11 12 15 . If t* is the closest (i.e., the most similar) such compound, its standard error (s _t*) is a reasonable first approximation to the predictive error s _u for a new compound. In most QSAR applications, a single response value Y _t is assigned to each compound in the training set, so the best estimate of s _t is simply |e _t|, where e _t is the deviation seen for t in the full, non-cross-validated model, i.e., the residual error of fitting.

Though the "true" dependence of predictive uncertainty on the Euclidean distance d _{t*, u} from t* is unknown, its dependence on distance can likely be approximated by a Taylor expansion in which all but the first, linear term in d is dropped. Taken together, these considerations yield the estimator defined by Eq. 7:

where d ₀₀ is the length of the vector x ₀₀ defined by the standard deviations of the descriptors; d ₀₀ = 1 when descriptors have been centered and autoscaled, as was the case here.

The problem then becomes one of estimating the predictive error γ _t associated with each compound t in the training set. PLS tends to overfit, so this term is likely to be greater than s _t*; otherwise Eq. 7 would parallel Eq. 4 exactly, except for the loss of the 1/n aggregation term within the brackets. Instead, one can turn to the squared predictive errors collected during cross-validation. In the calculation of the aggregate predictive uncertainty s _CV (Eq. 5), these are lumped into a single sum – the PRESS. If, however, one assumes that contributions from nearby training set compounds dominate the predictive error and, further, that the value of γ will be comparable for the training subset compounds closest to each individual compound u, the contribution that cross-validation of the i ^th compound makes to the PRESS can more appropriately be distributed across the training subset in inverse proportion to the distances between X _i and the n - 1 compounds used to predict Y _i (Eq. 8 and Fig. 1). A similar approach is taken to distributing response variance across the various sources of deviation from the mean in classical analysis of variance (ANOVA).

The normalization factor α _i in Eq. 9 is necessary to ensure that the distribution is a partition – i.e., that the contributions from the cross-validation step in which compound i was set aside sum to the observed cross-validation error in prediction .

A small constant (1/n) needs to be included to prevent the reciprocal from "exploding" at small distances. Basically, it dictates the distance at which error is expected to distribute evenly. The choice of this particular value is somewhat arbitrary, but 1/n works well and nicely accommodates the tendency of data points to get closer together as the training set gets larger. Taken together, Eqs. 7–9 define the Distributed PRedictive Error Sum of Squares (DPRESS) approach to estimating predictive uncertainty.

The set of structurally diverse cyclooxygenase inhibitors examined here was originally compiled by Chavatte et al. 38 . It includes data on five major and three minor structural classes (Fig. 2) of inhibitors of the inducible form of the enzyme (COX-2). This data set is attractive because the target has been a major focus of research on anti-inflammatory drugs and because it combines substantial structural variation with a few key shared elements such as the distal sulfonyl (SO₂CH₃) or sulfamoyl (SO₂NH₂) group. In addition, regression models based on this data set are well-characterized in terms of predictive robustness 25 28 and with respect to variations in how training subsets are selected 39 . Finally, the uneven representation of the different core structures reflects a sampling bias that is typical of the data sets used to build QSARs.

Fig. 3A shows how activity is distributed across the various structural classes when the compounds in the data set are projected into two dimensions using embedded non-linear mapping 40 41 based on the similarity in their molecular fields: symbols are colored by structural class and sized by activity. Clearly, no one structural class has a monopoly on high activity. Fig. 3B shows the distribution of activity across the descriptor space defined by the compounds' molecular holograms. Molecular fields are 3D descriptors, which are more generalized than holograms – 2D descriptors derived from substructure counts. The more literal character of holograms leads to smaller distances between inhibitors within classes relative to the distances between classes, which accounts for the greater between-class resolution in Fig. 3B. It also accounts for the fact that the sulfonyl and sulfamoyl subclasses are cleanly separated in the hologram space (Fig. 3B) but not in the space defined by the corresponding molecular fields (Fig. 3A).

The main goal of the work reported here was to see how well local estimates of predictive error obtained by DPRESS reflect the actual distribution of predictive error across the descriptor space. Simple random sampling produces a biased training set because, as in most such data sets, the major structural classes are not evenly represented (Fig. 2). Therefore diverse but representative ("boosted" 39 ) training sets were generated by independently drawing five training (sub)sets of 75 compounds from the full set using optimizable k-dissimilarity (OptiSim) selection 39 42 43 . Models based on those training sets were then used to predict the activities of the 229 inhibitors not used to construct them. Three additional training sets were drawn at random, only one of which gave acceptable internal cross-validation statistics. Representation in the full data set is biased, so such simple random subsets are biased as well. The results obtained using that training set (set R) are included here to illustrate the effect of sampling bias due to structural redundancy 39 44 45 .

The optimal number of components p* for the CoMFA models obtained for the boosted training sets ranged from three to seven. It is not appropriate to compare models that differ in complexity directly, however, so a consensus complexity of p = 6 was used in all cases. The corresponding leave-one-out (LOO) cross-validated standard errors (s _CV) ranged from 0.681 to 0.762, corresponding to internal predictivities (q ²) of 0.537 to 0.337. The non-cross-validated models exhibited standard errors of regression (s _FIT) ranging from 0.279 to 0.398, corresponding to r ² values between 0.901 and 0.827. Calculating the root mean square error for external predictions yielded s _PRED = 0.633 to 0.655 – i.e., the internal cross-validated error underestimated the overall accuracy of external prediction somewhat.

In contrast, the biased training set R yielded a cross-validated standard error (s _CV) of 0.489, corresponding to a q ² of 0.696. The overall goodness-of-fit statistics for the non-cross-validated model were s _FIT = 0.279 and r ² = 0.901. As expected, however, the predictive performance on those compounds not in R was substantially worse than that of the boosted training sets, with s _PRED = 0.744.

Fig. 4A shows the same projection as Fig. 3A, but here symbol sizes are based on the error in predicted pIC50 rather than on pIC50 itself. The top panels in Fig. 4(A–C) show the distributions of the individual observed errors in predicted activity (|e|) across the descriptor space, whereas the bottom panels (D-F) show distributions of the corresponding predictive uncertainties ( ) estimated using DPRESS. The leftmost panels (4A and 4D) were obtained for the model based on the boosted training set (set A) that had the lowest aggregate external predictive standard error (s _PRED), whereas the middle panels (4B and 4E) are results for the boosted training set (set B) that had the lowest aggregate internal (cross-validated) predictive standard error (s _CV) overall. The right-most panels (4C and 4F) display the results for the biased training set R.

Several conclusions can be drawn by comparing the distribution of errors to each other and to the distribution of activities. Firstly, though the distributions of observed predictive errors for the three models differ from one another (Fig. 4A vs 4B vs 4C), they resemble each other more than they resemble the distribution of activity itself (Fig. 3A). Secondly, the larger observed errors are not particularly concentrated among the singletons or at the edges of the descriptor space, as would be expected for the ordinary least squares distribution expected based on Eq. 6 and in most published approaches to establishing applicability domains. Thirdly, the distributions of predictive uncertainty seen for the boosted training sets are in good overall agreement with the observed errors with respect to the regions of descriptor space where the observed error is relatively high or low (Fig. 4D vs 4A and 4E vs 4B). Though somewhat less evident, the same is true for the model constructed using the biased training set R (Fig. 4F vs 4C). Finally, the smaller errors predicted by the boosted training with the better internal predictivity (Fig. 4E vs 4D) do seem to be realized in the localized errors actually observed (Fig. 4B vs 4A), even though this was not obvious in the aggregate statistics (s _PRED = 0.637 and s _PRED = 0.633, respectively).

Interpretation of the plots shown in Fig. 4 is complicated because the uncertainty s _u is a measure of the spread in predictive error at X _u; the expected value of the error is still 0. If is an accurate prediction of uncertainty, the magnitude of the observed error (|e _u|) can be expected to be less than about 68% of the time and to almost always (about 95% of the time) be less than 2 . The plots in Fig. 5 – in which the predicted uncertainty (which is always positive) is shown as a function of the observed error (which can be positive or negative) represent a more quantitative way to see how well the predicted uncertainties track the spreads in error actually observed outside the training set.

Eq. 7 implies that = |e _t| for each member t in the training set. The corresponding points are represented by filled stars in each panel in Fig. 5, which therefore define the lines = |e _u|. Unbiased and normally distributed error should only fall outside these lines about 32% of the time and should fall outside the dotted lines corresponding to = 2|e _u| less than 5% of the time. This is clearly not the case when the cross-validated error for the most similar compound t* in the training set is taken as a direct estimate of , i.e., when γ _t is set equal to 0 for all t (Fig. 5A). There are fewer unduly low predicted uncertainties for the biased training set R, but still more than would be expected by chance (Fig. 5B). Note that the bias evident in the model constructed from R comes mostly in the form of negative residuals, i.e., predicted activities that are larger than the observed activities. Such false positives account for most of the "extra" out-of-bounds errors seen in Fig. 5B. The distributions of errors for the boosted training sets are much better behaved; indeed, the predicted uncertainties are slightly more conservative than necessary for large errors in prediction (Fig. 5C and 5D).

HQSAR analyses were carried out as a complement to the results obtained in the CoMFA studies described above. The 2D molecular holograms used were built up from the number of each kind of substructure comprised of between four and seven heavy atoms, the counts being mapped down into count vectors of various lengths by hashing 37 . HQSAR models were then constructed by applying PLS analysis to holograms of length 97, 151, 199, 257, 307 and 353. The optimal complexity for the full model (N = 304) was six components for all hash lengths. The s _CV values obtained ranged from 0.609 to 0.640; the median and average were both 0.622. The value of q ² ranged from 0.547 to 0.582, with a median of 0.564 and an average of 0.563. Based on these results, a hash length of 353 (s _CV = 0.609 and q ² = 0.582) was chosen for evaluating the behavior of the various training sets. The corresponding non-cross-validated analysis gave s _FIT = 0.527 and r ² = 0.687.

The consensus optimal complexity across the boosted training subsets was five components, in keeping with the full data set's having nearly four times as many compounds and, therefore, containing substantially more information. The s _CV values obtained ranged from 0.691 to 0.776 versus a value of 0.540 for the biased training set R; the respective q ² values were 0.386 to 0.514 and 0.623. The s _PRED for the boosted subsets ranged from 0.619 to 0.669 and the corresponding value for the biased subset was 0.735. Hence HQSAR performance followed the trend seen for CoMFA: cross-validation under-estimated the predictive error substantially for the biased subset (i.e., was overly optimistic about the extensibility of the model) and over-estimated the predictive error slightly for the boosted training sets. It differed in that it was the boosted training set B which gave the better external predictive performance.

The distribution of observed predictive errors and predicted uncertainties across the hologram descriptor space are shown in Fig. 6 for the model based on boosted training set B, and the corresponding plots of as a function of e _u are shown in Fig. 7. Note that the predicted uncertainties for the boosted HQSAR models were more conservative than those for the CoMFA models discussed above, with the result that the magnitudes of nearly all errors above 0.75 log units were less than the corresponding . This effect is probably a side-effect of the exaggerated separation between classes seen in the hologram space (Fig. 3B).

Boosted training sets were obtained by applying OptiSim selection to the full data set. OptiSim selection entails drawing a series of random subsamples of size k from the data set of interest. For each subsample in the series, the individual that is most different from those selected from previous subsamples is extracted and added to the selection set S. This procedure results in a representative but diverse selection set that samples the full data set space both efficiently and effectively 42 . Here the structural space was defined in terms of the Tanimoto similarity T(a, b) between the corresponding UNITY substructural fingerprints 48 . The individual a in the i ^th subsample for which max(T(a, b): b ∈ S) is smallest was added to S. Candidates with a Tanimoto similarity greater than 0.8 to any compound already in the selection set were deemed redundant and were excluded from subsamples.

The selection process was repeated five times with n = 75 and k = 4, using a different random number seed each time. Five inhibitors appeared in every boosted training set, including the thiophene, cyclopentadiene and isoxazole analogs that fall outside the five major classes. A total of 113 inhibitors were not selected for any of the boosted training sets, whereas 191 were selected for at least one of them.

CoMFA involves using PLS to identify correlations of biological activity with variations in steric and electrostatic molecular fields, which requires that the molecules under consideration be put into similar conformations and into a common frame of reference as a key part of the process. Here, conformations were set and molecular structures aligned based on the homologous atoms in their central and peripheral rings, as has been described in detail elsewhere 39 . Charges were calculated using the method of Marsili and Gasteiger 49 , as extended in SYBYL 50 to take the distribution of π electrons into account ("Gasteiger-Hückel charges"). Coulombic and Lennard-Jones interaction energies were calculated on a 2 Å rectilinear grid extending 4 Å beyond the edge of any molecule in the full data set. The probe atom used to calculate the fields was an sp ³-hybridized carbon monocation. Interaction energies were truncated at nominal values above 30 kcal/mol, and electrostatics were ignored within the steric envelope of each inhibitor.

The first step in constructing a molecular hologram is to identify all substructures in a molecule that fall within a specified size range – here, all fragments made up of four to seven atoms, with hydrogens ignored and bond types taken into account. Each fragment is then mapped into a compressed count vector of specified length using a hashing function, so that the elements of that count vector can be used as descriptors in subsequent PLS analyses 36 . The hashing means that different fragments may map to the same position in the final count vector. The fragments overlap, however, so each substructure contributes to many fragment counts. The result is that the noise introduced by "collisions" for a few subfragments constitutes a relatively minor perturbation that is, on average, self-limiting. Overfit PLS models are characteristically unstable to such perturbations, however, so surveying a range of hash lengths and picking one with good but representative statistical properties is a good way to avoid picking a length whose cross-validation statistics are overly optimistic. This is a non-parametric perturbation analysis analogous to looking at the effect of small perturbations in response to assess model stability 28 .

2D depictions of the relationship between different compounds were obtained using the embedded non-linear mapping (eNLM) facility 40 in Benchware DataMiner 51 . "Ordinary" NLM can be thought of as placing springs between all pairs of points in the original descriptor space, then compressing the ensemble into two dimensions in such a way that the residual tension in those springs is minimized. Embedded NLM differs in that parts of springs longer than some specified threshold length (horizon) are treated as elastic to extension, i.e., they do not contribute to the overall stress in the system. Here, spring "tensions" were based on the block-wise autoscaled Euclidean distances ("CoMFA Standard scaling" 32 ) between the molecular fields or between the molecular holograms of different compounds.

CoMFA and HQSAR analyses were carried out in SYBYL. The distances d _{t, i} used to partition the PRESS were taken from the SAMPLS.dist file generated by the SYBYL interface as input to the SAMPLS program 52 and represent inter-observation distances in the descriptor space after autoscaling has been applied. The descriptors used here are already either fully commensurate (HQSAR) or are piecewise commensurate (within steric and electrostatic fields but not between them, for CoMFA), so "CoMFA standard" (block) autoscaling was used 33 . Observed and predicted responses were taken from the SAMPLS.out file generated by the SAMPLS program.

Localized predictivity estimates were calculated by combining scripts written in SYBYL programming language (SPL) with spreadsheet manipulations carried out in Excel. For each compound t in the training set, the scaling factor γ _t was calculated based on the observed predictive variance (squared cross-validation error of prediction, δ _i ²) for every other compound in the training set (i ≠ t) weighted inversely by the square of the Euclidean distance between the two (d _{t, i}) in the descriptor space (Eq. 8). A normalization factor α _i for each compound i was calculated as the sum of squared distances to all other compounds in the training set (Eq. 9). A limiting proximity term of 1/n was included to ensure reasonable behavior for closely-spaced compounds where d _{t, i} approaches 0; this works well when the descriptors have been autoscaled in some way before use.

The individual scaling factors γ _t obtained from the n LOO cross-validation errors for the training set were used to calculate an estimate for the predictive uncertainty associated with each new structure u based on the observed cross-validation error of the training set compound (t*) lying closest to it in the descriptor space, its distance from t*, and the scaling factor γ _t* derived from the model cross-validation analyses (Eqs. 7–9).