To illustrate, consider a network with three layers, as shown in Figure 1 for the Jersey cow data. In the left-most layer, there are input variables, 297 in Figure 1, or transformations thereof (called features) that enter into the network as predictors. In the middle ("hidden") layer there is a varying number of neurons; 5 are shown in Figure 1, but the number used is a model selection issue, with this addressed via an evaluation of predictive performance. In the right-most layer, there is a single ("output") node, at least for quantitative response variables. Each input (or feature) connects to each neuron with a strength represented by an unknown coefficient w. The collected input into a given neuron can be transformed (or not, in which case one speaks of an identity or linear activation function), and this activated net input is emitted to the output layer with a strength represented by another unknown coefficient. A similar process takes place for every neuron.

Algebraically, the process can be represented as follows. Let t_i (the target phenotype) be a quantitative trait measured in individual i (i = 1,2,...,n) and let p_i = {p_ij} be a vector of inputs or explanatory variables, e.g., marker genotypes or any other covariate measured in each of such individuals, with allowance made for inclusion of a 1, corresponding to the indicator variable for an intercept in a regression model. Suppose there are S neurons in the hidden layer of the architecture. The input into neuron k (k = 1,2,...,S) prior to activation, as described subsequently, is the linear function w'_k p_i, where w'_k ={w_kj } is a vector of unknown connection strengths ("regressions") peculiar to neuron k, including an intercept (called "bias" in the machine learning literature) in w'_k. This input is transformed ("activated") using some linear or non-linear function f(.), which can be neuron-specific or common to all neurons; this yields f_k(w'_k p_i) (k = 1,2,...,S). Subsequently, the so activated emission from neuron k is sent to the output layer, with the collection of emissions over all neurons being b + c g ∑ k = 1 s w k f k ( w ′ k p i ) , where b is an overall bias parameter, c is a regression on an activated emission, g(.) is another activation function, possibly non-linear, and w₁,w₂,...,w_S are regressions on each of the activated emissions f_k(w'_k p_i). The link between the response variable (phenotype) and the inputs is provided by the model

t i = b + c g ∑ k = 1 s w k f k ( w k ′ p i ) + e i ; i = 1 , 2 , … , n

where e_i ~ (0, σ²) and σ² is a variance parameter. If g(.) is a linear or identity activation function, the model is a linear regression on the adaptive covariates f_k(w'_k p_i); if, further, f_k(.), is also linear, the regression model is entirely linear. The term "adaptive" means that the covariates are functions of unknown parameters, the {w_kj} connection strengths, so the networks can "learn" the relationship between explanatory variables and phenotypes, as opposed to posing it arbitrarily, as it is the case in standard regression models. In this manner, this type of neural network can also be viewed as a regression model, but with the extent of non-linearity dictated by the type of activation functions used. Since the number of parameters increases linearly with the number of neurons, and the number of predictors given by the length of p (e.g., the number of markers) can amply exceed sample size, it is necessary to treat the connection strengths as random effects in which case the Bayesian connection is immediate 1516. This approach is called "Bayesian regularization".

Let t represent an n × 1 vector of phenotypic values and u ~ (0, Aσ²u ) be a vector of infinitesimal additive genetic effects, where σ²u is the additive genetic variance, A = CC' = {a_ij} is the numerator relationship matrix and C is its lower triangular Cholesky factor decomposition. Fisher's linear model on additive genetic effects (ignoring an overall mean and nuisance fixed effects, for simplicity) admits at least three representations:

I) t = u + e = Czσ_u + e = Cu* + e,

where z is a vector of independent standard normal deviates, u* = zσ_u ~ (0, Iσ²_u) and e ~ (0, Iσ²) is a residual vector with σ² interpretable as environmental variance.

II) t = AA^-1u + e = Au** + e,

where u** = A^-1u ~ (0, A^-1 σ²_u), and

III) t = A^-1 Au + e = A^-1u*** + e,

where u*** = Au ~ (0, A³σ²u).

In each of these formulations Fisher's model can be viewed as a neural network with a single neuron in the middle layer, where g(.) is an identity or linear activation function. The respective representations for the three models given above are

t i = b + g ( ∑ j = 1 n c i j u j * ) + e i , t i = b + g ( ∑ j = 1 n a i j u j * * ) + e i ,

and

t i = b + g ( ∑ j = 1 n a i j u j * * * ) + e i .

Here, a bias parameter b is included for the sake of generality. Hence, the additive model can be viewed as a single-neuron network regression on either elements of the Cholesky decomposition of the numerator relationship matrix, on the relationships themselves or on the elements of the inverse of A, with the strengths of the connections represented by the corresponding entries of u*, u** and u***, respectively.

Is it possible to exploit knowledge of relationships in a fuller manner? Since a neural network is a universal approximator, the predictive performance of the classical infinitesimal linear model can be enhanced, at least potentially, by taking a model on, say, S neurons, while effecting non-linear transformations simultaneously. The rationale is that Fisher's model holds under some assumptions which may be violated, such as linkage equilibrium, e.g., entries of the numerator relationship matrix are expected values in the absence of selection and under linkage equilibrium. For instance, using the second representation above one could write

t i = b + c g ∑ k = 1 s w k g k ( b k + ∑ j = 1 n a i j u j * * [ k ] ) + e i ; i = 1 , 2 , … , n .

Here, the inputs are entries a_ij of the relationship matrix, connecting individual i to all other individuals in the genealogy; the u j * * [ k ] coefficient is the connection strength for input j in neuron k; b_k is the bias parameter associated with neuron k; g_k is an activation function peculiar to neuron k; w_k is the connection strength between the activated emission from neuron k and the output layer, b is the outer layer bias parameter and g(.) is the outer activation function, which may be linear or non-linear, although it is typically taken as linear for quantitative responses. The nonlinear transformations modify the connection strengths between additive relationships and phenotypes in an adaptive manner, underlining the potential for an improvement in predictive ability.

Given the availability of dense markers in humans and animals, an alternative or complementary source of input that can be used in equation (2) consists of the elements of a marker-based relationship matrix, as in 17; in this case the a_ij coefficients are replaced by g_ij, i.e., elements of some genome or marker-derived relationship matrix G. As noted by 2, when G is proportional to XX', where X is the incidence matrix of a linear regression model on markers, this is equivalent to Bayesian ridge regression. Of course, nothing precludes using both pedigree-derived and marker-derived inputs in the construction of a neural network.

The objective in ANNs is to arrive at some configuration that fits the training data well but that it also has a reasonable ability of predicting yet to be seen observations. This can be achieved by placing constraints on the size of the network connection strengths, e.g., via shrinkage, and the process is known as regularization. A natural way of attaining this compromise between goodness of fit and predictive ability is by means of Bayesian methods 2111518. In this section, an approach used often for Bayesian regularization in neural networks 1819 is presented along the lines of the hierarchical models employed by quantitative geneticists 15.

Conditionally on m network parameters, the n phenotypes or outputs (represented as D for data) are assumed to be mutually independent, with density function (inputs p are omitted in the notation)

p ( D | b , w , σ 2 , M ) = ∏ i = 1 n N t i | b , w , σ 2 , M

where N(.) denotes a normal density; b is the outer bias parameter; w denotes all connection strength coefficients (including all neuron-specific biases); σ² is the residual variance and M represents a given neural network architecture (i.e., a choice of number of neurons and activation functions). The mean of this distribution is the conditional (given all regression coefficients) expectation function b + c g ∑ k = 1 s w k g k ( b k + ∑ j = 1 n a j j u j * * [ k ] ) , i = 1,2,...,n. The bias parameter b can be eliminated simply by taking deviations from the mean, or assigned a flat prior; for simplicity the first of the two options was employed in this study. The Bayesian approach used in regularized neural networks software (e.g., MATLAB) assigns the same normal prior distribution to each of the connection strengths, assumed independent a priori, such that p(w| σ²_w) = N(0, Iσ²_w), where σ²_w is the variance of connection strengths. More general specifications can be posed, but currently available software (public or commercially) lacks flexibility for doing so. Assuming that the two variance parameters are known, the posterior density of the connection strengths is

P ( w | D , σ 2 , σ w 2 , M ) = P ( D | w , σ 2 , M ) P ( w | σ w 2 , M ) P ( D | σ 2 , σ w 2 , M ) ,

where the denominator is the marginal density of the data, that is

P ( D | σ 2 , σ w 2 , M ) = ∫ P ( D | w , σ 2 , M ) P ( w | σ w 2 , M ) d w .

For a neural network with a least one non-linear activation function, the integral is expressible as

p ( D | σ 2 , σ w 2 , M ) = ( 1 2 π σ 2 ) n 2 ( 1 2 π σ w 2 ) m 2 × ∫ exp [ − 1 2 σ 2 ∑ i = 1 n ( t i − b − c g [ ∑ k = 1 s w k g k ( b k + ∑ j = 1 n a i j u j * * [ k ] ) ] ) − 1 2 σ w 2 w ' w ] d w

which does not have closed form, because of non-linearity. Recall that b can be set to 0 provided the observations are suitably centered.

Although a Bayesian neural network can be fitted using Markov chain Monte Carlo sampling, the computations are taxing because of the enormous non-linearities present coupled with the high-dimensionality of w, such as it is the case with genomic data. An alternative approach is based on computing conditional posterior modes of connection strengths, given some likelihood-based estimates of the variance parameters, i.e., as in best linear unbiased prediction (when viewed as a posterior mode) coupled with restricted maximum likelihood (where estimates of variances are the maximizers of a marginal likelihood). The conditional (given σ² and σ w 2 ) log-posterior density of w is from equation (4)

L ( w | D , σ 2 , σ w 2 , M ) = K + log P ( D | w , σ 2 , M ) + log P ( w | σ w 2 , M ) .

Let β = 1 2 σ 2 and α = 1 2 σ w 2 (a standard notation in neural networks literature), and

F ( α , β ) = β ∑ i = 1 n t i - b - c g ∑ k = 1 s w k g k ( b k + ∑ j = 1 n a i j u j * * [ k ] ) 2 + α w ′ w , (1) = β E D + α E w (2) (3)

where

E D = ∑ i = 1 n ( t i − b − c g [ ∑ k = 1 s w k g k ( b k + ∑ j = 1 n a i j u j * * [ k ] ] ) 2 ,

and E_w = w'w. It follows that maximizing L ( w | D , σ 2 , σ w 2 , M ) is equivalent to minimizing F(α, β). This function is often referred to as a "penalized" sum of squares, and it embeds a compromise between goodness of fit, given by the sum of squares of the residuals E_D, and the degree of model complexity, given by the sum of squares of the network weights E_w. The value w = w^MAP that maximizes L ( w | D , σ 2 , σ w 2 , M ) is the mode of the conditional (given the variances) posterior density of the connection strengths; MAP stands for "maximum a posteriori".

If the additive infinitesimal model is represented as a neural network, the coefficient of heritability is given by h 2 = 1 2 α ∕ 1 2 α + 1 2 β = β α + β . As it can be seen in equation (6), if α<<β, the fitting or training algorithm places more weight on goodness of fit. If α>>β, the algorithm emphasizes reduction in the values of w (shrinkage), which produces a less wiggly output function 20.

Given α and β, the w = w^MAP estimates can be found via any non-linear maximization algorithm as in, e.g., the threshold and survival analysis models of quantitative genetics 21.

A standard procedure used in neural networks (and in the software employed here) infers α and β by maximizing the marginal likelihood of the data in equation (5); this corresponds to what is often known as empirical Bayes. Because (5) does not have a closed form (except in linear neural networks), the marginal likelihood is approximated using a Laplacian integration done in the vicinity of the current value w = w^MAP, which depends in turn on the values of the tuning parameters at which the expansion is made. This type of approach for non-linear mixed models has been used in animal breeding for almost two decades 22.

The Laplacian approximation to the marginal density in equation (5) leads to the representation

log [ p ( D | α , β , M ) ] ≈ K + n 2 log ( β ) + m 2 log ( α ) - F ( α , β ) w = w ( α , β ) M A P - 1 2 log H w = w ( α , β ) M A P

where K is a constant and H = ∂ 2 ∂ w ∂ w ′ F ( α , β ) is the Hessian matrix. A grid search can be used to locate the α, β maximizers of the marginal likelihood in the training set. An alternative approach described by 1823 involves the iteration (updating is from right to left, with w^MAP evaluated at the "old" values of the tuning parameters)

α n e w = m 2 ( w M A P ′ w M A P + t r H M A P - 1 )

and

β n e w = n - m + 2 α M A P t r H M A P - 1 2 ∑ i = 1 n t i - b - ∑ k = 1 s w k g k ( b k + ∑ j = 1 n a i j u j * * [ k ] ) w = w ( α , β ) M A P 2

These expressions, as well as (7), are similar to those that arise in maximum likelihood estimation of variance components 242526, which vary depending on the algorithm used. Since β is a positive parameter, it must be that n > m - 2 α M A P t r H M A P - 1 . The quantity γ = m - 2 α M A P t r H M A P - 1 is called "effective number of parameters" in the network 20 and its value ranges from 0 (or 1, if an overall intercept b is fitted) to m, the total number of connection coefficients and bias parameters in the network. If γ is close to n over-fitting results, leading to poor generalization. It follows that the computations are similar to those used in the linear and non-linear models employed by quantitative geneticists, the salient aspect being that a neural network can be strongly non-linear.

More details on computing procedures for neural networks are in 121418232728. Typically, an algorithm proceeds as follows: 1) initialize α, β and all elements in w. 2) Take one step of the Levenberg-Marquardt algorithm to minimize the objective function F(α, β) and find the current value of w. 3) Compute γ, the effective number of parameters, using the Gauss-Newton approximation to the Hessian matrix in the Levenberg-Marquardt training algorithm. 4) Compute updates α_new and β_new; and 5) iterate steps 2-4 until convergence.

A prototype of the networks considered is in Figure 1; as already noted, the architecture shown has five neurons in the hidden (middle) layer. The ANN examined had from 1 through 6 neurons in the hidden layer. In architectures with a single neuron, two variants were considered. In one, the activation functions were linear (identity) throughout. In this case, e.g., when additive relationships a_ij are used as inputs, the network becomes a random regression on such relationships. If regularization were based on w ~ N ( 0 , A - 1 σ w 2 ) as prior distribution, this would yield the standard additive ("animal") model of quantitative genetics; this was not the case here because the MATLAB software used (see below) bases regularization on w ~ N ( 0 , I σ w 2 ) . The second single-neuron architecture was based on equation (1) with a single outer bias parameter but with a non-linear activation g of the emission made by the sole neuron in the architecture. The algebraic representation of this network is

t i = b + c g ( b + ∑ j = 1 n p i j u j * * ) + e i , i = 1 , 2 , … , n ,

where c is the regression of t_i on the activated emission g ( b + ∑ j = 1 n p i j u j * * ) . The objective here was to explore non-linearities between the inputs (additive or genomic relationships in the Jersey data, or markers genotypes in the wheat data) and the targets (phenotypes); the standard additive genetic model assumes that these relationships are linear. The activation function chosen was the hyperbolic tangent transformation g ( x i ) = e x i - e - x i e x i + e - x i , where x i = b + ∑ j = 1 n p i j u j * * ; here, x can take any value in the real line and g(x_i) is the neuron emission for cow or wheat line i, which takes values between -1 and 1. Given the inputs, the predicted phenotype or network output is

t ^ i = b ^ + ĉ e b ^ + Σ j = 1 n p i j u j * * - e - b ^ - Σ j = 1 n p i j u j * * e b ^ + Σ j = 1 n p i j u j * * + e - b ^ - Σ j = 1 n p i j u j * * i = 1 , 2 , … , n .

In models with 2-6 neurons the emissions were always assigned a hyperbolic tangent activation (the choice of function can be based on, e.g., cross-validation); these activations were summed and collected linearly as shown in Figure 1. For example, with 2 neurons the predictions are obtained as

t ^ i = b ^ + ĉ 1 e b ^ 1 + Σ j = 1 n p i j u j * * [ 1 ] - e - b ^ 1 - Σ j = 1 n p i j u j * * [ 1 ] e b ^ 1 + Σ j = 1 n p i j u j * * [ 1 ] + e - b ^ 1 - ∑ j = 1 n p i j u j * * [ 1 ] + ĉ 2 e b ^ 2 + Σ j = 1 n p i j u j * * [ 2 ] - e b ^ 2 - Σ j = 1 n p i j u j * * [ 2 ] e b ^ 2 + Σ j = 1 n p i j u j * * [ 2 ] + e b ^ 2 + ∑ j = 1 n p i j u j * * [ 2 ] i = 1 , 2 , … , n

where the ĉ coefficients are the estimated linear regressions of the traits on the activated emissions fired by each of the two neurons.

MATLAB's neural networks toolbox 29 was used for fitting the architectures studied, using Bayesian regularization in all cases. As mentioned earlier, two combinations of activation functions were tried: 1) the hyperbolic tangent sigmoid function for activating emissions from each neuron in the hidden layer, plus a linear activation function from the hidden to the output layer, and 2) a linear activation throughout, this corresponding to a linear model with random regression coefficients. To avoid spurious effects caused by starting values in each iterative sequence, the networks were trained 20 times in the Jersey data and 50 times in the wheat data set, for each of the architectures. In Jerseys, each run randomly allocated 60% of the animals to a training set, 20% to a validation set and 20% to a testing set; results reported are the average of the 20 runs for each of the configurations. In wheat, the records were randomly partitioned into a training (480 lines) and a testing (119 lines) set. Each of the 50 random repeats matched exactly those of 28, to provide a comparison with the predictive ability of the Bayesian Lasso and of support vector regression models used with the wheat data set by 30.

The neural networks were fitted to data in the training set, with the α and β parameters, connection strengths and biases modified iteratively. In the Jersey data, as parameters changed in the course of training, the predictive ability of the network was gauged in parallel in the validation set, which was expected to be similar in structure to the testing set, because they were randomly constructed. The same was done with the wheat data, except that there was no "intermediate" validation set. Once the mean squared error of prediction reached an optimal level, training stopped, and this led to the best estimates of the network coefficients. This estimated network was then used for predicting the testing set; predictive correlations (Pearson) and mean-squared errors were evaluated.

Before processing, MATLAB rescales all input and output variables such that they reside in the [-1, +1] range, to enhance numerical stability; this is done automatically using the "mapminmax" function. To illustrate, consider the vector x' = 364 so that x_min = 3 and x_max = 6. If values are to range between A_min = -1 and A_max = +1, one sets temporarily x_temp' = [-1,1,4], so only x₃ = 4 needs to be rescaled. This is done via the formula

x 3 , n e w = A min + x 3 - x min x max - x min ( A max - A min ) = - 1 + 4 - 3 6 - 3 2 = - 1 3 .

MATLAB uses the Levenberg-Marquardt algorithm (based on linearization) for computing the posterior modes in Bayesian regularization, and back-propagation is employed to minimize the penalized residual sum of squares. The maximum number of iterations (called epochs) in back-propagation was set to 1000, and iteration stopped earlier if the gradient of the objective function was below a suitable level or when there were obvious problems with the algorithm 282931. Each of these settings corresponds to the default values provided by MATLAB.

The effective number of parameters (γ) associated with each of the networks examined in the Jersey data is presented in Table 1 and shown graphically in Figure 2, by trait and type of input considered, i.e., additive or genomic relationships. Clearly, use of genomic relationships resulted in a larger number of effective parameters than use of pedigree, for all traits and architectures. When using pedigree relationships, the average (over runs, but note the large standard errors) effective number of parameters ranged from 91 (fat yield, one-neuron model with non-linear activation), to 136 (protein yield, 6 neurons). This illustrates the impact of shrinkage, and of how regularized neural networks cope with the "curse of dimensionality. For example, a 6-neuron network has close to 1800 nominal parameters. Likewise, when using genomic relationships as inputs, the average effective number of parameters ranged from 127 to 166 (fat yield). Similar results were obtained in the wheat data (Table 2). The effective number of parameters ranged from 220 (nonlinear ANN with 4 neurons) to 299 (linear ANN).

The effective number of parameters behaved differentially with respect to model architecture and this depended on the input variables used. When using pedigrees in the Jersey data, the hyperbolic tangent activation function in the 1-neuron model reduced γ drastically, relative to the linear model (1 neuron with linear activation throughout). Then, an increment in number of neurons from 2 to 6 increased model complexity relative to that of the 1 neuron model with non-linear activation, but not beyond that attained with the linear model, save for protein yield. For this trait, γ was 118 for the linear model, and ranged from 126 to 136 in models with 3 through 6 neurons. When genomic relationships were used as inputs, γ was largest for the linear model for fat and protein yield, and for the 1-neuron model with a non-linear activation function in the case of milk yield. In wheat, the effective number of parameters decreased as architectures became more complex. There was large variation among runs in effective number of parameters for both data sets, but there was not a clear pattern in the variability.

Results pertaining to predictive ability evaluated in the testing sets are shown in Table 2 for wheat and Tables 3 and 4 for the Jersey data. Figures 3 and 4 depict mean of squared errors of prediction and correlations coefficients in the Jersey cows.

The predictive correlations in wheat (Table 2) ranged from 0.48 with the linear ANN (equivalent to Bayesian ridge regression) to 0.59 for the nonlinear ANN with 4 neurons. Clear and significant differences between linear and nonlinear architectures were observed. The improvements over the linear ANN were 11.2, 14.3, 15.8 and 18.6% in predictive correlation for 1, 2, 3 and 4 neurons in the hidden layer, respectively. Mean squared errors were also 23-29% smaller than in the linear ANN.

In the Jerseys, the large variability in mean squared error among runs (Table 3) precludes making strong statements about differences among architectures. However, predictive correlations (Table 4) were clearly larger for the non-linear ANN. For fat yield, the results with pedigrees employed as input suggest that a non-linear, adaptive use of additive relationships (as done in all networks with the hyperbolic tangent activation function) can improve predictive performance beyond that of the infinitesimal model. Further, use of genomic relationships led to more reliable prediction of phenotypes than use of pedigree information as measured by the predictive correlations in Table 4. The relative increase in strength of association, as measured by the correlation, is much larger than the ones that have been reported, e.g., in dairy cattle 3237, when prediction of breeding values of bulls was made from genomic information, as opposed to from pedigrees. Our result is encouraging and suggests that genomic data may also play an important role in prediction of individual outcomes (as opposed to breeding value), an issue of relevance in medicine 4.

The distribution of connection strengths in a network gives an indication of the extent of regularization attained. Typically, weight values decrease with model complexity, in the same manner that estimates of marker effects become smaller in Bayesian regression models when p increases and training sample size is kept constant. Further, the distribution of weights is often linked to predictive ability; small values tend to lead to better generalization. Figure 5 depicts the distributions of weights for the linear models and for the nonlinear regularized networks that produced the largest predictive correlations for pedigree and genomic relationships in the Jersey data. The weights for the linear model were larger and more variable than for the nonlinear networks, where distributions were patently leptokurtic, indicating strong shrinkage towards 0. For example, the average (over runs) sum of squares of weights for the linear model and for the non-linear network with 6 neurons when using genomic relationships as predictors of milk yield were 7.5 and 8.5, respectively; however, the 7.5 for the linear model was the sum of squares of 297 weights whereas the 8.5 for the nonlinear model with 6 neurons was the sum of squares of 1782 weights (6 × 297). The same picture was observed in the wheat data (results are not reported).