Animal Breeding and Genomics Centre, Wageningen University, Wageningen, The Netherlands

Biometris, Plant Research International, Wageningen, The Netherlands

Plant Breeding, Wageningen University, Wageningen, The Netherlands

Animal Breeding and Genomics Centre, Wageningen UR Livestock Research, Lelystad, The Netherlands

Abstract

Background

Genomic selection, the use of markers across the whole genome, receives increasing amounts of attention and is having more and more impact on breeding programs. Development of statistical and computational methods to estimate breeding values based on markers is a very active area of research. A simulated dataset was analyzed by participants of the QTLMAS XIII workshop, allowing a comparison of the ability of different methods to estimate genomic breeding values.

Methods

A best case scenario was analyzed by the organizers where QTL genotypes were known. Participants submitted estimated breeding values for 1000 unphenotyped individuals together with a description of the applied method(s). The submitted breeding values were evaluated for correlation with the simulated values (accuracy), rank correlation of the best 10% of individuals and error in predictions. Bias was tested by regression of simulated on estimated breeding values.

Results

The accuracy obtained from the best case scenario was 0.94. Six research groups submitted 19 sets of estimated breeding values. Methods that assumed the same variance for markers showed accuracies, measured as correlations between estimated and simulated values, ranging from 0.75 to 0.89 and rank correlations between 0.58 and 0.70. Methods that allowed different marker variances showed accuracies ranging from 0.86 to 0.94 and rank correlations between 0.69 and 0.82. Methods assuming equal marker variances were generally more biased and showed larger prediction errors.

Conclusions

The best performing methods achieved very high accuracies, close to accuracies achieved in a best case scenario where QTL genotypes were known without error. Methods that allowed different marker variances generally outperformed methods that assumed equal marker variances. Genomic selection methods performed well compared to traditional, pedigree only, methods; all methods showed higher accuracies than those obtained for breeding values estimated solely on pedigree relationships.

Background

When methods for selection based on many markers across the genome, or genomic selection, were first described

Methods to deal with these large number of markers in breeding programs were first proposed by

The organizers of the previous QTL-MAS workshop initiated a comparison of methods using a simulated dataset which resembled a population one might encounter in litter-bearing animals. They concluded that models that include markers as fixed effects were unlikely to provide any gain from the use of markers, while random effects models and especially the Bayesian analyses were most promising

We aimed to compare methods that estimate genomic breeding values (GEBV) in a dataset that one might encounter in both plant and animal breeding programs and added the complexity of repeated measures over time. Participants of the current QTL-MAS workshop 2009 were invited to predict GEBV and describe their methods and results. Predicted GEBV from the different methods were submitted to the QTL-MAS workshop and compared to the simulated or true breeding values (TBV).

Methods

Simulated data

18 QTL were simulated affecting a trait called yield that followed a logistic growth curve. The growth curve was determined by 3 parameters and for each parameter, 6 QTL determined the genetic value with one large QTL (50% of genetic variance) and 5 smaller QTL. Phenotypic values for the parameters were simulated with a heritability of 0.50. Workshop participants were provided with genotypes for a set of biallelic markers that did not include the genotypes for the 18 QTL. Data available to participants of QTLMAS XIII consisted of 100 full-sib families which resulted from factorial mating of 20 female and 5 male parents. Each full-sib family consisted of 20 offspring. Parent-offspring relationships were provided, but relationships between parents were not. All offspring had genotypes for 453 markers, distributed over 5 chromosomes of 1 Morgan each. Phenotypes were provided for the offspring of 50 full-sib families and consisted of cumulative yield values at 5 different points in time, the last time point being 530. Further details of the simulation are described elsewhere

Best-case analysis

The workshop organizers applied a "best-case" analysis to the simulated data to provide an upper bound of the expected accuracies of the contributed analyses. This best-case analysis made use of additional information which was not provided to the workshop participants. The correct model, a logistic growth curve, was used to estimate 3 growth curve parameters from phenotypes for each individual. More importantly, the true genotypes of the QTL were used. Workshop participants could apply the correct growth model without knowing this was the case, but the actual genotypes could not be used. In the best-case analysis, the true QTL genotypes were used as the only variables in a multitrait fixed regression model to estimate the QTL effects on the 3 growth curve parameters. The estimated growth curve QTL effects were subsequently used to predict breeding value for yield on time point 600 for the unphenotyped individuals.

Prediction strategies

QTLMAS XIII participants were asked to predict breeding values for the unphenotyped offspring (n = 1000) at time point 600. Timepoint 600 was outside the range of time points for which phenotypes were provided. Several strategies could be followed: 1) predict phenotypes at time point 600, using any of several methods, and use these to predict breeding values, 2) use a function to describe the observed phenotypes, predict breeding values for the parameters of this function and use those to calculate EBV at time point 600, 3) predict breeding values for the 5 different time points and use these to extrapolate to time point 600, using any of several methods.

Comparison of predicted breeding values

Accuracies of GEBV, reported by workshop participants on unphenotyped offspring, were calculated as the correlation between the GEBV and the TBV. Bias was assessed from the regression of TBV on the GEBV. The ability of methods to identify the best individuals was assessed from the rank correlations of predicted and TBV of individuals in the top 10% of TBV. Mean squared prediction error was calculated after predicted and TBV were centered on zero. The variances of GEBV were calculated and reported as a proportion of the variance of TBV.

Results

Best-case analysis

The regression model applied directly to the QTL genotypes resulted in the highest accuracy (0.985) and rank correlation (0.935) of GEBV with TBV of unphenotyped individuals. However, regression of TBV on GEBV resulted in a regression coefficient of 0.847, relatively far away from 1 compared to other methods, The variance of GEBV was also higher (34.3) than the variance of the TBV (25.3) and higher than the variance from any other estimation method.

Prediction strategies

All authors applied a procedure with two or three steps. In most cases two steps were used with one step to predict phenotypes or breeding values at time point 600 and one step to estimate genetic effects for the markers. The order of these two steps varied and various methods were applied for both extrapolation to time point 600 as well as for the estimation of marker effects.

Two step strategies

Three authors

Three authors

Three step strategy

One author

Comparison of predicted breeding values

A number of different estimation methods were applied by participants (Table

Extrapolation, prediction and estimation methods

**Step 1**

**Step 2**

**Method**

**Paper**

**Model**

**Extrapolation**

**Prediction**

1

Cleveland

Bayes-A

Gompertz

Bayes-A

2

Cleveland

Lasso

Gompertz

Bayes-Lasso

3

Cleveland

Student- t

Gompertz

Bayes-Student-t

4

Pong-Wong

GEBV2

Gompertz

Bayes-B

5

Schulz-Streeck

Ridge

Logistic

BLUP \ RR

**Step 1**

**Step 2**

**Method**

**Paper**

**Model**

**Prediction**

**Extrapolation**

6

Mucha

SNPL

GRM

quadratic 5 points

7

Veerkamp

14.SNP

Bayes-C

linear 3 points

8

Veerkamp

BLUP

A matrix

linear 3 points

9

Veerkamp

IBD

Bayes-C

linear 3 points

10

Veerkamp

IBS2

Bayes-C

linear 3 points

11

Veerkamp

IBS5

Bayes-C

linear 3 points

12

Veerkamp

SNP1

Bayes-A

linear 3 points

13

Veerkamp

SNP2

Bayes-C

linear 3 points

14

Veerkamp

SNP3

Bayes-C (3dist)

linear 3 points

15

Verbyla

Bayes.BLUP

Bayes

linear 3 points

16

Verbyla

BayesA

Bayes-A

linear 3 points

17

Verbyla

BayesA.B

Bayes

linear 3 points

18

Verbyla

BayesC

Bayes-C

linear 3 points

**Step 1**

**Step 2**

**Step 3**

**Method**

**Paper**

**Model**

**Estimation**

**Prediction**

**Extrapolation**

19

Pong-Wong

GEBV1

Gompertz

Bayes-B

Gompertz

20

QTLMAS

Best case

Logistic

Regression

Logistic

Accuracy

The lowest accuracy was 0.647, obtained with pedigree BLUP (model 8), clearly below the second lowest model (model 6) with an accuracy of 0.751. All other methods performed markedly better than the genomic BLUP model 6 (Figure

Accuracy of contributed GEBV

**Accuracy of contributed GEBV** Ordered values for accuracy of predicted breeding values for phenotype at time point 600. Prediction using pedigree (▼), genomic BLUP (○), Bayes (●), best case (▲) models and true breeding values (♦).

Rank correlations

Rank correlations were calculated from the ranking of the top 10% of individuals, based on TBV. The range of rank correlations for the Bayes methods was 0.691 to 0.816 (Figure

Rank correlations of contributed GEBV

**Rank correlations of contributed GEBV** Ordered values for rank correlations between the predicted and true breeding values of the top 100 TBV individuals. Prediction using pedigree (▼), genomic BLUP (○), Bayes (●), best case (▲) models and true breeding values (♦).

Bias

The pedigree BLUP analysis was found to have a regression coefficient that was closest to 1 (Figure

Bias of contributed GEBV

**Bias of contributed GEBV** Ordered values for regression of simulated on predicted breeding values. Prediction using pedigree (▼), genomic BLUP (○), Bayes (●), best case (▲) models and true breeding values (♦).

Results from comparison to simulated breeding values (TBV)

**Analysis**

**Method**

**Var**

**Acc**

**MSEP**

**Rank**

**Regr**

0

TBV

25.346

1

0

1

1

1

Cle_Bayes.A

26.490

0.916

4.369

0.749

0.896

2

Cle_Lasso

32.909

0.916

5.337

0.716

0.804

3

Cle_Student.t

30.554

0.945

3.322

0.791

0.860

4

Pon_GEBV2^{1}

18.638

0.901

4.794

0.726

1.051

5

Sch_EBV600

20.354

0.889

5.303

0.700

0.992

6

Muc_SNPL

23.204

0.751

12.101

0.578

0.785

7

Vee_14.SNP

20.166

0.930

3.460

0.764

1.043

8

Vee_BLUP

10.763

0.647

14.728

0.485

0.993

9

Vee_IBD

19.009

0.931

3.475

0.767

1.075

10

Vee_IBS2

18.344

0.932

3.504

0.816

1.095

11

Vee_IBS5

19.579

0.929

3.517

0.781

1.057

12

Vee_SNP1

24.975

0.912

4.433

0.719

0.919

13

Vee_SNP2

17.974

0.925

3.828

0.793

1.098

14

Vee_SNP3

17.928

0.927

3.744

0.779

1.102

15

Ver_Bayes.BLUP

20.721

0.885

5.479

0.691

0.979

16

Ver_BayesA

13.783

0.857

7.092

0.696

1.162

17

Ver_BayesA.B

17.124

0.889

5.435

0.730

1.081

18

Ver_BayesC

17.914

0.861

6.561

0.710

1.024

19

Pon_GEBV1

19.726

0.897

4.971

0.709

1.016

20

best_case

34.256

0.985

1.572

0.935

0.847

^{1}After the QTLMAS workshop an error in the implemented software was found by the participant. The adjusted implementation yielded an accuracy of 0.947.

Variance of contributed GEBV

**Variance of contributed GEBV** Ordered values for variance of predicted breeding values. Prediction using pedigree (▼), genomic BLUP (○), Bayes (●), best case (▲) models and true breeding values (♦).

Prediction error

Average prediction error was largest for the pedigree BLUP EBV, which is due to the relatively low accuracies for this method and hence significant shrinkage of the resulting EBV (Figure

Prediction error of contributed GEBV

**Prediction error of contributed GEBV** Ordered values for mean squared prediction error of predicted breeding values. Prediction using pedigree (▼), genomic BLUP (○), Bayes (●), best case (▲) models and true breeding values (♦).

Discussion

2-step and 3-step methods

Accuracy of GEBV from the 3-step method (model 19) was 0.897, which was very similar to the average accuracy, 0.893, of all the other Bayes methods. Because the simulated QTL affect the parameters of the true logistic growth curve it might have been expected that methods that look for associations of markers with these underlying parameters had an advantage. In this dataset this does however not appear to be the case. It was found that extrapolation to time point 600 was not a big challenge because for most individuals this time point was within the part of the growth curve where growth was almost linear. In fact, two authors extrapolated exactly this way, by linear regression on the last three time-points and obtained high accuracies. The impact of methods for extrapolation would have been bigger when a time-point closer to the asymptote would have been chosen. However, extrapolation of the data was a secondary objective of the QTLMAS workshop whereas comparison of methods to obtain GEBV was a primary objective.

Bayes and genomic BLUP methods

QTL contributing to the three parameters of the growth curve were unequal in size. This was expected to favor Bayes methods over genomic BLUP methods that assume the same variance for all markers. This difference was not directly apparent from the results presented in this comparison as one of the genomic BLUP models (model 6) resulted in a lower accuracy (0.751) compared to all other marker methods while the accuracy (0.889) from the other genomic BLUP method (model 5) was higher than some of the Bayes methods. The GEBV from model 6 were obtained using a genetic covariance matrix build from a subset of 90 markers that were selected from only the first of the five chromosomes. The resulting GEBV may have reduced accuracies compared to other methods because not all information was used. However, a method which only selected 14 markers (model 7) still gave a high accuracy (0.930). These 14 markers were selected based on a QTL analysis while the 90 markers selected for model 6 were selected solely on the fact that they map to chromosome one.

Most likely the use of a polygenic component increased the accuracy of model 5. While a polygenic component was not used by model 6, some of the Bayes methods did include polygenes (models 7 to 18). In addition to the relatively high accuracy, the best genomic BLUP model (model 5) produced unbiased GEBV where many of the other methods showed moderately to severely biased results.

The structure of the data was such, that full sib families were either completely genotyped, or completely not genotyped. This structure was chosen, to avoid that models would benefit too much from close relationships between the animals in the training and validation data. The larger the distance between training and validation, the more emphasis on LD information to predict GEBV of animals without phenotypes

Number of markers included in the model

The proportion of markers, π, selected into the model or into the distribution of large marker effects was reported by Pong-Wong and found to be relatively high, even close to 1 for one of their methods where they tried to let the model decide on the value of π . Nevertheless, the proportion of markers that was included in the distribution of markers with large effect in more than half of the cycles (i.e. that had a posterior probability > 0.5), was limited. The posterior proportions in the various distributions were not reported by the other authors. Results obtained with a method that solely included 14 markers (model 7) show that very high accuracies could be obtained, at least in this dataset, with a small number, and a small proportion of the total number of markers selected into the model. The 14 SNPs included in model 7 were selected based on their association with the phenotype and analysed with a Bayes model. Preselection of SNPs and analysing them as fixed effects has not been considered by any of the participants. Results from QTLMAS comparison in 2008

Conclusions

Accuracies of GEBV were always higher than those estimated based on pedigree alone (model 8). Methods that allow different variances of markers generally performed better than genomic BLUP methods that assume equal variance for all markers but differences were not very large, except when only a portion of the genome was used (model 6). The best Bayes method achieved an accuracy that was 0.056 higher than the best genomic BLUP method. The simulated QTL varied strongly in size which will have favored the Bayes methods in the comparison of accuracies between the two types of methods. The highest accuracies obtained were very close to those from the best case analysis, where knowledge about QTL genotypes was used. Methods to extrapolate to time point 600 from the observed phenotypes at time points up to 530 appear to have had a minor impact on the accuracies of GEBV.

Competing interests

The authors declare that they have no competing interests.

Authors' contributions

JB performed the comparative analyses, MC performed the best-case analysis, JB and MC drafted the paper. All authors contributed to the design of the study and the methods for comparative analyses. CM, MB and AC contributed to discussion of the results and revision of the paper.

Acknowledgement

This article has been published as part of BMC Proceedings Volume 4 Supplement 1, 2009: Proceedings of 13th European workshop on QTL mapping and marker assisted selection.

The full contents of the supplement are available online at