On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practice

Variant pathogenicity classifiers such as SIFT, PolyPhen-2, CADD, and MetaLR assist in interpretation of the hundreds of rare, missense variants in the typical patient genome by deprioritizing some variants as likely benign. These widely used methods misclassify 26 to 38% of known pathogenic mutations, which could lead to missed diagnoses if the classifiers are trusted as definitive in a clinical setting. We developed M-CAP, a clinical pathogenicity classifier that outperforms existing methods at all thresholds and correctly dismisses 60% of rare, missense variants of uncertain significance in a typical genome at 95% sensitivity.

M-CAP eliminates a majority of variants of uncertain significance in clinical exomes at high sensitivity

Background: As more and more researchers are turning to big data for new opportunities of biomedical discoveries, machine learning models, as the backbone of big data analysis, are mentioned more often in biomedical journals. However, owing to the inherent complexity of machine learning methods, they are prone to misuse. Because of the flexibility in specifying machine learning models, the results are often insufficiently reported in research articles, hindering reliable assessment of model validity and consistent interpretation of model outputs. Objective: To attain a set of guidelines on the use of machine learning predictive models within clinical settings to make sure the models are correctly applied and sufficiently reported so that true discoveries can be distinguished from random coincidence. Methods: A multidisciplinary panel of machine learning experts, clinicians, and traditional statisticians were interviewed, using an iterative process in accordance with the Delphi method. Results: The process produced a set of guidelines that consists of (1) a list of reporting items to be included in a research article and (2) a set of practical sequential steps for developing predictive models. Conclusions: A set of guidelines was generated to enable correct application of machine learning models and consistent reporting of model specifications and results in biomedical research. We believe that such guidelines will accelerate the adoption of big data analysis, particularly with machine learning methods, in the biomedical research community. [J Med Internet Res 2016;18(12):e323]

/pdf/guidelines-for-developing-and-reporting-machine-learning-3pbgsbjz5b.pdf

Guidelines for Developing and Reporting Machine Learning Predictive Models in Biomedical Research: A Multidisciplinary View.

https://www.cell.com/trends/plant-science/pdf/S1360-1385(14)00141-1.pdf

Genomic selection: genome-wide prediction in plant improvement

An enhanced extreme learning machine model for river flow forecasting: State-of-the-art, practical applications in water resource engineering area and future research direction

Background
Genomic selection (GS) is emerging as an efficient and cost-effective method for estimating breeding values using molecular markers distributed over the entire genome. In essence, it involves estimating the simultaneous effects of all genes or chromosomal segments and combining the estimates to predict the total genomic breeding value (GEBV). Accurate prediction of GEBVs is a central and recurring challenge in plant and animal breeding. The existence of a bewildering array of approaches for predicting breeding values using markers underscores the importance of identifying approaches able to efficiently and accurately predict breeding values. Here, we comparatively evaluate the predictive performance of six regularized linear regression methods-- ridge regression, ridge regression BLUP, lasso, adaptive lasso, elastic net and adaptive elastic net-- for predicting GEBV using dense SNP markers.

/pdf/genomic-selection-using-regularized-linear-regression-models-2hlsmdbsrk.pdf

Genomic selection using regularized linear regression models: ridge regression, lasso, elastic net and their extensions

Genomic selection (GS) involves estimating breeding values using molecular markers spanning the entire genome. Accurate prediction of genomic breeding values (GEBVs) presents a central challenge to contemporary plant and animal breeders. The existence of a wide array of marker-based approaches for predicting breeding values makes it essential to evaluate and compare their relative predictive performances to identify approaches able to accurately predict breeding values. We evaluated the predictive accuracy of random forests (RF), stochastic gradient boosting (boosting) and support vector machines (SVMs) for predicting genomic breeding values using dense SNP markers and explored the utility of RF for ranking the predictive importance of markers for pre-screening markers or discovering chromosomal locations of QTLs. We predicted GEBVs for one quantitative trait in a dataset simulated for the QTLMAS 2010 workshop. Predictive accuracy was measured as the Pearson correlation between GEBVs and observed values using 5-fold cross-validation and between predicted and true breeding values. The importance of each marker was ranked using RF and plotted against the position of the marker and associated QTLs on one of five simulated chromosomes. The correlations between the predicted and true breeding values were 0.547 for boosting, 0.497 for SVMs, and 0.483 for RF, indicating better performance for boosting than for SVMs and RF. Accuracy was highest for boosting, intermediate for SVMs and lowest for RF but differed little among the three methods and relative to ridge regression BLUP (RR-BLUP).

/pdf/a-comparison-of-random-forests-boosting-and-support-vector-zqeq6db5ki.pdf

A comparison of random forests, boosting and support vector machines for genomic selection

Plant breeders and variety testing agencies routinely test candidate genotypes (crop varieties, lines, test hybrids) in multiple environments. Such multi-environment trials can be efficiently analysed by mixed models. A single-stage analysis models the entire observed data at the level of individual plots. This kind of analysis is usually considered as the gold standard. In practice, however, it is more convenient to use a two-stage approach, in which experiments are first analysed per environment, yielding adjusted means per genotype, which are then summarised across environments in the second stage. Stage-wise approaches suggested so far are approximate in that they cannot fully reproduce a single-stage analysis, except in very simple cases, because the variance-covariance matrix of adjusted means from individual environments needs to be approximated by a diagonal matrix. This paper proposes a fully efficient stage-wise method, which carries forward the full variance-covariance matrix of adjusted means from the individual environments to the analysis across the series of trials. Provided the variance components are known, this method can fully reproduce the results of a single-stage analysis. Computations are made efficient by a diagonalisation of the residual variance-covariance matrix, which necessitates a corresponding linear transformation of both the first-stage estimates (e.g. adjusted means and regression slopes for plot covariates) and the corresponding design matrices for fixed and random effects. We also exemplify the extension of the general approach to a three-stage analysis. The method is illustrated using two datasets, one real and the other simulated. The proposed approach has close connections with meta-analysis, where environments correspond to centres and genotypes to medical treatments. We therefore compare our theoretical results with recently published results from a meta-analysis.

A stage-wise approach for the analysis of multi-environment trials.

Using different populations in genomic selection raises the possibility of marker effects varying across populations. However, common models for genomic selection only account for the main marker effects, assuming that they are consistent across populations. We present an approach in which the main plus population-specific marker effects are simultaneously estimated in a single mixed model. Cross-validation is used to compare the predictive ability of this model to that of the ridge regression best linear unbiased prediction (RR-BLUP) method involving only either the main marker effects or the population-specific marker effects. We used a maize (Zea mays L.) data set with 312 genotypes derived from five biparental populations, which were genotyped with 39,339 markers. A combined analysis incorporating genotypes for all the populations and hence using a larger training set was better than separate analyses for each population. Modeling the main plus the population-specific marker effects simultaneously improved predictive ability only slightly compared with modeling only the main marker effects. The performance of the RR-BLUP method was comparable to that of two regularization methods, namely the ridge regression and the elastic net, and was more accurate than that of the least absolute shrinkage and selection operator (LASSO). Overall, combining information from related populations and increasing the number of genotypes improved predictive ability, but further allowing for population-specific marker effects made minor improvement.

Genomic Selection using Multiple Populations

Genomic selection (GS) is a method for predicting breeding values of plants or animals using many molecular markers that is commonly implemented in two stages. In plant breeding the first stage usually involves computation of adjusted means for genotypes which are then used to predict genomic breeding values in the second stage. We compared two classical stage-wise approaches, which either ignore or approximate correlations among the means by a diagonal matrix, and a new method, to a single-stage analysis for GS using ridge regression best linear unbiased prediction (RR-BLUP). The new stage-wise method rotates (orthogonalizes) the adjusted means from the first stage before submitting them to the second stage. This makes the errors approximately independently and identically normally distributed, which is a prerequisite for many procedures that are potentially useful for GS such as machine learning methods (e.g. boosting) and regularized regression methods (e.g. lasso). This is illustrated in this paper using componentwise boosting. The componentwise boosting method minimizes squared error loss using least squares and iteratively and automatically selects markers that are most predictive of genomic breeding values. Results are compared with those of RR-BLUP using fivefold cross-validation. The new stage-wise approach with rotated means was slightly more similar to the single-stage analysis than the classical two-stage approaches based on non-rotated means for two unbalanced datasets. This suggests that rotation is a worthwhile pre-processing step in GS for the two-stage approaches for unbalanced datasets. Moreover, the predictive accuracy of stage-wise RR-BLUP was higher (5.0–6.1 %) than that of componentwise boosting.

Torben Schulz-Streeck

Papers

Genomic selection using regularized linear regression models: ridge regression, lasso, elastic net and their extensions

A comparison of random forests, boosting and support vector machines for genomic selection

A stage-wise approach for the analysis of multi-environment trials.

Genomic Selection using Multiple Populations

Comparisons of single-stage and two-stage approaches to genomic selection.