Archive | Ensemble Learning

Box Plot of Soft Voting Ensemble Compared to Standalone Models for Binary Classification

How to Develop Voting Ensembles With Python

Voting is an ensemble machine learning algorithm. For regression, a voting ensemble involves making a prediction that is the average of multiple other regression models. In classification, a hard voting ensemble involves summing the votes for crisp class labels from other models and predicting the class with the most votes. A soft voting ensemble involves […]

Continue Reading 43
How to Use One-vs-Rest and One-vs-One for Multi-Class Classification

One-vs-Rest and One-vs-One for Multi-Class Classification

Not all classification predictive models support multi-class classification. Algorithms such as the Perceptron, Logistic Regression, and Support Vector Machines were designed for binary classification and do not natively support classification tasks with more than two classes. One approach for using binary classification algorithms for multi-classification problems is to split the multi-class classification dataset into multiple […]

Continue Reading 48
Box Plot of Standalone and Stacking Model Accuracies for Binary Classification

Stacking Ensemble Machine Learning With Python

Stacking or Stacked Generalization is an ensemble machine learning algorithm. It uses a meta-learning algorithm to learn how to best combine the predictions from two or more base machine learning algorithms. The benefit of stacking is that it can harness the capabilities of a range of well-performing models on a classification or regression task and […]

Continue Reading 74
Gradient Boosting with Scikit-Learn, XGBoost, LightGBM, and CatBoost

Gradient Boosting with Scikit-Learn, XGBoost, LightGBM, and CatBoost

Gradient boosting is a powerful ensemble machine learning algorithm. It’s popular for structured predictive modeling problems, such as classification and regression on tabular data, and is often the main algorithm or one of the main algorithms used in winning solutions to machine learning competitions, like those on Kaggle. There are many implementations of gradient boosting […]

Continue Reading 36
How to Develop Multioutput Regression Models in Python

How to Develop Multi-Output Regression Models with Python

Multioutput regression are regression problems that involve predicting two or more numerical values given an input example. An example might be to predict a coordinate given an input, e.g. predicting x and y values. Another example would be multi-step time series forecasting that involves predicting multiple future time series of a given variable. Many machine […]

Continue Reading 142
How to Develop Super Learner Ensembles in Python

How to Develop Super Learner Ensembles in Python

Selecting a machine learning algorithm for a predictive modeling problem involves evaluating many different models and model configurations using k-fold cross-validation. The super learner is an ensemble machine learning algorithm that combines all of the models and model configurations that you might investigate for a predictive modeling problem and uses them to make a prediction […]

Continue Reading 95
How to Use Out-of-Fold Predictions in Machine Learning

How to Use Out-of-Fold Predictions in Machine Learning

Machine learning algorithms are typically evaluated using resampling techniques such as k-fold cross-validation. During the k-fold cross-validation process, predictions are made on test sets comprised of data not used to train the model. These predictions are referred to as out-of-fold predictions, a type of out-of-sample predictions. Out-of-fold predictions play an important role in machine learning […]

Continue Reading 10