Ensemble Learning Algorithms With Python Crash Course.
Get on top of ensemble learning with Python in 7 days.
Ensemble learning refers to machine learning models that combine the predictions from two or more models.
Ensembles are an advanced approach to machine learning that are often used when the capability and skill of the predictions are more important than using a simple and understandable model. As such, they are often used by top and winning participants in machine learning competitions like the One Million Dollar Netflix Prize and Kaggle Competitions.
Modern machine learning libraries like scikit-learn Python provide a suite of advanced ensemble learning methods that are easy to configure and use correctly without data leakage, a common concern when using ensemble algorithms.
In this crash course, you will discover how you can get started and confidently bring ensemble learning algorithms to your predictive modeling project with Python in seven days.
This is a big and important post. You might want to bookmark it.
Kick-start your project with my new book Ensemble Learning Algorithms With Python, including step-by-step tutorials and the Python source code files for all examples.
Let’s get started.
Ensemble Machine Learning With Python (7-Day Mini-Course) Photo by anoldent, some rights reserved.
Who Is This Crash-Course For?
Before we get started, let’s make sure you are in the right place.
This course is for developers who may know some applied machine learning. Maybe you know how to work through a predictive modeling problem end to end, or at least most of the main steps, with popular tools.
The lessons in this course do assume a few things about you, such as:
You know your way around basic Python for programming.
You may know some basic NumPy for array manipulation.
You may know some basic scikit-learn for modeling.
You do NOT need to be:
A math wiz!
A machine learning expert!
This crash course will take you from a developer who knows a little machine learning to a developer who can effectively and competently apply ensemble learning algorithms on a predictive modeling project.
Note: This crash course assumes you have a working Python 3 SciPy environment with at least NumPy installed. If you need help with your environment, you can follow the step-by-step tutorial here:
This crash course is broken down into seven lessons.
You could complete one lesson per day (recommended) or complete all of the lessons in one day (hardcore). It really depends on the time you have available and your level of enthusiasm.
Below is a list of the seven lessons that will get you started and productive with data preparation in Python:
Lesson 01: What Is Ensemble Learning?
Lesson 02: Bagging Ensembles
Lesson 03: Random Forest Ensemble
Lesson 04: AdaBoost Ensemble
Lesson 05: Gradient Boosting Ensemble
Lesson 06: Voting Ensemble
Lesson 07: Stacking Ensemble
Each lesson could take you 60 seconds or up to 30 minutes. Take your time and complete the lessons at your own pace. Ask questions, and even post results in the comments below.
The lessons might expect you to go off and find out how to do things. I will give you hints, but part of the point of each lesson is to force you to learn where to go to look for help with and about the algorithms and the best-of-breed tools in Python. (Hint: I have all of the answers on this blog; use the search box.)
Post your results in the comments; I’ll cheer you on!
Hang in there; don’t give up.
Lesson 01: What Is Ensemble Learning?
In this lesson, you will discover what ensemble learning is and why it is important.
Applied machine learning often involves fitting and evaluating models on a dataset.
Given that we cannot know which model will perform best on the dataset beforehand, this may involve a lot of trial and error until we find a model that performs well or best for our project.
An alternate approach is to prepare multiple different models, then combine their predictions.
This is called an ensemble machine learning model, or simply an ensemble, and the process of finding a well-performing ensemble model is referred to as “ensemble learning.”
Although there is nearly an unlimited number of ways that this can be achieved, there are perhaps three classes of ensemble learning techniques that are most commonly discussed and used in practice.
Their popularity is due in large part to their ease of implementation and success on a wide range of predictive modeling problems.
They are:
Bagging, e.g. bagged decision trees and random forest.
Boosting, e.g. adaboost and gradient boosting
Stacking, e.g. voting and using a meta-model.
There are two main reasons to use an ensemble over a single model, and they are related; they are:
Reliability: Ensembles can reduce the variance of the predictions.
Skill: Ensembles can achieve better performance than a single model.
These are both important concerns on a machine learning project and sometimes we may prefer one or both properties from a model.
Your Task
For this lesson, you must list three applications of ensemble learning.
These may be famous examples, like a machine learning competition, or examples you have come across in tutorials, books, or research papers.
Post your answer in the comments below. I would love to see what you come up with.
In the next lesson, you will discover how to develop and evaluate a bagging ensemble.
Lesson 02: Bagging Ensembles
In this lesson, you will discover the bootstrap aggregation, or bagging, ensemble.
Bagging works by creating samples of the training dataset and fitting a decision tree on each sample.
The differences in the training datasets result in differences in the fit decision trees, and in turn, differences in predictions made by those trees. The predictions made by the ensemble members are then combined using simple statistics, such as voting or averaging.
Key to the method is the manner in which each sample of the dataset is prepared to train ensemble members. Examples (rows) are drawn from the dataset at random, although with replacement. Replacement means that if a row is selected, it is returned to the training dataset for potential re-selection in the same training dataset.
This is called a bootstrap sample, giving the technique its name.
Bagging is available in scikit-learn via the BaggingClassifier and BaggingRegressor classes, which use a decision tree as the base-model by default and you can specify the number of trees to create via the “n_estimators” argument.
The complete example of evaluating a bagging ensemble for classification is listed below.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# example of evaluating a bagging ensemble for classification
from numpy import mean
from numpy import std
from sklearn.datasets import make_classification
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import RepeatedStratifiedKFold
For this lesson, you must run the example and review the results of the evaluated model.
For bonus points, evaluate the effect of using more decision trees in the ensemble or even change the base learner that is used.
Post your answer in the comments below. I would love to see what you come up with.
In the next lesson, you will discover how to develop and evaluate a random forest ensemble.
Lesson 03: Random Forest Ensemble
In this lesson, you will discover the random forest ensemble.
Random forest is an extension of the bagging ensemble.
Like bagging, the random forest ensemble fits a decision tree on different bootstrap samples of the training dataset.
Unlike bagging, random forest will also sample the features (columns) of each dataset.
Specifically, split points are chosen in the data while constructing each decision tree. Rather than considering all features when choosing a split point, random forest limits the features to a random subset of features, such as 3 if there were 10 features.
The random forest ensemble is available in scikit-learn via the RandomForestClassifier and RandomForestRegressor classes. You can specify the number of trees to create via the “n_estimators” argument and the number of randomly selected features to consider at each split point via the “max_features” argument, which is set to the square root of the number of features in your dataset by default.
The complete example of evaluating a random forest ensemble for classification is listed below.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# example of evaluating a random forest ensemble for classification
from numpy import mean
from numpy import std
from sklearn.datasets import make_classification
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import RepeatedStratifiedKFold
from sklearn.ensemble import RandomForestClassifier
For this lesson, you must run the example and review the results of the evaluated model.
For bonus points, evaluate the effect of using more decision trees in the ensemble or tuning the number of features to consider at each split point.
Post your answer in the comments below. I would love to see what you come up with.
In the next lesson, you will discover how to develop and evaluate an AdaBoost ensemble.
Lesson 04: AdaBoost Ensemble
In this lesson, you will discover the adaptive boosting or AdaBoost ensemble.
Boosting involves adding models sequentially to the ensemble where new models attempt to correct the errors made by prior models already added to the ensemble. As such, the more ensemble members that are added, the fewer errors the ensemble is expected to make, at least to a limit supported by the data and before overfitting the training dataset.
The idea of boosting was first developed as a theoretical idea, and the AdaBoost algorithm was the first successful approach to realizing a boosting-based ensemble algorithm.
AdaBoost works by fitting decision trees on versions of the training dataset weighted so that the tree pays more attention to examples (rows) that the prior members got wrong, and less attention to those that the prior models got correct.
Rather than full decision trees, AdaBoost uses very simple trees that make a single decision on one input variable before making a prediction. These short trees are referred to as decision stumps.
AdaBoost is available in scikit-learn via the AdaBoostClassifier and AdaBoostRegressor classes, which use a decision tree (decision stump) as the base-model by default and you can specify the number of trees to create via the “n_estimators” argument.
The complete example of evaluating an AdaBoost ensemble for classification is listed below.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# example of evaluating an adaboost ensemble for classification
from numpy import mean
from numpy import std
from sklearn.datasets import make_classification
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import RepeatedStratifiedKFold
For this lesson, you must run the example and review the results of the evaluated model.
For bonus points, evaluate the effect of using more decision trees in the ensemble or even change the base learner that is used (note, it must support weighted training data).
Post your answer in the comments below. I would love to see what you come up with.
In the next lesson, you will discover how to develop and evaluate a gradient boosting ensemble.
Lesson 05: Gradient Boosting Ensemble
In this lesson, you will discover the gradient boosting ensemble.
Gradient boosting is a framework for boosting ensemble algorithms and an extension to AdaBoost.
It re-frames boosting as an additive model under a statistical framework and allows for the use of arbitrary loss functions to make it more flexible and loss penalties (shrinkage) to reduce overfitting.
Gradient boosting also introduces ideas of bagging to the ensemble members, such as sampling of the training dataset rows and columns, referred to as stochastic gradient boosting.
It is a very successful ensemble technique for structured or tabular data, although it can be slow to fit a model given that models are added sequentially. More efficient implementations have been developed, such as the popular extreme gradient boosting (XGBoost) and light gradient boosting machines (LightGBM).
Gradient boosting is available in scikit-learn via the GradientBoostingClassifier and GradientBoostingRegressor classes, which use a decision tree as the base-model by default. You can specify the number of trees to create via the “n_estimators” argument and the learning rate that controls the contribution from each tree via the “learning_rate” argument that defaults to 0.1.
The complete example of evaluating a gradient boosting ensemble for classification is listed below.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# example of evaluating a gradient boosting ensemble for classification
from numpy import mean
from numpy import std
from sklearn.datasets import make_classification
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import RepeatedStratifiedKFold
from sklearn.ensemble import GradientBoostingClassifier
For this lesson, you must run the example and review the results of the evaluated model.
For bonus points, evaluate the effect of using more decision trees in the ensemble or try different learning rate values.
Post your answer in the comments below. I would love to see what you come up with.
In the next lesson, you will discover how to develop and evaluate a voting ensemble.
Lesson 06: Voting Ensemble
In this lesson, you will discover the voting ensemble.
Voting ensembles use simple statistics to combine the predictions from multiple models.
Typically, this involves fitting multiple different model types on the same training dataset, then calculating the average prediction in the case of regression or the class label with the most votes for classification, called hard voting.
Voting can also be used when predicting the probability of class labels on classification problems by summing predicted probabilities and selecting the label with the largest summed probability. This is called soft voting and is preferred when the base-models used in the ensemble natively support predicting class probabilities as it can result in better performance.
Voting ensembles are available in scikit-learn via the VotingClassifier and VotingRegressor classes. A list of base-models can be provided as an argument to the model and each model in the list must be a tuple with a name and the model, e.g. (‘lr’, LogisticRegression()). The type of voting used for classification can be specified via the “voting” argument and set to either ‘soft‘ or ‘hard‘.
The complete example of evaluating a voting ensemble for classification is listed below.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
# example of evaluating a voting ensemble for classification
from numpy import mean
from numpy import std
from sklearn.datasets import make_classification
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import RepeatedStratifiedKFold
from sklearn.ensemble import VotingClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.linear_model import LogisticRegression
For this lesson, you must run the example and review the results of the evaluated model.
For bonus points, evaluate the effect of trying different types of models in the ensemble or even change the type of voting from soft voting to hard voting.
Post your answer in the comments below. I would love to see what you come up with.
In the next lesson, you will discover how to develop and evaluate a stacking ensemble.
Lesson 07: Stacking Ensemble
In this lesson, you will discover the stacked generalization or stacking ensemble.
Stacking involves combining the predictions of multiple different types of base-models, much like voting.
The important difference from voting is that another machine learning model is used to learn how to best combine the predictions of the base-models. This is often a linear model, such as a linear regression for regression problems or logistic regression for classification, but can be any machine learning model you like.
The meta-model is trained on the predictions made by base-models on out-of-sample data.
This involves using k-fold cross-validation for each base-model and storing all of the out-of-fold predictions. The base-models are then trained on the entire training dataset, and the meta-model is trained on the out-of-fold predictions and learns which model to trust, the degree to trust them, and under which circumstances.
Although internally stacking uses k-fold cross-validation to train the meta model, you can evaluate stacking models any way you like, such as via a train-test split or k-fold cross-validation. The evaluation of the model is separate from this internal resampling-for-training process.
Stacking ensembles are available in scikit-learn via the StackingClassifier and StackingRegressor classes. A list of base-models can be provided as an argument to the model and each model in the list must be a tuple with a name and the model, e.g. (‘lr’, LogisticRegression()). The meta-learner can be specified via the “final_estimator” argument and the resampling strategy can be specified via the “cv” argument and can be simply set to an integer indicating the number of cross-validation folds.
The complete example of evaluating a stacking ensemble for classification is listed below.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
# example of evaluating a stacking ensemble for classification
from numpy import mean
from numpy import std
from sklearn.datasets import make_classification
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import RepeatedStratifiedKFold
from sklearn.ensemble import StackingClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.linear_model import LogisticRegression
HOMEWORK 1.
Bagging: Machine Learning for Medical Diagnosis / Prognosis-Algorithm Rule Extraction for MEdical Diagnosis (REMED). For this type of domain, which requires accuracy in its predictions, as well as comprehensibility and reliability in the patterns obtained, symbolic classifiers are used (e.g. decision trees and rules), which is based on three stages: (i ) selection of the most relevant attributes; (ii) selection of the partitions that separate the examples of each class; and (iii) construction of the model that allows the examples to be classified; This is due to the fact that through them, intelligible models for human beings will be generated, capable of providing medical personnel with a new point of view that helps them make correct decisions in the difficult task of medical diagnosis / prognosis.
Boosting: Prediction of financial fragility for Colombian corporations through the application of classification trees and boosting techniques. Apply the techniques for the prediction of business bankruptcy: classification trees and boosting. It details the methodological procedure, the data purification and treatment criteria, as well as the model estimation process, to finally present the set of financial indicators that explain business bankruptcy to a greater extent: solvency and level of indebtedness; thus, as the most efficient prediction model, making use of criteria such as the classification rate and the metric based on the area under the ROC curve.
Stacking: Application of aggregate methods in the detection of outliers in meteorological time series. Application that allows adapting 3 non-aggregated classifiers (vector support regression, ARIMA, Bayesian networks) and 3 aggregated classifiers (stacking, bagging and AdaBoost) to 3 data sets of meteorological measurements (precipitation, maximum temperature and solar radiation). Compare the classifiers. In the first instance, the classifiers were averaged by performing multiple tests on each data set. Then, by means of a statistical test of hypothesis, the means obtained by the classifiers were compared to determine if the observed differences were significant. Finally, an analysis of the results was carried out, focused on comparing the performance of the aggregated classifiers against the performance of the best non-aggregated classifier in each data set. In general, it was found that it is possible to significantly improve performance by detecting point outliers in some uni-variable time series using aggregate methods. However, to achieve this improvement, conditions must be met that, although they vary depending on the method added, generally aim to improve the diversity of the base classifiers. When these conditions are not met, the aggregated methods did not have a significant difference in performance from the non-aggregated algorithm that performed best in the data set.
Some of the applications of ensemble learning are listed below
Detection of Lung abnormalities
Speech emotion recognition
Forcasting cryptocurrency time series
Hyperspectral Image Analysis
The issue of ensemble is its robustness. Any comment on this: For instance, assume you have done your ensemble for a data from date 2015 to date 2016. You got your model setup and calibration obtained using the above data (2015-2016).
Now, with such fixed model setup/calibration, use the data from 2014 to 2015.
How good/bad is the outcome from 2014 to 2015 when compared to 2015-2016?
Applications of Ensemble Learning
Ensemble learning is a machine learning technique that combines the predictions from multiple individual models to produce a more accurate and robust prediction. Ensemble learning is a fairly common strategy in deep learning and has been applied to tackle a myriad of problems. Classification and Regression: Ensemble methods are widely used for both classification and regression tasks. Algorithms like Random Forest and Gradient Boosting create an ensemble of decision trees to make predictions. These ensembles often outperform individual decision trees by reducing overfitting and improving predictive accuracy.
1. Anomaly Detection
2. Disease detection
3. Remote Sensing
4. Fraud Detection
Ensemble techniques can be used to build a model that is more robust to noisy data and can better distinguish between normal and anomalous instances. One common approach is to use a combination of different anomaly detection algorithms as an ensemble to improve overall performance.
# example of evaluating a bagging ensemble for classification
from numpy import mean
from numpy import std
from sklearn.datasets import make_classification
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import RepeatedStratifiedKFold
from sklearn.ensemble import BaggingClassifier
# create the synthetic classification dataset
X, y = make_classification(random_state=1)
# configure the ensemble model
model = BaggingClassifier(n_estimators=50)
# configure the resampling method
cv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)
# evaluate the ensemble on the dataset using the resampling method
n_scores = cross_val_score(model, X, y, scoring=’accuracy’, cv=cv, n_jobs=-1)
# report ensemble performance
print(‘Mean Accuracy: %.3f (%.3f)’ % (mean(n_scores), std(n_scores)))
A mean accuracy of 0.950 with a standard deviation of 0.072 suggests that the model or system being evaluated is performing quite well. An accuracy of 0.950 means that it is correctly classifying or predicting outcomes with high accuracy, and the low standard deviation indicates that the performance is consistent across different runs or samples.
# example of evaluating a random forest ensemble for classification
from numpy import mean
from numpy import std
from sklearn.datasets import make_classification
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import RepeatedStratifiedKFold
from sklearn.ensemble import RandomForestClassifier
# create the synthetic classification dataset
X, y = make_classification(random_state=1)
# configure the ensemble model
model = RandomForestClassifier(n_estimators=50)
# configure the resampling method
cv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)
# evaluate the ensemble on the dataset using the resampling method
n_scores = cross_val_score(model, X, y, scoring=’accuracy’, cv=cv, n_jobs=-1)
# report ensemble performance
print(‘Mean Accuracy: %.3f (%.3f)’ % (mean(n_scores), std(n_scores)))
# Additionally, we can evaluate the effect of changing the number of decision trees (n_estimators) and the number of features considered at each split point (max_features)
# Number of decision trees in the ensemble (I tryed different values)
n_estimators_values = [10, 50, 100, 200]
here is the results
n_estimators=10, max_features=auto
Mean Accuracy: 0.940 (0.076)
n_estimators=10, max_features=sqrt
Mean Accuracy: 0.940 (0.076)
n_estimators=10, max_features=log2
Mean Accuracy: 0.940 (0.076)
n_estimators=50, max_features=auto
Mean Accuracy: 0.950 (0.067)
n_estimators=50, max_features=sqrt
Mean Accuracy: 0.950 (0.067)
n_estimators=50, max_features=log2
Mean Accuracy: 0.950 (0.067)
n_estimators=100, max_features=auto
Mean Accuracy: 0.953 (0.067)
n_estimators=100, max_features=sqrt
Mean Accuracy: 0.953 (0.067)
n_estimators=100, max_features=log2
Mean Accuracy: 0.953 (0.067)
n_estimators=200, max_features=auto
Mean Accuracy: 0.957 (0.067)
n_estimators=200, max_features=sqrt
Mean Accuracy: 0.957 (0.067)
n_estimators=200, max_features=log2
Mean Accuracy: 0.957 (0.067)
As you can see from the results, there are some variations. As the ‘n’ value increases, the accuracy also increases some extent.
# example of evaluating an AdaBoost ensemble for classification
from numpy import mean
from numpy import std
from sklearn.datasets import make_classification
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import RepeatedStratifiedKFold
from sklearn.ensemble import AdaBoostClassifier
# create the synthetic classification dataset
X, y = make_classification(random_state=1)
# configure the ensemble model
model = AdaBoostClassifier(n_estimators=50)
# configure the resampling method
cv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)
# evaluate the ensemble on the dataset using the resampling method
n_scores = cross_val_score(model, X, y, scoring=’accuracy’, cv=cv, n_jobs=-1)
# report ensemble performance
print(‘Mean Accuracy: %.3f (%.3f)’ % (mean(n_scores), std(n_scores)))
Mean Accuracy: 0.947 (0.088)
The mean accuracy of the AdaBoost ensemble with 50 decision trees on this synthetic dataset is approximately 0.94/0, with a standard deviation of around 0.088.
For bonus points, let’s experiment with different numbers of decision trees in the ensemble and change the base learner:
Changing the Number of Decision Trees:
We can try different values for n_estimators to see how it affects the performance. Let’s try with 100 decision trees:
# Change the number of decision trees to 100
model = AdaBoostClassifier(n_estimators=100)
n_scores = cross_val_score(model, X, y, scoring=’accuracy’, cv=cv, n_jobs=-1)
print(‘Mean Accuracy with 100 Decision Trees: %.3f (%.3f)’ % (mean(n_scores), std(n_scores)))
Mean Accuracy with 100 Decision Trees: 0.950 (0.089)
Changing the Base Learner:
AdaBoost allows you to change the base learner. Let’s try using a different base learner, such as a Decision Tree with a maximum depth of 2:
from sklearn.tree import DecisionTreeClassifier
# Change the base learner to a Decision Tree with max depth 2
base_estimator = DecisionTreeClassifier(max_depth=2)
model = AdaBoostClassifier(base_estimator=base_estimator, n_estimators=50)
n_scores = cross_val_score(model, X, y, scoring=’accuracy’, cv=cv, n_jobs=-1)
print(‘Mean Accuracy with Decision Tree Base Learner (Max Depth 2): %.3f (%.3f)’ % (mean(n_scores), std(n_scores)))
Mean Accuracy with Decision Tree Base Learner (Max Depth 2): 0.940 (0.076)
To run the provided code and evaluate the Gradient Boosting Classifier on the synthetic classification dataset, let use the following steps:
# example of evaluating a gradient boosting ensemble for classification
from numpy import mean
from numpy import std
from sklearn.datasets import make_classification
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import RepeatedStratifiedKFold
from sklearn.ensemble import GradientBoostingClassifier
# create the synthetic classification dataset
X, y = make_classification(random_state=1)
# configure the ensemble model
model = GradientBoostingClassifier(n_estimators=50)
# configure the resampling method
cv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)
# evaluate the ensemble on the dataset using the resampling method
n_scores = cross_val_score(model, X, y, scoring=’accuracy’, cv=cv, n_jobs=-1)
# report ensemble performance
print(‘Mean Accuracy: %.3f (%.3f)’ % (mean(n_scores), std(n_scores)))
Mean Accuracy: 0.930 (0.090)
Let evaluate the effect of using more decision trees in the ensemble or different learning rate values, by modifying the model definition.
model = GradientBoostingClassifier(n_estimators=100, learning_rate=0.1)
# example of evaluating a gradient boosting ensemble for classification
from numpy import mean
from numpy import std
from sklearn.datasets import make_classification
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import RepeatedStratifiedKFold
from sklearn.ensemble import GradientBoostingClassifier
# create the synthetic classification dataset
X, y = make_classification(random_state=1)
# configure the ensemble model
model = GradientBoostingClassifier(n_estimators=100, learning_rate=0.1)
# configure the resampling method
cv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)
# evaluate the ensemble on the dataset using the resampling method
n_scores = cross_val_score(model, X, y, scoring=’accuracy’, cv=cv, n_jobs=-1)
# report ensemble performance
print(‘Mean Accuracy: %.3f (%.3f)’ % (mean(n_scores), std(n_scores)))
Mean Accuracy: 0.923 (0.099)
To run the provided code and evaluate the Gradient Boosting Classifier on the synthetic classification dataset, let use the following steps:
# example of evaluating a gradient boosting ensemble for classification
from numpy import mean
from numpy import std
from sklearn.datasets import make_classification
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import RepeatedStratifiedKFold
from sklearn.ensemble import GradientBoostingClassifier
# create the synthetic classification dataset
X, y = make_classification(random_state=1)
# configure the ensemble model
model = GradientBoostingClassifier(n_estimators=50)
# configure the resampling method
cv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)
# evaluate the ensemble on the dataset using the resampling method
n_scores = cross_val_score(model, X, y, scoring=’accuracy’, cv=cv, n_jobs=-1)
# report ensemble performance
print(‘Mean Accuracy: %.3f (%.3f)’ % (mean(n_scores), std(n_scores)))
Mean Accuracy: 0.930 (0.090)
Let evaluate the effect of using more decision trees in the ensemble or different learning rate values, by modifying the model definition.
model = GradientBoostingClassifier(n_estimators=100, learning_rate=0.1)
# example of evaluating a gradient boosting ensemble for classification
from numpy import mean
from numpy import std
from sklearn.datasets import make_classification
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import RepeatedStratifiedKFold
from sklearn.ensemble import GradientBoostingClassifier
# create the synthetic classification dataset
X, y = make_classification(random_state=1)
# configure the ensemble model
model = GradientBoostingClassifier(n_estimators=100, learning_rate=0.1)
# configure the resampling method
cv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)
# evaluate the ensemble on the dataset using the resampling method
n_scores = cross_val_score(model, X, y, scoring=’accuracy’, cv=cv, n_jobs=-1)
# report ensemble performance
print(‘Mean Accuracy: %.3f (%.3f)’ % (mean(n_scores), std(n_scores)))
Mean Accuracy: 0.923 (0.099)
Increasing the number of trees may lead to longer training times but can potentially improve accuracy
# example of evaluating a voting ensemble for classification
from numpy import mean
from numpy import std
from sklearn.datasets import make_classification
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import RepeatedStratifiedKFold
from sklearn.ensemble import VotingClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.linear_model import LogisticRegression
# create the synthetic classification dataset
X, y = make_classification(random_state=1)
# configure the models to use in the ensemble
models = [(‘lr’, LogisticRegression()), (‘nb’, GaussianNB())]
# configure the ensemble model
model = VotingClassifier(models, voting=’soft’)
# configure the resampling method
cv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)
# evaluate the ensemble on the dataset using the resampling method
n_scores = cross_val_score(model, X, y, scoring=’accuracy’, cv=cv, n_jobs=-1)
# report ensemble performance
print(‘Mean Accuracy: %.3f (%.3f)’ % (mean(n_scores), std(n_scores)))
Mean Accuracy: 0.960 (0.061)
example of changing the ensemble to use hard voting and adding a Support Vector Machine (SVM) classifier to the ensemble:
from sklearn.ensemble import VotingClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC # Import SVC
# create the synthetic classification dataset
X, y = make_classification(random_state=1)
# configure the models to use in the ensemble (including SVM)
models = [(‘lr’, LogisticRegression()), (‘nb’, GaussianNB()), (‘svm’, SVC())]
# configure the ensemble model (hard voting)
model = VotingClassifier(models, voting=’hard’)
# configure the resampling method
cv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)
# evaluate the ensemble on the dataset using the resampling method
n_scores = cross_val_score(model, X, y, scoring=’accuracy’, cv=cv, n_jobs=-1)
HOMEWORK 1.
Bagging: Machine Learning for Medical Diagnosis / Prognosis-Algorithm Rule Extraction for MEdical Diagnosis (REMED). For this type of domain, which requires accuracy in its predictions, as well as comprehensibility and reliability in the patterns obtained, symbolic classifiers are used (e.g. decision trees and rules), which is based on three stages: (i ) selection of the most relevant attributes; (ii) selection of the partitions that separate the examples of each class; and (iii) construction of the model that allows the examples to be classified; This is due to the fact that through them, intelligible models for human beings will be generated, capable of providing medical personnel with a new point of view that helps them make correct decisions in the difficult task of medical diagnosis / prognosis.
Boosting: Prediction of financial fragility for Colombian corporations through the application of classification trees and boosting techniques. Apply the techniques for the prediction of business bankruptcy: classification trees and boosting. It details the methodological procedure, the data purification and treatment criteria, as well as the model estimation process, to finally present the set of financial indicators that explain business bankruptcy to a greater extent: solvency and level of indebtedness; thus, as the most efficient prediction model, making use of criteria such as the classification rate and the metric based on the area under the ROC curve.
Stacking: Application of aggregate methods in the detection of outliers in meteorological time series. Application that allows adapting 3 non-aggregated classifiers (vector support regression, ARIMA, Bayesian networks) and 3 aggregated classifiers (stacking, bagging and AdaBoost) to 3 data sets of meteorological measurements (precipitation, maximum temperature and solar radiation). Compare the classifiers. In the first instance, the classifiers were averaged by performing multiple tests on each data set. Then, by means of a statistical test of hypothesis, the means obtained by the classifiers were compared to determine if the observed differences were significant. Finally, an analysis of the results was carried out, focused on comparing the performance of the aggregated classifiers against the performance of the best non-aggregated classifier in each data set. In general, it was found that it is possible to significantly improve performance by detecting point outliers in some uni-variable time series using aggregate methods. However, to achieve this improvement, conditions must be met that, although they vary depending on the method added, generally aim to improve the diversity of the base classifiers. When these conditions are not met, the aggregated methods did not have a significant difference in performance from the non-aggregated algorithm that performed best in the data set.
Well done!
Lesson 01: What Is Ensemble Learning?
Thank you for your great post,
In order not to make a long text, I just list the name of three applications of ensemble learning.
1. Medicine: neuroscience, Proterozoic, medical diagnosis, and …
2. Face recognition
3. Fraud detection
Task 2:
n_estimators=50 > Mean Accuracy: 0.953 (0.067)
n_estimators=70 > Mean Accuracy: 0.947 (0.072) –
n_estimators=100 > Mean Accuracy: 0.943 (0.080) –
n_estimators=130 > Mean Accuracy: 0.947 (0.072) +
n_estimators=140 > Mean Accuracy: 0.950 (0.067) +
n_estimators=150 > Mean Accuracy: 0.943 (0.080) –
n_estimators=190 > Mean Accuracy: 0.953 (0.067) +
n_estimators=200 > Mean Accuracy: 0.950 (0.067) –
n_estimators=300 > Mean Accuracy: 0.943 (0.080) –
n_estimators=350 > Mean Accuracy: 0.953 (0.067) +
: ‘ (
Well done.
Task 3:
n_estimators=50 , n_splits=10 > Mean Accuracy: 0.957 (0.067)
n_estimators=100 , n_splits=10 > Mean Accuracy: 0.960 (0.066) +
n_estimators=200 , n_splits=10 > Mean Accuracy: 0.957 (0.067) –
n_estimators=250 , n_splits=10 > Mean Accuracy: 0.960 (0.066) +
n_estimators=300 , n_splits=10 > Mean Accuracy: 0.957 (0.072) –
n_estimators=500 , n_splits=10 > Mean Accuracy: 0.957 (0.072)
n_estimators=999 , n_splits=10 > Mean Accuracy: 0.957 (0.072)
n_estimators=50 , n_splits=2 > Mean Accuracy: 0.963 (0.014)
n_estimators=50 , n_splits=5 > Mean Accuracy: 0.953 (0.046) –
n_estimators=50 , n_splits=10 > Mean Accuracy: 0.957 (0.067) +
n_estimators=50 , n_splits=15 > Mean Accuracy: 0.958 (0.084) +
n_estimators=50 , n_splits=20 > Mean Accuracy: 0.960 (0.080) +
n_estimators=50 , n_splits=30 > Mean Accuracy: 0.965 (0.102) ++
n_estimators=50 , n_splits=45 > Mean Accuracy: 0.963 (0.126) –
n_estimators=50 , n_splits=49 > Mean Accuracy: 0.959 (0.137) –
n_estimators=100 , n_splits=30 > Mean Accuracy: 0.961 (0.107)
n_estimators=250 , n_splits=30 > Mean Accuracy: 0.965 (0.102) +++
Well done!
Task 4: AdaBoost Ensemble
::: n_splits=10 :::
n_estimators=50 > Mean Accuracy: 0.947 (0.088)
n_estimators=70 > Mean Accuracy: 0.953 (0.088) +
n_estimators=100 > Mean Accuracy: 0.953 (0.088) .
n_estimators=130 > Mean Accuracy: 0.957 (0.088) +++
n_estimators=140 > Mean Accuracy: 0.957 (0.088) .
n_estimators=150 > Mean Accuracy: 0.957 (0.088) .
n_estimators=190 > Mean Accuracy: 0.957 (0.088) .
n_estimators=200 > Mean Accuracy: 0.957 (0.088) .
n_estimators=300 > Mean Accuracy: 0.953 (0.099) –
n_estimators=350 > Mean Accuracy: 0.953 (0.099) .
::: n_estimators=130 :::
n_splits=2 > Mean Accuracy: 0.950 (0.015) –
n_splits=8 > Mean Accuracy: 0.944 (0.088) –
n_splits=16 > Mean Accuracy: 0.957 (0.111) +
n_splits=20 > Mean Accuracy: 0.960 (0.088) +
n_splits=24 > Mean Accuracy: 0.963 (0.102) +
n_splits=28 > Mean Accuracy: 0.966 (0.095) +++
n_splits=30 > Mean Accuracy: 0.965 (0.102) –
n_splits=32 > Mean Accuracy: 0.957 (0.124) –
::: n_estimators=130 , n_splits=28 :::
n_repeats=1 > Mean Accuracy: 0.973 (0.077) +++
n_repeats=3 > Mean Accuracy: 0.966 (0.095) –
n_repeats=5 > Mean Accuracy: 0.963 (0.099) –
n_repeats=7 > Mean Accuracy: 0.963 (0.098) .
n_repeats=10 > Mean Accuracy: 0.962 (0.104) –
n_repeats=12 > Mean Accuracy: 0.962 (0.104) .
n_repeats=18 > Mean Accuracy: 0.959 (0.108) –
n_repeats=22 > Mean Accuracy: 0.958 (0.108) –
n_estimators=130 , n_splits=28, n_repeats=1 > Mean Accuracy: 0.973 (0.077)
B )
Well done!
Task 5:
::: n_splits=10 :::
n_estimators=50 > Mean Accuracy: 0.927 (0.100)
n_estimators=70 > Mean Accuracy: 0.927 (0.089) .
n_estimators=100 > Mean Accuracy: 0.927 (0.089) .
n_estimators=130 > Mean Accuracy: 0.923 (0.099) –
n_estimators=140 > Mean Accuracy: 0.923 (0.099) .
n_estimators=150 > Mean Accuracy: 0.923 (0.099) .
n_estimators=190 > Mean Accuracy: 0.923 (0.099) .
n_estimators=200 > Mean Accuracy: 0.927 (0.089) +++
n_estimators=220 > Mean Accuracy: 0.923 (0.099) –
n_estimators=250 > Mean Accuracy: 0.923 (0.099) .
n_estimators=270 > Mean Accuracy: 0.920 (0.098) –
n_estimators=300 > Mean Accuracy: 0.920 (0.098) .
n_estimators=350 > Mean Accuracy: 0.923 (0.099) +
n_estimators=360 > Mean Accuracy: 0.927 (0.089) +
n_estimators=370 > Mean Accuracy: 0.923 (0.099) – .
::: n_estimators=200 :::
n_splits=2 > Mean Accuracy: 0.933 (0.034)
n_splits=8 > Mean Accuracy: 0.934 (0.064) +
n_splits=16 > Mean Accuracy: 0.934 (0.099) .
n_splits=20 > Mean Accuracy: 0.927 (0.115) –
n_splits=24 > Mean Accuracy: 0.935 (0.126) +
n_splits=28 > Mean Accuracy: 0.947 (0.112) +++
n_splits=30 > Mean Accuracy: 0.941 (0.126) –
n_splits=32 > Mean Accuracy: 0.938 (0.140) –
::: n_estimators=200 , n_splits=28 :::
n_repeats=1 > Mean Accuracy: 0.946 (0.103)
n_repeats=3 > Mean Accuracy: 0.943 (0.116) –
n_repeats=5 > Mean Accuracy: 0.943 (0.114) .
n_repeats=7 > Mean Accuracy: 0.940 (0.120) –
n_repeats=10 > Mean Accuracy: 0.938 (0.128) –
n_repeats=12 > Mean Accuracy: 0.934 (0.132) –
n_repeats=18 > Mean Accuracy: 0.937 (0.131) +
n_repeats=22 > Mean Accuracy: 0.938 (0.131) +++
n_repeats=24 > Mean Accuracy: 0.934 (0.134) –
n_repeats=26 > Mean Accuracy: 0.936 (0.132) +
n_repeats=30 > Mean Accuracy: 0.936 (0.134) .
Well done.
Task 6:
GNB: GaussianNB
LR: LogisticRegression
RFC: RandomForestClassifier
SGD: SGDClassifier
models=LR, GNB | voting=soft > Mean Accuracy: 0.960 (0.061) +++
models=LR, GNB | voting=hard > Mean Accuracy: 0.957 (0.062) –
models=LR, GNB, RFC | voting=soft > Mean Accuracy: 0.960 (0.061) +++
models=LR, GNB, RFC | voting=hard > Mean Accuracy: 0.953 (0.067) –
models=SGD, LR, SVC | voting=hard > Mean Accuracy: 0.943 (0.067)
Great work!
Task 7:
DTC: DecisionTreeClassifier
KNC: KNeighborsClassifier
LR: LogisticRegression
LSVC: LinearSVC
RFC: RandomForestClassifier
::: final_estimator=LR :::
models=KNC, DTC > Mean Accuracy: 0.930 (0.090)
models=RFC, LSVR > Mean Accuracy: 0.910 (0.098)
models=KNC, RFC, LSVR > Mean Accuracy: 0.910 (0.098)
models= KNC, DTC, RFC, LSVR > Mean Accuracy: 0.943 (0.067)
::: final_estimator=SVC :::
models=KNC, DTC > Mean Accuracy: 0.933 (0.083)
models=RFC, LSVR > Mean Accuracy: 0.910 (0.098)
models=KNC, RFC, LSVR > Mean Accuracy: 0.913 (0.088)
models= KNC, DTC, RFC, LSVR > Mean Accuracy: 0.943 (0.072)
::: final_estimator=LinearSVC :::
models=KNC, DTC > Mean Accuracy: 0.923 (0.088)
models=RFC, LSVR > Mean Accuracy: 0.943 (0.067)
models=KNC, RFC, LSVR > Mean Accuracy: 0.947 (0.067) +++
models= KNC, DTC, RFC, LSVR > Mean Accuracy: 0.947 (0.067)+++
That is a great post! Thank you so much!
Well done.
An application of ensemble machine learning may be multi-class classification.
Well done.
Your Crash course always helps understanding concepts in very clear way. Thank you for your great work.
You’re welcome.
Some of the applications of ensemble learning are listed below
Detection of Lung abnormalities
Speech emotion recognition
Forcasting cryptocurrency time series
Hyperspectral Image Analysis
Thankyou for this valuable lecture sir
Thank you for your comment. Hope you enjoy other posts too!
The issue of ensemble is its robustness. Any comment on this: For instance, assume you have done your ensemble for a data from date 2015 to date 2016. You got your model setup and calibration obtained using the above data (2015-2016).
Now, with such fixed model setup/calibration, use the data from 2014 to 2015.
How good/bad is the outcome from 2014 to 2015 when compared to 2015-2016?
Any comment on this?
Hi Mecobio…I have not experimented with the data as you described. I would be eager to know what your findings would be from such an implementation.
Can AdaBoost Apply in Naive Bayes? where Naive bayes as base estimator?
Hi helmi…You may find the following resource of interest:
https://dergipark.org.tr/en/download/article-file/2013021
can I apply boosting to SMOTE ? which better to improve SMOTE boosting or bagging?
Hi maram…The following resource may be of interest to you:
https://machinelearningmastery.com/smote-oversampling-for-imbalanced-classification/
Applications of Ensemble Learning
Ensemble learning is a machine learning technique that combines the predictions from multiple individual models to produce a more accurate and robust prediction. Ensemble learning is a fairly common strategy in deep learning and has been applied to tackle a myriad of problems. Classification and Regression: Ensemble methods are widely used for both classification and regression tasks. Algorithms like Random Forest and Gradient Boosting create an ensemble of decision trees to make predictions. These ensembles often outperform individual decision trees by reducing overfitting and improving predictive accuracy.
1. Anomaly Detection
2. Disease detection
3. Remote Sensing
4. Fraud Detection
Ensemble techniques can be used to build a model that is more robust to noisy data and can better distinguish between normal and anomalous instances. One common approach is to use a combination of different anomaly detection algorithms as an ensemble to improve overall performance.
Day 2: Bagging Ensembles
# example of evaluating a bagging ensemble for classification
from numpy import mean
from numpy import std
from sklearn.datasets import make_classification
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import RepeatedStratifiedKFold
from sklearn.ensemble import BaggingClassifier
# create the synthetic classification dataset
X, y = make_classification(random_state=1)
# configure the ensemble model
model = BaggingClassifier(n_estimators=50)
# configure the resampling method
cv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)
# evaluate the ensemble on the dataset using the resampling method
n_scores = cross_val_score(model, X, y, scoring=’accuracy’, cv=cv, n_jobs=-1)
# report ensemble performance
print(‘Mean Accuracy: %.3f (%.3f)’ % (mean(n_scores), std(n_scores)))
Here the answer
Mean Accuracy: 0.950 (0.072)
A mean accuracy of 0.950 with a standard deviation of 0.072 suggests that the model or system being evaluated is performing quite well. An accuracy of 0.950 means that it is correctly classifying or predicting outcomes with high accuracy, and the low standard deviation indicates that the performance is consistent across different runs or samples.
Day 3: Random Forest Ensemble
# example of evaluating a random forest ensemble for classification
from numpy import mean
from numpy import std
from sklearn.datasets import make_classification
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import RepeatedStratifiedKFold
from sklearn.ensemble import RandomForestClassifier
# create the synthetic classification dataset
X, y = make_classification(random_state=1)
# configure the ensemble model
model = RandomForestClassifier(n_estimators=50)
# configure the resampling method
cv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)
# evaluate the ensemble on the dataset using the resampling method
n_scores = cross_val_score(model, X, y, scoring=’accuracy’, cv=cv, n_jobs=-1)
# report ensemble performance
print(‘Mean Accuracy: %.3f (%.3f)’ % (mean(n_scores), std(n_scores)))
# Additionally, we can evaluate the effect of changing the number of decision trees (n_estimators) and the number of features considered at each split point (max_features)
# Number of decision trees in the ensemble (I tryed different values)
n_estimators_values = [10, 50, 100, 200]
here is the results
n_estimators=10, max_features=auto
Mean Accuracy: 0.940 (0.076)
n_estimators=10, max_features=sqrt
Mean Accuracy: 0.940 (0.076)
n_estimators=10, max_features=log2
Mean Accuracy: 0.940 (0.076)
n_estimators=50, max_features=auto
Mean Accuracy: 0.950 (0.067)
n_estimators=50, max_features=sqrt
Mean Accuracy: 0.950 (0.067)
n_estimators=50, max_features=log2
Mean Accuracy: 0.950 (0.067)
n_estimators=100, max_features=auto
Mean Accuracy: 0.953 (0.067)
n_estimators=100, max_features=sqrt
Mean Accuracy: 0.953 (0.067)
n_estimators=100, max_features=log2
Mean Accuracy: 0.953 (0.067)
n_estimators=200, max_features=auto
Mean Accuracy: 0.957 (0.067)
n_estimators=200, max_features=sqrt
Mean Accuracy: 0.957 (0.067)
n_estimators=200, max_features=log2
Mean Accuracy: 0.957 (0.067)
As you can see from the results, there are some variations. As the ‘n’ value increases, the accuracy also increases some extent.
Day 4: AdaBoost Ensemble
# example of evaluating an AdaBoost ensemble for classification
from numpy import mean
from numpy import std
from sklearn.datasets import make_classification
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import RepeatedStratifiedKFold
from sklearn.ensemble import AdaBoostClassifier
# create the synthetic classification dataset
X, y = make_classification(random_state=1)
# configure the ensemble model
model = AdaBoostClassifier(n_estimators=50)
# configure the resampling method
cv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)
# evaluate the ensemble on the dataset using the resampling method
n_scores = cross_val_score(model, X, y, scoring=’accuracy’, cv=cv, n_jobs=-1)
# report ensemble performance
print(‘Mean Accuracy: %.3f (%.3f)’ % (mean(n_scores), std(n_scores)))
Mean Accuracy: 0.947 (0.088)
The mean accuracy of the AdaBoost ensemble with 50 decision trees on this synthetic dataset is approximately 0.94/0, with a standard deviation of around 0.088.
For bonus points, let’s experiment with different numbers of decision trees in the ensemble and change the base learner:
Changing the Number of Decision Trees:
We can try different values for n_estimators to see how it affects the performance. Let’s try with 100 decision trees:
# Change the number of decision trees to 100
model = AdaBoostClassifier(n_estimators=100)
n_scores = cross_val_score(model, X, y, scoring=’accuracy’, cv=cv, n_jobs=-1)
print(‘Mean Accuracy with 100 Decision Trees: %.3f (%.3f)’ % (mean(n_scores), std(n_scores)))
Mean Accuracy with 100 Decision Trees: 0.950 (0.089)
Changing the Base Learner:
AdaBoost allows you to change the base learner. Let’s try using a different base learner, such as a Decision Tree with a maximum depth of 2:
from sklearn.tree import DecisionTreeClassifier
# Change the base learner to a Decision Tree with max depth 2
base_estimator = DecisionTreeClassifier(max_depth=2)
model = AdaBoostClassifier(base_estimator=base_estimator, n_estimators=50)
n_scores = cross_val_score(model, X, y, scoring=’accuracy’, cv=cv, n_jobs=-1)
print(‘Mean Accuracy with Decision Tree Base Learner (Max Depth 2): %.3f (%.3f)’ % (mean(n_scores), std(n_scores)))
Mean Accuracy with Decision Tree Base Learner (Max Depth 2): 0.940 (0.076)
Day 5: Gradient Boosting Ensemble
To run the provided code and evaluate the Gradient Boosting Classifier on the synthetic classification dataset, let use the following steps:
# example of evaluating a gradient boosting ensemble for classification
from numpy import mean
from numpy import std
from sklearn.datasets import make_classification
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import RepeatedStratifiedKFold
from sklearn.ensemble import GradientBoostingClassifier
# create the synthetic classification dataset
X, y = make_classification(random_state=1)
# configure the ensemble model
model = GradientBoostingClassifier(n_estimators=50)
# configure the resampling method
cv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)
# evaluate the ensemble on the dataset using the resampling method
n_scores = cross_val_score(model, X, y, scoring=’accuracy’, cv=cv, n_jobs=-1)
# report ensemble performance
print(‘Mean Accuracy: %.3f (%.3f)’ % (mean(n_scores), std(n_scores)))
Mean Accuracy: 0.930 (0.090)
Let evaluate the effect of using more decision trees in the ensemble or different learning rate values, by modifying the model definition.
model = GradientBoostingClassifier(n_estimators=100, learning_rate=0.1)
# example of evaluating a gradient boosting ensemble for classification
from numpy import mean
from numpy import std
from sklearn.datasets import make_classification
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import RepeatedStratifiedKFold
from sklearn.ensemble import GradientBoostingClassifier
# create the synthetic classification dataset
X, y = make_classification(random_state=1)
# configure the ensemble model
model = GradientBoostingClassifier(n_estimators=100, learning_rate=0.1)
# configure the resampling method
cv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)
# evaluate the ensemble on the dataset using the resampling method
n_scores = cross_val_score(model, X, y, scoring=’accuracy’, cv=cv, n_jobs=-1)
# report ensemble performance
print(‘Mean Accuracy: %.3f (%.3f)’ % (mean(n_scores), std(n_scores)))
Mean Accuracy: 0.923 (0.099)
Day 5: Gradient Boosting Ensemble
To run the provided code and evaluate the Gradient Boosting Classifier on the synthetic classification dataset, let use the following steps:
# example of evaluating a gradient boosting ensemble for classification
from numpy import mean
from numpy import std
from sklearn.datasets import make_classification
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import RepeatedStratifiedKFold
from sklearn.ensemble import GradientBoostingClassifier
# create the synthetic classification dataset
X, y = make_classification(random_state=1)
# configure the ensemble model
model = GradientBoostingClassifier(n_estimators=50)
# configure the resampling method
cv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)
# evaluate the ensemble on the dataset using the resampling method
n_scores = cross_val_score(model, X, y, scoring=’accuracy’, cv=cv, n_jobs=-1)
# report ensemble performance
print(‘Mean Accuracy: %.3f (%.3f)’ % (mean(n_scores), std(n_scores)))
Mean Accuracy: 0.930 (0.090)
Let evaluate the effect of using more decision trees in the ensemble or different learning rate values, by modifying the model definition.
model = GradientBoostingClassifier(n_estimators=100, learning_rate=0.1)
# example of evaluating a gradient boosting ensemble for classification
from numpy import mean
from numpy import std
from sklearn.datasets import make_classification
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import RepeatedStratifiedKFold
from sklearn.ensemble import GradientBoostingClassifier
# create the synthetic classification dataset
X, y = make_classification(random_state=1)
# configure the ensemble model
model = GradientBoostingClassifier(n_estimators=100, learning_rate=0.1)
# configure the resampling method
cv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)
# evaluate the ensemble on the dataset using the resampling method
n_scores = cross_val_score(model, X, y, scoring=’accuracy’, cv=cv, n_jobs=-1)
# report ensemble performance
print(‘Mean Accuracy: %.3f (%.3f)’ % (mean(n_scores), std(n_scores)))
Mean Accuracy: 0.923 (0.099)
Increasing the number of trees may lead to longer training times but can potentially improve accuracy
Day 6: Voting Ensemble
# example of evaluating a voting ensemble for classification
from numpy import mean
from numpy import std
from sklearn.datasets import make_classification
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import RepeatedStratifiedKFold
from sklearn.ensemble import VotingClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.linear_model import LogisticRegression
# create the synthetic classification dataset
X, y = make_classification(random_state=1)
# configure the models to use in the ensemble
models = [(‘lr’, LogisticRegression()), (‘nb’, GaussianNB())]
# configure the ensemble model
model = VotingClassifier(models, voting=’soft’)
# configure the resampling method
cv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)
# evaluate the ensemble on the dataset using the resampling method
n_scores = cross_val_score(model, X, y, scoring=’accuracy’, cv=cv, n_jobs=-1)
# report ensemble performance
print(‘Mean Accuracy: %.3f (%.3f)’ % (mean(n_scores), std(n_scores)))
Mean Accuracy: 0.960 (0.061)
example of changing the ensemble to use hard voting and adding a Support Vector Machine (SVM) classifier to the ensemble:
from sklearn.ensemble import VotingClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC # Import SVC
# create the synthetic classification dataset
X, y = make_classification(random_state=1)
# configure the models to use in the ensemble (including SVM)
models = [(‘lr’, LogisticRegression()), (‘nb’, GaussianNB()), (‘svm’, SVC())]
# configure the ensemble model (hard voting)
model = VotingClassifier(models, voting=’hard’)
# configure the resampling method
cv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)
# evaluate the ensemble on the dataset using the resampling method
n_scores = cross_val_score(model, X, y, scoring=’accuracy’, cv=cv, n_jobs=-1)
# report ensemble performance
print(‘Mean Accuracy (Hard Voting): %.3f (%.3f)’ % (mean(n_scores), std(n_scores)))
Mean Accuracy (Hard Voting): 0.957 (0.062)