# Gradient Boosting with Scikit-Learn, XGBoost, LightGBM, and CatBoost

Gradient boosting is a powerful ensemble machine learning algorithm.

It’s popular for structured predictive modeling problems, such as classification and regression on tabular data, and is often the main algorithm or one of the main algorithms used in winning solutions to machine learning competitions, like those on Kaggle.

There are many implementations of gradient boosting available, including standard implementations in SciPy and efficient third-party libraries. Each uses a different interface and even different names for the algorithm.

In this tutorial, you will discover how to use gradient boosting models for classification and regression in Python.

Standardized code examples are provided for the four major implementations of gradient boosting in Python, ready for you to copy-paste and use in your own predictive modeling project.

After completing this tutorial, you will know:

• Gradient boosting is an ensemble algorithm that fits boosted decision trees by minimizing an error gradient.
• How to evaluate and use gradient boosting with scikit-learn, including gradient boosting machines and the histogram-based algorithm.
• How to evaluate and use third-party gradient boosting algorithms, including XGBoost, LightGBM, and CatBoost.

Kick-start your project with my new book Ensemble Learning Algorithms With Python, including step-by-step tutorials and the Python source code files for all examples.

Let’s get started. Gradient Boosting with Scikit-Learn, XGBoost, LightGBM, and CatBoost
Photo by John, some rights reserved.

## Tutorial Overview

This tutorial is divided into five parts; they are:

1. Library Installation
2. Test Problems
1. Library Installation
2. XGBoost for Classification
3. XGBoost for Regression
1. Library Installation
2. LightGBM for Classification
3. LightGBM for Regression
1. Library Installation
2. CatBoost for Classification
3. CatBoost for Regression

Gradient boosting refers to a class of ensemble machine learning algorithms that can be used for classification or regression predictive modeling problems.

Ensembles are constructed from decision tree models. Trees are added one at a time to the ensemble and fit to correct the prediction errors made by prior models. This is a type of ensemble machine learning model referred to as boosting.

Models are fit using any arbitrary differentiable loss function and gradient descent optimization algorithm. This gives the technique its name, “gradient boosting,” as the loss gradient is minimized as the model is fit, much like a neural network.

Gradient boosting is an effective machine learning algorithm and is often the main, or one of the main, algorithms used to win machine learning competitions (like Kaggle) on tabular and similar structured datasets.

Note: We will not be going into the theory behind how the gradient boosting algorithm works in this tutorial.

For more on the gradient boosting algorithm, see the tutorial:

The algorithm provides hyperparameters that should, and perhaps must, be tuned for a specific dataset. Although there are many hyperparameters to tune, perhaps the most important are as follows:

• The number of trees or estimators in the model.
• The learning rate of the model.
• The row and column sampling rate for stochastic models.
• The maximum tree depth.
• The minimum tree weight.
• The regularization terms alpha and lambda.

Note: We will not be exploring how to configure or tune the configuration of gradient boosting algorithms in this tutorial.

For more on tuning the hyperparameters of gradient boosting algorithms, see the tutorial:

There are many implementations of the gradient boosting algorithm available in Python. Perhaps the most used implementation is the version provided with the scikit-learn library.

Additional third-party libraries are available that provide computationally efficient alternate implementations of the algorithm that often achieve better results in practice. Examples include the XGBoost library, the LightGBM library, and the CatBoost library.

Do you have a different favorite gradient boosting implementation?
Let me know in the comments below.

When using gradient boosting on your predictive modeling project, you may want to test each implementation of the algorithm.

This tutorial provides examples of each implementation of the gradient boosting algorithm on classification and regression predictive modeling problems that you can copy-paste into your project.

Let’s take a look at each in turn.

Note: We are not comparing the performance of the algorithms in this tutorial. Instead, we are providing code examples to demonstrate how to use each different implementation. As such, we are using synthetic test datasets to demonstrate evaluating and making a prediction with each implementation.

This tutorial assumes you have Python and SciPy installed. If you need help, see the tutorial:

### Want to Get Started With Ensemble Learning?

Take my free 7-day email crash course now (with sample code).

Click to sign-up and also get a free PDF Ebook version of the course.

In this section, we will review how to use the gradient boosting algorithm implementation in the scikit-learn library.

### Library Installation

First, let’s install the library.

Don’t skip this step as you will need to ensure you have the latest version installed.

You can install the scikit-learn library using the pip Python installer, as follows:

Next, let’s confirm that the library is installed and you are using a modern version.

Run the following script to print the library version number.

Running the example, you should see the following version number or higher.

### Test Problems

We will demonstrate the gradient boosting algorithm for classification and regression.

As such, we will use synthetic test problems from the scikit-learn library.

#### Classification Dataset

We will use the make_classification() function to create a test binary classification dataset.

The dataset will have 1,000 examples, with 10 input features, five of which will be informative and the remaining five that will be redundant. We will fix the random number seed to ensure we get the same examples each time the code is run.

An example of creating and summarizing the dataset is listed below.

Running the example creates the dataset and confirms the expected number of samples and features.

#### Regression Dataset

We will use the make_regression() function to create a test regression dataset.

Like the classification dataset, the regression dataset will have 1,000 examples, with 10 input features, five of which will be informative and the remaining five that will be redundant.

Running the example creates the dataset and confirms the expected number of samples and features.

Next, let’s look at how we can develop gradient boosting models in scikit-learn.

The scikit-learn library provides the GBM algorithm for regression and classification via the GradientBoostingClassifier and GradientBoostingRegressor classes.

Let’s take a closer look at each in turn.

#### Gradient Boosting Machine for Classification

The example below first evaluates a GradientBoostingClassifier on the test problem using repeated k-fold cross-validation and reports the mean accuracy. Then a single model is fit on all available data and a single prediction is made.

The complete example is listed below.

Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.

Running the example first reports the evaluation of the model using repeated k-fold cross-validation, then the result of making a single prediction with a model fit on the entire dataset.

#### Gradient Boosting Machine for Regression

The example below first evaluates a GradientBoostingRegressor on the test problem using repeated k-fold cross-validation and reports the mean absolute error. Then a single model is fit on all available data and a single prediction is made.

The complete example is listed below.

Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.

Running the example first reports the evaluation of the model using repeated k-fold cross-validation, then the result of making a single prediction with a model fit on the entire dataset.

The scikit-learn library provides an alternate implementation of the gradient boosting algorithm, referred to as histogram-based gradient boosting.

This is an alternate approach to implement gradient tree boosting inspired by the LightGBM library (described more later). This implementation is provided via the HistGradientBoostingClassifier and HistGradientBoostingRegressor classes.

The primary benefit of the histogram-based approach to gradient boosting is speed. These implementations are designed to be much faster to fit on training data.

At the time of writing, this is an experimental implementation and requires that you add the following line to your code to enable access to these classes.

Without this line, you will see an error like:

or

Let’s take a close look at how to use this implementation.

#### Histogram-Based Gradient Boosting Machine for Classification

The example below first evaluates a HistGradientBoostingClassifier on the test problem using repeated k-fold cross-validation and reports the mean accuracy. Then a single model is fit on all available data and a single prediction is made.

The complete example is listed below.

Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.

Running the example first reports the evaluation of the model using repeated k-fold cross-validation, then the result of making a single prediction with a model fit on the entire dataset.

#### Histogram-Based Gradient Boosting Machine for Regression

The example below first evaluates a HistGradientBoostingRegressor on the test problem using repeated k-fold cross-validation and reports the mean absolute error. Then a single model is fit on all available data and a single prediction is made.

The complete example is listed below.

Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.

Running the example first reports the evaluation of the model using repeated k-fold cross-validation, then the result of making a single prediction with a model fit on the entire dataset.

XGBoost, which is short for “Extreme Gradient Boosting,” is a library that provides an efficient implementation of the gradient boosting algorithm.

The main benefit of the XGBoost implementation is computational efficiency and often better model performance.

For more on the benefits and capability of XGBoost, see the tutorial:

### Library Installation

You can install the XGBoost library using the pip Python installer, as follows:

Next, let’s confirm that the library is installed and you are using a modern version.

Run the following script to print the library version number.

Running the example, you should see the following version number or higher.

The XGBoost library provides wrapper classes so that the efficient algorithm implementation can be used with the scikit-learn library, specifically via the XGBClassifier and XGBregressor classes.

Let’s take a closer look at each in turn.

### XGBoost for Classification

The example below first evaluates an XGBClassifier on the test problem using repeated k-fold cross-validation and reports the mean accuracy. Then a single model is fit on all available data and a single prediction is made.

The complete example is listed below.

Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.

Running the example first reports the evaluation of the model using repeated k-fold cross-validation, then the result of making a single prediction with a model fit on the entire dataset.

### XGBoost for Regression

The example below first evaluates an XGBRegressor on the test problem using repeated k-fold cross-validation and reports the mean absolute error. Then a single model is fit on all available data and a single prediction is made.

The complete example is listed below.

Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.

Running the example first reports the evaluation of the model using repeated k-fold cross-validation, then the result of making a single prediction with a model fit on the entire dataset.

LightGBM, short for Light Gradient Boosted Machine, is a library developed at Microsoft that provides an efficient implementation of the gradient boosting algorithm.

The primary benefit of the LightGBM is the changes to the training algorithm that make the process dramatically faster, and in many cases, result in a more effective model.

For more technical details on the LightGBM algorithm, see the paper:

### Library Installation

You can install the LightGBM library using the pip Python installer, as follows:

Next, let’s confirm that the library is installed and you are using a modern version.

Run the following script to print the library version number.

Running the example, you should see the following version number or higher.

The LightGBM library provides wrapper classes so that the efficient algorithm implementation can be used with the scikit-learn library, specifically via the LGBMClassifier and LGBMRegressor classes.

Let’s take a closer look at each in turn.

### LightGBM for Classification

The example below first evaluates an LGBMClassifier on the test problem using repeated k-fold cross-validation and reports the mean accuracy. Then a single model is fit on all available data and a single prediction is made.

The complete example is listed below.

Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.

Running the example first reports the evaluation of the model using repeated k-fold cross-validation, then the result of making a single prediction with a model fit on the entire dataset.

### LightGBM for Regression

The example below first evaluates an LGBMRegressor on the test problem using repeated k-fold cross-validation and reports the mean absolute error. Then a single model is fit on all available data and a single prediction is made.

The complete example is listed below.

Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.

Running the example first reports the evaluation of the model using repeated k-fold cross-validation, then the result of making a single prediction with a model fit on the entire dataset.

CatBoost is a third-party library developed at Yandex that provides an efficient implementation of the gradient boosting algorithm.

The primary benefit of the CatBoost (in addition to computational speed improvements) is support for categorical input variables. This gives the library its name CatBoost for “Category Gradient Boosting.”

For more technical details on the CatBoost algorithm, see the paper:

### Library Installation

You can install the CatBoost library using the pip Python installer, as follows:

Next, let’s confirm that the library is installed and you are using a modern version.

Run the following script to print the library version number.

Running the example, you should see the following version number or higher.

The CatBoost library provides wrapper classes so that the efficient algorithm implementation can be used with the scikit-learn library, specifically via the CatBoostClassifier and CatBoostRegressor classes.

Let’s take a closer look at each in turn.

### CatBoost for Classification

The example below first evaluates a CatBoostClassifier on the test problem using repeated k-fold cross-validation and reports the mean accuracy. Then a single model is fit on all available data and a single prediction is made.

The complete example is listed below.

Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.

Running the example first reports the evaluation of the model using repeated k-fold cross-validation, then the result of making a single prediction with a model fit on the entire dataset.

### CatBoost for Regression

The example below first evaluates a CatBoostRegressor on the test problem using repeated k-fold cross-validation and reports the mean absolute error. Then a single model is fit on all available data and a single prediction is made.

The complete example is listed below.

Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.

Running the example first reports the evaluation of the model using repeated k-fold cross-validation, then the result of making a single prediction with a model fit on the entire dataset.

This section provides more resources on the topic if you are looking to go deeper.

## Summary

In this tutorial, you discovered how to use gradient boosting models for classification and regression in Python.

Specifically, you learned:

• Gradient boosting is an ensemble algorithm that fits boosted decision trees by minimizing an error gradient.
• How to evaluate and use gradient boosting with scikit-learn, including gradient boosting machines and the histogram-based algorithm.
• How to evaluate and use third-party gradient boosting algorithms including XGBoost, LightGBM and CatBoost.

Do you have any questions?

## Get a Handle on Modern Ensemble Learning! #### Improve Your Predictions in Minutes

...with just a few lines of python code

Discover how in my new Ebook:
Ensemble Learning Algorithms With Python

It provides self-study tutorials with full working code on:
Stacking, Voting, Boosting, Bagging, Blending, Super Learner, and much more...

### 59 Responses to Gradient Boosting with Scikit-Learn, XGBoost, LightGBM, and CatBoost

1. jaehyeong April 2, 2020 at 11:49 am #

Thank you for your good writing!

• Jason Brownlee April 2, 2020 at 1:30 pm #

I’m happy it helps.

2. Santiago April 3, 2020 at 6:41 am #

Excellent post, thanks a lot!

• Jason Brownlee April 3, 2020 at 6:59 am #

Thanks!

3. TOUNSI Youssef April 3, 2020 at 10:52 am #

Well organized, keep up the good work!

• Jason Brownlee April 3, 2020 at 1:16 pm #

Thanks!

4. Ben April 4, 2020 at 1:24 am #

Hi Jason, all of my work is time series regression with utility metering data. And I always just look at RSME because its in the units that make sense to me. Basically when using from sklearn.metrics import mean_squared_error I just take the math.sqrt(mse) I notice that you use mean absolute error in the code above… Is there anything wrong with what I am doing to achieve best model results only viewing RSME?

• Jason Brownlee April 4, 2020 at 6:20 am #

No problem! I used to use RMSE all the time myself.

Recently I prefer MAE – can’t say why. Perhaps taste. Perhaps because no sqrt step is required.

5. Svetlana April 26, 2020 at 2:13 am #

The best article. Thanks for such a mindblowing article.

When you use RepeatedStratifiedKFold mostly the accuracy is calculated to know the best performing model. What if one whats to calculate the parameters like recall, precision, sensitivity, specificity. Then how do we calculate it for each of these repeated folds and also the final mean of all of them like how accuracy is calculated?

Thanks!

• Jason Brownlee April 26, 2020 at 6:15 am #

You can specify any metric you like for stratified k-fold cross-validation.

6. Edivaldo April 27, 2020 at 2:18 am #

Excelent.

• Jason Brownlee April 27, 2020 at 5:37 am #

Thanks!

7. Svetlana April 27, 2020 at 2:24 am #

Do you have and example for the same? Or can you show how to do that?

• Jason Brownlee April 27, 2020 at 5:37 am #

Same what?

8. Fernando May 3, 2020 at 8:03 pm #

Thanks! Good tutorials

• Jason Brownlee May 4, 2020 at 6:19 am #

Thanks! You’re welcome.

9. Engr. Wasiu Ajao May 9, 2020 at 12:50 am #

What a job! Thank you Jason.

• Jason Brownlee May 9, 2020 at 6:16 am #

Thanks!

10. Jie May 11, 2020 at 8:31 pm #

Hi Jason, I have a question regarding the generating the dataset.

So if you set the informative to be 5, does it mean that the classifier will detect these 5 attributes during the feature importance at high scores while as the other 5 redundant will be calculated as low?

If you set informative at 5 and redundant at 2, then the other 3 attributes will be random important?

Thanks

• Jason Brownlee May 12, 2020 at 6:44 am #

Not really.

We change informative/redundant to make the problem easier/harder – at least in the general sense.

Trees are great at sifting out redundant features automatically.

11. Fábio Albuquerque August 26, 2020 at 7:14 am #

Any of Gradient Boosting Methods can work with multi-dimensional arrays for target values (y)?

12. Tony September 1, 2020 at 12:15 am #

Hi Jason,
I am confused how a light gradient boosting model works, since in the API they use “num_round = 10
bst = lgb.train(param, train_data, num_round, valid_sets=[validation_data])” to fit the model with the training data.

Why is it that the .fit method works in your code? Is it just because you imported the LGBMRegressor model?

Thanks,

• Jason Brownlee September 1, 2020 at 6:34 am #

Yes, I recommend using the scikit-learn wrapper classes – it makes using the model much simpler.

13. Atis September 14, 2020 at 11:24 pm #

Hello Jason – I am not quite happy with the regression results of my LSTM neural network. In particular, the far ends of the y-distribution are not predicted very well.

I am wondering if I could use the principle of gradient boosting to train successive networks to correct the remaining error the previous ones have made.

What do you think of this idea? What would the risks be?

• Jason Brownlee September 15, 2020 at 5:26 am #

Try it!

14. Paolina October 22, 2020 at 3:03 am #

hello
I have created used XGBoost and I have making tuning parameters by search grid (even I know that Bayesian optimization is better but I was obliged to use search grid)

The question is I must answer this question:(robustness of the system is not clear, you have to specify it) But I have no idea how to estimate robustness and what should I read to answer it

• Jason Brownlee October 22, 2020 at 6:47 am #

One estimate of model robustness is the variance or standard deviation of the performance metric from repeated evaluation on the same test harness.

15. Ron February 15, 2021 at 6:33 am #

Jason,

I did not find any reference to your article. This is the second one I know of. He seems to have omitted Histogram Based Gradient Boosting in here.

Ron

• Jason Brownlee February 15, 2021 at 8:11 am #

Thanks for letting me know, very disappointing that people rip me off so blatantly.

16. Sam February 16, 2021 at 6:27 am #

I also had to comment on his post because it is really shameful. He could cite you and add his own comments but recycling and adding something to the code. But really, that was a sad copy-paste.

• Jason Brownlee February 16, 2021 at 8:01 am #

Thank you!!!

Yes, people have no shame and I see it more and more – direct copy-paste of my tutorials on medium or similar.

I believe google can detect the duplicate content and punishes the copy cats with low rankings.

17. SSS February 19, 2021 at 9:54 am #

• Jason Brownlee February 19, 2021 at 10:51 am #

Great suggestion, thanks!

18. MS March 7, 2021 at 3:59 am #

Hi
Is catboost familiar with scikitlearn api?

• Jason Brownlee March 7, 2021 at 5:14 am #

Catboost can be used via the scikit-learn wrapper class, as in the above example.

19. MS March 9, 2021 at 3:49 pm #

thanks

• Jason Brownlee March 10, 2021 at 4:37 am #

You’re welcome.

20. Nick March 13, 2021 at 11:11 am #

Hey Jason, just wondering how you can incorporate early stopping with catboost and lightgbm? I’m getting an error which is asking for a validation set to be generated. I’m wondering if cross_val_score isn’t compatible with early stopping.
Cheers!

21. Saima April 26, 2021 at 4:00 am #

how to find precision, recall,f1 scores from here?

22. Mehdi May 3, 2021 at 1:21 pm #

• Jason Brownlee May 4, 2021 at 6:43 am #

You’re welcome!

23. jtm January 7, 2022 at 10:34 pm #

Thanks for the concise post. However, when trying to reproduce the classification results here, either I get an error from joblib or the run hangs forever. Any ideas on this issue? Thanks!

• James Carmichael January 8, 2022 at 11:00 am #

Hi JTM…Are you trying to run a specific code listing from our materials? If so, please indicate the specific code listing and provide the exact error message.

Regards,

24. Maya April 6, 2022 at 8:04 pm #

Thanks! I wanted to ask when you are reporting the MAE values for regression, the bracketed values represent the cross validation? If yes, what does it mean when the value is more than 1? Ideally the max value should be 1?

Also, when I tested a model based that was made using gbr = GradientBoostingRegressor(parameters), the function gbr.score(X_test, y_test) gave a negative value like -1.08 this means that the model is a blunder? What do these negative values mean?

• James Carmichael April 7, 2022 at 9:43 am #

Hi Maya…the following resource may help add clarity:

https://machinelearningmastery.mystagingwebsite.com/regression-metrics-for-machine-learning/

Some model evaluation metrics such as mean squared error (MSE) are negative when calculated in scikit-learn.

This is confusing, because error scores like MSE cannot actually be negative, with the smallest value being zero or no error.

The scikit-learn library has a unified model scoring system where it assumes that all model scores are maximized. In order this system to work with scores that are minimized, like MSE and other measures of error, the sores that are minimized are inverted by making them negative.

This can also be seen in the specification of the metric, e.g. ‘neg‘ is used in the name of the metric ‘neg_mean_squared_error‘.

When interpreting the negative error scores, you can ignore the sign and use them directly.

Model evaluation: quantifying the quality of predictions

25. Faiy V. May 21, 2022 at 1:10 am #

Hi !

Can we use the same code for LightGBM Ranker and XGBoost Ranker by changing only the model fit and some of the params?

Sofia

• James Carmichael May 21, 2022 at 11:46 pm #

Hi Faiy V…There would be a great deal of reuse of code. Have you implemented models for both and compared the results? Let us know what you find!

• Faiy V. May 27, 2022 at 11:14 pm #

Yes I tried. requires to define a group for ranking! I hope will work!

26. Faiy V. May 27, 2022 at 11:15 pm #

Yes I tried. requires to define a group for ranking! I hope will work!

27. Sepideh December 13, 2022 at 6:53 am #

There’s a question I’d like to ask.
You first made a model for each algorithm, implemented K-fold cross-validation on it, then made another model and predicted the targets with it.
In other words, you made 2 different models for each algorithm and cross-validate only one of them, and use the other one for predicting.
May I ask you why did you do that?
Could you please explain why you did not create only one model for cross-validation and prediction?

28. • 