XGBoost is a popular implementation of Gradient Boosting because of its speed and performance.

Internally, XGBoost models represent all problems as a regression predictive modeling problem that only takes numerical values as input. If your data is in a different form, it must be prepared into the expected format.

In this post, you will discover how to prepare your data for using with gradient boosting with the XGBoost library in Python.

After reading this post you will know:

- How to encode string output variables for classification.
- How to prepare categorical input variables using one hot encoding.
- How to automatically handle missing data with XGBoost.

Let’s get started.

**Update Sept/2016**: I updated a few small typos in the impute example.**Update Jan/2017**: Updated to reflect changes in scikit-learn API version 0.18.1.**Update Jan/2017**: Updated breast cancer example to converted input data to strings.

### Need help with XGBoost in Python?

Take my free 7-day email course and discover configuration, tuning and more (with sample code).

Click to sign-up now and also get a free PDF Ebook version of the course.

## Label Encode String Class Values

The iris flowers classification problem is an example of a problem that has a string class value.

This is a prediction problem where given measurements of iris flowers in centimeters, the task is to predict to which species a given flower belongs.

Below is a sample of the raw dataset. You can learn more about this dataset and download the raw data in CSV format from the UCI Machine Learning Repository.

1 2 3 4 5 |
5.1,3.5,1.4,0.2,Iris-setosa 4.9,3.0,1.4,0.2,Iris-setosa 4.7,3.2,1.3,0.2,Iris-setosa 4.6,3.1,1.5,0.2,Iris-setosa 5.0,3.6,1.4,0.2,Iris-setosa |

XGBoost cannot model this problem as-is as it requires that the output variables be numeric.

We can easily convert the string values to integer values using the LabelEncoder. The three class values (Iris-setosa, Iris-versicolor, Iris-virginica) are mapped to the integer values (0, 1, 2).

1 2 3 4 |
# encode string class values as integers label_encoder = LabelEncoder() label_encoder = label_encoder.fit(Y) label_encoded_y = label_encoder.transform(Y) |

We save the label encoder as a separate object so that we can transform both the training and later the test and validation datasets using the same encoding scheme.

Below is a complete example demonstrating how to load the iris dataset. Notice that Pandas is used to load the data in order to handle the string class values.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 |
# multiclass classification import pandas import xgboost from sklearn import model_selection from sklearn.metrics import accuracy_score from sklearn.preprocessing import LabelEncoder # load data data = pandas.read_csv('iris.csv', header=None) dataset = data.values # split data into X and y X = dataset[:,0:4] Y = dataset[:,4] # encode string class values as integers label_encoder = LabelEncoder() label_encoder = label_encoder.fit(Y) label_encoded_y = label_encoder.transform(Y) seed = 7 test_size = 0.33 X_train, X_test, y_train, y_test = cross_validation.train_test_split(X, label_encoded_y, test_size=test_size, random_state=seed) # fit model no training data model = xgboost.XGBClassifier() model.fit(X_train, y_train) print(model) # make predictions for test data y_pred = model.predict(X_test) predictions = [round(value) for value in y_pred] # evaluate predictions accuracy = accuracy_score(y_test, predictions) print("Accuracy: %.2f%%" % (accuracy * 100.0)) |

Running the example produces the following output:

1 2 3 4 5 6 |
XGBClassifier(base_score=0.5, colsample_bylevel=1, colsample_bytree=1, gamma=0, learning_rate=0.1, max_delta_step=0, max_depth=3, min_child_weight=1, missing=None, n_estimators=100, nthread=-1, objective='multi:softprob', reg_alpha=0, reg_lambda=1, scale_pos_weight=1, seed=0, silent=True, subsample=1) Accuracy: 92.00% |

Notice how the XGBoost model is configured to automatically model the multiclass classification problem using the **multi:softprob** objective, a variation on the softmax loss function to model class probabilities. This suggests that internally, that the output class is converted into a one hot type encoding automatically.

## One Hot Encode Categorical Data

Some datasets only contain categorical data, for example the breast cancer dataset.

This dataset describes the technical details of breast cancer biopsies and the prediction task is to predict whether or not the patient has a recurrence of cancer, or not.

Below is a sample of the raw dataset. You can learn more about this dataset at the UCI Machine Learning Repository and download it in CSV format from mldata.org.

1 2 3 4 5 |
'40-49','premeno','15-19','0-2','yes','3','right','left_up','no','recurrence-events' '50-59','ge40','15-19','0-2','no','1','right','central','no','no-recurrence-events' '50-59','ge40','35-39','0-2','no','2','left','left_low','no','recurrence-events' '40-49','premeno','35-39','0-2','yes','3','right','left_low','yes','no-recurrence-events' '40-49','premeno','30-34','3-5','yes','2','left','right_up','no','recurrence-events' |

We can see that all 9 input variables are categorical and described in string format. The problem is a binary classification prediction problem and the output class values are also described in string format.

We can reuse the same approach from the previous section and convert the string class values to integer values to model the prediction using the LabelEncoder. For example:

1 2 3 4 |
# encode string class values as integers label_encoder = LabelEncoder() label_encoder = label_encoder.fit(Y) label_encoded_y = label_encoder.transform(Y) |

We can use this same approach on each input feature in X, but this is only a starting point.

1 2 3 4 5 6 7 8 |
# encode string input values as integers features = [] for i in range(0, X.shape[1]): label_encoder = LabelEncoder() feature = label_encoder.fit_transform(X[:,i]) features.append(feature) encoded_x = numpy.array(features) encoded_x = encoded_x.reshape(X.shape[0], X.shape[1]) |

XGBoost may assume that encoded integer values for each input variable have an ordinal relationship. For example that ‘left-up’ encoded as 0 and ‘left-low’ encoded as 1 for the breast-quad variable have a meaningful relationship as integers. In this case, this assumption is untrue.

Instead, we must map these integer values onto new binary variables, one new variable for each categorical value.

For example, the breast-quad variable has the values:

1 2 3 4 5 |
left-up left-low right-up right-low central |

We can model this as 5 binary variables as follows:

1 2 3 4 5 6 |
left-up, left-low, right-up, right-low, central 1,0,0,0,0 0,1,0,0,0 0,0,1,0,0 0,0,0,1,0 0,0,0,0,1 |

This is called one hot encoding. We can one hot encode all of the categorical input variables using the OneHotEncoder class in scikit-learn.

We can one hot encode each feature after we have label encoded it. First we must transform the feature array into a 2-dimensional NumPy array where each integer value is a feature vector with a length 1.

1 |
feature = feature.reshape(X.shape[0], 1) |

We can then create the OneHotEncoder and encode the feature array.

1 2 |
onehot_encoder = OneHotEncoder(sparse=False) feature = onehot_encoder.fit_transform(feature) |

Finally, we can build up the input dataset by concatenating the one hot encoded features, one by one, adding them on as new columns (axis=2). We end up with an input vector comprised of 43 binary input variables.

1 2 3 4 5 6 7 8 9 10 11 12 13 |
# encode string input values as integers encoded_x = None for i in range(0, X.shape[1]): label_encoder = LabelEncoder() feature = label_encoder.fit_transform(X[:,i]) feature = feature.reshape(X.shape[0], 1) onehot_encoder = OneHotEncoder(sparse=False) feature = onehot_encoder.fit_transform(feature) if encoded_x is None: encoded_x = feature else: encoded_x = numpy.concatenate((encoded_x, feature), axis=1) print("X shape: : ", encoded_x.shape) |

Ideally, we may experiment with not one hot encode some of input attributes as we could encode them with an explicit ordinal relationship, for example the first column age with values like ’40-49′ and ’50-59′. This is left as an exercise, if you are interested in extending this example.

Below is the complete example with label and one hot encoded input variables and label encoded output variable.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 |
# binary classification, breast cancer dataset, label and one hot encoded import numpy from pandas import read_csv from xgboost import XGBClassifier from sklearn.model_selection import train_test_split from sklearn.metrics import accuracy_score from sklearn.preprocessing import LabelEncoder from sklearn.preprocessing import OneHotEncoder # load data data = read_csv('datasets-uci-breast-cancer.csv', header=None) dataset = data.values # split data into X and y X = dataset[:,0:9] X = X.astype(str) Y = dataset[:,9] # encode string input values as integers encoded_x = None for i in range(0, X.shape[1]): label_encoder = LabelEncoder() feature = label_encoder.fit_transform(X[:,i]) feature = feature.reshape(X.shape[0], 1) onehot_encoder = OneHotEncoder(sparse=False) feature = onehot_encoder.fit_transform(feature) if encoded_x is None: encoded_x = feature else: encoded_x = numpy.concatenate((encoded_x, feature), axis=1) print("X shape: : ", encoded_x.shape) # encode string class values as integers label_encoder = LabelEncoder() label_encoder = label_encoder.fit(Y) label_encoded_y = label_encoder.transform(Y) # split data into train and test sets seed = 7 test_size = 0.33 X_train, X_test, y_train, y_test = train_test_split(encoded_x, label_encoded_y, test_size=test_size, random_state=seed) # fit model no training data model = XGBClassifier() model.fit(X_train, y_train) print(model) # make predictions for test data y_pred = model.predict(X_test) predictions = [round(value) for value in y_pred] # evaluate predictions accuracy = accuracy_score(y_test, predictions) print("Accuracy: %.2f%%" % (accuracy * 100.0)) |

Running this example we get the following output:

1 2 3 4 5 6 7 |
('X shape: : ', (285, 43)) XGBClassifier(base_score=0.5, colsample_bylevel=1, colsample_bytree=1, gamma=0, learning_rate=0.1, max_delta_step=0, max_depth=3, min_child_weight=1, missing=None, n_estimators=100, nthread=-1, objective='binary:logistic', reg_alpha=0, reg_lambda=1, scale_pos_weight=1, seed=0, silent=True, subsample=1) Accuracy: 71.58% |

Again we can see that the XGBoost framework chose the ‘**binary:logistic**‘ objective automatically, the right objective for this binary classification problem.

## Support for Missing Data

XGBoost can automatically learn how to best handle missing data.

In fact, XGBoost was designed to work with sparse data, like the one hot encoded data from the previous section, and missing data is handled the same way that sparse or zero values are handled, by minimizing the loss function.

For more information on the technical details for how missing values are handled in XGBoost, see Section 3.4 “*Sparsity-aware Split Finding*” in the paper XGBoost: A Scalable Tree Boosting System.

The Horse Colic dataset is a good example to demonstrate this capability as it contains a large percentage of missing data, approximately 30%.

You can learn more about the Horse Colic dataset and download the raw data file from the UCI Machine Learning repository.

The values are separated by whitespace and we can easily load it using the Pandas function read_csv.

1 |
dataframe = read_csv("horse-colic.csv", delim_whitespace=True, header=None) |

Once loaded, we can see that the missing data is marked with a question mark character (‘?’). We can change these missing values to the sparse value expected by XGBoost which is the value zero (0).

1 2 |
# set missing values to 0 X[X == '?'] = 0 |

Because the missing data was marked as strings, those columns with missing data were all loaded as string data types. We can now convert the entire set of input data to numerical values.

1 2 |
# convert to numeric X = X.astype('float32') |

Finally, this is a binary classification problem although the class values are marked with the integers 1 and 2. We model binary classification problems in XGBoost as logistic 0 and 1 values. We can easily convert the Y dataset to 0 and 1 integers using the LabelEncoder, as we did in the iris flowers example.

1 2 3 4 |
# encode Y class values as integers label_encoder = LabelEncoder() label_encoder = label_encoder.fit(Y) label_encoded_y = label_encoder.transform(Y) |

The full code listing is provided below for completeness.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 |
# binary classification, missing data from pandas import read_csv from xgboost import XGBClassifier from sklearn.model_selection import train_test_split from sklearn.metrics import accuracy_score from sklearn.preprocessing import LabelEncoder # load data dataframe = read_csv("horse-colic.csv", delim_whitespace=True, header=None) dataset = dataframe.values # split data into X and y X = dataset[:,0:27] Y = dataset[:,27] # set missing values to 0 X[X == '?'] = 0 # convert to numeric X = X.astype('float32') # encode Y class values as integers label_encoder = LabelEncoder() label_encoder = label_encoder.fit(Y) label_encoded_y = label_encoder.transform(Y) # split data into train and test sets seed = 7 test_size = 0.33 X_train, X_test, y_train, y_test = train_test_split(X, label_encoded_y, test_size=test_size, random_state=seed) # fit model no training data model = XGBClassifier() model.fit(X_train, y_train) print(model) # make predictions for test data y_pred = model.predict(X_test) predictions = [round(value) for value in y_pred] # evaluate predictions accuracy = accuracy_score(y_test, predictions) print("Accuracy: %.2f%%" % (accuracy * 100.0)) |

Running this example produces the following output.

1 2 3 4 5 6 |
XGBClassifier(base_score=0.5, colsample_bylevel=1, colsample_bytree=1, gamma=0, learning_rate=0.1, max_delta_step=0, max_depth=3, min_child_weight=1, missing=None, n_estimators=100, nthread=-1, objective='binary:logistic', reg_alpha=0, reg_lambda=1, scale_pos_weight=1, seed=0, silent=True, subsample=1) Accuracy: 83.84% |

We can tease out the effect of XGBoost’s automatic handling of missing values, by marking the missing values with a non-zero value, such as 1.

1 |
X[X == '?'] = 1 |

Re-running the example demonstrates a drop in accuracy for the model.

1 |
Accuracy: 79.80% |

We can also impute the missing data with a specific value.

It is common to use a mean or a median for the column. We can easily impute the missing data using the scikit-learn Imputer class.

1 2 3 |
# impute missing values as the mean imputer = Imputer() imputed_x = imputer.fit_transform(X) |

Below is the full example with missing data imputed with the mean value from each column.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 |
# binary classification, missing data, impute with mean import numpy from pandas import read_csv from xgboost import XGBClassifier from sklearn.model_selection import train_test_split from sklearn.metrics import accuracy_score from sklearn.preprocessing import LabelEncoder from sklearn.preprocessing import Imputer # load data dataframe = read_csv("horse-colic.csv", delim_whitespace=True, header=None) dataset = dataframe.values # split data into X and y X = dataset[:,0:27] Y = dataset[:,27] # set missing values to 0 X[X == '?'] = numpy.nan # convert to numeric X = X.astype('float32') # impute missing values as the mean imputer = Imputer() imputed_x = imputer.fit_transform(X) # encode Y class values as integers label_encoder = LabelEncoder() label_encoder = label_encoder.fit(Y) label_encoded_y = label_encoder.transform(Y) # split data into train and test sets seed = 7 test_size = 0.33 X_train, X_test, y_train, y_test = train_test_split(imputed_x, label_encoded_y, test_size=test_size, random_state=seed) # fit model no training data model = XGBClassifier() model.fit(X_train, y_train) print(model) # make predictions for test data y_pred = model.predict(X_test) predictions = [round(value) for value in y_pred] # evaluate predictions accuracy = accuracy_score(y_test, predictions) print("Accuracy: %.2f%%" % (accuracy * 100.0)) |

Running this example we see results equivalent to the fixing the value to one (1). This suggests that at least in this case we are better off marking the missing values with a distinct value of zero (0) rather than a valid value (1) or an imputed value.

1 |
Accuracy: 79.80% |

It is a good lesson to try both approaches (automatic handling and imputing) on your data when you have missing values.

## Summary

In this post you discovered how you can prepare your machine learning data for gradient boosting with XGBoost in Python.

Specifically, you learned:

- How to prepare string class values for binary classification using label encoding.
- How to prepare categorical input variables using a one hot encoding to model them as binary variables.
- How XGBoost automatically handles missing data and how you can mark and impute missing values.

Do you have any questions about how to prepare your data for XGBoost or about this post? Ask your questions in the comments and I will do my best to answer.

hi Jason, the train data for the last example should be imputed_x, but you use the original X which has missing data. I tried with data imputed_x and get the accuracy 79.8%

Thanks Ralph. Fixed.

Hi Jason,

Thanks for the tutorial with such useful information! I have one question regarding the label encoding and the one hot encoding you applied on the breast cancer dataset.

You perform label encoding and one hot encoding for the whole dataset and then split into train and test set. This way it can be ensured that all the data are transformed with the same encoding configuration.

However, if we have new unseen data with the raw dataset type, how can we ensure that label encoding and one hot encoding is still transforming the unseen data in the same way? Do we need to save the encoders for the sake of processing unseen data?

Thanks in advance!

Great question Qichang.

I would prepare the encodings on the training data, store the mappings (or pickle the objects), then reuse the encodings on the test data.

It means we must be confident that the training data is representative of the data we may need to predict in the future.

how exactly would this be done?

Which part are you having trouble with?

Thanks Jason for the prompt reply.

Besides this kinds of data transformation, do we need to consider scaling or normalisation of the input variables before passing to XGBoost? We know that it generally yields better result for SVM especially with kernel function.

Generally no scaling. You may see some benefit by spreading out a univariate distribution to highlight specific features (e.g. with a square, log, square root, etc.)

Hi Jason, your post is very helpful. Thanks a lot!

I had a question around how to treat “default” values of continuous predictors for XGBoost. For example, let’s say attributer X may take continuous values as such (say in range of 1 -100). But certain records may have some default value (say 9999) which denotes certain segment of customers for whom that predictor X cannot be calculated or is unavailable. Can we directly use predictor X as input variable for an XGBoost model? Or, should we do some data treatment for X? If so, what would that be?

TIA

Great question JChen.

I would try modeling the data as-is to start with. XGBoost will figure it out.

You could then try some feature engineering (maybe add a new flag variable for such cases) and see if you can further list performance.

Thanks for your reply! This is helpful

I’m glad to hear it JChen.

For your missing data part you replaced ‘?’ with 0. But you have not mentioned while defining XGBClassifier model that in your dataset treat 0 as missing value. And by default ‘missing’ parameter value is none which is equivalent to treating NaN as missing value. So i don’t think your model is handling missing values.

Thanks for the note Sargam.

Hi Jason,

I came to this page from another : IRIS ,

source: http://machinelearningmastery.com/multi-class-classification-tutorial-keras-deep-learning-library/#comment-396630

In that webPage, your code classifies IRIS with 96.6% accuracy, which is very good.

In comments section, you told this guy (Abhilash Menon) to use gradient boosting with XGBoost.

Which is this tutorial. Here is also IRIS classification.

Well I do not understand:

Here, the IRIS classification with gradient boosting with XGBoost yields 79% or 83% accuracy (only).

Why should we use gradient boosting with XGBoost then?

Accuracy is too low.

The tutorial is a demonstration of how to use the algorithm.

Your job as the machine learning practitioner is to try many algorithms on your problem and see what works best.

See this post:

http://machinelearningmastery.com/a-data-driven-approach-to-machine-learning/

Hi, Jason. First of all i would like to thank you for the wonderful material. I have been trying the XGBoost algorithm, it seems its acting weird on my pc. The first iris species dataset i got score of 21.2 %. Now this one of the breast cancer my accuracy is 2.1%. I really dont know wats wrong, can you please, please help me

I’m sorry to hear that. Perhaps the library is not installed correctly?

I would recommend posting a question to stackoverflow or even the xgboost users group.

Let me know how you go.

Hello Jason, I know that for regression models we should drop the first dummy variable to avoid the “dummy trap”. I don’t see you doing that in this case. Is is because the dummy variable trap only applies to linear regression models and not gradient boosting algorithms?

I believe the dummy variable trap applies with linear models and when the variables are multicollinear.

Hello Jason.

What happens if I have a string column with NA’s?

Should I first put 0’s instead of the NA’s and then use the OneHotEncounter ?

XGBoost can handle the NAs. No need to do anything.

If you do want to impute or similar, see this post:

https://machinelearningmastery.com/handle-missing-data-python/

Thank you, Jason.

Very useful article.

I met with library CatBoost which has interest options to configure categorical variables. Maybe, this will be interest for your journey.

Thanks, do you have a link?

Link on “parameter tuning” section: https://tech.yandex.com/catboost/doc/dg/concepts/parameter-tuning-docpage/#choosing-cat-features

You can see many interest materials on this site.

+ If it is interest for you, there are other interest libraries which are based on ensembles of trees. I.e., realization of genetic algorithm in “TPOT” (tree based pipeline optimization technique).

Thanks Alexander.

Dear Jason,

Thank you for the great tutorial!

Unfortunately I could not get the “datasets-uci-breast-cancer.csv” – looks like it was removed from the website. I search on the web but could not find a similar file to try out your example.

Is there a possibility you can send me this file?

Greatly appreciated

Dani

Here it is:

https://gist.github.com/jbrownlee/ceb34e3014be83da5f8a255c75b026d7

Thank you very much!

Do we need to deal with outliers before passing features to XGBoost?

Perhaps. Try with and without outlier removal and compare model skill.

Hi Jason,

Thanks for taking the time to map this all out. Prior to reading your tutorial, I used the DataCamp course on XGBoost as a guide, where they use two steps for encoding categorical variables: LabelEncoder followed by OneHotEncoder.

So a categorical variable with 5 levels is converted to values 0-4 and then these are one-hot encoded into columns five columns.

I am interested in the feature importance, so xgb.plot_importance is a great tool. However, the features are two steps removed from their original state.

How would you undo this two-step encoding to get the original variable names?

The one hot encoding can be reversed with an argmax() on each vector.

The label encoding has an inverse_transform() function in sklearn.

Could you discuss this and its’ applicability to XGBoost:

http://roamanalytics.com/2016/10/28/are-categorical-variables-getting-lost-in-your-random-forests/

Thanks for the link.

Jason, Do you have any idea about applying XGBoost on a multilabel classification problem? I just need some help from you on the data preparation part. Some suggestions from stackoverflow I found is massage your data a little by making k copies of every data point that has k correct/positive labels. Then you can hack your way to a simpler multi class problem…. Any suggestions would be highly appreciated!

Sorry, I don’t have any multi-label examples.