You must prepare your raw data using data transforms prior to fitting a machine learning model.

This is required to ensure that you best expose the structure of your predictive modeling problem to the learning algorithms.

Applying data transforms like scaling or encoding categorical variables is straightforward when all input variables are the same type. It can be challenging when you have a dataset with mixed types and you want to selectively apply data transforms to some, but not all, input features.

Thankfully, the scikit-learn Python machine learning library provides the **ColumnTransformer** that allows you to selectively apply data transforms to different columns in your dataset.

In this tutorial, you will discover how to use the ColumnTransformer to selectively apply data transforms to columns in a dataset with mixed data types.

After completing this tutorial, you will know:

- The challenge of using data transformations with datasets that have mixed data types.
- How to define, fit, and use the ColumnTransformer to selectively apply data transforms to columns.
- How to work through a real dataset with mixed data types and use the ColumnTransformer to apply different transforms to categorical and numerical data columns.

Let’s get started.

## Tutorial Overview

This tutorial is divided into three parts; they are:

- Challenge of Transforming Different Data Types
- How to use the ColumnTransformer
- Data Preparation for the Abalone Regression Dataset

## Challenge of Transforming Different Data Types

It is important to prepare data prior to modeling.

This may involve replacing missing values, scaling numerical values, and one hot encoding categorical data.

Data transforms can be performed using the scikit-learn library; for example, the SimpleImputer class can be used to replace missing values, the MinMaxScaler class can be used to scale numerical values, and the OneHotEncoder can be used to encode categorical variables.

For example:

1 2 3 4 5 6 7 |
... # prepare transform scaler = MinMaxScaler() # fit transform on training data scaler.fit(train_X) # transform training data train_X = scaler.transform(train_X) |

Sequences of different transforms can also be chained together using the Pipeline, such as imputing missing values, then scaling numerical values.

For example:

1 2 3 4 5 |
... # define pipeline pipeline = Pipeline(steps=[('i', SimpleImputer(strategy='median')), ('s', MinMaxScaler())]) # transform training data train_X = scaler.fit_transform(train_X) |

It is very common to want to perform different data preparation techniques on different columns in your input data.

For example, you may want to impute missing numerical values with a median value, then scale the values and impute missing categorical values using the most frequent value and one hot encode the categories.

Traditionally, this would require you to separate the numerical and categorical data and then manually apply the transforms on those groups of features before combining the columns back together in order to fit and evaluate a model.

Now, you can use the ColumnTransformer to perform this operation for you.

## How to use the ColumnTransformer

The ColumnTransformer is a class in the scikit-learn Python machine learning library that allows you to selectively apply data preparation transforms.

For example, it allows you to apply a specific transform or sequence of transforms to just the numerical columns, and a separate sequence of transforms to just the categorical columns.

To use the ColumnTransformer, you must specify a list of transformers.

Each transformer is a three-element tuple that defines the name of the transformer, the transform to apply, and the column indices to apply it to. For example:

- (Name, Object, Columns)

For example, the ColumnTransformer below applies a OneHotEncoder to columns 0 and 1.

1 |
transformer = ColumnTransformer(transformers=[('cat', OneHotEncoder(), [0, 1])]) |

The example below applies a SimpleImputer with median imputing for numerical columns 0 and 1, and SimpleImputer with most frequent imputing to categorical columns 2 and 3.

1 2 |
t = [('num', SimpleImputer(strategy='median'), [0, 1]), ('cat', SimpleImputer(strategy='most_frequent'), [2, 3])] transformer = ColumnTransformer(transformers=t) |

Any columns not specified in the list of “*transformers*” are dropped from the dataset by default; this can be changed by setting the “*remainder*” argument.

Setting *remainder=’passthrough’* will mean that all columns not specified in the list of “*transformers*” will be passed through without transformation, instead of being dropped.

For example, if columns 0 and 1 were numerical and columns 2 and 3 were categorical and we wanted to just transform the categorical data and pass through the numerical columns unchanged, we could define the ColumnTransformer as follows:

1 |
transformer = ColumnTransformer(transformers=[('cat', OneHotEncoder(), [2, 3])], remainder='passthrough') |

Once the transformer is defined, it can be used to transform a dataset.

For example:

1 2 3 4 |
... transformer = ColumnTransformer(transformers=[('cat', OneHotEncoder(), [0, 1])]) # transform training data train_X = transformer.fit_transform(train_X) |

A ColumnTransformer can also be used in a Pipeline to selectively prepare the columns of your dataset before fitting a model on the transformed data.

This is the most likely use case as it ensures that the transforms are performed automatically on the raw data when fitting the model and when making predictions, such as when evaluating the model on a test dataset via cross-validation or making predictions on new data in the future.

For example:

1 2 3 4 5 6 7 8 9 10 11 |
... # define model model = LogisticRegression() # define transform transformer = ColumnTransformer(transformers=[('cat', OneHotEncoder(), [0, 1])]) # define pipeline pipeline = Pipeline(steps=[('t', transformer), ('m',model)]) # fit the model on the transformed data model.fit(train_X, train_y) # make predictions yhat = model.predict(test_X) |

Now that we are familiar with how to configure and use the ColumnTransformer in general, let’s look at a worked example.

## Data Preparation for the Abalone Regression Dataset

The abalone dataset is a standard machine learning problem that involves predicting the age of an abalone given measurements of an abalone.

You can download the dataset and learn more about it here:

The dataset has 4,177 examples, 8 input variables, and the target variable is an integer.

A naive model can achieve a mean absolute error (MAE) of about 2.363 (std 0.092) by predicting the mean value, evaluated via 10-fold cross-validation.

We can model this as a regression predictive modeling problem with a support vector machine model (SVR).

Reviewing the data, you can see the first few rows as follows:

1 2 3 4 5 6 |
M,0.455,0.365,0.095,0.514,0.2245,0.101,0.15,15 M,0.35,0.265,0.09,0.2255,0.0995,0.0485,0.07,7 F,0.53,0.42,0.135,0.677,0.2565,0.1415,0.21,9 M,0.44,0.365,0.125,0.516,0.2155,0.114,0.155,10 I,0.33,0.255,0.08,0.205,0.0895,0.0395,0.055,7 ... |

We can see that the first column is categorical and the remainder of the columns are numerical.

We may want to one hot encode the first column and normalize the remaining numerical columns, and this can be achieved using the ColumnTransformer.

First, we need to load the dataset. We can load the dataset directly from the URL using the read_csv() Pandas function, then split the data into two data frames: one for input and one for the output.

The complete example of loading the dataset is listed below.

1 2 3 4 5 6 7 8 9 |
# load the dataset from pandas import read_csv # load dataset url = 'https://raw.githubusercontent.com/jbrownlee/Datasets/master/abalone.csv' dataframe = read_csv(url, header=None) # split into inputs and outputs last_ix = len(dataframe.columns) - 1 X, y = dataframe.drop(last_ix, axis=1), dataframe[last_ix] print(X.shape, y.shape) |

**Note**: if you have trouble loading the dataset from a URL, you can download the CSV file with the name ‘*abalone.csv*‘ and place it in the same directory as your Python file and change the call to *read_csv()* as follows:

1 2 |
... dataframe = read_csv('abalone.csv', header=None) |

Running the example, we can see that the dataset is loaded correctly and split into eight input columns and one target column.

1 |
(4177, 8) (4177,) |

Next, we can use the *select_dtypes()* function to select the column indexes that match different data types.

We are interested in a list of columns that are numerical columns marked as ‘*float64*‘ or ‘*int64*‘ in Pandas, and a list of categorical columns, marked as ‘*object*‘ or ‘*bool*‘ type in Pandas.

1 2 3 4 |
... # determine categorical and numerical features numerical_ix = X.select_dtypes(include=['int64', 'float64']).columns categorical_ix = X.select_dtypes(include=['object', 'bool']).columns |

We can then use these lists in the ColumnTransformer to one hot encode the categorical variables, which should just be the first column.

We can also use the list of numerical columns to normalize the remaining data.

1 2 3 4 |
... # define the data preparation for the columns t = [('cat', OneHotEncoder(), categorical_ix), ('num', MinMaxScaler(), numerical_ix)] col_transform = ColumnTransformer(transformers=t) |

Next, we can define our SVR model and define a Pipeline that first uses the ColumnTransformer, then fits the model on the prepared dataset.

1 2 3 4 5 |
... # define the model model = SVR(kernel='rbf',gamma='scale',C=100) # define the data preparation and modeling pipeline pipeline = Pipeline(steps=[('prep',col_transform), ('m', model)]) |

Finally, we can evaluate the model using 10-fold cross-validation and calculate the mean absolute error, averaged across all 10 evaluations of the pipeline.

1 2 3 4 5 6 7 8 9 |
... # define the model cross-validation configuration cv = KFold(n_splits=10, shuffle=True, random_state=1) # evaluate the pipeline using cross validation and calculate MAE scores = cross_val_score(pipeline, X, y, scoring='neg_mean_absolute_error', cv=cv, n_jobs=-1) # convert MAE scores to positive values scores = absolute(scores) # summarize the model performance print('MAE: %.3f (%.3f)' % (mean(scores), std(scores))) |

Tying this all together, the complete example is listed below.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 |
# example of using the ColumnTransformer for the Abalone dataset from numpy import mean from numpy import std from numpy import absolute from pandas import read_csv from sklearn.model_selection import cross_val_score from sklearn.model_selection import KFold from sklearn.compose import ColumnTransformer from sklearn.pipeline import Pipeline from sklearn.preprocessing import OneHotEncoder from sklearn.preprocessing import MinMaxScaler from sklearn.svm import SVR # load dataset url = 'https://raw.githubusercontent.com/jbrownlee/Datasets/master/abalone.csv' dataframe = read_csv(url, header=None) # split into inputs and outputs last_ix = len(dataframe.columns) - 1 X, y = dataframe.drop(last_ix, axis=1), dataframe[last_ix] print(X.shape, y.shape) # determine categorical and numerical features numerical_ix = X.select_dtypes(include=['int64', 'float64']).columns categorical_ix = X.select_dtypes(include=['object', 'bool']).columns # define the data preparation for the columns t = [('cat', OneHotEncoder(), categorical_ix), ('num', MinMaxScaler(), numerical_ix)] col_transform = ColumnTransformer(transformers=t) # define the model model = SVR(kernel='rbf',gamma='scale',C=100) # define the data preparation and modeling pipeline pipeline = Pipeline(steps=[('prep',col_transform), ('m', model)]) # define the model cross-validation configuration cv = KFold(n_splits=10, shuffle=True, random_state=1) # evaluate the pipeline using cross validation and calculate MAE scores = cross_val_score(pipeline, X, y, scoring='neg_mean_absolute_error', cv=cv, n_jobs=-1) # convert MAE scores to positive values scores = absolute(scores) # summarize the model performance print('MAE: %.3f (%.3f)' % (mean(scores), std(scores))) |

Running the example evaluates the data preparation pipeline using 10-fold cross-validation.

Your specific results may vary given the stochastic learning algorithm and differences in library versions.

In this case, we achieve an average MAE of about 1.4, which is better than the baseline score of 2.3.

1 2 |
(4177, 8) (4177,) MAE: 1.465 (0.047) |

You now have a template for using the ColumnTransformer on a dataset with mixed data types that you can use and adapt for your own projects in the future.

## Further Reading

This section provides more resources on the topic if you are looking to go deeper.

### API

- sklearn.compose.ColumnTransformer API.
- pandas.read_csv API.
- sklearn.impute.SimpleImputer API.
- sklearn.preprocessing.OneHotEncoder API.
- sklearn.preprocessing.MinMaxScaler API
- sklearn.pipeline.Pipeline API.

## Summary

In this tutorial, you discovered how to use the ColumnTransformer to selectively apply data transforms to columns in datasets with mixed data types.

Specifically, you learned:

- The challenge of using data transformations with datasets that have mixed data types.
- How to define, fit, and use the ColumnTransformer to selectively apply data transforms to columns.
- How to work through a real dataset with mixed data types and use the ColumnTransformer to apply different transforms to categorical and numerical data columns.

Do you have any questions?

Ask your questions in the comments below and I will do my best to answer.

Wow this function with the pipeline seems to be magical. I really wanted to know what pipelines can do(the power of pipelines). I have seen the documentation of pipeline(), it’s not as simple as you explain 🙂 I really likes your way of explanation. Have you written any blogs on introduction to pipelines which can start with simple and explain complicated pipelines. if so, please share the links.

if not, Can you please share some references where we can learn more about the sklearn pipeline?

Thanks.

Yes, maybe here:

https://machinelearningmastery.com/automate-machine-learning-workflows-pipelines-python-scikit-learn/