The scikit-learn Python library for machine learning offers a suite of data transforms for changing the scale and distribution of input data, as well as removing input features (columns).
There are many simple data cleaning operations, such as removing outliers and removing columns with few observations, that are often performed manually to the data, requiring custom code.
The scikit-learn library provides a way to wrap these custom data transforms in a standard way so they can be used just like any other transform, either on data directly or as a part of a modeling pipeline.
In this tutorial, you will discover how to define and use custom data transforms for scikit-learn.
After completing this tutorial, you will know:
That custom data transforms can be created for scikit-learn using the FunctionTransformer class.
How to develop and apply a custom transform to remove columns with few unique values.
How to develop and apply a custom transform that replaces outliers for each column.
Kick-start your project with my new book Data Preparation for Machine Learning, including step-by-step tutorials and the Python source code files for all examples.
Let’s get started.
How to Create Custom Data Transforms for Scikit-Learn Photo by Berit Watkin, some rights reserved.
Tutorial Overview
This tutorial is divided into four parts; they are:
Custom Data Transforms in Scikit-Learn
Oil Spill Dataset
Custom Transform to Remove Columns
Custom Transform to Replace Outliers
Custom Data Transforms in Scikit-Learn
Data preparation refers to changing the raw data in some way that makes it more appropriate for predictive modeling with machine learning algorithms.
The scikit-learn Python machine learning library offers many different data preparation techniques directly, such as techniques for scaling numerical input variables and changing the probability distribution of variables.
These transforms can be fit and then applied on a dataset or used as part of a predictive modeling pipeline, allowing a sequence of transforms to be applied correctly without data leakage when evaluating model performance with data sampling techniques, such as k-fold cross-validation.
Although the data preparation techniques available in scikit-learn are extensive, there may be additional data preparation steps that are required.
Typically, these additional steps are performed manually prior to modeling and require writing custom code. The risk is that these data preparation steps may be performed inconsistently.
The solution is to create a custom data transform in scikit-learn using the FunctionTransformer class.
This class allows you to specify a function that is called to transform the data. You can define the function and perform any valid change, such as changing values or removing columns of data (not removing rows).
The class can then be used just like any other data transform in scikit-learn, e.g. to transform data directly, or used in a modeling pipeline.
The catch is that the transform is stateless, meaning that no state can be kept.
This means that the transform cannot be used to calculate statistics on the training dataset that are then used to transform the train and test datasets.
In addition to custom scaling operations, this can be helpful for standard data cleaning operations, such as identifying and removing columns with few unique values and identifying and removing relative outliers.
We will explore both of these cases, but first, let’s define a dataset that we can use as the basis for exploration.
Want to Get Started With Data Preparation?
Take my free 7-day email crash course now (with sample code).
Click to sign-up and also get a free PDF Ebook version of the course.
Oil Spill Dataset
The so-called “oil spill” dataset is a standard machine learning dataset.
The task involves predicting whether a patch contains an oil spill or not, e.g. from the illegal or accidental dumping of oil in the ocean, given a vector that describes the contents of a patch of a satellite image.
There are 937 cases. Each case is composed of 48 numerical computer vision derived features, a patch number, and a class label.
The normal case is no oil spill assigned the class label of 0, whereas an oil spill is indicated by a class label of 1. There are 896 cases for no oil spill and 41 cases of an oil spill.
We can see that the first column contains integers for the patch number. We can also see that the computer vision derived features are real-valued with differing scales, such as thousands in the second column and fractions in other columns.
This dataset contains columns with very few unique values and columns with outliers that provide a good basis for data cleaning.
The example below downloads the dataset and loads it as a numPy array and summarizes the number of rows and columns.
Running the example loads the dataset and confirms the expected number of rows and columns.
1
(937, 49) (937,)
Now that we have a dataset that we can use as the basis for data transforms, let’s look at how we can define some custom data cleaning transforms using the FunctionTransformer class.
Custom Transform to Remove Columns
Columns that have few unique values are probably not contributing anything useful to predicting the target value.
This is not absolutely true, but it is true enough that you should test the performance of your model fit on a dataset with columns of this type removed.
This is a type of data cleaning, and there is a data transform provided in scikit-learn called the VarianceThreshold that attempts to address this using the variance of each column.
Another approach is to remove columns that have fewer than a specified number of unique values, such as 1.
We can develop a function that applies this transform and use the minimum number of unique values as a configurable default argument. We will also add some debugging to confirm it is working as we expect.
First, the number of unique values for each column can be calculated. Ten columns with equal or fewer than the minimum number of unique values can be identified. Finally, those identified columns can be removed from the dataset.
The cust_transform() function below implements this.
# select all but the columns that are being removed
ix=[iforiinrange(X.shape[1])ifinotinto_del]
result=X[:,ix]
returnresult
We can then use this function in the FunctionTransformer.
A limitation of this transform is that it selects columns to delete based on the provided data. This means if a train and test dataset differ greatly, then it is possible for different columns to be removed from each, making model evaluation challenging (unstable!?). As such, it is best to keep the minimum number of unique values small, such as 1.
We can use this transform on the oil spill dataset. The complete example is listed below.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
# custom data transform for removing columns with few unique values
from numpy import unique
from pandas import read_csv
from sklearn.preprocessing import FunctionTransformer
Running the example first reports the number of rows and columns in the raw dataset.
Next, a list is printed that shows the number of unique values observed for each column in the dataset. We can see that many columns have very few unique values.
The columns with one (or fewer) unique values are then identified and reported. In this case, column index 22. This column is removed from the dataset.
Finally, the shape of the transformed dataset is reported, showing 48 instead of 49 columns, confirming that the column with a single unique value was deleted.
There are many extensions you could explore to this transform, such as:
Ensure that it is only applied to numerical input variables.
Experiment with a different minimum number of unique values.
Use a percentage rather than an absolute number of unique values.
If you explore any of these extensions, let me know in the comments below.
Next, let’s look at a transform that replaces values in the dataset.
Custom Transform to Replace Outliers
Outliers are observations that are different or unlike the other observations.
If we consider one variable at a time, an outlier would be a value that is far from the center of mass (the rest of the values), meaning it is rare or has a low probability of being observed.
There are standard ways for identifying outliers for common probability distributions. For Gaussian data, we can identify outliers as observations that are three or more standard deviations from the mean.
This may or may not be a desirable way to identify outliers for data that has many input variables, yet can be effective in some cases.
We can identify outliers in this way and replace their value with a correction, such as the mean.
Each column is considered one at a time and mean and standard deviation statistics are calculated. Using these statistics, upper and lower bounds of “normal” values are defined, then all values that fall outside these bounds can be identified. If one or more outliers are identified, their values are then replaced with the mean value that was already calculated.
The cust_transform() function below implements this as a function applied to the dataset, where we parameterize the number of standard deviations from the mean and whether or not debug information will be displayed.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
# replace outliers
def cust_transform(X,n_stdev=3,verbose=True):
# copy the array
result=X.copy()
# enumerate each column
foriinrange(result.shape[1]):
# retrieve values for column
col=X[:,i]
# calculate statistics
mu,sigma=mean(col),std(col)
# define bounds
lower,upper=mu-(sigma*n_stdev),mu+(sigma*n_stdev)
# select indexes that are out of bounds
ix=where(logical_or(col<lower,col>upper))[0]
ifverbose andlen(ix)>0:
print('>col=%d, outliers=%d'%(i,len(ix)))
# replace values
result[ix,i]=mu
returnresult
We can then use this function in the FunctionTransformer.
The method of outlier detection assumes a Gaussian probability distribution and applies to each variable independently, both of which are strong assumptions.
An additional limitation of this implementation is that the mean and standard deviation statistics are calculated on the provided dataset, meaning that the definition of an outlier and its replacement value are both relative to the dataset. This means that different definitions of outliers and different replacement values could be used if the transform is used on the train and test sets.
We can use this transform on the oil spill dataset. The complete example is listed below.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
# custom data transform for replacing outliers
from numpy import mean
from numpy import std
from numpy import where
from numpy import logical_or
from pandas import read_csv
from sklearn.preprocessing import FunctionTransformer
Running the example first reports the shape of the dataset prior to any change.
Next, the number of outliers for each column is calculated and only those columns with one or more outliers are reported in the output. We can see that a total of 32 columns in the dataset have one or more outliers.
The outliers are then removed and the shape of the resulting dataset is reported, confirming no change in the number of rows or columns.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
(937, 49) (937,)
>col=0, outliers=10
>col=1, outliers=8
>col=3, outliers=8
>col=5, outliers=7
>col=6, outliers=1
>col=7, outliers=12
>col=8, outliers=15
>col=9, outliers=14
>col=10, outliers=19
>col=11, outliers=17
>col=12, outliers=22
>col=13, outliers=2
>col=14, outliers=16
>col=15, outliers=8
>col=16, outliers=8
>col=17, outliers=6
>col=19, outliers=12
>col=20, outliers=20
>col=27, outliers=14
>col=28, outliers=18
>col=29, outliers=2
>col=30, outliers=13
>col=32, outliers=3
>col=34, outliers=14
>col=35, outliers=15
>col=37, outliers=13
>col=40, outliers=18
>col=41, outliers=13
>col=42, outliers=12
>col=43, outliers=12
>col=44, outliers=19
>col=46, outliers=21
(937, 49)
There are many extensions you could explore to this transform, such as:
Ensure that it is only applied to numerical input variables.
Experiment with a different number of standard deviations from the mean, such as 2 or 4.
Use a different definition of outlier, such as the IQR or a model.
If you explore any of these extensions, let me know in the comments below.
Further Reading
This section provides more resources on the topic if you are looking to go deeper.
It provides self-study tutorials with full working code on: Feature Selection, RFE, Data Cleaning, Data Transforms, Scaling, Dimensionality Reduction,
and much more...
Bring Modern Data Preparation Techniques to Your Machine Learning Projects
They are named arguments with default value – you must specify them by name.
…
cust_transform(X, min_values=36, verbose=True)
Perhaps read-up on python function arguments.
We can use the function directly, but the purpose of the tutorial is to show how to use a custom transform object that can be used anyway you like – such as directly or in a pipeline.
Dear Dr Jason,
Thank you for that, it works! The kw_args as you said is a dictionary, so the kw_args must be assigned with dictionary braces such as that {}.
Dr Dr Jason,
Thank you for your time.
In the 2nd paragraph under the heading “Custom Data Transforms in Scikit-Learn”, the FunctionTransformer can be integrated into a pipeline.
What is the correct implementation of FunctionTransformers with pipeline? Here is an example with other functions incorporated in the pipeline.
Dear Dr Jason,
It seems to work.
I made two pipelines – one that includes a custom transform to exclude outliers and the other does not have a custom transform. Quite interesting that by excluding outliers, your score is less!
Conclusion: leaving outliers you have a greater score than excluding outliers.
Great!
Now you will ask yourself what happens if I do something like pipeline.fit_transform?
You may notice that some functions within the pipeline to use fit_transform won’t work because some of those functions do not contain a fit_transform function: Let’s see
AttributeError:'DecisionTreeClassifier'objecthas no attribute'transform'
DecisionTreeClassifier does not have a transform function. BUT
BUT trans1 does have an inverse
Recall trans1 has an inverse function as specified in the inverse_func
#It's compulsory to have one argument in the function else it spews.
print("Demonstrates what happens within the function")
return"blah blah blah blah blah blah blah blah"
So the lesson is: if you want to invoke the pipeline’s inverse_transform function, make sure that the functions in the pipeline have an inverse.
Recall of other lesson:
Removing outliers from your dataset does not necessarily improve the score. In the pima-indians-diabetes case, the score decreased.
Dear Dr Jason,
I made a mistake in the first lesson.
Where I mentioned in the first lesson that if you want to invoke the pipeline’s inverse_transform, was meant to be invoke the pipeline’s transform function, make sure that the other functions in the pipeline have the transform function. In this case, the DecisionTreeClassifier does not have a transform function.
Dear Dr Jason,
To put this generally about pipelines.
If you invoke the pipeline’s transform function or any other method, THEN all the functions in the steps must have a corresponding function. In this case the transform method.
Put it another way. If you invoke pipeline.particular_method() then all the listed steps must have a corresponding particular_method().
I think you meant to say “The example below downloads the dataset and loads it as a Pandas DataFrame and summarizes the number of rows and columns.” instead of “loads it as a numPy array”. Correct me if I’m wrong.
Dear Sir,
I have a question , let say i have a a dataset, in which i need to generate a new column by elementwise division of two existing columns. Now after test train split and missing value imputation. can i generate this column with the help of function transformer in the training set and later in the test set ?
Sir,
I am wriing an example how can i add new columns in an existing numpy array using function transformer
>>> import numpy as np
>>> from sklearn.preprocessing import FunctionTransformer
>>> def col_add(x):
x1 = x[:, 0] + x[:, 1]
x2 = x[:, 0] * x[:, 1]
x3 = x[:, 0] / x[:, 1]
return np.c_[x, x1, x2, x3]
I have made such a transformer, above given is an example that generates 3 new columns, from existing 2 columns of a numpy array , first column is for element wise addition, second is for element wise multiplication and third is for element wise division .
So in this way a function transformer can be used to add new features generated from existing columns ?
Dear Dr Jason,
In the section “Custom Transform to Remove Columns”, I understand how the function
The function accepts three parameters with min_values and verbose default values are 1 and True.
Need clarification on why I cannot change the default values to something else.
For example while I can do this:
I cannot do this:
Even when I fit two variables, the FunctionTransformer(X,36) will IGNORE 36 and use the default 1.
Questions please:
How do you get FunctionTransformer to bypass the two default values of cust_transform?
Why not use cust_transform directly? I could set all three parameters successfully this way without a problem?
It looks ‘useless’ using FunctionTransformer when I can invoke the function directly without problem. Is there something that I am missing?
Thank you again in advance,
Anthony of Sydney
They are named arguments with default value – you must specify them by name.
…
cust_transform(X, min_values=36, verbose=True)
Perhaps read-up on python function arguments.
We can use the function directly, but the purpose of the tutorial is to show how to use a custom transform object that can be used anyway you like – such as directly or in a pipeline.
Dear Dr Jason,
Thank you for your response.
I understand how function arguments work when using the custom function.
BUT BUT I could not pass the arguments via the FunctionTransformer.
To demonstrate:
Why couldn’t I use the FunctionTransformer to pass arguments in the same way as the cust_transformer?
Thank you,
Anthony of Sydney
Good question, you must specify arguments to the function as a dictionary to the “kw_args” argument, see the API here for more information:
https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.FunctionTransformer.html
e.g. I would expect this would work:
Dear Dr Jason,
Thank you for that, it works! The kw_args as you said is a dictionary, so the kw_args must be assigned with dictionary braces such as that {}.
While the documentation at scikit-learn.org site at times appears vague, this site, in the context of FunctionTransformer elucidates many examples including the inverse transformer function, ref https://www.programcreek.com/python/example/93354/sklearn.preprocessing.FunctionTransformer
Thank you again,
Anthony of Sydney
Dear Dr Jason,
Apologies. I printed the wrong X. I’ll explain this later.
The above X should is much larger
This is what should have been printed
Explanation for the error: I was experimenting with an inverse function and used a simple matrix and printed the wrong thing. Source of inspiration, https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.FunctionTransformer.html
How to get an inverse function using simple matrix.
Thank you again,
Anthony of Sydney
Dear Dr Jason,
I wanted to make an inverse function, but I had to make the inverse function accept an input parameter.
The transform and inverse_transform work. I know that.
BUT I cannot have an inverse_transform that takes NO arguments.
Because if I make an inverse function with no arguments, this is what happens
Don’t know why myInverseFunction requires an argument when I defined it not to accept an argument.
BUT if myInverseFunction is defined to accept an argument doNothingVariable, it works – please refer to successful example at the top of this reply.
Thank you again,
Anthony of Sydney
The inverse function must take the data to invert the transform on.
As per the documentation, you must specify another function to calculate the inverse and arguments to that function.
Right, I was too quick off the mark. Nice work!
Dr Dr Jason,
Thank you for your time.
In the 2nd paragraph under the heading “Custom Data Transforms in Scikit-Learn”, the FunctionTransformer can be integrated into a pipeline.
What is the correct implementation of FunctionTransformers with pipeline? Here is an example with other functions incorporated in the pipeline.
Given that pipeline, trans1, trans2, rfe and model each have the fit function when I invoke
Are all the fit functions in trans1, trans2, rfe and model carried out in the order within the pipeline?
Thank you,
Anthony of Sydney
Looks about right to me.
Dear Dr Jason,
It seems to work.
I made two pipelines – one that includes a custom transform to exclude outliers and the other does not have a custom transform. Quite interesting that by excluding outliers, your score is less!
Conclusion: leaving outliers you have a greater score than excluding outliers.
Great!
Now you will ask yourself what happens if I do something like pipeline.fit_transform?
You may notice that some functions within the pipeline to use fit_transform won’t work because some of those functions do not contain a fit_transform function: Let’s see
DecisionTreeClassifier does not have a transform function. BUT
BUT trans1 does have an inverse
Recall trans1 has an inverse function as specified in the inverse_func
Here is myInverseFunction
So the lesson is: if you want to invoke the pipeline’s inverse_transform function, make sure that the functions in the pipeline have an inverse.
Recall of other lesson:
Removing outliers from your dataset does not necessarily improve the score. In the pima-indians-diabetes case, the score decreased.
Thank you,
Anthony of Sydney
You can only fit_transform a data transform, not a pipeline. You can fit a pipeline.
Dear Dr Jason,
I made a mistake in the first lesson.
Where I mentioned in the first lesson that if you want to invoke the pipeline’s inverse_transform, was meant to be invoke the pipeline’s transform function, make sure that the other functions in the pipeline have the transform function. In this case, the DecisionTreeClassifier does not have a transform function.
Thank you,
Anthony of Sydney
Dear Dr Jason,
To put this generally about pipelines.
If you invoke the pipeline’s transform function or any other method, THEN all the functions in the steps must have a corresponding function. In this case the transform method.
Put it another way. If you invoke pipeline.particular_method() then all the listed steps must have a corresponding particular_method().
Thank you,
Anthony of Sydney
Yes.
This seems straightforward.
I think you meant to say “The example below downloads the dataset and loads it as a Pandas DataFrame and summarizes the number of rows and columns.” instead of “loads it as a numPy array”. Correct me if I’m wrong.
Thanks. Not much functional difference.
Dear Sir,
I have a question , let say i have a a dataset, in which i need to generate a new column by elementwise division of two existing columns. Now after test train split and missing value imputation. can i generate this column with the help of function transformer in the training set and later in the test set ?
Perhaps try it.
Sir,
I am wriing an example how can i add new columns in an existing numpy array using function transformer
>>> import numpy as np
>>> from sklearn.preprocessing import FunctionTransformer
>>> def col_add(x):
x1 = x[:, 0] + x[:, 1]
x2 = x[:, 0] * x[:, 1]
x3 = x[:, 0] / x[:, 1]
return np.c_[x, x1, x2, x3]
>>> col_adder = FunctionTransformer(col_add)
>>> arr = np.array([[2, 7], [4, 9], [3, 5]])
>>> arr
array([[2, 7],
[4, 9],
[3, 5]])
>>> col_adder.transform(arr) # will add 3 columns
array([[ 2. , 7. , 9. , 14. , 0.28571429],
[ 4. , 9. , 13. , 36. , 0.44444444],
[ 3. , 5. , 8. , 15. , 0.6 ]])
>>>
I have made such a transformer, above given is an example that generates 3 new columns, from existing 2 columns of a numpy array , first column is for element wise addition, second is for element wise multiplication and third is for element wise division .
So in this way a function transformer can be used to add new features generated from existing columns ?
You can use hstack() to add a new column to an existing matrix.
sklearn transforms cannot add columns.