Preparing data is required to get the best results from machine learning algorithms.
In this post you will discover how to transform your data in order to best expose its structure to machine learning algorithms in R using the caret package.
You will work through 8 popular and powerful data transforms with recipes that you can study or copy and paste int your current or next machine learning project.
Kick-start your project with my new book Machine Learning Mastery With R, including step-by-step tutorials and the R source code files for all examples.
Let’s get started.
Pre-Process Your Machine Learning Dataset in R Photo by Fraser Cairns, some rights reserved.
Need For Data Pre-Processing
You want to get the best accuracy from machine learning algorithms on your datasets.
Some machine learning algorithms require the data to be in a specific form. Whereas other algorithms can perform better if the data is prepared in a specific way, but not always. Finally, your raw data may not be in the best format to best expose the underlying structure and relationships to the predicted variables.
It is important to prepare your data in such a way that it gives various different machine learning algorithms the best chance on your problem.
You need to pre-process your raw data as part of your machine learning project.
Data Pre-Processing Methods
It is hard to know which data-preprocessing methods to use.
You can use rules of thumb such as:
Instance based methods are more effective if the input attributes have the same scale.
Regression methods can work better of the input attributes are standardized.
These are heuristics, but not hard and fast laws of machine learning, because sometimes you can get better results if you ignore them.
You should try a range of data transforms with a range of different machine learning algorithms. This will help you discover both good representations for your data and algorithms that are better at exploiting the structure that those representations expose.
It is a good idea to spot check a number of transforms both in isolation as well as combinations of transforms.
In the next section you will discover how you can apply data transforms in order to prepare your data in R using the caret package.
Need more Help with R for Machine Learning?
Take my free 14-day email course and discover how to use R on your project (with sample code).
Click to sign-up and also get a free PDF Ebook version of the course.
Data Pre-Processing With Caret in R
The caret package in R provides a number of useful data transforms.
These transforms can be used in two ways.
Standalone: Transforms can be modeled from training data and applied to multiple datasets. The model of the transform is prepared using the preProcess() function and applied to a dataset using the predict() function.
Training: Transforms can prepared and applied automatically during model evaluation. Transforms applied during training are prepared using the preProcess() and passed to the train() function via the preProcess argument.
A number of data preprocessing examples are presented in this section. They are presented using the standalone method, but you can just as easily use the prepared preprocessed model during model training.
All of the preprocessing examples in this section are for numerical data. Note that the preprocessing functions will skip over non-numeric data without raising an error.
You can learn more about the data transforms provided by the caret package by reading the help for the preProcess function by typing ?preProcess and by reading the Caret Pre-Processing page.
The data transforms presented are more likely to be useful for algorithms such as regression algorithms, instance-based methods (like kNN and LVQ), support vector machines and neural networks. They are less likely to be useful for tree and rule based methods.
Summary of Transform Methods
Below is a quick summary of all of the transform methods supported in the method argument of the preProcess() function in caret.
“BoxCox“: apply a Box–Cox transform, values must be non-zero and positive.
“YeoJohnson“: apply a Yeo-Johnson transform, like a BoxCox, but values can be negative.
“expoTrans“: apply a power transform like BoxCox and YeoJohnson.
“zv“: remove attributes with a zero variance (all the same value).
“nzv“: remove attributes with a near zero variance (close to the same value).
“center“: subtract mean from values.
“scale“:Â divide values by standard deviation.
“range“: normalize values.
“pca“: transform data to the principal components.
“ica“: transform data to the independent components.
“spatialSign“: project data onto a unit circle.
The following sections will demonstrate some of the more popular methods.
1. Scale
The scale transform calculates the standard deviation for an attribute and divides each value by that standard deviation.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
# load libraries
library(caret)
# load the dataset
data(iris)
# summarize data
summary(iris[,1:4])
# calculate the pre-process parameters from the dataset
When an attribute has a Gaussian-like distribution but is shifted, this is called a skew. The distribution of an attribute can be shifted to reduce the skew and make it more Gaussian. The BoxCox transform can perform this operation (assumes all values are positive).
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# load libraries
library(mlbench)
library(caret)
# load the dataset
data(PimaIndiansDiabetes)
# summarize pedigree and age
summary(PimaIndiansDiabetes[,7:8])
# calculate the pre-process parameters from the dataset
# summarize the transformed dataset (note pedigree and age)
summary(transformed)
Running the recipe, you will see:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
pedigree age
Min. :0.0780 Min. :21.00
1st Qu.:0.2437 1st Qu.:24.00
Median :0.3725 Median :29.00
Mean :0.4719 Mean :33.24
3rd Qu.:0.6262 3rd Qu.:41.00
Max. :2.4200 Max. :81.00
Created from 768 samples and 2 variables
Pre-processing:
- ignored (0)
- Yeo-Johnson transformation (2)
Lambda estimates for Yeo-Johnson transformation:
-2.25, -1.15
pedigree age
Min. :0.0691 Min. :0.8450
1st Qu.:0.1724 1st Qu.:0.8484
Median :0.2265 Median :0.8524
Mean :0.2317 Mean :0.8530
3rd Qu.:0.2956 3rd Qu.:0.8580
Max. :0.4164 Max. :0.8644
7. Principal Component Analysis
Transform the data to the principal components. The transform keeps components above the variance threshold (default=0.95) or the number of components can be specified (pcaComp). The result is attributes that are uncorrelated, useful for algorithms like linear and generalized linear regression.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
# load the libraries
library(mlbench)
# load the dataset
data(iris)
# summarize dataset
summary(iris)
# calculate the pre-process parameters from the dataset
PCA needed 2 components to capture 95 percent of the variance
Species PC1 PC2
setosa :50 Min. :-2.7651 Min. :-2.67732
versicolor:50 1st Qu.:-2.0957 1st Qu.:-0.59205
virginica :50 Median : 0.4169 Median :-0.01744
Mean : 0.0000 Mean : 0.00000
3rd Qu.: 1.3385 3rd Qu.: 0.59649
Max. : 3.2996 Max. : 2.64521
8. Independent Component Analysis
Transform the data to the independent components. Unlike PCA, ICA retains those components that are independent. You must specify the number of desired independent components with the n.comp argument. Useful for algorithms such as naive bayes.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# load libraries
library(mlbench)
library(caret)
# load the dataset
data(PimaIndiansDiabetes)
# summarize dataset
summary(PimaIndiansDiabetes[,1:8])
# calculate the pre-process parameters from the dataset
Below are some tips for getting the most out of data transforms.
Actually Use Them. You are a step ahead if you are thinking about and using data transforms to prepare your data. It is an easy step to forget or skip over and often has a huge impact on the accuracy of your final models.
Use a Variety. Try a number of different data transforms on your data with a suite of different machine learning algorithms.
Review a Summary. It is a good idea to summarize your data before and after a transform to understand the effect it had. The summary() function can be very useful.
Visualize Data. It is also a good idea to visualize the distribution of your data before and after to get a spatial intuition for the effect of the transform.
Summary
In this section you discovered 8 data preprocessing methods that you can use on your data in R via the caret package:
Data scaling
Data centering
Data standardization
Data normalization
The Box-Cox Transform
The Yeo-Johnson Transform
PCA Transform
ICA Transform
You can practice with the recipes presented in this section or apply them on your current or next machine learning project.
Next Step
Did you try out these recipes?
Start your R interactive environment.
Type or copy-paste the recipes above and try them out.
Use the built-in help in R to learn more about the functions used.
Do you have a question. Ask it in the comments and I will do my best to answer it.
Under the section Summary of Transform Methods, it is mentioned that,
“center“: divide values by standard deviation.
“scale“: subtract mean from values.
But when it comes to demonstration of methods, the functionality of center and scale is interchanged as,
1. Scale
The scale transform calculates the standard deviation for an attribute and divides each value by that standard deviation.
2. Center
The center transform calculates the mean for an attribute and subtracts it from each value.
So which is the correct functionality of these methods.
Yes, I would recommend experimenting with turning them into binary variables, also called one hot encoding or dummy variables. You can learnmore here: https://topepo.github.io/caret/pre-processing.html
When should we be applying standardization ?
I am currently applying normalization ( variables expressed in different units). Should I apply standardization next ?
Strictly, standardization helps when your univariate distribution is Gaussian, the units vary across features and your algorithm assumes units do not vary and univariate distributions are Gaussian.
Results can be good or even better when you break this strict heuristic.
Generally, I recommend trying a suite of different data scaling and transforms for a problem in order to flush out what representations work well for your problem.
I have some doubt for the flow: should we first check the correlations between predictors to see collinearity, then do data treatment (normalization, etc)? I read your book and it follows this flow. Thank you!
How is the preprocessing different when we have data from accelerometer signals which measure gait. For example the data consists of x,y,z which are the measures of the accelerometer, another column with miliseconds and a dependend variable which states the event, for example walking or sitting. In this case, do we have to create windows first and then start extracting features?
Hi Jason, I trained my NN model on pre-processed (normalized) data and then used the model to predict (the data I fed is also normalized). How do I convert the prediction results to be un-normalized so that it makes sense? Thanks.
I used preProcess to standardize my train data set(data frame). Then, I used it and developed a model.
Then,to test my model on my test dataset(data frame), I need to standardize variables in test dataset with the same mean and std of variables in train dataset. (just, correct me if i am wrong!?) If so, Is there any package or method to do this?
hi Jason, thanks for the tutorial, however i have one problem i am facing, after training and testing, i supply new inputs in standardized form to the neural networks to get new known outputs, these outputs are in standardized range as well, how do i get the outputs back to the original unstandardized range.
If I have a dataset of categorical variables in the same units but with vastly different values (like height at age 8 and age 25): is it recommendable to standardize them? I mean, I don’t know what to do because for all of them the unit represent the same distance.
Also, is there a kind of dataset where normalization is highly recommended to be applied?
I’m struggling with all this concepts and, although now I’m starting to understand them, I don’t know when I should apply each one.
I am struggling with reversing these transformations after predictions on the test sets. Is there a way in R to do the reversing the same way as Python’s fit_transform() and inverse_transform ()? Thank you
Would it be possible for you to edit the current or write a separate blog post on how to use caret to inverse the pre-processing transformations after prediction. It would greatly enhance this post
Hi Jason,
How can I see the results of the preProcess nn my input data? I just want to check what the different parameters do to my dataset (i.e., range vs scale).
Dear Jason!
If I use standalone method for pre-processing data, the pre-processing step will be performed again in the training step as a default option or not?
[1] “———————————- test fold[1] | length 430”
# from 862 863 864 865 866 867
### to 1286 1287 1288 1289 1290 1291
[1] “———————————- test fold[2] | length 430”
# from 1292 1293 1294 1295 1296 1297
### to 1716 1717 1718 1719 1720 1721
[1] “———————————- test fold[3] | length 430”
# from 1722 1723 1724 1725 1726 1727
### to 2146 2147 2148 2149 2150 2151
and so on…
Now caret starts train on “train fold[1]”. With “center+scale” it uses “train fold[1]” for getting the mean and the stdDev of ALL train data from “train fold[1]”.
Caret calculates mean & stdDev from the COMPLETE TRAIN data, so if caret trains a model – this model already knows the future mean and stdDev?
My models are over-fitting the train data, i know there are a lot of possible reasons. But can this also be caused by this kind of data-snooping? Is that data-snooping?
Could you write a post in which you perform the standardization technique into the “Your First Machine Learning Project in R Step-By-Step (tutorial and template for future projects)”.
Hii Jason , I am currently working on the loan dataset which has the attribute interest rate (entries in %) . I want to normalize the data . so how should i handle this attribute?
Is it required to remove percentage before normalization ?
Great stuff Jason! But do you have a tutorial on one hot encoding in R similar to the one you have written for Python? That would be great and would allow (me) to compare certain differences in the two languages.
Python has StandardScaler which seems to be a combination of Center and Scale:
output = (x – u)/s
Is that that the equivalent of where you have:
method=c(“center”, “scale”))
Thanks!
Is their any example you came across for KNN algorithm for mixed data types. How to handle factors(categorical data) with just 0 and 1. Gender “Male” and Female”. It is a classification problem.
Yes, typically a hamming distance or boolean (same == 0 different == 1) is useful for categorical data, which can be added to the euclidean distance for the normalized numerical values.
Thanks a lot for this amazing article, I was thinking if there is a possibility or a way of using median and median absolute deviation to center the data especially where there are extreme outliers. I haven’t seen any data normalization technique or routine explicitly catering to this phenomenon during the normalization process.
thank you for your sharing, i have on issue, every time, we conducted the center, there will be the negative values, which is not desired, but wheher this method is something errors in that
Hi Jason,
Thanks for this. How does one transform the preprocessed (eg scaled, centered, range) values back to the original after modelling and prediction. Is there a method for that in the caret package.
THanks.
Hi Jason,
What are you doing in the predict step below?
library(caret)
# load the dataset
data(iris)
# summarize data
summary(iris[,1:4])
# calculate the pre-process parameters from the dataset
preprocessParams <- preProcess(iris[,1:4], method=c("range"))
# summarize transform parameters
print(preprocessParams)
# transform the dataset using the parameters
transformed <- predict(preprocessParams, iris[,1:4])<—–What is being predicted here?
# summarize the transformed dataset
summary(transformed)
Hi,
Under the section Summary of Transform Methods, it is mentioned that,
“center“: divide values by standard deviation.
“scale“: subtract mean from values.
But when it comes to demonstration of methods, the functionality of center and scale is interchanged as,
1. Scale
The scale transform calculates the standard deviation for an attribute and divides each value by that standard deviation.
2. Center
The center transform calculates the mean for an attribute and subtracts it from each value.
So which is the correct functionality of these methods.
Quite right, I have fixed the typo. Sorry about that.
Centre: subtract mean from values.
Scale: divide values by standard deviation.
Hi Jason
What do you do if some of your variables are category or factors, will preprocessing ignore these?
Hi Michael, good question.
It may just ignore them, I believe that is the default behavior.
Hi Jason
I love your books, they are well written and
I am doing a Kaggle competition ie
https://www.kaggle.com/c/allstate-claims-severity
Alot of the predictors are categorical, which I have turned to factors etc ie a,b,c etc with a few continuous.
Could you please point me in the right direction re preprocessing this data with caret.
I want to use random forest on the model.
Cheers
Michael
Thnaks Michael.
Yes, I would recommend experimenting with turning them into binary variables, also called one hot encoding or dummy variables. You can learnmore here:
https://topepo.github.io/caret/pre-processing.html
Hi,
After the preprocessing, how i can transformed back my data to original values?
Kind regards,
Zoraze
Great question Zoraze.
You will need to do the transform manually, like normalize. Then use the same coefficients to inverse the transform later.
If you use caret, it can perform these operations automatically during training if needed.
Hello,
When should we be applying standardization ?
I am currently applying normalization ( variables expressed in different units). Should I apply standardization next ?
Great question Goran.
Strictly, standardization helps when your univariate distribution is Gaussian, the units vary across features and your algorithm assumes units do not vary and univariate distributions are Gaussian.
Results can be good or even better when you break this strict heuristic.
Generally, I recommend trying a suite of different data scaling and transforms for a problem in order to flush out what representations work well for your problem.
Moreover, is this the correct flow –
Outlier removal -> Impute missing values -> Data treatment (Normalize etc) -> Check for correlations ?
Looks good to me Goran!
Thank you, Jason.
Hi Jason,
I have some doubt for the flow: should we first check the correlations between predictors to see collinearity, then do data treatment (normalization, etc)? I read your book and it follows this flow. Thank you!
Yes.
Or perhaps perform treatment regardless and compare to a model fit on the raw data and see if it results in a lift in performance.
How is the preprocessing different when we have data from accelerometer signals which measure gait. For example the data consists of x,y,z which are the measures of the accelerometer, another column with miliseconds and a dependend variable which states the event, for example walking or sitting. In this case, do we have to create windows first and then start extracting features?
Consider taking a look in the literature to see how this type of data has been prepared in other experiments.
Hi Jason, I trained my NN model on pre-processed (normalized) data and then used the model to predict (the data I fed is also normalized). How do I convert the prediction results to be un-normalized so that it makes sense? Thanks.
Yes, you can inverse the scaling transform. Sorry I do not have an example on hand.
Hi Jason,
I used preProcess to standardize my train data set(data frame). Then, I used it and developed a model.
Then,to test my model on my test dataset(data frame), I need to standardize variables in test dataset with the same mean and std of variables in train dataset. (just, correct me if i am wrong!?) If so, Is there any package or method to do this?
I have examples of this in Python, but not R, sorry.
Dear Brownlee,
Thank you for recipes they helped me so much. I am asking if there are R recipes to use
ISOMAP (nonlinear ) preprocessing.
Thank you in advance
Moumtana
Perhaps check the documentation for the function:
?isomap
hi Jason, thanks for the tutorial, however i have one problem i am facing, after training and testing, i supply new inputs in standardized form to the neural networks to get new known outputs, these outputs are in standardized range as well, how do i get the outputs back to the original unstandardized range.
You can reverse the standardization process.
Hi Jason,
If I have a dataset of categorical variables in the same units but with vastly different values (like height at age 8 and age 25): is it recommendable to standardize them? I mean, I don’t know what to do because for all of them the unit represent the same distance.
Also, is there a kind of dataset where normalization is highly recommended to be applied?
I’m struggling with all this concepts and, although now I’m starting to understand them, I don’t know when I should apply each one.
Those sound like numerical and not categorical values.
Standardizing is a great idea if the variables are Gaussian, try it and see how it impacts model skill. If not, try normalizing.
Hi Jason,
I am struggling with reversing these transformations after predictions on the test sets. Is there a way in R to do the reversing the same way as Python’s fit_transform() and inverse_transform ()? Thank you
Often the caret package will handle this for you.
Would it be possible for you to edit the current or write a separate blog post on how to use caret to inverse the pre-processing transformations after prediction. It would greatly enhance this post
Thanks for the suggestion.
Hi Jason,
I just want to know what are the dimension reduction technique for SVM using R and Python.
Same as other methods, pca, svd, feature selection, sammons, som, tsne, etc.
Hi Jason,
How can I see the results of the preProcess nn my input data? I just want to check what the different parameters do to my dataset (i.e., range vs scale).
You may have to perform the data preparation operation separately.
Dear Jason!
If I use standalone method for pre-processing data, the pre-processing step will be performed again in the training step as a default option or not?
Not sure I understand, sorry, can you please elaborate?
Dear Jason, thank you for your great articles.
I have question regarding caret’s pre-processing in Timeslice-Mode.
We split our data in equal train/test folds like this:
1] “———————————- train fold[1] | length 861”
# from 1 2 3 4 5 6
### to 856 857 858 859 860 861
[1] “———————————- train fold[2] | length 861”
# from 431 432 433 434 435 436
### to 1286 1287 1288 1289 1290 1291
[1] “———————————- train fold[3] | length 861”
# from 861 862 863 864 865 866
### to 1716 1717 1718 1719 1720 1721
[1] “———————————- test fold[1] | length 430”
# from 862 863 864 865 866 867
### to 1286 1287 1288 1289 1290 1291
[1] “———————————- test fold[2] | length 430”
# from 1292 1293 1294 1295 1296 1297
### to 1716 1717 1718 1719 1720 1721
[1] “———————————- test fold[3] | length 430”
# from 1722 1723 1724 1725 1726 1727
### to 2146 2147 2148 2149 2150 2151
and so on…
Now caret starts train on “train fold[1]”. With “center+scale” it uses “train fold[1]” for getting the mean and the stdDev of ALL train data from “train fold[1]”.
Caret calculates mean & stdDev from the COMPLETE TRAIN data, so if caret trains a model – this model already knows the future mean and stdDev?
My models are over-fitting the train data, i know there are a lot of possible reasons. But can this also be caused by this kind of data-snooping? Is that data-snooping?
Thank you!
Not quite, it it uses k-fold cross validation. You can learn more here:
https://machinelearningmastery.com/k-fold-cross-validation/
Hi Jason,
Could you write a post in which you perform the standardization technique into the “Your First Machine Learning Project in R Step-By-Step (tutorial and template for future projects)”.
Kind redargs,
Mano Brown
Thanks for the suggestion Mano. Perhaps in the future.
Hii Jason , I am currently working on the loan dataset which has the attribute interest rate (entries in %) . I want to normalize the data . so how should i handle this attribute?
Is it required to remove percentage before normalization ?
Yes, machine learning algorithms must work with numbers.
Thanks a lot for all the effort on teaching me ML 🙂
How do I apply scaling information on Train data to Test set? Or we can independently scale Train and test?
I believe caret will manage this for you.
Thanks 🙂
And in python?
In Python, you can use a Pipeline:
https://machinelearningmastery.com/automate-machine-learning-workflows-pipelines-python-scikit-learn/
Great stuff Jason! But do you have a tutorial on one hot encoding in R similar to the one you have written for Python? That would be great and would allow (me) to compare certain differences in the two languages.
Best,
Shivendra
I don’t believe so, sorry.
Python has StandardScaler which seems to be a combination of Center and Scale:
output = (x – u)/s
Is that that the equivalent of where you have:
method=c(“center”, “scale”))
Thanks!
I believe so.
Hi Jason,
Is their any example you came across for KNN algorithm for mixed data types. How to handle factors(categorical data) with just 0 and 1. Gender “Male” and Female”. It is a classification problem.
Thanks
Bharat
Yes, typically a hamming distance or boolean (same == 0 different == 1) is useful for categorical data, which can be added to the euclidean distance for the normalized numerical values.
Thanks a lot for this amazing article, I was thinking if there is a possibility or a way of using median and median absolute deviation to center the data especially where there are extreme outliers. I haven’t seen any data normalization technique or routine explicitly catering to this phenomenon during the normalization process.
Sure, perhaps try it on your problem and see if it helps?
thank you for your sharing, i have on issue, every time, we conducted the center, there will be the negative values, which is not desired, but wheher this method is something errors in that
If you don’t want negative values, you can use normalization instead.
Hi Jason,
Thanks for this. How does one transform the preprocessed (eg scaled, centered, range) values back to the original after modelling and prediction. Is there a method for that in the caret package.
THanks.
Good question, I’m not sure off hand.
Hi Jason,
What are you doing in the predict step below?
library(caret)
# load the dataset
data(iris)
# summarize data
summary(iris[,1:4])
# calculate the pre-process parameters from the dataset
preprocessParams <- preProcess(iris[,1:4], method=c("range"))
# summarize transform parameters
print(preprocessParams)
# transform the dataset using the parameters
transformed <- predict(preprocessParams, iris[,1:4])<—–What is being predicted here?
# summarize the transformed dataset
summary(transformed)
We are transforming the input variables, specifically, we are scaling their values.