The machine learning model that we use to make predictions on new data is called the final model.

There can be confusion in applied machine learning about how to train a final model.

This error is seen with beginners to the field who ask questions such as:

*How do I predict with cross validation?**Which model do I choose from cross-validation?**Do I use the model after preparing it on the training dataset?*

This post will clear up the confusion.

In this post, you will discover how to finalize your machine learning model in order to make predictions on new data.

Let’s get started.

## What is a Final Model?

A final machine learning model is a model that you use to make predictions on new data.

That is, given new examples of input data, you want to use the model to predict the expected output. This may be a classification (assign a label) or a regression (a real value).

For example, whether the photo is a picture of a *dog* or a *cat*, or the estimated number of sales for tomorrow.

The goal of your machine learning project is to arrive at a final model that performs the best, where “best” is defined by:

**Data**: the historical data that you have available.**Time**: the time you have to spend on the project.**Procedure**: the data preparation steps, algorithm or algorithms, and the chosen algorithm configurations.

In your project, you gather the data, spend the time you have, and discover the data preparation procedures, algorithm to use, and how to configure it.

The final model is the pinnacle of this process, the end you seek in order to start actually making predictions.

## The Purpose of Train/Test Sets

Why do we use train and test sets?

Creating a train and test split of your dataset is one method to quickly evaluate the performance of an algorithm on your problem.

The training dataset is used to prepare a model, to train it.

We pretend the test dataset is new data where the output values are withheld from the algorithm. We gather predictions from the trained model on the inputs from the test dataset and compare them to the withheld output values of the test set.

Comparing the predictions and withheld outputs on the test dataset allows us to compute a performance measure for the model on the test dataset. This is an estimate of the skill of the algorithm trained on the problem when making predictions on unseen data.

### Let’s unpack this further

When we evaluate an algorithm, we are in fact evaluating all steps in the procedure, including how the training data was prepared (e.g. scaling), the choice of algorithm (e.g. kNN), and how the chosen algorithm was configured (e.g. k=3).

The performance measure calculated on the predictions is an estimate of the skill of the whole procedure.

We generalize the performance measure from:

- “
*the skill of the procedure on the*“**test set**

to

- “
*the skill of the procedure on*“.**unseen data**

This is quite a leap and requires that:

- The procedure is sufficiently robust that the estimate of skill is close to what we actually expect on unseen data.
- The choice of performance measure accurately captures what we are interested in measuring in predictions on unseen data.
- The choice of data preparation is well understood and repeatable on new data, and reversible if predictions need to be returned to their original scale or related to the original input values.
- The choice of algorithm makes sense for its intended use and operational environment (e.g. complexity or chosen programming language).

A lot rides on the estimated skill of the whole procedure on the test set.

In fact, using the train/test method of estimating the skill of the procedure on unseen data often has a high variance (unless we have a heck of a lot of data to split). This means that when it is repeated, it gives different results, often very different results.

The outcome is that we may be quite uncertain about how well the procedure actually performs on unseen data and how one procedure compares to another.

Often, time permitting, we prefer to use k-fold cross-validation instead.

## The Purpose of k-fold Cross Validation

Why do we use k-fold cross validation?

Cross-validation is another method to estimate the skill of a method on unseen data. Like using a train-test split.

Cross-validation systematically creates and evaluates multiple models on multiple subsets of the dataset.

This, in turn, provides a population of performance measures.

- We can calculate the mean of these measures to get an idea of how well the procedure performs on average.
- We can calculate the standard deviation of these measures to get an idea of how much the skill of the procedure is expected to vary in practice.

This is also helpful for providing a more nuanced comparison of one procedure to another when you are trying to choose which algorithm and data preparation procedures to use.

Also, this information is invaluable as you can use the mean and spread to give a confidence interval on the expected performance on a machine learning procedure in practice.

Both train-test splits and k-fold cross validation are examples of resampling methods.

## Why do we use Resampling Methods?

The problem with applied machine learning is that we are trying to model the unknown.

On a given predictive modeling problem, the ideal model is one that performs the best when making predictions on new data.

We don’t have new data, so we have to pretend with statistical tricks.

The train-test split and k-fold cross validation are called resampling methods. Resampling methods are statistical procedures for sampling a dataset and estimating an unknown quantity.

In the case of applied machine learning, we are interested in estimating the skill of a machine learning procedure on unseen data. More specifically, the skill of the predictions made by a machine learning procedure.

Once we have the estimated skill, we are finished with the resampling method.

- If you are using a train-test split, that means you can discard the split datasets and the trained model.
- If you are using k-fold cross-validation, that means you can throw away all of the trained models.

They have served their purpose and are no longer needed.

You are now ready to finalize your model.

## How to Finalize a Model?

You finalize a model by applying the chosen machine learning procedure on all of your data.

That’s it.

With the finalized model, you can:

- Save the model for later or operational use.
- Make predictions on new data.

What about the cross-validation models or the train-test datasets?

They’ve been discarded. They are no longer needed. They have served their purpose to help you choose a procedure to finalize.

## Common Questions

This section lists some common questions you might have.

### Why not keep the model trained on the training dataset?

and

### Why not keep the best model from the cross-validation?

You can if you like.

You may save time and effort by reusing one of the models trained during skill estimation.

This can be a big deal if it takes days, weeks, or months to train a model.

Your model will likely perform better when trained on all of the available data than just the subset used to estimate the performance of the model.

This is why we prefer to train the final model on all available data.

### Won’t the performance of the model trained on all of the data be different?

I think this question drives most of the misunderstanding around model finalization.

Put another way:

- If you train a model on all of the available data, then how do you know how well the model will perform?

You have already answered this question using the resampling procedure.

If well designed, the performance measures you calculate using train-test or k-fold cross validation suitably describe how well the finalized model trained on all available historical data will perform in general.

If you used k-fold cross validation, you will have an estimate of how “wrong” (or conversely, how “right”) the model will be on average, and the expected spread of that wrongness or rightness.

This is why the careful design of your test harness is so absolutely critical in applied machine learning. A more robust test harness will allow you to lean on the estimated performance all the more.

### Each time I train the model, I get a different performance score; should I pick the model with the best score?

Machine learning algorithms are stochastic and this behavior of different performance on the same data is to be expected.

Resampling methods like repeated train/test or repeated k-fold cross-validation will help to get a handle on how much variance there is in the method.

If it is a real concern, you can create multiple final models and take the mean from an ensemble of predictions in order to reduce the variance.

I talk more about this in the post:

## Summary

In this post, you discovered how to train a final machine learning model for operational use.

You have overcome obstacles to finalizing your model, such as:

- Understanding the goal of resampling procedures such as train-test splits and k-fold cross validation.
- Model finalization as training a new model on all available data.
- Separating the concern of estimating performance from finalizing the model.

Do you have another question or concern about finalizing your model that I have not addressed?

Ask in the comments and I will do my best to help.

Hi Jason,

Thank you for this very informative post. I have a question regarding the train-test split for classification problems: Can we perform a rain/test split in a stratified way for classification or does this introduce what is called data snooping (a biased estimate of test error)?

Thanks

Elie

The key is to ensure that fitting your model does not use any information about the test dataset, including min/max values if you are scaling.

Hi Jason,

Do you have any blog on How to Deploy the Final ML model

This post will give you some ideas:

http://machinelearningmastery.com/deploy-machine-learning-model-to-production/

How to save final model in Tenorflow and use it in Tenorflow.js

“Also, this information is invaluable as you can use the mean and spread to give a confidence interval on the expected performance on a machine learning procedure in practice.”

I have to assume a normal distribution for that right? But is this the always the case? Or should i normalize my data in a preprocessing step and then it would be correct to assume that? Thanks

Hi Dan, great question!

Yes, we are assuming results are Gaussian to report results using mean and standard deviation.

Repeating experiments and gathering info on the min, max and central tendency (median, percentiles) regardless of the distribution of results is a valuable exercise in reporting on model performance.

Great post….my little experience teached me that:

a) for classification you can use your final trained model with no risk

b) for regression, you have to rerun your model againt all data (using the parameters tuned during training)

b) specifically for time series regression, you can’t use normal cross validation – it should respect the cronology of the data (from old to new always) and you have to rerun your model againt all data (using the parameters tuned during training) as well, as the latest data are the crucial ones for the model to learn.

Cheers!

Thanks for the tips Kleyn.

Hi Jason

So, my question is how to predict the time series data not only for the test set but for further future forecast?

As I cannot used the saved model to predict the time series data.

Thank you

Fit a final model on all available data, save it, load it and use it to make predictions.

If new data becomes available, you must decide whether you want to refit the model or use it as is.

Great post! I really learned a lot from your post and applied it to my academic project. However, there are few questions still in my mind. In our project, we want to compare different machine algorithms with and without 10-fold cv, including logistics regression, SVM, random forest, and ANN. We can get the cv score of each model with 10-fold cross validation, but the problem is how can we get the final model with 10-fold? Does the cross-validation function as finding best parameter of the different model? (such determine k in kNN?) I am still a little bit confused about the purpose of cross-validation. Thanks

Hi Hank, the above directly answers this question.

Cross-validation is a tool to help you estimate the skill of models. We calculate these estimates so we can compare models and configs.

After we have chosen a model and it’s config, we throw away all of the CV models. We’re done estimating.

We can now fit the “final model” on all available data and use it to make predictions.

Does that make sense?

Please ask more questions if this in not clear. This is really important to understand and I thought I answered all of this in the post.

Hi Jason,

Thank you so much! Does that mean cross-validation is just a tool to help us compare different models based on cross-validation score?

After we are done with evaluation, we would apply original model to whole dataset and make predictions. Since I read a paper where the author compare auc, true positive rate, true negative rate, false positive rate and false negative rate between those models with and without cross-validation. It turns out that logistic regression with 10fold perform best. So I though we will apply logistics regression with 10-fold to test data. Is my understanding incorrectly? Thanks!

Yes, CV is just a tool to compare configs for a model or compare models.

Hi Jason,

Great post.

It took me awhile to get this but when the penny dropped about 18 months ago it was liberating. I liken cross validation to experimenting a process which you want to emulate against all your train data. One idea though.

When you cross validate you might say 10 folds of 3 repeats for each combination of parameters. Now say with whatever measure you are taking for accuracy you typically taken the mean from these 30. Is it sensible to bootstrap with replacement, particularly if it is not Gaussian, from this sample of 30 say 1000 times and from their calculate the median and 2.5/97.5 percentiles?

What does everyone else think!

PK

Yes, I like to use the bootstrap + empirical confidence intervals to report final model skill.

I have a post showing how to do this scheduled for later in the month.

Thanks for the very informative post. Just one question: When you train the final model, are you learning a completely new model or is some or all of the value of the previously learned models somehow retained?

Yes, generally, you are training an entirely new model. All the CV models are discarded.

Thanks Jason. Very Useful info & insight , helping lot to take right approach .

I’m glad it helped Muralidhar.

Thank you very much Jason. I found in this post answers to many questions.

I’m so glad to hear that Imene.

Hi Jason

I want tank you for this informative post . I m working in project “emotion recognition on image” I want to know how can I create my model and train it.

thanks in advance

I’m glad it helped issam.

Very informative, thanks alot, am also trying to see if this will be useful in a project I would like to do, and how it can be applied in biometrics and pattern recognition

Thanks.

Thanks for the article. What about the parameters. You will likely do tunning on the a development set or via cross-validation. The optimum parameter set you find is the best for that particular split or fold. Wouldn’t it be left to chance for our optimized parameters to be the optimum with the whole training data as well?

Hi Ras,

k-fold cross-validation is generally the best practice for using the training dataset to find a “good” configuration of a model.

Does that help? Is that clearer?

thanks for this post

i know this may be useful but i don’t know what we do in training phase using KNN

if u can write the details step that is done during training phase

i will be so grateful

There is no training of knn, only predictions.

See this post:

http://machinelearningmastery.com/tutorial-to-implement-k-nearest-neighbors-in-python-from-scratch/

Thanks for this post, it has given a clear explanation for most of my questions. However, I still have one question: if I have used undersampling duting CV, how should apply it to my whole data. To be clearer

– I have a training set of around 1 million positive (+) and 130 thousand negative (-) examples. I also have an independent test data set with a hundred thousand (+) and 4000 (-) examples.

– I have estimated performance with 10-fold CV and applied undersampling (I have used R gmlnet package, logit regression with LASSO, training for AUC). It gave me super results for the CV.

And now I’m lost a bit. Training for all data would mean to randomly select 130 thousand (+) from the 1 million and only use this ~260 thousand examples? Should I evaluate my model after training on my test data set?

Thank you for your help!

If you can, I would suggest evaluating the model on all data and see if skill improves.

In fact, it is a good idea to understand the data set size and model skill relationship to find the point of diminishing returns.

I have a question. In the training process using gausion naive bayes, can you say what are the steps to be taken to train the model.

Yes, see here:

http://machinelearningmastery.com/naive-bayes-classifier-scratch-python/

Hi Jason. Thanks for a great article!

When you say that “You finalize a model by applying the chosen machine learning procedure on all of your data”, does this mean that before deploying the model you should train a completely new model with the best hyperparameters from the validation phase, but now using training data + validation data + testing data, i.e. including the completely unseen testing data that you had never touched before?

This is how I interpret it, and it makes sense to me given that the whole the whole point of validation is to estimate the performance of a method of generating a model, rather than the performance of the model itself. Some people may argue, though, that because you’re now training on previously unseen data, it is impossible to know how the new trained model is actually performing and whether or not the new, real-world results will be in line with those estimated during validation and testing.

If I am interpreting this correctly, is there a good technical description anywhere for why this works in theory, or a good explanation for convincing people that this is the correct approach?

Yes. Correct.

Yes. The prior results are estimates of the performance of the final model in practice.

Thanks Jason. It’s great to have confirmation of that. Do you know of any published papers or sources out there that spell this out explicitly or go into the theory as to why this is theoretically sound?

Not off hand sorry.

Thanks Jason for this explanation. I would like to ask how to deal with test sets when I would like to compare the performance of my model to existing models. Do I have to hold out a test set, train my model on the remaining data and compare all models using my test set?

After that, can I merge this held out set to my original training set and use all data for training a final model?

What other solutions can be used?

Yes. Choose models based on skill on the test set. Then re-fit the model on all available data (if this makes sense for your chosen model and data).

Does that make sense?

Yes, it makes sense, thank you.

Great!

Thank you for the great post Jason.

I have a question about forecasting unseen data in RNN with LSTM.

I’ve built complete model using RNN with LSTM by using the post(http://machinelearningmastery.com/time-series-prediction-lstm-recurrent-neural-networks-python-keras).

How can we forecast unseen data(like ahead of current) from complete model?

I mean we don’t have any base data except time though.

I already saw some comments that you replied “You can make predictions on new data by calling Y = model.predict(X)” on that post. However, I couldn’t understand.. :'(

I mean in real-time. 🙂

Thanks in advance.

Best,

Paul

In real time, the same applies, but you can decide whether you re-train a new model, update the model or do nothing and just make predictions.

You can predict the next step beyond the available data by training the model on all current data, then calling predict with whatever input your model takes taken from the end of the training data.

Does that help?

Which part is confusing?

Hi Jason.

Thanks a lot for this great and informative post. I have 2 questions I would be thankful if you can help me with them:

1- Is that possible to refresh (update) a model without retraining it in full? To elaborate more, I have a model built using 9 weeks of data (weekly snapshots). As the size of the dataset is very large, when I want to update the model on a weekly basis, it takes a lot of time. Is that possible to update the model with the new snapshot (say for week 10), without retraining the model on the whole 10 weeks (9 old snapshots + 1 new snapshot)?

2- When I train my model and evaluate it using cross-validation, I get errors (or alternatively, I get AUCs) which are consistently better than what I get when I score serving data and test the real performance of the model. Why is that so, and how can I treat it? To elaborate more, taking the 9 snapshots explained in the first question, I use snapshot_date column as cross-validation fold column. Therefore, in each round, the algorithm uses 8 weeks of data for training, and test the model on the remaining unseen week. Therefore, I would end up with 9 different models and 9 different AUCs on the validation frame. All the AUCs are between 0.83 to 0.91. So I would expect that the real performance of the model built using whole data should be at minimum AUC 0.83. However, when I score the serving data, and the next week I assess the performance of the model, I see no better than AUC 0.78. I have experienced it for 3 weeks (3 times), so I don’t think it’s just random variation. Additionally, I am quite sure there is no data leakage and there is no future variable in my data. Also, I tune the model quite well and there is no overfitting.

Your help is highly appreciated.

You can update a model. The amount of updating depends on your data and domain. I have some posts on this and more in my LSTM book.

Model evaluation on test data is biased and optimistic generally. You may want to further refine your test harness to make the reported scores less biases for your specific dataset (e.g. fewer folds, more folds, more data, etc. depends on your dataset).

Hello Jason, very interesting this post! what do you get

I finished my model (with a score of 91%), but how do I do or how to evaluate this model with the new dataset?

I have saved the model in model.pkl but in my new data (for example iris.csv) How to predict the field “species”? (in my datasets do I need to put this field blank?) how is this step?

Thks for your help because I’m confused

Load the model and call model.predict(X) where X is your new input data.

Hi Jason – Great post. This cleared things for me in settling with a final model.

fitControl <- trainControl(

method = "repeatedcv",

number = 10,

savePredictions = 'final',

verboseIter = T,

summaryFunction = twoClassSummary,

classProbs = T)

glm_fit <- caret::train(dv ~. , data = dataset

,method = "glm", family=binomial, trControl = fitControl, metric = "ROC")

It says that the glm_fit now becomes the final model as it runs 10 fold based on trControl and finally trains model using entire data. Setting verboseIter = T, gives me a summary during this run a message at the end – "Fitting final model on full training set". So can I use this as a final model?

Perhaps.

Hi there,

This article makes a lot of sense, but one thing I am surprised was not addressed was the problem of over-fitting. If there is no test/validation data used in the final model generation, and if the model being used has been seen to over-fit the data in testing, then we need to know when to stop training without over-fitting. A simple approach would be to guess from the ‘correct’ training times from the previous tests, but of course the final model with all data will naturally need longer training times. Is there a statistical approach we could use to determine the best time to stop training without using a validation set?

Concerns of overfitting are addressed prior to finalizing the model as part of model selection.

If I was using early stopping during k-fold cross-validation, is it correct to average the number of epoch and apply it to the finalized model? Since there is no validation set in the finalized model for early stopping, so I thought of using average number of epoch to train the final model. Please help me 🙂

Yes, you could try that.

Hi Jason! Thank you so much for this informative post.

A little question though. If we don’t have the luxury to acquire another dataset (because it’s only for a little college project, for example), how do you apply k-fold cross validation (or test-training split) to evaluate models then?

My understanding is that once you apply, let’s say, k-fold cross validation for choosing which model to use and then tuning the parameters to suit your need, you will run your model on another different dataset hoping the model you have built and tuned will give you your expected result.

You can split your original dataset prior to using CV.

Hi Jason,

Glad to meet with your tutorial as these are one of the best in teaching deep wuth keras.

I have already read the notes which people asked you questions about using k-fold cv for training a final deep model but as I am a naive in working with deep learning models I could not understand some things.

I wanna train (or finalized) CNN,LSTM & RNN for text dataset (it is a sentiment analysis). In fact my teacher told me to apply k-fold cross validation for training=finalizing model to be able to predict the proability of belonging unseen data to each class (binary class).

my question is this:[ is it wrong to apply k-fold cross validation to train a final deep model?]

as I wrote commands 15 epoches run in each fold. is there any thing wrong with it?

I am so sorry for my naive question as i am not a english native to understand perfect the above comments U all wrote about it.

my written code is like this:

[

from sklearn.model_selection import KFold

kf = KFold(10)

f1_lstm_kfld=[]

oos_y_lstm = []

oos_pred_lstm = []

fold = 0

for train, test in kf.split(x_train):

fold += 1

print(“Fold #{}”.format(fold))

print(‘train’, train)

print(‘test’, test)

x_train1 = x_train[train]

y_train1 = y_train[train]

x_test1 = x_train[test]

y_test1 = y_train[test]

print(x_train1.shape, y_train1.shape, x_test1.shape, y_test1.shape)

print(‘Build model…’)

model_lstm = Sequential()

model_lstm.add(Embedding(vocab_dic_size, 128))

model_lstm.add(LSTM(128, dropout=0.2, recurrent_dropout=0.2))

model_lstm.add(Dense(1, activation=’sigmoid’))

model_lstm.compile(loss=’binary_crossentropy’,

optimizer=’adam’,

metrics=[‘accuracy’])

print(‘Train…’)

model_lstm.fit(x_train1, y_train1,

batch_size=32,

epochs=15,

validation_data=(x_test1, y_test1))

score_lstm, acc_lstm = model_lstm.evaluate(x_test1, y_test1,

batch_size=32)

sum_f1_lstm_kfld=0

for i in f1_lstm_kfld:

sum_f1_lstm_kfld =sum_f1_lstm_kfld+i

print (‘sum_f1_lstm_kfld’,sum_f1_lstm_kfld)

mean_f1_lstm_kfld=(sum_f1_lstm_kfld)/10

print (‘mean_f1_lstm_kfld’, mean_f1_lstm_kfld)

Please guide me as i get confused.

Thank U in advanced.

Rose

You cannot train a final model via CV.

I recommend re-reading the above post as to why.

Hi Jason,

I am afraid to ask about this issue again but when I re-read the above post , I saw this sentence: Resampling methods like repeated train/test or repeated k-fold cross-validation will help to get a handle on how much variance there is in the method. If it is a real concern, you can create multiple final models and take the mean from an ensemble of predictions in order to reduce the variance.

What do U mean by this sentence: “create multiple final models” do you mean applying k-fold cross-validation to achieve multiple models??

and also about this one: take the mean from an ensemble of predictions, you mean can we use “this take the mean” in finalizing model?

I wanna train a model=finalizing model.

You mentioned: “Why not keep the best model from the cross-validation?

You can if you like.You may save time and effort by reusing one of the models trained during skill estimation.Your model will likely perform better when trained on all of the available data than just the subset used to estimate the performance of the model.

what do u mean by saying above three sentences especially this one:”when trained on all of the available data than just the subset used to estimate the performance of the model?

you mean by “training on all of the available data” the procedure which we do not use k-fold cross validation?? and you mean applying k-fold cross validation from this sentence:”just the subset used to estimate the performance of the model”?

If I want to ask my question clearly I shold say in this way: [ I wanna train a CNN ,LSTM and RNN deep model to define a deep model inorder to estimating the proability of unseen data, what should I do? applying splitting data set into train and test or any other procedures?

Any guidance will be appreciate.

I mean, if there is a lot of variance in the model predictions (e.g. due to randomness in the model itself), you can train multiple final models on all training data and use them in an ensemble.

Sometimes training a model can take days or weeks. You may not want to retrain a model, hence, reuse a model from the time when you estimated model skill.

Does that help?

can u provide a sample code for prediction

I have many examples on my blog for different platforms.

Try a search and let me know if you don’t find what you’re looking for.

Do you happen to have an example with code where you train with all your data and then predict unknown future data?

Yes, you can learn how to make predictions on new data here:

https://machinelearningmastery.com/faq/single-faq/how-do-i-make-predictions

Thanks for this post Jason, and two additional questions:

1) Is there a peer-reviewed article that can be cited to demonstrate the validity of this approach?

2) Do I understand correctly that if the uncertainty in the relationship derived for the training data is correctly propagated to the test data set, the “best” model can be selected based solely on cross-validation statistics? That is, goodness of fit measure for the training relationship don’t really matter?

Thanks!

Of finalizing a model? There may be, I don’t know sorry. It might be tacit knowledge.

Yes, skill estimated using a well configured k-fold cross-validation may be sufficient, but i the score is reviewed too often (e.g. to tune hyperparams), you can still overfit.

this post is very helpful for my final project to get better understanding,

i have question, after done with train/test split, and next step is training with all available dataset. Do i have to use all of the dataset for data train no need to split again? and using the best hyperparameter or configurations from train/test split or cross validation before?

Thank you very much

I’m glad to hear that.

Correct, you would use all available data with hyperparameters chosen via testing on your train/test/validation sets.

Thanks Jason for the great post. I have one question though ..

During training I pre-process data e.g. scaling, feature reengineering etc. And then I train the model using train / validation /test set. Now I have the final model which I would like to use for prediction.

Now my prediction system is different (written using Java and TF) and there I import the trained model – incidentally all my training code are in Keras and Python. But in my prediction system I get the data points one at a time and I have to do prediction.

My question is how can I do the data pre-processing during prediction ? Pre-processing like scaling and feature extraction do not make sense on a single data point. With my use case prediction looks good if I accumulate all the data that I receive (unseen before), do similar pre-processing as in training, once I have quite a bit of them and then submit to the trained model for scoring. Otherwise I get very different and inaccurate results.

Would love to hear some suggestions on how to tackle this issue.

Excellent question!

The single data point must be prepared using the same methods used to prepare the training data.

Specifically, the coefficients for scaling (min/max or mean/stdev) are calculated on the training dataset, used to scale the training dataset, then used going forward to scale any points that you are predicting.

Does that help?

Hi Jason, Thank you for the awesome tutorial.

As I see, you emphasize on training a neural network on the entire data set without taking apart a sub-set of the whole dataset as a Test data set in order to train and finalizing a data set.

I have already trained cnn_model on the entire data set (( I mean I did not separate some samples for test set)) but I separate 20% of whole data set as the validation set via this statement:

‘model_cnn.fit(x_datasetpad, y_datasetpad, validation_split=0.2, epochs=5, batch_size=32)’

I think I made a mistake about putting ((validation_split=0.2)) in fitting network process.

Do I remove validation set to finalizing the cnn network??

Should I train a network on the entire data set { I mean Should I delete validation_split=0.2???}

Yes, remove the validation split for the final model.

Hi Jason,

I am so grateful for the quick answer.

No problem.

How to save final model in Tenorflow and use it in Tenorflow.js

Sorry, I don’t have tensorflow or tensorflow.js examples.

I Read Post, all Questions & answers.So finally want to summarise and approve from you 🙂

Ex: I have Training data 100k values, test data: 50k values.

1. We try various models like linear regr, decision tree, random forest, neural net on K fold validation with 150k values and see what model gives better performance measure(mean error etc…). We now decided what algorithm/procedure works best on data .

Ex: Decision Tree.

2. Now let us run K fold validation with 150k values on Decision tree with different hyperparamter values and check what value gives better performance measure.

3. we know what model and what hyperparameter “generally” works across the data.

4. Let us use all the data 150K values and train final Decision tree (FDT) with hyperparameter that we selected(which worked best) previously.

As model and hyperparameters are checked previously , the above post believes they will and should works best on the unseen data.

My thoughts: I might take a safer approach at the end by double checking, which means rather than train on all the data that i have i will keep 5% for testing (Unseen Data) and 95% for training.

Thank You for the Great Post . I thought this might help people who are concerned about hyper parameter tuning post model/ML procedure selection.

All good except the final check is redundant and could be misleading. What if skill on the 5% is poor, what do you do and why?

I thought to keep 5% as a double check but after your question i began to ponder what if skill is poor – I have two things to say.

1. This 5% is a sample that is not representative of data . i.e.. Occurred by chance. So i should have other approach to test on representative of the data.

2. Model is not good enough or over-fitted – Even this time i cannot come to conclusion as 5% sample may not be representative of data.

Understood finally that Cross Fold validation is solution for above 2 points which we already did on the whole data prior and so ” final check is redundant”.

Thank You So much Jason Brownlee.

Nice reasoning!

Keep an open mind and adapt methods for your specific problem. There are no “rules”, it is an empirical discipline.

Thank You. Understood.

Thank you Jason. I am trying to get probabilities of whether an employee is going to leave or stay with the company. I have 1,500 records of individuals that left the company and 500 that are currently with us. I need to get probabilities for all 500 associates that are still with us.

The issue is that the model is technically seeing all the data that it is training on in order to get probabilities for the entire data set. I don’t have “new” data I can apply the trained model to since we know all the employees that are currently with us. How do I get probabilities for the 500 current associates without overfitting? Is it as simple as making the predictors more generic? Thank you for your advice in advance.

You can fit models on some of your data and evaluate it on the rest.

Once you find a model that works, you can train it on all of your data and use it to make predictions on new data.

I assume you have historical records for people that have stayed or left, you train on that. Then you have people now that you want to know if they will stay or leave, this is new data for which you want to generate a prediction.

I have as ome problem.I am trying to predict currency exchange rte using historical data.I’m trying to predict tomorrow exchange rate using yesterday rate..I am littlebit confused.What artifical neural network should i used?And ilike to use K-fold cross validation to sampling..I like to know your ideas.

Don’t waste your time:

https://machinelearningmastery.com/faq/single-faq/can-you-help-me-with-machine-learning-for-finance-or-the-stock-market

Hi Jason,

This is a fantastic article. Cleared a lot of things.

I used to think that number of folds for k-fold is another hyperparameter to find the best model. For example we test two models with k=[3,5,7,9]. But after reading your post I guess that is pretty unnecessary as k fold validation does not choose a final model anyway.

So, do I just pick a single value of say, k=10 and run with it?

Thanks

Luv

Yes.

More details here:

https://machinelearningmastery.com/k-fold-cross-validation/

Hi Jason,

I have become more knowledgeable via your tutorial.

I wanna write a paper so i have to mention a valid resource for this point you mentioned below:

“You finalize a model by applying the chosen machine learning procedure on all of your data.”

in which paper or resource you have seen that we should apply whole data set for training model in order to finalizing it??

please gimme the paper or book as a resource.

waiting for the reply.

Best

Sarah

It is tacit knowledge, not written down in a paper.

Hello Jason,

How to Re-Train a model with new data which is already trained with another data?

Thanks,

KALYAN.

I have an example here:

https://machinelearningmastery.com/update-lstm-networks-training-time-series-forecasting/

Hi Jason,

Great post! Thanks!

A quick question here.

If I have a training dataset of n=500 instances.

Then I use 10-fold cross-validation and feature selection to identify an optimal machine learning algorithms based on this n=500 training dataset.

What would be a reasonable number of instances in the independent testing dataset that I can use to evaluate/test a performance of this machine learning algorithm on the un-seen dataset?

Thanks,

Albert Tu

It really depends on the problem.

Perhaps try some different sized dataset and evaluate the stability of their assessment?

Hello Jason,

thank you very much for your deep and crystal clear explanations.

I have one question about two modelling Setups:

Setup I:

– Split the entire data into a Training-Set (80%) and Test-Set (20%)

– Make a 10-Fold Cross Validation on the Training-Set to find the optimal parameter configuration

– Train the model with the determined parameter configuration on the Trainings-Set

– Final Evaluation of the model on the Testset (“Holdout”, untouched data)

Setup II:

– Split the entire data into a Training-Set (80%) and Test-Set (20%)

– Make a 10-Fold Cross Validation on the entire data set to find the optimal parameter configuration

– Train the model with the determined parameter configuration on the Trainings-Set

– Final Evaluation of the model on the Testset (“Holdout”, untouched data)

The only difference between setup I and II is that I make in setup I the CV on the Training-Set and in setup II I’m doing it on the entire dataset.

Which setup is do you think better or do you think both approaches are valid?

Thanks in advance!

Kind regards from Germany

Jonathan

There is no notion of better. Use an approach that gives you confidence in the findings of your experiment, enough that you can make decisions.

i would like to know how to measure the performance of the skill of the model using cross validation. since we have k different models how do we measure the performance ? do we get any aggregate score of performance on all models ? please explain me.. to be specific i use h20 to train my model

Average the performance across each model.

More here:

https://machinelearningmastery.com/k-fold-cross-validation/

Thank you for this very informative post.

You’re welcome Steve. I’m happy that it helped.

Why do you use the test data as the validation data in almost all of your examples.Are we not suppose to have two different test and validation data

Validation is often a subset of train, more here:

https://machinelearningmastery.com/difference-test-validation-datasets/

I try to keep my examples simple and often reuse test as validation.

Would it help to strengthen the ability of classifier to be trained on various data sets (such as from Kaggle, like stock data, car accidents, crime data etc.)…. While we do so, we may tune underlying algorithms or math construct to deal with different issues such as over-fit, low accuracy, etc.

This approach could be used to learn generic features for a class of problem. E.g. like unsupervised pre-training or an autoencoder.

I think this approach is the future for applied ML. I have a post scheduled on this approach being used in time series forecasting with an LSTM autoencoder. Very exciting stuff.

Hey Jason,

Very Very nice article. Archived it for my next project. Thanks for sharing such an informative articles.

I’m happy that it helped.

Thank you so much for your great article.

I understood that we should use all the data which we have to train our final model. However, when should I stop training when I train my final model with dataset including train+validation+test? Especially for a deep learning model. Let me explain it with an example:

I have a CNN model with 100,000 examples. I will do the following procedure:

1. I split this dataset into training data 80,000, validation data 10,000 and test data 10,000.

2. I used my validation dataset to guide my training and hyperparameter tuning. Here I used early stopping to prevent overfitting.

3. Then I got my best performance and hyperparameters. From early stopping setting, I got that when I trained my model 37 epochs, the losses were low and performance using test data to evaluate was good.

4 I will finalize my model, train my final model with all my 100,000 data.

Here is a problem. Without validation dataset, how can I know when I should stop training, that is how many epochs I should choose when I train my final models. Will I use the same epochs which are used before finalizing the model? or should I match the loss which I got before?

I think for the machine learning models without early stopping training, they are no problems. But like deep learning models, when to stop training is a critical issue. Any advice? Thanks.

Great question. It is really a design decision.

You can try to re-fit on all data, without early stopping by perhaps performing a sensitivity analysis on how many epochs are required on average.

You could sacrifice the a new validation set and refit a new final model on train-test.

There is no single answer, find an approach that makes the most sense for your project and what you know about your model performance and variance.

Thank you, Jason.

Did you have a chance to read this from Quora?

https://www.quora.com/Should-we-train-neural-networks-on-the-training-set-combined-with-the-validation-set-after-tuning

and also this one

https://stackoverflow.com/questions/39459203/combine-training-data-and-validation-data-how-to-select-hyper-parameters

What are your opinions? Thanks

At the end of the day it’s a design decision that you must make based on the specifics of your problem.

You are surely right! Thank you again.

Hi Jason Brownlee,

I am reading your tutorials and writing the code the understand the knots and bolts of it. Two reasons I do it. To understand Applied Machine Learning and how scikit-learn works. I had an extensive class on machine learning 1.5 years ago and getting back to my notes I feel like I understand most of the algorithms (I need workout statistics and probability as bit). My Calculus and Linear Algebra is fine. I am working as a student data scientist. My pandas and numpy knowledge would be the same as a beginner and I believe I would do good when using them per need. What would you recommend in such a situation how should I proceed with Data Science? I have my rough route but I would love to hear your comments.

Work through small projects and build up your skills.

Hi Jason, thank you for the wealth of info you’re sharing here! Such an awesome resource! One thing I’m having trouble with is selecting the most important variables to use in a final Random Forest model. I ran 5-fold cross-validation and was able to get feature importance values from each of the 5 models. It is acceptable to then take the average of the 5 importance values for each of the features and use that average to determine the top N features? If I then train a model on those top features and do another 5-fold cross-validation on that model, have I introduced leakage and risk overfitting? Is it better to just take the top N features using the feature importance of just one model from the initial cross-validation? Thanks for any input!

Generally, the random forest will perform the feature selection for you as part of building the model, I would not expect a lift from using feature importance to select features prior to modeling with RF.

Thank you, Dr. Jason, for the great website, I read many of your articles and learn a lot.

Regarding this article, you said: “Why not keep the best model from the cross-validation?

You can if you like.”

My questions are:

1- in python, how can I save one or the best model returned by cross-validation? as far as I know that the function cross_validation.cross_val_score (from scikit-learn package) don’t return a trained model, it just returns the scores? is there another package or function in python that return 10 trained model and I could save one of them?

2- What about R, does cross-validation return models and I can save any one of them? if yes, then how?

I read your articles related to cross-validation, I read books and I searched too, but I did not find answers yet.

some of the links that I find about this issue:

https://stackoverflow.com/questions/32700797/saving-a-cross-validation-trained-model-in-scikit

https://stats.stackexchange.com/questions/52274/how-to-choose-a-predictive-model-after-k-fold-cross-validation

appreciate your help

You can step over the folds manually and save the models, for example:

https://machinelearningmastery.com/k-fold-cross-validation/

Hi Jason,

Could you explain how nested CV comes into play here? And where in the process does hyperparameter tuning come into play as well?

Thanks!

All models created for evaluating using CV are discarded.

They occur prior to fitting a final model.

Hi Jason,

Good post. So far I have been using the train/validation/test split, and used the validation set to select hyper parameters and avoid overfitting and finally evaluated the model on the test set. Based on this post I guess I can do the same here and train the final model on all the data, or at least on the training set and validation set together? Because I waste a lot of data, not being used for training.

In k-fold CV you mention that overfitting are addressed prior to finalizing the model. Can you elaborate on this? Because there is no validation set to stop the training. I assume you still use the validation set to stop training, for each fold.

Do you have a post where you use both k-fold CV and the train/validation/test split and compare the results and derive a final model? If not, it would be very interesting.

Yes. Re-fit using same parameters on all data.

Not sure what you’re referring to. CV is for estimating model performance only. If you use early stopping, a validation dataset must be used, even with the final model.

Great suggestion, thanks.

Dear Jason,

first of all, thank you for helping!

I wonder if you can provide as some references regarding what you explained in this post.

It would be great!

Bye

It’s too practical to be written up in a paper, if that is what you mean. Academics don’t talk about a final model for use in operations.

Hello Jason,

Thanks for your great blog, it’s very helpful.

I have a question about this part “If you train a model on all of the available data, then how do you know how well the model will perform? You have already answered this question using the resampling procedure.””

1.So is that a correct procedure to train the model on all the available dataset and just use resampling method like k-fold to see the accuracy of our mode?

2.If we still split the dataset into train/test sets, test model accuracy and then on the other hand, also use k-fold on the train set. Is there any possibility that we get a big difference between the mean accuracy in K-fold and the model accuracy?

Thanks!

Yes.

There could be, it depends on the model and the data. Perhaps test how sensitive your model model is to changes in the size of your dataset.

hi, Jason,

I still wonder about k-fold cross validation. Support I have a dataset D. When I do k-fold cv, should I split D into training dataset and test dataest first, and then split the train dataset into k-fold? Or just split the whole dataset D into k-fold?

Another question is how to report final confuse matrix when using k-fold cv. Because I will get k confuse matrix.

You can learn more about how k-fold cross-validation works here:

https://machinelearningmastery.com/k-fold-cross-validation/

Confusion matrix can only calculated for a single test set, not cross validation.

Hi Jason,

Thanks for a great post.

I need a clarification with the following code.

ensemble = VotingClassifier(estimators=[

(model1), (model2), (model3)], voting=’hard’)

ensemble = ensemble.fit(X_train, Y_train)

predictions=ensemble.predict(X_validation)

As you have said in this post do I have to discard the X_train and Y_train subsets created using 10-fold Cross validation for fitting the ensemble model for making predictions or is this code correct. Do I need to use the entire dataset for fitting the ensemble model for making prediction.

Kindly help me.

The code appear to define a voting ensemble using 3 models.

It is often a good idea to use different datasets to fit the ensemble vs the submodels.

One approach involves using the out of sample data during cross validation.

Great post. It answered all my questions I couldn’t find in any other websites. Keep going!

Thanks, I’m happy it helped!

Hey Jason, nice article! Do you maybe know about more research on this topic? Any literature written on this, specifically for time series data?

No, it is an engineering consideration. E.g. how to use a model.

Hi Jason,

Great post. I’m fairly new to the machine learning and python and still learning. I have built few ML models, but once i retrain the model how do i deploy it.

Like I’m not sure how to proceed in terms of how to save the models, retrain the ML model with entire training set and how to deploy it by code on new/future test data. Greatly appreciated

If you are using sklearn, perhaps this will help:

https://machinelearningmastery.com/save-load-machine-learning-models-python-scikit-learn/

Jason,

Yes that partially helped. Part where I’m really stuck is, once i train/test my model, cross validate it and decide which model to use.

How do i go back and essentially retrain my model on the entire data (training data set, instead of just on train/test cut of the data)

Collect all of the data into a single dataset and call model.fit().

Perhaps I don’t understand the difficulty you are having?

Thanks for great tutorial, I have a question, I have some pictures with labels, I make 3 copies by adding Gaussian white noise to pictures and 3 copies by making non align pictures, initial dataset was centered on our image, then I shuffle them and split to test and train, accuracy is around one, my question is that good accuracy could be because of overlap in train and test dataset? should I do any refinment? or is ok

You cannot have copies of the same image in train and test, it would be an invalid evaluation of the model.

Data augmentation is only used on the training set.

Hi Jason,

Recently I am always reading your excellent posts since I a new ML learner. These posts are clear enough to convey useful information. In recent days, I am confused and struggled to understand how cross-validation works.

I have 500 datasets (may be a little), below is my procedure how I employ ML algorithms. Plz correct me if somewhere is wrong. It is binary classification question.

1. Collect data to form 500 datasets

2. Split the 500 datasets into training (80 %, 400) and test (20 %, 100) datasets

3. Use 10-fold cross-validation to check 9 machine learning algorithms (Logistic Regression, Support Vector Machine, Decision Tree, Random Forest, and AdaBoost) on whole 500 datasets. I find AdaBoost has the highest predictive accuracy from Step 3. This means that AdaBoost has more generalized capacity when applied to my question.

4. Then I use GridSearchCV to find the best hyperparameter for AdaBoost on training (80 %, 400) datasets. Using test (20 %, 100) datasets to test the AdaBoost model with best hyperparameter. I can get the accuracy score, confusion matrix, AUC-ROC, TPR, FPR, ROC curve, PR curve.

My question:

(1) For step 3, it is whole 500 datasets or training (80 %, 400) datasets?

(2) For step 4, do I need to finalize the model using whole 500 datasets? It is for academic question, not for industrial question.

Looks good.

Step 3 would just use the training set, step 4 would use the test/validation set. Finalizing is on all data.

More on train/test/validation datasets here:

https://machinelearningmastery.com/difference-test-validation-datasets/

Dear Jason

Request your opinion.

Assume we build an initial Model using,say 1000 features. The output of variables of importance lists only 800 of these 1000 features.

As I understand, while building final Model we would use the same parameters as earlier (e.g. optimal nrounds,etc..), but with complete data. I also use the same random seed as was used to build initial model.

But what about features, should we supply 1000 or 800 features (as obtained above) – while building final model?

Thanks

You would use the same framing of the problem (inputs and outputs) as was used during your experiment.

Hi. There is a publicly available dataset with pre-determined train(1,75,341) and test (82,332) splits. The performance of the model is always good with this experimental approach and I got 98% accuracy. However, in order to get a reliable assessment of the model, I combined train+test (2,57,673), then applied 80:20 split with cross validation and performed classification by stacking classifiers which resulted in an accuracy of 86%. Can I infer that although the accuracy is less, this method yields reliable predictions than the first method? Pls clarify.

Generally, it is a good idea to estimate the performance of a model across multiple train/test splits, ideally k-fold cross validation.

Thanks Jason very useful information on how to get your final model.

I am working on a churn predictive model. I already got my final model done. Do only thing that is a little confusing is when I use my model to predict with new data, how do I know what time period does predictions are likely to happen? If I use a years accumulation of data to train the model, and 6 months worth of new data to make my predictions. Should I expect my predictions to happen in the next 6 months?

NOTE: all of my predictor variables are averaged per month.

Thanks in advance,

*those instead of does

Not sure I follow, sorry. Can you elaborate?

For example, in my churn project if I am trying to predict clients that are likely to cancel. Lets say the model generated an output of 100 clients that are likely to cancel. Approximately when should I expect those clients to leave the company? Is it possible to know?

Thanks for your attention

It could be a time series classification task.

It could also be a survival analysis:

https://en.wikipedia.org/wiki/Survival_analysis

Thanks Jason I really appreciate it!

Hello Jason,

I went through all the 150+ reviews concerning this particular article.Based on the reviews I can easily understand that cross validation is i very important topic.

As far as my understanding goes cross validation is to relate all the independent variables amongst each other to check their importance regards to model building hope I am correct.But this in turn also refers a little bit of multicollinearity. Isnt it???

Can you please clarify.

Thanks.

Best Regards

Sandipan Sarkar

What do you mean exactly? I don’t follow sorry.

Jason,

According to this exchange it claims that you can’t report performance metrics of a model trained on a full data set using averages of the K-folds. Wouldn’t it be more appropriate to train on the full data set and then have a test data set to report the model performance?

https://stats.stackexchange.com/questions/184095/should-final-production-ready-model-be-trained-on-complete-data-or-just-on-tra

Your summary goes against many years of findings, and I’m not interested in debates.

My best advice is to prepare a test harness that give you and project stakeholders confidence in the estimate of model performance on new data.

Thank you for this great post, it’s very informative.

However, I can’t imagine how the K trained models be applied for predication.

I have also followed this post: https://machinelearningmastery.com/train-final-machine-learning-model/

It has mentioned that, “You finalize a model by applying the chosen machine learning procedure on all of your data.”

But I’m sorry that, I still have no idea how to finalize the model. Just pick the best model on the K trained model? Combine the result to build up a better model? Or anything else?

Can you give some practical example to finalize a model based on the K-Flod Cross Validation? Thanks.

The simple approach is once you have a algorithm and config that is reliable on your test harness/procedures, you use it and fit a model on all available data then use that model to start making predictions on new data.

If you have high variance in the final model (see seen on your test harness), you can reduce this variance by fitting K models on all training data, and using them together as an ensemble when making a prediction on new data.

Does that help?

Thanks for the explanation.

I just start self-study on machine leanring with python for few days, it seems that I have mixed up something in k -fold Cross Validation.

Before I hit the topic k-fold Cross Validation, and I learn by splitting the known data into training and testing data, and we can estimate the performance by compare the predicated value with testing data.

So, I guess the k-fold Cross Validation is used to find the best model by using different training and testing data group. it seems wrong.

Can I said that, the k-fold cross validation is used to compare between different algorithm / config? Then based on the score to choose the best algorithm, and build the model using this algorithm.

Thanks a lot.

k-fold cross validation is just another approach to estimate the performance of a model when making predictions on new data.

Just like a train/test split, but the result is less biased (optimistic).

You can learn more here:

https://machinelearningmastery.com/k-fold-cross-validation/

I’m little confused about k-fold CV in general.

Initially I thought its purpose for evaluation to validate the created model. It was about using one particular ML model and repeat its evaluation multiple times to get multiple different scores in order to get average score. This is good since it avoids a “lucky” situation which happens if model training and evaluation was done once.

But then here k-fold CV is used for finding hyperparameters (such as best ML technique to train model) before doing actual training or evaluation.

Also, in some other sources the k-fold CV is used in whole dataset (training set + test set), in other sources k-fold CV is used only on training set.

I am quite confused… I appreciate if you are able to clarify these things about k-fold CV.

You are correct. In all cases, it is used to estimate the performance of a model on a dataset.

https://machinelearningmastery.com/k-fold-cross-validation/

It can operate at different scales, across configs, across models, for one model. At some higher orders, we may need to hold back some additional data to validate with in order that we don’t overfit the model to the dataset.

You can learn about a validation dataset here:

https://machinelearningmastery.com/difference-test-validation-datasets/