The machine learning model that we use to make predictions on new data is called the final model.

There can be confusion in applied machine learning about how to train a final model.

This error is seen with beginners to the field who ask questions such as:

*How do I predict with cross validation?**Which model do I choose from cross-validation?**Do I use the model after preparing it on the training dataset?*

This post will clear up the confusion.

In this post, you will discover how to finalize your machine learning model in order to make predictions on new data.

Let’s get started.

## What is a Final Model?

A final machine learning model is a model that you use to make predictions on new data.

That is, given new examples of input data, you want to use the model to predict the expected output. This may be a classification (assign a label) or a regression (a real value).

For example, whether the photo is a picture of a *dog* or a *cat*, or the estimated number of sales for tomorrow.

The goal of your machine learning project is to arrive at a final model that performs the best, where “best” is defined by:

**Data**: the historical data that you have available.**Time**: the time you have to spend on the project.**Procedure**: the data preparation steps, algorithm or algorithms, and the chosen algorithm configurations.

In your project, you gather the data, spend the time you have, and discover the data preparation procedures, algorithm to use, and how to configure it.

The final model is the pinnacle of this process, the end you seek in order to start actually making predictions.

## The Purpose of Train/Test Sets

Why do we use train and test sets?

Creating a train and test split of your dataset is one method to quickly evaluate the performance of an algorithm on your problem.

The training dataset is used to prepare a model, to train it.

We pretend the test dataset is new data where the output values are withheld from the algorithm. We gather predictions from the trained model on the inputs from the test dataset and compare them to the withheld output values of the test set.

Comparing the predictions and withheld outputs on the test dataset allows us to compute a performance measure for the model on the test dataset. This is an estimate of the skill of the algorithm trained on the problem when making predictions on unseen data.

### Let’s unpack this further

When we evaluate an algorithm, we are in fact evaluating all steps in the procedure, including how the training data was prepared (e.g. scaling), the choice of algorithm (e.g. kNN), and how the chosen algorithm was configured (e.g. k=3).

The performance measure calculated on the predictions is an estimate of the skill of the whole procedure.

We generalize the performance measure from:

- “
*the skill of the procedure on the*“**test set**

to

- “
*the skill of the procedure on*“.**unseen data**

This is quite a leap and requires that:

- The procedure is sufficiently robust that the estimate of skill is close to what we actually expect on unseen data.
- The choice of performance measure accurately captures what we are interested in measuring in predictions on unseen data.
- The choice of data preparation is well understood and repeatable on new data, and reversible if predictions need to be returned to their original scale or related to the original input values.
- The choice of algorithm makes sense for its intended use and operational environment (e.g. complexity or chosen programming language).

A lot rides on the estimated skill of the whole procedure on the test set.

In fact, using the train/test method of estimating the skill of the procedure on unseen data often has a high variance (unless we have a heck of a lot of data to split). This means that when it is repeated, it gives different results, often very different results.

The outcome is that we may be quite uncertain about how well the procedure actually performs on unseen data and how one procedure compares to another.

Often, time permitting, we prefer to use k-fold cross-validation instead.

## The Purpose of k-fold Cross Validation

Why do we use k-fold cross validation?

Cross-validation is another method to estimate the skill of a method on unseen data. Like using a train-test split.

Cross-validation systematically creates and evaluates multiple models on multiple subsets of the dataset.

This, in turn, provides a population of performance measures.

- We can calculate the mean of these measures to get an idea of how well the procedure performs on average.
- We can calculate the standard deviation of these measures to get an idea of how much the skill of the procedure is expected to vary in practice.

This is also helpful for providing a more nuanced comparison of one procedure to another when you are trying to choose which algorithm and data preparation procedures to use.

Also, this information is invaluable as you can use the mean and spread to give a confidence interval on the expected performance on a machine learning procedure in practice.

Both train-test splits and k-fold cross validation are examples of resampling methods.

## Why do we use Resampling Methods?

The problem with applied machine learning is that we are trying to model the unknown.

On a given predictive modeling problem, the ideal model is one that performs the best when making predictions on new data.

We don’t have new data, so we have to pretend with statistical tricks.

The train-test split and k-fold cross validation are called resampling methods. Resampling methods are statistical procedures for sampling a dataset and estimating an unknown quantity.

In the case of applied machine learning, we are interested in estimating the skill of a machine learning procedure on unseen data. More specifically, the skill of the predictions made by a machine learning procedure.

Once we have the estimated skill, we are finished with the resampling method.

- If you are using a train-test split, that means you can discard the split datasets and the trained model.
- If you are using k-fold cross-validation, that means you can throw away all of the trained models.

They have served their purpose and are no longer needed.

You are now ready to finalize your model.

## How to Finalize a Model?

You finalize a model by applying the chosen machine learning procedure on all of your data.

That’s it.

With the finalized model, you can:

- Save the model for later or operational use.
- Make predictions on new data.

What about the cross-validation models or the train-test datasets?

They’ve been discarded. They are no longer needed. They have served their purpose to help you choose a procedure to finalize.

## Common Questions

This section lists some common questions you might have.

### Why not keep the model trained on the training dataset?

and

### Why not keep the best model from the cross-validation?

You can if you like.

You may save time and effort by reusing one of the models trained during skill estimation.

This can be a big deal if it takes days, weeks, or months to train a model.

Your model will likely perform better when trained on all of the available data than just the subset used to estimate the performance of the model.

This is why we prefer to train the final model on all available data.

### Won’t the performance of the model trained on all of the data be different?

I think this question drives most of the misunderstanding around model finalization.

Put another way:

- If you train a model on all of the available data, then how do you know how well the model will perform?

You have already answered this question using the resampling procedure.

If well designed, the performance measures you calculate using train-test or k-fold cross validation suitably describe how well the finalized model trained on all available historical data will perform in general.

If you used k-fold cross validation, you will have an estimate of how “wrong” (or conversely, how “right”) the model will be on average, and the expected spread of that wrongness or rightness.

This is why the careful design of your test harness is so absolutely critical in applied machine learning. A more robust test harness will allow you to lean on the estimated performance all the more.

### Each time I train the model, I get a different performance score; should I pick the model with the best score?

Machine learning algorithms are stochastic and this behavior of different performance on the same data is to be expected.

Resampling methods like repeated train/test or repeated k-fold cross-validation will help to get a handle on how much variance there is in the method.

If it is a real concern, you can create multiple final models and take the mean from an ensemble of predictions in order to reduce the variance.

I talk more about this in the post:

## Summary

In this post, you discovered how to train a final machine learning model for operational use.

You have overcome obstacles to finalizing your model, such as:

- Understanding the goal of resampling procedures such as train-test splits and k-fold cross validation.
- Model finalization as training a new model on all available data.
- Separating the concern of estimating performance from finalizing the model.

Do you have another question or concern about finalizing your model that I have not addressed?

Ask in the comments and I will do my best to help.

Hi Jason,

Thank you for this very informative post. I have a question regarding the train-test split for classification problems: Can we perform a rain/test split in a stratified way for classification or does this introduce what is called data snooping (a biased estimate of test error)?

Thanks

Elie

The key is to ensure that fitting your model does not use any information about the test dataset, including min/max values if you are scaling.

Hi Jason,

Do you have any blog on How to Deploy the Final ML model

This post will give you some ideas:

http://machinelearningmastery.com/deploy-machine-learning-model-to-production/

can you hide the comments show below all post on this website, or at least make them appear on demand, here there are more comments then the post content.

Good suggestion, thanks.

How to save final model in Tenorflow and use it in Tenorflow.js

“Also, this information is invaluable as you can use the mean and spread to give a confidence interval on the expected performance on a machine learning procedure in practice.”

I have to assume a normal distribution for that right? But is this the always the case? Or should i normalize my data in a preprocessing step and then it would be correct to assume that? Thanks

Hi Dan, great question!

Yes, we are assuming results are Gaussian to report results using mean and standard deviation.

Repeating experiments and gathering info on the min, max and central tendency (median, percentiles) regardless of the distribution of results is a valuable exercise in reporting on model performance.

Great post….my little experience teached me that:

a) for classification you can use your final trained model with no risk

b) for regression, you have to rerun your model againt all data (using the parameters tuned during training)

b) specifically for time series regression, you can’t use normal cross validation – it should respect the cronology of the data (from old to new always) and you have to rerun your model againt all data (using the parameters tuned during training) as well, as the latest data are the crucial ones for the model to learn.

Cheers!

Thanks for the tips Kleyn.

Hi Jason

So, my question is how to predict the time series data not only for the test set but for further future forecast?

As I cannot used the saved model to predict the time series data.

Thank you

Fit a final model on all available data, save it, load it and use it to make predictions.

If new data becomes available, you must decide whether you want to refit the model or use it as is.

Great post! I really learned a lot from your post and applied it to my academic project. However, there are few questions still in my mind. In our project, we want to compare different machine algorithms with and without 10-fold cv, including logistics regression, SVM, random forest, and ANN. We can get the cv score of each model with 10-fold cross validation, but the problem is how can we get the final model with 10-fold? Does the cross-validation function as finding best parameter of the different model? (such determine k in kNN?) I am still a little bit confused about the purpose of cross-validation. Thanks

Hi Hank, the above directly answers this question.

Cross-validation is a tool to help you estimate the skill of models. We calculate these estimates so we can compare models and configs.

After we have chosen a model and it’s config, we throw away all of the CV models. We’re done estimating.

We can now fit the “final model” on all available data and use it to make predictions.

Does that make sense?

Please ask more questions if this in not clear. This is really important to understand and I thought I answered all of this in the post.

Hi Jason,

Thank you so much! Does that mean cross-validation is just a tool to help us compare different models based on cross-validation score?

After we are done with evaluation, we would apply original model to whole dataset and make predictions. Since I read a paper where the author compare auc, true positive rate, true negative rate, false positive rate and false negative rate between those models with and without cross-validation. It turns out that logistic regression with 10fold perform best. So I though we will apply logistics regression with 10-fold to test data. Is my understanding incorrectly? Thanks!

Yes, CV is just a tool to compare configs for a model or compare models.

Hi Jason,

Still trying to wrap my head around the CV concept. So essentially, let’s say we have a data set and we want to use an xgboost model. So all CV really does is create 10 different variations of train-test splits of same data set, test them on 10 different model of xgboost, give you a mean performance score and then discard all the 10 models and variations of data sets that were created. After this, you sort of like start fresh and fit your model on the entire data set where the model has no past memory of it having being test on data historically. Is my understanding correct?

Correct. But we will now have an estimate of how well we expect the model or modeling pipeline to perform on average.

Hi Jason

I was under the impression that, during k-fold cross validation, one of the folds are used to tune the hyperparameters. The performance on this fold is therefore not equivalent to performance on unseen data since the unseen data is used to improve the model.

I would recommend that you use k-folding on your training set to not have a validation set that might not be representative of the entire data. But, this is not a real indication of your model performance, this is only a manner in which to obtain the best hyperparameters.

Once you have obtained these optimal hyperparameters, you retrain the model on the entire training data (the hold-out test set is still kept safely locked away). Then you report the performance on this hold-out test set.

If you fear that the test set is maybe not representative of the training set, you can do one of two things:

1) Compare the distribution of your test set values to that of your training set.

2) Repeat the entire training and testing procedure, described above, multiple times. Each time randomly selecting a training and testing set and report the average performance on these test sets.

Would you agree with the above?

You can tune the model within cv if you like, it is called nested cv and you can learn more here:

https://machinelearningmastery.com/nested-cross-validation-for-machine-learning-with-python/

Nested cv is not typical, it is a more advanced technique. A typical approach might be to first evaluate and compare algorithms, choose one, then tune it.

You can use any procedure you want – that gives you confidence you are choosing an effective model for your problem.

Hi Jason,

Great post.

It took me awhile to get this but when the penny dropped about 18 months ago it was liberating. I liken cross validation to experimenting a process which you want to emulate against all your train data. One idea though.

When you cross validate you might say 10 folds of 3 repeats for each combination of parameters. Now say with whatever measure you are taking for accuracy you typically taken the mean from these 30. Is it sensible to bootstrap with replacement, particularly if it is not Gaussian, from this sample of 30 say 1000 times and from their calculate the median and 2.5/97.5 percentiles?

What does everyone else think!

PK

Yes, I like to use the bootstrap + empirical confidence intervals to report final model skill.

I have a post showing how to do this scheduled for later in the month.

Awesome, thanks. Could you please clarify two more things-

1) How is cross validation score different from let’s say other performance metrics (confusion matrix/MSE)? What is the cross validation score based on as in what is it actually – the error? some type of unique metric?

2) In one of your articles that I later came across yesterday, you mentioned that during CV for lets say xgboost, if k=10, then 10 variations of the original dataset is tested on the same xgboost model. However, in my previous comment, my understanding was that 10 variations are tested on 10 models of xgboost and you did not refute that comment. So I’m just trying to understand is the same model exposed to 10 variations of dataset or are 10 different models exposed to each of the unique variation from K fold CV.

Thanks a lot,

You can learn more about cross-validation and how it works here:

https://machinelearningmastery.com/k-fold-cross-validation/

Thanks for the very informative post. Just one question: When you train the final model, are you learning a completely new model or is some or all of the value of the previously learned models somehow retained?

Yes, generally, you are training an entirely new model. All the CV models are discarded.

Thanks Jason. Very Useful info & insight , helping lot to take right approach .

I’m glad it helped Muralidhar.

Thank you very much Jason. I found in this post answers to many questions.

I’m so glad to hear that Imene.

Hi Jason

I want tank you for this informative post . I m working in project “emotion recognition on image” I want to know how can I create my model and train it.

thanks in advance

I’m glad it helped issam.

Very informative, thanks alot, am also trying to see if this will be useful in a project I would like to do, and how it can be applied in biometrics and pattern recognition

Thanks.

Thanks for the article. What about the parameters. You will likely do tunning on the a development set or via cross-validation. The optimum parameter set you find is the best for that particular split or fold. Wouldn’t it be left to chance for our optimized parameters to be the optimum with the whole training data as well?

Hi Ras,

k-fold cross-validation is generally the best practice for using the training dataset to find a “good” configuration of a model.

Does that help? Is that clearer?

thanks for this post

i know this may be useful but i don’t know what we do in training phase using KNN

if u can write the details step that is done during training phase

i will be so grateful

There is no training of knn, only predictions.

See this post:

http://machinelearningmastery.com/tutorial-to-implement-k-nearest-neighbors-in-python-from-scratch/

Thanks for this post, it has given a clear explanation for most of my questions. However, I still have one question: if I have used undersampling duting CV, how should apply it to my whole data. To be clearer

– I have a training set of around 1 million positive (+) and 130 thousand negative (-) examples. I also have an independent test data set with a hundred thousand (+) and 4000 (-) examples.

– I have estimated performance with 10-fold CV and applied undersampling (I have used R gmlnet package, logit regression with LASSO, training for AUC). It gave me super results for the CV.

And now I’m lost a bit. Training for all data would mean to randomly select 130 thousand (+) from the 1 million and only use this ~260 thousand examples? Should I evaluate my model after training on my test data set?

Thank you for your help!

If you can, I would suggest evaluating the model on all data and see if skill improves.

In fact, it is a good idea to understand the data set size and model skill relationship to find the point of diminishing returns.

I have a question. In the training process using gausion naive bayes, can you say what are the steps to be taken to train the model.

Yes, see here:

http://machinelearningmastery.com/naive-bayes-classifier-scratch-python/

Hi Jason. Thanks for a great article!

When you say that “You finalize a model by applying the chosen machine learning procedure on all of your data”, does this mean that before deploying the model you should train a completely new model with the best hyperparameters from the validation phase, but now using training data + validation data + testing data, i.e. including the completely unseen testing data that you had never touched before?

This is how I interpret it, and it makes sense to me given that the whole the whole point of validation is to estimate the performance of a method of generating a model, rather than the performance of the model itself. Some people may argue, though, that because you’re now training on previously unseen data, it is impossible to know how the new trained model is actually performing and whether or not the new, real-world results will be in line with those estimated during validation and testing.

If I am interpreting this correctly, is there a good technical description anywhere for why this works in theory, or a good explanation for convincing people that this is the correct approach?

Yes. Correct.

Yes. The prior results are estimates of the performance of the final model in practice.

Thanks Jason. It’s great to have confirmation of that. Do you know of any published papers or sources out there that spell this out explicitly or go into the theory as to why this is theoretically sound?

Not off hand sorry.

Thanks Jason for this explanation. I would like to ask how to deal with test sets when I would like to compare the performance of my model to existing models. Do I have to hold out a test set, train my model on the remaining data and compare all models using my test set?

After that, can I merge this held out set to my original training set and use all data for training a final model?

What other solutions can be used?

Yes. Choose models based on skill on the test set. Then re-fit the model on all available data (if this makes sense for your chosen model and data).

Does that make sense?

Yes, it makes sense, thank you.

Great!

Thank you for the great post Jason.

I have a question about forecasting unseen data in RNN with LSTM.

I’ve built complete model using RNN with LSTM by using the post(http://machinelearningmastery.com/time-series-prediction-lstm-recurrent-neural-networks-python-keras).

How can we forecast unseen data(like ahead of current) from complete model?

I mean we don’t have any base data except time though.

I already saw some comments that you replied “You can make predictions on new data by calling Y = model.predict(X)” on that post. However, I couldn’t understand.. :'(

I mean in real-time. 🙂

Thanks in advance.

Best,

Paul

In real time, the same applies, but you can decide whether you re-train a new model, update the model or do nothing and just make predictions.

You can predict the next step beyond the available data by training the model on all current data, then calling predict with whatever input your model takes taken from the end of the training data.

Does that help?

Which part is confusing?

Hi Jason.

Thanks a lot for this great and informative post. I have 2 questions I would be thankful if you can help me with them:

1- Is that possible to refresh (update) a model without retraining it in full? To elaborate more, I have a model built using 9 weeks of data (weekly snapshots). As the size of the dataset is very large, when I want to update the model on a weekly basis, it takes a lot of time. Is that possible to update the model with the new snapshot (say for week 10), without retraining the model on the whole 10 weeks (9 old snapshots + 1 new snapshot)?

2- When I train my model and evaluate it using cross-validation, I get errors (or alternatively, I get AUCs) which are consistently better than what I get when I score serving data and test the real performance of the model. Why is that so, and how can I treat it? To elaborate more, taking the 9 snapshots explained in the first question, I use snapshot_date column as cross-validation fold column. Therefore, in each round, the algorithm uses 8 weeks of data for training, and test the model on the remaining unseen week. Therefore, I would end up with 9 different models and 9 different AUCs on the validation frame. All the AUCs are between 0.83 to 0.91. So I would expect that the real performance of the model built using whole data should be at minimum AUC 0.83. However, when I score the serving data, and the next week I assess the performance of the model, I see no better than AUC 0.78. I have experienced it for 3 weeks (3 times), so I don’t think it’s just random variation. Additionally, I am quite sure there is no data leakage and there is no future variable in my data. Also, I tune the model quite well and there is no overfitting.

Your help is highly appreciated.

You can update a model. The amount of updating depends on your data and domain. I have some posts on this and more in my LSTM book.

Model evaluation on test data is biased and optimistic generally. You may want to further refine your test harness to make the reported scores less biases for your specific dataset (e.g. fewer folds, more folds, more data, etc. depends on your dataset).

Hello Jason, very interesting this post! what do you get

I finished my model (with a score of 91%), but how do I do or how to evaluate this model with the new dataset?

I have saved the model in model.pkl but in my new data (for example iris.csv) How to predict the field “species”? (in my datasets do I need to put this field blank?) how is this step?

Thks for your help because I’m confused

Load the model and call model.predict(X) where X is your new input data.

Hi Jason – Great post. This cleared things for me in settling with a final model.

fitControl <- trainControl(

method = "repeatedcv",

number = 10,

savePredictions = 'final',

verboseIter = T,

summaryFunction = twoClassSummary,

classProbs = T)

glm_fit <- caret::train(dv ~. , data = dataset

,method = "glm", family=binomial, trControl = fitControl, metric = "ROC")

It says that the glm_fit now becomes the final model as it runs 10 fold based on trControl and finally trains model using entire data. Setting verboseIter = T, gives me a summary during this run a message at the end – "Fitting final model on full training set". So can I use this as a final model?

Perhaps.

Hi there,

This article makes a lot of sense, but one thing I am surprised was not addressed was the problem of over-fitting. If there is no test/validation data used in the final model generation, and if the model being used has been seen to over-fit the data in testing, then we need to know when to stop training without over-fitting. A simple approach would be to guess from the ‘correct’ training times from the previous tests, but of course the final model with all data will naturally need longer training times. Is there a statistical approach we could use to determine the best time to stop training without using a validation set?

Concerns of overfitting are addressed prior to finalizing the model as part of model selection.

If I was using early stopping during k-fold cross-validation, is it correct to average the number of epoch and apply it to the finalized model? Since there is no validation set in the finalized model for early stopping, so I thought of using average number of epoch to train the final model. Please help me 🙂

Yes, you could try that.

Hi Jason! Thank you so much for this informative post.

A little question though. If we don’t have the luxury to acquire another dataset (because it’s only for a little college project, for example), how do you apply k-fold cross validation (or test-training split) to evaluate models then?

My understanding is that once you apply, let’s say, k-fold cross validation for choosing which model to use and then tuning the parameters to suit your need, you will run your model on another different dataset hoping the model you have built and tuned will give you your expected result.

You can split your original dataset prior to using CV.

Hi Jason,

Glad to meet with your tutorial as these are one of the best in teaching deep wuth keras.

I have already read the notes which people asked you questions about using k-fold cv for training a final deep model but as I am a naive in working with deep learning models I could not understand some things.

I wanna train (or finalized) CNN,LSTM & RNN for text dataset (it is a sentiment analysis). In fact my teacher told me to apply k-fold cross validation for training=finalizing model to be able to predict the proability of belonging unseen data to each class (binary class).

my question is this:[ is it wrong to apply k-fold cross validation to train a final deep model?]

as I wrote commands 15 epoches run in each fold. is there any thing wrong with it?

I am so sorry for my naive question as i am not a english native to understand perfect the above comments U all wrote about it.

my written code is like this:

[

from sklearn.model_selection import KFold

kf = KFold(10)

f1_lstm_kfld=[]

oos_y_lstm = []

oos_pred_lstm = []

fold = 0

for train, test in kf.split(x_train):

fold += 1

print(“Fold #{}”.format(fold))

print(‘train’, train)

print(‘test’, test)

x_train1 = x_train[train]

y_train1 = y_train[train]

x_test1 = x_train[test]

y_test1 = y_train[test]

print(x_train1.shape, y_train1.shape, x_test1.shape, y_test1.shape)

print(‘Build model…’)

model_lstm = Sequential()

model_lstm.add(Embedding(vocab_dic_size, 128))

model_lstm.add(LSTM(128, dropout=0.2, recurrent_dropout=0.2))

model_lstm.add(Dense(1, activation=’sigmoid’))

model_lstm.compile(loss=’binary_crossentropy’,

optimizer=’adam’,

metrics=[‘accuracy’])

print(‘Train…’)

model_lstm.fit(x_train1, y_train1,

batch_size=32,

epochs=15,

validation_data=(x_test1, y_test1))

score_lstm, acc_lstm = model_lstm.evaluate(x_test1, y_test1,

batch_size=32)

sum_f1_lstm_kfld=0

for i in f1_lstm_kfld:

sum_f1_lstm_kfld =sum_f1_lstm_kfld+i

print (‘sum_f1_lstm_kfld’,sum_f1_lstm_kfld)

mean_f1_lstm_kfld=(sum_f1_lstm_kfld)/10

print (‘mean_f1_lstm_kfld’, mean_f1_lstm_kfld)

Please guide me as i get confused.

Thank U in advanced.

Rose

You cannot train a final model via CV.

I recommend re-reading the above post as to why.

Hi Jason,

I am afraid to ask about this issue again but when I re-read the above post , I saw this sentence: Resampling methods like repeated train/test or repeated k-fold cross-validation will help to get a handle on how much variance there is in the method. If it is a real concern, you can create multiple final models and take the mean from an ensemble of predictions in order to reduce the variance.

What do U mean by this sentence: “create multiple final models” do you mean applying k-fold cross-validation to achieve multiple models??

and also about this one: take the mean from an ensemble of predictions, you mean can we use “this take the mean” in finalizing model?

I wanna train a model=finalizing model.

You mentioned: “Why not keep the best model from the cross-validation?

You can if you like.You may save time and effort by reusing one of the models trained during skill estimation.Your model will likely perform better when trained on all of the available data than just the subset used to estimate the performance of the model.

what do u mean by saying above three sentences especially this one:”when trained on all of the available data than just the subset used to estimate the performance of the model?

you mean by “training on all of the available data” the procedure which we do not use k-fold cross validation?? and you mean applying k-fold cross validation from this sentence:”just the subset used to estimate the performance of the model”?

If I want to ask my question clearly I shold say in this way: [ I wanna train a CNN ,LSTM and RNN deep model to define a deep model inorder to estimating the proability of unseen data, what should I do? applying splitting data set into train and test or any other procedures?

Any guidance will be appreciate.

I mean, if there is a lot of variance in the model predictions (e.g. due to randomness in the model itself), you can train multiple final models on all training data and use them in an ensemble.

Sometimes training a model can take days or weeks. You may not want to retrain a model, hence, reuse a model from the time when you estimated model skill.

Does that help?

can u provide a sample code for prediction

I have many examples on my blog for different platforms.

Try a search and let me know if you don’t find what you’re looking for.

Do you happen to have an example with code where you train with all your data and then predict unknown future data?

Yes, you can learn how to make predictions on new data here:

https://machinelearningmastery.com/faq/single-faq/how-do-i-make-predictions

Thanks for this post Jason, and two additional questions:

1) Is there a peer-reviewed article that can be cited to demonstrate the validity of this approach?

2) Do I understand correctly that if the uncertainty in the relationship derived for the training data is correctly propagated to the test data set, the “best” model can be selected based solely on cross-validation statistics? That is, goodness of fit measure for the training relationship don’t really matter?

Thanks!

Of finalizing a model? There may be, I don’t know sorry. It might be tacit knowledge.

Yes, skill estimated using a well configured k-fold cross-validation may be sufficient, but i the score is reviewed too often (e.g. to tune hyperparams), you can still overfit.

this post is very helpful for my final project to get better understanding,

i have question, after done with train/test split, and next step is training with all available dataset. Do i have to use all of the dataset for data train no need to split again? and using the best hyperparameter or configurations from train/test split or cross validation before?

Thank you very much

I’m glad to hear that.

Correct, you would use all available data with hyperparameters chosen via testing on your train/test/validation sets.

Thanks Jason for the great post. I have one question though ..

During training I pre-process data e.g. scaling, feature reengineering etc. And then I train the model using train / validation /test set. Now I have the final model which I would like to use for prediction.

Now my prediction system is different (written using Java and TF) and there I import the trained model – incidentally all my training code are in Keras and Python. But in my prediction system I get the data points one at a time and I have to do prediction.

My question is how can I do the data pre-processing during prediction ? Pre-processing like scaling and feature extraction do not make sense on a single data point. With my use case prediction looks good if I accumulate all the data that I receive (unseen before), do similar pre-processing as in training, once I have quite a bit of them and then submit to the trained model for scoring. Otherwise I get very different and inaccurate results.

Would love to hear some suggestions on how to tackle this issue.

Excellent question!

The single data point must be prepared using the same methods used to prepare the training data.

Specifically, the coefficients for scaling (min/max or mean/stdev) are calculated on the training dataset, used to scale the training dataset, then used going forward to scale any points that you are predicting.

Does that help?

Hi Jason, Thank you for the awesome tutorial.

As I see, you emphasize on training a neural network on the entire data set without taking apart a sub-set of the whole dataset as a Test data set in order to train and finalizing a data set.

I have already trained cnn_model on the entire data set (( I mean I did not separate some samples for test set)) but I separate 20% of whole data set as the validation set via this statement:

‘model_cnn.fit(x_datasetpad, y_datasetpad, validation_split=0.2, epochs=5, batch_size=32)’

I think I made a mistake about putting ((validation_split=0.2)) in fitting network process.

Do I remove validation set to finalizing the cnn network??

Should I train a network on the entire data set { I mean Should I delete validation_split=0.2???}

Yes, remove the validation split for the final model.

Hi Jason,

I am so grateful for the quick answer.

No problem.

How to save final model in Tenorflow and use it in Tenorflow.js

Sorry, I don’t have tensorflow or tensorflow.js examples.

I Read Post, all Questions & answers.So finally want to summarise and approve from you 🙂

Ex: I have Training data 100k values, test data: 50k values.

1. We try various models like linear regr, decision tree, random forest, neural net on K fold validation with 150k values and see what model gives better performance measure(mean error etc…). We now decided what algorithm/procedure works best on data .

Ex: Decision Tree.

2. Now let us run K fold validation with 150k values on Decision tree with different hyperparamter values and check what value gives better performance measure.

3. we know what model and what hyperparameter “generally” works across the data.

4. Let us use all the data 150K values and train final Decision tree (FDT) with hyperparameter that we selected(which worked best) previously.

As model and hyperparameters are checked previously , the above post believes they will and should works best on the unseen data.

My thoughts: I might take a safer approach at the end by double checking, which means rather than train on all the data that i have i will keep 5% for testing (Unseen Data) and 95% for training.

Thank You for the Great Post . I thought this might help people who are concerned about hyper parameter tuning post model/ML procedure selection.

All good except the final check is redundant and could be misleading. What if skill on the 5% is poor, what do you do and why?

I thought to keep 5% as a double check but after your question i began to ponder what if skill is poor – I have two things to say.

1. This 5% is a sample that is not representative of data . i.e.. Occurred by chance. So i should have other approach to test on representative of the data.

2. Model is not good enough or over-fitted – Even this time i cannot come to conclusion as 5% sample may not be representative of data.

Understood finally that Cross Fold validation is solution for above 2 points which we already did on the whole data prior and so ” final check is redundant”.

Thank You So much Jason Brownlee.

Nice reasoning!

Keep an open mind and adapt methods for your specific problem. There are no “rules”, it is an empirical discipline.

Thank You. Understood.

In Vaddi Ajay Kumar’s step 1 and step 2 with cross validation, different algorithms(step 1) and different hyper-parameters (step 2) with different data. Questtons are:

1.Why not just use train/test split method?at least ensure use the same train data and test data. What is the advantage of cross validation against train/test method?

2.I know K shoud not be too small or too big, Some books recommend 10-fold (no mention how much optional values), then how to choose train/test in 10-fold cross validation (given each part named 0,1,2…9 ) if the optional hyper-parameter has only 5 values according to prior? If it is not right, how to choose K if the optional hyper-parameter has only 5 values according to prior? and 20 values?

K for k-CV is unrelated to the hyperparameters.

CV or repeated CV give a less biased estimate of model skill than a single train/test split. K=10 has been found to be a good trade-off (less optimistic) when tested by many people on many differently sized dataset.

@Vaddi Ajay Kumar, why did you split your data in train and test set if you always use 150k that is the sum of the two to do your computation? For what do you use the test set? Thanks.

Thank you Jason. I am trying to get probabilities of whether an employee is going to leave or stay with the company. I have 1,500 records of individuals that left the company and 500 that are currently with us. I need to get probabilities for all 500 associates that are still with us.

The issue is that the model is technically seeing all the data that it is training on in order to get probabilities for the entire data set. I don’t have “new” data I can apply the trained model to since we know all the employees that are currently with us. How do I get probabilities for the 500 current associates without overfitting? Is it as simple as making the predictors more generic? Thank you for your advice in advance.

You can fit models on some of your data and evaluate it on the rest.

Once you find a model that works, you can train it on all of your data and use it to make predictions on new data.

I assume you have historical records for people that have stayed or left, you train on that. Then you have people now that you want to know if they will stay or leave, this is new data for which you want to generate a prediction.

I have as ome problem.I am trying to predict currency exchange rte using historical data.I’m trying to predict tomorrow exchange rate using yesterday rate..I am littlebit confused.What artifical neural network should i used?And ilike to use K-fold cross validation to sampling..I like to know your ideas.

Don’t waste your time:

https://machinelearningmastery.com/faq/single-faq/can-you-help-me-with-machine-learning-for-finance-or-the-stock-market

Hi Jason,

This is a fantastic article. Cleared a lot of things.

I used to think that number of folds for k-fold is another hyperparameter to find the best model. For example we test two models with k=[3,5,7,9]. But after reading your post I guess that is pretty unnecessary as k fold validation does not choose a final model anyway.

So, do I just pick a single value of say, k=10 and run with it?

Thanks

Luv

Yes.

More details here:

https://machinelearningmastery.com/k-fold-cross-validation/

Hi Jason,

I have become more knowledgeable via your tutorial.

I wanna write a paper so i have to mention a valid resource for this point you mentioned below:

“You finalize a model by applying the chosen machine learning procedure on all of your data.”

in which paper or resource you have seen that we should apply whole data set for training model in order to finalizing it??

please gimme the paper or book as a resource.

waiting for the reply.

Best

Sarah

It is tacit knowledge, not written down in a paper.

Hello Jason,

How to Re-Train a model with new data which is already trained with another data?

Thanks,

KALYAN.

I have an example here:

https://machinelearningmastery.com/update-lstm-networks-training-time-series-forecasting/

Hi Jason,

Great post! Thanks!

A quick question here.

If I have a training dataset of n=500 instances.

Then I use 10-fold cross-validation and feature selection to identify an optimal machine learning algorithms based on this n=500 training dataset.

What would be a reasonable number of instances in the independent testing dataset that I can use to evaluate/test a performance of this machine learning algorithm on the un-seen dataset?

Thanks,

Albert Tu

It really depends on the problem.

Perhaps try some different sized dataset and evaluate the stability of their assessment?

Hello Jason,

thank you very much for your deep and crystal clear explanations.

I have one question about two modelling Setups:

Setup I:

– Split the entire data into a Training-Set (80%) and Test-Set (20%)

– Make a 10-Fold Cross Validation on the Training-Set to find the optimal parameter configuration

– Train the model with the determined parameter configuration on the Trainings-Set

– Final Evaluation of the model on the Testset (“Holdout”, untouched data)

Setup II:

– Split the entire data into a Training-Set (80%) and Test-Set (20%)

– Make a 10-Fold Cross Validation on the entire data set to find the optimal parameter configuration

– Train the model with the determined parameter configuration on the Trainings-Set

– Final Evaluation of the model on the Testset (“Holdout”, untouched data)

The only difference between setup I and II is that I make in setup I the CV on the Training-Set and in setup II I’m doing it on the entire dataset.

Which setup is do you think better or do you think both approaches are valid?

Thanks in advance!

Kind regards from Germany

Jonathan

There is no notion of better. Use an approach that gives you confidence in the findings of your experiment, enough that you can make decisions.

i would like to know how to measure the performance of the skill of the model using cross validation. since we have k different models how do we measure the performance ? do we get any aggregate score of performance on all models ? please explain me.. to be specific i use h20 to train my model

Average the performance across each model.

More here:

https://machinelearningmastery.com/k-fold-cross-validation/

Thank you for this very informative post.

You’re welcome Steve. I’m happy that it helped.

Why do you use the test data as the validation data in almost all of your examples.Are we not suppose to have two different test and validation data

Validation is often a subset of train, more here:

https://machinelearningmastery.com/difference-test-validation-datasets/

I try to keep my examples simple and often reuse test as validation.

Would it help to strengthen the ability of classifier to be trained on various data sets (such as from Kaggle, like stock data, car accidents, crime data etc.)…. While we do so, we may tune underlying algorithms or math construct to deal with different issues such as over-fit, low accuracy, etc.

This approach could be used to learn generic features for a class of problem. E.g. like unsupervised pre-training or an autoencoder.

I think this approach is the future for applied ML. I have a post scheduled on this approach being used in time series forecasting with an LSTM autoencoder. Very exciting stuff.

Hey Jason,

Very Very nice article. Archived it for my next project. Thanks for sharing such an informative articles.

I’m happy that it helped.

Thank you so much for your great article.

I understood that we should use all the data which we have to train our final model. However, when should I stop training when I train my final model with dataset including train+validation+test? Especially for a deep learning model. Let me explain it with an example:

I have a CNN model with 100,000 examples. I will do the following procedure:

1. I split this dataset into training data 80,000, validation data 10,000 and test data 10,000.

2. I used my validation dataset to guide my training and hyperparameter tuning. Here I used early stopping to prevent overfitting.

3. Then I got my best performance and hyperparameters. From early stopping setting, I got that when I trained my model 37 epochs, the losses were low and performance using test data to evaluate was good.

4 I will finalize my model, train my final model with all my 100,000 data.

Here is a problem. Without validation dataset, how can I know when I should stop training, that is how many epochs I should choose when I train my final models. Will I use the same epochs which are used before finalizing the model? or should I match the loss which I got before?

I think for the machine learning models without early stopping training, they are no problems. But like deep learning models, when to stop training is a critical issue. Any advice? Thanks.

Great question. It is really a design decision.

You can try to re-fit on all data, without early stopping by perhaps performing a sensitivity analysis on how many epochs are required on average.

You could sacrifice the a new validation set and refit a new final model on train-test.

There is no single answer, find an approach that makes the most sense for your project and what you know about your model performance and variance.

Thank you, Jason.

Did you have a chance to read this from Quora?

https://www.quora.com/Should-we-train-neural-networks-on-the-training-set-combined-with-the-validation-set-after-tuning

and also this one

https://stackoverflow.com/questions/39459203/combine-training-data-and-validation-data-how-to-select-hyper-parameters

What are your opinions? Thanks

At the end of the day it’s a design decision that you must make based on the specifics of your problem.

You are surely right! Thank you again.

Hi Jason Brownlee,

I am reading your tutorials and writing the code the understand the knots and bolts of it. Two reasons I do it. To understand Applied Machine Learning and how scikit-learn works. I had an extensive class on machine learning 1.5 years ago and getting back to my notes I feel like I understand most of the algorithms (I need workout statistics and probability as bit). My Calculus and Linear Algebra is fine. I am working as a student data scientist. My pandas and numpy knowledge would be the same as a beginner and I believe I would do good when using them per need. What would you recommend in such a situation how should I proceed with Data Science? I have my rough route but I would love to hear your comments.

Work through small projects and build up your skills.

Hi Jason, thank you for the wealth of info you’re sharing here! Such an awesome resource! One thing I’m having trouble with is selecting the most important variables to use in a final Random Forest model. I ran 5-fold cross-validation and was able to get feature importance values from each of the 5 models. It is acceptable to then take the average of the 5 importance values for each of the features and use that average to determine the top N features? If I then train a model on those top features and do another 5-fold cross-validation on that model, have I introduced leakage and risk overfitting? Is it better to just take the top N features using the feature importance of just one model from the initial cross-validation? Thanks for any input!

Generally, the random forest will perform the feature selection for you as part of building the model, I would not expect a lift from using feature importance to select features prior to modeling with RF.

Hi Taylor, I know it is quite late now. But I think in your case, assume you do not use RF (SVM for example), I think your approach to determine the top N features is introducing leakage. You should not do CV again.

My question here is, without doing CV again, can we just take those top N features as a feature selection method, skip CV, and jump right into test phase? Is it valid?

Thank you, Dr. Jason, for the great website, I read many of your articles and learn a lot.

Regarding this article, you said: “Why not keep the best model from the cross-validation?

You can if you like.”

My questions are:

1- in python, how can I save one or the best model returned by cross-validation? as far as I know that the function cross_validation.cross_val_score (from scikit-learn package) don’t return a trained model, it just returns the scores? is there another package or function in python that return 10 trained model and I could save one of them?

2- What about R, does cross-validation return models and I can save any one of them? if yes, then how?

I read your articles related to cross-validation, I read books and I searched too, but I did not find answers yet.

some of the links that I find about this issue:

https://stackoverflow.com/questions/32700797/saving-a-cross-validation-trained-model-in-scikit

https://stats.stackexchange.com/questions/52274/how-to-choose-a-predictive-model-after-k-fold-cross-validation

appreciate your help

You can step over the folds manually and save the models, for example:

https://machinelearningmastery.com/k-fold-cross-validation/

Hi Jason,

Could you explain how nested CV comes into play here? And where in the process does hyperparameter tuning come into play as well?

Thanks!

All models created for evaluating using CV are discarded.

They occur prior to fitting a final model.

Hi Jason,

This post was the best that I found in the Internet, Thank you very much.

However, I have the same problem. Why do some people use nested (double) CV then?

Thank you very much for your informative posts!

Thanks.

CV and nested CV are all used to find a model and set of configurations.

After that you can fit a final model on the chosen model and config.

Hi Jason,

Good post. So far I have been using the train/validation/test split, and used the validation set to select hyper parameters and avoid overfitting and finally evaluated the model on the test set. Based on this post I guess I can do the same here and train the final model on all the data, or at least on the training set and validation set together? Because I waste a lot of data, not being used for training.

In k-fold CV you mention that overfitting are addressed prior to finalizing the model. Can you elaborate on this? Because there is no validation set to stop the training. I assume you still use the validation set to stop training, for each fold.

Do you have a post where you use both k-fold CV and the train/validation/test split and compare the results and derive a final model? If not, it would be very interesting.

Yes. Re-fit using same parameters on all data.

Not sure what you’re referring to. CV is for estimating model performance only. If you use early stopping, a validation dataset must be used, even with the final model.

Great suggestion, thanks.

Dear Jason,

first of all, thank you for helping!

I wonder if you can provide as some references regarding what you explained in this post.

It would be great!

Bye

It’s too practical to be written up in a paper, if that is what you mean. Academics don’t talk about a final model for use in operations.

Hello Jason,

Thanks for your great blog, it’s very helpful.

I have a question about this part “If you train a model on all of the available data, then how do you know how well the model will perform? You have already answered this question using the resampling procedure.””

1.So is that a correct procedure to train the model on all the available dataset and just use resampling method like k-fold to see the accuracy of our mode?

2.If we still split the dataset into train/test sets, test model accuracy and then on the other hand, also use k-fold on the train set. Is there any possibility that we get a big difference between the mean accuracy in K-fold and the model accuracy?

Thanks!

Yes.

There could be, it depends on the model and the data. Perhaps test how sensitive your model model is to changes in the size of your dataset.

hi, Jason,

I still wonder about k-fold cross validation. Support I have a dataset D. When I do k-fold cv, should I split D into training dataset and test dataest first, and then split the train dataset into k-fold? Or just split the whole dataset D into k-fold?

Another question is how to report final confuse matrix when using k-fold cv. Because I will get k confuse matrix.

You can learn more about how k-fold cross-validation works here:

https://machinelearningmastery.com/k-fold-cross-validation/

Confusion matrix can only calculated for a single test set, not cross validation.

Hi Jason,

Thanks for a great post.

I need a clarification with the following code.

ensemble = VotingClassifier(estimators=[

(model1), (model2), (model3)], voting=’hard’)

ensemble = ensemble.fit(X_train, Y_train)

predictions=ensemble.predict(X_validation)

As you have said in this post do I have to discard the X_train and Y_train subsets created using 10-fold Cross validation for fitting the ensemble model for making predictions or is this code correct. Do I need to use the entire dataset for fitting the ensemble model for making prediction.

Kindly help me.

The code appear to define a voting ensemble using 3 models.

It is often a good idea to use different datasets to fit the ensemble vs the submodels.

One approach involves using the out of sample data during cross validation.

Great post. It answered all my questions I couldn’t find in any other websites. Keep going!

Thanks, I’m happy it helped!

Hey Jason, nice article! Do you maybe know about more research on this topic? Any literature written on this, specifically for time series data?

No, it is an engineering consideration. E.g. how to use a model.

Hi Jason,

Great post. I’m fairly new to the machine learning and python and still learning. I have built few ML models, but once i retrain the model how do i deploy it.

Like I’m not sure how to proceed in terms of how to save the models, retrain the ML model with entire training set and how to deploy it by code on new/future test data. Greatly appreciated

If you are using sklearn, perhaps this will help:

https://machinelearningmastery.com/save-load-machine-learning-models-python-scikit-learn/

Jason,

Yes that partially helped. Part where I’m really stuck is, once i train/test my model, cross validate it and decide which model to use.

How do i go back and essentially retrain my model on the entire data (training data set, instead of just on train/test cut of the data)

Collect all of the data into a single dataset and call model.fit().

Perhaps I don’t understand the difficulty you are having?

Thanks for great tutorial, I have a question, I have some pictures with labels, I make 3 copies by adding Gaussian white noise to pictures and 3 copies by making non align pictures, initial dataset was centered on our image, then I shuffle them and split to test and train, accuracy is around one, my question is that good accuracy could be because of overlap in train and test dataset? should I do any refinment? or is ok

You cannot have copies of the same image in train and test, it would be an invalid evaluation of the model.

Data augmentation is only used on the training set.

Hi Jason,

Recently I am always reading your excellent posts since I a new ML learner. These posts are clear enough to convey useful information. In recent days, I am confused and struggled to understand how cross-validation works.

I have 500 datasets (may be a little), below is my procedure how I employ ML algorithms. Plz correct me if somewhere is wrong. It is binary classification question.

1. Collect data to form 500 datasets

2. Split the 500 datasets into training (80 %, 400) and test (20 %, 100) datasets

3. Use 10-fold cross-validation to check 9 machine learning algorithms (Logistic Regression, Support Vector Machine, Decision Tree, Random Forest, and AdaBoost) on whole 500 datasets. I find AdaBoost has the highest predictive accuracy from Step 3. This means that AdaBoost has more generalized capacity when applied to my question.

4. Then I use GridSearchCV to find the best hyperparameter for AdaBoost on training (80 %, 400) datasets. Using test (20 %, 100) datasets to test the AdaBoost model with best hyperparameter. I can get the accuracy score, confusion matrix, AUC-ROC, TPR, FPR, ROC curve, PR curve.

My question:

(1) For step 3, it is whole 500 datasets or training (80 %, 400) datasets?

(2) For step 4, do I need to finalize the model using whole 500 datasets? It is for academic question, not for industrial question.

Looks good.

Step 3 would just use the training set, step 4 would use the test/validation set. Finalizing is on all data.

More on train/test/validation datasets here:

https://machinelearningmastery.com/difference-test-validation-datasets/

Dear Jason

Request your opinion.

Assume we build an initial Model using,say 1000 features. The output of variables of importance lists only 800 of these 1000 features.

As I understand, while building final Model we would use the same parameters as earlier (e.g. optimal nrounds,etc..), but with complete data. I also use the same random seed as was used to build initial model.

But what about features, should we supply 1000 or 800 features (as obtained above) – while building final model?

Thanks

You would use the same framing of the problem (inputs and outputs) as was used during your experiment.

Hi. There is a publicly available dataset with pre-determined train(1,75,341) and test (82,332) splits. The performance of the model is always good with this experimental approach and I got 98% accuracy. However, in order to get a reliable assessment of the model, I combined train+test (2,57,673), then applied 80:20 split with cross validation and performed classification by stacking classifiers which resulted in an accuracy of 86%. Can I infer that although the accuracy is less, this method yields reliable predictions than the first method? Pls clarify.

Generally, it is a good idea to estimate the performance of a model across multiple train/test splits, ideally k-fold cross validation.

Thanks Jason very useful information on how to get your final model.

I am working on a churn predictive model. I already got my final model done. Do only thing that is a little confusing is when I use my model to predict with new data, how do I know what time period does predictions are likely to happen? If I use a years accumulation of data to train the model, and 6 months worth of new data to make my predictions. Should I expect my predictions to happen in the next 6 months?

NOTE: all of my predictor variables are averaged per month.

Thanks in advance,

*those instead of does

Not sure I follow, sorry. Can you elaborate?

For example, in my churn project if I am trying to predict clients that are likely to cancel. Lets say the model generated an output of 100 clients that are likely to cancel. Approximately when should I expect those clients to leave the company? Is it possible to know?

Thanks for your attention

It could be a time series classification task.

It could also be a survival analysis:

https://en.wikipedia.org/wiki/Survival_analysis

Thanks Jason I really appreciate it!

Hello Jason,

I went through all the 150+ reviews concerning this particular article.Based on the reviews I can easily understand that cross validation is i very important topic.

As far as my understanding goes cross validation is to relate all the independent variables amongst each other to check their importance regards to model building hope I am correct.But this in turn also refers a little bit of multicollinearity. Isnt it???

Can you please clarify.

Thanks.

Best Regards

Sandipan Sarkar

What do you mean exactly? I don’t follow sorry.

Jason,

According to this exchange it claims that you can’t report performance metrics of a model trained on a full data set using averages of the K-folds. Wouldn’t it be more appropriate to train on the full data set and then have a test data set to report the model performance?

https://stats.stackexchange.com/questions/184095/should-final-production-ready-model-be-trained-on-complete-data-or-just-on-tra

Your summary goes against many years of findings, and I’m not interested in debates.

My best advice is to prepare a test harness that give you and project stakeholders confidence in the estimate of model performance on new data.

Thank you for this great post, it’s very informative.

However, I can’t imagine how the K trained models be applied for predication.

I have also followed this post: https://machinelearningmastery.com/train-final-machine-learning-model/

It has mentioned that, “You finalize a model by applying the chosen machine learning procedure on all of your data.”

But I’m sorry that, I still have no idea how to finalize the model. Just pick the best model on the K trained model? Combine the result to build up a better model? Or anything else?

Can you give some practical example to finalize a model based on the K-Flod Cross Validation? Thanks.

The simple approach is once you have a algorithm and config that is reliable on your test harness/procedures, you use it and fit a model on all available data then use that model to start making predictions on new data.

If you have high variance in the final model (see seen on your test harness), you can reduce this variance by fitting K models on all training data, and using them together as an ensemble when making a prediction on new data.

Does that help?

Thanks for the explanation.

I just start self-study on machine leanring with python for few days, it seems that I have mixed up something in k -fold Cross Validation.

Before I hit the topic k-fold Cross Validation, and I learn by splitting the known data into training and testing data, and we can estimate the performance by compare the predicated value with testing data.

So, I guess the k-fold Cross Validation is used to find the best model by using different training and testing data group. it seems wrong.

Can I said that, the k-fold cross validation is used to compare between different algorithm / config? Then based on the score to choose the best algorithm, and build the model using this algorithm.

Thanks a lot.

k-fold cross validation is just another approach to estimate the performance of a model when making predictions on new data.

Just like a train/test split, but the result is less biased (optimistic).

You can learn more here:

https://machinelearningmastery.com/k-fold-cross-validation/

I’m little confused about k-fold CV in general.

Initially I thought its purpose for evaluation to validate the created model. It was about using one particular ML model and repeat its evaluation multiple times to get multiple different scores in order to get average score. This is good since it avoids a “lucky” situation which happens if model training and evaluation was done once.

But then here k-fold CV is used for finding hyperparameters (such as best ML technique to train model) before doing actual training or evaluation.

Also, in some other sources the k-fold CV is used in whole dataset (training set + test set), in other sources k-fold CV is used only on training set.

I am quite confused… I appreciate if you are able to clarify these things about k-fold CV.

You are correct. In all cases, it is used to estimate the performance of a model on a dataset.

https://machinelearningmastery.com/k-fold-cross-validation/

It can operate at different scales, across configs, across models, for one model. At some higher orders, we may need to hold back some additional data to validate with in order that we don’t overfit the model to the dataset.

You can learn about a validation dataset here:

https://machinelearningmastery.com/difference-test-validation-datasets/

Hi,

Thanks for this great post.

I have followed through the whole process of building a machine learning project. This includes standardization, feature engineering and even saving the model in a pickle file. However when I want to use my model to make predictions in the real world, i need to prepare the new data in the same manner I prepared the one for training. Knowing full well that my standardization was done based on the values in the training model, how do i standardize my new data according to the values in the training model?

Does the standardization save rules for new data? How do I access them and also can I save some data into the model file because I would need to do some rounding off of numbers and those numbers were used in the training set.

How do I get all these? Does the model save them?

You must use the same coefficients used to prepare the training data or the same objects if that is easier. You may need to pickle the object/coefficients as well.

Hi Jason,

Thank you for your post.

I am confused with the step where we deploy the model. Are we keeping the ‘old’ coefficients that we get from trained data and applying it onto the new data? Or do we re-fit the model (that we conclude is ‘best’ through validation steps) on the new data?

The coefficients prepared on the training data is the model.

The model is used in a read-only manner to make predictions on new data.

Does that help?

Thank you! Makes sense.

You’re welcome.

Hi Jason,

I have trained a model and save it, but the problem is that when i am going access it from my test code, the test code not getting the training model classes name. So, is there any trick via. i could solve the problem

What problem are you having exactly?

Hi Jason:

This series on cross-validation and training a final model have been very enlightening. I found both the information and your consistency in comment responses very helpful in my development of understanding this concept. I think my biggest hang-up was in hearing statements like “once you have a model that you like…” As I now understand CV from your explanation, we are not trying to optimize the result of model training during CV (model parameters, scaling factors, features, etc. ), but instead we want to optimize the process that we apply to each fold that gives us the most consistent “score” across all folds. For example, we might try a method of feature selection based each feature’s correlation to the predicted result. Applying that to each fold may in-fact result in different features per fold based on that fold’s training data. But in the end, we are not interested in the best list of features, but the best method by which we programmatically choose those features based on how it affects the variance of results across all folds. Once we have assembled all of the methods we “like” we then apply those against the entire dataset to produce a trained and deployable model. Am I understanding this correctly?

Exactly!!!

We are searching for the process that reliably gives a good model for our data.

This will really help you now – tie it all together:

https://machinelearningmastery.com/applied-machine-learning-as-a-search-problem/

Jason — I emailed you a long question yesterday after not finding the answer among your many posts. Then I suddenly found the topic elsewhere, especially in this posting on the “final model”. no need to reply to my post of yesterday. I now understand that the sampling through splitting and cross-validation yields both a configuration and algorithm that prove effective for the problem at hand. I am sure the application to a new data set presents opportunities for further refinement. So, when I asked, “where does the intelligence live,” I can suggest this answer: “it lives in the refined algorithm and configuration proven through experimentation.” Thinks for providing the excellent resources that you do.

Thanks Patrick.

Hi Jason. Thanks for the article.

I recently had the following problem. I have two classifiers – Random Forest and GBoost . I use CV to find the optimal parameters. Initially I used as the final model the models trained only on the training set and I used the accuracy metrics from the test set to assess them (accuracy is around 95%).

Recently after reading your post, I decided to train the final models on the full data set. After doing that I estimate the accuracy of the final models on the full dataset, and while GBoost accuracy is around 95%, I get 100% accuracy for the Random Forest model. In other words, the final RF model classifies all the points in the full data set correctly.

How is this possible? Have you seen something like that?

Thanks,

Egor

You cannot estimate the accuracy of the model on data used to train it.

You estimate accuracy using cross validation, then fit a final model. No need to evaluate it again as you already have the estimate.

Is there any reason not to just ensemble the models trained during cross-validation to produce a final model?

Surely, 10 models each trained with a different 90% of the data, has to produce better results than 1 model trained on all of it?

The data I work with is noisy and non-stationary, so I presume lots of models will help reduce the variance in my results.

Off hand: one model is simpler.

Ensemble is good only if there is ROI – e.g. a sufficient lift is skill. This is not always the case, can be neural or even negative.

Reduction in variance is an excellent reason also.

Try it and see!

Hello Jason Brownlee,

I have developed my own model in robust regression, I also did some programming for my proposed model. But now I want to validate the proposed model by using cross validation, to find the optimal values from a defined grid for the tuning and hyper parameters. If you can provide me any tutorial or codes related to cross validation for our own models which we can do manually, rather than for available models.

Perhaps start here:

https://machinelearningmastery.com/faq/single-faq/how-do-i-evaluate-a-machine-learning-algorithm

And here:

https://machinelearningmastery.com/k-fold-cross-validation/

Hi Jason, Thank you for this great article!

I am still a bit confused about the final model. here is my example:

I have 40 observations.

1. I apply kfold cross validation with K=10 and then repeat it 50 times and at the end I have 500 models

2. I calculate the mean of error metrics such as ame, rmse, standard deviation for the 500 models

3. I chose my best configuration based on cross validation

4. I train my data on all 40 observation to get the final model parameters

so here is my question: how I can test my model to see how well it predicts? I do not have any observation left for prediction. do I need to keep some observations for my model predication. let say I train my final model on 30 values and then evaluate its prediction on the 10 left observation? if yes, how these 10 values should be selected among my dataset to be representative?

Regards,

Amir

No need to test the final model, you tested that config using cross validation already. That measure tells you how well the model is expected to perform on average.

Hi Jason, I already save and load the model but I have a problem to proceed a single prediction. Do you have an idea on how to continue for a single prediction? Sorry for asking cause I’m juat a beginner of this machine learning. Thank you Jason

Yes, see these examples:

https://machinelearningmastery.com/make-predictions-scikit-learn/

Thank you Jason. This was really helpful for me. 🙂

You’re welcome!

Hey Jason,

Thanks for your post. Can you answer a really important question for me?

I am working on my capstone project right now that does text classification from text scraped through the internet. In order to do so I had to do perform feature engineering to train an SVM (SVC) classifier. The features include: avg tfidf value of the entire sentence, presences of root word and presence of NER.

Now that I have trained and saved the model I face a dilemma. I wanna use this pre-trained model to generate labels for unknown data that would be scraped at a later time. But the problem is, I had to create the feature called tfidf avg value (sum of all tfidf values of tokens in a sentence/ total tokens in the sentence). So if I have to generate these features for the unknown data later, the words and values will be vastly different. As such, the model may or may not perform well.

How should I avoid this and ensure that the model works fine? Is there a way to get standard values for it?

Good question, perhaps you need to store the data prep objects used on the training data along with the model, and reuse them to prepare new data.

Great post! Very informative..

i have a question though.

as fas as i know we don’t have to do k-fold cv if we have loads of data, we can go with train/tests datasets. But when we don’t have enough data, we should prefer k-fold(or other cv methods).

Given these, do we really need to train our final model with the whole data(say consists of billions of observations), which we didn’t use even on training?

Thanks.

Yes, generally.

Perhaps not if your training dataset is sufficiently large and representative.

Hi Jason!

I am really grateful and amazed by how you explain very complicated concepts in a simple and understandable way.

I wanted to thank you and wish you the best.

Bless you!

Thanks, you’re very kind!

Hi Jason,

Really nice post to answer the common confusion! Though I still have one confusion: I noticed that people sometimes use the entire dataset to do cv, sometimes they first split the entire dataset to the training and test, and only use the training to do cv. which one is correct or they have different purposes?

If I understand your post correctly, we should first apply the second one (first split the data to the training and test, and only use the training to do cv), by doing this we compare different ML methods, and choose the best one. Then use the best one and use the entire dataset to train the model to get the final model. Is it correct?

Many thanks and look forward to your answers!

Thanks.

Both are correct, and neither. You must choose what makes sense for your specific project and goals.

No, this post is saying that after you choose a model/config, throw all the models away, train a final model on all data and start using it to make predictions on new data.

i am a student trying to complete my research is in classification in rough set theory want a few basic answers

how to find a reduct

how to build a test model as mine is an algorithmic approach (i am using a tool RSES)

and how to check the accuracy of the model

will be

Sorry, I don’t have tutorials on rough set theory, I cannot give you advice on the topic.

hi

i am continuing the above post

will be grateful and look forward for your answer

thanks …..

hey,

I wanted to know if I want to re-train the ML model with some new and updated data how do I do it?

and If I want to remove some of the training data from the existing model for e-g- old data which is not so important now. is it possible to omit it while re-training the ML model?

Could you provide some information or suitable links?

Best Regards,

Great question.

I recommend testing diffrent update strategies and see what works best for your specific data and model.

what are the different update strategies? could you please elaborate your explanation a little.

i am using encoder decoder model with numerical data like temp etc

This will give you ideas:

https://machinelearningmastery.com/update-lstm-networks-training-time-series-forecasting/

Hi Jason,

Thanks for the great post. I think this is one of the very few online posts that I’ve found useful re: cross validation and model building. I’m fairly new to this topic of cross-validation and still have some questions; wonder if you might share your expert opinion on this.

I’m trying to build a multinomial logistic regression model for prediction purposes and asked to do cross-validation. A set of candidate independent variables (IVs) are given to me.

I use all the IVs and do multinomial logistic regression using k-fold cross validation; I see that the model performs poorly: Pseudo R2 are very low and only a few IVs are significant. So I need to find a better model with reduced number of predictors. My question is:

• Each time I try a new model (with a different set of IVs), do I do k-fold cross-validation? Then identify the best model and fit it on the entire dataset?

• Or do I 1st do these procedures (i.e. trying different models with different set of predictors) on a particular sub-set of the entire sample, identify the most relevant/useful model, and then use k-fold cross-validation on the remaining sample only for this model?

• Or do I do those steps of finding the final model on the entire sample and then do k-fold cross-validation only for the final model?

The concerns I have regarding the options #1 and #3 above are that, data that are used to develop the model should not be used for testing/validating the model as it will give too “optimistic” results. Options #1 and #3 above will include a portion of the dataset, that might overlap on both the model ‘development’ and ‘validation’ samples…

Would appreciate your opinion. Thanks!

Yes, each model should be evaluated in an identical manner (same folds).

Hi Jason,

Thank you so much for your prompt answer! I really appreciate it! Just wanted to confirm that I understood what you suggested: “each model should be evaluated in an identical manner (same folds)” using k-fold cross validation and then the final model be fit on the entire dataset…? Thanks again!

Sincerely

Rimi

Yes.

Thank you!!!

You’re welcome.

Dear Dr. Jason

thanks again for your information.

What I understand from your page is that when somebody so as me using

sklearn.model_selection import train_test_split

1) it does not need to use K-fold?

2) others, I using GBRT for classification and regression and I divide 20% to test data

does it possible the overfitting or underfitting take place for my data?

how I can understand and prevent over and underfitting?

My dataset has 980 data with 14 features

Good question, this will give you ideas:

https://machinelearningmastery.com/introduction-to-regularization-to-reduce-overfitting-and-improve-generalization-error/

Typically you use either cross-validation or a train/test split to evaluate your model.

You must choose an approach that is appropriate for your dataset, if you are unsure use 10-fold cross validation for small data and train/test for very large data or slow models.

thank you for your guidance

according to your information, I reach to this hyperparameters in GBRT for overcome over and under fitting

params = {‘n_estimators’: 1000,

‘max_depth’: 4,’subsample’:0.25,

‘min_samples_split’: 5,

‘learning_rate’:0.004,

‘loss’: ‘ls’,’min_samples_leaf’:5,

‘random_state’:7}

does this learning rate is reaonabel because most literate talk about minimum 0.01 lr?

Perhaps test it and see?

yes, I test and I just want to know there is no minimum learning rate that’s right?

my train deviance chart is so that because of learning rate 0.004 has many fluctuated in contrast of test that moving too smooth over it.

There is no minimum learning rate.

Thanks for your great article!

If I want to select the best one from various types of model as well as the best combination of hyperparameters, should I first perform CV for each type of model to get the optimal set of hyperparameters, then perform another CV to select the best model with their optimal hyperparameters?

One approach is to use tuning within each cv of each model. This is called nested cross-validation.

Another is to quickly test a suite of models then tune the top performing to get the most out of them. Ideally, the first and second step would use diffrent data, but that may not always be possible.

Dear Jason,

Thank you very much for your precious contributions!

Could you help me on several questions?

I found that you said we have to decide whether to refit the model or use it as is if new data becomes available in a reply.

>. if we decide to refit, do we need to redo all the works for choosing the best procedure then finalize the model from the very beginning?

>. If so, after how many new data is available, then we redo the works?

>. if we don’t redo, does it mean that we assume the pattern in the data doesn’t change over time?

Regards,

Tom

It is up to you how much work you want to do or not do.

The simples approach is to assume the approach remains viable and refit it on new data.

thanks for this really useful post:

A question about implementing the final model after cross_validation: I want to retrain model now after CV on entirety of the data.

Presumably now there is no holdout data (validation or independent testing data).

Is there a strategy for choosing the number of epochs to train to prevent over/under fitting? During 10x cross_val, the average epochs taken to converge (with early stopping) is 25-30. Is taking the average epochs taken to converge over the 10 folds also a viable strategy for training on the entire dataset?

so could the final model.fit call look like:

model.fit(allcountdata, allonehotlabels, epochs=25)

many thanks again for this incredibly useful post!

One approach is to use your test harness to find the number of epochs.

Another approach is to hold out a validation set and use early stopping to train the final model.

Thanks for your tutorial. However I have a silly problem. I am training my model in keras using many data sets of the same dimension. It runs very nicely and finally i want to save the model…However my question is for every data sets the model is updated correct me if i am wrong…. then which model should i save to .h5 and .json file…is it the model which will obtain after running the programme over all datasets…

Save the model that is trained on all data.

Hi, and thx for this really good post !

I have a question about categorical data for the final model :

– I splitted data into train/test split

– I fitted a sklearn OnehotEncoder on train data and transform train/test data

– I made cross-validation to select my model

Now I fit my model on all data to finalize it but… Do I need te ‘refit’ my OneHotEncoder with ALL data before transforming or must I reuse the fitted one on the train data ???

Yes.

Hi Jason,

a great post, again! Thank you!

I hope this is neither too naive nor too off-topic:

If I finalize my model (lstm-time-series prediction) like done here and then ran model.predict() on all the data again (to see in how far the predictions would have been correct in the past), is that supposed to give me a nicely aligned pair of y and yhat curves? Or will the predictions be off, as the model is not in the state to predict an early state in the time-series again? (I tried, and it was off. So I am worried I build a crappy model…) – Thanks.

I think you’re asking will the model make good predictions on the training set. If so, the answer is it should, but they won’t be perfect – there’s always some error when we try to generalize.

Does that help?

To choose a good model, select a metric and optimize it, and it is a great idea to review both the predictions made by the model vs expected values in a hold out set (test set) as well as a plot of the errors. It can give you ideas of the range of error and how you might model the problem better.

Hi Jason,

thank you. That certainly helps. The prediction was far off, really … What threw me off, was that I achieved an excellent RMSE of around 2% during the train and test procedure. I will rethink the model I used and hope to find the wrong turn. However, I still wonder how the graphed RMSE for train and test can look so promising, while the final prediction is so far off.

I will dig deeper into my model 😉

Good luck!

Great post. But I have a question. How to train a neural network on all data?

We need a validation set to stop the training of NN to prevent overfitting. Even, let’s say we split the final data to train-validation set with 10% validation to stop the training of the network, then the question would be how to pick that validation set?

Different train-validations may result in different accuracies for the trained model. Should we try another 10-fold CV wihtout any test data and take the best one as the final model?

Thanks.

If you need data for early stopping, then hold some data back for early stopping.

Yes, fit n models and ensemble their predictions if you want to counter the high variance.

Yes, the ensemble model is always possible. But what if we need only one model to present as the final solution not an esnamble?

Then how to choose the validation set (for early stoping?) also in what ratio? 10%? 15%? …

If I fit n final models, then is it wise to take the best one or the worst one as the final product?

A random sample. Then perhaps do a sensitivity analysis to determine the appropriate ratio.

Thanks for your invaluable and truly practical material.

They have helped me a lot in both implementation of different algorithms and also understanding the underlying concepts.

Please keep it up.

Thanks for you efforts.

Erfan

You’re welcome!

I have compared and tuned 3 different models using 5-fold CV.

I tuned the hyper parameters of the models by looking at the averaged 5f_CV scores on validation set (not a training set). And I found model#2 is the best model having the highest averaged 5f-CV score on validation sets.

Now I am going to pick the tuned model#2 to fit over all my data set to get my finial model.

Am I right?

Thanks in advance for your response.

It is a good idea to choose a model based on their mean performance relative to the performance of other models tested on the same test harness.

Hi, Jason:

I have an imbalanced dataset. After applying an Stratified10Fold (with a 3NN and undersamppling every fold) I obtain an f1_score average of 83%. Finally, I train my model on all my available data and apply it to new unseen data, getting a 67% of f1_score. Is this happening because of the imbalance of the classes? Should I undersample all my data and apply the 3NN? Or is it better to keep the model I used on the last fold?

Thanks in advance!

You should only evaluate the model on unseen data, not on training data.

Perhaps this framework will help:

https://machinelearningmastery.com/framework-for-imbalanced-classification-projects/

Hi, Jason:

Thanks for your post, it’s great. I’m a beginner. I’m confusing that for a single fold of K-fold CV, the test set is unseen data to the model training on train data, so if we don’t attempt to use K-fold CV to choose the best parameters, can I think that K-fold CV is combined of K final model?

Sorry, I’m not sure I follow your question sorry.

Perhaps you can rephrase it?