Training to the test set is a type of overfitting where a model is prepared that intentionally achieves good performance on a given test set at the expense of increased generalization error.
It is a type of overfitting that is common in machine learning competitions where a complete training dataset is provided and where only the input portion of a test set is provided. One approach to training to the test set involves constructing a training set that most resembles the test set and then using it as the basis for training a model. The model is expected to have better performance on the test set, but most likely worse performance on the training dataset and on any new data in the future.
Although overfitting the test set is not desirable, it can be interesting to explore as a thought experiment and provide more insight into both machine learning competitions and avoiding overfitting generally.
In this tutorial, you will discover how to intentionally train to the test set for classification and regression problems.
After completing this tutorial, you will know:
Training to the test set is a type of data leakage that may occur in machine learning competitions.
One approach to training to the test set involves creating a training dataset that is most similar to a provided test set.
How to use a KNN model to construct a training dataset and train to the test set with a real dataset.
Kick-start your project with my new book Data Preparation for Machine Learning, including step-by-step tutorials and the Python source code files for all examples.
Let’s get started.
How to Train to the Test Set in Machine Learning Photo by ND Strupler, some rights reserved.
Tutorial Overview
This tutorial is divided into three parts; they are:
Train to the Test Set
Train to Test Set for Classification
Train to Test Set for Regression
Train to the Test Set
In applied machine learning, we seek a model that learns the relationship between the input and output variables using the training dataset.
The hope and goal is that we learn a relationship that generalizes to new examples beyond the training dataset. This goal motivates why we use resampling techniques like k-fold cross-validation to estimate the performance of the model when making predictions on data not used during training.
In the case of machine learning competitions, like those on Kaggle, we are given access to the complete training dataset and the inputs of the test dataset and are required to make predictions for the test dataset.
This leads to a possible situation where we may accidentally or choose to train a model to the test set. That is, tune the model behavior to achieve the best performance on the test dataset rather than develop a model that performs well in general, using a technique like k-fold cross-validation.
Another, more overt path to information leakage, can sometimes be seen in machine learning competitions where the training and test set data are given at the same time.
It is an explicit type of data leakage. Nevertheless, it is an interesting thought experiment.
One approach to training to the test set is to contrive a training dataset that is most similar to the test set. For example, we could discard all rows in the training set that are too different from the test set and only train on those rows in the training set that are maximally similar to rows in the test set.
While the test set data often have the outcome data blinded, it is possible to “train to the test” by only using the training set samples that are most similar to the test set data. This may very well improve the model’s performance scores for this particular test set but might ruin the model for predicting on a broader data set.
We would expect the model to overfit the test set, but this is the whole point of this thought experiment.
Let’s explore this approach to training to the test set in this tutorial.
We can use a k-nearest neighbor model to select those instances of the training set that are most similar to the test set. The KNeighborsRegressor and KNeighborsClassifier both provide the kneighbors() function that will return indexes into the training dataset for rows that are most similar to a given data, such as a test set.
1
2
3
4
...
# get the most similar neighbor for each point in the test set
Given that we are using a KNN model to construct the training set from the test set, we will also use the same type of model to make predictions on the test set. This is not required, but it makes the examples simpler.
Using this approach, we can now experiment with training to the test set for both classification and regression datasets.
Want to Get Started With Data Preparation?
Take my free 7-day email crash course now (with sample code).
Click to sign-up and also get a free PDF Ebook version of the course.
Train to Test Set for Classification
We will use the diabetes dataset as the basis for exploring training for the test set for classification problems.
Each record describes the medical details of a female and the prediction is the onset of diabetes within the next five years.
The dataset has eight input variables and 768 rows of data; the input variables are all numeric and the target has two class labels, e.g. it is a binary classification task.
Below provides a sample of the first five rows of the dataset.
1
2
3
4
5
6
6,148,72,35,0,33.6,0.627,50,1
1,85,66,29,0,26.6,0.351,31,0
8,183,64,0,0,23.3,0.672,32,1
1,89,66,23,94,28.1,0.167,21,0
0,137,40,35,168,43.1,2.288,33,1
...
First, we can load the dataset directly from the URL, split it into input and output elements, then split the dataset into train and test sets, holding thirty percent back for the test set. We can then evaluate a KNN model with default model hyperparameters by training it on the training set and making predictions on the test set.
The complete example is listed below.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
# example of evaluating a knn model on the diabetes classification dataset
from pandas import read_csv
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
from sklearn.neighbors import KNeighborsClassifier
Running the example first loads the dataset and summarizes the number of rows and columns, matching our expectations. The shape of the train and test sets are then reported, showing we have about 230 rows in the test set.
Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.
Finally, the classification accuracy of the model is reported to be about 77.056 percent.
1
2
3
(768, 8) (768,)
(537, 8) (231, 8) (537,) (231,)
Accuracy: 77.056
Now, let’s see if we can achieve better performance on the test set by preparing a model that is trained directly for it.
First, we will construct a training dataset using the simpler example in the training set for each row in the test set.
1
2
3
4
5
6
7
8
9
10
...
# select examples that are most similar to the test set
knn=KNeighborsClassifier()
knn.fit(X_train,y_train)
# get the most similar neighbor for each point in the test set
Running the example, we can see that the reported size of the new training dataset is the same size as the test set, as we expected.
Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.
We can see that we have achieved a lift in performance by training to the test set over training the model on the entire training dataset. In this case, we achieved a classification accuracy of about 79.654 percent compared to 77.056 percent when the entire training dataset is used.
1
2
3
4
(768, 8) (768,)
(537, 8) (231, 8) (537,) (231,)
(231, 8) (231,)
Accuracy: 79.654
You might want to try selecting different numbers of neighbors from the training set for each example in the test set to see if you can achieve better performance.
Also, you might want to try keeping unique row indexes in the training set and see if that makes a difference.
Finally, it might be interesting to hold back a final validation dataset and compare how different “train-to-the-test-set” techniques affect performance on the holdout dataset. E.g. see how training to the test set impacts generalization error.
Report your findings in the comments below.
Now that we know how to train to the test set for classification, let’s look at an example for regression.
Train to Test Set for Regression
We will use the housing dataset as the basis for exploring training for the test set for regression problems.
The housing dataset involves the prediction of a house price in thousands of dollars given details of the house and its neighborhood.
First, we can load the dataset, split it, and evaluate a KNN model on it directly using the entire training dataset. We will report performance on this regression class using mean absolute error (MAE).
The complete example is listed below.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
# example of evaluating a knn model on the housing regression dataset
from pandas import read_csv
from sklearn.model_selection import train_test_split
Running the example first loads the dataset and summarizes the number of rows and columns, matching our expectations. The shape of the train and test sets are then reported, showing we have about 150 rows in the test set.
Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.
Finally, the MAE of the model is reported to be about 4.488.
1
2
3
(506, 13) (506,)
(354, 13) (152, 13) (354,) (152,)
MAE: 4.488
Now, let’s see if we can achieve better performance on the test set by preparing a model that is trained to it.
First, we will construct a training dataset using the simpler example in the training set for each row in the test set.
1
2
3
4
5
6
7
8
9
10
...
# select examples that are most similar to the test set
knn=KNeighborsClassifier()
knn.fit(X_train,y_train)
# get the most similar neighbor for each point in the test set
Running the example, we can see that the reported size of the new training dataset is the same size as the test set, as we expected.
Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.
We can see that we have achieved a lift in performance by training to the test set over training the model on the entire training dataset. In this case, we achieved a MAE of about 4.433 compared to 4.488 when the entire training dataset is used.
Again, you might want to explore using a different number of neighbors when constructing the new training set and see if keeping unique rows in the training dataset makes a difference. Report your findings in the comments below.
1
2
3
4
(506, 13) (506,)
(354, 13) (152, 13) (354,) (152,)
(152, 13) (152,)
MAE: 4.433
Further Reading
This section provides more resources on the topic if you are looking to go deeper.
It provides self-study tutorials with full working code on: Feature Selection, RFE, Data Cleaning, Data Transforms, Scaling, Dimensionality Reduction,
and much more...
Bring Modern Data Preparation Techniques to Your Machine Learning Projects
Hi Jason. Loved the interesting tutorial like all others.
My real life data without labels are very different in nature than my training data. With 10 fold CV on my training data I am getting 90% accuracy where in real life it is 70%. I checked over and underfitting using my training data and also tuned hyperparameters.
No. The approach listed in the above tutorial is perhaps appropriate for competitions (or not), or for demonstrating the danger in general of training to the test set intentionally or accidentally.
Seems, a typo: in your Train to Test Set for Regression you have
…
# define model
model = KNeighborsClassifier()
# fit model
model.fit(X_train_neigh, y_train_neigh)
Dear Dr Jason,
Where you suggested to try for different neighbors under the section heading “Train to Test Set for Classification” where you said “…you might want to explore using a different number of neighbors…” I found no difference.
Is what you meant “differernt number of neighbors” changing the parameter 1 to various values in the following
I’m interested in Machine learning for multivariate time series (e.g., EEG, ECG). I’m looking for python based courses and tutorials for processing and analysis these kind of signals.
I’m looking forward to your reply and appreciate your help.
Thanks very much for great explanation and examples on the subject. I have been a fan of your blog and learnt a lot from your publications.
I have some questions on the supervised learning approach when the training and testing datasets are disjoined. Say for example, a training set consists of label1, label2, label3 and the testing set consists of labels from 1 to 5. In this case, labels 4 and 5 were not presented in the training set and thus a trained model never seen these classes before.
In this case, my questions are:
i) Do we have any techniques that are able to differentiate what have been trained vs untrained, i.e., labels 1-3 vs label 4-5?
ii) To predict untrained labels from a train model, how do we assign (or create) labels for them?
I assume you probably can refer to any unsupervised methods to address this problem, but in this case just want to use supervised learning instead.
Any thoughts or suggestions are greatly appreciated.
Not really – training data is generally assumed to be representative of the problem/all data. Perhaps you can train a model to determine if the data looks like the training data or not, and only use the model if it does.
Yeah, clustering might help in sorting out natural structure in the data which may or may not relate to labels.
The idea of using a binary classifier requires labels for both classes of trained/untrained and I still don’t know how to effectively/accurately create data for the untrained class…
Perhaps, we can use soft decisions from the prediction, but I found this approach is hard to generalize.
We train and evaluate models/configs with a dataset that has known labels.
We use the results to choose a model and config.
We then train the model on all data and use it to predict labels on new data where we do not know the label. This is the whole point of choosing a model.
Allow me to start by briefly explaining my problem. I am training neural network models on structured data. This data increases on weekly basis and so I train my models on the current and the incremented set (combined) everytime. I do internal split for the data I receive: train 70% and the rest 30 % for validation and test splits (each 15 %). This is in production.
My question:
– Do you think, “in production”, it is much worth to use the entire (all data splits) data to train the model?
I am afraid that the model will overfit in that case (especially that currently I use early stopping technique that monitors validation loss on my validation data).
– Or you think its much better to create representive validation/test sets using nearest neighbors algorithm? and continue with this training process in production (not using validation set for training).
This has been a confusion. I am looking forward to your answer.
Hi Jason. Loved the interesting tutorial like all others.
My real life data without labels are very different in nature than my training data. With 10 fold CV on my training data I am getting 90% accuracy where in real life it is 70%. I checked over and underfitting using my training data and also tuned hyperparameters.
Do you think this approach might be useful?
Thanks!
No. The approach listed in the above tutorial is perhaps appropriate for competitions (or not), or for demonstrating the danger in general of training to the test set intentionally or accidentally.
Seems, a typo: in your Train to Test Set for Regression you have
…
# define model
model = KNeighborsClassifier()
# fit model
model.fit(X_train_neigh, y_train_neigh)
What is the typo exactly?
Thanks, Jason Brownlee for this wonderful tutorial. You’ve have inspired me since day 1. I’m very grateful. Keep on writing. 🙂
You’re welcome.
Well, no doubt it takes time to read this all but a great piece of work. Thanks for sharing.
Thanks!
Dear Dr Jason,
Where you suggested to try for different neighbors under the section heading “Train to Test Set for Classification” where you said “…you might want to explore using a different number of neighbors…” I found no difference.
Is what you meant “differernt number of neighbors” changing the parameter 1 to various values in the following
If so I have the code which shows that changing the parameter 1 to other values did nothing:
Output:
So Is what you meant “differernt number of neighbors” changing the parameter 1 to various values in the following? The results are identical.
Thank you,
Anthony of Sydney
Nice finding, thanks for sharing.
Hello Jason,
I’m interested in Machine learning for multivariate time series (e.g., EEG, ECG). I’m looking for python based courses and tutorials for processing and analysis these kind of signals.
I’m looking forward to your reply and appreciate your help.
Thanks
Mohammed
See this book on deep learning for time series forecasting:
https://machinelearningmastery.com/deep-learning-for-time-series-forecasting/
Thanks for replying
You’re welcome.
Hi Jason,
Thanks very much for great explanation and examples on the subject. I have been a fan of your blog and learnt a lot from your publications.
I have some questions on the supervised learning approach when the training and testing datasets are disjoined. Say for example, a training set consists of label1, label2, label3 and the testing set consists of labels from 1 to 5. In this case, labels 4 and 5 were not presented in the training set and thus a trained model never seen these classes before.
In this case, my questions are:
i) Do we have any techniques that are able to differentiate what have been trained vs untrained, i.e., labels 1-3 vs label 4-5?
ii) To predict untrained labels from a train model, how do we assign (or create) labels for them?
I assume you probably can refer to any unsupervised methods to address this problem, but in this case just want to use supervised learning instead.
Any thoughts or suggestions are greatly appreciated.
Thanks,
Dave
You’re welcome.
Not really – training data is generally assumed to be representative of the problem/all data. Perhaps you can train a model to determine if the data looks like the training data or not, and only use the model if it does.
Yeah, clustering might help in sorting out natural structure in the data which may or may not relate to labels.
Thank you for your response Jason.
The idea of using a binary classifier requires labels for both classes of trained/untrained and I still don’t know how to effectively/accurately create data for the untrained class…
Perhaps, we can use soft decisions from the prediction, but I found this approach is hard to generalize.
Again, much appreciated your reply.
-D.
We train and evaluate models/configs with a dataset that has known labels.
We use the results to choose a model and config.
We then train the model on all data and use it to predict labels on new data where we do not know the label. This is the whole point of choosing a model.
Does that help?
Hi Jason,
Thank you, as always, for your amazing articles.
Allow me to start by briefly explaining my problem. I am training neural network models on structured data. This data increases on weekly basis and so I train my models on the current and the incremented set (combined) everytime. I do internal split for the data I receive: train 70% and the rest 30 % for validation and test splits (each 15 %). This is in production.
My question:
– Do you think, “in production”, it is much worth to use the entire (all data splits) data to train the model?
I am afraid that the model will overfit in that case (especially that currently I use early stopping technique that monitors validation loss on my validation data).
– Or you think its much better to create representive validation/test sets using nearest neighbors algorithm? and continue with this training process in production (not using validation set for training).
This has been a confusion. I am looking forward to your answer.
Thanks 🙂