#### How to predict classification or regression outcomes

with scikit-learn models in Python.

Once you choose and fit a final machine learning model in scikit-learn, you can use it to make predictions on new data instances.

There is some confusion amongst beginners about how exactly to do this. I often see questions such as:

How do I make predictions with my model in scikit-learn?

In this tutorial, you will discover exactly how you can make classification and regression predictions with a finalized machine learning model in the scikit-learn Python library.

After completing this tutorial, you will know:

- How to finalize a model in order to make it ready for making predictions.
- How to make class and probability predictions in scikit-learn.
- How to make regression predictions in scikit-learn.

Letâ€™s get started.

## Tutorial Overview

This tutorial is divided into 3 parts; they are:

- First Finalize Your Model
- How to Predict With Classification Models
- How to Predict With Regression Models

## 1. First Finalize Your Model

Before you can make predictions, you must train a final model.

You may have trained models using k-fold cross validation or train/test splits of your data. This was done in order to give you an estimate of the skill of the model on out-of-sample data, e.g. new data.

These models have served their purpose and can now be discarded.

You now must train a final model on all of your available data.

You can learn more about how to train a final model here:

## 2. How to Predict With Classification Models

Classification problems are those where the model learns a mapping between input features and an output feature that is a label, such as “*spam*” and “*not spam*.”

Below is sample code of a finalized LogisticRegression model for a simple binary classification problem.

Although we are using *LogisticRegression* in this tutorial, the same functions are available on practically all classification algorithms in scikit-learn.

1 2 3 4 5 6 7 8 |
# example of training a final classification model from sklearn.linear_model import LogisticRegression from sklearn.datasets.samples_generator import make_blobs # generate 2d classification dataset X, y = make_blobs(n_samples=100, centers=2, n_features=2, random_state=1) # fit final model model = LogisticRegression() model.fit(X, y) |

After finalizing your model, you may want to save the model to file, e.g. via pickle. Once saved, you can load the model any time and use it to make predictions. For an example of this, see the post:

For simplicity, we will skip this step for the examples in this tutorial.

There are two types of classification predictions we may wish to make with our finalized model; they are class predictions and probability predictions.

### Class Predictions

A class prediction is: given the finalized model and one or more data instances, predict the class for the data instances.

We do not know the outcome classes for the new data. That is why we need the model in the first place.

We can predict the class for new data instances using our finalized classification model in scikit-learn using the *predict()* function.

For example, we have one or more data instances in an array called *Xnew*. This can be passed to the *predict()* function on our model in order to predict the class values for each instance in the array.

1 2 |
Xnew = [[...], [...]] ynew = model.predict(Xnew) |

### Multiple Class Predictions

Let’s make this concrete with an example of predicting multiple data instances at once.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
# example of training a final classification model from sklearn.linear_model import LogisticRegression from sklearn.datasets.samples_generator import make_blobs # generate 2d classification dataset X, y = make_blobs(n_samples=100, centers=2, n_features=2, random_state=1) # fit final model model = LogisticRegression() model.fit(X, y) # new instances where we do not know the answer Xnew, _ = make_blobs(n_samples=3, centers=2, n_features=2, random_state=1) # make a prediction ynew = model.predict(Xnew) # show the inputs and predicted outputs for i in range(len(Xnew)): print("X=%s, Predicted=%s" % (Xnew[i], ynew[i])) |

Running the example predicts the class for the three new data instances, then prints the data and the predictions together.

1 2 3 |
X=[-0.79415228 2.10495117], Predicted=0 X=[-8.25290074 -4.71455545], Predicted=1 X=[-2.18773166 3.33352125], Predicted=0 |

### Single Class Prediction

If you had just one new data instance, you can provide this as instance wrapped in an array to the *predict()* function; for example:

1 2 3 4 5 6 7 8 9 10 11 12 13 |
# example of making a single class prediction from sklearn.linear_model import LogisticRegression from sklearn.datasets.samples_generator import make_blobs # generate 2d classification dataset X, y = make_blobs(n_samples=100, centers=2, n_features=2, random_state=1) # fit final model model = LogisticRegression() model.fit(X, y) # define one new instance Xnew = [[-0.79415228, 2.10495117]] # make a prediction ynew = model.predict(Xnew) print("X=%s, Predicted=%s" % (Xnew[0], ynew[0])) |

Running the example prints the single instance and the predicted class.

1 |
X=[-0.79415228, 2.10495117], Predicted=0 |

### A Note on Class Labels

When you prepared your data, you will have mapped the class values from your domain (such as strings) to integer values. You may have used a LabelEncoder.

This *LabelEncoder* can be used to convert the integers back into string values via the *inverse_transform()* function.

For this reason, you may want to save (pickle) the *LabelEncoder* used to encode your y values when fitting your final model.

### Probability Predictions

Another type of prediction you may wish to make is the probability of the data instance belonging to each class.

This is called a probability prediction where given a new instance, the model returns the probability for each outcome class as a value between 0 and 1.

You can make these types of predictions in scikit-learn by calling the *predict_proba()* function, for example:

1 2 |
Xnew = [[...], [...]] ynew = model.predict_proba(Xnew) |

This function is only available on those classification models capable of making a probability prediction, which is most, but not all, models.

The example below makes a probability prediction for each example in the *Xnew* array of data instance.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
# example of making multiple probability predictions from sklearn.linear_model import LogisticRegression from sklearn.datasets.samples_generator import make_blobs # generate 2d classification dataset X, y = make_blobs(n_samples=100, centers=2, n_features=2, random_state=1) # fit final model model = LogisticRegression() model.fit(X, y) # new instances where we do not know the answer Xnew, _ = make_blobs(n_samples=3, centers=2, n_features=2, random_state=1) # make a prediction ynew = model.predict_proba(Xnew) # show the inputs and predicted probabilities for i in range(len(Xnew)): print("X=%s, Predicted=%s" % (Xnew[i], ynew[i])) |

Running the instance makes the probability predictions and then prints the input data instance and the probability of each instance belonging to the first and second classes (0 and 1).

1 2 3 |
X=[-0.79415228 2.10495117], Predicted=[0.94556472 0.05443528] X=[-8.25290074 -4.71455545], Predicted=[3.60980873e-04 9.99639019e-01] X=[-2.18773166 3.33352125], Predicted=[0.98437415 0.01562585] |

This can be helpful in your application if you want to present the probabilities to the user for expert interpretation.

## 3. How to Predict With Regression Models

Regression is a supervised learning problem where, given input examples, the model learns a mapping to suitable output quantities, such as “0.1” and “0.2”, etc.

Below is an example of a finalized *LinearRegression* model. Again, the functions demonstrated for making regression predictions apply to all of the regression models available in scikit-learn.

1 2 3 4 5 6 7 8 |
# example of training a final regression model from sklearn.linear_model import LinearRegression from sklearn.datasets import make_regression # generate regression dataset X, y = make_regression(n_samples=100, n_features=2, noise=0.1, random_state=1) # fit final model model = LinearRegression() model.fit(X, y) |

We can predict quantities with the finalized regression model by calling the *predict()* function on the finalized model.

As with classification, the predict() function takes a list or array of one or more data instances.

### Multiple Regression Predictions

The example below demonstrates how to make regression predictions on multiple data instances with an unknown expected outcome.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
# example of training a final regression model from sklearn.linear_model import LinearRegression from sklearn.datasets import make_regression # generate regression dataset X, y = make_regression(n_samples=100, n_features=2, noise=0.1) # fit final model model = LinearRegression() model.fit(X, y) # new instances where we do not know the answer Xnew, _ = make_regression(n_samples=3, n_features=2, noise=0.1, random_state=1) # make a prediction ynew = model.predict(Xnew) # show the inputs and predicted outputs for i in range(len(Xnew)): print("X=%s, Predicted=%s" % (Xnew[i], ynew[i])) |

Running the example makes multiple predictions, then prints the inputs and predictions side-by-side for review.

1 2 3 |
X=[-1.07296862 -0.52817175], Predicted=-61.32459258381131 X=[-0.61175641 1.62434536], Predicted=-30.922508147981667 X=[-2.3015387 0.86540763], Predicted=-127.34448527071137 |

### Single Regression Prediction

The same function can be used to make a prediction for a single data instance as long as it is suitably wrapped in a surrounding list or array.

For example:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
# example of training a final regression model from sklearn.linear_model import LinearRegression from sklearn.datasets import make_regression # generate regression dataset X, y = make_regression(n_samples=100, n_features=2, noise=0.1) # fit final model model = LinearRegression() model.fit(X, y) # define one new data instance Xnew = [[-1.07296862, -0.52817175]] # make a prediction ynew = model.predict(Xnew) # show the inputs and predicted outputs print("X=%s, Predicted=%s" % (Xnew[0], ynew[0])) |

Running the example makes a single prediction and prints the data instance and prediction for review.

1 |
X=[-1.07296862, -0.52817175], Predicted=-77.17947088762787 |

## Further Reading

This section provides more resources on the topic if you are looking to go deeper.

- How to Train a Final Machine Learning Model
- Save and Load Machine Learning Models in Python with scikit-learn
- scikit-learn API Reference

### Summary

In this tutorial, you discovered how you can make classification and regression predictions with a finalized machine learning model in the scikit-learn Python library.

Specifically, you learned:

- How to finalize a model in order to make it ready for making predictions.
- How to make class and probability predictions in scikit-learn.
- How to make regression predictions in scikit-learn.

Do you have any questions?

Ask your questions in the comments below and I will do my best to answer.

Once again, Jason… you’re answering all the questions that need answering.

I was just working through yesterday how to actually use these highly developed models (which I’ve learned to do expediently from your book by the way) to predict my new input variables. And here in my inbox, you’ve delivered this great article on it!

Thank you for making us all better at Machine Learning. You’re work here is stupendous and appreciated!

Thanks Mitch, I’m glad it helps!

Shoot/post me questions any time.

sureshkumar0707@gmail.com

Segmentation can be performed using Python? machine learning can be applied on the segmentation ?

Yes, I recommend looking into OpenCV.

How does one turn predictions into actions? Say I am predicting user fraud, how would you go about taking any given prediction point and determine the customer for that particular prediction.

Great question.

The use of the predictive model would be embedded within an application that is aware of the current customer for which a prediction is being made.

Great post. Love to see an example of the same in R.

Thanks for the suggestion.

I like your explanation but I am missing one thing.

How do you encode and scale features for Xnew so they match trained data?

You must sale new data using the procedure you used to scale training data.

This might mean keeping objects or coefficients used to prepare training data to then apply on new data in the future, such as min/max, a vocab, etc. depending on the problem type.

Thank you very much for the explanation

But my question is how to use the Source Code as an .exe Application to use it later without a script engine

You can use code in your application as you would any other software engineering project.

I’m sorry, I am not an expert at creating executable files on Windows. I have not used the platform in nearly 2 decades.

how to save a label encoder and reuse it again across different python files.

i have encoded my data in training phase and while i am trying to predict labels in testing phase i am not able to get same label encoders so i am getting wrong prediction

please help..

How to save (pickle) the LabelEncoder used to encode your y values when fitting your final model.

You can use the pickle dump/load functions directly.

Hi Jason, (relatively new to ML)

I have a data frame with,

1 ID column

6 feature columns

1 target column

when I train/test split the feature and target columns and do predictions etc, that is where I need to map back to the ID.

I want to be able to view something like this after my predictions:

A data frame with,

1 ID column

6 feature columns

1 target column

1 predicted column

Would you be able to help me with this? Really appreciate it,

kevin

The predictions are made in the order of the inputs.

You can take the array of predictions and align them with your inputs directly and start using them.

Does that help? If not, what is the specific problem you are having?

Thanks Jason, I suppose I have reached a point where I can get my final model and I cannot seem to find any information as to what next, i.e. real specifics regarding making predictions with new datasets.

There are a trillion examples of how to work with train/test split and refining models, but my end goal is taking a ‘complete’ dataset, plugging it into my model prediction and producing back my initial ‘complete’ dataset PLUS my predicted column(s).

I work for a credit union in DC and I have a list of member data, such things like member number, name, phone number, address, account balances and various other features I would use for prediction. I would like to feed this ‘complete’ dataset into my prediction model and have it spit out my initial ‘complete’ dataset PLUS my predicted column(s) that someone could then use to reach out marketing related messages to the member, depending on the prediction off course.

Hope that makes sense..

thanks again Jason, appreciate your time (how do you find the time?!!)

Yes, that makes sense.

You will need to write code to take the input to the model (X) and pass it to the model to make predictions model.predict(X) to get the prediction column (yhat).

You then have the dataset X and the predictions yhat and the rows in one correspond to rows in the other. You can hstack() the arrays to have one large matrix of inputs and predictions.

What problem specifically are you having in achieving this?

Thank you Jason, not so much a problem as a lack of experience trying to tie it all together, but with your help we’ll get there!!

thanks again.

Hang in there Kevin!

Hello Jason, I’ve got started working with scikit-learn models to predict further values but there is something I don’t clearly understand: Let’s suppose I do have a Stock Exchange price datasets with Date, Open Price, Close Price, and the variation rate from the previous date, for a single asset or position. In order to ‘fit’ a good prediction, I decided to use a Multiple Linear Regression and a Polynomial Feature also: I can obtain a formula even used a support vector machine (SVR) but I don’t know how to predict a NEW dataset, since the previous one has more than one variable (Open Price, Variation Rate, Date). How can I simulate further values?

Thanks for your response.

The tutorial above shows how to make a prediction with new data.

What problem are you having exactly?

Thank you so much, Jason, for this Great post! can you please tell me, I have used LabelEncoder for three Cities. So now I have to take input from a user as a string and convert them into int using LabelEncoder and provide it to trained model for prediction. Is it correct?

I believe so.

Hi Jason, always a pleasure seeing your blogs.

I’m thinking of a few things in regard to measuring the “accuracy” of a regression model and making use of such a model, would love to hear your thoughts.

I have problem that can either be framed as a classification problem (discrete labels) but also as a regression problem (a similar example could be price range or exact price ). After trying out a few models, I liked the use of a (random forest) regression model.

Besides evaluating the model on things like R^2 and RMSE I’m doing a sort of pseudo accuracy evaluation.

Say I have a prediction and a true value of

[19.8, 20]

So by true accuracy as in a classification problem the above is wrong, but if I define a new measure that tolerates answers within something fitting to the problem like +/- 2 or something like +/- 10% of the predicted value then the prediction is correct and the model will have greater accuracy. And then the prediction of a given sample would read something like x +/- y .

Or how would you display/interpret the predictions made by a regression model? Is it “correct” to measure the success as a pseudo accuracy as above? Or is it more correct and robust to express a prediction using e.g. RMSE as pred = x +/- RMSE ? Should I avoid this line of thinking when it comes to regression problems completely? And if such, how would I display my prediction of a given sample with a fitting confidence since the regression model typically is close but not always spot on the true value?

It sounds like evaluating the mode as a regression model would be better.

You would use MAE or RMSE to describe the expected error on average in the same units as the output variable.

E.g. The RMSE is xx.x on average +/- x.

You can use a confidence interval, I explain more here:

https://machinelearningmastery.com/confidence-intervals-for-machine-learning/

Hi Jason,

when I am assigning the X_Test to y_pred, it is returning the below shown error, can you please explain why?

y_pred = classifier.predict(X_Test)

Error:

NotFittedError Traceback (most recent call last)

in ()

—-> 1 y_pred = classifier.predict(X_Test)

C:\Anaconda\lib\site-packages\sklearn\neighbors\classification.py in predict(self, X)

143 X = check_array(X, accept_sparse=’csr’)

144

–> 145 neigh_dist, neigh_ind = self.kneighbors(X)

146

147 classes_ = self.classes_

C:\Anaconda\lib\site-packages\sklearn\neighbors\base.py in kneighbors(self, X, n_neighbors, return_distance)

325 “””

326 if self._fit_method is None:

–> 327 raise NotFittedError(“Must fit neighbors before querying.”)

328

329 if n_neighbors is None:

NotFittedError: Must fit neighbors before querying.

It suggests that perhaps your model has not been fit on the data.