Multioutput regression are regression problems that involve predicting two or more numerical values given an input example.

An example might be to predict a coordinate given an input, e.g. predicting x and y values. Another example would be multi-step time series forecasting that involves predicting multiple future time series of a given variable.

Many machine learning algorithms are designed for predicting a single numeric value, referred to simply as regression. Some algorithms do support multioutput regression inherently, such as linear regression and decision trees. There are also special workaround models that can be used to wrap and use those algorithms that do not natively support predicting multiple outputs.

In this tutorial, you will discover how to develop machine learning models for multioutput regression.

After completing this tutorial, you will know:

- The problem of multioutput regression in machine learning.
- How to develop machine learning models that inherently support multiple-output regression.
- How to develop wrapper models that allow algorithms that do not inherently support multiple outputs to be used for multiple-output regression.

Let’s get started.

## Tutorial Overview

This tutorial is divided into three parts; they are:

- Problem of Multioutput Regression
- Check Scikit-Learn Version
- Multioutput Regression Test Problem

- Inherently Multioutput Regression Algorithms
- Linear Regression for Multioutput Regression
- k-Nearest Neighbors for Multioutput Regression
- Random Forest for Multioutput Regression
- Evaluate Multioutput Regression With Cross-Validation

- Wrapper Multioutput Regression Algorithms
- Separate Model for Each Output (MultiOutputRegressor)
- Chained Models for Each Output (RegressorChain)

## Problem of Multioutput Regression

Regression refers to a predictive modeling problem that involves predicting a numerical value.

For example, predicting a size, weight, amount, number of sales, and number of clicks are regression problems. Typically, a single numeric value is predicted given input variables.

Some regression problems require the prediction of two or more numeric values. For example, predicting an x and y coordinate.

These problems are referred to as multiple-output regression, or multioutput regression.

**Regression**: Predict a single numeric output given an input.**Multioutput Regression**: Predict two or more numeric outputs given an input.

In multioutput regression, typically the outputs are dependent upon the input and upon each other. This means that often the outputs are not independent of each other and may require a model that predicts both outputs together or each output contingent upon the other outputs.

Multi-step time series forecasting may be considered a type of multiple-output regression where a sequence of future values are predicted and each predicted value is dependent upon the prior values in the sequence.

There are a number of strategies for handling multioutput regression and we will explore some of them in this tutorial.

### Check Scikit-Learn Version

First, confirm that you have a modern version of the scikit-learn library installed.

This is important because some of the models we will explore in this tutorial require a modern version of the library.

You can check the version of the library with the following code example:

1 2 3 |
# check scikit-learn version import sklearn print(sklearn.__version__) |

Running the example will print the version of the library.

At the time of writing, this is about version 0.22. You need to be using this version of scikit-learn or higher.

1 |
0.22.1 |

### Multioutput Regression Test Problem

We can define a test problem that we can use to demonstrate the different modeling strategies.

We will use the make_regression() function to create a test dataset for multiple-output regression. We will generate 1,000 examples with 10 input features, five of which will be redundant and five that will be informative. The problem will require the prediction of two numeric values.

**Problem Input**: 10 numeric variables.**Problem Output**: 2 numeric variables.

The example below generates the dataset and summarizes the shape.

1 2 3 4 5 6 |
# example of multioutput regression test problem from sklearn.datasets import make_regression # create datasets X, y = make_regression(n_samples=1000, n_features=10, n_informative=5, n_targets=2, random_state=1) # summarize dataset print(X.shape, y.shape) |

Running the example creates the dataset and summarizes the shape of the input and output elements of the dataset for modeling, confirming the chosen configuration.

1 |
(1000, 10) (1000, 2) |

Next, let’s look at modeling this problem directly.

## Inherently Multioutput Regression Algorithms

Some regression machine learning algorithms support multiple outputs directly.

This includes most of the popular machine learning algorithms implemented in the scikit-learn library, such as:

- LinearRegression (and related)
- KNeighborsRegressor
- DecisionTreeRegressor
- RandomForestRegressor (and related)

Let’s look at a few examples to make this concrete.

### Linear Regression for Multioutput Regression

The example below fits a linear regression model on the multioutput regression dataset, then makes a single prediction with the fit model.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
# linear regression for multioutput regression from sklearn.datasets import make_regression from sklearn.linear_model import LinearRegression # create datasets X, y = make_regression(n_samples=1000, n_features=10, n_informative=5, n_targets=2, random_state=1) # define model model = LinearRegression() # fit model model.fit(X, y) # make a prediction data_in = [[-2.02220122, 0.31563495, 0.82797464, -0.30620401, 0.16003707, -1.44411381, 0.87616892, -0.50446586, 0.23009474, 0.76201118]] yhat = model.predict(data_in) # summarize prediction print(yhat[0]) |

Running the example fits the model and then makes a prediction for one input, confirming that the model predicted two required values.

1 |
[-93.147146 23.26985013] |

### k-Nearest Neighbors for Multioutput Regression

The example below fits a k-nearest neighbors model on the multioutput regression dataset, then makes a single prediction with the fit model.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
# k-nearest neighbors for multioutput regression from sklearn.datasets import make_regression from sklearn.neighbors import KNeighborsRegressor # create datasets X, y = make_regression(n_samples=1000, n_features=10, n_informative=5, n_targets=2, random_state=1) # define model model = KNeighborsRegressor() # fit model model.fit(X, y) # make a prediction data_in = [[-2.02220122, 0.31563495, 0.82797464, -0.30620401, 0.16003707, -1.44411381, 0.87616892, -0.50446586, 0.23009474, 0.76201118]] yhat = model.predict(data_in) # summarize prediction print(yhat[0]) |

Running the example fits the model and then makes a prediction for one input, confirming that the model predicted two required values.

1 |
[-109.74862659 0.38754079] |

### Random Forest for Multioutput Regression

The example below fits a random forest model on the multioutput regression dataset, then makes a single prediction with the fit model.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
# random forest for multioutput regression from sklearn.datasets import make_regression from sklearn.ensemble import RandomForestRegressor # create datasets X, y = make_regression(n_samples=1000, n_features=10, n_informative=5, n_targets=2, random_state=1) # define model model = RandomForestRegressor() # fit model model.fit(X, y) # make a prediction data_in = [[-2.02220122, 0.31563495, 0.82797464, -0.30620401, 0.16003707, -1.44411381, 0.87616892, -0.50446586, 0.23009474, 0.76201118]] yhat = model.predict(data_in) # summarize prediction print(yhat[0]) |

Running the example fits the model and then makes a prediction for one input, confirming that the model predicted two required values.

1 |
[-76.79505796 27.16551641] |

### Evaluate Multioutput Regression With Cross-Validation

We may want to evaluate a multioutput regression using k-fold cross-validation.

This can be achieved in the same way as evaluating any other machine learning model.

We will fit and evaluate a *DecisionTreeRegressor* model on the test problem using 10-fold cross-validation with three repeats. We will use the mean absolute error (MAE) performance metric as the score.

The complete example is listed below.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
# evaluate multioutput regression model with k-fold cross-validation from numpy import absolute from numpy import mean from numpy import std from sklearn.datasets import make_regression from sklearn.tree import DecisionTreeRegressor from sklearn.model_selection import cross_val_score from sklearn.model_selection import RepeatedKFold # create datasets X, y = make_regression(n_samples=1000, n_features=10, n_informative=5, n_targets=2, random_state=1) # define model model = DecisionTreeRegressor() # evaluate model cv = RepeatedKFold(n_splits=10, n_repeats=3, random_state=1) n_scores = cross_val_score(model, X, y, scoring='neg_mean_absolute_error', cv=cv, n_jobs=-1, error_score='raise') # summarize performance n_scores = absolute(n_scores) print('Result: %.3f (%.3f)' % (mean(n_scores), std(n_scores))) |

Running the example evaluates the performance of the decision tree model for multioutput regression on the test problem. The mean and standard deviation of the MAE is reported calculated across all folds and all repeats.

Importantly, error is reported across both output variables, rather than separate error scores for each output variable.

1 |
Result: 51.659 (3.455) |

## Wrapper Multioutput Regression Algorithms

Not all regression algorithms support multioutput regression.

One example is the support vector machine, although for regression, it is referred to as support vector regression, or SVR.

This algorithm does not support multiple outputs for a regression problem and will raise an error. We can demonstrate this with an example, listed below.

1 2 3 4 5 6 7 8 9 |
# failure of support vector regression for multioutput regression from sklearn.datasets import make_regression from sklearn.svm import LinearSVR # create datasets X, y = make_regression(n_samples=1000, n_features=10, n_informative=5, n_targets=2, random_state=1) # define model model = LinearSVR() # fit model model.fit(X, y) |

Running the example reports an error message indicating that the model does not support multioutput regression.

1 |
ValueError: bad input shape (1000, 2) |

There are two workarounds that we can adopt in order to use an algorithm like SVR for multioutput regression.

They are to create a separate model for each output and to create a linear sequence of models, one for each output, where the output of each model is dependent upon the output of the previous models.

Thankfully, the scikit-learn library supports both of these cases. Let’s take a closer look at each.

### Separate Model for Each Output (MultiOutputRegressor)

We can create a separate model for each output of the problem.

This assumes that the outputs are independent of each other, which might not be a correct assumption. Nevertheless, this approach can provide surprisingly effective predictions on a range of problems and may be worth trying, at least as a performance baseline.

You never know. The outputs for your problem may, in fact, be mostly independent, if not completely independent, and this strategy can help you find out.

This approach is supported by the MultiOutputRegressor class that takes a regression model as an argument. It will then create one instance of the provided model for each output in the problem.

The example below demonstrates using the *MultiOutputRegressor* class with linear SVR for the test problem.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
# example of linear SVR with the MultiOutputRegressor wrapper for multioutput regression from sklearn.datasets import make_regression from sklearn.multioutput import MultiOutputRegressor from sklearn.svm import LinearSVR # create datasets X, y = make_regression(n_samples=1000, n_features=10, n_informative=5, n_targets=2, random_state=1) # define model model = LinearSVR() wrapper = MultiOutputRegressor(model) # fit model wrapper.fit(X, y) # make a prediction yhat = wrapper.predict(data_in) # summarize prediction print(yhat[0]) |

Running the example fits a separate LinearSVR for each of the outputs in the problem using the *MultiOutputRegressor* wrapper class.

This wrapper can then be used directly to make a prediction on new data, confirming that multiple outputs are supported.

1 |
[-93.147146 23.26985013] |

### Chained Models for Each Output (RegressorChain)

Another approach to using single-output regression models for multioutput regression is to create a linear sequence of models.

The first model in the sequence uses the input and predicts one output; the second model uses the input and the output from the first model to make a prediction; the third model uses the input and output from the first two models to make a prediction, and so on.

This can be achieved using the RegressorChain class in the scikit-learn library.

The order of the models may be based on the order of the outputs in the dataset (the default) or specified via the “*order*” argument. For example, *order=[0,1]* would first predict the 0th output, then the 1st output, whereas an *order=[1,0]* would first predict the last output variable and then the first output variable in our test problem.

The example below uses the *RegressorChain* with the default output order to fit a linear SVR on the multioutput regression test problem.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
# example of fitting a chain of linear SVR for multioutput regression from sklearn.datasets import make_regression from sklearn.multioutput import RegressorChain from sklearn.svm import LinearSVR # create datasets X, y = make_regression(n_samples=1000, n_features=10, n_informative=5, n_targets=2, random_state=1) # define model model = LinearSVR() wrapper = RegressorChain(model) # fit model wrapper.fit(X, y) # make a prediction yhat = wrapper.predict(data_in) # summarize prediction print(yhat[0]) |

Running the example first fits a linear SVR to predict the first output variable, then a second linear SVR to predict the second output variable using the input and the output of the first model. These models are fit on the entire dataset.

The fit chain of models is then used directly to make a prediction on a new test instance, predicting the required two output variables.

1 |
[-93.147146 23.26938475] |

## Further Reading

This section provides more resources on the topic if you are looking to go deeper.

### APIs

- Multiclass and multilabel algorithms, API.
- sklearn.datasets.make_regression API.
- sklearn.multioutput.MultiOutputRegressor API.
- sklearn.multioutput.RegressorChain API.

## Summary

In this tutorial, you discovered how to develop machine learning models for multioutput regression.

Specifically, you learned:

- The problem of multioutput regression in machine learning.
- How to develop machine learning models that inherently support multiple-output regression.
- How to develop wrapper models that allow algorithms that do not inherently support multiple outputs to be used for multiple-output regression.

Do you have any questions?

Ask your questions in the comments below and I will do my best to answer.

Thank you for this post. I was not aware that scikit-learn had those wrapper classes. That is very handy.

Thanks for show how to use them in a very clear straightforward way.

You’re welcome.

Sereasly interested.

Thanks.

It is just amazing. Thank you

You’re very welcome!

Hi Dr. Jason,

Thanks for the meaningful tutorial article, it helps me a lot.

I just tested and found all of the code works well in sklearn 0.20, so we don’t necessarily need to update to 0.22 or higher.

Thanks again for the excellent article, looking forward to the next one.

Thanks, great tip!

Hi Dr. Brownlee,

Thanks for the excellent work!

I have a question, can the wrapped model (either using MultiOutputRegressor or RegressorChain ) work with cross_val_score function? I tested the following code and found the score is 0.0, is there anything wrong with that?

from numpy import absolute

from numpy import mean

from numpy import std

from sklearn.datasets import make_regression

from sklearn.multioutput import MultiOutputRegressor

from sklearn.svm import LinearSVR

from sklearn.model_selection import cross_val_score

from sklearn.model_selection import RepeatedKFold

# create datasets

X, y = make_regression(n_samples=1000, n_features=10, n_informative=5, n_targets=2, random_state=1)

# define model

model = LinearSVR()

wrapper = MultiOutputRegressor(model)

# evaluate model

cv = RepeatedKFold(n_splits=10, n_repeats=3, random_state=1)

n_scores = cross_val_score(wrapper, X, y, scoring=’neg_mean_absolute_error’, cv=cv, n_jobs=-1, error_score=’raise’)

# summarize performance

n_scores = absolute(n_scores)

print(‘Result: %.3f (%.3f)’ % (mean(n_scores), std(n_scores)))

Thank you in advance!

Yes.

A score of 0 means perfect predictions.

Thanks for a really good and pedagogical tutorial. Fits my need exactly! 🙂

Thanks, I’m very happy to hear that!

Hi Jason!

Excellent tutorial. Is it possible to apply a Transformed Target Regressor with this multi-output regression models?

Thanks in advance!

Probably. You might have to experiment to confirm it works as expected.

Hi Jason,

Thanks a lot for this tutorial. It suits my need perfectly.

I have a question regarding the correlation among different Y variables. Do the KNN or Random Forest models automatically consider the correlation among different Y while predicting? If they do that, is there any documentation on how it is done? I tried to look for it online, but found nothing. Could you please guide me to some resources? Thanks

You’re welcome.

No, but some algorithms are not bothered by colinear inputs (ensembles of trees I think), and some are (linear models).

Thanks for this very interesting tutorial. I have a question for some cases where we have for example about 500 inputs and 200 outputs. Does this Multi-Output Regression way work with such problems?(In fact I have a about 20 features and 500 input points that are extracted form scan-1 and 200 output points from scan-2.)

You’re welcome.

Perhaps develop a quick prototype and see if the methods are appropriate and effective?

Thanks for your reply:), the problem is my current data is not completely ready and I have to wait, so I was also thinking about Deep-learning methods such as convolution, but I have to do some research on it. Do you have any suggestion for such types of problems? It is something like image processing because it the input data and the output data are points that are extracted from scan(images).

If the input data are images, then CNNs would be a good method to explore.

Actually input data are digits that are extracted from images. But as the number of my sample data are limited (about 30 samples) I cannot use CNNs. And I decided to convert one image into about 500 points/digits. Can you please let me know your opinion?

Test a suite of data preparations and models and discover what works best for your dataset.

This is great. But i have a question on the metrics of these. Based on your test, which one would you say provided the best prediction?

It depends on your dataset.

You must use controlled experiments to discover what works best on your project.

Why not simply use a neural network with multiple neurons in output layer or build a multi-target decision tree and let the model figure out patterns instead of us (the modelers) having to decide whether or not the outputs are correlated/ordered? If there’s a strong rationale for ordered outputs, then RegressorChain might be a good option.

PS: I believe by default sklearn’s decision trees are amenable to this as per this post https://stackoverflow.com/questions/46062774/does-scikit-learns-decisiontreeregressor-do-true-multi-output-regression

You can use a neural net, but we cannot know which model will best for a given dataset, so we must test many different methods and select the simplest well performing model.

Jason your tutorials is simply amazing. I love them!

Congrats for this awesome material.

Thanks!

It seems wrapper method doesn’t work on some ensemble, I tried this:

model = GradientBoostingRegressor(ExtraTreesRegressor())

wrapper = MultiOutputRegressor(model)

wrapper.fit(x,y)

it said “TypeError: unsupported format string passed to ExtraTreesRegressor.__format__”

But it works on AdaBoostRegressor.

Anyway, this is a good tutorial, and it gave many ideas to my final year project, thank you Jason!

You’re welcome.

Interesting.

I believe ensembles of trees support multi-output regression directly – e.g. not wrapper required.

Thank you very much !! I was looking for something like this and was unable to find ways for multi output predictions other than the four you mentioned ,now i did .

You’re welcome!

Thank you

Also i wanted to ask a question:

I have coordinates in table (in time series)

like

a ,b,c->d

b,c,d->e

and so on

where each point is in the form of [lat, long]

So this is where I wanted to use a multiple regression output

And I want to calculate error as well , which error metric would you suggest be best?

I wanted to use MAPE but it has the form like diff/actual… where actual is of the form ([lat, long)]

so I cant divide it with a list, it needs a single value. Any suggestions?

You’re welcome.

I recommend choosing a metric that best captures the goals of the project for you and project stakeholders. Also, perhaps check the literature to see what others have done on the same type of problem before you.

If you are unsure MAE and RMSE are a great place to start.

Very helpful!

When using Chained models using wrapper class, can we use separate models and then wrap them all to fit?

What do you mean exactly? Perhaps you can elaborate.

Thanks for this tutorial which I used in conjunction with the eBook.

Can I ask – i am working on a multi-input/multioutput problem, with 3 sets of input (each set with 7 different variables with >10000 values). I have one output set (8 variables) for the combined 3 input sets.

What’s the best way to import/organise this data (vectors/arrays?) at the beginning of the process to make it easy to use with scripts?

Well, generally sklearn expects input as a vector of rows and columns. So working with data in that form might be easier for you.

Really good post. It really helps me a lot. I have two questions:

For Linear Regression for Multioutput in scikit-learn, when call fit(X, y). It is actually fit Separate Model for Each Output. Am I right?

Second question, I am using kernel_ridge linear regression(RBF kernel). I use cross validation to choose 2 hyper-parameter- alpha: the parameter for L2 regulazation, and gamma:the parameter for RBF kernel. When I call

kr = GridSearchCV(KernelRidge(kernel=’rbf’, gamma=0.1),

param_grid={“alpha”: np.logspace(-2, 0, 10),

“gamma”: np.logspace(-2, 0, 10)},

scoring=’neg_mean_squared_error’)

It fit Separate Model for each output ,but the Separate models share the same ‘alpha’ and ‘gamma’ in each CV parameter search?

Thanks!

Yes, MultiOutputRegressor fits a separate model for each target as described in the tutorial.

Correct, they will both use the same hyperparameters. If this is not desirable, you can fit separate models manually.

Thanks for your reply. Also I found Gaussian process regression model in scikit learn- GaussianProcessRegressor- support mutioutput. Is the model actually model each outhput dimension as a single gaussian process regression problem?

Perhaps check the documentation for the model.