Exploratory Configuration of a Multilayer Perceptron Network for Time Series Forecasting

Last Updated on

It can be difficult when starting out on a new predictive modeling project with neural networks.

There is so much to configure, and no clear idea where to start.

It is important to be systematic. You can break bad assumptions and quickly hone in on configurations that work and areas for further investigation likely to payoff.

In this tutorial, you will discover how to use exploratory configuration of multilayer perceptron (MLP) neural networks to find good first-cut models for time series forecasting.

After completing this tutorial, you will know:

  • How to design a robust experimental test harness to evaluate MLP models for time series forecasting.
  • Systematic experimental designs for varying epochs, neurons, and lag configurations.
  • How to interpret results and use diagnostics to learn more about well-performing models.

Discover how to build models for multivariate and multi-step time series forecasting with LSTMs and more in my new book, with 25 step-by-step tutorials and full source code.

Let’s get started.

  • Updated Jul/2017: Changed function for creating models to be more descriptive.
  • Updated Apr/2019: Updated the link to dataset.

 

Exploratory Configuration of a Multilayer Perceptron Network for Time Series Forecasting

Exploratory Configuration of a Multilayer Perceptron Network for Time Series Forecasting
Photo by Lachlan Donald, some rights reserved.

Tutorial Overview

This tutorial is broken down into 6 parts. They are:

  1. Shampoo Sales Dataset
  2. Experimental Test Harness
  3. Vary Training Epochs
  4. Vary Hidden Layer Neurons
  5. Vary Hidden Layer Neurons with Lag
  6. Review of Results

Environment

This tutorial assumes you have a Python SciPy environment installed. You can use either Python 2 or 3 with this example.

This tutorial assumes you have Keras v2.0 or higher installed with either the TensorFlow or Theano backend.

This tutorial also assumes you have scikit-learn, Pandas, NumPy, and Matplotlib installed.

Next, let’s take a look at a standard time series forecasting problem that we can use as context for this experiment.

If you need help setting up your Python environment, see this post:

Shampoo Sales Dataset

This dataset describes the monthly number of sales of shampoo over a 3-year period.

The units are a sales count and there are 36 observations. The original dataset is credited to Makridakis, Wheelwright, and Hyndman (1998).

The example below loads and creates a plot of the loaded dataset.

Running the example loads the dataset as a Pandas Series and prints the first 5 rows.

A line plot of the series is then created showing a clear increasing trend.

Line Plot of Shampoo Sales Dataset

Line Plot of Shampoo Sales Dataset

Next, we will take a look at the model configuration and test harness used in the experiment.

Experimental Test Harness

This section describes the test harness used in this tutorial.

Data Split

We will split the Shampoo Sales dataset into two parts: a training and a test set.

The first two years of data will be taken for the training dataset and the remaining one year of data will be used for the test set.

Models will be developed using the training dataset and will make predictions on the test dataset.

The persistence forecast (naive forecast) on the test dataset achieves an error of 136.761 monthly shampoo sales. This provides a lower acceptable bound of performance on the test set.

Model Evaluation

A rolling-forecast scenario will be used, also called walk-forward model validation.

Each time step of the test dataset will be walked one at a time. A model will be used to make a forecast for the time step, then the actual expected value from the test set will be taken and made available to the model for the forecast on the next time step.

This mimics a real-world scenario where new Shampoo Sales observations would be available each month and used in the forecasting of the following month.

This will be simulated by the structure of the train and test datasets.

All forecasts on the test dataset will be collected and an error score calculated to summarize the skill of the model. The root mean squared error (RMSE) will be used as it punishes large errors and results in a score that is in the same units as the forecast data, namely monthly shampoo sales.

Data Preparation

Before we can fit an MLP model to the dataset, we must transform the data.

The following three data transforms are performed on the dataset prior to fitting a model and making a forecast.

  1. Transform the time series data so that it is stationary. Specifically, a lag=1 differencing to remove the increasing trend in the data.
  2. Transform the time series into a supervised learning problem. Specifically, the organization of data into input and output patterns where the observation at the previous time step is used as an input to forecast the observation at the current timestep
  3. Transform the observations to have a specific scale. Specifically, to rescale the data to values between -1 and 1.

These transforms are inverted on forecasts to return them into their original scale before calculating and error score.

MLP Model

We will use a base MLP model with 1 neuron hidden layer, a rectified linear activation function on hidden neurons, and linear activation function on output neurons.

A batch size of 4 is used where possible, with the training data truncated to ensure the number of patterns is divisible by 4. In some cases a batch size of 2 is used.

Normally, the training dataset is shuffled after each batch or each epoch, which can aid in fitting the training dataset on classification and regression problems. Shuffling was turned off for all experiments as it seemed to result in better performance. More studies are needed to confirm this result for time series forecasting.

The model will be fit using the efficient ADAM optimization algorithm and the mean squared error loss function.

Experimental Runs

Each experimental scenario will be run 30 times and the RMSE score on the test set will be recorded from the end each run.

Let’s dive into the experiments.

Need help with Deep Learning for Time Series?

Take my free 7-day email crash course now (with sample code).

Click to sign-up and also get a free PDF Ebook version of the course.

Download Your FREE Mini-Course

Vary Training Epochs

In this first experiment, we will investigate varying the number of training epochs for a simple MLP with one hidden layer and one neuron in the hidden layer.

We will use a batch size of 4 and evaluate training epochs 50, 100, 500, 1000, and 2000.

The complete code listing is provided below.

This code listing will be used as the basis for all following experiments, with only the changes to this code provided in subsequent sections.

Running the experiment prints the test set RMSE at the end of each experimental run.

At the end of all runs, a table of summary statistics is provided, one row for each statistic and one configuration for each column.

The summary statistics suggest that on average 1000 training epochs resulted in the better performance with a general decreasing trend in error with the increase of training epochs.

A box and whisker plot of the distribution of test RMSE scores for each configuration was also created and saved to file.

The plot highlights that each configuration shows the same general spread in test RMSE scores (box), with the median (green line) trending downward with the increase of training epochs.

The results confirm that the configured MLP trained for 1000 is a good starting point on this problem.

Box and Whisker Plot of Vary Training Epochs for Time Series Forecasting on the Shampoo Sales Dataset

Box and Whisker Plot of Varying Training Epochs for Time Series Forecasting on the Shampoo Sales Dataset

Another angle to consider with a network configuration is how it behaves over time as the model is being fit.

We can evaluate the model on the training and test datasets after each training epoch to get an idea as to if the configuration is overfitting or underfitting the problem.

We will use this diagnostic approach on the top result from each set of experiments. A total of 10 repeats of the configuration will be run and the train and test RMSE scores after each training epoch plotted on a line plot.

In this case, we will use this diagnostic on the MLP fit for 1000 epochs.

The complete diagnostic code listing is provided below.

As with the previous code listing, the code listing below will be used as the basis for all diagnostics in this tutorial and only the changes to this listing will be provided in subsequent sections.

Running the diagnostic prints the final train and test RMSE for each run. More interesting is the final line plot created.

The line plot shows the train RMSE (blue) and test RMSE (orange) after each training epoch.

In this case, the diagnostic plot shows little difference in train and test RMSE after about 400 training epochs. Both train and test performance level out on a near flat line.

This rapid leveling out suggests the model is reaching capacity and may benefit from more information in terms of lag observations or additional neurons.

Diagnostic Line Plot of Train and Test Performance of 400 Epochs on the Shampoo Sales Dataset

Diagnostic Line Plot of Train and Test Performance of 1000 Epochs on the Shampoo Sales Dataset

Vary Hidden Layer Neurons

In this section, we will look at varying the number of neurons in the single hidden layer.

Increasing the number of neurons can increase the learning capacity of the network at the risk of overfitting the training data.

We will explore increasing the number of neurons from 1 to 5 and fit the network for 1000 epochs.

The differences in the experiment script are listed below.

Running the experiment prints summary statistics for each configuration.

Looking at the average performance, it suggests a decrease of test RMSE with an increase in the number of neurons in the single hidden layer.

The best results appear to be with 3 neurons.

A box and whisker plot is also created to summarize and compare the distributions of results.

The plot confirms the suggestion of 3 neurons performing well compared to the other configurations and suggests in addition that the spread of results is also smaller. This may indicate a more stable configuration.

Box and Whisker Plot of Varying Hidden Neurons for Time Series Forecasting on the Shampoo Sales Dataset

Box and Whisker Plot of Varying Hidden Neurons for Time Series Forecasting on the Shampoo Sales Dataset

Again, we can dive a little deeper by reviewing diagnostics of the chosen configuration of 3 neurons fit for 1000 epochs.

The changes to the diagnostic script are limited to the run() function and listed below.

Running the diagnostic script provides a line plot of train and test RMSE for each training epoch.

The diagnostics suggest a flattening out of model skill, perhaps around 400 epochs. The plot also suggests a possible situation of overfitting where there is a slight increase in test RMSE over the last 500 training epochs, but not a strong increase in training RMSE.

Diagnostic Line Plot of Train and Test Performance of 3 Hidden Neurons on the Shampoo Sales Dataset

Diagnostic Line Plot of Train and Test Performance of 3 Hidden Neurons on the Shampoo Sales Dataset

Vary Hidden Layer Neurons with Lag

In this section, we will look at increasing the lag observations as input, whilst at the same time increasing the capacity of the network.

Increased lag observations will automatically scale the number of input neurons. For example, 3 lag observations as input will result in 3 input neurons.

The added input will require additional capacity in the network. As such, we will also scale the number of neurons in the one hidden layer with the number of lag observations used as input.

We will use odd numbers of lag observations as input from 1, 3, 5, and 7 and use the same number of neurons respectively.

The change to the number of inputs affects the total number of training patterns during the conversion of the time series data to a supervised learning problem. As such, the batch size was reduced from 4 to 2 for all experiments in this section.

A total of 1000 training epochs are used in each experimental run.

The changes from the base experiment script are limited to the experiment() function and the running of the experiment, listed below.

Running the experiment summarizes the results using descriptive statistics for each configuration.

The results suggest that all increases in lag input variables with increases with hidden neurons decrease performance.

Of note is the 1 neuron and 1 input configuration, which compared to the results from the previous section resulted in a similar mean and standard deviation.

It is possible that the decrease in performance is related to the smaller batch size and that the results from the 1-neuron/1-lag case are insufficient to tease this out.

A box and whisker plot of the distribution of results was also created allowing configurations to be compared.

Interestingly, the use of 3 neurons and 3 input variables shows a tighter spread compared to the other configurations. This is similar to the observation from 3 neurons and 1 input variable seen in the previous section.

Box and Whisker Plot of Varying Lag Features and Hidden Neurons for Time Series Forecasting on the Shampoo Sales Dataset

Box and Whisker Plot of Varying Lag Features and Hidden Neurons for Time Series Forecasting on the Shampoo Sales Dataset

We can also use diagnostics to tease out how the dynamics of the model might have changed while fitting the model.

The results for 3-lags/3-neurons are interesting and we will investigate them further.

The changes to the diagnostic script are confined to the run() function.

Running the diagnostics script creates a line plot showing the train and test RMSE after each training epoch for 10 experimental runs.

The results suggest good learning during the first 500 epochs and perhaps overfitting in the remaining epochs with the test RMSE showing an increasing trend and the train RMSE showing a decreasing trend.

Diagnostic Line Plot of Train and Test Performance of 3 Hidden Neurons and Lag Features on the Shampoo Sales Dataset

Diagnostic Line Plot of Train and Test Performance of 3 Hidden Neurons and Lag Features on the Shampoo Sales Dataset

Review of Results

We have covered a lot of ground in this tutorial. Let’s review.

  • Epochs. We looked at how model skill varied with the number of training epochs and found that 1000 might be a good starting point.
  • Neurons. We looked at varying the number of neurons in the hidden layer and found that 3 neurons might be a good configuration.
  • Lag Inputs. We looked at varying the number of lag observations as inputs whilst at the same time increasing the number of neurons in the hidden layer and found that results generally got worse, but again, 3 neurons in the hidden layer shows interest. Poor results may have been related to the change of batch size from 4 to 2 compared to other experiments.

The results suggest using a 1 lag input, 3 neurons in the hidden layer, and fit for 1000 epochs as a first-cut model configuration.

This can be improved upon in many ways; the next section lists some ideas.

Extensions

This section lists extensions and follow-up experiments you might like to explore.

  • Shuffle vs No Shuffle. No shuffling was used, which is abnormal. Develop an experiment to compare shuffling to no shuffling of the training set when fitting the model for time series forecasting.
  • Normalization Method. Data was rescaled to -1 to 1, typical for a tanh activation function, not used in the model configurations. Explore other rescaling, such as 0-1 normalization and standardization and the impact on model performance.
  • Multiple Layers. Explore the use of multiple hidden layers to add network capacity to learn more complex multi-step patterns.
  • Feature Engineering. Explore the use of additional features, such as an error time series and even elements of the date-time of each observation.

Also, check out the post:

Did you try any of these extensions?
Post your results in the comments below.

Summary

In this tutorial, you discovered how to use systematic experiments to explore the configuration of a multilayer perceptron for time series forecasting and develop a first-cut model.

Specifically, you learned:

  • How to develop a robust test harness for evaluating MLP models for time series forecasting.
  • How to systematically evaluate training epochs, hidden layer neurons, and lag inputs.
  • How to use diagnostics to help interpret results and suggest follow-up experiments.

Do you have any questions about this tutorial?
Ask your questions in the comments below and I will do my best to answer.

Develop Deep Learning models for Time Series Today!

Deep Learning for Time Series Forecasting

Develop Your Own Forecasting models in Minutes

...with just a few lines of python code

Discover how in my new Ebook:
Deep Learning for Time Series Forecasting

It provides self-study tutorials on topics like:
CNNs, LSTMs, Multivariate Forecasting, Multi-Step Forecasting and much more...

Finally Bring Deep Learning to your Time Series Forecasting Projects

Skip the Academics. Just Results.

See What's Inside

38 Responses to Exploratory Configuration of a Multilayer Perceptron Network for Time Series Forecasting

  1. elina April 26, 2017 at 11:00 pm #

    what is mean by transforming observations to have specific scale? can u more elaborate it with simple example?

  2. Kunpeng Zhang April 28, 2017 at 3:12 am #

    Hi Jason,
    I have a question regarding the prediction result evaluation.
    In my code, I get MSE MAE MAPE calculated by keras as follows:
    model.compile(loss=’mse’, optimizer=optimizer, metrics=[‘mae’, ‘mape’])

    Epoch 150/150
    0s – loss: 0.0038 – mean_absolute_error: 0.0455 – mean_absolute_percentage_error: 164095.5176
    mse=0.003538, mae=0.043654, mape=58238.251235
    ————————————————————————-
    But when I compute these values by the code:
    mse = mean_squared_error(test_Y, predicted_output)
    rmse = math.sqrt(mse)
    mae = mean_absolute_error(test_Y, predicted_output)
    mape = np.mean(np.abs(np.divide(np.subtract(test_Y, predicted_output), test_Y))) * 100
    RMSE: 14.992
    MAE: 10.462
    MAPE: 2.208

    The two results are very different from each other. What happened?

    • Jason Brownlee April 28, 2017 at 7:53 am #

      That is interesting. I’m not sure what is going on here.

      I would trust the manual results. I have not had any issues like this myself, my epoch scores always seem to match my post-training evaluation.

      Consider preparing a small self-contained example and posting it as a bug to the Keras project:
      https://github.com/fchollet/keras

      • Kunpeng Zhang April 29, 2017 at 11:46 pm #

        I give it a try on your example.
        http://machinelearningmastery.com/time-series-prediction-lstm-recurrent-neural-networks-python-keras/

        Change
        model.compile(loss=’mean_squared_error’, optimizer=’adam’)
        to
        model.compile(loss=’mean_squared_error’, optimizer=’adam’, metrics=[‘mae’, ‘mape’])
        Got the same result.
        0s – loss: 0.0019 – mean_absolute_error: 0.0343 – mean_absolute_percentage_error: 620693.8651
        Epoch 96/100
        0s – loss: 0.0020 – mean_absolute_error: 0.0351 – mean_absolute_percentage_error: 326543.9207
        Epoch 97/100
        0s – loss: 0.0019 – mean_absolute_error: 0.0345 – mean_absolute_percentage_error: 488762.9108
        Epoch 98/100
        0s – loss: 0.0019 – mean_absolute_error: 0.0345 – mean_absolute_percentage_error: 514091.1566
        Epoch 99/100
        0s – loss: 0.0019 – mean_absolute_error: 0.0345 – mean_absolute_percentage_error: 531419.0410
        Epoch 100/100
        0s – loss: 0.0019 – mean_absolute_error: 0.0341 – mean_absolute_percentage_error: 454424.3737
        Train Score: 22.34 RMSE
        Test Score: 45.66 RMSE

  3. Hans April 28, 2017 at 6:12 pm #

    What is the most robust sign for overfitting?

    • Jason Brownlee April 29, 2017 at 7:22 am #

      Skill on the test set is worse than the training set.

      • Hans April 30, 2017 at 3:50 pm #

        Is there a list of overfitting signs available?

        Alternative:

        If I want to prepare such a list, how should it look like?

        1. Skill on the test set is worse than the training set.
        2. ?
        3. ?
        4. ?
        5. ?

        • Hans April 30, 2017 at 3:51 pm #

          Is there a function available for our code, which ‘alerts’ overfitting?

          • Hans April 30, 2017 at 4:00 pm #

            I monitor every result and multiple parameters of your examples in a Sqlite database.

            Therefore I could easelly tagging overfitting.

            For example if a test result is worse then the training set.

            Could there be more relations of parameters and results to monitor for overfitting in general?

          • Jason Brownlee May 1, 2017 at 5:55 am #

            It is a trend you are seeking for overfitting, not necessarily one result being worse.
            https://en.wikipedia.org/wiki/Overfitting

          • Jason Brownlee May 1, 2017 at 5:54 am #

            No.

        • Jason Brownlee May 1, 2017 at 5:54 am #

          Nope. The first is all you need.

          Instead, you develop a list of 100s of ways to address overfitting/early convergence.

  4. Hans April 30, 2017 at 4:52 pm #

    Could we assume that the best performing epoches, neurons and lag inputs found with this MLP setup are also the best performing ones in a LSTM model or any other model?

    • Jason Brownlee May 1, 2017 at 5:56 am #

      No, generally findings are not transferable to other problems or other algorithms.

  5. Hans May 1, 2017 at 8:59 pm #

    Is it possible to save a trained model on HD, to predict unseen data later, in a shorter time?

  6. Magnus May 12, 2017 at 11:22 pm #

    Would it be possible to achieve the same study using GridSearchCV, like you did in another post? Or is it not possible for time series?

  7. Jussi December 4, 2017 at 6:07 am #

    Hi Jason,

    Thanks for a great tutorial.

    Could you give me an advice how to model, a multivariate timeseries with MLP. For example, if we model a timeseries such an energy consumption with window approach like in the example above. How do we add/model additional features such as dates, temperature, etc, which can be also timeseries?

    Is the only option to create a single array with all the different timeseries one after another and hope that the network can learn their relations? If yes, it seems strange that there are no way to indicate which inputs relate to each other timewise… But I guess the LSTM and other RNN are just for this purpose…

    • Jason Brownlee December 4, 2017 at 8:00 am #

      You could try modeling each series with separate networks and ensemble the models, you could try one large model for all series, I’d encourage you to experiment and see what works best for your data.

      • kingsley udeh August 2, 2018 at 9:21 pm #

        Hi Jason,

        Please do you have any example of this:

        “You could try modeling each series with separate networks and ensemble the models…”

        Could you please give more explanation of how we could combine the models to form an ensample?

        In this case, any need for a final model? That is, fitting the ensembled model(s) with all available data?

  8. Steven February 20, 2018 at 6:13 am #

    Hi Jason,

    Thanks for the great tutorials you offer. I have both tried the LSTM examples and this MLP example. All these examples gave me good insights and helped me to configure models and do optimizations for my own data.

    A characteristic of time series is that the predictions (forecasts) are based on history data. But in this MLP example I observed the testing data is used to make the predictions:
    output = model.predict(test_reshaped, batch_size=batch_size)
    I think this essentially is a matter of regression and not time series. The model can only predict values based on input values and the model is not able to predict solely based on history data. Can you please comment on this?

  9. Amelie November 26, 2018 at 11:23 pm #

    Thank you for your demo,

    I want to know please, the difference between a NAR and a MLP?
    and how a MLP can take into consideration the time aspect of the series..

    Can you lead me to an example of application of a BPTT (Backpropagation through time)

  10. Ehsan December 8, 2018 at 7:04 am #

    Thank you Jason. I run your code and got the info that you said it is important.(n neurons, epochs and etc)so now how can I use these parameters to run the model and plot the future price predictions? in auto arima I got the plot that shows future vs original values but here I don’t know how to do it. thanks

  11. mk December 29, 2018 at 4:53 pm #

    Can MLP model deal with multivariate inputs/outputs,Multi-step Time Series?

    Your work is wonderful in this post https://machinelearningmastery.com/how-to-develop-convolutional-neural-networks-for-multi-step-time-series-forecasting/#comment-460415

    Multi-step Time Series, multivariate inputs and outputs are necessary for my work.

  12. Hala October 28, 2019 at 10:16 pm #

    How can plotting rmse boxplot for repeating range with constant epoch

    • Jason Brownlee October 29, 2019 at 5:24 am #

      Sorry, I don’t understand, can you elaborate?

  13. Hala October 29, 2019 at 8:48 pm #

    You,here for example, plots Box and Whisker Plot of Varying Training Epochs
    I want to plot box and whisker plot for rmse error for varying the repeats time for test experiment.

Leave a Reply