Time Series data must be re-framed as a supervised learning dataset before we can start using machine learning algorithms.

There is no concept of input and output features in time series. Instead, we must choose the variable to be predicted and use feature engineering to construct all of the inputs that will be used to make predictions for future time steps.

In this tutorial, you will discover how to perform feature engineering on time series data with Python to model your time series problem with machine learning algorithms.

After completing this tutorial, you will know:

- The rationale and goals of feature engineering time series data.
- How to develop basic date-time based input features.
- How to develop more sophisticated lag and sliding window summary statistics features.

Let’s dive in.

**Update June/2017**: Fixed a typo in the expanding window code example.

## Feature Engineering for Time Series

A time series dataset must be transformed to be modeled as a supervised learning problem.

That is something that looks like:

1 2 3 |
time 1, value 1 time 2, value 2 time 3, value 3 |

To something that looks like:

1 2 3 |
input 1, output 1 input 2, output 2 input 3, output 3 |

So that we can train a supervised learning algorithm.

Input variables are also called features in the field of machine learning, and the task before us is to create or invent new input features from our time series dataset. Ideally, we only want input features that best help the learning methods model the relationship between the inputs (**X**) and the outputs (**y**) that we would like to predict.

In this tutorial, we will look at three classes of features that we can create from our time series dataset:

**Date Time Features**: these are components of the time step itself for each observation.**Lag Features**: these are values at prior time steps.**Window Features**: these are a summary of values over a fixed window of prior time steps.

Before we dive into methods for creating input features from our time series data, let’s first review the goal of feature engineering.

### Stop learning Time Series Forecasting the *slow way*!

Take my free 7-day email course and discover data prep, modeling and more (with sample code).

Click to sign-up and also get a free PDF Ebook version of the course.

## Goal of Feature Engineering

The goal of feature engineering is to provide strong and ideally simple relationships between new input features and the output feature for the supervised learning algorithm to model.

In effect, we are are moving complexity.

Complexity exists in the relationships between the input and output data. In the case of time series, there is no concept of input and output variables; we must invent these too and frame the supervised learning problem from scratch.

We may lean on the capability of sophisticated models to decipher the complexity of the problem. We can make the job for these models easier (and even use simpler models) if we can better expose the inherent relationship between inputs and outputs in the data.

The difficulty is that we do not know the underlying inherent functional relationship between inputs and outputs that we’re trying to expose. If we did know, we probably would not need machine learning.

Instead, the only feedback we have is the performance of models developed on the supervised learning datasets or “views” of the problem we create. In effect, the best default strategy is to use all the knowledge available to create many good datasets from your time series dataset and use model performance (and other project requirements) to help determine what good features and good views of your problem happen to be.

For clarity, we will focus on a univariate (one variable) time series dataset in the examples, but these methods are just as applicable to multivariate time series problems. Next, let’s take a look at the dataset we will use in this tutorial.

## Minimum Daily Temperatures Dataset

In this post, we will use the Minimum Daily Temperatures dataset.

This dataset describes the minimum daily temperatures over 10 years (1981-1990) in Melbourne, Australia.

The units are in degrees Celsius and there are 3,650 observations. The source of the data is credited as the Australian Bureau of Meteorology.

Below is a sample of the first 5 rows of data, including the header row.

1 2 3 4 5 6 |
"Date","Temperature" "1981-01-01",20.7 "1981-01-02",17.9 "1981-01-03",18.8 "1981-01-04",14.6 "1981-01-05",15.8 |

Below is a plot of the entire dataset taken from Data Market.

The dataset shows an increasing trend and possibly some seasonal components.

Download and learn more about the dataset here.

**Note**: The downloaded file contains some question mark (“?”) characters that must be removed before you can use the dataset. Open the file in a text editor and remove the “?” characters. Also remove any footer information in the file.

## Date Time Features

Let’s start with some of the simplest features that we can use.

These are features from the date/time of each observation. In fact, these can start off simply and head off into quite complex domain-specific areas.

Two features that we can start with are the integer month and day for each observation. We can imagine that supervised learning algorithms may be able to use these inputs to help tease out time-of-year or time-of-month type seasonality information.

The supervised learning problem we are proposing is to predict the daily minimum temperature given the month and day, as follows:

1 2 3 |
Month, Day, Temperature Month, Day, Temperature Month, Day, Temperature |

We can do this using Pandas. First, the time series is loaded as a Pandas *Series*. We then create a new Pandas *DataFrame* for the transformed dataset.

Next, each column is added one at a time where month and day information is extracted from the time-stamp information for each observation in the series.

Below is the Python code to do this.

1 2 3 4 5 6 7 8 |
from pandas import Series from pandas import DataFrame series = Series.from_csv('daily-minimum-temperatures-in-me.csv', header=0) dataframe = DataFrame() dataframe['month'] = [series.index[i].month for i in range(len(series))] dataframe['day'] = [series.index[i].day for i in range(len(series))] dataframe['temperature'] = [series[i] for i in range(len(series))] print(dataframe.head(5)) |

Running this example prints the first 5 rows of the transformed dataset.

1 2 3 4 5 6 |
month day temperature 0 1 1 20.7 1 1 2 17.9 2 1 3 18.8 3 1 4 14.6 4 1 5 15.8 |

Using just the month and day information alone to predict temperature is not sophisticated and will likely result in a poor model. Nevertheless, this information coupled with additional engineered features may ultimately result in a better model.

You may enumerate all the properties of a time-stamp and consider what might be useful for your problem, such as:

- Minutes elapsed for the day.
- Hour of day.
- Business hours or not.
- Weekend or not.
- Season of the year.
- Business quarter of the year.
- Daylight savings or not.
- Public holiday or not.
- Leap year or not.

From these examples, you can see that you’re not restricted to the raw integer values. You can use binary flag features as well, like whether or not the observation was recorded on a public holiday.

In the case of the minimum temperature dataset, maybe the season would be more relevant. It is creating domain-specific features like this that are more likely to add value to your model.

Date-time based features are a good start, but it is often a lot more useful to include the values at previous time steps. These are called lagged values and we will look at adding these features in the next section.

## Lag Features

Lag features are the classical way that time series forecasting problems are transformed into supervised learning problems.

The simplest approach is to predict the value at the next time (t+1) given the value at the previous time (t-1). The supervised learning problem with shifted values looks as follows:

1 2 3 |
Value(t-1), Value(t+1) Value(t-1), Value(t+1) Value(t-1), Value(t+1) |

The Pandas library provides the shift() function to help create these shifted or lag features from a time series dataset. Shifting the dataset by 1 creates the t-1 column, adding a NaN (unknown) value for the first row. The time series dataset without a shift represents the t+1.

Let’s make this concrete with an example. The first 3 values of the temperature dataset are 20.7, 17.9, and 18.8. The shifted and unshifted lists of temperatures for the first 3 observations are therefore:

1 2 3 4 |
Shifted, Original NaN, 20.7 20.7, 17.9 17.9, 18.8 |

We can concatenate the shifted columns together into a new DataFrame using the concat() function along the column axis (*axis=1*).

Putting this all together, below is an example of creating a lag feature for our daily temperature dataset. The values are extracted from the loaded series and a shifted and unshifted list of these values is created. Each column is also named in the *DataFrame* for clarity.

1 2 3 4 5 6 7 8 |
from pandas import Series from pandas import DataFrame from pandas import concat series = Series.from_csv('daily-minimum-temperatures-in-me.csv', header=0) temps = DataFrame(series.values) dataframe = concat([temps.shift(1), temps], axis=1) dataframe.columns = ['t-1', 't+1'] print(dataframe.head(5)) |

Running the example prints the first 5 rows of the new dataset with the lagged feature.

1 2 3 4 5 6 |
t-1 t+1 0 NaN 20.7 1 20.7 17.9 2 17.9 18.8 3 18.8 14.6 4 14.6 15.8 |

You can see that we would have to discard the first row to use the dataset to train a supervised learning model, as it does not contain enough data to work with.

The addition of lag features is called the sliding window method, in this case with a window width of 1. It is as though we are sliding our focus along the time series for each observation with an interest in only what is within the window width.

We can expand the window width and include more lagged features. For example, below is the above case modified to include the last 3 observed values to predict the value at the next time step.

1 2 3 4 5 6 7 8 |
from pandas import Series from pandas import DataFrame from pandas import concat series = Series.from_csv('daily-minimum-temperatures-in-me.csv', header=0) temps = DataFrame(series.values) dataframe = concat([temps.shift(3), temps.shift(2), temps.shift(1), temps], axis=1) dataframe.columns = ['t-3', 't-2', 't-1', 't+1'] print(dataframe.head(5)) |

Running this example prints the first 5 rows of the new lagged dataset.

1 2 3 4 5 6 |
t-3 t-2 t-1 t+1 0 NaN NaN NaN 20.7 1 NaN NaN 20.7 17.9 2 NaN 20.7 17.9 18.8 3 20.7 17.9 18.8 14.6 4 17.9 18.8 14.6 15.8 |

Again, you can see that we must discard the first few rows that do not have enough data to train a supervised model.

A difficulty with the sliding window approach is how large to make the window for your problem.

Perhaps a good starting point is to perform a sensitivity analysis and try a suite of different window widths to in turn create a suite of different “views” of your dataset and see which results in better performing models. There will be a point of diminishing returns.

Additionally, why stop with a linear window? Perhaps you need a lag value from last week, last month, and last year. Again, this comes down to the specific domain.

In the case of the temperature dataset, a lag value from the same day in the previous year or previous few years may be useful.

We can do more with a window than include the raw values. In the next section, we’ll look at including features that summarize statistics across the window.

## Rolling Window Statistics

A step beyond adding raw lagged values is to add a summary of the values at previous time steps.

We can calculate summary statistics across the values in the sliding window and include these as features in our dataset. Perhaps the most useful is the mean of the previous few values, also called the rolling mean.

For example, we can calculate the mean of the previous two values and use that to predict the next value. For the temperature data, we would have to wait 3 time steps before we had 2 values to take the average of before we could use that value to predict a 3rd value.

For example:

1 2 3 |
mean(t-2, t-1), t+1 mean(20.7, 17.9), 18.8 19.3, 18.8 |

Pandas provides a rolling() function that creates a new data structure with the window of values at each time step. We can then perform statistical functions on the window of values collected for each time step, such as calculating the mean.

First, the series must be shifted. Then the rolling dataset can be created and the mean values calculated on each window of two values.

Here are the values in the first three rolling windows:

1 2 3 4 |
#, Window Values 1, NaN 2, NaN, 20.7 3, 20.7, 17.9 |

This suggests that we will not have usable data until the 3rd row.

Finally, as in the previous section, we can use the *concat()* function to construct a new dataset with just our new columns.

The example below demonstrates how to do this with Pandas with a window size of 2.

1 2 3 4 5 6 7 8 9 10 11 |
from pandas import Series from pandas import DataFrame from pandas import concat series = Series.from_csv('daily-minimum-temperatures-in-me.csv', header=0) temps = DataFrame(series.values) shifted = temps.shift(1) window = shifted.rolling(window=2) means = window.mean() dataframe = concat([means, temps], axis=1) dataframe.columns = ['mean(t-2,t-1)', 't+1'] print(dataframe.head(5)) |

Running the example prints the first 5 rows of the new dataset. We can see that the first two rows are not useful.

- The first NaN was created by the shift of the series.
- The second because NaN cannot be used to calculate a mean value.
- Finally, the third row shows the expected value of 19.30 (the mean of 20.7 and 17.9) used to predict the 3rd value in the series of 18.8.

1 2 3 4 5 6 |
mean(t-2,t-1) t+1 0 NaN 20.7 1 NaN 17.9 2 19.30 18.8 3 18.35 14.6 4 16.70 15.8 |

There are more statistics we can calculate and even different mathematical ways of calculating the definition of the “window.”

Below is another example that shows a window width of 3 and a dataset comprised of more summary statistics, specifically the minimum, mean, and maximum value in the window.

You can see in the code that we are explicitly specifying the sliding window width as a named variable. This lets us use it both in calculating the correct shift of the series and in specifying the width of the window to the *rolling()* function.

In this case, the window width of 3 means we must shift the series forward by 2 time steps. This makes the first two rows NaN. Next, we need to calculate the window statistics with 3 values per window. It takes 3 rows before we even have enough data from the series in the window to start calculating statistics. The values in the first 5 windows are as follows:

1 2 3 4 5 6 |
#, Window Values 1, NaN 2, NaN, NaN 3, NaN, NaN, 20.7 4, NaN, 20.7, 17.9 5, 20.7, 17.9, 18.8 |

This suggests that we would not expect usable data until at least the 5th row (array index 4)

1 2 3 4 5 6 7 8 9 10 11 |
from pandas import Series from pandas import DataFrame from pandas import concat series = Series.from_csv('daily-minimum-temperatures-in-me.csv', header=0) temps = DataFrame(series.values) width = 3 shifted = temps.shift(width - 1) window = shifted.rolling(window=width) dataframe = concat([window.min(), window.mean(), window.max(), temps], axis=1) dataframe.columns = ['min', 'mean', 'max', 't+1'] print(dataframe.head(5)) |

Running the code prints the first 5 rows of the new dataset.

We can spot check the correctness of the values on the 5th row (array index 4). We can see that indeed 17.9 is the minimum and 20.7 is the maximum of values in the window of [20.7, 17.9, 18.8].

1 2 3 4 5 6 |
min mean max t+1 0 NaN NaN NaN 20.7 1 NaN NaN NaN 17.9 2 NaN NaN NaN 18.8 3 NaN NaN NaN 14.6 4 17.9 19.133333 20.7 15.8 |

## Expanding Window Statistics

Another type of window that may be useful includes all previous data in the series.

This is called an expanding window and can help with keeping track of the bounds of observable data. Like the *rolling()* function on *DataFrame*, Pandas provides an expanding() function that collects sets of all prior values for each time step.

These lists of prior numbers can be summarized and included as new features. For example, below are the lists of numbers in the expanding window for the first 5 time steps of the series:

1 2 3 4 5 6 |
#, Window Values 1, 20.7 2, 20.7, 17.9, 3, 20.7, 17.9, 18.8 4, 20.7, 17.9, 18.8, 14.6 5, 20.7, 17.9, 18.8, 14.6, 15.8 |

Again, you can see that we must shift the series one-time step to ensure that the output value we wish to predict is excluded from these window values. Therefore the input windows look as follows:

1 2 3 4 5 6 |
#, Window Values 1, NaN 2, NaN, 20.7 3, NaN, 20.7, 17.9, 4, NaN, 20.7, 17.9, 18.8 5, NaN, 20.7, 17.9, 18.8, 14.6 |

Thankfully, the statistical calculations exclude the NaN values in the expanding window, meaning no further modification is required.

Below is an example of calculating the minimum, mean, and maximum values of the expanding window on the daily temperature dataset.

1 2 3 4 5 6 7 8 9 10 |
# create expanding window features from pandas import Series from pandas import DataFrame from pandas import concat series = Series.from_csv('daily-minimum-temperatures.csv', header=0) temps = DataFrame(series.values) window = temps.expanding() dataframe = concat([window.min(), window.mean(), window.max(), temps.shift(-1)], axis=1) dataframe.columns = ['min', 'mean', 'max', 't+1'] print(dataframe.head(5)) |

Running the example prints the first 5 rows of the dataset.

Spot checking the expanding minimum, mean, and maximum values shows the example having the intended effect.

1 2 3 4 5 6 |
min mean max t+1 0 20.7 20.700000 20.7 17.9 1 17.9 19.300000 20.7 18.8 2 17.9 19.133333 20.7 14.6 3 14.6 18.000000 20.7 15.8 4 14.6 17.560000 20.7 15.8 |

## Summary

In this tutorial, you discovered how to use feature engineering to transform a time series dataset into a supervised learning dataset for machine learning.

Specifically, you learned:

- The importance and goals of feature engineering time series data.
- How to develop date-time and lag-based features.
- How to develop sliding and expanding window summary statistic features.

**Do you know of more feature engineering methods for time series?
**Let me know in the comments below.

**Do you have any questions?
**Ask your questions in the comments below and I will do my best to answer.

Awesome introduction to Feature Engineering with Time Series. I thought that it was something only for scientific minds but actually it is so easy. I enjoyed the easy way that you have explained it. I am ready now for any challenge using Time Series in my domain of expertise. Thanks!!!!

I’m glad to hear it Dario.

min mean max t+1

0 NaN NaN NaN 20.7

1 NaN NaN NaN 17.9

2 NaN NaN NaN 18.8

3 NaN NaN NaN 14.6

4 17.9 19.133333 20.7 15.8

why you shift 2 step instead of 1? to make it

min mean max t+1

0 NaN NaN NaN 20.7

1 NaN NaN NaN 17.9

2 NaN NaN NaN 18.8

3 17.9 19.133333 20.7 14.6

great post btw… thanks !!

Hi Fai, the shifting can get confusing.

I try to explain it above by showing how the groups of observations in the window look each row as we build them up. I guess it didn’t work as an explanatory tool:

Thx for the reply. Thats a good explanatory tool. But i was wondering

if shift by 1 step, we could have the following window observations,

and we can have enough data in row 4 for calculating statistic with 3 values per window.

1, NaN

2, NaN, 20.7

3, NaN, 20.7, 17.9

4, 20.7, 17.9, 18.8

so why should we shift by 2 step instead (we can only start calculating at row5)

1, NaN

2, NaN, NaN

3, NaN, NaN, 20.7

4, NaN, 20.7, 17.9

5, 20.7, 17.9, 18.8

If you’re asking why a window size of “3” or a window size of “2” or even “1”, no reason. Just for demonstration purposes.

We can use autocorrelation to find good window sizes (ACF (AutoCorrelation Function) and correlograms) and I will get into this future posts.

Does this help at all? Or am I still misunderstanding?

Hi Jason,

Actually, I have the same concern. There are two concepts we are applying here.

1. Rolling Window. The size is 3, and that’s not the concern.

2. Lagging. The shift is 2, and the question is: why don’t we use time steps 1, 2, and 3 to create features for time step 4? Why did we use time steps 1, 2, and 3 to create features in time step 5? By features I mean (min, max, & mean of 3 leading measurements)

Instead of: shifted = temps.shift(width – 1)

why don’t we use: shifted = temps.shift(1)

Hi Jason, nice introduction. If you are interested in more statistical features tale a look at

http://rsif.royalsocietypublishing.org/content/10/83/20130048.full and the associted github repo.

I find it the most complete work for ts features engineering

Thanks for sharing Hicham.

Hi,

In rolling mean why do you take lag of raw variables.why do you not take raw variables instead?

In this case, the data is univariate, there are no other variables beyond those that are “invented” like lag vars.

Hi Jason:

When you write “predict the daily maximum temperature” did you mean “predict the daily minimum temperature” ?

I did Daniel, thanks. Fixed.

I got error when reading the data. Where exactly did you spot the questionmarks?

You can use a text editor and the find-replace feature.

how to create lagged variable for test data? we do not know any info in test data, right?

The lag for the test data will be the end of the train data.

You can prepare the data using the function in this post:

http://machinelearningmastery.com/convert-time-series-supervised-learning-problem-python/

It would be greate if you added some comments about forecasting time series with machine learning techniques like Nearest Neighbor or SVR methods.

Great suggestion, I hope to cover it in the future.

Hello !

There is a task for the Time Series Forecasting with the LSTM.

Multiple input variables:

1 year

2. Month

3. Day of the week

4. Date of the month

5. Hour of the day

…

Hour of day.

Business hours or not.

Weekend or not.

Imprint – the number of sales

It is required to help – how to implement the algorithm in Python with Keras.

Thank you !

I would recommend using any of my many posts on LSTMs for time series as a starting point and adapt it to your problem.

I love this sentence: complexity exists in the relationships between input and output data!

Because the output data may or may not be a straightforward outcome of the input data, what we are trying to do is to establish a predictable relationship between A and B. By feature engineering, we are hoping to reduce the complexity and difficulty of the establishment with intermediate features that may be more informative and closer to B.

We always hope to build a closer bridge to save time and energy.

Thank you so much for that punctual summary and great explanation! It is very inspiring and helpful.

By the way, what do you mean here “In the case of time series, there is no concept of input and output variables”? Aren’t we always provided a time series that comes with its time step (input) and its value (Y)? Do you mean concept of input and output variables for unknown new features we are going to invent?

Thanks.

Thanks!

I guess I meant that we have really only have lags obs in univariate time series.

Hi, Jason! This is a great piece!

I have a question on how to sample this kind of data set.

My understanding is that since the features are generated from a time series, there is time dependency among rows and across the variables. So it may require different treatment than the normal data sets which we assume the rows are independent between one another.

Could you please help to advise on the sampling method applied on this kind of data set?

Thank you very much!

Yes, you can use walk-forward validation to evaluate models on sequence prediction problems.

See this post:

https://machinelearningmastery.com/backtest-machine-learning-models-time-series-forecasting/

Hi, Jason.

What do you think about combining Date Time Features with Lag Features? Is this approach justified?

Thanks a lot.

Perhaps.

I would recommend doing the experiment and using the results/data to decide whether it is a good idea for your specific forecasting problem.

Hello,

I have a time series data(csv), which is multimodal(obtained from different sensors).My task is to implement multiclass classification, by extracting features from this data.Can you give me a starting point? any posts for classification

Good question.

I don’t have post on time series classification. I hope to have some in the future.