It can be difficult to understand how to prepare your sequence data for input to an LSTM model.

Often there is confusion around how to define the input layer for the LSTM model.

There is also confusion about how to convert your sequence data that may be a 1D or 2D matrix of numbers to the required 3D format of the LSTM input layer.

In this tutorial, you will discover how to define the input layer to LSTM models and how to reshape your loaded input data for LSTM models.

After completing this tutorial, you will know:

- How to define an LSTM input layer.
- How to reshape a one-dimensional sequence data for an LSTM model and define the input layer.
- How to reshape multiple parallel series data for an LSTM model and define the input layer.

Let’s get started.

## Tutorial Overview

This tutorial is divided into 4 parts; they are:

- LSTM Input Layer
- Example of LSTM with Single Input Sample
- Example of LSTM with Multiple Input Features
- Tips for LSTM Input

### LSTM Input Layer

The LSTM input layer is specified by the “*input_shape*” argument on the first hidden layer of the network.

This can make things confusing for beginners.

For example, below is an example of a network with one hidden LSTM layer and one Dense output layer.

1 2 3 |
model = Sequential() model.add(LSTM(32)) model.add(Dense(1)) |

In this example, the LSTM() layer must specify the shape of the input.

The input to every LSTM layer must be three-dimensional.

The three dimensions of this input are:

**Samples**. One sequence is one sample. A batch is comprised of one or more samples.**Time Steps**. One time step is one point of observation in the sample.**Features**. One feature is one observation at a time step.

This means that the input layer expects a 3D array of data when fitting the model and when making predictions, even if specific dimensions of the array contain a single value, e.g. one sample or one feature.

When defining the input layer of your LSTM network, the network assumes you have 1 or more samples and requires that you specify the number of time steps and the number of features. You can do this by specifying a tuple to the “*input_shape*” argument.

For example, the model below defines an input layer that expects 1 or more samples, 50 time steps, and 2 features.

1 2 3 |
model = Sequential() model.add(LSTM(32, input_shape=(50, 2))) model.add(Dense(1)) |

Now that we know how to define an LSTM input layer and the expectations of 3D inputs, let’s look at some examples of how we can prepare our data for the LSTM.

## Example of LSTM With Single Input Sample

Consider the case where you have one sequence of multiple time steps and one feature.

For example, this could be a sequence of 10 values:

1 |
0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0 |

We can define this sequence of numbers as a NumPy array.

1 2 |
from numpy import array data = array([0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]) |

We can then use the *reshape()* function on the NumPy array to reshape this one-dimensional array into a three-dimensional array with 1 sample, 10 time steps, and 1 feature at each time step.

The *reshape()* function when called on an array takes one argument which is a tuple defining the new shape of the array. We cannot pass in any tuple of numbers; the reshape must evenly reorganize the data in the array.

1 |
data = data.reshape((1, 10, 1)) |

Once reshaped, we can print the new shape of the array.

1 |
print(data.shape) |

Putting all of this together, the complete example is listed below.

1 2 3 4 |
from numpy import array data = array([0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]) data = data.reshape((1, 10, 1)) print(data.shape) |

Running the example prints the new 3D shape of the single sample.

1 |
(1, 10, 1) |

This data is now ready to be used as input (*X*) to the LSTM with an input_shape of (10, 1).

1 2 3 |
model = Sequential() model.add(LSTM(32, input_shape=(10, 1))) model.add(Dense(1)) |

## Example of LSTM with Multiple Input Features

Consider the case where you have multiple parallel series as input for your model.

For example, this could be two parallel series of 10 values:

1 2 |
series 1: 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0 series 2: 1.0, 0.9, 0.8, 0.7, 0.6, 0.5, 0.4, 0.3, 0.2, 0.1 |

We can define these data as a matrix of 2 columns with 10 rows:

1 2 3 4 5 6 7 8 9 10 11 12 |
from numpy import array data = array([ [0.1, 1.0], [0.2, 0.9], [0.3, 0.8], [0.4, 0.7], [0.5, 0.6], [0.6, 0.5], [0.7, 0.4], [0.8, 0.3], [0.9, 0.2], [1.0, 0.1]]) |

This data can be framed as 1 sample with 10 time steps and 2 features.

It can be reshaped as a 3D array as follows:

1 |
data = data.reshape(1, 10, 2) |

Putting all of this together, the complete example is listed below.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
from numpy import array data = array([ [0.1, 1.0], [0.2, 0.9], [0.3, 0.8], [0.4, 0.7], [0.5, 0.6], [0.6, 0.5], [0.7, 0.4], [0.8, 0.3], [0.9, 0.2], [1.0, 0.1]]) data = data.reshape(1, 10, 2) print(data.shape) |

Running the example prints the new 3D shape of the single sample.

1 |
(1, 10, 2) |

This data is now ready to be used as input (*X*) to the LSTM with an input_shape of (10, 2).

1 2 3 |
model = Sequential() model.add(LSTM(32, input_shape=(10, 2))) model.add(Dense(1)) |

## Longer Worked Example

For a complete end-to-end worked example of preparing data, see this post:

## Tips for LSTM Input

This section lists some tips to help you when preparing your input data for LSTMs.

- The LSTM input layer must be 3D.
- The meaning of the 3 input dimensions are: samples, time steps, and features.
- The LSTM input layer is defined by the
*input_shape*argument on the first hidden layer. - The
*input_shape*argument takes a tuple of two values that define the number of time steps and features. - The number of samples is assumed to be 1 or more.
- The
*reshape()*function on NumPy arrays can be used to reshape your 1D or 2D data to be 3D. - The
*reshape()*function takes a tuple as an argument that defines the new shape.

## Further Reading

This section provides more resources on the topic if you are looking go deeper.

- Recurrent Layers Keras API
- Numpy reshape() function API
- How to Convert a Time Series to a Supervised Learning Problem in Python
- Time Series Forecasting as Supervised Learning

## Summary

In this tutorial, you discovered how to define the input layer for LSTMs and how to reshape your sequence data for input to LSTMs.

Specifically, you learned:

- How to define an LSTM input layer.
- How to reshape a one-dimensional sequence data for an LSTM model and define the input layer.
- How to reshape multiple parallel series data for an LSTM model and define the input layer.

Do you have any questions?

Ask your questions in the comments below and I will do my best to answer.

Great explanation of the dimensions! Just wanted to say this explanation also works for LSTM models in Tensorflow as well.

Thanks Steven.

Hi Jason,

Thanks a lot for your explanations .

I have a confusion below:

Assuming that we have multiple parallel series as input for out model.The first step is to define these data as a matrix of M columns with N rows.To be 3D(samples, time steps, and features),is this means that,samples :1 sample ,time steps: row numbers of the matrix ,and features: column numbers of the matrix ? Must it be like this?Looking forward to your reply.Thank you

Sorry, I’m not sure I follow your question.

If you have parallel time series, then each series would need the same number of time steps and be represented as a separate feature (e.g. observation at a time).

Does that help?

Agree with Yuan on this issue

Hello Sir,

I have used your multivariates code for testing lstm model.Its working fine but I did not understand why lstm one row is hidden from train as well as test dataset. If I test two rows in test then got one prediction as a output. And datetime output this one prediction result?

Please help me.

Thanks in advance.

Azad

Sorry, I don’t follow.

Perhaps you can elaborate on your question?

hi jason,your artical is very inspire.

Now i am working with loan risk control. i have client repayment time difference for each period. i am working on use time series modle on time difference before to predite the client will overdue.

suppose i have 10000 client and for each client i have 6 period repayment time difference. as the artical you write my input data orginal is (10000*6).before i send it to LSTM model i should reshape it to 10000*6*1 3D dataset? is that the right way to do it? if you have any other experience of this problem you can tell me as well.

Sounds like it would be [10000, 6, 1]

HI, I think wanghy wants to make a loan risk forecast.

The customer should pay 6 repayments for each loan.

So if you use [10000,6,1] to train the model,

how do you feed data when making prediction after 3 replayments have done

Perhaps you can make 1 step forecasts based on zero or more input time steps and use zero padding to fill out the inputs?

e.g.

thanks jason

I am still confused about the timesteps.

Suppose the customer pays 240 repayments for each loan (20 years) and the loan risk is related to the latest year’s repayments.

So, the train data should be [rows, 12, 1] instead of [rows, 240, 1], right?

Sometimes, the values of timesteps and features are very large, such as 1024, 512.

How to choose the value of the timesteps when the memory is not enough?

This explains it well:

https://machinelearningmastery.com/faq/single-faq/what-is-the-difference-between-samples-timesteps-and-features-for-lstm-input

Trial and error is a good approach, test different framings of the problem and see what works best.

thanks very much

Hi Jason,

thanks a lot for all the explanations you gave!

I tried to understand the effect of the reshape parameters and the effect in the spyder/variable explorer. But I do not understand the result shown in the data window.

I used the code from a different tutorial:

data = array([

[0.1, 1.0],

[0.2, 0.9],

[0.3, 0.8],

[0.4, 0.7],

[0.5, 0.6],

[0.6, 0.5],

[0.7, 0.4],

[0.8, 0.3],

[0.9, 0.2],

[1.0, 0.1]])

data_re = data.reshape(1, 10, 2)

When checking the result in the variable explorer of spyder I see 3 dimensions of the array but can not connect it to the paramters sample, timestep, feature.

On axis 0 of data_re I see the complete dataset

On axis 1 of the data_re I get 0.1 and 1.0 in column 1

On axis 2 of the data_re I see the column 1 of axis 0 transposed to row 1

Would you give me a hint how to interpret it?

Regards,

Oliver.

There are no named parameters, I am referring to the dimensions by those names because that is how the LSTM model uses the data.

Sorry for the confusion.

Hi Jason,

Thanks so much for the article (and the whole series in fact!). The documentation in Keras is not very clear on many things on its own.

I have been trying to implement a model that receives multiple samples of multivariate timeseries as input. The twist is that the length of the series, i.e. the “time steps” dimension is different for different samples. I have tried to train a model on each sample individually and then merge, (but then each LSTM is going to be extremely prone to overfitting). Another idea was to scale the samples to have the same time steps but this comes with a scaling factor of time steps for each sample which is not ideal either.

Is there a way to provide the LSTM with samples of dynamic time steps? maybe using a lower-level API?

Regards,

Saga

A way I use often is to pad all sequences to the same length and use a masking layer on the front end to ignore masked time steps.

HI Jason,

Thank you for this amazing article.

I have the same problem here. which is the samples have many different lengths.

I did not get the idea you said.

“A way I use often is to pad all sequences to the same length and use a masking layer on the front end to ignore masked time steps.”

can you please provide more details about that?

or maybe provide articles explain how to solve this problem.

Thank you in advance.

These examples will show you how to truncate and pad sequence data:

https://machinelearningmastery.com/data-preparation-variable-length-input-sequences-sequence-prediction/

Hi Jason,

Thanks very much for your tutorials on LSTM. I am trying to predict one time series from 10 different parallel time series. All of them are different 1D series. So, the shape of my X_train is (50000,10) and Y_train is (50000,1). I couldn’t figure out how to reshape my dataset and the input shape of LSTM if I want to use let’s say 100 time steps or look back as 100.

Thanks.

This post will help you formulates your series as a supervised learning problem:

https://machinelearningmastery.com/convert-time-series-supervised-learning-problem-python/

Respected Sir

I want to use LSTM RNN GRU to check changes in facial expression of the person who is watching a movie. Want to check his mental state whether he is a boar or interested to continue this movie or at what time he is a boar. Can you please help me how can I start to work on same.

That sounds like a great problem. I would recommend starting by collecting a ton of training data.

Then think of using a CNN on the front end of your LSTM.

Hi,

I have around 12,000 tweets for sentiment classification totally. Do you think 16GB CPU RAM will be enough?

Sure.

Hi Jason

Thanks for the simple explanation.

However, I have a doubt. What if you don’t know the no of time steps? How do you proceed then?

Is that why we use the embedding layer?

I intend to use it for sentiment analysis of imDb movie review dataset.

You can force all input sequences to be the same length by padding/truncating.

You could also use a model that does not specify the input length, for example:

https://machinelearningmastery.com/develop-encoder-decoder-model-sequence-sequence-prediction-keras/

Hi Jason,

I finally understood the input shape requirements.

Just a quick question: batch_size would be a certain number of samples inside a group e.g if we have 100 samples we can divide it into batches of 10. Batching helps with a faster training time right?

Correct, and weight updates occur and state is reset at the end of each batch.

Hi Jason,

About sample (the first argument in reshape): if I have two sequences with different number of values (let’s suppose one with 10 values and another with 8) and want them to be considered as two distinct samples (not 2 features), a zero-padding is necessary?

series 1: 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0

series 2: 1.0, 0.9, 0.8, 0.7, 0.6, 0.5, 0.4, 0.3, 0.0, 0.0

If I do:

data = data.reshape(2, 10, 1)

It is going to understand them as 2 different samples?

Yes, padding to 10 time steps.

Yes, your reshape looks good.

Explore pre and post padding to see if it makes a difference for your model.

With this input, the model is going to understand two different series?

Why to don’t use (1, 10,2) shape?

You could treat them as two features as you suggestion, I thought they were separate samples.

Hi Jason,

Thanks a lot for the tutorial!

I am trying to understand the input shape for LSTM data (No. of timesteps & no. of features). Could I ask what each will be in the context of the iris dataset, please?

Am I correct to say that in the iris dataset, the timesteps can be 2, 3, 5, 6 – as long as it neatly divides the dataset into equal number of rows (iris has 150 rows).

And the number of features will be the number of columns (apart from the target column/class)?

Thanks ever so much!

The iris dataset is not a sequence classification problem. It does not have time steps, only samples and features.

Hello!

First of all, thank you very much for your posts, I have learned a lot.

My question is because I’m not sure how to focus the next type of problems: multiples sequences of multiple features.

For example, predict the amount that a user could spend given the previous purchases (here I can consider different features such as the previous amounts, products, day of week, etc.). If I have a dataset with data of 1000 users and I want to predict the amount for each user, how should be addressed?

Can I use a lstm for all users or each user will have a model/lstm?

I understand that a lstm for all users could see things more interesting.. But I don’t know how to organize the input of different users.. because the example of two sequences (1,10,2) I don’t know how to apply.. I want to include more features for each sequence..

I’m very lost..

Thank you in advance

Perhaps start off by modeling individual users?

Thanks!

By modelling individual users do you mean a lstm per user?

I have users with 200 purchases but others only with only 10.. would be enough?

I will try!

Thanks!

Or a user per sample.

Hi Jason,

Thanks for your tutorial and for your book!

I am not sure how to design the input shape of the following table or dataframe:

date, product, store, hasPromotion, attrib1, attrib2, quantity (t)

The first three columns are the key. We have 50000 products in 20 stores and I would like to predict the quantity (per product per store) at least 14 days ahead with LSTM.

What is the good start for the 3D input?

I am wondering if creating new features from date (as there are repetition), like day of week, day of month, month of year, etc. + the existing features + quantity (t), quantity (t+1) would do…

Thank you for your help in advance!

Br,

Drop date and you have 6 features, does that help?

OK, thanks. If there are seasonality and trend in sales, should I remove them before train the LSTM, too?

Yes, I would recommend that.

Howdy!

Thank you so much for the great amount of tutorials on LSTMs

Im trying to build an LSTM in keras using your examples and keep running into shape issues.

I have time series data set with prices for different things, and am trying to predict the price of item4 for time t+1

Item4 is a lagged value so that you can use previous set of prices to predict the next.

The data set has 400 sequential observations.

variables: datetime price1 price2 price3 item4_price

since the data variable has uniform interval of observations and none are missing, i am dropping the datetime variable.

So now i have 4 variables and 400 observations.

trainX = train[:, 0:-1] #use first 3 variables

trainY = train[:,-1] #use the last variable

so now the trainX data set has price1 price2 and price3 variables (its my undestanding that this means there are 3 “feautres” in keras)

trainY is the predictor data set and only cointains item4_price

trainX = numpy.reshape(trainX, (1, 400, 3)) #reshape, this means there is 1 sample, 400 timestamps, and 3 features

model = Sequential()

model.add(LSTM(5, input_shape=(1, 400, 3), return_sequences=True))

model.add(Dense(1))

model.compile(loss=’mean_squared_error’, optimizer=’adam’)

model.fit(trainX, trainY, epochs=100, batch_size=1, verbose=2)

Keep getting various shape errors all the time, no matter what i do. I tried switching it around, and even ommiting the first dimension.

I was wondering if you could point me in the right dirrection of what it is that i keep missing in my understnading of keras/lstm shapes.

I also dont know if the trainY set needs shaping? I tried to shape it too but python was also not happy with that.

Let me know what you think!

Thanks,

Vic

Perhaps start with one series and really nail down what is required.

Did you try this tutorial:

https://machinelearningmastery.com/prepare-univariate-time-series-data-long-short-term-memory-networks/

Hi Dr. Brownlee,

I have previously read that tutorial and feel as though i understand it fine.

But when applying what I learned to the problem in a way as described previously, find that Im running into some trouble.

So i was hoping I was just overlooking something, but at this point im not really sure what. Is what Im doing seem reasonable?

Thanks!

Perhaps, but I don’t know your problem as well as you and there is no set way to solve any ml problem.

I would encourage you to brainstorm and try a suite of approaches to see what works best.

Hi Jason,

I have gone through this tutorial but i have a input size of 1762 X 4 and output size 1762X 1.

I did as follows but the shape of y train is giving as (1394, 4) , which should be 1394,1

Can you help me on this?

Sorry, I cannot debug your code for you. I simply do not have the capacity, I’m sure you can understand.

Perhaps post your error to stackoverflow or cross validated?

I got an exception “ValueError: Input arrays should have the same number of samples as target arrays. Found 1 input samples and 21 target samples”.

=>print X_train

[[ 0.15699646 -1.59383227]

[-0.31399291 -0.03680409]

[ 0.15699646 -1.59383227]

[-0.31399291 0.78456757]

[ 0.15699646 -1.59383227]

[ 4.39590078 -1.59383227]

[-0.31399291 1.38764971]

[-0.31399291 -0.03680409]

[-0.31399291 -0.32252408]

[-0.31399291 0.6081381 ]

[-0.31399291 -0.32252408]

[-0.31399291 1.38764971]

[-0.31399291 0.78456757]

[-0.31399291 -0.03680409]

[-0.31399291 0.78456757]

[ 0.15699646 1.24889926]

[-0.31399291 -0.32252408]

[-0.31399291 1.24889926]

[-0.31399291 -0.69488163]

[-0.31399291 -0.69488163]

[-0.31399291 0.6081381 ]]

=>print y_train

0 1

1 1

2 1

3 1

4 1

5 1

6 1

7 0

8 0

9 0

10 0

11 0

12 0

13 0

14 1

15 1

16 1

17 1

18 0

19 0

20 0

Name: out, dtype: int64

=>print(y_train.shape)

(21,)

=>print X_train.shape

(21, 2)

=>print X_test.shape

(8, 2)

I have reshaped the inputs to 3dimensional input. I have followed you steps.

=>X_train = X_train.reshape(1,21, 2)

print(X_train.shape)

(1, 21, 2)

=>

model = Sequential()

model.add(LSTM(32, input_shape=(21, 2)))

model.add(Dense(1))

model.compile(optimizer=’rmsprop’,loss=’categorical_crossentropy’,metrics=[‘categorical_accuracy’])

history = model.fit(X_train,y_train,batch_size =13, epochs = 14)

—————————————————————————

ValueError Traceback (most recent call last)

in ()

—-> 1 history = model.fit(X_train,y_train,batch_size =13, epochs = 14)

/home/siji/anaconda2/lib/python2.7/site-packages/keras/models.pyc in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, **kwargs)

891 class_weight=class_weight,

892 sample_weight=sample_weight,

–> 893 initial_epoch=initial_epoch)

894

895 def evaluate(self, x, y, batch_size=32, verbose=1,

/home/siji/anaconda2/lib/python2.7/site-packages/keras/engine/training.pyc in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, **kwargs)

1553 class_weight=class_weight,

1554 check_batch_axis=False,

-> 1555 batch_size=batch_size)

1556 # Prepare validation data.

1557 do_validation = False

/home/siji/anaconda2/lib/python2.7/site-packages/keras/engine/training.pyc in _standardize_user_data(self, x, y, sample_weight, class_weight, check_batch_axis, batch_size)

1419 for (ref, sw, cw, mode)

1420 in zip(y, sample_weights, class_weights, self._feed_sample_weight_modes)]

-> 1421 _check_array_lengths(x, y, sample_weights)

1422 _check_loss_and_target_compatibility(y,

1423 self._feed_loss_fns,

/home/siji/anaconda2/lib/python2.7/site-packages/keras/engine/training.pyc in _check_array_lengths(inputs, targets, weights)

249 ‘the same number of samples as target arrays. ‘

250 ‘Found ‘ + str(list(set_x)[0]) + ‘ input samples ‘

–> 251 ‘and ‘ + str(list(set_y)[0]) + ‘ target samples.’)

252 if len(set_w) > 1:

253 raise ValueError(‘All sample_weight arrays should have ‘

ValueError: Input arrays should have the same number of samples as target arrays. Found 1 input samples and 21 target samples.

Please solve my problem. I am new in this area. What is the mistake

Perhaps cut the example back to a few lines to help expose the fault?

Hi Jason,

I’m trying to understand the input_shape but I think I’m totally confused about the time step variable. I have a multivariate time series with 18,000 samples and 720 features. I created a 10 lagged observation dataset to forecast the next 5 time steps so my dataset goes from t-10 to t+5, being the feature dataset from t-10 to t and the label dataset from t+1 to t+5.

Assuming that I take 15,000 samples for training, what will be the values of the reshape function? I think it should be [15000, 1, 7200 (720 features * 10)]. Regarding the time step, is the value “1” correct or it should be the number of lagged observations, that is, 10?

Thank you in advance.

Generally, I would recommend about 200-400 time steps.

Here’s some more advice on how to handle a very long time series:

https://machinelearningmastery.com/handle-long-sequences-long-short-term-memory-recurrent-neural-networks/

And here:

https://machinelearningmastery.com/prepare-univariate-time-series-data-long-short-term-memory-networks/

Hi Jason!

Thank you so much for all the tutorials on LSTMs, I’ve learned a lot.

I’m trying to implement the LSTM Architecture from the paper “Dropout improves Recurrent Neural Networks for Handwriting Recognition” for resolving the handwritten recognition problem.

Basically I have to train the network giving in input variable-sized images (different W and H but always 3 channels) and to predict what is the word written in the image. What I can’t understand is how to deal with variable sized images? Can I consider images as some sequences (for ex. a 50×30 image considered as 50 sequences with 30 features?). The authors say I give in input a block of image of size 2×2 scaning in 4 different directions (multidirectional multidimensional LSTM).

What do I have to specify here : input_size(Samples,Time Steps,Features) ? The Samples refers to the number of all images I have in training set or the number of miniblocks 2×2 ? What about time steps and features? I don’t get it and its very confusing. Can you please help with any idea? I am new in this area and Im stacked in this problem.

Thanks a lot 🙂

I would recommend padding the inputs to a fixed size.

Hi Jason!

Thanks so much for your tutorials on LSTM!

I’m trying to predict trajectory with LSTM and ARIMA now. After reading this tutorial, I’ve got some questions.

(1) Do we must transfer time series to lag observations if we want to do forecast work with LSTM?

(2) After transfering time series to supervised learning problems, the forecast is only related to “order” or “lag” rather than “time”(like ARIMA do)? Why the input is not time/date? And the time interval of data must be even?

Thanks a lot in advance!

No, LSTMs can work with the time steps directly.

The order of the observations is sufficient for the model, if the time steps are consistently spaced it does not need the absolute date/time information.

Hello Jason! Congratulations on the LSTM input tutorial!

Could you please answer three questions?

I’m working with 500 samples that have varying sizes. My doubts are related to the organization of these 500 samples within this 3-dimensional input, mainly in relation to the Samples dimension.

The dimension “Features” has already defined that it will have size 26, the dimension “Time Steps” will have to have size 100 but the dimension “Samples” is that I still do not know what its value will be.

Doubt 1: In these cases of samples with different sizes to know the dimension “Samples” I have to be based on the larger sample and for the other samples I fill in the value 0 (zero) in the additional spaces?

Doubt 2: Can I have more than one line in the “Samples” dimension representing the same sample?

Doubt 3: How do samples have varying sizes, there are possibilities to work with 4 dimensions, for example: “Samples” x “Part of Samples” x “Time Steps” x “Features”?

Thank you for your attention!

One sample is one sequence.

Each sample must have the same length, you can use zero-padding to achieve this and use Masking to ignore the padded obs.

This tutorial will make it clearer I think:

https://machinelearningmastery.com/prepare-univariate-time-series-data-long-short-term-memory-networks/

Hi Jason,

Thank you for such a good tutorial! This really helps!

I am not sure if I understand the model correctly:

The sequence of samples does matter in lstm because the state of current one is affected by last one in the sequence.

If this is the case, can you let me know how I should deal with the following scenarios?

I have non-equally spaced trajectory data. The interval varies from seconds to days. Solutions I come up with are interpolation or adding time feature. What do you think is a good way to prepare the data?

Assuming last problem is solved, how can I organize the input if my data contains trajectories of different people? For example, trajectory of one person is (100, 5, 2) and trajectory of another one is (200, 5, 2). How to train both sequence in one model?

Thank you very much!

Thanks.

Perhaps you can interpolate the data?

Perhaps you can use zero padding with a masking input layer to ignore the zeros?

I have more on preparing data for LSTMs here:

https://machinelearningmastery.com/faq/single-faq/how-do-i-prepare-my-data-for-an-lstm

If each trajectory is a sample, and samples are independent, then perhaps use a batch size of 1 or reset at the end of each sample. Compare the skill of this model to one that does not reset state so often.

Hello Jason, you are making a good job Dr.! I am a bit confused about my data shape for the network: I have 300 different samples, where the next one always is measured lets say in 1 min steps, so I have in total 300 timesteps, and each file is containing 1 column, 2.000 rows. When we say I want to reduce the ‘features’, rows I am thinking that my inputshape must be therefore (1,300,2000) and than I can reduce to something e.g.200. with the lstm decoder ?

How can 1 sample have 300 time steps, 1 column and 2k rows? I do not understand sorry.

Sorrry for the obscurity, 1 sample has 2000 rows all in one column so only one type of value temperature is measured. in total I have 300 samples and time distance between their recording is 1 min

I still don’t follow.

Are the 2000 rows related in time for one feature? Or are the 2000 rows separate features at one time?

I think if I understood you right, the 2000 rows are related to one measurement (so I measure in 1 second 2000 times the temperature). But when you regard it with an lstm autoencoder I try to reduce the “features” to learn from them and make than the prediction. I do not know the shape for the lstm encoder decoder either it should be (1, 300,2000) or (2000,300,1) but for the last one I got strange results, the first one is closer to the real data. Which one is right ?

The input to the encoder will be [300, ?, 2000] where ? represents the number of time steps you wish to model.

The encoder decoder is not appropriate for all sequence prediction problems, it is suited to sequence output that differs in length to the input. If you are doing straight sequence classification/regression it might not be appropriate.

HI Jason, the example is really good. Besides this I have a question for my data. I have temperature values measured for a sampling rate of 1 second with a sampling frequency of 10.000. So I measure in 1 second 10.000 different values but same unit(lets say force). This I repeat with certain time intervals. Do I have than 10.000 different features or only one feature as input dimension ?

Sounds like 10K features at each time step.

this means that one time step can have 10k features?

An NLP problem may have 100K features or more.

great tutorial jason .. but i have a problem in the reshaping my RNN model,,

this my code

import numpy as np

from keras.datasets import imdb

from keras.models import Sequential

from keras.layers import Dense

from keras.layers import LSTM

from keras.layers import Bidirectional

from keras.preprocessing import sequence

# fix random seed for reproducibility

np.random.seed(7)

train = np.loadtxt(“featwithsignalsTRAIN.txt”, delimiter=”,”)

test = np.loadtxt(“featwithsignalsTEST.txt”, delimiter=”,”)

x_train = train[:,[2,3,4,5,6,7]]

x_test = test[:,[2,3,4,5,6,7]]

y_train = train[:,8]

y_test = test[:,8]

# create the model

model = Sequential()

model.add(LSTM(20, input_shape=(10,6)))

model.add(Dense(1415684, activation = ‘sigmoid’))

model.compile(loss=’binary_crossentropy’, optimizer=’adam’, metrics=[‘accuracy’])

model.fit(x_train, y_train, epochs = 2)

What problem?

a problem of reshaping the dataset..

this is a sample of my dataset

a sample of my dataset (patient number, time in mill/sec., normalization of X Y and Z, kurtosis, skewness, pitch, roll and yaw, label) respectively.

1,15,-0.248010047716,0.00378335508419,-0.0152548459993,-86.3738760481,0.872322164158,-3.51314800063,0

1,31,-0.248010047716,0.00378335508419,-0.0152548459993,-86.3738760481,0.872322164158,-3.51314800063,0

1,46,-0.267422664673,0.0051143782875,-0.0191247001961,-85.7662354031,1.0928406847,-4.08015176908,0

1,62,-0.267422664673,0.0051143782875,-0.0191247001961,-85.7662354031,1.0928406847,-4.08015176908,0

Yes, what is the problem that you are having exactly?

i didn’t even know how to reshape my dataset to fit the RNN model.

can you please help me?

Here is a ton more help on the topic:

https://machinelearningmastery.com/faq/single-faq/how-do-i-prepare-my-data-for-an-lstm

i didn’t kow how td do it !

Take it slow, one step at a time.

this is what i have accomplished

train = np.loadtxt(“featwithsignalsTRAIN.txt”, delimiter=”,”)

test = np.loadtxt(“featwithsignalsTEST.txt”, delimiter=”,”)

x_train = train[:,[2,3,4,5,6,7]]

x_test = test[:,[2,3,4,5,6,7]]

y_train = train[:,8]

y_test = test[:,8]

model = Sequential()

model.add(LSTM(64,activation=’relu’,batch_input_shape=(100, 10, 1),

stateful=True,

return_sequences=False))

model.add(Dense(1, activation=’linear’))

model.compile(loss=’mean_squared_error’, optimizer=’adam’)

is that true ??

Nice work.

What do you mean by true?

Our job is to find a model that gives “good enough” results when making predictions. This requires careful experimentation.

thank you. i have tried the following code

np.random.seed(7)

train = np.loadtxt(“featwithsignalsTRAIN.txt”, delimiter=”,”)

test = np.loadtxt(“featwithsignalsTEST.txt”, delimiter=”,”)

x_train = train[:,[2,3,4,5,6,7]]

x_test = test[:,[2,3,4,5,6,7]]

y_train = train[:,8]

y_test = test[:,8]

x_train = x_train.reshape((-1,1,6))

model = Sequential()

model.add(LSTM(64,activation=’relu’,input_shape=(1, 6)))

model.add(Dense(1, activation=’softmax’))

model.compile(loss=’binary_crossentropy’,

optimizer=’adam’,

metrics=[‘accuracy’])

model.fit(x_train, y_train, batch_size = 128, epochs = 10, verbose = 2)

but it gets a very low accuracy with very high loss

Epoch 1/20 – 63s – loss: 15.0343 – acc: 0.0570

Epoch 2/20 – 60s – loss: 15.0343 – acc: 0.0570

Epoch 3/20 – 60s – loss: 15.0343 – acc: 0.0570

Epoch 4/20 – 60s – loss: 15.0343 – acc: 0.0570

Perhaps try other configurations, here are some ideas:

http://machinelearningmastery.com/improve-deep-learning-performance/

What does 32 in model.add(LSTM(32)) mean?

It means 32 LSTM units in the layer.

Would have been nice if you have added this info in your article.

Thanks for the suggestion, you can get started with the basics of LSTMs here:

https://machinelearningmastery.com/start-here/#lstm

Hi, Jason,

Thank you for the great tutorial. It helps me to predict time series data sequences with the lstm model.

However, I have a question about how to determine the length of look_back time steps.

For example, there is a time series sequence X1, X2, X3, …, Xn. When I apply ARIMA for prediction Xn+1, I can use ACF and PCF to determine the parameter pi and qi. The number of pi indicates the look_back time steps. Then, the ARIMA equation can be used to predict Xn+1.

But for lstm, I do not know how to determine the look_back time steps, in other words, the reshape size for a time series sequence. Is there any way to get an appropriate look_back time steps in reshaping the time series sequence data for lstm? Could you pls give me some suggestions about it?

Thanks a lot.

Suzi

Looking at ACF/PACF plots might be a good start to get an idea of the number of lag obs that are significant.

Thanks for your quick reply.

I am still confused about your suggestion. Do you mean that I need plot the ACF/PACF to find the number of time lag for applying lstm?

I do not think the the ACF/PACF can be used for determining the look_back time steps for lstm. These two criteria explain the linear correlation of time series.

For those nonlinear correlation time series sequences, the ACF/PACF is not truncating or tailing and ARIMA cannot be used to model them.

Then I use lstm to model the nonlinear correlation time series sequences and lstm is good at it. Unfortunately, the ACF/PACF is not able to find the time lag in applying lstm.

Before applying lstm for a time series prediction, I must decide the reshape size. However, I cannot find any information on the internet about how to determine it. Is there any book or tutorial can help me to solve this problem?

Thank you very much.

Yes, it is a useful guide / starting point.

I have a ton of advice on preparing data for LSTMs here:

https://machinelearningmastery.com/faq/single-faq/how-do-i-prepare-my-data-for-an-lstm

Hi Jason,

Thanks for the article and clarifying the dimensions which some of us have trouble with them.

However, my question goes to something I didn’t find anybody asked in the Q&A:

Why do you put 32 units in the input LSTM layer?

I mean, if you have 2 features in each of the 10 time steps and one sample example, why would we want to have more than 10 neurons in the first input layer?

As I understand LSTMs, each neuron gets feed with the features of one specific time step (in the cell images of colah’s blog it is stated as Xt, as you will surely know).

If you feed the first one with “t” and continue like “t+1,t+2,t+3…t+10”, what time step will we use in the case of t+17 for example which would be the 17th neuron?

In fully connected ANN the first input layer has the same number of neurons that of features. Is there anything I’m missing or is there any rule to select the number of neurons if we choose that our input layer is a LSTMs one?

Thanks for the attention and for correcting any error that I may be not understanding.

The number of units in the hidden layer is unrelated to the number of input or output time steps.

We configure neural nets by trial and error, more here:

https://machinelearningmastery.com/faq/single-faq/how-many-layers-and-nodes-do-i-need-in-my-neural-network

Hi again Jason,

Thanks for the quick reply.

Let me please introduce some numbers:

Input_shape = (300, 10, 2)

Batch_size = 1

Num_units in input/first LSTM layer = 32

So, as you say, if the units in the input LSTM layer (I am supposing that it is the first layer we use) are not related to the time steps, each time we feed a batch of data into that layer through “Xt” we will feed one row (one sample) of those 300 with 10 columns and we will do it two times: one for the first feature and another for the second feature, and the important point, this feeding will be to every unit of those 32 that compose the LSTM layer. Am I getting the point?

I get confused because in normal feedforward ANN, the first layer (the input layer) has as many nodes as features we have, so that we can feed each feature in one node.

If you could clarify this for me, you would be doing me a big favour because there is not much insight about this details elsewhere.

Thanks in advance,

If your batch size is 1, then each batch will contain one sample (sequence).

Yes, the sequence will be exposed to each unit in the first hidden layer.

Hi Jason,

Okey, perfect. Now I get almost all the points.

Thank you for your kindness,

Hey Jason,

i think i have the same question as Eriz had but i’m not sure whether i understood his explanation right, so i would be great if you could tell me if i got it right.

So the question is: How is the data fed into the first layer of a lstm/rnn ? (i hope there aren’t any differences)

Let’s take Eriz example: 32 Units in the first layer and an input shape of (300,10,2)

I understood Eriz like this:

For one example e (from the 300 examples) all 32 units in the first layer get the time series with length 10 of example e. And this seperately for each feature one after the other (in this case two times) before the network processes the next example.

Is this correct?

Also if we look at this typical illustration of a rnn:

http://www.wildml.com/wp-content/uploads/2015/09/rnn.jpg

Am i right that in this case the variable t in the image would be in the range of 1 <= x <= 10 ? (because of the length of the time series)

Thank you very much in advance, because i couldn't find any detailed description on how this works.

Each of the 32 units in the hidden layer are separate and do not interact.

For a given unit, it receives one time step of data at a time with 2 features. This continues until all 10 time steps have been shown, the final activation is then passed on to the next layer.

Does that help?

Somewhat. I now understand what happens for a single unit.

You said that all units are independent. Does that mean each of the units recieves the same data (the complete time series of an example) in the way you described in your second sentence?

Again, thank you for your effort.

Hi Jason, thank you for the articles and books… I just have some open questions about shape. Since I have a 2D multivariate data ex: (samples = 1024, features = 6) , and make a supervised learning dataset with ten (10) lags, the shape will be (samples = 1024, features = 60).

The question is: The shape for LSTM is (samples, timesteps, features) so it will be data.reshape(1024, 10, 60) ? I dont, understand why some tutorials use something lile (1, 10, 1) and how to reshape/split train/test on the new shape. The steps are:

1 – convert to supervised problem.

2 – reshape the entire 2D dataset or split here and reshape after?

3 – how about shape of Y to make predictions?

I just need a step with these key points… Thanks for the excelent posts.

From your description, there is no need to worry about the lag, the time steps take care of that.

The shape would be (1024, 10, 6)

Hi again Jason, thanks for the reply. On your example for one and multiple features, you say:

– consider a matrix of 2 columns with 10 rows, This data can be framed as 1 sample with 10 timesteps and 2 features.

So, this “1 sample” is drive me crazy. When I have a 2D data like lines vs columns (sample, features), I thought the number of samples will be the number of lines of a matrix 2D data; So it always be (sample, features) –> (sample, timesteps, features). On your example, the rows turned into timesteps and I can’t realize this sample = 1 in this post. Why one sample? Why rows become timesteps here and become samples on other examples?

Another question is: After rechape input data, how to reshape X_train, y_train and new data for predictions.

It is challenging in the beginning.

Think about it like this: you are taking a 2D dataset and projecting it into a 3D space.

Hello,

Why do we need to reshape data on 3 dimensions for the LSTM

Thank you

Because LSTMs expect data as input in 3 dimensions.

Hi Thank you so much for this article, it helped me understand Keras and Overall Input thing.

I am really confused how can I prepare the output data. Overall the output

at one point we use

model.fit(X_train, y_train, validation_data=(X_test, y_test), epochs=3, batch_size=64)

I have (25000, 15) input shape, How can I prepare the overall output

Why, what problem are you having exactly?

Thanks for sharing. I am confused about the padding and “sliding window” method. Suppose the dataset contains two sequences s1,s2 and time_step is set to 3, then s1=(1,2,3,4,5) should have 2 subsequences: [(1,2,3), (2,3,4)], s2=(6,7,8,9,10,11,12) should have 4 subsequences: [(6,7,8), (7,8,9), (8,9,10), (9,10,11)]. Theses 6 subsequences have the same length equal to time_step, so it can be reshaped to 3D tensor (6, 3, 1) without padding s1 and s2 into same length. If all the sequences length are greater than the time_step, then we don’t need to pad the sequences into same length. Am I right?

Padding is only required if the number of time steps differ and/or if obs for a time step are missing.

Hi, thanks for the great article!

Say I have a normalized 2D array data with a shape of (10,2)

but when I want to reshape the data to a 3D array of (10,3,2), I got an error saying:

“ValueError: cannot reshape array of size 20 into shape (10,3,2)”

It seems that the previous 2D array multiply the samples of data of 10 with the input dimensions of 2 before reshaping it to a 3D array, and perhaps that caused the error?

Thanks in advance,

You need more data to go from (10,2) to (10,3,2), think about it, maybe even draw a pic of it. You are invention dimensions that don’t exist in your original data.

You would beed (10,6) to go to (10,3,2)

Ok I think I got it, I should actually divide the samples with the timesteps, because doing this finally solved my problem! Thanks Jason!

Ok I see. So, does reframed first the data with lagged t-n solved the issue?

If you may I Have a Question : I Have 20 Topics (classes) each topic have 700 files each file is a represent a document but in word embedding representation (size of each file : number of words X 300 features ) I want to train a LSTM Network is it possible and how ?

You can get started with LSTMs and text data here:

https://machinelearningmastery.com/start-here/#nlp

Thank you so much I will look it up …

I have another question please, so for my problem does your book “Deep Learning for Natural Language Processing” have LSTM in it because I don’t only want to take the word embedding only but I want to take word’s order in consideration .

or I’ll need your book “Long Short-Term Memory Networks With Python” also ?

Sorry for bothering you with my questions but I’m really stuck and I don’t have anyone who can help me in this matter.

Thanks in advance…

Best Regards,

I give examples of addressing NLP problems with LSTMs as well as other networks like MLPs and CNNs in “deep learning for nlp”.

Thanks, Jason, for your wonderful blog posts!

I have a question regarding the input shape which I cannot find a solid answer to. I don’t know how much this question is related to this blog post, but would appreciate to hear your answer to my question:

I have a training set which contains sequences of images (say n is the number of the images in the sequence and c, h, w are channel , height, and width). I have trained a CNN-LSTM on that with the input shape of (n,c,h,w).

Now, for predicting through this network, it seems I have to feed sequences of data to it at each time (not a single frame). That is, with each new frame I need to update the sequence and feed it to the network to get the results.

However, I was under the impression that when dealing with RNN or LSTM, we can feed one frame at a time (because of recurrency), rather than feeding the whole sequence. Was this impression wrong?

So, briefly, when having an LSTM network for real time prediction, do we need to feed sequences to the network, or are there cases that we may feed a single signal/frame/datapoint?

Thanks a lot in advance!

Yes, you can feed one frame of video at a time and have the CNN interpret the frames, then the LSTM put the sequence together and interpret them all.

I have an example of this in my LSTM book. I have a summary of how to do this in Keras here:

https://machinelearningmastery.com/cnn-long-short-term-memory-networks/

Thanks! I went through your other blog post before (and now again). But still I don’t see how I can feed one frame at a time. How about the input size?

Do you mean I can have a CNN and a seperate LSTM. Feed frames one at a time to CNN and then in a sequence to LSTM? This means, again, I have to create the sequence myself to feed to the network?

What I don’t understand is that the input-shape of the trained network is defined to be (n, c, h, w), how can I feed an input of shape (1,c,h,w) when n is not 1?

By the way, I have already wrapped my CNN in TimeDistributed layer. my code is as below

model.add(TimeDistributed(Conv2D(24, (5, 5), padding=’same’, activation=’relu’, kernel_constraint = maxnorm(3), kernel_initializer=’he_normal’), input_shape=(5,1, 125, 150)))

model.add(TimeDistributed(MaxPooling2D(pool_size=(2, 2), strides = (2,2))))

model.add(TimeDistributed(Dropout(0.4)))

model.add(TimeDistributed(Conv2D(36, (5, 5), activation=’relu’, padding=’same’, kernel_constraint=maxnorm(3))))

model.add(TimeDistributed(MaxPooling2D(pool_size=(2, 2), strides = (2,2))))

model.add(TimeDistributed(Dropout(0.6)))

model.add(TimeDistributed(Conv2D(50, (5, 5), padding=’same’, activation=’relu’, kernel_constraint = maxnorm(3))))

model.add(TimeDistributed(MaxPooling2D(pool_size=(2, 2), strides = (2,2))))

model.add(TimeDistributed(Dropout(0.4)))

model.add(TimeDistributed(Conv2D(70, (5, 5), padding=’same’, activation=’relu’, kernel_constraint = maxnorm(3))))

model.add(TimeDistributed(MaxPooling2D(pool_size=(2, 2), strides = (2,2))))

model.add(TimeDistributed(Dropout(0.5)))

model.add(TimeDistributed(Flatten()))

# define LSTM model

model.add(LSTM(128, return_sequences=True))

model.add(LSTM(32))

model.add(Dense(2, activation=’sigmoid’))

No, it is one model. The size to the CNN is the size of one image. Images are exposed to the model in a sequence, something like:

[samples, frames, width, height, channels]

Thanks, Jason, for your wonderful post!

I created a model before to read your post, and I see that I made a mistake. I swapped time step by feature. How much this can impact in model performance?

Best regards!

The model learns over time. Time is key to the models understanding of the sequence.

Hi, thanks for the great article 🙂

One thing I did’t understand on LSTM network-

If the output of each time step suppose to predict the next input,

how come that the input vector dimension (witch related to the features number) not equal to output vector dimension (witch related to the number of units in the layer)

Other way around.

The output at one step may be used as input at the next step.

Perhaps this will help:

https://machinelearningmastery.com/convert-time-series-supervised-learning-problem-python/

Hi Jason,

Many thanks for the post – it was really useful!

I would just like to run my problem through with you just to verify you feel the approach outlined in this tutorial is suitable for me:

I have two columns of data – one is resonance energies and the other is corresponding neutron widths. I want to feed 70% of this data to the network i.e values of resonance energies and neutron neutrons. Then ask the network to predict the neutron width values given the remaining 30% unseen resonance energies.

I wanted an LSTM layer as it may help use previous computations in its current prediction.

So I believe I have 2 inputs and one output.

If i have 300 values of [resonance energies, neutron widths] (i.e 300 rows of data) would my reshape be:

reshape(1, 300, 2) or reshape(1, 300, 1) ? I’m not sure if the second column is technically a feature as its meant to be the output.

Also would i need any explicit pairing given each resonance energy is related to the neutron width on the same row? Or should i use some key-value pair?

(This is also the first experiment, I also hope to then use resonance energy and neutron width to predict another variable but essentially in exactly the same way as described in this problem just the new experiment contains one more feature)

It sounds like you have 300 time steps of 1 input feature.

You could have one sample of [1,300,1] but that would not be enough to train a model.

Perhaps you can split the input samples into smaller subsequences of 5 or 10 time steps?

Perhaps this worked example will make things clearer for you:https://machinelearningmastery.com/prepare-univariate-time-series-data-long-short-term-memory-networks/

Thank you very much for the swift reply.

Ah my data is here

https://pastebin.com/index/9qwJU3AQ

I have read the link you provided however I am unclear as to whether my data allows me to drop the time variable as mentioned in your article. If so, I could perhaps have the sample of [1,300,1] as opposed to [1,300,2]

One query I have is I’m getting a score of ‘Test Score: 0.00 MSE (0.01 RMSE)” for my test set (which is 30% of my samples) would not having enough samples really be shown by such low RMSE scores? If anything doesn’t that show the predictions are almost too good (or overfitting)?

Sorry one final thing from reading this tutorial – If one uses an LSTM layer, is it still possible to use look_back? (an argument used quite frequently in your other tutorials when creating a dataset). If I am correct, an LSTM layer essentially allows for previous calculations to be examined when determining the current calculation, but look_back determines how many previous timesteps can be consider at each timestep calculation?

I do not have the capacity to shape the data for you. I believe you have everything you need to shape your data for an LSTM model.

A look-back refers to the number of prior time steps of data to feed to the model in one sample. E.g. the “timesteps” in [samples, timesteps, features].

Hi Jason,

Sure that sounds good I will have a go at that.

One quick question I had is when I plot my results in many of your tutorial you tend to use the lines:

testPredictPlot = numpy.empty_like(dataset)

testPredictPlot[:, :] = numpy.nan

testPredictPlot[len(trainPredict)+(look_back)+1:len(dataset)-1, :] = testPredict

so the first line simply creates the numpy matrix like dataset,

but does the second line fill it with nan values? (if so why, or is it just a check?)

The third line then shift the test predict plot?

Many thanks

You’re explanation seems correct.

Hi Jason,

Thank you for this post. I’ve learned a lot from it.

I have a question about my LSTM model for classification.

My input data is 4842 samples, 34 time steps, 254 features. In other words, it’s (4842,34,254).

I have trained it with proper parameters. Although I got a decent result with 98% accuracy on validation data, I got pretty low accuracy at around 20% on test data (from separate data).

My first thought is overfitting but I also tried callback function such as earlystopping or reducelronplateau. It does not give me a better result.

Could you give me any suggestions on this issue?

Many thanks!

Yes, sounds like overfitting.

I have a suite of ideas here to try:

http://machinelearningmastery.com/improve-deep-learning-performance/

Hi Jason,

I learn a lot from your blog posts. For this specific post, I have three specific questions.

Q1: How do we decide the value of FIRST parameter of the constructor LSTM. You used LSTM (32, …). How was that value of 32 decided for the representative problem that is being addressed here? For word embedding input, is a vlaue between 200 and 500 reasonable?

Q3: What is the significane of this parameter? Is it number of LSTM cells and should it be matched with the value of dimension of input layer of the Keras model (in case of work embedding, value b/w 200 and 500)?

Q3: What would be perfformance impact of choosing a value of 500 for this parameter?

Thanks and Regards

The number of nodes in the first hidden layer is found via trial and error or tuning, more here:

https://machinelearningmastery.com/faq/single-faq/how-many-layers-and-nodes-do-i-need-in-my-neural-network

I just don’t understand reshape. I do not see how you calculate the array and enter it to batch size or train data on it. I have been stuck on this all day and googling. Any help from anyone is greatly appreciated

Perhaps go back to basics and learn how to use the reshape function:

https://machinelearningmastery.com/index-slice-reshape-numpy-arrays-machine-learning-python/

hello Jason, thanks for your great explanation

can you help me with this question:

https://stackoverflow.com/questions/52317804/recurrent-neural-network-using-different-time-steps-with-keras

Perhaps you can summarize it for me in a sentence?

Hi Jason,

I have multidimensional timeseries data with a sample size of 200,000 and 50 dimensions.

I want to train a sequence to sequence autoencoder on normal data to use it later to detect anomalies. I want to have a go at this task using a LSTM autoencoder, using the example from the keras site:

https://blog.keras.io/building-autoencoders-in-keras.html

inputs = Input(shape=(timesteps, input_dim))

encoded = LSTM(latent_dim)(inputs)

decoded = RepeatVector(timesteps)(encoded)

decoded = LSTM(input_dim, return_sequences=True)(decoded)

sequence_autoencoder = Model(inputs, decoded)

encoder = Model(inputs, encoded)

I am confused about how to convert my data.

When generating the sequences lets say with a timestep of 100: do I convert this 200k data into separate sequences of 100, or use a sliding window to generate my sequences?

Many thanks for your help.

Yes, you split the long sequence into subsequences.

No need to overlap, but you can if you want and see if it improves detection.

Hi Jason,

Excellent tutorial. I have a question that I wanted to ask.

I have a total of 3 sequences.

Sequence 1: It has a shape of 800×2500 (800 observations and 2500 features) It falls into category 1

Sequence 2: It has a shape of 1000×2500. It falls into category 2

Sequence 3: It has a shape of 600×2500. It falls into category 3.

I have combined these 3 sequences into 1 array which has 2400×2500 features. I want to train an LSTM network on this array. Want it to learn pattern of these sequences and predict the category (1,2 r 3) given a new test sequence of any length (? x 2500) shape.

What should my input shape be? Should it be (1,2400,2500)?

Each sequence would be a separate sample and the number of time steps would have to be padded with zeros to match.

The shape would be: [3, 1000, 2500]

Training a model on 3 samples does not sound useful, you might need 3 thousand or 3 million samples.

Also, 1000 time steps might be a little long, 200-400 is preferred.

model.fit(trainX,trainY,epochs=100,verbose=0)

ValueError: Error when checking input: expected lstm_2_input to have shape (3, 1) but got array with shape (1, 1) this error occur

I have some suggestions here:

https://machinelearningmastery.com/faq/single-faq/why-does-the-code-in-the-tutorial-not-work-for-me

Hi Jason,

thank you so much for this tutorial. I have a question conserning Conv2D

i want to devlop model for binery image classification with size (256*256)

i put my image in liste numpy with lenth (10000) and each index have np.array (256*256)

i get error when i start fit the model with input_shape = (256,256,1)

what i should define input shape

thank you

A 2D CNN will require input with the shape: rows x cols x channels. [256,256,1] sounds right.

Hello, thank you for your tutorials.

I am trying to understand data inputs to LSTM autoencoder, but I am lost.

I want use autoencoder for anomaly detection in time series (falling detection).

Dataset:

500 samples (data from gyroscope – walking, jumping, running…)

600-10000 time steps in each sample (6-100seconds).

3 features.

Example of one sample(one file):

time x y z

1 1.3 9.6 1.3

2 1.2 9.3 1.5

3 0.9 8.0 -2

. . . .

. . . .

. . . .

1000 1.4 9.8 2

Idea:

Train autoencoder to reproduce normal data(walking,jumping,running…).

Anomaly detection: Reproduction error.

I want to check anomaly (fall) in 2second time windows(200 time steps) -> Reproduce every 2second window and check reproduction error.

Questions:

Can you please explain how to prepare dataset for this task?(dimensions, structure of dataset…)

How big batches and time steps(seq_length) shoud I use?

Should I generate batches randomly, or from start of dataset? (batch1: 0 – batch_size, batch2: batch_size-batch_size*2 …) -> I saw that someone generated random batches in every iteration. Couldn’t that cause some data to be used multiple times and others is not used at all?

Thank you.

I have an example of preparing activity recognition data or an LSTM here that might be useful:

https://machinelearningmastery.com/how-to-model-human-activity-from-smartphone-data/

Hello, thanks a lot for this tutorial. I’m working on some project but still stuck after reading this. Here’s the description

I have a dataframe of 2 columns, both text – one is title and other is the label to it.

Unique label count is around 40k so one hot encode was out of question.

I used word2vec with size=150 for both, trained and used the created model to encode both title and label.

e.g. for hello world

I split them and then use their respective word vectors of size 150, add them and normalize to create a vector that represents hello world.

So both columns in my dataframe had been changed and each column of each row has a vector.

dataframe.shape shows (len, 2) where len is length of dataframe

and then I did

X = df[‘title’].values

y = df[‘label’].values

and I got two numpy.ndarray with following shapes

X.shape –> (len,)

X[0].shape –> (150,)

y.shape –> (len,)

y[0].shape –> (150,)

After this I’m stuck with input and output shapes for the network.

I tried with LSTM and I got the error that lstm expected 3 dimensions but got array

I tried with Dense layers and still got shape errors.

Basically I’m struggling as to what the input and output should be for the network.

Any help in this regard is appreciated. I can provide more details and code if needed.

What do you mean you used word2vec for the label? Does that mean you are predicting a vector that is then mapped to words? Sounds odd.

I would recommend an embedding on the front end and softmax on the output with 40k one hot encoded vectors.

hi jason, thank you for the tutorial.

I am trying to feed 3 column(3316 rows) merged encoded text Train_data and Train_Labels categorizing class 1-9(3316 rows) to an lstm network.

Train_input last column is output of word embedded vector of 50 dimensions,(3316,50)

Train_input first and second columns are words – one-hot encoded text data (3316,2)

After merging three columns the shape is (samples=1, time_steps=3316, features=3)

TraIN_LABELS categorizing above train_data into class 1-9. encoded it using label encoder.

train_input=(1,3316,3), train_label=(1,3316,9) facing error with this data shape

reshaping labels to (1,3316,3) is not happening

how do i reshape labels to feed to lstm?

The labels are typically a 1D array with one element per input sample.

Thank you for the reply jason.

LSTM network returned error as “expecting dense 3dimensional shape instead recieved

(3316,)” when given 1D array.

Is there any other way that i can feed? Have i done any mistake in reshaping train data?

Really appreciated your explanation!

Could you explain how to feed multi-input the LSTM, let’s say:

you have: data = data.reshape(1, 10, 2)

data = array([

[0.1, 1.0],

[0.2, 0.9],

[0.3, 0.8],

[0.4, 0.7],

[0.5, 0.6],

[0.6, 0.5],

[0.7, 0.4],

[0.8, 0.3],

[0.9, 0.2],

[1.0, 0.1]])

and model.add(LSTM(32, input_shape=(10, 2)))

So in1 iteration of epoch, the value in the first column: 0.1,0.2,….. 1.0 will be fed into xt, xt+1,…, xt+9 of input gate of LSTM. And the 2nd column: 1.0, 0.9…0.1, will they be also fed into xt,xt+1,…,xt+9 or they will fed into another input gate of LSTM: xxt, xxt+1,….,xxt+9 ?

Not quite, if there is 1 sample, and your batch size is 1 sample, then all time steps in the 1 input sample will be fed into the model for epoch 1.

Sorry, I am still not clear !

So model.add(LSTM(32, input_shape=(10, 2)))

what is the number 2 in the input_shape(10,2) mean ?

(10,2) means that each sample will have 10 timesteps with 2 features.

Hello Jason, thanks a lot for your great post.

I had understood the post and made a dataset as follow.

x_train_shape : (35849, 100, 3)

y_train_shape : (35849, )

x_validation_shape : (8963, 100, 3)

y_validation_shape : (8963, )

x_train_shape : (11204, 100, 3)

y_train_shape : (11204, )

It is a time series sensor data of three-channel.

And, the actual form of the data is as follow.

(2067, 1976, 1964)

(2280, 1994, 1952)

(2309, 1976, 1968)

.

.

.

(2020, 2160, 1979)

(1994, 2181, 2064)

I did labeling the data for particular section by [window size : 100, stride : 15] per a channel.

For example, if the particular section is from 251 to 750, 27 pieces of cutted-data are made as follows.

251~350, 266~365, 281~380, …, 641~740

With this data, I was able to proceed learning with DFN and CNN to perform effectively.

However, when I do learning using LSTM, the learning does not proceed that the loss does not decrease and the accuracy is around 50%.

So I would like to hear your opinion about this phenomenon, which has a broad and deep knowledge in this field.

Should I change the data structure differently?

Or, is there a particular LSTM-containing model structure suitable for data like this structure?

Or, do you have another new opinion?

Thank you so much for your interest in my problem.

LSTMs are generally poor a time series forecasting. You may need to carefully tune the model.

Thank you for your reply.

Last night, I was contemplating a lot. I was able to figure out why my dataset is not suitable for this model.

This is because a unit of data cannot affect the decision.

So I’m going to find a way to increase the data processing unit on LSTM.

I have been able to think a lot through your material and reply.

Thank you so much again.

Nice work.

Hi,

I am really confused with Input shape. Assume I have a data set of some info about houses.

Features are [size, room_no, floor_no]

My data set contains 4 samples as follow:

dataX = [[200, 3, 1], [150, 1, 1], [270, 4, 2], [320, 3, 2]]. Which for example dataX[0] is house number0 with size of 200, 3 rooms and 1 floor.

Now I want to train my LSTM. Are (samples, time_step, feature) different from the ones I defined here? I mean I have 4 samples with 3 features! How do you tell that to LSTM? For example would you please say that in the first 2 or 3 time steps what data is fed into LSTM?

Thank you very much. 🙂

Each of size, rooms and floor are “features”. Features=3

If you have 3 houses, these are “samples”. Samples=3

But you did not mention any time steps. If you have no time steps (observations over time), an LSTM is a poor choice and I would recommend an MLP instead.

Hello Jason,

First of all, thanks for all your work. It is grateful to have a support like you give us in all your tutorials!

On the other hand, I have a question. For LSTMs we need a 3D array (samples, features, timesteps). But I still don’t understand what “timesteps” means? Is it the same of loop_back variable?

Thanks,

Time steps are observations (features) made over time (e.g. minutes, hours, etc.). Or, they can be items in a sequence like words in a sentence.

Does that help?

Like others, I wanna say thank you for this and other useful articles.

I need to connect output of ConvLSTM2D to a LSTM!

The output of ConvLSTM2D is (samples, time, output_row, output_col, filters) which return_sequences is True.

I’m confused here how this 5-D input can be feed to a LSTM!

I will be thankful if any help!

Are you sure the output of the ConvLSTM2D is as you describe?

If so, perhaps you can use a lambda to flatten rows/cols and filters.

Hello Jason,

I have a dataframe of n,p rows (n for the different samples and p for q timesteps and r static feature p = q+r ). For each row, I have q values for the measure of interest and r static features.

Basically, I want to do a time series classification that is based also on the static features.

From what I understood, the input shape should be (n,q,r) but I cannot transform my dataframe from (n,p) to (n,q,r) since p = q+r

Thank you a lot for your help !

I don’t follow what you mean by p = q+r?

Dear Jason,

why do we need to reshape the data in numpy before feeding the data to the lstm. Why Keras doesn’t do it automatically?

if I have a sequence of 10 values and I want to predict the 11th value, I guess Keras LSTM has already all the information in a numpy array of size (1,10) to answer the task of predicting the 11th value . Why do I need to reshape it to (1,10,1) ?

regards

Guido

Why – that is the expectation of the API. Don’t fight it, meet it.

Hi Jason,

Thank you for your great post.

I have problem about padding.

I use the pre-trained sentence embedding(dim:300),

and 1 sample means a document in my application.

Now I have variable sentence number in documents.

(Not sentence sequences with variable words length)

E.g., doc1:[sent1, sent2]

doc2:[sent1]

doc3:[sent1, sent2, sent3]

The zero-padding in variable sentence sequences means pad 0 after sequence of word_index

But pre-trained sentence embedding can’t map sentence to index

How do I pad the sentence number to a fixed size?

Does it just pad 300 zeros in each timestep, and use mask layer to skip those zeros?

E.g., pad_sent = np.zeros((1,300))

doc1:[sent1, sent2, pad_sent]

doc2:[sent1, pad_sent, pad_sent]

doc3:[sent1, sent2, sent3]

Thank you.

The padded values can be mapped to the index of “unknown” in the embedding, which is often also 0.

Hi Jason Hopefully you have not already answered this. To prep the x data for LSTM input,

the data is reshaped to a 3 d array. Should something similiar also be done to the y data? Should it also be divided up into subsequences so that the output is synced up with the input?

If so, should that be done prior to transforming the data with one hot encoding? Or does the

one hot encoding come after? Thanks for all your help.

No, not unless you are predicting a sequence output.

If you are predicting a multi-class label, a one hot encoding is used.

Hi Jason,

great article – always come here when needed a boiled-down explanation. Thank you!

Regarding input shapes – have been using LSTM for a while and didn’t have any problems with it but now I tried 1D convolutional layers for speeding up processing and now I run into trouble – can you see what the problem is with the following? (Dummy data used here)

#load packages

import numpy as np

import pandas as pd

from keras.models import Sequential

from keras.layers import Dense, Dropout, Activation, GRU, TimeDistributed

from keras.layers import Conv1D, MaxPooling1D, Flatten, GlobalAveragePooling1D

from keras.layers import Conv2D, MaxPooling2D

from keras.utils import np_utils

nfeat, kernel, timeStep, length, fs = 36, 8, 20, 100, 100

#data (dummy)

data = np.random.rand(length*fs,nfeat)

classes = 0*data[:,0]

classes[:int(length/2*fs)] = 1

#splitting matrix

#data

X = np.asarray([data[i*timeStep:(i + 1)*timeStep,:] for i in range(0,length * fs // timeStep)])

#classes

Y = np.asarray([classes[i*timeStep:(i + 1)*timeStep] for i in range(0,length * fs // timeStep)])

#split into training and test set

from sklearn.model_selection import train_test_split

trainX, testX, trainY, testY = train_test_split(X,Y,test_size=0.2,random_state=0)

#one-hot-encoding

trainY_OHC = np_utils.to_categorical(trainY)

trainY_OHC.shape, trainX.shape

#set up model with simple 1D convnet

model = Sequential()

model.add(Conv1D(8,10,activation=’relu’,input_shape=(timeStep,nfeat)))

model.add(MaxPooling1D(3))

model.add(Flatten())

model.add(Dense(10,activation=’tanh’))

model.add(Dense(2,activation=’softmax’))

model.summary()

#compile model

model.compile(loss=’mse’,optimizer=’Adam’ ,metrics=[‘accuracy’])

#train model

model.fit(trainX,trainY_OHC,epochs=5,batch_size=4,validation_split=0.2)

I get an error for the fitting:

ValueError: Error when checking target: expected dense_17 to have 2 dimensions, but got array with shape (400, 20, 2)

I cannot see what is wrong here?!

I suspect your output variable y does not match the expectation of the model of 2 values per sample. Instead, you’ve provided [400,20,2]

Hello Jason,

Thanks for the tutorial. Your tutorials are a life-saver! I have a simple doubt.

I have extracted speech features from 597 .wav files into arrays. Each array is of shape Nx40, where N varies between 500-1100. In other words, for each of the 597 .wav files, I have an array of N rows (varying between 500-1100) and 40 columns (40 acoustic features).

1. So, what should be my 3D input shape to the LSTM? Is it (1, ?, 40), where ? = N.

2. If it is (1, ?, 40), then should I pick a particular length and pad the rest of them?

Really stuck with this. Any response will be of immense help. Thanks!

Sounds like you have nearly 600 samples, 500-1100 time steps and about 40 features.

I’d recommend padding the time steps and having a shape of something like [600, 1100, 40] as a start, then try truncating time steps to see how it impacts the model.

Hey Jason,

I have a Time series dataset for 38,000 distinct patients, where each patient has 30 physiological parameters recorded for an hour. If I want to extract 48 hours of information of a patient for every hour, then technically I’ll have 48 rows for a single patient, every row containing observations for 30 feature for every hour! suppose I want to extract similar data for the other 29,000 patients, then ill end up with over a lakh rows i.e(29,000 * 48 rows). So should my input shape be (30,000, 48, 30) ?

Maybe, I’m not sure I follow about “48 hours every hour”. Perhaps try your approach?

What should be the output layer shape? Particularly, if I have only one sample, do I need to reshape it into three dimensions? For example, if I have the input data of shape [m,n] and output also has [m,n] shape. Do I need to change the input into [m,1,n] shape? Also, can I keep the output shape [m,n] if I am using return_sequences=False? I am dealing with time series data. Below are the input and output at different time steps for example

Input: Output:

[1,2] [3,4]

[2,3] [4,5]

[4,5] [6,7]

Thank you

Output shape is often [samples, features] for a normal model or [sample, time steps, features] for an encoder-decoder model.

Hi Jason,

Your article is so good. I want to apply this to my data set.

I have 3000 samples and 88 features(columns).In that number of feature columns, I have 20 features called A, B, C, D etc.For each one of them I have 3 columns A1,A2,A3 , B1,B2,B3 , C1,C2,C3 etc as lag columns. Thus 20+20*3 = 80 are the feature columns with lags

and 8 other features which do not have lag values.

How to convert that into a shape of 3D array to feed as input to LSTM model?

Your reply is greatly appreciated.

Good question, this will help:

https://machinelearningmastery.com/faq/single-faq/what-is-the-difference-between-samples-timesteps-and-features-for-lstm-input

hi jason

I want to write code for functionalities of input gate ,output gate and forget gate for LSTM.help me sir

Sorry, I don’t have an example of coding an LSTM cell from scratch, thanks for the suggestion.

sir,

directly we are using LSTM function to process the data .instead we can write the code for input gate ,forget gate and output gate by creating own lstm function

I recommend using Keras and not coding an LSTM from scratch as it will almost certainly have bugs and be less efficient.

Hi

Thanks so much for this, your stuff is really helping me get my head around LSTMs

I’m struggling to understand how to load more than just one sample though. This works fine:

samples = np.array([

[0.1, 1.0],

[0.2, 0.9],

[0.3, 0.8],

[0.4, 0.7],

[0.5, 0.6],

[0.6, 0.5],

[0.7, 0.4],

[0.8, 0.3],

[0.9, 0.2],

[1.0, 0.1]])

samples = samples.reshape(1, 10, 2)

But how should I present my data to use something with (2,10,2) or (3,10,2)? Is it an array of arrays?

Ultimately I want to be able to go from an hdf5 file to a numpy array to this 3d shape, but I’m struggling to tell my model that there is more than one sample, even when i’m just making up and typing the data as a toy example. Any tips?

Load more data into memory, then reshape it.

Perhaps I don’t understand the problem you’re having.

Also, this may help with mindset:

https://machinelearningmastery.com/faq/single-faq/what-is-the-difference-between-samples-timesteps-and-features-for-lstm-input

I don’t think I explained it very well. Thanks for taking the time to reply. I think my question is what should my data look like if I have more than one sample and I’m trying to use reshape?

What I mean is at the moment ‘samples’ is a numpy array representing just one example of 2 features and 10 timesteps. when really, samples should be a dataset of many examples. And I don’t understand how to write that so that it can be reshaped. Should samples be a list of numpy arrays. or does it need to be an array of arrays or something else? I’ve tried both but neither seems to work.

Reading this https://machinelearningmastery.com/gentle-introduction-n-dimensional-arrays-python-numpy/ suggests it could be that i need to use vstack?

Yes, a vstack, dstack or hstack will do it.

I cannot know, I don’t have your code/data and it is completely specific to your data.

If you’re having trouble, try tinkering with a few contrived samples in a separate python file until you get it right.

Hi!

I have a df like this https://imgur.com/a/hmg2Ng4

If I want use temporal component for a LSTM, i think that my sequence, will be make by date_col.

But, If I select date_col, I will have a array of new information. I mean, it’s not a row typical secuence, I think this is more complex.

My secuence will be [day1,day2,day3] and, in each day, I have a array with [product1,product2,product3], and each product [feat1, feat2].

For day 1: [[feat1,feat2],[feat1,feat2],[feat1,feat2]]

Secuence will be:

[ [[feat1,feat2],[feat1,feat2],[feat1,feat2]],

[[feat1,feat2],[feat1,feat2],[feat1,feat2]],

[[feat1,feat2],[feat1,feat2],[feat1,feat2]] ]

This is correct? This will work with neural networks?

I’m seeing that I’m writing so bad.

‘My secuence will be [day1,day2,day3] and, in each day, I have a array with [product1,product2,product3], and each product [feat1, feat2]. ‘

when I say productX, i want to say idX

Perhaps.

I think this explanation will help you develop an intuition for data prep with LSTMs:https://machinelearningmastery.com/faq/single-faq/what-is-the-difference-between-samples-timesteps-and-features-for-lstm-input

Hi Jason. Thank you for your informative articles. They have been very helpful to me.

I was wondering if you could give me your recommendation on setting up a LSTM model. I’m dealing with three features that are measured when a passive RFID tag is read by an RFID reader. I understand that before setting up my model, I need to decide how many time steps I will be dealing with for each input. Let’s say I set that number at 32 time steps. This means my input shape will be (32, 3). What I’m not clear on is if I need to add a feature for a time stamp of when that read occurred. Tag reads can happen at any rate. There could be one read per second which would mean my input instance of (32, 3) would span 32 seconds, or there could be one read per minute which would mean my input instance of (32, 3) would span 32 minutes. Most importantly, the read rate will not be constant. I could get 3 reads in 1 second and then wait 30 seconds for the fourth read to come in.

Does the LSTM need to know about this, or is it enough to simply give it the time ordered sequence of reads without it knowing the actual time span those 32 reads occurred over?

If I did have to add a time stamp to my data, the input shape would now be (32, 4). As a follow up question, does the LSTM require me to define a fixed time step between the feature inputs? What I mean is, am I forced to pick a time span of say 1 second between the input feature list? If I am, and I only have reads at time 0 seconds, 4 seconds, and 6 seconds, do I then have to generate my (32, 4) input values as follows:

(0, x0, y0, z0)

(1, missing, missing, missing)

(2, missing, missing, missing)

(3, missing, missing, missing)

(4, x4, y4, z4)

(5, missing, missing, missing)

(6, x6, y6, z6)

…..

OR

is there no assumption needed on the time between inputs so I can I simply stack them without inserting missing values, such as:

(0, x0, y0, z0)

(4, x4, y4, z4)

(6, x6, y6, z6)

Any input you can provide would be appreciated.

Thank you.

Perhaps try modeling as-is, then try with padded/missing values and see if there is any difference in model skill.

Also try other model types. LSTMs are mostly terrible at time series.

Thanks Jason. I was surprised to hear you say LSTM’s are terrible at time series. What I’m trying to implement is a binary sequence classification model using the setup I described above. If you think LSTM’s are not the right approach given what I’ve described, what type of model do you think would work best? I want to process the data as new reads come in and give a classification output every time I get a new read.

I was planning to use your post on: Sequence Classification with LSTM Recurrent Neural Networks in Python with Keras as my starting point, but now you’ve got me concerned I’m going down the wrong path.

LSTMs can be good for time series classification.

In general, I recommend this process:

https://machinelearningmastery.com/how-to-develop-a-skilful-time-series-forecasting-model/

Hi,Jason. I want add a dense layer before LSTM layer, but don’t know how to reshape the data. The input data (train_x) is 3D (batch_size,time_step,input_dim), I firstly want to reshape the data to 2D so as to apply to dense layer,.After operating the dense layer ,I have to reshape the 2D outcome of dense layer to 3D so as to apply in the LSTM layer. I am using keras function API, but I can not find a reshape layer to do that (keras.layers.reshape can not do that.).Do you have any idea?