Recurrent neural networks are a type of neural network that add the explicit handling of order in input observations. This capability suggests that the promise of recurrent neural networks is to learn the temporal context of input sequences in order to make better predictions. That is, that the suite of lagged observations required to make […]

# Archive | Deep Learning

## How to Learn to Add Numbers with seq2seq Recurrent Neural Networks

Long Short-Term Memory (LSTM) networks are a type of Recurrent Neural Network (RNN) that are capable of learning the relationships between elements in an input sequence. A good demonstration of LSTMs is to learn how to combine multiple terms together using a mathematical operation like a sum and outputting the result of the calculation. A […]

## How to Use the TimeDistributed Layer for Long Short-Term Memory Networks in Python

Long Short-Term Networks or LSTMs are a popular and powerful type of Recurrent Neural Network, or RNN. They can be quite difficult to configure and apply to arbitrary sequence prediction problems, even with well defined and “easy to use” interfaces like those provided in the Keras deep learning library in Python. One reason for this […]

## How to use Different Batch Sizes when Training and Predicting with LSTMs

Keras uses fast symbolic mathematical libraries as a backend, such as TensorFlow and Theano. A downside of using these libraries is that the shape and size of your data must be defined once up front and held constant regardless of whether you are training your network or making predictions. On sequence prediction problems, it may […]

## Demonstration of Memory with a Long Short-Term Memory Network in Python

Long Short-Term Memory (LSTM) networks are a type of recurrent neural network capable of learning over long sequences. This differentiates them from regular multilayer neural networks that do not have memory and can only learn a mapping between input and output patterns. It is important to understand the capabilities of complex neural networks like LSTMs […]

## Multi-step Time Series Forecasting with Long Short-Term Memory Networks in Python

The Long Short-Term Memory network or LSTM is a recurrent neural network that can learn and forecast long sequences. A benefit of LSTMs in addition to learning long sequences is that they can learn to make a one-shot multi-step forecast which may be useful for time series forecasting. A difficulty with LSTMs is that they […]

## How to Use Weight Regularization with LSTM Networks for Time Series Forecasting

Long Short-Term Memory (LSTM) models are a recurrent neural network capable of learning sequences of observations. This may make them a network well suited to time series forecasting. An issue with LSTMs is that they can easily overfit training data, reducing their predictive skill. Weight regularization is a technique for imposing constraints (such as L1 […]

## How to Use Dropout with LSTM Networks for Time Series Forecasting

Long Short-Term Memory (LSTM) models are a type of recurrent neural network capable of learning sequences of observations. This may make them a network well suited to time series forecasting. An issue with LSTMs is that they can easily overfit training data, reducing their predictive skill. Dropout is a regularization method where input and recurrent […]

## Exploratory Configuration of a Multilayer Perceptron Network for Time Series Forecasting

It can be difficult when starting out on a new predictive modeling project with neural networks. There is so much to configure, and no clear idea where to start. It is important to be systematic. You can break bad assumptions and quickly hone in on configurations that work and areas for further investigation likely to […]

## Instability of Online Learning for Stateful LSTM for Time Series Forecasting

Some neural network configurations can result in an unstable model. This can make them hard to characterize and compare to other model configurations on the same problem using descriptive statistics. One good example of a seemingly unstable model is the use of online learning (a batch size of 1) for a stateful Long Short-Term Memory […]