Archive | Long Short-Term Memory Networks

How to One Hot Encode Sequence Classification Data in Python

How to One Hot Encode Sequence Data in Python

Machine learning algorithms cannot work with categorical data directly. Categorical data must be converted to numbers. This applies when you are working with a sequence classification type problem and plan on using deep learning methods such as Long Short-Term Memory recurrent neural networks. In this tutorial, you will discover how to convert your input or […]

Continue Reading 168
Attentional Interpretation of Words in the Input Document to the Output Summary

Attention in Long Short-Term Memory Recurrent Neural Networks

The Encoder-Decoder architecture is popular because it has demonstrated state-of-the-art results across a range of domains. A limitation of the architecture is that it encodes the input sequence to a fixed length internal representation. This imposes limits on the length of input sequences that can be reasonably learned and results in worse performance for very […]

Continue Reading 30
How to Prepare Sequence Prediction for Truncated Backpropagation Through Time in Keras

How to Prepare Sequence Prediction for Truncated BPTT in Keras

Recurrent neural networks are able to learn the temporal dependence across multiple timesteps in sequence prediction problems. Modern recurrent neural networks like the Long Short-Term Memory, or LSTM, network are trained with a variation of the Backpropagation algorithm called Backpropagation Through Time. This algorithm has been modified further for efficiency on sequence prediction problems with […]

Continue Reading 26
A Gentle Introduction to Backpropagation Through Time

A Gentle Introduction to Backpropagation Through Time

Backpropagation Through Time, or BPTT, is the training algorithm used to update weights in recurrent neural networks like LSTMs. To effectively frame sequence prediction problems for recurrent neural networks, you must have a strong conceptual understanding of what Backpropagation Through Time is doing and how configurable variations like Truncated Backpropagation Through Time will affect the […]

Continue Reading 35
Data Preparation for Variable Length Input Sequences for Sequence Prediction

Data Preparation for Variable Length Input Sequences

Deep learning libraries assume a vectorized representation of your data. In the case of variable length sequence prediction problems, this requires that your data be transformed such that each sequence has the same length. This vectorization allows code to efficiently perform the matrix operations in batch for your chosen deep learning algorithms. In this tutorial, […]

Continue Reading 99
Line Plot of Log Loss for an LSTM, Reversed LSTM and a Bidirectional LSTM

How to Develop a Bidirectional LSTM For Sequence Classification in Python with Keras

Bidirectional LSTMs are an extension of traditional LSTMs that can improve model performance on sequence classification problems. In problems where all timesteps of the input sequence are available, Bidirectional LSTMs train two instead of one LSTMs on the input sequence. The first on the input sequence as-is and the second on a reversed copy of […]

Continue Reading 153
How to use an Encoder-Decoder LSTM to Echo Sequences of Random Integers

How to use an Encoder-Decoder LSTM to Echo Sequences of Random Integers

A powerful feature of Long Short-Term Memory (LSTM) recurrent neural networks is that they can remember observations over long sequence intervals. This can be demonstrated by contriving a simple sequence echo problem where the entire input sequence or partial contiguous blocks of the input sequence are echoed as an output sequence. Developing LSTM recurrent neural […]

Continue Reading 27