Long Short-Term Memory Networks With Python
Develop Deep Learning Models for your Sequence Prediction Problems
The Long Short-Term Memory network, or LSTM for short, is a type of recurrent neural network that achieves state-of-the-art results on challenging prediction problems.
In this laser-focused Ebook written in the friendly Machine Learning Mastery style that you’re used to, finally cut through the math, research papers and patchwork descriptions about LSTMs.
Using clear explanations, standard Python libraries and step-by-step tutorial lessons you will discover what LSTMs are, and how to develop a suite of LSTM models to get the most out of the method on your sequence prediction problems.
About the Ebook:
- PDF format Ebook.
- 3 parts, 14 step-by-step tutorial lessons, 246 pages.
- 6 LSTM model architectures.
- 45 Python (.py) files.
Clear and Complete Examples.
No Math. Nothing Hidden.
Click to jump straight to the packages.
Sequence Prediction is…important, overlooked, and HARD
Sequence prediction is different to other types of supervised learning problems.
The sequence imposes an order on the observations that must be preserved when training models and making predictions.
There are 4 main types of sequence prediction problems:
1. Sequence Prediction
Given an input sequence, predict the next value in the sequence.
- Weather Forecasting
- Stock Market Prediction
- Product Recommendation
2. Sequence Classification
Given an input sequence, classify the sequence.
- DNA Sequence Classification
- Anomaly Detection
- Sentiment Analysis
3. Sequence Generation
Given an observation, generate an output sequence.
- Text Generation
- Music Generation
- Image Captioning
4. Sequence-to-Sequence Prediction
Given an input sequence, generate an output sequence.
- Multi-Step Time Series Forecasting
- Text Summarization
- Language Translation
Long Short-Term Memory Networks
UNLOCK Sequence Prediction for Deep Learning
Classical neural networks called Multilayer Perceptrons, or MLPs for short, can be applied to sequence prediction problems.
The application of MLPs to sequence prediction requires that the input sequence be divided into smaller overlapping subsequences called windows that are shown to the network in order to generate a prediction.
This can work well on some problems but suffers some critical limitations such as being stateless and having a fixed number of inputs and outputs.
Promise of Recurrent Neural Networks
The Long Short-Term Memory, or LSTM, network is a type of Recurrent Neural Network (RNN) designed for sequence problems.
Given a standard feedforward MLP network, an RNN can be thought of as the addition of loops to the architecture. The recurrent connections add state or memory to the network and allow it to learn and harness the ordered nature of observations within input sequences.
The internal memory means outputs of the network are conditional on the recent context in the input sequence, not what has just been presented as input to the network.
In a sense, this capability unlocks sequence prediction for neural networks and deep learning.
Impressive Applications of LSTMs
We are interested in LSTMs for the elegant solutions they can provide to challenging sequence prediction problems.
Let’s look at 3 examples to give you a snapshot of the results that LSTMs are capable of achieving.
Automatic Image Caption Generation
Automatic image captioning is the task where, given an image, the system must generate a caption that describes the contents of the image.
Automatic Translation of Text
Automatic text translation is the task where you are given sentences of text in one language and must translate them into text in another language.
Automatic Handwriting Generation
This is a task where, given a corpus of handwriting examples, new handwriting for a given word or phrase is generated.
There are a number of RNNs, but it is the LSTM that delivers on the promise of RNNs for sequence prediction. It is why there is so much buzz and application of LSTMs at the moment.
So, how can you get started and get good at using LSTMs fast?
Introducing my new Ebook:
“Long Short-Term Memory Networks With Python“
This is the book I wish I had when I was getting started with LSTMs.
This book was born out of one thought:
What would I teach if I had to get a machine learning practitioner proficient with LSTMs in two weeks?
I had been researching and applying LSTMs for some time and wanted to write something on the topic, but struggled for months on how exactly to present it. The above question crystallized it for me and this whole book came together.
The above motivating question for this book is clarifying. It means that the lessons that I teach are focused only on the topics that you need to know in order to understand (1) what LSTMs are, (2) why we need LSTMs and (3) how to develop LSTM models in Python.
I developed a program to take you on the critical path:
From…a practitioner interested in LSTMs (e.g. you right now).
To…a practitioner that can confidently apply LSTMs (e.g. you after reading the book).
I want you to get proficient with LSTMs as quickly as you can. I want you using LSTMs on your project.
This also means not covering some topics, even topics covered by “everyone else“, like LSTM math.
This book is not for everyone
…so is this book right for YOU?
Let’s make sure you are in the right place.
This book is for developers that know some applied machine learning and need to get good at LSTMs fast.
Maybe you want or need to start using LSTMs on your research project or on a project at work. This guide was written to help you do that quickly and efficiently by compressing years worth of knowledge and experience into a laser-focused course of 14 lessons.
The lessons in this book assume a few things about you, such as:
- You know your way around basic Python.
- You know your way around basic NumPy.
- You know your way around basic scikit-learn.
For some bonus points, perhaps some of the below points apply to you (don’t panic if they don’t).
- You may know how to work through a predictive modeling problem.
- You may know a little bit of deep learning.
- You may know a little bit of Keras.
This guide was written in the top-down and results-first machine learning style that you’re used to from Machine Learning Mastery.
This book is not a panacea
…so what will YOU know after reading it?
This book will teach you how to get results as a machine learning practitioner interested in using LSTMs on your project.
After reading and working through this book, you will know:
- What LSTMs are.
- Why LSTMs are important.
- How LSTMs work.
- How to develop a suite of LSTM architectures.
- How to get the most out of your LSTM models.
This book will NOT teach you how to be a research scientist and all the theory behind why LSTMs work. For that, I would recommend good research papers and textbooks. See the Further Reading section at the end of the first lesson for a good starting point.
Exactly What You Need to Know
…14 carefully designed lessons to take you from Beginner to Practitioner
This book was designed to be a 14-day crash course into LSTMs for machine learning practitioners.
There are a lot of things you could learn about LSTMs, from theory to applications to Keras API. My goal is to take you straight to getting results with LSTMs in Keras with 14 laser-focused lessons.
I designed the lessons to focus on the LSTM models and their implementation in the Keras deep learning library. They give you the tools to both rapidly understand each model and apply them to your own sequence prediction problems.
Each of the 14 lessons are designed to take you about one hour to read through and complete, excluding the extensions and further reading.
You can choose to work through the lessons one per day, one per week, or at your own pace. I think momentum is critically important, and this book was intended to be read and used, not to sit idle. I would recommend picking a schedule and sticking to it.
The lessons are divided into three parts:
- Part 1: Foundations. The lessons in this section are designed to give you an understanding of how LSTMs work, how to prepare data, and the life-cycle of LSTM models in the Keras library.
- Part 2: Models. The lessons in this section are designed to teach you about the different types of LSTM architectures and how to implement them in Keras.
- Part 3: Advanced. The lessons in this section are designed to teach you how to get the most from your LSTM models.
You can see that these parts provide a theme for the lessons with focus on the different types of LSTM models.
Here is an overview of the 14 step-by-step tutorial lessons you will complete:
Each lesson was designed to be completed in about 30-to-60 minutes by the average developer.
Part I. Foundations
- Lesson 01: What are LSTMs.
- Lesson 02: How to Train LSTMs.
- Lesson 03: How to Prepare Data for LSTMs.
- Lesson 04: How to Develop LSTMs in Keras.
- Lesson 05: Models for Sequence Prediction.
Part II. Models
- Lesson 06: How to Develop Vanilla LSTMs.
- Lesson 07: How to Develop Stacked LSTMs.
- Lesson 08: How to Develop CNN LSTMs.
- Lesson 09: How to Develop Encoder-Decoder LSTMs.
- Lesson 10: How to Develop Bidirectional LSTMs.
- Lesson 11: How to Develop Generative LSTMs.
Part III. Advanced
- Lesson 12: How to Diagnose and Tune LSTMs.
- Lesson 13: How to Make Predictions with LSTMs.
- Lesson 14: How to Update LSTM Models.
You can see that each lesson has a targeted learning outcome. This acts as a filter to ensure you are only focused on the things you need to know to get to a specific result and not get bogged down in the math or near-infinite number of configuration parameters.
These lessons were not designed to teach you everything there is to know about each of the LSTM models. They were designed to give you an understanding of how they work, how to use them on your projects the fastest way I know how: to learn by doing.
Discover 4 Different Sequence Prediction Models
There are 4 main types of sequence prediction models that you need to know.
Each of these model types are presented in the book with code examples showing you how to implement them in Python.
1. One-to-One Model
2. One-to-Many Model
3. Many-to-One Model
4. Many-to-Many Model
Discover 6 Different LSTM Architectures
The LSTM network is the starting point. What you are really interested in is how to use the LSTM to address sequence prediction problems.
The way that the LSTM network is used as layers in sophisticated network architectures. The way that you will get good at applying LSTMs is by knowing about the different useful LSTM networks and how to use them.
The whole middle section of this book focuses on teaching you about the different LSTM architectures.
1. Vanilla LSTM
Memory cells of a single LSTM layer are used in a simple network structure.
2. Stacked LSTM
LSTM layers are stacked one on top of another into deep recurrent neural networks.
3. CNN LSTM
A Convolutional Neural Network is used to learn features in spatial input and the LSTM is used to support a sequence of inputs (e.g. video of images).
4. Encoder-Decoder LSTM
One LSTM network encodes input sequences and a separate LSTM network decodes the encoding into an output sequence.
5. Bidirectional LSTM
Input sequences are presented and learned both forward and backward.
6. Generative LSTM
LSTMs learn the structure relationship in input sequences so well that they can generate new plausible sequences.
Take a Sneak Peek Inside The Ebook
Click image to Enlarge.
BONUS: LSTM RNN Code Recipes
…you also get 45 fully working LSTM scripts
Sample Code Recipes
Each recipe presented in the book is standalone, meaning that you can copy and paste it into your project and use it immediately.
- You get one Python script (.py) for each example provided in the book.
This means that you can follow along and compare your answers to a known working implementation of each example in the provided Python files.
This helps a lot to speed up your progress when working through the details of a specific task.
The provided code was developed in a text editor and intended to be run on the command line. No special IDE or notebooks are required.
All code examples were tested with Python 2 and Python 3 with Keras 2.
All code examples will run on modest and modern computer hardware and were executed on a CPU. No GPUs are required to run the presented examples, although a GPU would make the code run faster.
Python Technical Details
This section provides some technical details about the code provided with the book.
- Python Version: You can use Python 2 or 3.
- SciPy: You will use NumPy, Pandas and scikit-learn.
- Keras: You will need Keras version 2 with either a Theano or TensorFlow backend.
- Operating System: You can use Windows, Linux or Mac OS X.
- Hardware: A standard modern workstation will do, no GPUs required.
- Editor: You can use a text editor and run the example from the command line.
Don’t have a Python environment?
The appendix contains step-by-step tutorials showing you exactly how to setup a Python deep learning environment.
About The Author
Hi, I'm Jason Brownlee.
I live in Australia with my wife and son and love to write and code.
I have a computer science background as well as a Masters and Ph.D. degree in Artificial Intelligence.
I’ve written books on algorithms, won and ranked in the top 10% in machine learning competitions, consulted for startups and spent a long time working on systems for forecasting tropical cyclones. (yes I have written tons of code that runs operationally)
I get a lot of satisfaction helping developers get started and get really good at machine learning.
I teach an unconventional top-down and results-first approach to machine learning where we start by working through tutorials and problems, then later wade into theory as we need it.
I'm here to help if you ever have any questions. I want you to be awesome at machine learning.
Download Your Sample Chapter
Do you want to take a closer look at the book? Download a free sample chapter PDF.
Enter your email address and your sample chapter will be sent to your inbox.
Check Out What Customers Are Saying:
You're Not Alone in Choosing Machine Learning Mastery
Trusted by Over 10,000 Practitioners
...including employees from companies like:
...students and faculty from universities like:
and many thousands more...
Absolutely No Risk with...
100% Money Back Guarantee
Plus, as you should expect of any great product on the market, every Machine Learning Mastery Ebook
comes with the surest sign of confidence: my gold-standard 100% money-back guarantee.
100% Money-Back Guarantee
If you're not happy with your purchase of any of the Machine Learning Mastery Ebooks,
just email me within 90 days of buying, and I'll give you your money back ASAP.
No waiting. No questions asked. No risk.
Bring Deep Learning with LSTM Models to your
Sequence Prediction Problems TODAY!
Choose Your Package:
You will get the Ebook:
- LSTM Networks With Python
(including bonus source code)
(a great deal!)
All prices are in US Dollars (USD).
(1) Click the button. (2) Enter your details. (3) Download immediately.
Secure Payment Processing With SSL Encryption
Are you a Student, Teacher or Retiree?
Do you have any Questions?
What Are Skills in Machine Learning Worth?
Your boss asks you:
Hey, can you build a predictive model for this?
Imagine you had the skills and confidence to say:
...and follow through.
I have been there. It feels great!
How much is that worth to you?
The industry is demanding skills in machine learning.
The market wants people that can deliver results, not write academic papers.
Business knows what these skills are worth and are paying sky-high starting salaries.
A Data Scientists Salary Begins at:
$100,000 to $150,000.
A Machine Learning Engineers Salary is Even Higher.
What Are Your Alternatives?
You made it this far.
You're ready to take action.
But, what are your alternatives? What options are there?
(1) A Theoretical Textbook for $100+
...it's boring, math-heavy and you'll probably never finish it.
(2) An On-site Boot Camp for $10,000+
...it's full of young kids, you must travel and it can take months.
(3) A Higher Degree for $100,000+
...it's expensive, takes years, and you'll be an academic.
For the Hands-On Skills You Get...
And the Speed of Results You See...
And the Low Price You Pay...
Machine Learning Mastery Ebooks are
And they work. That's why I offer the money-back guarantee.
You're A Professional
The field moves quickly,
...how long can you wait?
You think you have all the time in the world, but...
- New methods are devised and algorithms change.
- New books get released and prices increase.
- New graduates come along and jobs get filled.
Right Now is the Best Time to make your start.
Bottom-up is Slow and Frustrating,
...don't you want a faster way?
Can you really go on another day, week or month...
- Scraping ideas and code from incomplete posts.
- Skimming theory and insight from short videos.
- Parsing Greek letters from academic textbooks.
Targeted Training is your Shortest Path to a result.
Professionals Use Training To Stay On Top Of Their Field
Get The Training You Need!
You don't want to fall behind or miss the opportunity.
Frequently Asked Questions
Why doesn't my payment work?
I am sorry to hear that you're having difficulty.
- Perhaps you can double check that your details are correct, just in case of a typo?
- Perhaps you could try a different payment method, such as PayPal or Credit Card?
- Perhaps you could try my alternative secure payment processor, click here?
- Perhaps you're able to talk to your bank, just in case they blocked the transaction?
If you're still having difficulty, please contact me and I can help investigate further.
Can I get your books for free?
Sorry, I don’t give away free copies of my books.
You can access all of my best free material on my blog.
Can I get a hard copy of your book?
Sorry, I don't have hard copies by design.
The books are written for immediate use, rather than references to sit on the shelf.
My students like to have the PDF open on their screen next to their editor so they can copy-paste code.
Also, the books are updated often to reflect changes to APIs. The field is moving very fast.
I hope that helps explain the rationale.
Are there Kindle or ePub versions of the books?
Sorry, just PDF Ebooks.
This is by design and I put a lot of thought into it. My rationale is as follows:
- I use LaTeX to layout the text and code to give a professional look and I am afraid that EBook readers would mess this up.
- The increase in supported formats would create a maintenance headache that would take a large amount of time away from updating the books and working on new books.
- Most critically, reading on an e-reader or iPad is antithetical to the book-open-next-to-code-editor approach the PDF format was chosen to support.
My materials are playbooks intended to be open on the computer, next to a text editor and a command line. They are not reference texts to be read away from the computer.
Will I get free updates to the books?
All updates are free.
Books are usually updated once every month or two to fix bugs, typos and keep abreast of API changes.
Contact me anytime and check if there have been updates. Let me know what version of the book you have (version is listed on the copyright page).
How do I get access to any bonuses?
After you complete your purchase you will receive an email with a link to download your bundle.
The download will include the book or books and any bonus material.
Is there any digital rights management (DRM)?
Can I print the PDF for my personal use?
In what order should I read your books?
My best advice is to pick a topic that most interests you and start there.
Can I get a customized bundle of books?
Sorry, I cannot create custom bundles of books for you, it would create a maintenance nightmare for me. I’m sure you can understand.
You can see the full catalog of my books and bundles here.
Can I get an evaluation copy of your books?
Sorry, I no longer distribute evaluation copies of my books due to some past abuse of the privilege.
If you are a teacher or lecturer, I’m happy to offer you a student discount.
Contact me and ask for the discount.
Can I get an invoice for my purchase?
Email me with the details of your order (order number or email address used to make the purchase) and details you would like to appear on the invoice (your name, company name and address).
I will create a PDF invoice for you and email it back.
How long do books take to ship?
There are no physical books, therefore no shipping is required.
All books are EBooks that you can download immediately after you complete your purchase.
Do you ship to my country?
There are no physical books, therefore no shipping is required.
All books are EBooks that you can download immediately after you complete your purchase.
I support purchases from any country via PayPal or Credit Card.
Can I have a discount?
I do offer a discount to students, teachers, and retirees.
Note: I only offer discounts on individual books, not on the bundles. This is because the bundles are already heavily discounted.
If you are a student, teacher or a retiree please contact me and ask for the discount.
Do you have any sales, deals, or coupons?
I generally don't do sales.
If I do have a special, such as around the launch of a new book, I only offer it to past customers and subscribers on my email list.
I do offer book bundles that offer a discount for a collection of related books.
Can I get a refund?
I am sorry to hear that you want a refund.
Please contact me directly with your purchase details (order number or email address used to make the purchase) and I will organize a refund.
Will you help me if I have questions?
Please contact me anytime with questions about machine learning or the books.
One question at a time please.
Also, each book has a final chapter on getting more help and further reading and points to resources that you can use to get more help.
Do I need to be a good programmer?
Not at all.
My material requires that you have a programmers mindset of thinking in procedures and learning by doing.
You do not need to be an excellent programmer to read and learn about machine learning algorithms.
How much math do I need to know?
No background in statistics, probability or linear algebra is required.
I teach using a top-down and results-first approach to machine learning. You will learn by doing, not learn by theory.
There are no derivations.
Any questions presented are explained in full and are only provided to make the explanation clearer, not more confusing.
How much machine learning do I need to know?
Only a little.
If you are a reader of my blog posts, then you know enough to get started.
I do my best to lead you through what you need to know, step-by-step.
How long will the book take me to complete?
I recommend reading one chapter per day.
Some students finish the book in a weekend.
Most students finish the book in a few weeks by working through it during nights and weekends.
How are your books different to other books?
My books are playbooks. Not textbooks.
They have no deep explanations of theory, just working examples that are laser-focused on the information that you need to know to bring machine learning to your project.
My books are not for everyone, they are carefully designed for practitioners that need to get results, fast.
How are your books different from the blog?
The books are a concentrated and more convenient version of what I put on the blog.
I design my books to be a combination of lessons and projects to teach you how to use a specific machine learning tool or library and then apply it to real predictive modeling problems.
The books get updated with bug fixes, updates for API changes and the addition of new chapters, and these updates are totally free.
I do put some of the book chapters on the blog as examples, but they are not tied to the surrounding chapters or the narrative that a book offers and do not offer the standalone code files.
With each book, you also get all of the source code files used in the book that you can use as recipes to jump-start your own predictive modeling problems.
How are the 2 algorithms books different?
The book “Master Machine Learning Algorithms” is for programmers and non-programmers alike that learn through worked examples. It teaches you how 10 top machine learning algorithms work, with worked examples in arithmetic, not code (and spreadsheets) that show how each model learns and makes predictions.
The book “Machine Learning Algorithms From Scratch” is for programmers that learn by writing code to understand. It provides step-by-step tutorials on how to implement top algorithms as well as how to load data, evaluate models and more. It has less on how the algorithms work, instead focusing exclusively on how to implement each in code.
The two books can support each other.
Is there a team or company-wide license?
Due to abuse of the privilege, I only support purchases by individuals.
Is there a license for libraries?
Sorry, I only support purchases by individuals.
Do you have videos?
I only have tutorial lessons and projects in text format.
This is by design. I used to have video content and I found the completion rate much lower.
I want you to put the material into practice. I have found that text-based tutorials are the best way of achieving this.
After reading and working through the tutorials you are far more likely to apply what you have learned.
What operating systems are supported?
Linux, Mac OS X and Windows.
Can you be my mentor or coach?
Thanks for asking. I would love to help, but I just don't have the capacity.
I try to help as many people as possible through my blog and books.
Can I purchase from Amazon (or elsewhere)?
My books can only be purchased from my website.
The reason is that I am a small business and I want a direct relationship with you, my customer, so that I can offer personal support and send out updates about your book and new stuff I am working on.
I hope you can understand my rationale.
What if my download link expires?
It is possible that your link to download your purchase will expire after a few days.
This is a security precaution.
Please contact me and I will resend you purchase receipt with an updated download link.
Can I use your code in my own project?
But, understand that all code was developed and provided for educational purposes only and that I take no responsibility for it, what it might do or how you might use it.
Do you have another question?