Developing neural networks is often referred to as a dark art.
The reason for this is that being skilled at developing neural network models comes from experience. There are no reliable methods to analytically calculate how to design a “good” or “best” model for your specific dataset. You must draw on experience and experiment in order to discover what works on your problem.
A lot of this experience can come from actually developing neural networks on test problems.
Nevertheless, many people have come before and recorded their discoveries, best practices, and preferred techniques. You can learn a lot about how to design and configure neural networks from some of the best books on the topic.
In this post, you will discover the three books that I recommend reading and having next to you when developing neural networks for your datasets.
Kick-start your project with my new book Better Deep Learning, including step-by-step tutorials and the Python source code files for all examples.
Let’s get started.
Three Recommended Books on Neural Networks
There are three books that I think you must own physical copies of if you are a neural network practitioner.
They are:
- Neural Networks for Pattern Recognition, 1995.
- Neural Smithing: Supervised Learning in Feedforward Artificial Neural Networks, 1999.
- Deep Learning, 2016.
These books are references, not tutorials.
You dip into them again and again before and during projects to ensure that you are getting everything you can out of your data and models.
These are the books that I read and reference all the time. If you have books that you recommend when developing neural network models, please let me know in the comments below.
Now, let’s take a closer look at each book in turn.
Neural Networks for Pattern Recognition
Neural Networks for Pattern Recognition by Christopher Bishop was released in 1995.
This great book was followed about a decade later by the still classic textbook Pattern Recognition and Machine Learning (fondly referred to as PRML). Christopher Bishop is both a professor at the University of Edinburgh and a director at Microsoft’s Cambridge research lab.
This book is a classic in the field of neural networks. It is a handbook that handily captures both the state of theory at the time, and techniques that remain just as relevant today nearly 25 years later.
Although reading the book cover to cover will provide you a robust foundation, I’d instead encourage you to use it as a reference for getting the most out of your neural network models.
I’d recommend dipping into the following chapters as needed:
- Chapter 7: Parameter Optimization Algorithms
- Chapter 8: Pre-Processing and Feature Extraction
- Chapter 9: Learning and Generalization.
Chapter 9 is worth the sticker price for the book alone, giving a laundry list of descriptions for regularization methods and ensemble methods you should be testing.
I recommend this book because given the description of new methods almost daily, practitioners often forget the tried and true basics.
I don’t think this book is in print anymore, but you can find secondhand and international versions everywhere online.
Want Better Results with Deep Learning?
Take my free 7-day email crash course now (with sample code).
Click to sign-up and also get a free PDF Ebook version of the course.
Neural Smithing: Supervised Learning in Feedforward Artificial Neural Networks
Neural Smithing: Supervised Learning in Feedforward Artificial Neural Networks by Russell Reed and Robert Marks was released in 1999.
I have a large soft spot for this book.
I purchased it soon after it was released and used it as a reference for many of my own implementations of neural network algorithms through the 2000s.
There are two things I like the most about this book:
- Code
- Plots
The book uses mathematics and descriptions to explain concepts, but importantly they also use snippets of pseudocode or ANSI C to show how things work. This is invaluable the first time you’re coding backpropagation of error or an activation functions.
The book also uses plots of the decision surface models. This is invaluable to understand what the models are doing/seeing during training under different learning algorithms and how things like regularization effect the model.
There is perhaps an over focus on pruning methods given the authors interest in the area; nevertheless, I’d recommend dipping into the following chapters when developing your own models:
- Chapter 14: Factors Influencing Generalization
- Chapter 15: Generalization Prediction and Assessment
- Chapter 16: Heuristics for Improving Generalization
- Chapter 17: Effects of Training with Noisy Inputs
Although I recommend buying this book and having it next to you (always), Robert Marks has a reprint of the book on his website in HTML format:
Deep Learning
Deep Learning by Ian Goodfellow, Yoshua Bengio, and Aaron Courville was released in 2016.
This is the missing bridge between the classic books of the 1990s and modern deep learning.
Importantly, neural networks are introduced with careful mention of the innovations and milestones that have made the field into what it is today. Specifically Chapter 6: Deep Feedforward Networks and Section 6.6 Historical Notes.
There are three chapters that are must-reads for neural network practitioners; they are:
- Chapter 7: Regularization for Deep Learning
- Chapter 8: Optimization for Training Deep Models
- Chapter 11: Practical Methodology
Chapter 11 especially is important as it ties together specific methods and how and when to use them in practice. It is by far worth the purchase price of the book alone.
This is a must have. You need a physical copy of this book. Nevertheless, the entire text is available on the books website here:
Further Reading
This section provides more resources on the topic if you are looking to go deeper.
Books
- Neural Networks for Pattern Recognition, 1995.
- Neural Smithing: Supervised Learning in Feedforward Artificial Neural Networks, 1999.
- Deep Learning, 2016.
Additional Books
Additional Links
Summary
In this post, you discovered the three reference books that I think that every neural network practitioner must own.
Do you use one or more of these books yourself?
What chapters do you reference heavily?
Are there other books that you reference a lot?
Let me know below.
Hi Jason,
Thanks for the recommendations. What books would you recommend for more practical deep learning (ie. A little less theory and more coding with real data)?
Thanks for everything,
David
There are few, perhaps start here:
https://machinelearningmastery.com/start-here/#deeplearning
I feel more motivated to continue reading the Deep Learning book. At chapter 5 right now but will jump to chapter 7, 8 and 11 as Jason said it is a must read chapter.
Thanks Jason as always.
I’m glad it helped.
As always Jason, thanks for your posts.
I hope the books are helpful.
Why do you say that we need physical copies of these books? What extra value do you see in the physical copies over an ebook version?
I like both ebooks and physical copies for different reasons, but I find physical copies of technical books invaluable. It is something about knowing where in the physical book a topic is covered and being able to turn to it – to know which page and where on the page. To hold it.
Only for reference information though. I don’t see the need for tutorial based material.
Sir do your bit for the eniornment by selecting ebooks as far as possible. Most Humbly,
Thanks for your suggestion.
Really valuable information… thanks for the posts…
You’re welcome, I’m glad they help.
What about the book (if it happens for you to know) “Neural and adaptive systems”, by Jose C. Principe, Neil R. Euliano, and W. Curt Lefebvre? Do you consider it as another one worthy to be in the list?
I would like to have your opinion.
Neural and Adaptive Systems: Fundamentals through Simulations
https://amzn.to/2XzNCwJ
I’ve not read it, sorry.
Really valuable information… thanks for the posts…,What about the Haykin Neural Networks & Learning Machines.Please give your valuable opinion.
Sorry, I don’t think I know it.
I’m curious why you recommend Bishop’s 1995 book more strongly than his 2006 book? It seems like his 2006 book is more well-known
Yes, but the 1995 book is 100% focused on neural nets, and is an excellent read still! The 2006 book is more broad on ML.