Line Plots of Classification Accuracy on Train and Test Datasets With Different Batch Sizes

How to Control the Speed and Stability of Training Neural Networks With Gradient Descent Batch Size

Neural networks are trained using gradient descent where the estimate of the error used to update the weights is calculated based on a subset of the training dataset. The number of examples from the training dataset used in the estimate of the error gradient is called the batch size and is an important hyperparameter that […]

Continue Reading 0
Line Plot Classification Accuracy of MLP With Batch Normalization After Activation Function on Train and Test Datasets Over Training Epochs

How to Accelerate Learning of Deep Neural Networks With Batch Normalization

Batch normalization is a technique designed to automatically standardize the inputs to a layer in a deep learning neural network. Once implemented, batch normalization has the effect of dramatically accelerating the training process of a neural network, and in some cases improves the performance of the model via a modest regularization effect. In this tutorial, […]

Continue Reading 3
Overview of Course Structure

Practical Deep Learning for Coders (Review)

Practical deep learning is a challenging subject in which to get started. It is often taught in a bottom-up manner, requiring that you first get familiar with linear algebra, calculus, and mathematical optimization before eventually learning the neural network techniques. This can take years, and most of the background theory will not help you to […]

Continue Reading 4
Line Plot of Train and Test Set Accuracy of Over Training Epochs for Deep MLP with ReLU with 15 Hidden Layers

How to Fix Vanishing Gradients Using the Rectified Linear Activation Function

The vanishing gradients problem is one example of unstable behavior that you may encounter when training a deep neural network. It describes the situation where a deep multilayer feed-forward network or a recurrent neural network is unable to propagate useful gradient information from the output end of the model back to the layers near the […]

Continue Reading 0
Line Plot of Rectified Linear Activation for Negative and Positive Inputs

A Gentle Introduction to the Rectified Linear Activation Function for Deep Learning Neural Networks

In a neural network, the activation function is responsible for transforming the summed weighted input from the node into the activation of the node or output for that input. The rectified linear activation function is a piecewise linear function that will output the input directly if is positive, otherwise, it will output zero. It has […]

Continue Reading 8
Line Plot of Cosine Annealing Learning Rate Schedule

How to Develop a Snapshot Ensemble Deep Learning Neural Network in Python With Keras

Model ensembles can achieve lower generalization error than single models but are challenging to develop with deep learning neural networks given the computational cost of training each single model. An alternative is to train multiple model snapshots during a single training run and combine their predictions to make an ensemble prediction. A limitation of this […]

Continue Reading 6
Four Scatter Plots of the Circles Dataset Varied by the Amount of Statistical Noise

Impact of Dataset Size on Deep Learning Model Skill And Performance Estimates

Supervised learning is challenging, although the depths of this challenge are often learned then forgotten or willfully ignored. This must be the case, because dwelling too long on this challenge may result in a pessimistic outlook. In spite of the challenge, we continue to wield supervised learning algorithms and they perform well in practice. Fundamental […]

Continue Reading 4