Archive | Better Deep Learning

A Gentle Introduction to Dropout for Regularizing Deep Neural Networks

A Gentle Introduction to Dropout for Regularizing Deep Neural Networks

Deep learning neural networks are likely to quickly overfit a training dataset with few examples. Ensembles of neural networks with different model configurations are known to reduce overfitting, but require the additional computational expense of training and maintaining multiple models. A single model can be used to simulate having a large number of different network […]

Continue Reading 8
Scatter Plot of Circles Dataset with Color Showing the Class Value of Each Sample

How to Reduce Generalization Error in Deep Neural Networks With Activity Regularization in Keras

Activity regularization provides an approach to encourage a neural network to learn sparse features or internal representations of raw observations. It is common to seek sparse learned representations in autoencoders, called sparse autoencoders, and in encoder-decoder models, although the approach can also be used generally to reduce overfitting and improve a model’s ability to generalize […]

Continue Reading 6
Activation Regularization for Reducing Generalization Error in Deep Learning Neural Networks

Activation Regularization for Reducing Generalization Error in Deep Learning Neural Networks

Deep learning models are capable of automatically learning a rich internal representation from raw input data. This is called feature or representation learning. Better learned representations, in turn, can lead to better insights into the domain, e.g. via visualization of learned features, and to better predictive models that make use of the learned features. A […]

Continue Reading 0
Scatter Plot of Moons Dataset With Color Showing the Class Value of Each Sample

How to Reduce Overfitting in Deep Neural Networks Using Weight Constraints in Keras

Weight constraints provide an approach to reduce the overfitting of a deep learning neural network model on the training data and improve the performance of the model on new data, such as the holdout test set. There are multiple types of weight constraints, such as maximum and unit vector norms, and some require a hyperparameter […]

Continue Reading 18
A Gentle Introduction to Weight Constraints to Reduce Generalization Error in Deep Learning

A Gentle Introduction to Weight Constraints to Reduce Generalization Error in Deep Learning

Weight regularization methods like weight decay introduce a penalty to the loss function when training a neural network to encourage the network to use small weights. Smaller weights in a neural network can result in a model that is more stable and less likely to overfit the training dataset, in turn having better performance when […]

Continue Reading 6
How to Configure the Number of Layers and Nodes in a Neural Network

How to Configure the Number of Layers and Nodes in a Neural Network

Artificial neural networks have two main hyperparameters that control the architecture or topology of the network: the number of layers and the number of nodes in each hidden layer. You must specify values for these parameters when configuring your network. The most reliable way to configure these hyperparameters for your specific predictive modeling problem is […]

Continue Reading 8
Comparison of Adam to Other Optimization Algorithms Training a Multilayer Perceptron

Gentle Introduction to the Adam Optimization Algorithm for Deep Learning

The choice of optimization algorithm for your deep learning model can mean the difference between good results in minutes, hours, and days. The Adam optimization algorithm is an extension to stochastic gradient descent that has recently seen broader adoption for deep learning applications in computer vision and natural language processing. In this post, you will […]

Continue Reading 47