Neural Network with More Hidden Neurons

Last Updated on January 10, 2023

The traditional model of neural network is called multilayer perceptrons. They are usually made up of a series of interconnected layers. The input layer is where the data enters the network, and the output layer is where the network delivers the output.

The input layer is usually connected to one or more hidden layers, which modify and process the data before it reaches the output layer. The hidden layers are what make neural networks so powerful: they can learn complicated functions that might be difficult for a programmer to specify in code.

In the previous tutorial, we built a neural network with only a couple of hidden neurons. Here, you will implement a neural network by adding more hidden neurons to it. This will estimate more complex function for us in order to fit the data.

During the implementation process, you’ll learn:

  • How to build a neural network with more hidden neurons in PyTorch.
  • How to estimate complex functions using neural networks by adding more hidden neurons to the network.
  • How to train a neural network in PyTorch.

Let’s get started.

Neural Network with More Hidden Neurons.
Picture by Kdwk Leung. Some rights reserved.

Overview

This tutorial is in three parts; they are

  • Preparing the Data
  • Build the Model Architecture
  • Train the Model

Preparing the Data

Let’s build a Data class that extends the Dataset class from PyTorch. You use it to create a dataset of 100 synthetic values ranging from $-50$ to $50$. The x tensor stores the values in the specified range, while the y tensor is a corresponding tensor of zeros with the same shape as x.

Next, you use a for loop to set the values in x and y tensors based on the values in x. If a value in x is between $-20$ and $20$, the corresponding value in y is set to 1 and if a value in x is between $-30$ and $-20$ or between $20$ and $30$, the corresponding value in y is set to 0. Similarly, If a value in x is between $-40$ and $-30$ or between $30$ and $40$, the corresponding value in y is set to 1. Otherwise, the corresponding value in y is set to 0.

In the Data class, the __getitem__() method has been used to retrieve the x and y values at a specified index in the dataset. The __len__() method returns the length of the dataset. With these, you can obtain a sample from the dataset using data[i] and tell the size of the dataset using len(data). This class can be used to create a data object that can be passed to a PyTorch data loader to train a machine learning model.

Note that you are building this complex data object to see how well our neural network with more hidden neurons estimates the function. Here is how the code of the data object will look like.

Let’s instantiate a data object.

And, write a function to visualize this data, which will also be useful when you train the model later.

If you run this function, you can see the data looks like the following:

Build the Model Architecture

Below, you will define a NeuralNetwork class to build a custom model architecture using nn.Module from PyTorch. This class represents a simple neural network with an input layer, a hidden layer, and an output layer.

The __init__() method is used to initialize the neural network by defining the layers in the network. The forward method is used to define the forward pass through the network. In this case, a sigmoid activation function is applied to the output of both input and output layers. This means that the output of the network will be a value between 0 and 1.

Finally, you will create an instance of the NeuralNetwork class and store it in the model variable. The model is initialized with an input layer having 1 input neuron, a hidden layer having 15 hidden neurons, and an output layer having 1 output neuron. This model is now ready to be trained on some data.

Train the Model

Let’s define the criterion, optimizer, and dataloader. You should use binary cross entropy loss as the dataset is a classification with two classes. Adam optimizer is used, with a batch size of 32. The learning rate is set to 0.01 which determines how model weights are updated during training. The loss function is used to evaluate the model performance, the optimizer updates the weights, and the data loader divides the data into batches for efficient processing.

Now, let’s build a training loop for 7000 epochs and visualize the results during training. You’ll see how well our model estimates the data points as the training progresses.

When you run this loop, you will see at the first epoch, the neural network modelled the dataset poorly, like the following:

But the accuracy improved as the training progressed. After the training loop completed, we can see the result as the neural network modelled the data like the following:

and the corresponding history of loss metric can be plot like the following:

As you can see, our model estimated the function quite well but not perfect. The input of range 20 to 40, for example, isn’t predicted right. You may try to expand the network to add one more layer, such as the following, and see if it will make any difference.

Putting everything together, the following is the complete code:

Summary

In this tutorial, you learned how we estimate complex functions by introducing more neurons into the neural networks. Particularly, you learned:

  • How to build a neural network with more hidden neurons in PyTorch.
  • How to estimate complex functions using neural networks by adding more hidden neurons to the network.
  • How to train a neural network in PyTorch.

One Response to Neural Network with More Hidden Neurons

  1. Mohammed January 11, 2023 at 6:56 am #

    How to choose the number of hidden layers?

Leave a Reply