How to Develop VGG, Inception and ResNet Modules from Scratch in Keras

Last Updated on

There are discrete architectural elements from milestone models that you can use in the design of your own convolutional neural networks.

Specifically, models that have achieved state-of-the-art results for tasks like image classification use discrete architecture elements repeated multiple times, such as the VGG block in the VGG models, the inception module in the GoogLeNet, and the residual module in the ResNet.

Once you able to implement parameterized versions of these architecture elements, you can use them in the design of your own models for computer vision and other applications.

In this tutorial, you will discover how to implement the key architecture elements from milestone convolutional neural network models, from scratch.

After completing this tutorial, you will know:

  • How to implement a VGG module used in the VGG-16 and VGG-19 convolutional neural network models.
  • How to implement the naive and optimized inception module used in the GoogLeNet model.
  • How to implement the identity residual module used in the ResNet model.

Discover how to build models for photo classification, object detection, face recognition, and more in my new computer vision book, with 30 step-by-step tutorials and full source code.

Let’s get started.

How to Implement Major Architecture Innovations for Convolutional Neural Networks

How to Implement Major Architecture Innovations for Convolutional Neural Networks
Photo by daveynin, some rights reserved.

Tutorial Overview

This tutorial is divided into three parts; they are:

  1. How to implement VGG Blocks
  2. How to implement the Inception Module
  3. How to implement the Residual Module

Want Results with Deep Learning for Computer Vision?

Take my free 7-day email crash course now (with sample code).

Click to sign-up and also get a free PDF Ebook version of the course.

Download Your FREE Mini-Course

How to Implement VGG Blocks

The VGG convolutional neural network architecture, named for the Visual Geometry Group at Oxford, was an important milestone in the use of deep learning methods for computer vision.

The architecture was described in the 2014 paper titled “Very Deep Convolutional Networks for Large-Scale Image Recognition” by Karen Simonyan and Andrew Zisserman and achieved top results in the LSVRC-2014 computer vision competition.

The key innovation in this architecture was the definition and repetition of what we will refer to as VGG-blocks. These are groups of convolutional layers that use small filters (e.g. 3×3 pixels) followed by a max pooling layer.

The image is passed through a stack of convolutional (conv.) layers, where we use filters with a very small receptive field: 3 x 3 (which is the smallest size to capture the notion of left/right, up/down, center). […] Max-pooling is performed over a 2 x 2 pixel window, with stride 2.

Very Deep Convolutional Networks for Large-Scale Image Recognition, 2014.

A convolutional neural network with VGG-blocks is a sensible starting point when developing a new model from scratch as it is easy to understand, easy to implement, and very effective at extracting features from images.

We can generalize the specification of a VGG-block as one or more convolutional layers with the same number of filters and a filter size of 3×3, a stride of 1×1, same padding so the output size is the same as the input size for each filter, and the use of a rectified linear activation function. These layers are then followed by a max pooling layer with a size of 2×2 and a stride of the same dimensions.

We can define a function to create a VGG-block using the Keras functional API with a given number of convolutional layers and with a given number of filters per layer.

To use the function, one would pass in the layer prior to the block and receive the layer for the end of the block that can be used to integrate into the model.

For example, the first layer might be an input layer which could be passed into the function as an argument. The function then returns a reference to the final layer in the block, the pooling layer, that could be connected to a flatten layer and subsequent dense layers for making a classification prediction.

We can demonstrate how to use this function by defining a small model that expects square color images as input and adds a single VGG block to the model with two convolutional layers, each with 64 filters.

Running the example creates the model and summarizes the structure.

We can see that, as intended, the model added a single VGG block with two convolutional layers each with 64 filters, followed by a max pooling layer.

A plot is also created of the model architecture that may help to make the model layout more concrete.

Note, creating the plot assumes that you have pydot and pygraphviz installed. If this is not the case, you can comment out the import statement and call to the plot_model() function in the example.

Plot of Convolutional Neural Network Architecture With a VGG Block

Plot of Convolutional Neural Network Architecture With a VGG Block

Using VGG blocks in your own models should be common because they are so simple and effective.

We can expand the example and demonstrate a single model that has three VGG blocks, the first two blocks have two convolutional layers with 64 and 128 filters respectively, the third block has four convolutional layers with 256 filters. This is a common usage of VGG blocks where the number of filters is increased with the depth of the model.

The complete code listing is provided below.

Again, running the example summarizes the model architecture and we can clearly see the pattern of VGG blocks.

A plot of the model architecture is created providing a different perspective on the same linear progression of layers.

Plot of Convolutional Neural Network Architecture With Multiple VGG Blocks

Plot of Convolutional Neural Network Architecture With Multiple VGG Blocks

How to Implement the Inception Module

The inception module was described and used in the GoogLeNet model in the 2015 paper by Christian Szegedy, et al. titled “Going Deeper with Convolutions.”

Like the VGG model, the GoogLeNet model achieved top results in the 2014 version of the ILSVRC challenge.

The key innovation on the inception model is called the inception module. This is a block of parallel convolutional layers with different sized filters (e.g. 1×1, 3×3, 5×5) and a and 3×3 max pooling layer, the results of which are then concatenated.

In order to avoid patch-alignment issues, current incarnations of the Inception architecture are restricted to filter sizes 1×1, 3×3 and 5×5; this decision was based more on convenience rather than necessity. […] Additionally, since pooling operations have been essential for the success of current convolutional networks, it suggests that adding an alternative parallel pooling path in each such stage should have additional beneficial effect, too

Going Deeper with Convolutions, 2015.

This is a very simple and powerful architectural unit that allows the model to learn not only parallel filters of the same size, but parallel filters of differing sizes, allowing learning at multiple scales.

We can implement an inception module directly using the Keras functional API. The function below will create a single inception module with a fixed number of filters for each of the parallel convolutional layers. From the GoogLeNet architecture described in the paper, it does not appear to use a systematic sizing of filters for parallel convolutional layers as the model is highly optimized. As such, we can parameterize the module definition so that we can specify the number of filters to use in each of the 1×1, 3×3, and 5×5 filters.

To use the function, provide the reference to the prior layer as input, the number of filters, and it will return a reference to the concatenated filters layer that you can then connect to more inception modules or a submodel for making a prediction.

We can demonstrate how to use this function by creating a model with a single inception module. In this case, the number of filters is based on “inception (3a)” from Table 1 in the paper.

The complete example is listed below.

Running the example creates the model and summarizes the layers.

We know the convolutional and pooling layers are parallel, but this summary does not capture the structure easily.

A plot of the model architecture is also created that helps to clearly see the parallel structure of the module as well as the matching shapes of the output of each element of the module that allows their direct concatenation by the third dimension (filters or channels).

Plot of Convolutional Neural Network Architecture With a Naive Inception Module

Plot of Convolutional Neural Network Architecture With a Naive Inception Module

The version of the inception module that we have implemented is called the naive inception module.

A modification to the module was made in order to reduce the amount of computation required. Specifically, 1×1 convolutional layers were added to reduce the number of filters before the 3×3 and 5×5 convolutional layers, and to increase the number of filters after the pooling layer.

This leads to the second idea of the Inception architecture: judiciously reducing dimension wherever the computational requirements would increase too much otherwise. […] That is, 1×1 convolutions are used to compute reductions before the expensive 3×3 and 5×5 convolutions. Besides being used as reductions, they also include the use of rectified linear activation making them dual-purpose

Going Deeper with Convolutions, 2015.

If you intend to use many inception modules in your model, you may require this computational performance-based modification.

The function below implements this optimization improvement with parameterization so that you can control the amount of reduction in the number of filters prior to the 3×3 and 5×5 convolutional layers and the number of increased filters after max pooling.

We can create a model with two of these optimized inception modules to get a concrete idea of how the architecture looks in practice.

In this case, the number of filter configurations are based on “inception (3a)” and “inception (3b)” from Table 1 in the paper.

The complete example is listed below.

Running the example creates a linear summary of the layers that does not really help to understand what is going on.

A plot of the model architecture is created that does make the layout of each module clear and how the first model feeds the second module.

Note that the first 1×1 convolution in each inception module is on the far right for space reasons, but besides that, the other layers are organized left to right within each module.

Plot of Convolutional Neural Network Architecture With a Efficient Inception Module

Plot of Convolutional Neural Network Architecture With a Efficient Inception Module

How to Implement the Residual Module

The Residual Network, or ResNet, architecture for convolutional neural networks was proposed by Kaiming He, et al. in their 2016 paper titled “Deep Residual Learning for Image Recognition,” which achieved success on the 2015 version of the ILSVRC challenge.

A key innovation in the ResNet was the residual module. The residual module, specifically the identity residual model, is a block of two convolutional layers with the same number of filters and a small filter size where the output of the second layer is added with the input to the first convolutional layer. Drawn as a graph, the input to the module is added to the output of the module and is called a shortcut connection.

We can implement this directly in Keras using the functional API and the add() merge function.

A limitation with this direct implementation is that if the number of filters in the input layer does not match the number of filters in the last convolutional layer of the module (defined by n_filters), then we will get an error.

One solution is to use a 1×1 convolution layer, often referred to as a projection layer, to either increase the number of filters for the input layer or reduce the number of filters for the last convolutional layer in the module. The former solution makes more sense, and is the approach proposed in the paper, referred to as a projection shortcut.

When the dimensions increase […], we consider two options: (A) The shortcut still performs identity mapping, with extra zero entries padded for increasing dimensions. This option introduces no extra parameter; (B) The projection shortcut […] is used to match dimensions (done by 1×1 convolutions).

Deep Residual Learning for Image Recognition, 2015.

Below is an updated version of the function that will use the identity if possible, otherwise a projection of the number of filters in the input does not match the n_filters argument.

We can demonstrate the usage of this module in a simple model.

The complete example is listed below.

Running the example first creates the model then prints a summary of the layers.

Because the module is linear, this summary is helpful to see what is going on.

A plot of the model architecture is also created.

We can see the module with the inflation of the number of filters in the input and the addition of the two elements at the end of the module.

Plot of Convolutional Neural Network Architecture With an Residual Module

Plot of Convolutional Neural Network Architecture With an Residual Module

The paper describes other types of residual connections such as bottlenecks. These are left as an exercise for the reader that can be implemented easily by updating the residual_module() function.

Further Reading

This section provides more resources on the topic if you are looking to go deeper.

Posts

Papers

API

Summary

In this tutorial, you discovered how to implement key architecture elements from milestone convolutional neural network models, from scratch.

Specifically, you learned:

  • How to implement a VGG module used in the VGG-16 and VGG-19 convolutional neural network models.
  • How to implement the naive and optimized inception module used in the GoogLeNet model.
  • How to implement the identity residual module used in the ResNet model.

Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.


Develop Deep Learning Models for Vision Today!

Deep Learning for Computer Vision

Develop Your Own Vision Models in Minutes

…with just a few lines of python code

Discover how in my new Ebook:
Deep Learning for Computer Vision

It provides self-study tutorials on topics like: classification, object detection (yolo and rcnn), face recognition (vggface and facenet), data preparation and much more…

Finally Bring Deep Learning to your Vision Projects

Skip the Academics. Just Results.

Click to learn more.


27 Responses to How to Develop VGG, Inception and ResNet Modules from Scratch in Keras

  1. Bejoscha April 26, 2019 at 8:06 am #

    I love your code-snippets and practical examples on implementation. These are exactly the time-savers and error-avoiders that a beginner needs to get experimenting. Thanks a lot. I’m still struggling a bit with the concept of the 1×1 convolution filters, though. A post on that topic alone – but in more detail on what the layer actually does and how it works – would be very helpful. (I will need to re-read and try this post here a few times, I guess. Comment written after first read-through.)

    • Jason Brownlee April 26, 2019 at 8:41 am #

      Thanks.

      I hope to have a post dedicated to the topic soon.

  2. William April 26, 2019 at 9:58 pm #

    Thank you for your effort. I learn a lot from you.

  3. Abkul April 26, 2019 at 10:12 pm #

    Thanks Jason,

    Your blogs and the explanations are always very useful and distilled.Your books are equally excellent.

    I have a lot of interest in combination of CNN and LSTM having read all your blogs, purchased time series and the latest book, particularly in handling temporal sequential problems concerning acquiring images from a plant for the diagnosis of disease development over time.

    Would appreciate if a blog on combination of this two models is done for clarity with a unique example.Your advice is more than welcome.

    • Jason Brownlee April 27, 2019 at 6:30 am #

      Thanks for the suggestion.

      I believe there is an example of processing images over time in the LSTM book. I hope to cover more examples in the future.

  4. Nassimleking May 21, 2019 at 11:25 am #

    I suppose that in the last section there is an error , when using merge_input the conv1 must be attached not to layer_in but to the merge_input , wasn’t it ?

    • Jason Brownlee May 21, 2019 at 2:43 pm #

      I don’t believe there’s an error.

      Which example are you referring to exactly?

      • Nassimleking May 21, 2019 at 11:33 pm #

        The last code in the 17th line .

        • Jason Brownlee May 22, 2019 at 8:07 am #

          You’re right, thanks! Fixed.

          Update, no, that is not an error.

          The 1×1 projection is only needed of the number of channels for the input does not match the number of channels in the output.

  5. Hamed June 30, 2019 at 9:37 am #

    Hi Jason! This is a great tutorial on VGG and inception. Thank you for that. I have a question, though. This way you developed your codes in this tutorial is kind of new to me. In your previous posts, you use sequential models and then you use ‘model.add’. Is this a specific way to develop VGG or we can still use that format (e.g. ‘model.add’) to develop these architectures? Thank you in advance!

  6. James July 2, 2019 at 6:39 pm #

    Thanks Jason for another great tutorial as always!
    Many models from Keras application requires 224×224 images.
    Is there a way to use bigger image with the pre-trained models without resizing?
    If not, is there a way to at least use their architectures without their weights, and train from scratch?
    Thanks!

  7. Jairo Junior Rangel Redondo July 4, 2019 at 2:29 am #

    Amazing Tutorials!!!

  8. Hamed July 28, 2019 at 10:16 am #

    Hi Jason,

    We added 1×1 convolutional layer prior to 3×3 and 5×5 layers in the projected inception module in order to reduce the amount of computation. We can, however, easily see that for example the number of parameters has increased to 110720 from 3584 for 128 filters of size 3×3 in the naive inception module. I wonder why it increased while we intended to reduce it! And would you mind if you give me a hint about how it calculates the number of parameters? Much obligied!

    • Jason Brownlee July 29, 2019 at 6:04 am #

      Nice observation. We are trading off space complexity for time complexity. E.g. less compute but more weights.

      • Hamed July 29, 2019 at 3:45 pm #

        Thank you! But I guess this way we’re putting the burden on memory rather than on GPU. Like I got OOM when I wanted to use this module for sequential data i.e. replacing CONV2D by CONVLSTM2D because I got >500M parameters! While with VGA I got 40M which was calculable by my RAM (16Gb)

        • Hamed July 29, 2019 at 3:48 pm #

          And I don’t know why but for some reasons when I used only one block, the number of parameters became negative!!

  9. Keshav Rawat August 21, 2019 at 10:34 pm #

    Your stuff is very helpful for the newcomers. Thank you!!

Leave a Reply