Looking at all of the very large convolutional neural networks such as ResNets, VGGs, and the like, it begs the question on how we can make all of these networks smaller with less parameters while still maintaining the same level of accuracy or even improving generalization of the model using a smaller amount of parameters. One approach is depthwise separable convolutions, also known by separable convolutions in TensorFlow and Pytorch (not to be confused with spatially separable convolutions which are also referred to as separable convolutions). Depthwise separable convolutions were introduced by Sifre in “Rigid-motion scattering for image classification” and has been adopted by popular model architectures such as MobileNet and a similar version in Xception. It splits the channel and spatial convolutions that are usually combined together in normal convolutional layers

In this tutorial, we’ll be looking at what depthwise separable convolutions are and how we can use them to speed up our convolutional neural network image models.

After completing this tutorial, you will learn:

- What is a depthwise, pointwise, and depthwise separable convolution
- How to implement depthwise separable convolutions in Tensorflow
- Using them as part of our computer vision models

Let’s get started!

## Overview

This tutorial is split into 3 parts:

- What is a depthwise separable convolution
- Why are they useful
- Using depthwise separable convolutions in computer vision model

## What is a Depthwise Separable Convolution

Before diving into depthwise and depthwise separable convolutions, it might be helpful to have a quick recap on convolutions. Convolutions in image processing is a process of applying a kernel over volume, where we do a weighted sum of the pixels with the weights as the values of the kernels. Visually as follows:

Now, let’s introduce a depthwise convolution. A depthwise convolution is basically a convolution along only one spatial dimension of the image. Visually, this is what a single depthwise convolutional filter would look like and do:

The key difference between a normal convolutional layer and a depthwise convolution is that the depthwise convolution applies the convolution along only one spatial dimension (i.e. channel) while a normal convolution is applied across all spatial dimensions/channels at each step.

If we look at what an entire depthwise layer does on all RGB channels,

Notice that since we are applying one convolutional filter for each output channel, the number of output channels is equal to the number of input channels. After applying this depthwise convolutional layer, we then apply a pointwise convolutional layer.

Simply put a pointwise convolutional layer is a regular convolutional layer with a `1x1`

kernel (hence looking at a single point across all the channels). Visually, it looks like this:

## Why are Depthwise Separable Convolutions Useful?

Now, you might be wondering, what’s the use of doing two operations with the depthwise separable convolutions? Given that the title of this article is to speed up computer vision models, how does doing two operations instead of one help to speed things up?

To answer that question, let’s look at the number of parameters in the model (there would be some additional overhead associated with doing two convolutions instead of one though). Let’s say we wanted to apply 64 convolutional filters to our RGB image to have 64 channels in our output. Number of parameters in normal convolutional layer (including bias term) is $ 3 \times 3 \times 3 \times 64 + 64 = 1792$. On the other hand, using a depthwise separable convolutional layer would only have $(3 \times 3 \times 1 \times 3 + 3) + (1 \times 1 \times 3 \times 64 + 64) = 30 + 256 = 286$ parameters, which is a significant reduction, with depthwise separable convolutions having less than 6 times the parameters of the normal convolution.

This can help to reduce the number of computations and parameters, which reduces training/inference time and can help to regularize our model respectively.

Let’s see this in action. For our inputs, let’s use the CIFAR10 image dataset of `32x32x3`

images,

1 2 3 4 5 |
import tensorflow.keras as keras from keras.datasets import mnist # load dataset (trainX, trainY), (testX, testY) = keras.datasets.cifar10.load_data() |

Then, we implement a depthwise separable convolution layer. There’s an implementation in Tensorflow but we’ll go into that in the final example.

1 2 3 4 5 6 7 8 9 |
class DepthwiseSeparableConv2D(keras.layers.Layer): def __init__(self, filters, kernel_size, padding, activation): super(DepthwiseSeparableConv2D, self).__init__() self.depthwise = DepthwiseConv2D(kernel_size = kernel_size, padding = padding, activation = activation) self.pointwise = Conv2D(filters = filters, kernel_size = (1, 1), activation = activation) def call(self, input_tensor): x = self.depthwise(input_tensor) return self.pointwise(x) |

Constructing a model with using a depthwise separable convolutional layer and looking at the number of parameters,

1 2 3 4 |
visible = Input(shape=(32, 32, 3)) depthwise_separable = DepthwiseSeparableConv2D(filters=64, kernel_size=(3,3), padding="valid", activation="relu")(visible) depthwise_model = Model(inputs=visible, outputs=depthwise_separable) depthwise_model.summary() |

which gives the output

1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
_________________________________________________________________ Layer (type) Output Shape Param # ================================================================= input_15 (InputLayer) [(None, 32, 32, 3)] 0 depthwise_separable_conv2d_ (None, 30, 30, 64) 286 11 (DepthwiseSeparableConv2 D) ================================================================= Total params: 286 Trainable params: 286 Non-trainable params: 0 _________________________________________________________________ |

which we can compare with a similar model using a regular 2D convolutional layer,

1 |
normal = Conv2D(filters=64, kernel_size=(3,3), padding=”valid”, activation=”relu”)(visible) |

which gives the output

1 2 3 4 5 6 7 8 9 10 11 12 |
_________________________________________________________________ Layer (type) Output Shape Param # ================================================================= input (InputLayer) [(None, 32, 32, 3)] 0 conv2d (Conv2D) (None, 30, 30, 64) 1792 ================================================================= Total params: 1,792 Trainable params: 1,792 Non-trainable params: 0 _________________________________________________________________ |

That corroborates with our initial calculations on the number of parameters done earlier and shows the reduction in number of parameters that can be achieved by using depthwise separable convolutions.

More specifically, let’s look at the number and size of kernels in a normal convolutional layer and a depthwise separable one. When looking at a regular 2D convolutional layer with $c$ channels as inputs, $w \times h$ kernel spatial resolution, and $n$ channels as output, we would need to have $(n, w, h, c)$ parameters, that is $n$ filters, with each filter having a kernel size of $(w, h, c)$. However, this is different for a similar depthwise separable convolution even with the same number of input channels, kernel spatial resolution, and output channels. First, there’s the depthwise convolution which involves $c$ filters, each with a kernel size of $(w, h, 1)$ which outputs $c$ channels since it acts on each filter. This depthwise convolutional layer has $(c, w, h, 1)$ parameters (plus some bias units). Then comes the pointwise convolution which takes in the $c$ channels from the depthwise layer, and outputs $n$ channels, so we have $n$ filters each with a kernel size of $(1, 1, n)$. This pointwise convolutional layer has $(n, 1, 1, n)$ parameters (plus some bias units).

You might be thinking right now, but why do they work?

One way of thinking about it, from the Xception paper by Chollet is that depthwise separable convolutions have the assumption that we can separately map cross-channel and spatial correlations. Given this, there will be bunch of redundant weights in the convolutional layer which we can reduce by separating the convolution into two convolutions of the depthwise and pointwise component. One way of thinking about it for those familiar with linear algebra is how we are able to decompose a matrix into outer product of two vectors when the column vectors in the matrix are multiples of each other.

## Using Depthwise Separable Convolutions in Computer Vision Models

Now that we’ve seen the reduction in parameters that we can achieve by using a depthwise separable convolution over a normal convolutional filter, let’s see how we can use it in practice with Tensorflow’s `SeparableConv2D`

filter.

For this example, we will be using the CIFAR-10 image dataset used in the above example, while for the model we will be using a model built off VGG blocks. The potential of depthwise separable convolutions is in deeper models where the regularization effect is more beneficial to the model and the reduction in parameters is more obvious as opposed to a lighter weight model such as LeNet-5.

Creating our model using VGG blocks using normal convolutional layers,

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 |
from keras.models import Model from keras.layers import Input, Conv2D, MaxPooling2D, Dense, Flatten, SeparableConv2D import tensorflow as tf # function for creating a vgg block def vgg_block(layer_in, n_filters, n_conv): # add convolutional layers for _ in range(n_conv): layer_in = Conv2D(filters = n_filters, kernel_size = (3,3), padding='same', activation="relu")(layer_in) # add max pooling layer layer_in = MaxPooling2D((2,2), strides=(2,2))(layer_in) return layer_in visible = Input(shape=(32, 32, 3)) layer = vgg_block(visible, 64, 2) layer = vgg_block(layer, 128, 2) layer = vgg_block(layer, 256, 2) layer = Flatten()(layer) layer = Dense(units=10, activation="softmax")(layer) # create model model = Model(inputs=visible, outputs=layer) # summarize model model.summary() model.compile(optimizer="adam", loss=tf.keras.losses.SparseCategoricalCrossentropy(), metrics="acc") history = model.fit(x=trainX, y=trainY, batch_size=128, epochs=10, validation_data=(testX, testY)) |

Then we look at the results of this 6-layer convolutional neural network with normal convolutional layers,

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 |
_________________________________________________________________ Layer (type) Output Shape Param # ================================================================= input_1 (InputLayer) [(None, 32, 32, 3)] 0 conv2d (Conv2D) (None, 32, 32, 64) 1792 conv2d_1 (Conv2D) (None, 32, 32, 64) 36928 max_pooling2d (MaxPooling2D (None, 16, 16, 64) 0 ) conv2d_2 (Conv2D) (None, 16, 16, 128) 73856 conv2d_3 (Conv2D) (None, 16, 16, 128) 147584 max_pooling2d_1 (MaxPooling (None, 8, 8, 128) 0 2D) conv2d_4 (Conv2D) (None, 8, 8, 256) 295168 conv2d_5 (Conv2D) (None, 8, 8, 256) 590080 max_pooling2d_2 (MaxPooling (None, 4, 4, 256) 0 2D) flatten (Flatten) (None, 4096) 0 dense (Dense) (None, 10) 40970 ================================================================= Total params: 1,186,378 Trainable params: 1,186,378 Non-trainable params: 0 _________________________________________________________________ Epoch 1/10 391/391 [==============================] - 11s 27ms/step - loss: 1.7468 - acc: 0.4496 - val_loss: 1.3347 - val_acc: 0.5297 Epoch 2/10 391/391 [==============================] - 10s 26ms/step - loss: 1.0224 - acc: 0.6399 - val_loss: 0.9457 - val_acc: 0.6717 Epoch 3/10 391/391 [==============================] - 10s 26ms/step - loss: 0.7846 - acc: 0.7282 - val_loss: 0.8566 - val_acc: 0.7109 Epoch 4/10 391/391 [==============================] - 10s 26ms/step - loss: 0.6394 - acc: 0.7784 - val_loss: 0.8289 - val_acc: 0.7235 Epoch 5/10 391/391 [==============================] - 10s 26ms/step - loss: 0.5385 - acc: 0.8118 - val_loss: 0.7445 - val_acc: 0.7516 Epoch 6/10 391/391 [==============================] - 11s 27ms/step - loss: 0.4441 - acc: 0.8461 - val_loss: 0.7927 - val_acc: 0.7501 Epoch 7/10 391/391 [==============================] - 11s 27ms/step - loss: 0.3786 - acc: 0.8672 - val_loss: 0.8279 - val_acc: 0.7455 Epoch 8/10 391/391 [==============================] - 10s 26ms/step - loss: 0.3261 - acc: 0.8855 - val_loss: 0.8886 - val_acc: 0.7560 Epoch 9/10 391/391 [==============================] - 10s 27ms/step - loss: 0.2747 - acc: 0.9044 - val_loss: 1.0134 - val_acc: 0.7387 Epoch 10/10 391/391 [==============================] - 10s 26ms/step - loss: 0.2519 - acc: 0.9126 - val_loss: 0.9571 - val_acc: 0.7484 |

Let’s try out the same architecture but replace the normal convolutional layers with Keras’ `SeparableConv2D`

layers instead:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
# depthwise separable VGG block def vgg_depthwise_block(layer_in, n_filters, n_conv): # add convolutional layers for _ in range(n_conv): layer_in = SeparableConv2D(filters = n_filters, kernel_size = (3,3), padding='same', activation='relu')(layer_in) # add max pooling layer layer_in = MaxPooling2D((2,2), strides=(2,2))(layer_in) return layer_in visible = Input(shape=(32, 32, 3)) layer = vgg_depthwise_block(visible, 64, 2) layer = vgg_depthwise_block(layer, 128, 2) layer = vgg_depthwise_block(layer, 256, 2) layer = Flatten()(layer) layer = Dense(units=10, activation="softmax")(layer) # create model model = Model(inputs=visible, outputs=layer) # summarize model model.summary() model.compile(optimizer="adam", loss=tf.keras.losses.SparseCategoricalCrossentropy(), metrics="acc") history_dsconv = model.fit(x=trainX, y=trainY, batch_size=128, epochs=10, validation_data=(testX, testY)) |

Running the above code gives us the result:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 |
_________________________________________________________________ Layer (type) Output Shape Param # ================================================================= input_1 (InputLayer) [(None, 32, 32, 3)] 0 separable_conv2d (Separab (None, 32, 32, 64) 283 leConv2D) separable_conv2d_2 (Separab (None, 32, 32, 64) 4736 leConv2D) max_pooling2d (MaxPoolin (None, 16, 16, 64) 0 g2D) separable_conv2d_3 (Separab (None, 16, 16, 128) 8896 leConv2D) separable_conv2d_4 (Separab (None, 16, 16, 128) 17664 leConv2D) max_pooling2d_2 (MaxPoolin (None, 8, 8, 128) 0 g2D) separable_conv2d_5 (Separa (None, 8, 8, 256) 34176 bleConv2D) separable_conv2d_6 (Separa (None, 8, 8, 256) 68096 bleConv2D) max_pooling2d_3 (MaxPoolin (None, 4, 4, 256) 0 g2D) flatten (Flatten) (None, 4096) 0 dense (Dense) (None, 10) 40970 ================================================================= Total params: 174,821 Trainable params: 174,821 Non-trainable params: 0 _________________________________________________________________ Epoch 1/10 391/391 [==============================] - 10s 22ms/step - loss: 1.7578 - acc: 0.3534 - val_loss: 1.4138 - val_acc: 0.4918 Epoch 2/10 391/391 [==============================] - 8s 21ms/step - loss: 1.2712 - acc: 0.5452 - val_loss: 1.1618 - val_acc: 0.5861 Epoch 3/10 391/391 [==============================] - 8s 22ms/step - loss: 1.0560 - acc: 0.6286 - val_loss: 0.9950 - val_acc: 0.6501 Epoch 4/10 391/391 [==============================] - 8s 21ms/step - loss: 0.9175 - acc: 0.6800 - val_loss: 0.9327 - val_acc: 0.6721 Epoch 5/10 391/391 [==============================] - 9s 22ms/step - loss: 0.7939 - acc: 0.7227 - val_loss: 0.8348 - val_acc: 0.7056 Epoch 6/10 391/391 [==============================] - 8s 22ms/step - loss: 0.7120 - acc: 0.7515 - val_loss: 0.8228 - val_acc: 0.7153 Epoch 7/10 391/391 [==============================] - 8s 21ms/step - loss: 0.6346 - acc: 0.7772 - val_loss: 0.7444 - val_acc: 0.7415 Epoch 8/10 391/391 [==============================] - 8s 21ms/step - loss: 0.5534 - acc: 0.8061 - val_loss: 0.7417 - val_acc: 0.7537 Epoch 9/10 391/391 [==============================] - 8s 21ms/step - loss: 0.4865 - acc: 0.8301 - val_loss: 0.7348 - val_acc: 0.7582 Epoch 10/10 391/391 [==============================] - 8s 21ms/step - loss: 0.4321 - acc: 0.8485 - val_loss: 0.7968 - val_acc: 0.7458 |

Notice that there are significantly less parameters in the depthwise separable convolution version (~200k vs ~1.2m parameters), along with a slightly lower train time per epoch. Depthwise separable convolutions is more likely to work better on deeper models that might face an overfitting problem and on layers with larger kernels since there is a greater decrease in parameters and computations that would offset the additional computational cost of doing two convolutions instead of one. Next, we plot the train and validation and accuracy of the two models, to see differences in the training performance of the models:

The highest validation accuracy is similar for both models, but the depthwise separable convolution appears to have less overfitting to the train set, which might help it generalize better to new data.

Combining all the code together for the depthwise separable convolutions version of the model,

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 |
import tensorflow.keras as keras from keras.datasets import mnist # load dataset (trainX, trainY), (testX, testY) = keras.datasets.cifar10.load_data() # depthwise separable VGG block def vgg_depthwise_block(layer_in, n_filters, n_conv): # add convolutional layers for _ in range(n_conv): layer_in = SeparableConv2D(filters = n_filters, kernel_size = (3,3), padding='same',activation='relu')(layer_in) # add max pooling layer layer_in = MaxPooling2D((2,2), strides=(2,2))(layer_in) return layer_in visible = Input(shape=(32, 32, 3)) layer = vgg_depthwise_block(visible, 64, 2) layer = vgg_depthwise_block(layer, 128, 2) layer = vgg_depthwise_block(layer, 256, 2) layer = Flatten()(layer) layer = Dense(units=10, activation="softmax")(layer) # create model model = Model(inputs=visible, outputs=layer) # summarize model model.summary() model.compile(optimizer="adam", loss=tf.keras.losses.SparseCategoricalCrossentropy(), metrics="acc") history_dsconv = model.fit(x=trainX, y=trainY, batch_size=128, epochs=10, validation_data=(testX, testY)) |

## Further Reading

This section provides more resources on the topic if you are looking to go deeper.

**Papers:**

- Rigid-Motion Scattering For Image Classification (depthwise separable convolutions)
- MobileNet
- Xception

**APIs:**

- Depthwise Separable Convolutions in Tensorflow (SeparableConv2D)

## Summary

In this post, you’ve seen what are depthwise, pointwise, and depthwise separable convolutions. You’ve also seen how using depthwise separable convolutions allows us to get competitive results while using a significantly smaller number of parameters.

Specifically, you’ve learnt:

- What is a depthwise, pointwise, and depthwise separable convolution
- How to implement depthwise separable convolutions in Tensorflow
- Using them as part of our computer vision models

Very useful article!

In your figures, it’s probably “Accuracy” in your y-axis, rather than “Loss”.

Hi George…Thank you for your feedback! The following resource will provide more detail on learning curves:

https://machinelearningmastery.mystagingwebsite.com/learning-curves-for-diagnosing-machine-learning-model-performance/