Running a Neural Network Model in OpenCV

Many machine learning models have been developed, each with strengths and weaknesses. This catalog is not complete without neural network models. In OpenCV, you can use a neural network model developed using another framework. In this post, you will learn about the workflow of applying a neural network in OpenCV. Specifically, you will learn:

• What OpenCV can use in its neural network model
• How to prepare a neural network model for OpenCV

Kick-start your project with my book Machine Learning in OpenCV. It provides self-study tutorials with working code.

Running a Neural Network Model in OpenCV
Photo by Nastya Dulhiier. Some rights reserved.

Overview of Neural Network Models

The other name of the neural network is multilayer perceptrons. It is inspired by the structure and function of the human brain. Imagine a web of interconnected nodes, each performing simple calculations on data that passes through it. These nodes, or “perceptrons,” communicate with each other, adjusting their connections based on the information they receive. These perceptrons are organized in a directed graph, and the calculations have a deterministic order from input to output. Their organization is often described in terms of sequential layers. The learning process allows the network to identify patterns and make predictions even with unseen data.

In computer vision, neural networks tackle tasks like image recognition, object detection, and image segmentation. Usually, within the model, three high-level operations are performed:

1. Feature extraction: The network receives an image as input. The first layers then analyze the pixels, searching for basic features like edges, curves, and textures. These features are like building blocks, giving the network a rudimentary understanding of the image’s content.
2. Feature learning: Deeper layers build upon these features, combining and transforming them to discover higher-level, more complex patterns. This could involve recognizing shapes or objects.
3. Output generation: Finally, the last layers of the network use the learned patterns to make their predictions. Depending on the task, it could classify the image (e.g., cat vs. dog) or identify the objects it contains.

These operations are learned rather than crafted. The power of neural networks lies in their flexibility and adaptivity. By fine-tuning the connections between neurons and providing large amounts of labeled data, we can train them to solve complex vision problems with remarkable accuracy. But also because of their flexibility and adaptivity, neural networks are usually not the most efficient model regarding memory and computation complexity.

Training a Neural Network

Because of the nature of the model, training a generic neural network is not trivial. There is no training facility in OpenCV. Therefore, you must train a model using another framework and load it in OpenCV. You want to use OpenCV in this case because you are already using OpenCV for other image processing tasks and do not want to introduce another dependency to your project or because OpenCV is a much lighter library.

For example, consider the classic MNIST handwritten digit recognition problem. Let’s use Keras and TensorFlow to build and train the model for simplicity. The dataset can be obtained from TensorFlow.

The two print statements give:

You can see that the dataset provides the digits in 28×28 grayscale format. The training set has 60,000 samples. You can show some random samples using matplotlib, which you should see an image like the following:

This dataset has a label of 0 to 9, denoting the digits on the image. There are many models you can use for this classification problem. The famous LeNet5 model is one of them. Let’s create one using Keras syntax:

The last line shows the neural network architecture as follows:

There are three convolutional layers followed by two dense layers in this network. The final dense layer output is a 10-element vector as a probability that the input image corresponds to one of the 10 digits.

Training such a network in Keras is not difficult.

First, you need to reformat the input from a 28×28 image pixels into a tensor of 28×28×1 such that the extra dimension is expected by the convolutional layers. Then, the labels should be converted into a one-hot vector to match the format of the network output.

Then, you can kickstart the training by providing the hyperparameters: The loss function should be cross entropy because it is a multi-class classification problem. Adam is used as the optimizer since it is the usual choice. And during training, you want to observe for its prediction accuracy. The training should be fast. So, let’s decide to run it for 100 epochs, but let it stop early if you can’t see the model improved on the loss metric in the validation set for four consecutive epochs.

The code is as follows:

Running this model would print the progress like the following:

This training stopped at epoch 15 because of the early stopping rule.

Once you finished the model training, you can save your Keras model in the HDF5 format, which will include both the model architecture and the layer weights:

The complete code to build a model is as follows:

Converting the Model for OpenCV

OpenCV supports neural networks in its dnn module. It can consume models saved by several frameworks, including TensorFlow 1.x. But for the Keras models saved above, it is better to first convert into the ONNX format.

The tool to convert a Keras model (HDF5 format) or generic TensorFlow model (Protocol Buffer format) is the Python module tf2onnx. You can install it in your environment with the following command:

Afterwards, you have the conversion command from the module. For example, since you saved a Keras model into HDF5 format, you can use the following command to convert it into an ONNX format:

Then, a file lenet5.onnx is created.

To use it in OpenCV, you need to load the model into OpenCV as a network object. Should it be a TensorFlow Protocol Buffer file, there is a function cv2.dnn.readNetFromTensorflow('frozen_graph.pb') for this. In this post, you are using an ONNX file. Hence, it should be cv2.dnn.readNetFromONNX('model.onnx')

This model assumes an input as a “blob”, and you should invoke the model with the following:

The blob is also a numpy array but reformatted to add the batch dimension.

Using the model in OpenCV only needs a few lines of code. For example, we get the images again from the TensorFlow dataset and check all test set samples to compute the model accuracy:

Running a neural network model in OpenCV is slightly different from running the model in TensorFlow in such a way that you need to assign the input and get the output in two separate steps.

In the code above, you convert the output into a “blob” with no scaling and shifting since this is how the model is trained. You set the input of a single image, and the output will be a 1×10 array. As a softmax output, you get the model’s prediction using the argmax function. The subsequent calculation of average accuracy over the test set is trivial. The above code prints:

Summary

In this post, you learned how to use a neural network in OpenCV via its dnn module. Specifically, you learned

• How to train a neural network model and convert it to ONNX format for the consumption of OpenCV
• How to use load the model in OpenCV and run the model

Get Started on Machine Learning in OpenCV!

Learn how to use machine learning techniques in image processing projects

...using OpenCV in advanced ways and work beyond pixels

Discover how in my new Ebook:
Machine Learing in OpenCV

It provides self-study tutorials with all working code in Python to turn you from a novice to expert. It equips you with
logistic regression, random forest, SVM, k-means clustering, neural networks, and much more...all using the machine learning module in OpenCV

Kick-start your deep learning journey with hands-on exercises

See What's Inside