How To Implement Learning Vector Quantization From Scratch With Python

A limitation of k-Nearest Neighbors is that you must keep a large database of training examples in order to make predictions.

The Learning Vector Quantization algorithm addresses this by learning a much smaller subset of patterns that best represent the training data.

In this tutorial, you will discover how to implement the Learning Vector Quantization algorithm from scratch with Python.

After completing this tutorial, you will know:

  • How to learn a set of codebook vectors from a training data set.
  • How to make predictions using learned codebook vectors.
  • How to apply Learning Vector Quantization to a real predictive modeling problem.

Let’s get started.

  • Update Jan/2017: Changed the calculation of fold_size in cross_validation_split() to always be an integer. Fixes issues with Python 3.
  • Update Aug/2018: Tested and updated to work with Python 3.6.
How To Implement Learning Vector Quantization From Scratch With Python

How To Implement Learning Vector Quantization From Scratch With Python
Photo by Tony Faiola, some rights reserved.

Description

This section provides a brief introduction to the Learning Vector Quantization algorithm and the Ionosphere classification problem that we will use in this tutorial

Learning Vector Quantization

The Learning Vector Quantization (LVQ) algorithm is a lot like k-Nearest Neighbors.

Predictions are made by finding the best match among a library of patterns. The difference is that the library of patterns is learned from training data, rather than using the training patterns themselves.

The library of patterns are called codebook vectors and each pattern is called a codebook. The codebook vectors are initialized to randomly selected values from the training dataset. Then, over a number of epochs, they are adapted to best summarize the training data using a learning algorithm.

The learning algorithm shows one training record at a time, finds the best matching unit among the codebook vectors and moves it closer to the training record if they have the same class, or further away if they have different classes.

Once prepared, the codebook vectors are used to make predictions using the k-Nearest Neighbors algorithm where k=1.

The algorithm was developed for classification predictive modeling problems, but can be adapted for use with regression problems.

Ionosphere Dataset

The Ionosphere dataset predicts the structure of the ionosphere given radar return data.

Each instance describes the properties of radar returns from the atmosphere and the task is to predict whether or not there is structure in the ionosphere.

There are 351 instances and 34 numerical input variables, 17 pairs of 2 for each radar pulse that generally have the same scale of 0-1. The class value is a string with a value of either a “g” for good return or “b” for a bad return.

Using the Zero Rule Algorithm that predicts the class with the most observations, a baseline accuracy of 64.286% can be achieved.

You can learn more and download the dataset from the UCI Machine Learning Repository.

Download the dataset and place it in your current working directory with the name ionosphere.csv.

Tutorial

This tutorial is broken down into 4 parts:

  1. Euclidean Distance.
  2. Best Matching Unit.
  3. Training Codebook Vectors.
  4. Ionosphere Case Study.

These steps will lay the foundation for implementing and applying the LVQ algorithm to your own predictive modeling problems.

1. Euclidean Distance

The first step needed is to calculate the distance between two rows in a dataset.

Rows of data are mostly made up of numbers and an easy way to calculate the distance between two rows or vectors of numbers is to draw a straight line. This makes sense in 2D or 3D and scales nicely to higher dimensions.

We can calculate the straight line distance between two vectors using the Euclidean distance measure. It is calculated as the square root of the sum of the squared differences between the two vectors.

Where x1 is the first row of data, x2 is the second row of data and i is the index for a specific column as we sum across all columns.

With Euclidean distance, the smaller the value, the more similar two records will be. A value of 0 means that there is no difference between two records.

Below is a function named euclidean_distance() that implements this in Python.

You can see that the function assumes that the last column in each row is an output value which is ignored from the distance calculation.

We can test this distance function with a small contrived classification dataset. We will use this dataset a few times as we construct the elements needed for the LVQ algorithm.

Putting this all together, we can write a small example to test our distance function by printing the distance between the first row and all other rows. We would expect the distance between the first row and itself to be 0, a good thing to look out for.

The full example is listed below.

Running this example prints the distances between the first row and every row in the dataset, including itself.

Now it is time to use the distance calculation to locate the best matching unit within a dataset.

2. Best Matching Unit

The Best Matching Unit or BMU is the codebook vector that is most similar to a new piece of data.

To locate the BMU for a new piece of data within a dataset we must first calculate the distance between each codebook to the new piece of data. We can do this using our distance function above.

Once distances are calculated, we must sort all of the codebooks by their distance to the new data. We can then return the first or most similar codebook vector.

We can do this by keeping track of the distance for each record in the dataset as a tuple, sort the list of tuples by the distance (in descending order) and then retrieve the BMU.

Below is a function named get_best_matching_unit() that implements this.

You can see that the euclidean_distance() function developed in the previous step is used to calculate the distance between each codebook and the new test_row.

The list of codebook and distance tuples is sorted where a custom key is used ensuring that the second item in the tuple (tup[1]) is used in the sorting operation.

Finally, the top or most similar codebook vector is returned as the BMU.

We can test this function with the small contrived dataset prepared in the previous section.

The complete example is listed below.

Running this example prints the BMU in the dataset to the first record. As expected, the first record is the most similar to itself and is at the top of the list.

Make predictions with a set of codebook vectors is the same thing.

We use the 1-nearest neighbor algorithm. That is, for each new pattern we wish to make a prediction for, we locate the most similar codebook vector in the set and return its associated class value.

Now that we know how to get the best matching unit from a set of codebook vectors, we need to learn how to train them.

3. Training Codebook Vectors

The first step in training a set of codebook vectors is to initialize the set.

We can initialize it with patterns constructed from random features in the training dataset.

Below is a function named random_codebook() that implements this. Random input and output features are selected from the training data.

After the codebook vectors are initialized to a random set, they must be adapted to best summarize the training data.

This is done iteratively.

  1. Epochs: At the top level, the process is repeated for a fixed number of epochs or exposures of the training data.
  2. Training Dataset: Within an epoch, each training pattern is used one at a time to update the set of codebook vectors.
  3. Pattern Features: For a given training pattern, each feature of a best matching codebook vector is updated to move it closer or further away.

The best matching unit is found for each training pattern and only this best matching unit is updated. The difference between the training pattern and the BMU is calculated as the error. The class values (assumed to be the last value in the list) are compared. If they match, the error is added to the BMU to bring it closer to the training pattern, otherwise, it is subtracted to push it further away.

The amount that the BMU is adjusted is controlled by a learning rate. This is a weighting on the amount of change made to all BMUs. For example, a learning rate of 0.3 means that BMUs are only moved by 30% of the error or difference between training patterns and BMUs.

Further, the learning rate is adjusted so that it has maximum effect in the first epoch and less effect as training continues until it has a minimal effect in the final epoch. This is called a linear decay learning rate schedule and can also be used in artificial neural networks.

We can summarize this decay in learning rate by epoch number as follows:

We can test this equation by assuming a learning rate of 0.3 and 10 epochs. The learning rate each epoch would be as follows:

We can put all of this together. Below is a function named train_codebooks() that implements the procedure for training a set of codebook vectors given a training dataset.

The function takes 3 additional arguments to the training dataset, the number of codebook vectors to create and train, the initial learning rate and the number of epochs for which to train the codebook vectors.

You can also see that the function keeps track of the sum squared error each epoch and prints a message showing the epoch number, effective learning rate and sum squared error score. This is helpful when debugging the training function or the specific configuration for a given prediction problem.

You can see the use of the random_codebook() to initialize the codebook vectors and the get_best_matching_unit() function to find the BMU for each training pattern within an epoch.

We can put this together with the examples above and learn a set of codebook vectors for our contrived dataset.

Below is the complete example.

Running this example trains a set of 2 codebook vectors for 10 epochs with an initial learning rate of 0.3. The details are printed each epoch and the set of 2 codebook vectors learned from the training data is displayed.

We can see that the changes to learning rate meet our expectations explored above for each epoch. We can also see that the sum squared error each epoch does continue to drop at the end of training and that there may be an opportunity to tune the example further to achieve less error.

Now that we know how to train a set of codebook vectors, let’s see how we can use this algorithm on a real dataset.

4. Ionosphere Case Study

In this section, we will apply the Learning Vector Quantization algorithm to the Ionosphere dataset.

The first step is to load the dataset and convert the loaded data to numbers that we can use with the Euclidean distance calculation. For this we will use the helper function load_csv() to load the file, str_column_to_float() to convert string numbers to floats and str_column_to_int() to convert the class column to integer values.

We will evaluate the algorithm using k-fold cross-validation with 5 folds. This means that 351/5=70.2 or just over 70 records will be in each fold. We will use the helper functions evaluate_algorithm() to evaluate the algorithm with cross-validation and accuracy_metric() to calculate the accuracy of predictions.

The complete example is listed below.

Running this example prints the classification accuracy on each fold and the mean classification accuracy across all folds.

We can see that the accuracy of 87.143% is better than the baseline of 64.286%. We can also see that our library of 20 codebook vectors is far fewer than holding the entire training dataset.

Extensions

This section lists extensions to the tutorial that you may wish to explore.

  • Tune Parameters. The parameters in the above example were not tuned, try different values to improve the classification accuracy.
  • Different Distance Measures. Experiment with different distance measures such as Manhattan distance and Minkowski distance.
  • Multiple-Pass LVQ. The codebook vectors may be updated by multiple training runs. Experiment by training with large learning rates followed by a large number of epochs with smaller learning rates to fine tune the codebooks.
  • Update More BMUs. Experiment with selecting more than one BMU when training and pushing and pulling them away from the training data.
  • More Problems. Apply LVQ to more classification problems on the UCI Machine Learning Repository.

Did you explore any of these extensions?
Share your experiences in the comments below.

Review

In this tutorial, you discovered how to implement the learning vector quantization algorithm from scratch in Python.

Specifically, you learned:

  • How to calculate the distance between patterns and locate the best matching unit.
  • How to train a set of codebook vectors to best summarize the training dataset.
  • How to apply the learning vector quantization algorithm to a real predictive modeling problem.

Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.


Want to Code Algorithms in Python Without Math?

Machine Learning Algorithms From Scratch

Code Your First Algorithm in Minutes

…with step-by-step tutorials on real-world datasets

Discover how in my new Ebook:
Machine Learning Algorithms From Scratch

It covers 18 tutorials with all the code for 12 top algorithms, like:
Linear Regression, k-Nearest Neighbors, Stochastic Gradient Descent and much more…

Finally, Pull Back the Curtain on
Machine Learning Algorithms

Skip the Academics. Just Results.

Click to learn more.


45 Responses to How To Implement Learning Vector Quantization From Scratch With Python

  1. Glen November 29, 2016 at 6:55 am #

    Hi Mr. Brownlee,
    I am used LVQ for my own dataset and it is giving me a Mean accuracy of 57.546%. Could you please suggest me the reasons for such low accuracy?

  2. barry December 2, 2016 at 1:45 am #

    I tried running it for the electricity data in this dataset: http://www.inescporto.pt/~jgama/ales/ales_5.html

    I have used the LVQ on an arff file and I am getting very bad performance from the algorithm. The accuracy is around 60%. Can you try to use the algorithm on this dataset and let me know your results or ways to improve it?

  3. Curious_Kid December 9, 2016 at 4:24 am #

    Hi, How can I use LVQ for doing the instance selection? Let’s say I want to select 5000 instances out of 15000 instances in the dataset? Please give some details.

    • Jason Brownlee December 9, 2016 at 8:46 am #

      Sorry, I’m not sure I understand your question, perhaps you could explain your intention or goal?

      • Curious_Kid December 10, 2016 at 12:56 am #

        I wanted to know if we can use LVQ for instance selection. For e.g. if there are 15000 instances in my dataset and I only want to use 5000 of those instances as the training dataset. There are various ‘Instance Selection’ algorithms and I read that we can LVQ as well. But I am not sure how can I do that using the example that you have given above.

        • Jason Brownlee December 10, 2016 at 8:06 am #

          I see, you mean to use LVQ as a data preparation or data cleaning method.

          Perhaps. I have not done this, but I love the idea. Say use a threshold and only pick instances within a distance of codebook vectors.

          Try it and see – you could be onto something very interesting there. Please share results if you get some!

          • Rahul January 5, 2017 at 2:54 am #

            Correct me if i am wrong.
            What i understand from the question asked by curious_kid is :
            Say we are given a dataset of 15000 instances..
            example
            0.000133,5,0.446809,0.089798,0.516662,0.003467,0.422915,0.414912,UP
            0.000133,5,0.468085,0.047676,0.50476,0.003467,0.422915,0.414912,DOWN
            0.000133,5,0.489362,0.146902,0.495537,0.003467,0.422915,0.414912,UP
            0.000133,5,0.510638,0.193287,0.485272,0.003467,0.422915,0.414912,UP
            .
            .
            .
            ………

            Now he wants to use LVQ for instance selection(data pre processing). That means that the final result should have 5000 best instances which he can later use for classification.

            Now as suggested by you, it could be done by taking some threshold and only pick instances within a distance of codebook vectors.

            Doing this means that we are picking instances from the dataset on the basis of distance from the random codebooks generated. For e.g. in the function “get_best_matching_unit”, if we use this distance “distances.append((codebook, dist))”(line 86), then we won’t be training the codebooks as done in your code in the lines 109-114 and hence the main functionality of LVQ won’t be implemented(i.e. to train the random codebooks over epochs and adjusting bmu according to the error). If we do train the codebooks, then we will lose the distance values, because you are returning “distances[0][0]” in line 88.

            The way I am thinking to approach this problem is by training the codebooks as done by you in the code and getting the final value of codebooks as done in line 115. Giving the value of n_codebooks as 5000, which will give us the best 5000 instances representing the data ( as one codebook represent an instance). And then use those 5000 instances for classification(using any classification algorithm).

            Do you have any suggestions for this approach? I would be glad to know about it.

          • Jason Brownlee January 5, 2017 at 9:33 am #

            Interesting. Using LVQ this way is performing a type of outlier removal.

            It will probably be quite effective at lower dimensions (in data without outlier observations) and fall apart at higher dimensions given the curse of dimensionality as distances and thresholds quickly become less meaningful. Generally LVQ and kNN breakdown at higher dimensions for the same reason:
            https://en.wikipedia.org/wiki/Curse_of_dimensionality

            Yes, the codebook vectors could be taken as centroids or a projection of the data. I would contrast this approach with projection methods like PCA, sammons mapping, tsne and others. Develop models from each projection and compare the performance on resulting models in a hold out dataset.

            I’d love to hear about the results.

  4. Curious_Kid December 11, 2016 at 10:31 am #

    I am thinking of using the euclidean distance between the codebook vectors and the new data and then calculate the mean of those distances which can act as the threshold. Then I am planning to select all those instances for which the euclidean distance is less than the calculated mean value. Does that sound good to you or do I need to think in a different way?

    • Jason Brownlee December 12, 2016 at 6:47 am #

      Try and see.

      It might be worth doing some simulations in 2d with contrived data.

      Keep in mind that the more features – the higher the dimensionality and the curse of dimensionality will make the distances less and less meaningful.

  5. Alex January 2, 2017 at 3:20 pm #

    I could not understand the functionality of “learning_vector_quantization” function. For e.g. where is it called in the code? And thus the “predict” function will not be called. I am getting the same result by commenting the predict function. Please correct me if I am wrong or if I am missing something here.

    • Jason Brownlee January 3, 2017 at 7:36 am #

      Hi Alex,

      The “learning_vector_quantization” function is passed as an argument to the “evaluate_algorithm” function where it is called on line 68 of the final code listing.

      Does that help? Try adding some print() debug statements to help you clarify further.

      • Alex January 5, 2017 at 3:52 am #

        Thank you Jason. I understood it now. I have one more doubt. You have mentioned in your blog that once the codebook vectors are prepared, those codebook vectors are used to make predictions using 1-NN. But I am not able to understand that how are you doing it in your code. Please elaborate it a bit.

        • Jason Brownlee January 5, 2017 at 9:40 am #

          Which part don’t you understand Alex?

          The specific lines of code?

          The kNN algorithm?

  6. Navin February 11, 2017 at 2:17 pm #

    How can I implement this for large image databases. I have a folder full of images, how do I convert it into arrays so that I can implement lqv on it?

    • Jason Brownlee February 12, 2017 at 5:34 am #

      Sorry Navin, I do not have an example of working with raw images and LVQ.

      • Rahat April 18, 2018 at 5:12 pm #

        Could you give any idea, how can I use your implementation for raw image.

  7. inzar May 24, 2017 at 1:00 am #

    can you share about kohonen Self-Organizing Maps metohod?
    i really need to know how unsupervice learning in this methode work.
    thanks, sorry about my bad english.

  8. Hugo June 20, 2017 at 5:20 pm #

    Hi Jason,

    I’ve created a LVQ network with trained data and two samples. It works, but how would I go about getting confidence values for untrained data? Ex. confidence of .98 means the sample is really close to the actual model of one of my samples.

    • Jason Brownlee June 21, 2017 at 8:09 am #

      Good question, I don’t know. Off the cuff, I expect you could devise a confidence interval based on distance measures and the known extent of the domain.

      • Hugo June 22, 2017 at 1:35 pm #

        Thanks for your reply. As I normalize all the values between 0 and 1, what would I use for a maximum? I tried using 1 – distance(bestUnitMatch, sample) and I get values of .96; and if that’s right I’m so amazed because I only trained the network with 30 samples for each 2 of the sets

        • Jason Brownlee June 23, 2017 at 6:38 am #

          Perhaps the largest distance between two points in your domain?

  9. Wija October 21, 2017 at 12:14 am #

    Hi, I’m learning about LVQ right now on my studies, can you send me the whole code? so I can learn it better. It is for educational purpose. Thanks

  10. Sushanth Suresh November 1, 2017 at 4:24 pm #

    Hi,

    In the “train_codebooks()” function where is the “codebooks” variable getting updated?.Only the variable “bmu” is moved towards or away from the training vector which is not used to update the “codebooks” vector.

    Please let me know if i am missing something.

    Thank you

    • Jason Brownlee November 2, 2017 at 5:04 am #

      Correct, only the BMU is being updated for each training pattern.

      • Sushanth Suresh November 4, 2017 at 8:06 am #

        I think that this updated BMU should be saved back into the codebooks to ensure the learning is taking place.

        • Jason Brownlee November 5, 2017 at 5:12 am #

          No need, we have a copy of the reference to it.

          Perhaps read up on “pass by reference” in modern programming languages?

  11. Sushanth Suresh November 5, 2017 at 10:23 am #

    Thank you clarifying it.
    I was not aware of how “pass by reference” worked in python.

    • Jason Brownlee November 6, 2017 at 4:47 am #

      No problem, it’s the same as most other languages.

  12. Vinay December 8, 2017 at 1:13 am #

    Hi, Thank you for this lovely illustration. I am not a pro in python but have some questions.
    On line 61, train_set.remove(fold) returns a funny error “ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()”
    Is there any alternative around this. ?

    Also i was wondering the reason for using lists and not numpy array like dataframe. Any particular reason?

    Thanks again.

    • Jason Brownlee December 8, 2017 at 5:43 am #

      I write the code using Python 2.7, perhaps double check your Python version?

  13. m.ragab April 25, 2018 at 7:56 pm #

    I get this errors when I run ? what I do ?

    Python 3.6.3 (v3.6.3:2c5fed8, Oct 3 2017, 17:26:49) [MSC v.1900 32 bit (Intel)] on win32
    Type “copyright”, “credits” or “license()” for more information.
    >>>
    ==== RESTART: H:/Ebooks/Ebooks/machine/project/New folder (3)/LVQ/lvq.py ====
    Traceback (most recent call last):
    File “H:/Ebooks/Ebooks/machine/project/New folder (3)/LVQ/lvq.py”, line 140, in
    scores = evaluate_algorithm(dataset, learning_vector_quantization, n_folds, n_codebooks, learn_rate, n_epochs)
    File “H:/Ebooks/Ebooks/machine/project/New folder (3)/LVQ/lvq.py”, line 68, in evaluate_algorithm
    predicted = algorithm(train_set, test_set, *args)
    File “H:/Ebooks/Ebooks/machine/project/New folder (3)/LVQ/lvq.py”, line 119, in learning_vector_quantization
    codebooks = train_codebooks(train, n_codebooks, lrate, epochs)
    File “H:/Ebooks/Ebooks/machine/project/New folder (3)/LVQ/lvq.py”, line 110, in train_codebooks
    error = row[i] – bmu[i]
    IndexError: list index out of range
    >>>

    • Jason Brownlee April 26, 2018 at 6:25 am #

      The code assumes you are using Python version 2, you are using Python 3.

      You can change your version of python or change the code.

  14. m.ragab April 26, 2018 at 2:58 pm #

    I also get this error ! although I use python 2.7

    Python 2.7.14 (v2.7.14:84471935ed, Sep 16 2017, 20:19:30) [MSC v.1500 32 bit (Intel)] on win32
    Type “copyright”, “credits” or “license()” for more information.
    >>>
    ==== RESTART: H:/Ebooks/Ebooks/machine/project/New folder (3)/LVQ/lvq.py ====

    Traceback (most recent call last):
    File “H:/Ebooks/Ebooks/machine/project/New folder (3)/LVQ/lvq.py”, line 140, in
    scores = evaluate_algorithm(dataset, learning_vector_quantization, n_folds, n_codebooks, learn_rate, n_epochs)
    File “H:/Ebooks/Ebooks/machine/project/New folder (3)/LVQ/lvq.py”, line 68, in evaluate_algorithm
    predicted = algorithm(train_set, test_set, *args)
    File “H:/Ebooks/Ebooks/machine/project/New folder (3)/LVQ/lvq.py”, line 119, in learning_vector_quantization
    codebooks = train_codebooks(train, n_codebooks, lrate, epochs)
    File “H:/Ebooks/Ebooks/machine/project/New folder (3)/LVQ/lvq.py”, line 110, in train_codebooks
    error = row[i] – bmu[i]
    IndexError: list index out of range
    >>>

  15. mo3az April 26, 2018 at 11:46 pm #

    I can’t use the code without spaces and I can’t guess it as i’am not expert in python plz help me

  16. m.ragab May 1, 2018 at 8:45 pm #

    I get this error

    error = row[i] – bmu[i]
    IndexError: list index out of range

  17. Mahendhiran PD August 2, 2018 at 5:18 pm #

    Dear Jason

    I am Mahendhiran. I am trying to execute LVQ with audio data(MFCC). I am getting error like ‘row[column] = float(row[column].strip())

    TypeError: ‘int’ object is not subscriptable’

    I request you to please help to solve this issue. I am using spyder(python 3.6).
    Thanks in advance….

Leave a Reply