A Gentle Introduction to Vectors for Machine Learning

By Jason Brownlee on October 17, 2021 in Linear Algebra 33

Vectors are a foundational element of linear algebra.

Vectors are used throughout the field of machine learning in the description of algorithms and processes such as the target variable (y) when training an algorithm.

In this tutorial, you will discover linear algebra vectors for machine learning.

After completing this tutorial, you will know:

What a vector is and how to define one in Python with NumPy.
How to perform vector arithmetic such as addition, subtraction, multiplication and division.
How to perform additional operations such as dot product and multiplication with a scalar.

Kick-start your project with my new book Linear Algebra for Machine Learning, including step-by-step tutorials and the Python source code files for all examples.

Let’s get started.

A Gentle Introduction to Vectors for Machine Learning
Photo by Lachlan Donald, some rights reserved.

Tutorial Overview

This tutorial is divided into 5 parts; they are:

What is a Vector?
Defining a Vector
Vector Arithmetic
Vector Dot Product
Vector-Scalar Multiplication

Need help with Linear Algebra for Machine Learning?

Take my free 7-day email crash course now (with sample code).

Click to sign-up and also get a free PDF Ebook version of the course.

What is a Vector?

A vector is a tuple of one or more values called scalars.

Vectors are built from components, which are ordinary numbers. You can think of a vector as a list of numbers, and vector algebra as operations performed on the numbers in the list.

— Page 69, No Bullshit Guide To Linear Algebra, 2017

Vectors are often represented using a lowercase character such as “v”; for example:

v = (v1, v2, v3)

1	v = (v1, v2, v3)

Where v1, v2, v3 are scalar values, often real values.

Vectors are also shown using a vertical representation or a column; for example:

      v1
v = ( v2 )
      v3

v = ( v2 )

It is common to represent the target variable as a vector with the lowercase “y” when describing the training of a machine learning algorithm.

It is common to introduce vectors using a geometric analogy, where a vector represents a point or coordinate in an n-dimensional space, where n is the number of dimensions, such as 2.

The vector can also be thought of as a line from the origin of the vector space with a direction and a magnitude.

These analogies are good as a starting point, but should not be held too tightly as we often consider very high dimensional vectors in machine learning. I find the vector-as-coordinate the most compelling analogy in machine learning.

Now that we know what a vector is, let’s look at how to define a vector in Python.

Defining a Vector

We can represent a vector in Python as a NumPy array.

A NumPy array can be created from a list of numbers. For example, below we define a vector with the length of 3 and the integer values 1, 2 and 3.

# create a vector
from numpy import array
v = array([1, 2, 3])
print(v)

# create a vector

from numpy import array

v = array([1, 2, 3])

print(v)

The example defines a vector with 3 elements.

Running the example prints the defined vector.

[1 2 3]

[1 2 3]

Vector Arithmetic

In this section will demonstrate simple vector-vector arithmetic, where all operations are performed element-wise between two vectors of equal length to result in a new vector with the same length

Vector Addition

Two vectors of equal length can be added together to create a new third vector.

c = a + b

c = a + b

The new vector has the same length as the other two vectors. Each element of the new vector is calculated as the addition of the elements of the other vectors at the same index; for example:

a + b = (a1 + b1, a2 + b2, a3 + b3)

1	a + b = (a1 + b1, a2 + b2, a3 + b3)

Or, put another way:

c[0] = a[0] + b[0]
c[1] = a[1] + b[1]
c[2] = a[2] + b[2]

c[0] = a[0] + b[0]

c[1] = a[1] + b[1]

c[2] = a[2] + b[2]

We can add vectors directly in Python by adding NumPy arrays.

# add vectors
from numpy import array
a = array([1, 2, 3])
print(a)
b = array([1, 2, 3])
print(b)
c = a + b
print(c)

# add vectors

from numpy import array

a = array([1, 2, 3])

print(a)

b = array([1, 2, 3])

print(b)

c = a + b

print(c)

The example defines two vectors with three elements each, then adds them together.

Running the example first prints the two parent vectors then prints a new vector that is the addition of the two vectors.

[1 2 3]

[1 2 3]

[2 4 6]

[1 2 3]

[2 4 6]

Vector Subtraction

One vector can be subtracted from another vector of equal length to create a new third vector.

c = a - b

c = a - b

As with addition, the new vector has the same length as the parent vectors and each element of the new vector is calculated as the subtraction of the elements at the same indices.

a - b = (a1 - b1, a2 - b2, a3 - b3)

1	a - b = (a1 - b1, a2 - b2, a3 - b3)

Or, put another way:

c[0] = a[0] - b[0]
c[1] = a[1] - b[1]
c[2] = a[2] - b[2]

c[0] = a[0] - b[0]

c[1] = a[1] - b[1]

c[2] = a[2] - b[2]

The NumPy arrays can be directly subtracted in Python.

# subtract vectors
from numpy import array
a = array([1, 2, 3])
print(a)
b = array([0.5, 0.5, 0.5])
print(b)
c = a - b
print(c)

# subtract vectors

from numpy import array

a = array([1, 2, 3])

print(a)

b = array([0.5, 0.5, 0.5])

print(b)

c = a - b

print(c)

The example defines two vectors with three elements each, then subtracts the first from the second.

Running the example first prints the two parent vectors then prints the new vector that is the first minus the second.

[1 2 3]

[ 0.5 0.5 0.5]

[ 0.5 1.5 2.5]

[1 2 3]

[ 0.5 0.5 0.5]

[ 0.5 1.5 2.5]

Vector Multiplication

Two vectors of equal length can be multiplied together.

c = a * b

c = a * b

As with addition and subtraction, this operation is performed element-wise to result in a new vector of the same length.

a * b = (a1 * b1, a2 * b2, a3 * b3)

1	a * b = (a1 * b1, a2 * b2, a3 * b3)

ab = (a1b1, a2b2, a3b3)

1	ab = (a1b1, a2b2, a3b3)

Or, put another way:

c[0] = a[0] * b[0]
c[1] = a[1] * b[1]
c[2] = a[2] * b[2]

c[0] = a[0] * b[0]

c[1] = a[1] * b[1]

c[2] = a[2] * b[2]

We can perform this operation directly in NumPy.

# multiply vectors
from numpy import array
a = array([1, 2, 3])
print(a)
b = array([1, 2, 3])
print(b)
c = a * b
print(c)

# multiply vectors

from numpy import array

a = array([1, 2, 3])

print(a)

b = array([1, 2, 3])

print(b)

c = a * b

print(c)

The example defines two vectors with three elements each, then multiplies the vectors together.

Running the example first prints the two parent vectors, then the new vector is printed.

[1 2 3]

[1 2 3]

[1 4 9]

[1 2 3]

[1 4 9]

Vector Division

Two vectors of equal length can be divided.

c = a / b

c = a / b

As with other arithmetic operations, this operation is performed element-wise to result in a new vector of the same length.

a / b = (a1 / b1, a2 / b2, a3 / b3)

1	a / b = (a1 / b1, a2 / b2, a3 / b3)

a / b = (a1b1, a2b2, a3b3)

1	a / b = (a1b1, a2b2, a3b3)

Or, put another way:

c[0] = a[0] / b[0]
c[1] = a[1] / b[1]
c[2] = a[2] / b[2]

c[0] = a[0] / b[0]

c[1] = a[1] / b[1]

c[2] = a[2] / b[2]

We can perform this operation directly in NumPy.

# divide vectors
from numpy import array
a = array([1, 2, 3])
print(a)
b = array([1, 2, 3])
print(b)
c = a / b
print(c)

# divide vectors

from numpy import array

a = array([1, 2, 3])

print(a)

b = array([1, 2, 3])

print(b)

c = a / b

print(c)

The example defines two vectors with three elements each, then divides the first by the second.

Running the example first prints the two parent vectors, followed by the result of the vector division.

[1 2 3]

[1 2 3]

[ 1. 1. 1.]

[1 2 3]

[ 1. 1. 1.]

Vector Dot Product

We can calculate the sum of the multiplied elements of two vectors of the same length to give a scalar.

This is called the dot product, named because of the dot operator used when describing the operation.

The dot product is the key tool for calculating vector projections, vector decompositions, and determining orthogonality. The name dot product comes from the symbol used to denote it.

— Page 110, No Bullshit Guide To Linear Algebra, 2017

c = a . b

c = a . b

The operation can be used in machine learning to calculate the weighted sum of a vector.

The dot product is calculated as follows:

a . b = (a1 * b1 + a2 * b2 + a3 * b3)

1	a . b = (a1 * b1 + a2 * b2 + a3 * b3)

a . b = (a1b1 + a2b2 + a3b3)

1	a . b = (a1b1 + a2b2 + a3b3)

We can calculate the dot product between two vectors in Python using the dot() function on a NumPy array. It can also be calculated using the newer @ operator, since Python version 3.5. The example below demonstrates both methods.

# dot product vectors
from numpy import array
a = array([1, 2, 3])
print(a)
b = array([1, 2, 3])
print(b)
c = a.dot(b)
print(c)
d = a @ b
print(d)

# dot product vectors

from numpy import array

a = array([1, 2, 3])

print(a)

b = array([1, 2, 3])

print(b)

c = a.dot(b)

print(c)

d = a @ b

print(d)

The example defines two vectors with three elements each, then calculates the dot product.

Running the example first prints the two parent vectors, then the scalar dot product.

[1 2 3]

Vector-Scalar Multiplication

A vector can be multiplied by a scalar, in effect scaling the magnitude of the vector.

To keep notation simple, we will use lowercase “s” to represent the scalar value.

c = s * v

c = s * v

c = sv

c = sv

The multiplication is performed on each element of the vector to result in a new scaled vector of the same length.

s * v = (s * v1, s * v2, s * v3)

1	s * v = (s * v1, s * v2, s * v3)

Or, put another way:

c[0] = a[0] * s
c[1] = a[1] * s
c[2] = a[2] * s

c[0] = a[0] * s

c[1] = a[1] * s

c[2] = a[2] * s

We can perform this operation directly with the NumPy array.

# vector-scalar multiplication
from numpy import array
a = array([1, 2, 3])
print(a)
s = 0.5
print(s)
c = s * a
print(c)

# vector-scalar multiplication

from numpy import array

a = array([1, 2, 3])

print(a)

s = 0.5

print(s)

c = s * a

print(c)

The example first defines the vector and the scalar then multiplies the vector by the scalar.

Running the example first prints the parent vector, then scalar, and then the result of multiplying the two together.

[1 2 3]

0.5

[ 0.5 1. 1.5]

[1 2 3]

0.5

[ 0.5 1. 1.5]

Similarly, vector-scalar addition, subtraction, and division can be performed in the same way.

Extensions

This section lists some ideas for extending the tutorial that you may wish to explore.

Create 5 examples using each operation using your own data.
Implement each vector operation manually for vectors defined as lists.
Search machine learning papers and find 1 example of each operation being used.

If you explore any of these extensions, I’d love to know.

Summary

In this tutorial, you discovered linear algebra vectors for machine learning.

Specifically, you learned:

What a vector is and how to define one in Python with NumPy.
How to perform vector arithmetic such as addition, subtraction, multiplication and division.
How to perform additional operations such as dot product and multiplication with a scalar.

Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.

33 Responses to A Gentle Introduction to Vectors for Machine Learning

Edward June 6, 2019 at 4:05 pm #

So how do you determine a vector to help in classification?

Reply
- Jason Brownlee June 7, 2019 at 7:50 am #
  
  What do you mean exactly Edward?
  
  Reply
  - Edward June 7, 2019 at 8:51 pm #
    
    When you have a feature vector and asked to determine the vector, what does that mean?
    
    Reply
    - Jason Brownlee June 8, 2019 at 6:53 am #
      
      If you have a feature vector, it can be classified with a model.
      
      A feature vector is just a row where each value is measurement for a different feature or column.
      
      Does that help?
      
      Reply
Ahmed Ramadan June 23, 2019 at 8:23 pm #

what is vector addition mean in machine learning?
I have two vector contain features, can I use vector add to preserve two features into single vector?

Reply
- Jason Brownlee June 24, 2019 at 6:24 am #
  
  Adding two vectors together.
  
  Typically we do not add features together unless it has a specific meaning in the domain, e.g. both are coordinates in some larger n-dimensional space.
  
  We can explore an embedding using vector arithmetic or a GAN latent space.
  
  Reply
Vania Todorova July 12, 2019 at 1:47 am #

Have you worked with vectors for data for the SageMaker? my data is in numpy arrays but the error msg i get is labels must be a Vector..

Reply
- Jason Brownlee July 12, 2019 at 8:45 am #
  
  I have not used SageMaker, sorry.
  
  Reply
Ricardo Lizano G August 8, 2019 at 11:02 am #

Very clear, thanks!

Reply
- Jason Brownlee August 8, 2019 at 2:21 pm #
  
  Thanks, I’m glad it helped.
  
  Reply
Aminzai August 9, 2019 at 12:51 am #

thanks
Jason Brownlee great explaination.

Reply
- Jason Brownlee August 9, 2019 at 8:16 am #
  
  You’re welcome Aminzai.
  
  Reply
mohara January 30, 2020 at 7:37 am #

hi, as far as I know for text classification we need some features and it is up to us to vectorized each sentences based on the specific teacher yes??
I mean we should write suitable program to convert each sentence as vector based on our feature yes?

Reply
- mohara January 30, 2020 at 7:44 am #
  
  ##i corrected my question sir
  hi, as far as I know for text classification we need some features and it is up to us to vectorized each sentences based on the specific feature yes??
  I mean we should write suitable program to convert each sentence as vector based on our feature yes?
  for feature 1 we should write a program to represent our sentences as a vector while for feature 2 we should consider another pieces of code to represent our sentences as a vector yea?
  
  Reply
- Jason Brownlee January 30, 2020 at 2:13 pm #
  
  You can use a bag of words model:
  https://machinelearningmastery.com/gentle-introduction-bag-words-model/
  
  Reply
Saurabh Singh April 17, 2020 at 3:39 am #

I had been looking for similar tutorials for a long time and now I have found. Thank you sincerely.

Reply
- Jason Brownlee April 17, 2020 at 6:27 am #
  
  Thanks!
  
  Reply
Faiqa July 2, 2020 at 10:49 pm #

Just like the way we have a feature vector, can we also possibly have a response vector in here? if yes then what it would consist of sir?

Reply
- Jason Brownlee July 3, 2020 at 6:16 am #
  
  Yes.
  
  You can define the composition of the feature vectors and target vectors for your project.
  
  Reply
Jim October 4, 2020 at 10:04 am #

I know squat about Liner Algebra and this made total sense to me. Thanks for a simple, clear and concise explanation.

Reply
- Jason Brownlee October 4, 2020 at 2:57 pm #
  
  You’re welcome.
  
  Reply
firoza October 19, 2020 at 2:53 am #

Explanation was awesome!! its easy to understand.

Reply
- Jason Brownlee October 19, 2020 at 6:39 am #
  
  Thanks!
  
  Reply
Tony December 19, 2020 at 5:12 am #

Great explanation. Thank you!

Reply
- Jason Brownlee December 19, 2020 at 6:21 am #
  
  Thanks!
  
  Reply

Terry Jerry January 27, 2021 at 12:38 am #

My code for vector arithmetic operations with lists. I know it’s not perfect.

import numpy as np
from typing import List

Vector = List[float]

def addition(v: Vector, x: Vector) -> Vector:
    assert len(v) == len(x), 'Length of vectors is not the same!'
    return [x + y for x, y in zip(v, x)]

a = [1, 2, 3]
b = [1, 2, 3]
                                                                        
assert addition(a, b) == [2, 4, 6]
#assert addition(a, [1, 2]) == [2, 4, 6]


def subtraction(v: Vector, x: Vector) -> Vector:
    assert len(v) == len(x), 'Length of vectors is not the same!'
    return [x - y for x, y in zip(v, x)]


assert subtraction(a, b) == [0, 0, 0]
#assert subtraction(a, [1, 2]) == [0, 0]

def multiplication(v: Vector, x: Vector) -> Vector:
    assert len(v) == len(x), 'Length of vectors is not the same!'
    return [x * y for x, y in zip(v, x)]


assert multiplication(a, b) == [1, 4, 9]


def division(v: Vector, x: Vector) -> Vector:
    assert len(v) == len(x), 'The length of the vector is not the same!'
    return [x / y for x, y in zip(v, x)]


assert division(a, [0.5, 0.5, 0.5]) == [2., 4., 6.]


def dot_production(v: Vector, x: Vector) -> Vector:
    assert len(v) == len(x), 'The length of the vectors is not the same'
    return sum([x * y for x, y in zip(v, x)])


def scalar_production(v: Vector, s: float) -> Vector:
    return [s * x for x in v]


assert scalar_production(a, 0.5) == [0.5, 1., 1.5]

import numpy as np

from typing import List

Vector = List[float]

def addition(v: Vector, x: Vector) -> Vector:

assert len(v) == len(x), 'Length of vectors is not the same!'

return [x + y for x, y in zip(v, x)]

a = [1, 2, 3]

b = [1, 2, 3]

assert addition(a, b) == [2, 4, 6]

#assert addition(a, [1, 2]) == [2, 4, 6]

def subtraction(v: Vector, x: Vector) -> Vector:

assert len(v) == len(x), 'Length of vectors is not the same!'

return [x - y for x, y in zip(v, x)]

assert subtraction(a, b) == [0, 0, 0]

#assert subtraction(a, [1, 2]) == [0, 0]

def multiplication(v: Vector, x: Vector) -> Vector:

assert len(v) == len(x), 'Length of vectors is not the same!'

return [x * y for x, y in zip(v, x)]

assert multiplication(a, b) == [1, 4, 9]

def division(v: Vector, x: Vector) -> Vector:

assert len(v) == len(x), 'The length of the vector is not the same!'

return [x / y for x, y in zip(v, x)]

assert division(a, [0.5, 0.5, 0.5]) == [2., 4., 6.]

def dot_production(v: Vector, x: Vector) -> Vector:

assert len(v) == len(x), 'The length of the vectors is not the same'

return sum([x * y for x, y in zip(v, x)])

def scalar_production(v: Vector, s: float) -> Vector:

return [s * x for x in v]

assert scalar_production(a, 0.5) == [0.5, 1., 1.5]

Terry Jerry January 27, 2021 at 12:40 am #

I forgot to add:

after import modules you should add line:

Vector = List[float]

Reply
- Jason Brownlee January 27, 2021 at 6:10 am #
  
  I added it for you.
  
  Reply
Jason Brownlee January 27, 2021 at 6:09 am #

Well done, thanks for sharing!

Reply

Maanas February 1, 2022 at 4:47 pm #

What is the intuition behind applying dot product on 2 vectors? For instance, why do we apply the dot product on the item vector and user vector in content-based filtering (recommender system)

Reply
- James Carmichael February 2, 2022 at 10:22 am #
  
  Hi Maanas…The following would be a great place to clarify your understanding:
  
  https://machinelearningmastery.com/linear-algebra-machine-learning-7-day-mini-course/
  
  Reply
Guillermo February 16, 2022 at 11:57 pm #

Hi.Could you explain the concept of reference vectors? I see that mentioned in Kohonen Self-Organizing maps, I cannot understand it.

Reply
- James Carmichael February 17, 2022 at 1:25 pm #
  
  Hi Guillermo…The following is great starting point regarding vectors and their role in machine learning.
  
  https://neptune.ai/blog/understanding-vectors-from-a-machine-learning-perspective
  
  Reply

Navigation

A Gentle Introduction to Vectors for Machine Learning

Tutorial Overview

Need help with Linear Algebra for Machine Learning?

What is a Vector?

Defining a Vector

Vector Arithmetic

Vector Addition

Vector Subtraction

Vector Multiplication

Vector Division

Vector Dot Product

Vector-Scalar Multiplication

Extensions

Further Reading

Books

API

Articles

Summary

Get a Handle on Linear Algebra for Machine Learning!

Develop a working understand of linear algebra

Finally Understand the Mathematics of Data

More On This Topic

33 Responses to A Gentle Introduction to Vectors for Machine Learning

Leave a Reply Click here to cancel reply.