A Gentle Introduction to Linear Algebra

What is Linear Algebra?

Linear algebra is a field of mathematics that is universally agreed to be a prerequisite to a deeper understanding of machine learning.

Although linear algebra is a large field with many esoteric theories and findings, the nuts and bolts tools and notations taken from the field are practical for machine learning practitioners. With a solid foundation of what linear algebra is, it is possible to focus on just the good or relevant parts.

In this tutorial, you will discover what exactly linear algebra is from a machine learning perspective.

After completing this tutorial, you will know:

  • Linear algebra is the mathematics of data.
  • Linear algebra has had a marked impact on the field of statistics.
  • Linear algebra underlies many practical mathematical tools, such as Fourier series and computer graphics.

Kick-start your project with my new book Linear Algebra for Machine Learning, including step-by-step tutorials and the Python source code files for all examples.

Let’s get started.

A Gentle Introduction to Linear Algebra

A Gentle Introduction to Linear Algebra
Photo by Steve Corey, some rights reserved.

Tutorial Overview

This tutorial is divided into 4 parts; they are:

  1. Linear Algebra
  2. Numerical Linear Algebra
  3. Linear Algebra and Statistics
  4. Applications of Linear Algebra

Need help with Linear Algebra for Machine Learning?

Take my free 7-day email crash course now (with sample code).

Click to sign-up and also get a free PDF Ebook version of the course.

Linear Algebra

Linear algebra is a branch of mathematics, but the truth of it is that linear algebra is the mathematics of data. Matrices and vectors are the language of data.

Linear algebra is about linear combinations. That is, using arithmetic on columns of numbers called vectors and arrays of numbers called matrices, to create new columns and arrays of numbers. Linear algebra is the study of lines and planes, vector spaces and mappings that are required for linear transforms.

It is a relatively young field of study, having initially been formalized in the 1800s in order to find unknowns in systems of linear equations. A linear equation is just a series of terms and mathematical operations where some terms are unknown; for example:

Equations like this are linear in that they describe a line on a two-dimensional graph. The line comes from plugging in different values into the unknown x to find out what the equation or model does to the value of y.

We can line up a system of equations with the same form with two or more unknowns; for example:

The column of y values can be taken as a column vector of outputs from the equation. The two columns of floating-point values are the data columns, say a1 and a2, and can be taken as a matrix A. The two unknown values x1 and x2 can be taken as the coefficients of the equation and together form a vector of unknowns b to be solved. We can write this compactly using linear algebra notation as:

Problems of this form are generally challenging to solve because there are more unknowns (here we have 2) than there are equations to solve (here we have 3). Further, there is often no single line that can satisfy all of the equations without error. Systems describing problems we are often interested in (such as a linear regression) can have an infinite number of solutions.

This gives a small taste of the very core of linear algebra that interests us as machine learning practitioners. Much of the rest of the operations are about making this problem and problems like it easier to understand and solve.

Numerical Linear Algebra

The application of linear algebra in computers is often called numerical linear algebra.

“numerical” linear algebra is really applied linear algebra.

— Page ix, Numerical Linear Algebra, 1997.

It is more than just the implementation of linear algebra operations in code libraries; it also includes the careful handling of the problems of applied mathematics, such as working with the limited floating point precision of digital computers.

Computers are good at performing linear algebra calculations, and much of the dependence on Graphical Processing Units (GPUs) by modern machine learning methods such as deep learning is because of their ability to compute linear algebra operations fast.

Efficient implementations of vector and matrix operations were originally implemented in the FORTRAN programming language in the 1970s and 1980s and a lot of code, or code ported from those implementations, underlies much of the linear algebra performed using modern programming languages, such as Python.

Three popular open source numerical linear algebra libraries that implement these functions are:

  • Linear Algebra Package, or LAPACK.
  • Basic Linear Algebra Subprograms, or BLAS (a standard for linear algebra libraries).
  • Automatically Tuned Linear Algebra Software, or ATLAS.

Often, when you are calculating linear algebra operations directly or indirectly via higher-order algorithms, your code is very likely dipping down to use one of these, or similar linear algebra libraries. The name of one of more of these underlying libraries may be familiar to you if you have installed or compiled any of Python’s numerical libraries such as SciPy and NumPy.

Linear Algebra and Statistics

Linear algebra is a valuable tool in other branches of mathematics, especially statistics.

Usually students studying statistics are expected to have seen at least one semester of linear algebra (or applied algebra) at the undergraduate level.

— Page xv, Linear Algebra and Matrix Analysis for Statistics, 2014.

The impact of linear algebra is important to consider, given the foundational relationship both fields have with the field of applied machine learning.

Some clear fingerprints of linear algebra on statistics and statistical methods include:

  • Use of vector and matrix notation, especially with multivariate statistics.
  • Solutions to least squares and weighted least squares, such as for linear regression.
  • Estimates of mean and variance of data matrices.
  • The covariance matrix that plays a key role in multinomial Gaussian distributions.
  • Principal component analysis for data reduction that draws many of these elements together.

As you can see, modern statistics and data analysis, at least as far as the interests of a machine learning practitioner are concerned, depend on the understanding and tools of linear algebra.

Applications of Linear Algebra

As linear algebra is the mathematics of data, the tools of linear algebra are used in many domains.

In his classical book on the topic titled “Introduction to Linear Algebra“, Gilbert Strang provides a chapter dedicated to the applications of linear algebra. In it, he demonstrates specific mathematical tools rooted in linear algebra. Briefly they are:

  • Matrices in Engineering, such as a line of springs.
  • Graphs and Networks, such as analyzing networks.
  • Markov Matrices, Population, and Economics, such as population growth.
  • Linear Programming, the simplex optimization method.
  • Fourier Series: Linear Algebra for functions, used widely in signal processing.
  • Linear Algebra for statistics and probability, such as least squares for regression.
  • Computer Graphics, such as the various translation, rescaling and rotation of images.

Another interesting application of linear algebra is that it is the type of mathematics used by Albert Einstein in parts of his theory of relativity. Specifically tensors and tensor calculus. He also introduced a new type of linear algebra notation to physics called Einstein notation, or the Einstein summation convention.

Linear Algebra Tutorials

Are you looking for some help to get started with linear algebra, then take a look at some of these tutorials:

Extensions

This section lists some ideas for extending the tutorial that you may wish to explore.

  • Search books and the web for 5 quotations defining the field of linear algebra.
  • Research and list 5 more applications or uses of linear algebra in the field probability and statistics.
  • List and write short definitions for 10 terms used in the description of linear algebra.

If you explore any of these extensions, I’d love to know.

Further Reading

This section provides more resources on the topic if you are looking to go deeper.

Books

Articles

Related Posts

Summary

In this tutorial, you discovered a gentle introduction to linear algebra from a machine learning perspective.

Specifically, you learned:

  • Linear algebra is the mathematics of data.
  • Linear algebra has had a marked impact on the field of statistics.
  • Linear algebra underlies many practical mathematical tools, such as Fourier series and computer graphics.

Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.

Get a Handle on Linear Algebra for Machine Learning!

Linear Algebra for Machine Learning

Develop a working understand of linear algebra

...by writing lines of code in python

Discover how in my new Ebook:
Linear Algebra for Machine Learning

It provides self-study tutorials on topics like:
Vector Norms, Matrix Multiplication, Tensors, Eigendecomposition, SVD, PCA and much more...

Finally Understand the Mathematics of Data

Skip the Academics. Just Results.

See What's Inside

28 Responses to A Gentle Introduction to Linear Algebra

  1. Avatar
    Dan January 26, 2018 at 7:12 pm #

    Thanks Jason for the ongoing contributions you make – this site is a wealth of resources for me to learn to apply ML in my academic research from scratch

  2. Avatar
    David January 29, 2018 at 4:33 am #

    Jason,

    Thank you for the linear algebra introduction, I found it very useful.

    As part of your Extensions exercise, I wrote a blog about my own linear algebra exploration https://agentcurry.com/2018/01/linear-algebra-exploration

    I only recently found your website and find it to be a great resource for machine learning. Thank you for your contributions.

    ~David

  3. Avatar
    SiVa January 30, 2018 at 4:32 pm #

    Hey brownlee,

    Thanks for the linear algebra introduction.

  4. Avatar
    Sameer Madamaiah July 11, 2018 at 2:30 pm #

    Thank you so much for this information

  5. Avatar
    Amit Mukherjee July 14, 2018 at 2:12 am #

    I am learning linear algebra to implement machine learning solutions for solving business problems.

    Thanks Jason for the systematic organization of the course.

  6. Avatar
    alvhin August 6, 2018 at 11:51 pm #

    sir can you help about my topic mastery level in linear algebra thesis tnx

  7. Avatar
    Buba March 14, 2019 at 5:35 pm #

    Problems of this form are generally challenging to solve because there are more unknowns (here we have 2) than there are equations to solve (here we have 3).

    But 2 is less than 3.

    • Avatar
      Jason Brownlee March 15, 2019 at 5:26 am #

      Yes, that makes 2 x 3 unknowns across the 3 equations. My point was that the unknowns scale with the number of equations.

  8. Avatar
    Saurabh April 22, 2019 at 5:57 am #

    Hello Sir,

    I am an Arts student, with a very little exposure to Maths, At the same time I am interseted in learning Analytics and Automation.

    As a beginner please can you suggest me how should I begin. I order to learn and understand Analytics and Automation.

    • Avatar
      Jason Brownlee April 22, 2019 at 6:26 am #

      I would recommend diving into the specific area that interests you.

  9. Avatar
    Harshdeep Prajapati June 25, 2019 at 3:19 pm #

    Hi sir, I’m really want to be a part of data revolution that will change the world. I’m really want to learn data analytics and BIG DATA that involves in a business decisions process. But the problem is that, I hate maths since my school days. But I think now I’ve to learn it. At the time commenting this, I’ve Zero knowledge of maths. However, I’ve been working in web development in python Django since mid 2016. And Knowledge of AWS. Technology wise I’ve sufficient knowledge to dive into Data Analyst and BIG DATA field. But, Maths Scares Me. Please Help.

  10. Avatar
    Alaa September 5, 2019 at 3:27 am #

    Thank you so much your books help me to learn a lot about machine learning and other subjects

  11. Avatar
    Matthew January 17, 2022 at 11:03 pm #

    Thx this really helped, I am still wondering, how is algebra used in statistics?

  12. Avatar
    Romaloh January 17, 2022 at 11:31 pm #

    The info helped a lot but I am wondering how algebra is used in statistics?

  13. Avatar
    FZS May 9, 2022 at 12:12 am #

    Thank you for explaining. This is very well written. Appreciate all your hard work in educating so many.

    • Avatar
      James Carmichael May 9, 2022 at 11:01 am #

      Great feedback FZS!

  14. Avatar
    Paul Leiby October 14, 2022 at 1:04 am #

    Thanks for doing this but I have a question seeking clarification, following on that of @Buba.
    The example you give (3 eqns and 2 unknowns) is actually _overspecified_, as are most linear regression problems. The example does indeed have “FEWER [not more] unknowns (here we have 2) than there are equations to solve (here we have 3).”
    You subsequently say “Systems describing problems we are often interested in (such as a linear regression) can have an infinite number of solutions.”

    Isn’t it actually that case the linear regression problems typically have _more_ observations (more equations or rows in A) than unknowns (parameters X to be estimated or columns in A)? These problems, being over-specified, more typically have NO solutions that exactly solve all equations (without error, as you say), not an infinite number of solutions.

    Isn’t it more typically the underspecified problems, where the number of equations/observations is less than the number of unknowns, that can have an infinite number of solutions?

    Luckily, linear algebra is helpful for both overspecified and upderspecified problems!
    Thank you.

Leave a Reply