How to Learn a Machine Learning Algorithm

The question of how to learn a machine learning algorithm has come up a few times on the email list.

In this post I’ll share with you the strategy I have been using for years to learn and build up a structured description of an algorithm in a step-by-step manner that I can add to, refine and refer back to again and again. I even used it to write a book.

This was just a strategy I used personally and I’ve been really surprised by the positive feedback.

Kick-start your project with my new book Master Machine Learning Algorithms, including step-by-step tutorials and the Excel Spreadsheet files for all examples.

Let’s get started.

Algorithm Descriptions are Broken

Learning a machine learning algorithm can be overwhelming. There are so many papers, books and websites describing how the algorithm works mathematically and textually. If you are really lucky you might find a pseudocode description of the algorithm.

Implement a Machine Leaning Algorithm

Implement a Machine Leaning Algorithm
Photo by Nic’s events, some rights reserved

If you are really really lucky you might find some suggested ways to configure the method for different situations. These descriptions are rare and typically buried deep in the original publication or in technical notes by the original authors.

A fact you learn quickly when you want to implement a method from research papers is that algorithms are almost never described in sufficient detail for you to reproduce them. The reasons vary, from the micro-decisions that are left out of the paper, to whole procedures that are summarized ambiguously in text, to symbols that are used inconsistently.

Get your FREE Algorithms Mind Map

Machine Learning Algorithms Mind Map

Sample of the handy machine learning algorithms mind map.

I've created a handy mind map of 60+ algorithms organized by type.

Download it, print it and use it. 


Also get exclusive access to the machine learning algorithms email mini-course.

 

 

Piece it Together

To understand an algorithm you have to piece together an understanding yourself from disparate descriptions. This is the only tactic that I have found to be successful.

Disparate descriptions means resources such as the original descriptions of the method in the primary sources as well as authoritative secondary interpretations made of original descriptions in review papers and books.

It is common for there to be prototype implementations of a method released with the primary sources and reading this code (typically C, FORTRAN, R or Matlab) can be very enlightening for the details you need to reproduce an algorithm.

Algorithm Descriptions

An algorithm is an island of research and in all reality it can be difficult to pin down the canonical definition. For example, is it the version described in the primary source or is it the version that includes all the fixes and enhancements that are “best practice”.

A solution is to consider a given algorithm from multiple perspectives, each of which can serve a different purpose. For example, the abstract information processing description of the algorithm could be realized by a variety of different specific computational implementations.

I like this approach because it defends the need to telescope in on a specific case of the algorithm from many possible cases at each step of the description while also leaving the option open for the description of variations.

There are many descriptions you could use of varying specificity depending on your needs. Some that I like to use include: inspiration for the algorithm, metaphor or analogy for the strategy, information processing objectives, pseudocode and code.

Algorithm Prescriptions

When I started my own research projects, I thought the answer to this problem was to read everything on an algorithm and create the definitive implementation in code. Nice idea perhaps, but code is just one way to communicate an algorithm, and it is limited.

There is more to an algorithm description than the computation. There is meta information around an algorithm that can be invaluable for certain use cases.

For example, usage heuristics for an algorithm are embedded in papers. Having a summary of usage heuristics collected together in one place can mean the difference of getting a good enough result quickly and running sensitivity analysis on the algorithm for days or weeks.

Other examples include the standard experimental datasets used to test the algorithm, the general classes of problem to which the algorithm is suited, and known limitations that have been identified and described for the algorithm.

Design an Algorithm Description Template

An algorithm description template provides a structured way for you to learn about a machine learning algorithm.

You can start with a blank document and list out the section headings for the types of descriptions you need of the algorithm, for example applied, implementation, or your own personal reference cheat sheet.

To figure out what sections to include in your template, list out questions you would like to answer about the algorithm, or algorithms if you are looking to build up a reference. Some questions you could use include:

  • What is the standard and abbreviations used for the algorithm?
  • What is the information processing strategy of the algorithm?
  • What is the objective or goal for the algorithm?
  • What metaphors or analogies are commonly used to describe the behavior of the algorithm?
  • What is the pseudocode or flowchart description of the algorithm?
  • What are the heuristics or rules of thumb for using the algorithm?
  • What classes of problem is the algorithm well suited?
  • What are common benchmark or example datasets used to demonstrate the algorithm?
  • What are useful resources for learning more about the algorithm?
  • What are the primary references or resources in which the algorithm was first described?

Once you have settled on some questions, turn them into section headings.

For each section heading clearly define the requirements of the section and the form that the description in that section will take. I like to include motivating questions for each section that once answered will satisfy the section at the minimum level of detail.

Start Small and Build it Up

The beauty of this approach is that you don’t need to be an expert in the algorithm or in research. As long as you can find some resources that mention the algorithm, you can start to capture notes about an algorithm in the template.

You can start really simply and collect high-level descriptions of the algorithm, its names and abbreviations and the resources you have found and what they talk about. From here you can decide to expand the description further, or not.

You will end up with a one-to-two page description of the algorithm very quickly.

I Use It

I’ve been using algorithm templates for a long time. Some examples where I have found this strategy practically useful include:

  • Implementing machine learning algorithms using a descriptive-focused template.
  • Applying a machine learning algorithm using an applied-focused template.
  • Building a catalog of algorithms to use and refer to using a general purpose template.

In this last case, I turned my catalog into a book of 45 nature inspired algorithms which I published in early 2011. The book is called Clever Algorithms: Nature-Inspired Programming Recipes.

Summary

In this post you learned about using an algorithm description template as a strategy for learning a machine learning algorithm.

You learned that algorithm descriptions are broken and the answer to learning an algorithm effectively is to design an algorithm template that meets your needs and to fill in the template as you read and learn about the algorithm.

You learned that the template is an efficient and structured way to tackle an overwhelming problem.

Discover How Machine Learning Algorithms Work!

Mater Machine Learning Algorithms

See How Algorithms Work in Minutes

...with just arithmetic and simple examples

Discover how in my new Ebook:
Master Machine Learning Algorithms

It covers explanations and examples of 10 top algorithms, like:
Linear Regression, k-Nearest Neighbors, Support Vector Machines and much more...

Finally, Pull Back the Curtain on
Machine Learning Algorithms

Skip the Academics. Just Results.

See What's Inside

27 Responses to How to Learn a Machine Learning Algorithm

  1. Avatar
    Preethi August 20, 2015 at 12:00 pm #

    I need some help in understanding the proofs of ML Algorithms. I’m a beginner and I’m trying to learn these Algorithms and their proofs.
    Please suggest any text books/resources which can use to understand things faster.
    Please share your tips too!

    • Avatar
      Jason Brownlee December 23, 2015 at 9:43 am #

      My best advice is to not start with the poofs of the algorithms.

      Start with a simple descriptive understanding, expand to a pseudocode description, then mathematical, then finally move onto relevant proofs.

      • Avatar
        Chitrang Talaviya January 9, 2016 at 12:34 pm #

        Yup Jason truly said. Appreciate your efforts. Your articles are very useful for me to get into Machine Learning.

    • Avatar
      Robin White March 31, 2016 at 10:02 am #

      Hi Preethi. I suggest you take the machine learning courses that I have taken before. I put the link below for you. I hope it will be helpful to you. http://www.thedevmasters.com/machine-learning-in-python/

      • Avatar
        sujal November 24, 2017 at 11:32 am #

        Hi robin white,

        Will it be beneficial to to spend money on thedevmasters course of machine learning , can you please tell me?

  2. Avatar
    Will September 27, 2016 at 1:16 am #

    Hi Jason!

    Would love to see an example of an algorithm template you have written if possible!

    Thanks

  3. Avatar
    Rabbani October 2, 2017 at 3:08 am #

    page not found when trying to download Algorithms Mind Map

  4. Avatar
    Anu October 11, 2017 at 1:38 am #

    unable to download Algorithms Mind Map

    “Oops, page not found!

    It seems that the page you are looking for does not exist.”

  5. Avatar
    sujal November 21, 2017 at 2:19 pm #

    Hi Jason,

    I am doing Ms in data analytics and learning machine learning & R can you give me some advice from where i have to start as i have no background on statistics so how to understand mathematical part to understand algorithms ?

  6. Avatar
    Jesús Martínez February 17, 2018 at 2:16 am #

    Thanks for the article, Jason! I usually find that the most efficient method to learn anything is to use a metaphor to understand it. Also, learning something with a purpose in mind is much more appealing than going through pages and pages of formulas and code without any underlying reason for doing so. In a nutshell: Learning implementing a side project is much more fun and quick!

    What do you think about it? I’d love to know.

  7. Avatar
    Eli Mead December 17, 2018 at 5:08 am #

    Under the “Piece it Together” section, 2nd paragraph down, you wrote “Desperate descriptions” rather than “Disparate descriptions.”

  8. Avatar
    Noman April 28, 2019 at 6:46 pm #

    hello professor , i want to use ML algorithms for Fault detection or performance optimization of fel cells , i am new in this field and know almost nothing about machine learning.

    • Avatar
      Jason Brownlee April 29, 2019 at 8:20 am #

      Perhaps start with some google searches on “fault detection” and review some recent papers to see what state of the art methods might be?

  9. Avatar
    DK November 25, 2019 at 7:19 pm #

    HI Jason,

    I am having a hard time understanding Machine learning algorithms , i want to learn the mathematics behind the algos , can you suggest some good books or courses that go over the top algorithms with proper mathematics on how they work ?

    Appreciate your help on this !

  10. Avatar
    Maria Horn March 30, 2020 at 5:11 pm #

    The brand new idea is here). I have read the post with great satisfaction and also could know something new that I will use for my further requirements.
    The guide is bright and clear, with no further worthless facts or else,
    it reminded me https://davidsmcdonough.com/how-to-write-a-compare-and-contrast-essay/. The language
    is both brilliant and vivid, so the more I read, the more I do enjoy it!
    Anyway, the information is quite cutting edge, so just like it.

  11. Avatar
    Becky Henderson May 1, 2020 at 4:43 am #

    Jason,

    I just wanted to thank you for your articles and guidance. I am a newcomer to the AI/ML arena and your published information has given me great places to start and get my mind working towards ways to come up with solutions for my current projects. Much appreciated and thank you!

Leave a Reply