How Machine Learning Algorithms Work (they learn a mapping of input to output)

How do machine learning algorithms work?

There is a common principle that underlies all supervised machine learning algorithms for predictive modeling.

In this post you will discover how machine learning algorithms actually work by understanding the common principle that underlies all algorithms.

How Machine Learning Algorithms Work

How Machine Learning Algorithms Work
Photo by GotCredit, some rights reserved.

Let’s get started.

Learning a Function

Machine learning algorithms are described as learning a target function (f) that best maps input variables (X) to an output variable (Y).

Y = f(X)

This is a general learning task where we would like to make predictions in the future (Y) given new examples of input variables (X).

We don’t know what the function (f) looks like or it’s form. If we did, we would use it directly and we would not need to learn it from data using machine learning algorithms.

It is harder than you think. There is also error (e) that is independent of the input data (X).

Y = f(X) + e

This error might be error such as not having enough attributes to sufficiently characterize the best mapping from X to Y. This error is called irreducible error because no matter how good we get at estimating the target function (f), we cannot reduce this error.

This is to say, that the problem of learning a function from data is a difficult problem and this is the reason why the field of machine learning and machine learning algorithms exist.

Get your FREE Algorithms Mind Map

Machine Learning Algorithms Mind Map

Sample of the handy machine learning algorithms mind map.

I've created a handy mind map of 60+ algorithms organized by type.

Download it, print it and use it. 

Download For Free


Also get exclusive access to the machine learning algorithms email mini-course.

 

 

Learning a Function To Make Predictions

The most common type of machine learning is to learn the mapping Y=f(X) to make predictions of Y for new X.

This is called predictive modeling or predictive analytics and our goal is to make the most accurate predictions possible.

As such, we are not really interested in the shape and form of the function (f) that we are learning, only that it makes accurate predictions.

We could learn the mapping of Y=f(X) to learn more about the relationship in the data and this is called statistical inference. If this were the goal, we would use simpler methods and value understanding the learned model and form of (f) above making accurate predictions.

When we learn a function (f) we are estimating its form from the data that we have available. As such, this estimate will have error. It will not be a perfect estimate for the underlying hypothetical best mapping from Y given X.

Much time in applied machine learning is spent attempting to improve the estimate of the underlying function and in term improve the performance of the predictions made by the model.

Techniques For Learning a Function

Machine learning algorithms are techniques for estimating the target function (f) to predict the output variable (Y) given input variables (X).

Different representations make different assumptions about the form of the function being learned, such as whether it is linear or nonlinear.

Different machine learning algorithms make different assumptions about the shape and structure of the function and how best to optimize a representation to approximate it.

This is why it is so important to try a suite of different algorithms on a machine learning problem, because we cannot know before hand which approach will be best at estimating the structure of the underlying function we are trying to approximate.

Summary

In this post you discovered the underlying principle that explains the objective of all machine learning algorithms for predictive modeling.

You learned that machine learning algorithms work to estimate the mapping function (f) of output variables (Y) given input variables (X), or Y=f(X).

You also learned that different machine learning algorithms make different assumptions about the form of the underlying function. And that when we don’t know much about the form of the target function we must try a suite of different algorithms to see what works best.

Do you have any questions about how machine learning algorithms or this post? Leave a comment and ask your question and I will do my best to answer it.


Frustrated With Machine Learning Math?

Mater Machine Learning Algorithms

See How Algorithms Work in Minutes

…with just arithmetic and simple examples

Discover how in my new Ebook: Master Machine Learning Algorithms

It covers explanations and examples of 10 top algorithms, like:
Linear Regression, k-Nearest Neighbors, Support Vector Machines and much more…

Finally, Pull Back the Curtain on
Machine Learning Algorithms

Skip the Academics. Just Results.

Click to learn more.


20 Responses to How Machine Learning Algorithms Work (they learn a mapping of input to output)

  1. Sreepal September 20, 2016 at 7:19 pm #

    Is it possible to learn Machine learning without prior guidance? I don’t have enough physical resources like a professor or a expert in Machine learning. I was just interested in learn programming which about prediction and feeding the data into computer to make to predict the circumstances and predict the future to take the right decisions. Well, as normal student having limited resources, is it really possible to dive into Machine learning. What are the prerequisites for Machine learning? What is the best alternative to get a live guidance to take Machine learning? Please help!

  2. Adly Thebaud December 1, 2017 at 7:24 am #

    Great read!
    I am just getting started in Machine Learning. My question after reading is, do the machine learning algorithms try to alter the mapping function f(X) to reduce error, or do they only try to create a mapping function from given data sets of (X,Y)? Or is it both?

    Also, what does the mapping function look like? For a standard set of X and Y variables that are floating point numbers, would it be something of the form (Y = mX + b)? More quadratic or even approaching differential equations or linear algebra? Does the mapping function come from trying to make a line of best fit on a graph from a set of data?

    Sorry for all my questions. I am eager to learn!
    Thank you!

    • Jason Brownlee December 1, 2017 at 7:47 am #

      Depends on the algorithm, often algorithms seek a mapping with min error.

      Algorithms like knn have no such optimization or functional form.

  3. Fauzan Taufik December 18, 2017 at 11:21 pm #

    How we know the value of error, since we dont know exactly the value of Y?

  4. Gowtham January 24, 2018 at 3:12 am #

    i didnt know about machine learning but i take the college project related to machine learning so i now started to learn machine learning its intresting and very well i love maths i learned python day and night watching tutorials and learn from websites

  5. sandhya February 20, 2018 at 9:04 pm #

    i am confused …………which algorithm is gives best results in privacy preserving for different data sets…….

    • Jason Brownlee February 21, 2018 at 6:39 am #

      It depends on the data. My advice is to test on your data and discover what works best.

  6. Shubhanker March 20, 2018 at 5:43 am #

    Sir, as referred to in the article the statistical inference, that is the mathematical relationship between the input data and the predicted values…or the mathematical function…how much of an importance does it have for an ML engineer? Thank you for your help!!!

    • Jason Brownlee March 20, 2018 at 6:33 am #

      A conceptual understanding of this relationship is of the highest importance for getting the most out of a given prediction problem.

      It provides a framework for thinking about your problem.

  7. tayeb May 2, 2018 at 3:27 am #

    Good evening
    if you can help me with the code and schema of algorithm “LSTM” because I need it in my own research in the master certificate
    Thank you

  8. Prateek sharma May 17, 2018 at 4:07 am #

    Good evening, I am a learner wants to start my work in the field of AI.And I have done some part in Soft computing.kindly guide me so that I can start my work as a beginner in the field of AI.

  9. Abhijit Mishra July 24, 2018 at 2:54 am #

    Sir, I need some basic operation of RBF kernel based learning and on Reproducing kernel hilbert spaces (RKHS) using GRAM Matrix along with their MATLAB implementation for my research work in Ph.D. Kindly guide me on above topics.

  10. Ajay September 4, 2018 at 6:48 pm #

    Hello sir,

    What is representation in above context?
    What is meant by shape and form of function?

    Thank You

    • Jason Brownlee September 5, 2018 at 6:32 am #

      We don’t know the shape and form of the function, we use algorithms to approximate it by minimizing loss.

      If we did know about the function, we would just use it directly and there would be no need to learn anything.

Leave a Reply