Basic Concepts in Machine Learning

What are the basic concepts in machine learning?

I found that the best way to discover and get a handle on the basic concepts in machine learning is to review the introduction chapters to machine learning textbooks and to watch the videos from the first model in online courses.

Pedro Domingos is a lecturer and professor on machine learning at the University of Washing and author of a new book titled “The Master Algorithm: How the Quest for the Ultimate Learning Machine Will Remake Our World“.

Domingos has a free course on machine learning online at courser titled appropriately “Machine Learning“. The videos for each module can be previewed on Coursera any time.

In this post you will discover the basic concepts of machine learning summarized from Week One of Domingos’ Machine Learning course.

Basic Concepts in Machine Learning

Basic Concepts in Machine Learning
Photo by Travis Wise, some rights reserved.

Machine Learning

The first half of the lecture is on the general topic of machine learning.

What is Machine Learning?

Why do we need to care about machine learning?

A breakthrough in machine learning would be worth ten Microsofts.

Bill Gates, Former Chairman, Microsoft

Machine Learning is getting computers to program themselves. If programming is automation, then machine learning is automating the process of automation.

Writing software is the bottleneck, we don’t have enough good developers. Let the data do the work instead of people. Machine learning is the way to make programming scalable.

  • Traditional Programming: Data and program is run on the computer to produce the output.
  • Machine Learning: Data and output is run on the computer to create a program. This program can be used in traditional programming.

Machine learning is like farming or gardening. Seeds is the algorithms, nutrients is the data, the gardner is you and plants is the programs.

Traditional Programming vs Machine Learning

Traditional Programming vs Machine Learning

Applications of Machine Learning

Sample applications of machine learning:

  • Web search: ranking page based on what you are most likely to click on.
  • Computational biology: rational design drugs in the computer based on past experiments.
  • Finance: decide who to send what credit card offers to. Evaluation of risk on credit offers. How to decide where to invest money.
  • E-commerce:  Predicting customer churn. Whether or not a transaction is fraudulent.
  • Space exploration: space probes and radio astronomy.
  • Robotics: how to handle uncertainty in new environments. Autonomous. Self-driving car.
  • Information extraction: Ask questions over databases across the web.
  • Social networks: Data on relationships and preferences. Machine learning to extract value from data.
  • Debugging: Use in computer science problems like debugging. Labor intensive process. Could suggest where the bug could be.

What is your domain of interest and how could you use machine learning in that domain?

Key Elements of Machine Learning

There are tens of thousands of machine learning algorithms and hundreds of new algorithms are developed every year.

Every machine learning algorithm has three components:

  • Representation: how to represent knowledge. Examples include decision trees, sets of rules, instances, graphical models, neural networks, support vector machines, model ensembles and others.
  • Evaluation: the way to evaluate candidate programs (hypotheses). Examples include accuracy, prediction and recall, squared error, likelihood, posterior probability, cost, margin, entropy k-L divergence and others.
  • Optimization: the way candidate programs are generated known as the search process. For example combinatorial optimization, convex optimization, constrained optimization.

All machine learning algorithms are combinations of these three components. A framework for understanding all algorithms.

Types of Learning

There are four types of machine learning:

  • Supervised learning: (also called inductive learning) Training data includes desired outputs.  This is spam this is not, learning is supervised.
  • Unsupervised learning: Training data does not include desired outputs. Example is clustering. It is hard to tell what is good learning and what is not.
  • Semi-supervised learning: Training data includes a few desired outputs.
  • Reinforcement learning: Rewards from a sequence of actions. AI types like it, it is the most ambitious type of learning.

Supervised learning is the most mature, the most studied and the type of learning used by most machine learning algorithms. Learning with supervision is much easier than learning without supervision.

Inductive Learning is where we are given examples of a function in the form of data (x) and the output of the function (f(x)). The goal of inductive learning is to learn the function for new data (x).

  • Classification: when the function being learned is discrete.
  • Regression: when the function being learned is continuous.
  • Probability Estimation: when the output of the function is a probability.

Machine Learning in Practice

Machine learning algorithms are only a very small part of using machine learning in practice as a data analyst or data scientist. In practice, the process often looks like:

  1. Start Loop
    1. Understand the domain, prior knowledge and goals. Talk to domain experts. Often the goals are very unclear. You often have more things to try then you can possibly implement.
    2. Data integration, selection, cleaning and pre-processing. This is often the most time consuming part. It is important to have high quality data. The more data you have, the more it sucks because the data is dirty. Garbage in, garbage out.
    3. Learning models. The fun part. This part is very mature. The tools are general.
    4. Interpreting results. Sometimes it does not matter how the model works as long it delivers results. Other domains require that the model is understandable. You will be challenged by human experts.
    5. Consolidating and deploying discovered knowledge. The majority of projects that are successful in the lab are not used in practice. It is very hard to get something used.
  2. End Loop

It is not a one-shot process, it is a cycle. You need to run the loop until you get a result that you can use in practice. Also, the data can change, requiring a new loop.

Inductive Learning

The second part of the lecture is on the topic of inductive learning. This is the general theory behind supervised learning.

What is Inductive Learning?

From the perspective of inductive learning, we are given input samples (x) and output samples (f(x)) and the problem is to estimate the function (f). Specifically, the problem is to generalize from the samples and the mapping to be useful to estimate the output for new samples in the future.

In practice it is almost always too hard to estimate the function, so we are looking for very good approximations of the function.

Some practical examples of induction are:

  • Credit risk assessment.
    • The x is the properties of the customer.
    • The f(x) is credit approved or not.
  • Disease diagnosis.
    • The x are the properties of the patient.
    • The f(x) is the disease they suffer from.
  • Face recognition.
    • The x are bitmaps of peoples faces.
    • The f(x) is to assign a name to the face.
  • Automatic steering.
    • The x are bitmap images from a camera in front of the car.
    • The f(x) is the degree the steering wheel should be turned.

When Should You Use Inductive Learning?

There are problems where inductive learning is not a good idea. It is important when to use and when not to use supervised machine learning.

4 problems where inductive learning might be a good idea:

  • Problems where there is no human expert. If people do not know the answer they cannot write a program to solve it. These are areas of true discovery.
  • Humans can perform the task but no one can describe how to do it. There are problems where humans can do things that computer cannot do or do well. Examples include riding a bike or driving a car.
  • Problems where the desired function changes frequently. Humans could describe it and they could write a program to do it, but the problem changes too often. It is not cost effective. Examples include the stock market.
  • Problems where each user needs a custom function. It is not cost effective to write a custom program for each user. Example is recommendations of movies or books on Netflix or Amazon.

The Essence of Inductive Learning

We can write a program that works perfectly for the data that we have. This function will be maximally overfit. But we have no idea how well it will work on new data, it will likely be very badly because we may never see the same examples again.

The data is not enough. You can predict anything you like. And this would be naive assume nothing about the problem.

In practice we are not naive. There is an underlying problem and we are interested in an accurate approximation of the function. There is a double exponential number of possible classifiers in the number of input states. Finding a good approximate for the function is very difficult.

There are classes of hypotheses that we can try. That is the form that the solution may take or the representation. We cannot know which is most suitable for our problem before hand. We have to use experimentation to discover what works on the problem.

Two perspectives on inductive learning:

  • Learning is the removal of uncertainty. Having data removes some uncertainty. Selecting a class of hypotheses we are removing more uncertainty.
  • Learning is guessing a good and small hypothesis class. It requires guessing. We don’t know the solution we must use a trial and error process. If you knew the domain with certainty, you don’t need learning. But we are not guessing in the dark.

You could be wrong.

  • Our prior knowledge could be wrong.
  • Our guess of the hypothesis class could be wrong.

In practice we start with a small hypothesis class and slowly grow the hypothesis class until we get a good result.

A Framework For Studying Inductive Learning

Terminology used in machine learning:

  • Training example: a sample from x including its output from the target function
  • Target function: the mapping function f from x to f(x)
  • Hypothesis: approximation of f, a candidate function.
  • Concept: A boolean target function, positive examples and negative examples for the 1/0 class values.
  • Classifier: Learning program outputs a classifier that can be used to classify.
  • Learner: Process that creates the classifier.
  • Hypothesis space: set of possible approximations of f that the algorithm can create.
  • Version space: subset of the hypothesis space that is consistent with the observed data.

Key issues in machine learning:

  • What are good hypothesis space?
  • What algorithms work with that space?
  • What can I do to optimize accuracy on unseen data?
  • How do we have confidence in the model?
  • Are there learning problems that are computationally intractable?
  • How can we formulate application problems as machine learning problems?

There are 3 concerns for a choosing a hypothesis space space:

  • Size: number of hypotheses to choose from
  • Randomness: stochastic or deterministic
  • Parameter: the number and type of parameters

There are 3 properties by which you could choose an algorithm:

  • Search procedure
    • Direct computation: No search, just calculate what is needed.
    • Local: Search though the hypothesis space to refine the hypothesis.
    • Constructive: Build the hypothesis piece by piece.
  • Timing
    • Eager: Learning performed up front. Most algorithms are eager.
    • Lazy: Learning performed at the time that it is needed
  • Online vs Batch
    • Online: Learning based on each pattern as it is observed.
    • Batch: Learning over groups of patterns. Most algorithms are batch.


In this post you discovered the basic concepts in machine learning.

In summary, these were:

  • What is machine learning?
  • Applications of Machine Learning
  • Key Elements of Machine Learning
  • Types of Learning
  • Machine Learning in Practice
  • What is Inductive Learning?
  • When Should You Use Inductive Learning?
  • The Essence of Inductive Learning
  • A Framework For Studying Inductive Learning

These are the basic concepts that are covered in the introduction to most machine learning courses and in the opening chapters of any good textbook on the topic.

Although targeted at academics, as a practitioner, it is useful to have a firm footing in these concepts in order to better understand how machine learning algorithms behave in the general sense.

113 Responses to Basic Concepts in Machine Learning

  1. Avatar
    Deepak July 17, 2016 at 2:28 pm #

    It is indeed very good article. I am beginner to Machine learning and this article helped me give basic information. Thank You Jason.

  2. Avatar
    Lal Thomas September 5, 2016 at 5:27 pm #

    Very nice introduction…

    • Avatar
      Jason Brownlee September 6, 2016 at 8:03 am #

      I’m glad you found it useful Lal.

    • Avatar
      Ms Anupama Deshpande February 18, 2022 at 7:42 am #

      Good to start

  3. Avatar
    Lal Thomas September 5, 2016 at 7:28 pm #

    @Jason I found a typo ‘martin’ which should be margin I think…

  4. Avatar
    Namnnb September 28, 2016 at 5:08 pm #

    This article is very useful, certainly. But as a newbie, I see that the final part, from framework for studying inductive learning is quite hard to understand because of too much new concepts. I know I have to learn more.
    Many thanks, Jason.
    Best wishes for you and your family.

  5. Avatar
    soufiane October 20, 2016 at 7:55 pm #

    @Jason Very useful article. thank you.

    • Avatar
      Jason Brownlee October 21, 2016 at 8:34 am #

      I’m glad to hear that soufiane.

      Which topics were most interesting?

  6. Avatar
    Anees Ahmed Awan November 13, 2016 at 6:53 am #

    Thnx Jason Brownlee Nice and Interesting Article …..Very help full.

  7. Avatar
    Jerry November 28, 2016 at 3:28 pm #

    Thanks for this AWESOME introduction to machine learning! I’ve always been interested in the subject but never gotten around to looking into it.

    Now that I’ve graduate from university (masters in physics [lasers]) I’ve a bit more time on my hands as I start to look for a job. I’m increasingly excited at the possibility of going into this large and growing field.

    Thank you again!

  8. Avatar
    Abhishek Dwivedi January 4, 2017 at 5:30 pm #

    As I am beginner so it makes me very confident,whatever I was expecting in machine learning it cover-up all those stuffs . Thank you very much and very helpful for beginner

  9. Avatar
    muhammed gabry February 8, 2017 at 6:10 pm #

    what’s the difference between inductive learning algorithm and analogy learning algorithm?

    • Avatar
      Jason Brownlee February 9, 2017 at 7:24 am #

      I’ve not heard of the analogy learning algorithm, sorry.

  10. Avatar
    Irfan Ahmad February 19, 2017 at 10:02 am #

    Very nice explanation. Can you help me to understand Artificial Intelligence and the difference between ML and AI.


    • Avatar
      Jason Brownlee February 20, 2017 at 9:24 am #

      Hi Irfan,

      ML is a subfield of AI concerned with making inferences from data. AI is concerned with all aspects of intelligence.

      The most useful part of ML I would recommend focusing on is predictive modeling.

  11. Avatar
    DEVENDRA BHADORIYA 16MCA1065 May 11, 2017 at 6:06 am #

    Thanks, MR Jason, such a wonderful knowledge about machine learning. It is very usedful for beginners.

  12. Avatar
    Sagar Gauda May 12, 2017 at 8:25 pm #

    Good Article Indeed, thanks for making me familiar with those new terms., Looking forward for more info.

  13. Avatar
    Majid May 16, 2017 at 12:40 am #

    Hi Jason,

    Like others, I should also say that this is a very nice conceptual introduction. Could you possibly add or refer to a practical and simple example of a solved problem using ML? So far I couldn’t have found any useful source giving sufficient details of different steps for ML, in particular the mathematics behind it.

  14. Avatar
    Addisie May 19, 2017 at 8:14 pm #

    Very nice article, i get relevant basic concepts about ML.

  15. Avatar
    Saksham Sinha May 23, 2017 at 7:52 pm #

    Great article for a beginner like me. I got to learn basic terminology and concepts in ML.

  16. Avatar
    Raghav May 27, 2017 at 3:22 pm #

    Thanks Dr. Jason.

    Your articles are very practical and comprehensive. In a way I am indebted.

    Regards Raghav

  17. Avatar
    Vijay June 16, 2017 at 9:35 pm #

    Great Work

  18. Avatar
    kevin June 24, 2017 at 5:24 am #

    so what do you suggest to go from here to get my feet a bit more wet?

  19. Avatar
    Mohit July 17, 2017 at 8:09 pm #

    Can you explain more regarding selecting an algorithm based on search procedure.

  20. Avatar
    meghana August 7, 2017 at 8:40 pm #

    very nice article. Thanks Jason!!

  21. Avatar
    Jann Krynauw August 8, 2017 at 2:06 pm #

    Thanks Jason, is online simply where batch-size = 1?

  22. Avatar
    Ramesh Veeramallu August 13, 2017 at 1:06 pm #

    Very informative article. Thank you Jason..

  23. Avatar
    pavan August 15, 2017 at 11:44 pm #

    Nice Article Jason.If you have a series of this, please let us know. We will follow this

  24. Avatar
    Kamila August 28, 2017 at 11:01 pm #

    I was about to read it and go further. But I can’t leave your website before saying that you have a great ability to write about very complex things in an easy matter. It shows that you have very big knowlege and with your articles it is easy to understand a lot of things. Continue what you’re doing because you’re doing it good. You were very helpful to me, thanks.

  25. Avatar
    Jeevan September 5, 2017 at 3:21 am #

    Hi Jason,

    Thanks for this wonderful start. May i know the pre-requistes for ML? Do we have need any programming experience?

  26. Avatar
    Dave October 16, 2017 at 10:52 pm #

    Hi Jason,

    Any tips on formulating a good hypothesis with the data owner?

    Mostly, it’s a case of “I want to know ‘this’ – here’s my data”.

  27. Avatar
    Neetirajsinh Chhasatia February 24, 2018 at 12:08 am #

    Can you discriminate following terminologies ?
    -Deep learning
    -Machine learning
    -Neural Network
    -Artificial Intelligence

    How they differ with each other ?

    • Avatar
      Jason Brownlee February 24, 2018 at 9:12 am #

      ML is a subfield of AI.
      DL and NN are the same thing and are a subfield of ML.

  28. Avatar
    Annu Choudhary February 28, 2018 at 6:13 am #

    Useful stuff (Y)

  29. Avatar
    Doug March 11, 2018 at 2:31 am #

    here’s a typo Jason

    “There are problems where inductive learning is not a hood idea”

    Should be “good” instead of “hood”.


  30. Avatar
    Dulaj Chathuranga April 15, 2018 at 5:06 pm #

    I am a newbie. Nice introduction. Thank you.

  31. Avatar
    Shashank May 14, 2018 at 2:01 pm #

    thank you sir
    Helped me for preparing my exams for AI
    I searched the web for inductive learning, got here with the aid of QUORA
    Thanks again!!

  32. Avatar
    Dr.M.Sujithra June 13, 2018 at 2:45 pm #


    well written introduction

    clear cut Definitions

    all the information are at to the point .

    very useful

    thank you so much

    awaiting for exploring more from you materials

  33. Avatar
    Mrb July 9, 2018 at 2:23 pm #

    The fact that the article still resonates with the audience after 2 years speaks on its own. Thank you!

  34. Avatar
    ankur July 18, 2018 at 2:02 am #

    Hi Jason,

    It was a nice and informative article. Could you please explain how version space learning works?

    Thanks and Regards,
    Ankur Gupta

  35. Avatar
    Phan Dat August 29, 2018 at 1:19 pm #

    Thank so much for great introduction!

  36. Avatar
    Lalitha September 21, 2018 at 5:22 pm #

    It is very clear and helpful..

  37. Avatar
    Amit Chaudhary September 23, 2018 at 4:26 pm #

    It’s a great tutorial on ML…

  38. Avatar
    TATHAGATA NATH October 14, 2018 at 5:39 am #

    very useful article .I am beginner in Machine Learning.can you send me other article.

    • Avatar
      Jason Brownlee October 14, 2018 at 6:05 am #


      You can access all of the articles on the blog.

  39. Avatar
    Star-Lord October 29, 2018 at 9:42 am #

    Thank you for the article.

    Some remarks :
    It could be interesting to add real life analogies / concrete examples to : terminology / the algorithm properties …
    Typo at the end ? “patters” instead of “patterns” ?
    Also some information in readers comments could be implemented in the article

  40. Avatar
    sanjeev February 11, 2019 at 5:17 pm #

    great article…

    what are the statistical approach we use in machine clearing while modeling….
    I mean suppose we have an data set,should we have an hypothesis to start with …what are the steps,it would be very helpful ,if you could throw some light on it…

  41. Avatar
    raybamford March 3, 2019 at 10:28 am #

    This is a very nice summary, Jason, thank you for sharing.

    Some additional topics that seem worthy of mention here:
    1. Generalization — the objective of a predictive model is to predict well on new data that the model has never seen, not to “fit” the data we already have.
    2. Model validation – how to assess model performance; dividing data into training, validation, and test sets; cross-validation; avoiding data snooping, selection bias, survivorship bias, look-ahead bias, and more. In my experience, model validation is one of the most challenging aspects of ML (and to do it well may vastly increase the challenges in constructing and managing your datasets)
    3. Bias / variance tradeoff — importance of balancing overfitting (high variance, complex models) and underfitting (high bias, simple models); the more complex the model, the more data that is required to avoid overfitting; deep learning models require vastly more labeled data than traditional (simpler) ML models
    4. Curse of dimensionality — as you increase the number of predictors (independent variables), you need exponentially more data to avoid underfitting; dimensionality reduction techniques
    5. Feature engineering — related to domain expertise and data preparation; with good domain experts, you can often construct features that perform vastly better than the raw data.

    Ok, that’s more than enough. I started my reply intending to mention only generalization and validation … This is such a rich topic! Hope this helps. Thanks again for your great work.

  42. Avatar
    AbdiMo May 22, 2019 at 10:25 am #

    Could you explain the types of error functions used in machine learning systems

  43. Avatar
    Shahbaaz July 29, 2019 at 6:13 pm #

    Hi Jason, this article was very helpful to me but i am beginnner in this feild and i dont even know prgramming please help me out

  44. Avatar
    Zainab Talib Drweesh August 5, 2019 at 9:57 pm #

    Thank you for the article. please can u send me some references about induction learning I needed so badly….

  45. Avatar
    Austin October 14, 2019 at 1:52 am #

    Hi Jason. I enjoyed your article, thanks for writing. I also wrote an article on machine learning that is geared towards beginners at Let me know what you think!

  46. Avatar
    gunasekaran December 10, 2019 at 4:29 pm #

    HI Jason.thanks for this article,and more clear about machine learning with example?

  47. Avatar
    Subhabrata December 19, 2019 at 8:32 pm #

    Very good overview for a beginner. Would you like to share some most commonly asked interview questions on ML? Thanks

    • Avatar
      Jason Brownlee December 20, 2019 at 6:44 am #

      Thanks for the suggestion.

      Sorry, I don’t know about interview questions. I’m an expert in using applied ML to solve problems, not job interviews.

  48. Avatar
    NIkhil January 23, 2020 at 7:39 am #

    Very detailed and informative in a single page

  49. Avatar
    Typo192 January 25, 2020 at 2:56 am #

    There is a typo under “The Essence of Inductive Learning”. The first paragraph has “de” instead of “be”.

  50. Avatar
    Saeid September 18, 2020 at 6:57 am #

    HI, Jason.thanks for this great article. do I need a strong statistical and algebra knowledge if I want to start learning ML?

  51. Avatar
    Jeetech Academy March 7, 2022 at 2:07 pm #

    Machine learning is the coolest field to build a better career. I was looking for some point which help me to become a machine learning. After reading your blog every picture become clear in my mind. Now I can design my career path. Thank you for best career advice. Your blog have fabulous information.

  52. Avatar
    Odipo July 15, 2023 at 1:03 pm #

    Wow, this is my first reading on Machine Learning! And I am seriously delving into it. A great introduction of a subject. Will seek your guide as I walk in the new world. Thanks

Leave a Reply