5 Mistakes Programmers Make when Starting in Machine Learning

There is no right way to get into machine learning. We all learn slightly different ways and have different objectives of what we want to do with or for machine learning.

A common goal is to get productive with machine learning quickly. If that is your goal then this post highlights five common mistakes programmers make on the path to quickly being productive machine learning practitioners.

Mistakes Programmers Make when Starting in Machine Learning

Mistakes Programmers Make when Starting in Machine Learning
Photo credited to aarontait, some rights reserved.

1. Put Machine Learning on a pedestal

Machine learning is just another bag of techniques that you can use to create solutions to complex problems.

Because it is a burgeoning field, machine learning is typically communicated in academic publications and textbooks for postgraduate students. This gives it the appearance that it is elite and impenetrable.

A mindset shift is required to be effective at machine learning, from technology to process, from precision to “good enough”, but the same could be said for other complex methods that programmers are interested in adopting.

2. Write Machine Learning Code

Starting in machine learning by writing code can make things difficult because it means that you are solving at least two problems rather than one: how a techniques so that you can implement it and how to apply the technique to a given problem.

It is much easier to work on one problem at a time and leverage machine learning and statistical environments and libraries of algorithms to learn how to apply a technique to a problem. This allows you to spot check and tune a variety of algorithms relatively quickly and tune the one or two that look promising rather than investing large amounts of time interpreting ambiguous research papers containing algorithm descriptions.

Implementing an algorithm can be treated as a separate project to be completed at a later time, such as for a learning exercise or if the prototype system needs to me put into operations. Learn one thing at a time, I recommend starting with a GUI based machine learning framework whether you’re a programmer or not.

3. Doing Things Manually

A process surrounds applied machine learning including problem definition, data preparation and presentation of results, among other tasks. These processes along with the testing and tuning of algorithms can and should be automated.

Automation is a big part of modern software development for builds, tests and deployment. There is great advantage in scripting data preparation, algorithm testing and tuning and the preparation of results in order to gain the benefits of rigor and speed of improvement. Remember and reuse the lessons learned in professional software development.

The failure to start with automation (such as Makefiles or similar build system) is likely due to the fact that many programmers come to machine learning from books and courses that have less focus on the applied nature of the field. In fact, brining automation to applied machine learning is a huge opportunity for programmers.

4. Reinvent Solutions to Common Problems

Hundreds and thousands of people have likely implemented the algorithm you are implementing before you or have solved a problem type similar to the problem you are solving, exploit their lessons learned.

There is a wealth of knowledge out there of solving applied machine learning. Granted much of it may be tied up in books and research publications, but you can access it. Do your homework and search Google, Google Books, Google Scholar and reach out to the machine learning community.

If you are implementing an algorithm:

  • Do you have to implement it? Can you reuse an existing open source algorithm implementation in a library or tool?
  • Do you have to implement from scratch? Can you code review, learn from or port an existing open source implementation?
  • Do you have to interpret the canonical algorithm description? Are there algorithm descriptions in other books, papers, theses, or blog posts that you can review and learn from?
Reinvent Solutions to Common Problems

Photo credited to justgrimes, some rights reserved

If you are addressing a problem:

  • Do you have to test all algorithms on the problem? Can you exploit studies on this or similar problem instances of the same general type that suggest algorithms and algorithm classes that perform well?
  • Do you have to collect your own data? Are their publicly available data sets or APIs that you can use directly or as a proxy for your problem to quickly learn which methods are likely to perform well?
  • Do you have to optimize the parameters of the algorithm? Are the heuristics you can use for configuring the algorithm presented in papers or studies of the algorithm?

What would be your strategy if you have a problem with a programming library or a specific type of data structure? Use the same tactics in the field of machine learning. Reach out to the community and ask for resources that you may be able to exploit to accelerate your learning and progress on your project. Consider forums and Q&A sites to start with and contact academics and specialists as the next step.

5. Ignoring the Math

You do not need the mathematical theory to get started, but maths is a big part of machine learning. The reason for this is it provides perhaps the most efficient and unambiguous way to describe problems and the behaviors of systems.

Ignoring the mathematical treatments of algorithms can lead to problems such as having a limited understanding of a method or adopting a limited interpretation of an algorithm. For example, many machine learning algorithms have an optimization at their core that is incrementally updated. Knowing about the nature of the optimization being solved (is the function convex) allows you to use efficient optimization algorithms that exploit this knowledge.

Internalizing the mathematical treatment of algorithms is slow and comes with mastery. Particularly if you are implementing advanced algorithms from scratch including the internal optimization algorithms, take the time to learn the algorithm from mathematical perspective.

Summary

In this post you learned about 5 common mistakes that programmers make when getting started in machine learning. The five lessons are:

  • Don’t put machine learning on a pedestal
  • Don’t write machine learning code
  • Don’t do things manually
  • Don’t reinvent solutions to common problems
  • Don’t ignore the math

UPDATE: Continue the conversation on HackerNews and DataTau.

10 Responses to 5 Mistakes Programmers Make when Starting in Machine Learning

  1. guy January 29, 2014 at 9:33 am #

    Typo in the summary:

    “Don’t put machine learning is put on a pedestal”

  2. Anubhav January 29, 2014 at 5:49 pm #

    Did you mean Don’t ignore the math in 5?

  3. Lelala January 29, 2014 at 8:31 pm #

    Heck, i will try it – even if i suck at math, sounds somewhat motivatin “you do not need the full theory” :-)

  4. Mathias Brandewinder January 31, 2014 at 4:07 am #

    100% agree with point 3, in particular your comment that it is a great opportunity for software engineers. One difference between statistics 20 years ago and machine learning today is that it has become a software engineering problem as much as it is a stats problem – and absolutely, automation is a life-saver there, if only because it allows you to replicate results later. I have had some very unpleasant moments where I “lost” a model because it wasn’t scripted, and I was unable to replicate the steps that led to a particular result a couple of days later.
    I am on the fence on point 2. On one hand I agree that implementing from scratch algorithms is a great way to waste time; on the other hand, you could make the argument that just like you need to dig a bit into the math to truly understand an algorithm, knowing how the implementation work is another angle into understanding what is really going on (and potentially easier than the math route for a software engineer). I am always a bit worried about people applying existing implementations without fully understanding what is happening. I would say, by default if there is an existing implementation and you know the algorithm well, use it, but otherwise, if you want to learn an approach, dig into the math AND the implementation – you don’t fully understand an algorithm until you’ve attempted to write it yourself ;)

  5. Martin February 3, 2014 at 6:08 am #

    “If that is your goal than this post highlights.. ” – should be “then”.

  6. Martin February 3, 2014 at 8:46 am #

    Two more:
    “learning by writing code can be make things difficult”
    and
    “Hundreds and thousands of people have likely implemented the algorithm you are implemented “

    • jasonb February 3, 2014 at 9:14 am #

      Thanks a lot martin, fixed. I need to take a lot more care proofing (or hire you!).

  7. Links February 4, 2014 at 12:31 am #

    I arrived to a similar conclusions when I started working on Machine Learning algorithms a few years ago, and coming from a heavy database/development background, it felt like a monumental task to take on.

    With all the available libraries and frameworks, It is surprising how much of the mathematical modelling is already abstracted for you.

    I do recommend tho subscribing to a few Coursera lectures on the topic, which provides great insight into benchmarking, measurement and the whole “researchy” approach (design => build => measure => iterate) towards building a machine learning system.

    Very nice write up!

Leave a Reply