[New Book] Click to get The Beginner's Guide to Data Science!
Use the offer code 20offearlybird to get 20% off. Hurry, sale ends soon!

Automated Machine Learning (AutoML) Libraries for Python

AutoML provides tools to automatically discover good machine learning model pipelines for a dataset with very little user intervention.

It is ideal for domain experts new to machine learning or machine learning practitioners looking to get good results quickly for a predictive modeling task.

Open-source libraries are available for using AutoML methods with popular machine learning libraries in Python, such as the scikit-learn machine learning library.

In this tutorial, you will discover how to use top open-source AutoML libraries for scikit-learn in Python.

After completing this tutorial, you will know:

  • AutoML are techniques for automatically and quickly discovering a well-performing machine learning model pipeline for a predictive modeling task.
  • The three most popular AutoML libraries for Scikit-Learn are Hyperopt-Sklearn, Auto-Sklearn, and TPOT.
  • How to use AutoML libraries to discover well-performing models for predictive modeling tasks in Python.

Let’s get started.

Automated Machine Learning (AutoML) Libraries for Python

Automated Machine Learning (AutoML) Libraries for Python
Photo by Michael Coghlan, some rights reserved.

Tutorial Overview

This tutorial is divided into four parts; they are:

  1. Automated Machine Learning
  2. Auto-Sklearn
  3. Tree-based Pipeline Optimization Tool (TPOT)
  4. Hyperopt-Sklearn

Automated Machine Learning

Automated Machine Learning, or AutoML for short, involves the automatic selection of data preparation, machine learning model, and model hyperparameters for a predictive modeling task.

It refers to techniques that allow semi-sophisticated machine learning practitioners and non-experts to discover a good predictive model pipeline for their machine learning task quickly, with very little intervention other than providing a dataset.

… the user simply provides data, and the AutoML system automatically determines the approach that performs best for this particular application. Thereby, AutoML makes state-of-the-art machine learning approaches accessible to domain scientists who are interested in applying machine learning but do not have the resources to learn about the technologies behind it in detail.

— Page ix, Automated Machine Learning: Methods, Systems, Challenges, 2019.

Central to the approach is defining a large hierarchical optimization problem that involves identifying data transforms and the machine learning models themselves, in addition to the hyperparameters for the models.

Many companies now offer AutoML as a service, where a dataset is uploaded and a model pipeline can be downloaded or hosted and used via web service (i.e. MLaaS). Popular examples include service offerings from Google, Microsoft, and Amazon.

Additionally, open-source libraries are available that implement AutoML techniques, focusing on the specific data transforms, models, and hyperparameters used in the search space and the types of algorithms used to navigate or optimize the search space of possibilities, with versions of Bayesian Optimization being the most common.

There are many open-source AutoML libraries, although, in this tutorial, we will focus on the best-of-breed libraries that can be used in conjunction with the popular scikit-learn Python machine learning library.

They are: Hyperopt-Sklearn, Auto-Sklearn, and TPOT.

Did I miss your favorite AutoML library for scikit-learn?
Let me know in the comments below.

We will take a closer look at each, providing the basis for you to evaluate and consider which library might be appropriate for your project.

Auto-Sklearn

Auto-Sklearn is an open-source Python library for AutoML using machine learning models from the scikit-learn machine learning library.

It was developed by Matthias Feurer, et al. and described in their 2015 paper titled “Efficient and Robust Automated Machine Learning.”

… we introduce a robust new AutoML system based on scikit-learn (using 15 classifiers, 14 feature preprocessing methods, and 4 data preprocessing methods, giving rise to a structured hypothesis space with 110 hyperparameters).

Efficient and Robust Automated Machine Learning, 2015.

The first step is to install the Auto-Sklearn library, which can be achieved using pip, as follows:

Once installed, we can import the library and print the version number to confirm it was installed successfully:

Running the example prints the version number. Your version number should be the same or higher.

Next, we can demonstrate using Auto-Sklearn on a synthetic classification task.

We can define an AutoSklearnClassifier class that controls the search and configure it to run for two minutes (120 seconds) and kill any single model that takes more than 30 seconds to evaluate. At the end of the run, we can report the statistics of the search and evaluate the best performing model on a holdout dataset.

The complete example is listed below.

Running the example will take about two minutes, given the hard limit we imposed on the run.

At the end of the run, a summary is printed showing that 599 models were evaluated and the estimated performance of the final model was 95.6 percent.

We then evaluate the model on the holdout dataset and see that a classification accuracy of 97 percent was achieved, which is reasonably skillful.

For more on the Auto-Sklearn library, see:

Tree-based Pipeline Optimization Tool (TPOT)

Tree-based Pipeline Optimization Tool, or TPOT for short, is a Python library for automated machine learning.

TPOT uses a tree-based structure to represent a model pipeline for a predictive modeling problem, including data preparation and modeling algorithms, and model hyperparameters.

… an evolutionary algorithm called the Tree-based Pipeline Optimization Tool (TPOT) that automatically designs and optimizes machine learning pipelines.

Evaluation of a Tree-based Pipeline Optimization Tool for Automating Data Science, 2016.

The first step is to install the TPOT library, which can be achieved using pip, as follows:

Once installed, we can import the library and print the version number to confirm it was installed successfully:

Running the example prints the version number. Your version number should be the same or higher.

Next, we can demonstrate using TPOT on a synthetic classification task.

This involves configuring a TPOTClassifier instance with the population size and number of generations for the evolutionary search, as well as the cross-validation procedure and metric used to evaluate models. The algorithm will then run the search procedure and save the best discovered model pipeline to file.

The complete example is listed below.

Running the example may take a few minutes, and you will see a progress bar on the command line.

The accuracy of top-performing models will be reported along the way.

Your specific results will vary given the stochastic nature of the search procedure.

In this case, we can see that the top-performing pipeline achieved the mean accuracy of about 92.6 percent.

The top-performing pipeline is then saved to a file named “tpot_best_model.py“.

Opening this file, you can see that there is some generic code for loading a dataset and fitting the pipeline. An example is listed below.

You can then retrieve the code for creating the model pipeline and integrate it into your project.

For more on TPOT, see the following resources:

Hyperopt-Sklearn

HyperOpt is an open-source Python library for Bayesian optimization developed by James Bergstra.

It is designed for large-scale optimization for models with hundreds of parameters and allows the optimization procedure to be scaled across multiple cores and multiple machines.

HyperOpt-Sklearn wraps the HyperOpt library and allows for the automatic search of data preparation methods, machine learning algorithms, and model hyperparameters for classification and regression tasks.

… we introduce Hyperopt-Sklearn: a project that brings the benefits of automatic algorithm configuration to users of Python and scikit-learn. Hyperopt-Sklearn uses Hyperopt to describe a search space over possible configurations of Scikit-Learn components, including preprocessing and classification modules.

Hyperopt-Sklearn: Automatic Hyperparameter Configuration for Scikit-Learn, 2014.

Now that we are familiar with HyperOpt and HyperOpt-Sklearn, let’s look at how to use HyperOpt-Sklearn.

The first step is to install the HyperOpt library.

This can be achieved using the pip package manager as follows:

Next, we must install the HyperOpt-Sklearn library.

This too can be installed using pip, although we must perform this operation manually by cloning the repository and running the installation from the local files, as follows:

We can confirm that the installation was successful by checking the version number with the following command:

This will summarize the installed version of HyperOpt-Sklearn, confirming that a modern version is being used.

Next, we can demonstrate using Hyperopt-Sklearn on a synthetic classification task.

We can configure a HyperoptEstimator instance that runs the search, including the classifiers to consider in the search space, the pre-processing steps, and the search algorithm to use. In this case, we will use TPE, or Tree of Parzen Estimators, and perform 50 evaluations.

At the end of the search, the best performing model pipeline is evaluated and summarized.

The complete example is listed below.

Running the example may take a few minutes.

The progress of the search will be reported and you will see some warnings that you can safely ignore.

At the end of the run, the best-performing model is evaluated on the holdout dataset and the Pipeline discovered is printed for later use.

Your specific results may differ given the stochastic nature of the learning algorithm and search process. Try running the example a few times.

In this case, we can see that the chosen model achieved an accuracy of about 84.8 percent on the holdout test set. The Pipeline involves a SGDClassifier model with no pre-processing.

The printed model can then be used directly, e.g. the code copy-pasted into another project.

For more on Hyperopt-Sklearn, see:

Summary

In this tutorial, you discovered how to use top open-source AutoML libraries for scikit-learn in Python.

Specifically, you learned:

  • AutoML are techniques for automatically and quickly discovering a well-performing machine learning model pipeline for a predictive modeling task.
  • The three most popular AutoML libraries for Scikit-Learn are Hyperopt-Sklearn, Auto-Sklearn, and TPOT.
  • How to use AutoML libraries to discover well-performing models for predictive modeling tasks in Python.

Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.

Discover Fast Machine Learning in Python!

Master Machine Learning With Python

Develop Your Own Models in Minutes

...with just a few lines of scikit-learn code

Learn how in my new Ebook:
Machine Learning Mastery With Python

Covers self-study tutorials and end-to-end projects like:
Loading data, visualization, modeling, tuning, and much more...

Finally Bring Machine Learning To
Your Own Projects

Skip the Academics. Just Results.

See What's Inside

25 Responses to Automated Machine Learning (AutoML) Libraries for Python

  1. Avatar
    Sam September 18, 2020 at 9:19 am #

    Check out AutoGluon too: https://autogluon.mxnet.io/

  2. Avatar
    xi September 18, 2020 at 12:14 pm #

    Hello, Jason. I use the command “pip install autosklearn”, it failed to install AutoML library for scikit-learn. Pls tell me why?

    • Avatar
      Jason Brownlee September 18, 2020 at 2:49 pm #

      I’m sorry to hear that.

      I don’t know why it failed, perhaps try posting your error on stackoverflow.com?

    • Avatar
      TM September 19, 2020 at 2:20 am #

      pip install auto-sklearn

  3. Avatar
    Atsushi Isobe September 18, 2020 at 12:18 pm #

    How can I use autosklearn on Google Colab?

    I did this
    > ! pip install autosklearn

    Below is the output.
    >
    Collecting autosklearn
    Using cached https://files.pythonhosted.org/packages/33/c3/bca73d2b9d839bdeb1a68aa9e15f90c5dccb2cbc7426262e188c80e7413b/autosklearn-0.1.1.tar.gz
    Building wheels for collected packages: autosklearn
    Building wheel for autosklearn (setup.py) … error
    ERROR: Failed building wheel for autosklearn
    Running setup.py clean for autosklearn
    Failed to build autosklearn
    Installing collected packages: autosklearn
    Running setup.py install for autosklearn … error
    ERROR: Command errored out with exit status 1: /usr/bin/python3 -u -c ‘import sys, setuptools, tokenize; sys.argv[0] = ‘”‘”‘/tmp/pip-install-34fbo8pn/autosklearn/setup.py'”‘”‘; __file__='”‘”‘/tmp/pip-install-34fbo8pn/autosklearn/setup.py'”‘”‘;f=getattr(tokenize, ‘”‘”‘open'”‘”‘, open)(__file__);code=f.read().replace(‘”‘”‘\r\n'”‘”‘, ‘”‘”‘\n'”‘”‘);f.close();exec(compile(code, __file__, ‘”‘”‘exec'”‘”‘))’ install –record /tmp/pip-record-lt1zshky/install-record.txt –single-version-externally-managed –compile Check the logs for full command output.

  4. Avatar
    Atsushi Isobe September 18, 2020 at 12:49 pm #

    Great blog Jason !!!!

    If we have such a tool, does it mean studying for “applied ML” is not much of value?

    I am studying ML now using Python. I am a long time programmer.

    I got to the point of applying ML pipeline, as below.
    1. read data
    2. check data
    3. do a bit of necessary data pre-prosessing
    (category to numeric conversion, fillna, drop, binning)
    4. split data
    5. use several algorithms to fit on training data, predict on validation data
    using selected evaluation method (accuracy, FScore, AUC….)
    Get a feel for the model performance (AUC was over 0.92 for example)
    6. predict on test data
    7. submit to Kaggle, for example (got 0.8xxx )

    But I get a lower result at step 7 compared to step 5.
    I assume I need to do better with data pre-processing.
    But feature engineering is very difficult for a beginner.

    It seems this AutoML will solve the problem. Is this correct?

    At the same time, I suddenly feel, is studying for “applied ML” worth it?
    Step 3 is the heart of ML. But it is difficult to have domain knowledge for everything.
    I want to apply ML to my job, not be a ML researcher.

    What do you think?

    • Avatar
      Jason Brownlee September 18, 2020 at 2:51 pm #

      Thanks.

      There’s no silver bullet. AutoML can help you get a good result quickly, but not win competitions.

  5. Avatar
    TM September 19, 2020 at 2:24 am #

    Getting below eror:
    [ERROR] [2020-09-18 16:16:25,132:AutoML(1791095845):77b13f0bfe0702b784d576f1db1241bb] ‘generator’ object is not subscriptable
    Traceback (most recent call last):
    File “/home/gautqm/py37/lib/python3.8/site-packages/autosklearn/automl.py”, line 615, in fit
    _proc_smac.run_smbo()
    File “/home/gautqm/py37/lib/python3.8/site-packages/autosklearn/smbo.py”, line 358, in run_smbo
    metalearning_configurations = self.get_metalearning_suggestions()
    File “/home/gautqm/py37/lib/python3.8/site-packages/autosklearn/smbo.py”, line 547, in get_metalearning_suggestions
    meta_base = MetaBase(self.config_space, self.metadata_directory)
    File “/home/gautqm/py37/lib/python3.8/site-packages/autosklearn/metalearning/metalearning/meta_base.py”, line 40, in __init__
    aslib_reader = aslib_simple.AlgorithmSelectionProblem(self.aslib_directory)
    File “/home/gautqm/py37/lib/python3.8/site-packages/autosklearn/metalearning/input/aslib_simple.py”, line 33, in __init__
    self._read_files()
    File “/home/gautqm/py37/lib/python3.8/site-packages/autosklearn/metalearning/input/aslib_simple.py”, line 73, in _read_files
    read_func(file_)
    File “/home/gautqm/py37/lib/python3.8/site-packages/autosklearn/metalearning/input/aslib_simple.py”, line 79, in _read_algorithm_runs
    if arff_dict[“attributes”][0][0].upper() != “INSTANCE_ID”:
    TypeError: ‘generator’ object is not subscriptable
    —————————————————————————
    TypeError Traceback (most recent call last)
    in
    10 model = AutoSklearnClassifier(time_left_for_this_task=2*60, per_run_time_limit=30, n_jobs=8)
    11 # perform the search
    —> 12 model.fit(X_train, y_train)
    13 # summarize
    14 print(model.sprint_statistics())

    ~/py37/lib/python3.8/site-packages/autosklearn/estimators.py in fit(self, X, y, X_test, y_test, feat_type, dataset_name)
    682 self.target_type = target_type
    683
    –> 684 super().fit(
    685 X=X,
    686 y=y,

    ~/py37/lib/python3.8/site-packages/autosklearn/estimators.py in fit(self, **kwargs)
    430 processes.append(p)
    431 p.start()
    –> 432 _fit_automl(
    433 automl=self._automl[0],
    434 kwargs=kwargs,

    ~/py37/lib/python3.8/site-packages/autosklearn/estimators.py in _fit_automl(automl, kwargs, load_models)
    14
    15 def _fit_automl(automl, kwargs, load_models):
    —> 16 return automl.fit(load_models=load_models, **kwargs)
    17
    18

    ~/py37/lib/python3.8/site-packages/autosklearn/automl.py in fit(self, X, y, X_test, y_test, feat_type, dataset_name, only_return_configuration_space, load_models)
    1154 self._metric = accuracy
    1155
    -> 1156 return super().fit(
    1157 X, y,
    1158 X_test=X_test,

    ~/py37/lib/python3.8/site-packages/autosklearn/automl.py in fit(self, X, y, task, X_test, y_test, feat_type, dataset_name, only_return_configuration_space, load_models)
    613 try:
    614 self.runhistory_, self.trajectory_, self._budget_type = \
    –> 615 _proc_smac.run_smbo()
    616 trajectory_filename = os.path.join(
    617 self._backend.get_smac_output_directory_for_run(self._seed),

    ~/py37/lib/python3.8/site-packages/autosklearn/smbo.py in run_smbo(self)
    356 # Initialize some SMAC dependencies
    357
    –> 358 metalearning_configurations = self.get_metalearning_suggestions()
    359
    360 if self.resampling_strategy in [‘partial-cv’,

    ~/py37/lib/python3.8/site-packages/autosklearn/smbo.py in get_metalearning_suggestions(self)
    545 self.logger.info(‘Metadata directory: %s’,
    546 self.metadata_directory)
    –> 547 meta_base = MetaBase(self.config_space, self.metadata_directory)
    548
    549 metafeature_calculation_time_limit = int(

    ~/py37/lib/python3.8/site-packages/autosklearn/metalearning/metalearning/meta_base.py in __init__(self, configuration_space, aslib_directory)
    38 self.aslib_directory = aslib_directory
    39
    —> 40 aslib_reader = aslib_simple.AlgorithmSelectionProblem(self.aslib_directory)
    41 self.metafeatures = aslib_reader.metafeatures
    42 self.algorithm_runs = aslib_reader.algorithm_runs

    ~/py37/lib/python3.8/site-packages/autosklearn/metalearning/input/aslib_simple.py in __init__(self, directory)
    31 # Read ASLib files
    32 self._find_files()
    —> 33 self._read_files()
    34
    35 def _find_files(self):

    ~/py37/lib/python3.8/site-packages/autosklearn/metalearning/input/aslib_simple.py in _read_files(self)
    71 read_func = self.read_funcs.get(os.path.basename(file_))
    72 if read_func:
    —> 73 read_func(file_)
    74
    75 def _read_algorithm_runs(self, filename):

    ~/py37/lib/python3.8/site-packages/autosklearn/metalearning/input/aslib_simple.py in _read_algorithm_runs(self, filename)
    77 arff_dict = arff.load(fh)
    78
    —> 79 if arff_dict[“attributes”][0][0].upper() != “INSTANCE_ID”:
    80 self.logger.error(
    81 “instance_id as first attribute is missing in %s” % (filename))

    TypeError: ‘generator’ object is not subscriptable

  6. Avatar
    Valentin Mayr September 29, 2020 at 4:50 pm #

    For all of us failing to install:
    auto-sklearn currently is currently only working seamlessly on linux systems. Check out the options for Win/Mac: https://automl.github.io/auto-sklearn/master/installation.html#windows-osx-compatibility.

    • Avatar
      Jason Brownlee September 30, 2020 at 6:23 am #

      Sorry to hear that.

      It seemed to install and work directly on macos and linux when I wrote the tutorual months back.

  7. Avatar
    Nick November 18, 2020 at 7:38 am #

    Great post!

    As of version 0.11.6, TPOT now has native support for GPU-accelerated models via RAPIDS cuML and DMLC XGBoost, which can give huge time savings and even lead to better ML pipelines for medium-to-large datasets.

    Blog post with a small case study if you’re interested in learning more: https://medium.com/rapids-ai/faster-automl-with-tpot-and-rapids-758455cd89e5

  8. Avatar
    Nithin Raju December 16, 2020 at 6:11 pm #

    Wow that was a great content.

  9. Avatar
    Abhinav B March 21, 2021 at 6:21 pm #

    The Autosklearn library is not supported on Windows
    See this thread

    https://stackoverflow.com/questions/54817301/installing-autosklearn-in-anaconda-environment

  10. Avatar
    Rei Balachandran April 7, 2021 at 1:31 pm #

    Thanks so much for all your invaluable tutorials! They are great! Please keep up the good work!

  11. Avatar
    Aykut Özdemir June 14, 2021 at 10:07 pm #

    Hi Jason! Thank you so much for this amazing content. Great introduction. Keep up the work!

  12. Avatar
    Tai Nguyen June 23, 2021 at 5:17 pm #

    when i run example Hyperopt exactly you show in notebook, but have problem.
    TypeError: __init__() got an unexpected keyword argument ‘presort’

Leave a Reply