Auto-Sklearn for Automated Machine Learning in Python

Last Updated on September 12, 2020

Automated Machine Learning (AutoML) refers to techniques for automatically discovering well-performing models for predictive modeling tasks with very little user involvement.

Auto-Sklearn is an open-source library for performing AutoML in Python. It makes use of the popular Scikit-Learn machine learning library for data transforms and machine learning algorithms and uses a Bayesian Optimization search procedure to efficiently discover a top-performing model pipeline for a given dataset.

In this tutorial, you will discover how to use Auto-Sklearn for AutoML with Scikit-Learn machine learning algorithms in Python.

After completing this tutorial, you will know:

  • Auto-Sklearn is an open-source library for AutoML with scikit-learn data preparation and machine learning models.
  • How to use Auto-Sklearn to automatically discover top-performing models for classification tasks.
  • How to use Auto-Sklearn to automatically discover top-performing models for regression tasks.

Let’s get started.

Auto-Sklearn for Automated Machine Learning in Python

Auto-Sklearn for Automated Machine Learning in Python
Photo by Richard, some rights reserved.

Tutorial Overview

This tutorial is divided into four parts; they are:

  1. AutoML With Auto-Sklearn
  2. Install and Using Auto-Sklearn
  3. Auto-Sklearn for Classification
  4. Auto-Sklearn for Regression

AutoML With Auto-Sklearn

Automated Machine Learning, or AutoML for short, is a process of discovering the best-performing pipeline of data transforms, model, and model configuration for a dataset.

AutoML often involves the use of sophisticated optimization algorithms, such as Bayesian Optimization, to efficiently navigate the space of possible models and model configurations and quickly discover what works well for a given predictive modeling task. It allows non-expert machine learning practitioners to quickly and easily discover what works well or even best for a given dataset with very little technical background or direct input.

Auto-Sklearn is an open-source Python library for AutoML using machine learning models from the scikit-learn machine learning library.

It was developed by Matthias Feurer, et al. and described in their 2015 paper titled “Efficient and Robust Automated Machine Learning.”

… we introduce a robust new AutoML system based on scikit-learn (using 15 classifiers, 14 feature preprocessing methods, and 4 data preprocessing methods, giving rise to a structured hypothesis space with 110 hyperparameters).

Efficient and Robust Automated Machine Learning, 2015.

The benefit of Auto-Sklearn is that, in addition to discovering the data preparation and model that performs for a dataset, it also is able to learn from models that performed well on similar datasets and is able to automatically create an ensemble of top-performing models discovered as part of the optimization process.

This system, which we dub AUTO-SKLEARN, improves on existing AutoML methods by automatically taking into account past performance on similar datasets, and by constructing ensembles from the models evaluated during the optimization.

Efficient and Robust Automated Machine Learning, 2015.

The authors provide a useful depiction of their system in the paper, provided below.

Overview of the Auto-Sklearn System

Overview of the Auto-Sklearn System.
Taken from: Efficient and Robust Automated Machine Learning, 2015.

Install and Using Auto-Sklearn

The first step is to install the Auto-Sklearn library, which can be achieved using pip, as follows:

Once installed, we can import the library and print the version number to confirm it was installed successfully:

Running the example prints the version number.

Your version number should be the same or higher.

Using Auto-Sklearn is straightforward.

Depending on whether your prediction task is classification or regression, you create and configure an instance of the AutoSklearnClassifier or AutoSklearnRegressor class, fit it on your dataset, and that’s it. The resulting model can then be used to make predictions directly or saved to file (using pickle) for later use.

There are a ton of configuration options provided as arguments to the AutoSklearn class.

By default, the search will use a train-test split of your dataset during the search, and this default is recommended both for speed and simplicity.

Importantly, you should set the “n_jobs” argument to the number of cores in your system, e.g. 8 if you have 8 cores.

The optimization process will run for as long as you allow, measure in minutes. By default, it will run for one hour.

I recommend setting the “time_left_for_this_task” argument for the number of seconds you want the process to run. E.g. less than 5-10 minutes is probably plenty for many small predictive modeling tasks (sub 1,000 rows).

We will use 5 minutes (300 seconds) for the examples in this tutorial. We will also limit the time allocated to each model evaluation to 30 seconds via the “per_run_time_limit” argument. For example:

You can limit the algorithms considered in the search, as well as the data transforms.

By default, the search will create an ensemble of top-performing models discovered as part of the search. Sometimes, this can lead to overfitting and can be disabled by setting the “ensemble_size” argument to 1 and “initial_configurations_via_metalearning” to 0.

At the end of a run, the list of models can be accessed, as well as other details.

Perhaps the most useful feature is the sprint_statistics() function that summarizes the search and the performance of the final model.

Now that we are familiar with the Auto-Sklearn library, let’s look at some worked examples.

Auto-Sklearn for Classification

In this section, we will use Auto-Sklearn to discover a model for the sonar dataset.

The sonar dataset is a standard machine learning dataset comprised of 208 rows of data with 60 numerical input variables and a target variable with two class values, e.g. binary classification.

Using a test harness of repeated stratified 10-fold cross-validation with three repeats, a naive model can achieve an accuracy of about 53 percent. A top-performing model can achieve accuracy on this same test harness of about 88 percent. This provides the bounds of expected performance on this dataset.

The dataset involves predicting whether sonar returns indicate a rock or simulated mine.

No need to download the dataset; we will download it automatically as part of our worked examples.

The example below downloads the dataset and summarizes its shape.

Running the example downloads the dataset and splits it into input and output elements. As expected, we can see that there are 208 rows of data with 60 input variables.

We will use Auto-Sklearn to find a good model for the sonar dataset.

First, we will split the dataset into train and test sets and allow the process to find a good model on the training set, then later evaluate the performance of what was found on the holdout test set.

The AutoSklearnClassifier is configured to run for 5 minutes with 8 cores and limit each model evaluation to 30 seconds.

The search is then performed on the training dataset.

Afterward, a summary of the search and best-performing model is reported.

Finally, we evaluate the performance of the model that was prepared on the holdout test dataset.

Tying this together, the complete example is listed below.

Running the example will take about five minutes, given the hard limit we imposed on the run.

Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.

At the end of the run, a summary is printed showing that 1,054 models were evaluated and the estimated performance of the final model was 91 percent.

We then evaluate the model on the holdout dataset and see that classification accuracy of 81.2 percent was achieved, which is reasonably skillful.

Auto-Sklearn for Regression

In this section, we will use Auto-Sklearn to discover a model for the auto insurance dataset.

The auto insurance dataset is a standard machine learning dataset comprised of 63 rows of data with one numerical input variable and a numerical target variable.

Using a test harness of repeated stratified 10-fold cross-validation with three repeats, a naive model can achieve a mean absolute error (MAE) of about 66. A top-performing model can achieve a MAE on this same test harness of about 28. This provides the bounds of expected performance on this dataset.

The dataset involves predicting the total amount in claims (thousands of Swedish Kronor) given the number of claims for different geographical regions.

No need to download the dataset; we will download it automatically as part of our worked examples.

The example below downloads the dataset and summarizes its shape.

Running the example downloads the dataset and splits it into input and output elements. As expected, we can see that there are 63 rows of data with one input variable.

We will use Auto-Sklearn to find a good model for the auto insurance dataset.

We can use the same process as was used in the previous section, although we will use the AutoSklearnRegressor class instead of the AutoSklearnClassifier.

By default, the regressor will optimize the R^2 metric.

In this case, we are interested in the mean absolute error, or MAE, which we can specify via the “metric” argument when calling the fit() function.

The complete example is listed below.

Running the example will take about five minutes, given the hard limit we imposed on the run.

You might see some warning messages during the run and you can safely ignore them, such as:

Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.

At the end of the run, a summary is printed showing that 1,759 models were evaluated and the estimated performance of the final model was a MAE of 29.

We then evaluate the model on the holdout dataset and see that a MAE of 26 was achieved, which is a great result.

Further Reading

This section provides more resources on the topic if you are looking to go deeper.

Summary

In this tutorial, you discovered how to use Auto-Sklearn for AutoML with Scikit-Learn machine learning algorithms in Python.

Specifically, you learned:

  • Auto-Sklearn is an open-source library for AutoML with scikit-learn data preparation and machine learning models.
  • How to use Auto-Sklearn to automatically discover top-performing models for classification tasks.
  • How to use Auto-Sklearn to automatically discover top-performing models for regression tasks.

Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.

Discover Fast Machine Learning in Python!

Master Machine Learning With Python

Develop Your Own Models in Minutes

...with just a few lines of scikit-learn code

Learn how in my new Ebook:
Machine Learning Mastery With Python

Covers self-study tutorials and end-to-end projects like:
Loading data, visualization, modeling, tuning, and much more...

Finally Bring Machine Learning To
Your Own Projects

Skip the Academics. Just Results.

See What's Inside

52 Responses to Auto-Sklearn for Automated Machine Learning in Python

  1. Alessandro F. September 7, 2020 at 7:29 pm #

    Hello Jason,

    I’ve tried to run Auto-Sklearn on Google CO-Lab without success.
    It seems there is a conflict with the current version of Pandas (1.0.x) while the package runs on a prior version (0.25.x).

    I’ll try to solve this issue.
    Anyway, your explanations are always the best !

    Regards,
    Alessandro

    • Jason Brownlee September 8, 2020 at 6:47 am #

      Perhaps try running on your workstation.

      • Gábor September 8, 2020 at 8:29 pm #

        I also had problem with Pandas >0.25.x. Downgrading to 0.25.3, substituting with the arff package with liac-arff fixed it.

        • Jason Brownlee September 9, 2020 at 6:47 am #

          Here are the versions I’m using if that helps at all:

          • Junaid Akhtar September 12, 2020 at 4:09 pm #

            how did you compute this list?

          • Jason Brownlee September 13, 2020 at 5:59 am #

            I printed the version of each library in turn with this script:

  2. Gabriel September 8, 2020 at 3:49 am #

    Is it possible to evaluate the automatically selected model by hand? I mean, is there a method to call for all the parameters and configuration set to the model at the end?

    • Jason Brownlee September 8, 2020 at 6:52 am #

      Yes, once defined you can evaluate it on your own test harness.

  3. shamik September 8, 2020 at 11:25 am #

    This is such informative information. Really very useful. I wanted to use this in my live project but could not move ahead.
    This package does not seem to be supported by windows as per their official website. Neither could find a way out for Anaconda which I am using at present.
    This command mentioned above does not work in Anaconda:
    sudo pip install autosklearn

    Kindly help how we can use it in Anaconda env.

    • Jason Brownlee September 8, 2020 at 1:36 pm #

      You’re welcome.

      Sorry to hear that. Perhaps try using conda to install the package?

  4. Manoj Ishi September 8, 2020 at 7:06 pm #

    TypeError: ‘generator’ object is not subscribable
    error is occurred while running the classification problem

  5. Gábor Stikkel September 9, 2020 at 4:55 am #

    Thank you for yet another great article Jason!
    I got an error for the regression part:

    —————————————————————————
    TypeError Traceback (most recent call last)
    in
    17 model = AutoSklearnRegressor(time_left_for_this_task=60, per_run_time_limit=30, n_jobs=8)#, metric=auto_mean_absolute_error)
    18 # perform the search
    —> 19 model.fit(X_train, y_train, metric=auto_mean_absolute_error)
    20 # summarize
    21 print(model.sprint_statistics())

    TypeError: fit() got an unexpected keyword argument ‘metric’

    I think the metric part shall be moved to the model definition (see commmented part above).

    [Keep rocking!]

    • Jason Brownlee September 9, 2020 at 6:53 am #

      Sorry to hear that, perhaps check that all of your libraries are up to date?

      • Adeliya September 10, 2020 at 11:26 pm #

        Hi!

        I got the same error. And actually looking at the documentation “metric” is a parameter of AutoSklearnRegressor(), not fit().

        Thanks for tutorial!

  6. Bill September 11, 2020 at 5:41 am #

    Thanks for the tutorial,

    We are really keen to see some tutorials on Knowledge Graphs and Knowledge Graph Embeddings 🙂

    Thanks,
    Bill

  7. Timothy September 11, 2020 at 8:02 am #

    Looks like just what I need. Unfortunately, install was not successful. Will have to revisit when I have lots of time to troubleshoot.

  8. Xu Zhang September 11, 2020 at 8:25 am #

    Thank you for your great post.

    Does Auto-Sklearn always got the better performance compared to the fine-tuned individual models?

  9. ct dyana September 11, 2020 at 8:47 am #

    Tq for the informative explanation…I like it much…

    but I am interested to know what algorithm does python used in auto sklearn for ml? Does it perform as same as using weka..
    How I want to know the algorithm for classification and regression that applicable in lib python?

    Tqvm

    • Jason Brownlee September 11, 2020 at 1:29 pm #

      You’re welcome.

      It uses Bayesian Optimization to search for an appropriate algorithm.

  10. Amit Barik September 11, 2020 at 4:13 pm #

    Hi Jason,

    I have windows10 laptop. When i tried installing auto-sklearn, it said it couldn’t becoz of system compatibility issues. Then i checked in git and got to know that we can’t install in windows machine. Am i right? Does it run only on unix machines?

    Please help!!!

    Regards,
    Amit

  11. Santosh September 11, 2020 at 6:06 pm #

    Can we achive the parameters what was internally used in to get the best results.

    • Jason Brownlee September 12, 2020 at 6:05 am #

      Yes, the best model include the hyperparameters used.

  12. Mark Littlewood September 11, 2020 at 6:30 pm #

    How do you know what algo it has selected eg GBM or is it an ensemble

    • Jason Brownlee September 12, 2020 at 6:06 am #

      You can use model.show_models() to show the ensemble of models.

  13. Anubha Pearline S September 11, 2020 at 8:25 pm #

    Hi Jason,
    How to check which model is chosen thro AutoSklearn?

    • Jason Brownlee September 12, 2020 at 6:11 am #

      You can use the following to show the models in the final ensemble:

  14. shaheen mohammed saleh September 12, 2020 at 1:12 am #

    Please how to install auto-sklearn in anaconda windows 10 because auto-sklearn has the following system requirements:

    Linux operating system (for example Ubuntu) (get Linux here),

    Python (>=3.5) (get Python here).

    C++ compiler (with C++11 supports) (get GCC here) and

    SWIG (version 3.0.* is required; >=4.0.0 is not supported) (get SWIG here).

    • Jason Brownlee September 12, 2020 at 6:18 am #

      Perhaps use an AWS EC2 instance or a linux virtual machine.

  15. Rahul Goswami September 12, 2020 at 2:51 am #

    For everyone facing issues installing auto sklearn on windows in Google Colab, should run on jupyter as well but have not tried out.Run the below codes and auto sklearn runs without issue

    !apt-get install swig -y
    !pip install Cython numpy

    # sometimes you have to run the next command twice on colab
    # I haven’t figured out why
    !pip install auto-sklearn

  16. Rahul Goswami September 12, 2020 at 3:14 am #

    Hi Jason,

    Gettting this error when trying out the classifier for auto sklearn

    ypeError: ‘generator’ object is not subscribable
    error is occurred while running the classification problem

    Kindly help if possible and thanks for all the great blogs.

    • Jason Brownlee September 12, 2020 at 6:20 am #

      Sorry, I have not seen this error. Perhaps try searching/posting on stackoverflow.

  17. TC September 18, 2020 at 2:04 pm #

    Hi Jason,

    Thanks for writing this blog as there are very fews articles online covering auto-sklearn.

    A question: does auto-sklearn really offer any “feature engineering” stuff? It look like feature preprocessors just do dimension reduction or compression. For example, if we have a datetime variable, can auto-sklearn automatically create hour, day, month, year engineered features out of this variable?

    • Jason Brownlee September 18, 2020 at 2:52 pm #

      Not really. You could add those modules.

      • TC September 18, 2020 at 6:06 pm #

        Could you provide an example of how to “add those modules”?

        • Jason Brownlee September 19, 2020 at 6:48 am #

          Thanks for the suggestion, perhaps in the future.

  18. TC September 18, 2020 at 6:13 pm #

    Hi Jason,

    “Using a test harness of repeated stratified 10-fold cross-validation with three repeats, a naive model can achieve an accuracy of about 53 percent. A top-performing model can achieve accuracy on this same test harness of about 88 percent. This provides the bounds of expected performance on this dataset.”

    Could you tell me what algorithms did you use to get the naive and top-performing models respectively?

  19. TC September 18, 2020 at 6:15 pm #

    Hi Jason,

    “At the end of the run, a summary is printed showing that 1,759 models were evaluated and the estimated performance of the final model was a MAE of 29.”

    Is there any way to see what algorithms auto-sklearn uses to generate these 1,759 models? Does auto-sklearn include xgboost as one of the algorithms to build models?

    • Jason Brownlee September 19, 2020 at 6:49 am #

      Yes, as follows:

      I don’t know if it supports xgboost off hand, sorry.

  20. Gustavo September 22, 2020 at 12:16 pm #

    Does it make Cross validation to choose best model?

    • Jason Brownlee September 22, 2020 at 1:39 pm #

      Good question, I’m not sure off the cuff.

      Perhaps check the library documentaiton?

Leave a Reply