SALE! Use code blackfriday for 40% off everything!
Hurry, sale ends soon! Click to see the full catalog.

Recursive Feature Elimination (RFE) for Feature Selection in Python

Last Updated on August 28, 2020

Recursive Feature Elimination, or RFE for short, is a popular feature selection algorithm.

RFE is popular because it is easy to configure and use and because it is effective at selecting those features (columns) in a training dataset that are more or most relevant in predicting the target variable.

There are two important configuration options when using RFE: the choice in the number of features to select and the choice of the algorithm used to help choose features. Both of these hyperparameters can be explored, although the performance of the method is not strongly dependent on these hyperparameters being configured well.

In this tutorial, you will discover how to use Recursive Feature Elimination (RFE) for feature selection in Python.

After completing this tutorial, you will know:

  • RFE is an efficient approach for eliminating features from a training dataset for feature selection.
  • How to use RFE for feature selection for classification and regression predictive modeling problems.
  • How to explore the number of selected features and wrapped algorithm used by the RFE procedure.

Kick-start your project with my new book Data Preparation for Machine Learning, including step-by-step tutorials and the Python source code files for all examples.

Let’s get started.

Recursive Feature Elimination (RFE) for Feature Selection in Python

Recursive Feature Elimination (RFE) for Feature Selection in Python
Taken by djandywdotcom, some rights reserved.

Tutorial Overview

This tutorial is divided into three parts; they are:

  1. Recursive Feature Elimination
  2. RFE With scikit-learn
    1. RFE for Classification
    2. RFE for Regression
  3. RFE Hyperparameters
    1. Explore Number of Features
    2. Automatically Select the Number of Features
    3. Which Features Were Selected
    4. Explore Base Algorithm

Recursive Feature Elimination

Recursive Feature Elimination, or RFE for short, is a feature selection algorithm.

A machine learning dataset for classification or regression is comprised of rows and columns, like an excel spreadsheet. Rows are often referred to as samples and columns are referred to as features, e.g. features of an observation in a problem domain.

Feature selection refers to techniques that select a subset of the most relevant features (columns) for a dataset. Fewer features can allow machine learning algorithms to run more efficiently (less space or time complexity) and be more effective. Some machine learning algorithms can be misled by irrelevant input features, resulting in worse predictive performance.

For more on feature selection generally, see the tutorial:

RFE is a wrapper-type feature selection algorithm. This means that a different machine learning algorithm is given and used in the core of the method, is wrapped by RFE, and used to help select features. This is in contrast to filter-based feature selections that score each feature and select those features with the largest (or smallest) score.

Technically, RFE is a wrapper-style feature selection algorithm that also uses filter-based feature selection internally.

RFE works by searching for a subset of features by starting with all features in the training dataset and successfully removing features until the desired number remains.

This is achieved by fitting the given machine learning algorithm used in the core of the model, ranking features by importance, discarding the least important features, and re-fitting the model. This process is repeated until a specified number of features remains.

When the full model is created, a measure of variable importance is computed that ranks the predictors from most important to least. […] At each stage of the search, the least important predictors are iteratively eliminated prior to rebuilding the model.

— Pages 494-495, Applied Predictive Modeling, 2013.

Features are scored either using the provided machine learning model (e.g. some algorithms like decision trees offer importance scores) or by using a statistical method.

The importance calculations can be model based (e.g., the random forest importance criterion) or using a more general approach that is independent of the full model.

— Page 494, Applied Predictive Modeling, 2013.

Now that we are familiar with the RFE procedure, let’s review how we can use it in our projects.

Want to Get Started With Data Preparation?

Take my free 7-day email crash course now (with sample code).

Click to sign-up and also get a free PDF Ebook version of the course.

Download Your FREE Mini-Course

RFE With scikit-learn

RFE can be implemented from scratch, although it can be challenging for beginners.

The scikit-learn Python machine learning library provides an implementation of RFE for machine learning.

It is available in modern versions of the library.

First, confirm that you are using a modern version of the library by running the following script:

Running the script will print your version of scikit-learn.

Your version should be the same or higher. If not, you must upgrade your version of the scikit-learn library.

The RFE method is available via the RFE class in scikit-learn.

RFE is a transform. To use it, first the class is configured with the chosen algorithm specified via the “estimator” argument and the number of features to select via the “n_features_to_select” argument.

The algorithm must provide a way to calculate important scores, such as a decision tree. The algorithm used in RFE does not have to be the algorithm that is fit on the selected features; different algorithms can be used.

Once configured, the class must be fit on a training dataset to select the features by calling the fit() function. After the class is fit, the choice of input variables can be seen via the “support_” attribute that provides a True or False for each input variable.

It can then be applied to the training and test datasets by calling the transform() function.

It is common to use k-fold cross-validation to evaluate a machine learning algorithm on a dataset. When using cross-validation, it is good practice to perform data transforms like RFE as part of a Pipeline to avoid data leakage.

Now that we are familiar with the RFE API, let’s take a look at how to develop a RFE for both classification and regression.

RFE for Classification

In this section, we will look at using RFE for a classification problem.

First, we can use the make_classification() function to create a synthetic binary classification problem with 1,000 examples and 10 input features, five of which are important and five of which are redundant.

The complete example is listed below.

Running the example creates the dataset and summarizes the shape of the input and output components.

Next, we can evaluate an RFE feature selection algorithm on this dataset. We will use a DecisionTreeClassifier to choose features and set the number of features to five. We will then fit a new DecisionTreeClassifier model on the selected features.

We will evaluate the model using repeated stratified k-fold cross-validation, with three repeats and 10 folds. We will report the mean and standard deviation of the accuracy of the model across all repeats and folds.

The complete example is listed below.

Running the example reports the mean and standard deviation accuracy of the model.

Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.

In this case, we can see the RFE that uses a decision tree and selects five features and then fits a decision tree on the selected features achieves a classification accuracy of about 88.6 percent.

We can also use the RFE model pipeline as a final model and make predictions for classification.

First, the RFE and model are fit on all available data, then the predict() function can be called to make predictions on new data.

The example below demonstrates this on our binary classification dataset.

Running the example fits the RFE pipeline on the entire dataset and is then used to make a prediction on a new row of data, as we might when using the model in an application.

Now that we are familiar with using RFE for classification, let’s look at the API for regression.

RFE for Regression

In this section, we will look at using RFE for a regression problem.

First, we can use the make_regression() function to create a synthetic regression problem with 1,000 examples and 10 input features, five of which are important and five of which are redundant.

The complete example is listed below.

Running the example creates the dataset and summarizes the shape of the input and output components.

Next, we can evaluate an REFE algorithm on this dataset.

As we did with the last section, we will evaluate the pipeline with a decision tree using repeated k-fold cross-validation, with three repeats and 10 folds.

We will report the mean absolute error (MAE) of the model across all repeats and folds. The scikit-learn library makes the MAE negative so that it is maximized instead of minimized. This means that larger negative MAE are better and a perfect model has a MAE of 0.

The complete example is listed below.

Running the example reports the mean and standard deviation accuracy of the model.

Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.

In this case, we can see the RFE pipeline with a decision tree model achieves a MAE of about 26.

We can also use the c as a final model and make predictions for regression.

First, the Pipeline is fit on all available data, then the predict() function can be called to make predictions on new data.

The example below demonstrates this on our regression dataset.

Running the example fits the RFE pipeline on the entire dataset and is then used to make a prediction on a new row of data, as we might when using the model in an application.

Now that we are familiar with using the scikit-learn API to evaluate and use RFE for feature selection, let’s look at configuring the model.

RFE Hyperparameters

In this section, we will take a closer look at some of the hyperparameters you should consider tuning for the RFE method for feature selection and their effect on model performance.

Explore Number of Features

An important hyperparameter for the RFE algorithm is the number of features to select.

In the previous section, we used an arbitrary number of selected features, five, which matches the number of informative features in the synthetic dataset. In practice, we cannot know the best number of features to select with RFE; instead, it is good practice to test different values.

The example below demonstrates selecting different numbers of features from 2 to 10 on the synthetic binary classification dataset.

Running the example first reports the mean accuracy for each configured number of input features.

Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.

In this case, we can see that performance improves as the number of features increase and perhaps peaks around 4-to-7 as we might expect, given that only five features are relevant to the target variable.

A box and whisker plot is created for the distribution of accuracy scores for each configured number of features.

Box Plot of RFE Number of Selected Features vs. Classification Accuracy

Box Plot of RFE Number of Selected Features vs. Classification Accuracy

Automatically Select the Number of Features

It is also possible to automatically select the number of features chosen by RFE.

This can be achieved by performing cross-validation evaluation of different numbers of features as we did in the previous section and automatically selecting the number of features that resulted in the best mean score.

The RFECV class implements this for us.

The RFECV is configured just like the RFE class regarding the choice of the algorithm that is wrapped. Additionally, the minimum number of features to be considered can be specified via the “min_features_to_select” argument (defaults to 1) and we can also specify the type of cross-validation and scoring to use via the “cv” (defaults to 5) and “scoring” arguments (uses accuracy for classification).

We can demonstrate this on our synthetic binary classification problem and use RFECV in our pipeline instead of RFE to automatically choose the number of selected features.

The complete example is listed below.

Running the example reports the mean and standard deviation accuracy of the model.

Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.

In this case, we can see the RFE that uses a decision tree and automatically selects a number of features and then fits a decision tree on the selected features achieves a classification accuracy of about 88.6 percent.

Which Features Were Selected

When using RFE, we may be interested to know which features were selected and which were removed.

This can be achieved by reviewing the attributes of the fit RFE object (or fit RFECV object). The “support_” attribute reports true or false as to which features in order of column index were included and the “ranking_” attribute reports the relative ranking of features in the same order.

The example below fits an RFE model on the whole dataset and selects five features, then reports each feature column index (0 to 9), whether it was selected or not (True or False), and the relative feature ranking.

Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.

Running the example lists of the 10 input features and whether or not they were selected as well as their relative ranking of importance.

Explore Base Algorithm

There are many algorithms that can be used in the core RFE, as long as they provide some indication of variable importance.

Most decision tree algorithms are likely to report the same general trends in feature importance, but this is not guaranteed. It might be helpful to explore the use of different algorithms wrapped by RFE.

The example below demonstrates how you might explore this configuration option.

Running the example first reports the mean accuracy for each wrapped algorithm.

Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.

In this case, the results suggest that linear algorithms like logistic regression might select better features more reliably than the chosen decision tree and ensemble of decision tree algorithms.

A box and whisker plot is created for the distribution of accuracy scores for each configured wrapped algorithm.

We can see the general trend of good performance with logistic regression, CART and perhaps GBM. This highlights that even thought the actual model used to fit the chosen features is the same in each case, the model used within RFE can make an important difference to which features are selected and in turn the performance on the prediction problem.

Box Plot of RFE Wrapped Algorithm vs. Classification Accuracy

Box Plot of RFE Wrapped Algorithm vs. Classification Accuracy

Further Reading

This section provides more resources on the topic if you are looking to go deeper.

Tutorials

Books

Papers

APIs

Articles

Summary

In this tutorial, you discovered how to use Recursive Feature Elimination (RFE) for feature selection in Python.

Specifically, you learned:

  • RFE is an efficient approach for eliminating features from a training dataset for feature selection.
  • How to use RFE for feature selection for classification and regression predictive modeling problems.
  • How to explore the number of selected features and wrapped algorithm used by the RFE procedure.

Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.

Get a Handle on Modern Data Preparation!

Data Preparation for Machine Learning

Prepare Your Machine Learning Data in Minutes

...with just a few lines of python code

Discover how in my new Ebook:
Data Preparation for Machine Learning

It provides self-study tutorials with full working code on:
Feature Selection, RFE, Data Cleaning, Data Transforms, Scaling, Dimensionality Reduction, and much more...

Bring Modern Data Preparation Techniques to
Your Machine Learning Projects


See What's Inside

101 Responses to Recursive Feature Elimination (RFE) for Feature Selection in Python

  1. Sidhu May 25, 2020 at 11:10 am #

    Hi Jason, great tutorial on feature selection. Can this be applied to ordinal data as well?
    My dataset contains wellbeing measures(mental health, nutritional quality, sleep quality etc.) as input features and the values for those can range from 0-10. I want to know the most useful features among them. Can I use this technique or some kind of correlation analysis?

    • Jason Brownlee May 25, 2020 at 1:24 pm #

      Thanks!

      Yes, RFE is agnostic to input data type.

      • Suneel June 5, 2020 at 3:21 am #

        Very nice tutorial on feature selection.

  2. Carlos May 26, 2020 at 5:59 am #

    Hello Jason. Can you please elaborate on the following? “When using cross-validation, it is good practice to perform data transforms like RFE as part of a Pipeline to avoid data leakage.”

    Why using data transforms will avoid data leakage?

  3. mlearn May 27, 2020 at 12:52 am #

    Thank you for the article. Where would you put cross-validation/model tuning?
    1. Do cross-validation at the beginning using all the features and then perform RFE;
    2. Use RFE and then do cross-validation using the model with the features selected;
    3. Include cross-validation inside RFE: at each iteration in RFE tune a model using the current subset of features, remove the least important, perform cross-validation again using the new subset and discard the least important, and so on.
    Which one is the correct approach?

    • Jason Brownlee May 27, 2020 at 7:58 am #

      Ideally CV is used to evaluate the entire modeling pipeline.

      Within the pipeline you might want to use a nested rfecv to automatically configure rfe. We call this nested cross-validation.

  4. Mathmatinho May 29, 2020 at 7:11 am #

    how is this idea different from backward selection? it’s well known that either backward, forward and stepwise selection are not preferred when collinearity exist which is more prevalent these days with decent amount of variables.

  5. Punit Kumar June 9, 2020 at 10:09 am #

    Brilliant, quite comprehensive. Thanks a lot, mate for your efforts.

  6. Luciano Gavinho June 21, 2020 at 3:10 pm #

    Hi Jason, Thanks a lot for the nice explanations.

    I would like to know, how to get the features selected after all models were tested. I mean, after a chose the winner model how I could get the winner features?

    • Jason Brownlee June 22, 2020 at 6:10 am #

      After the model is fit on your data, you can access the RFE object and report the selected features.

      See the section “Which Features Were Selected”

  7. CNguyen June 25, 2020 at 2:16 am #

    Hi Jason,

    Great tutorial.

    Would you help me to understand why those selected column (2,3,4,6,8) in “Which Features Were Selected” are different from the previous RFE explore number of features where significant columns are (4-7)?

    Thanks,

    • Jason Brownlee June 25, 2020 at 6:27 am #

      It is possible that features that RFE think are not important do in fact contribute to model skill.

  8. shaheen June 25, 2020 at 6:41 pm #

    Is it possible to extract final regression formula or equation from any successful prediction models like conventional regression models ? Thank Very Much.

  9. shaheen June 25, 2020 at 6:44 pm #

    Is there any robust tutorial about nonlinear curve estimation with many input variables

    • Jason Brownlee June 26, 2020 at 5:31 am #

      Sorry, I don’t have a tutorial on this topic, hopefully in the future.

  10. Iram June 26, 2020 at 4:42 pm #

    Hi Jason,

    Nice post. In Section ‘RFE with Scikit learn’ you explained that RFE can be used with fit and transform method using ‘rfe.fit(X,y)’ and ‘rfe.transform(X,y)’. isn’t it be only ‘rfe.transform(X)’ without class labels? Thanks!

  11. Anthony The Koala June 29, 2020 at 7:34 am #

    Dear Dr Jason,
    I had a go of applying the above to the iris data using the Pipeline and not using Pipeline.

    Using Pipeline

    This is application of the iris data without pipeline

    From the results there is little difference between using the Pipeline and not using the Pipeline.

    What is the point of implementing a Pipeline when there is little difference between the mean and stddev of the n_scores?

    Thank you,
    Anthony of Sydney

    • Anthony The Koala June 29, 2020 at 7:43 am #

      Dear Dr Jason,
      Apologies, I used the mean(std). I should have used the mean(n_score) and std(n_score)

      The same question applies – what is the point of the Pipeline, where it produces little differences in the results of the scores?

      Thank you,
      Anthony of Sydney

    • Jason Brownlee June 29, 2020 at 1:19 pm #

      We use a pipeline to avoid data leakage:
      https://machinelearningmastery.com/data-preparation-without-data-leakage/

      • Anthony The Koala June 29, 2020 at 1:43 pm #

        Dear Dr Jason,
        Thank you for referring me to the article on “data preparation without data leakage”.

        Generally it is about the way the data is prepared. In the above ‘experiment’, the relevant headings are “Cross-Validation Evaluation With Naive Data Preparation” and “Cross-Validation Evaluation With Correct Data Preparation”.

        I understand the pipeline method is to put the operations in a list.

        The two lines from the ‘naive’ and ‘correct’ data preparation methods respectively are

        Yes, the mean and stddev of the scores results were slightly different.

        Please excuse my concept of leakage in computing. I thought ‘leakage’ meant something to do with garbage collection in C or Java. Obviously it is not the case in the data preparation tutorial

        Question please:
        How can an assignment of testing and training data leak into each other when you make an assignment of a variable to another variable, you wouldn’t expect that when you assign X_test and y_test to a k-fold operation to mix with the X_train and y_train.

        Or put it another way: although pipelines are not the same as threads, if you don’t funnel a set of procedures in a certain order, you won’t get accurate answers, in the same way that if you don’t have threads the execution of a particular block of code you won’t get accurate answers?

        Thank you,
        Anthony of Sydney

        • Jason Brownlee June 30, 2020 at 6:10 am #

          It has nothing to do with threads or programming.

          Instead, it has to do with making use of data by the model that it should not have access to. E.g. access to “information” from the test set when training the model. Perhaps re-read the tutorial on data leakage.

          • Anthony The Koala June 30, 2020 at 7:15 am #

            Dear Dr Jason,
            Thank you for the reply. I have re-read and still have a ‘mental’ block’
            Split this into background and questions.

            Background:
            Piplines have to do “…with making use of data by the model that it should not have access to…”
            From the blog.
            “….knowledge of the hold out test set leaks into the dataset used to train the model…”
            “….information about the holdout dataset, such as a test or validation dataset, is made available to the model in the training dataset….”
            “….This could happen when test data is leaked into the training set,…”

            If I split the data into train and test, how does the information leak from the train to test or test to train set after I split the data.

            Then once I split the data into train test, how does test or train data leak back?

            Furthermore when I do the model fitting,

            I am fitting X_train and y_train.
            I don’t see how X_test, y_test leaks into X_train or y_train and vice versa.

            And when data preparation is performed for using the MinMaxScaler() you are transforming the data using MinMaxScaler amongst the scalers MinMaxScaler, RobustScaler, StandardScaler and Normalizer, you want better convergence of the model.
            In other words how does info from the X_train interfere with X_test and vice versa. Same question for leakage into y_train and y_test and vice versa.

            In addition:

            From the naive model, I don’t get how cv = RepeatedKFold causes leaks when cv is already assigned similarly, when the data is used in scores=cross_val_score(model,X,y,……..cv=cv….) that there is leaking.

            From the Pipeline model

            I don’t understand how information from X, y is leaked given that there is no split of X and y.

            Questions/Summary – of leakage:
            There are two methods
            * one uses data preparation for faster convergence of model, BUT I don’t understand how transformation reduces the leakage when the data is already assigned.
            * The other uses a Pipeline. Again I don’t understand how putting a pipeline with a list containing a sequence of commands causes interference between X and y.

            I am sure that there is something simple that how a mere assignment of variables STILL causes leaks unless I either prepare the data and/or use pipelines.

            Thank you,
            Anthony of Sydney

          • Jason Brownlee June 30, 2020 at 1:03 pm #

            You can leak from test to train if you scale train using knowledge of test, e.g. you normalize and calculate min/max using the entire dataset.

            A pipeline ensures that the transforms are only ever fit on the training set.

          • Anthony The Koala June 30, 2020 at 3:33 pm #

            Dear Dr Jason,
            This is what I understand about based on your answer and the blog.

            Under the heading “Train-Test Evaluation With Correct Data Preparation” and subheading “Tying this together, the complete example is listed below”, the anti-leakage preparation by transformation was performed on SEPARATE X_train and X_test. .NOT ON THE WHOLE X features.

            As I understand it, the standard deviation of the X_train may not necessarily be the same as the standard deviation of the X_test, NEITHER WHICH ARE THE SAME as the std deviation of the whole X.

            Then in order to make a prediction, you transform the values you wish to predict.

            Thank you again, it is appreciated.
            Anthony of Sydney

          • Jason Brownlee July 1, 2020 at 5:49 am #

            Yes, if we fit the transform on the whole dataset we get leakage. If we fit the transform on the training set only, we don’t get leakage.

  12. Anthony The Koala July 1, 2020 at 6:56 am #

    Dear Dr Jason,
    Thank you for that, it is appreciated.
    Anthony of Sydney

    • Jason Brownlee July 1, 2020 at 11:16 am #

      You’re welcome.

      • Anthony The Koala August 1, 2020 at 7:13 am #

        Dear Dr Jason,
        When I submitted the question, I had errors on the web browser due to a slow response. I submitted the same question at the bottom of the page. Please ignore the above submission as the same submission is asked at the bottom.
        Anthony of Sydney

  13. abc July 6, 2020 at 9:02 pm #

    great!

  14. suvarna punalekar July 14, 2020 at 10:38 pm #

    Thanks Jason, Really ueful stuff!!
    I have one questions. Once you run feature selection, cross validation and grid search through pipeline, how do you access the best model for predictions on x_test?

    • Jason Brownlee July 15, 2020 at 8:18 am #

      You fit a new final model using the techniques and configuration you discovered work best on your dataset:
      https://machinelearningmastery.com/train-final-machine-learning-model/

      • suvarna punalekar July 15, 2020 at 9:20 pm #

        Thanks Jason, Yes that makes sense to me. But I am not sure how do I access selected features when I use ‘cross_val_score’ and the ‘pipeline’ in a loop (as you show in “RFE for Classification”).

        What I am trying to do through a loop is:

        1. For every combination of hyperparameters and RFE, run a model fit and cross_val_score

        2. Select best features with reference to this model and transform inputs

        3. Fit a new model using selected features only and use it to predict with test data

        4. Check accuracy, so that in a box plot I can also visualise for every model run how it performed on the test data.

        I have completely independent validation data that I would use at the end for independent validation for the ‘best model’.
        But I don’t know how I get selected features after calling cross_val_score.

        Thanks!!

        • Jason Brownlee July 16, 2020 at 6:36 am #

          You don’t need to access the features as RFE becomes part of your modeling pipeline.

          You can select the features chosen by RFE manually, but the point is you don’t need to. RFE does it for you.

  15. Vinayak Shanawad July 17, 2020 at 1:12 am #

    Nice article. Thanks

    1. When we use RFECV to Automatically Select the Number of Features then how can we know what features are selected using RFECV method?

    2. Can we first use RFECV and then do cross-validation using the model with the selected features from RFECV?

    • Jason Brownlee July 17, 2020 at 6:22 am #

      You can run the method manually if you like and have it print the features it selected.

      Yes.

  16. Vinayak Shanawad July 17, 2020 at 4:49 pm #

    Thanks. You mean manually means using RFE method?

    • Jason Brownlee July 18, 2020 at 5:59 am #

      I mean that you can run the RFE or RFECV method in a standalone manner and review what it is doing.

      • Vinayak Shanawad July 18, 2020 at 2:31 pm #

        Sure. Thanks a lot.

  17. Margo July 24, 2020 at 3:09 am #

    Very informative article. Thank you for posting.

    I have a question related to having a mixed data set i.e it has both numerical and categorical inputs and requires a categorical output. If I split the data set into two files one containing the numerical data and another containing the categorical data, and then run the appropriate feature selection method (eg ANOVA and Chi Squared) on each data set, then is it appropriate to use the information obtained on the most important features of each data type, to alter the original data set to select the appropriate fields? I am wondering if this is appropriate or if it introduces bias?

    • Jason Brownlee July 24, 2020 at 6:35 am #

      Thanks!

      Try it and see if it performs better than an RFE or using all features.

      It is bias, hopefully can find a way to do it within a cv fold.

  18. Anthony The Koala August 1, 2020 at 6:47 am #

    Dear Dr Jason,
    The code under subheading “Explore Base Algorithm”, the lines in particular are:
    lines 25-27. This is similar to the code used in your book, listing 15.21 p187

    We have the RFE select up to five features using LogisticRegression.
    How does the DecisionTreeClassifier work with the RFE(LogisticRegression).

    Thank you,
    Anthony of Sydney

    • Jason Brownlee August 1, 2020 at 1:27 pm #

      The decision tree will take the features selected by the RFE and fit a model.

      The logistic regression model is only used by the RFE to evaluate different subsets of features selected by the RFE.

      • Anthony The Koala August 1, 2020 at 7:42 pm #

        Dear Dr Jason,
        Thank you for the elaboration. It is appreciated.
        Anthony of Sydney

      • Anthony The Koala August 1, 2020 at 8:57 pm #

        Dear Dr Jason,
        Again thank you for the reply. I would like to apply your reply to listing 15.21 on page 186 (203 of 398) of “Data Preparation For Machine Learning” a book I highly recommend.

        Applying your previous answer in order to get a better understanding:
        The first line, rfe selects features using differerent subsets based on the DecisionTreeClassifier. The selected features are evaluated in the DecisionTreeClassifier = model.

        Thank you,
        Anthony of Sydney

        • Jason Brownlee August 2, 2020 at 5:43 am #

          Yes, it can be a good idea to use the same model within RFE as in following RFE.

          • Anthony The Koala August 2, 2020 at 6:25 am #

            Dear Dr Jason,
            Thank you, it has answered my question, it is appreciated.
            Anthony of Sydney

  19. Diego August 14, 2020 at 5:01 am #

    Thanks for sharing this. Your blog is better that sklearn documentation 🙂

    Quick question: When I tune n_features_to_select parameter and use a DecisionTreeClassifier as estimator I get similar results to yours, however when I use a LogisticRegression instead, I always get the same results, no matter the value of n_features_to_select is.

    With DecisionTreeClassifier: (f-score)
    >2 0.742 (0.009)
    >3 0.742 (0.009)
    >4 0.741 (0.009)
    >5 0.741 (0.009)
    >6 0.741 (0.009)
    >7 0.740 (0.010)
    >8 0.739 (0.010)
    >9 0.739 (0.010)

    With LogisticRegression: (f-score)
    >2 0.742 (0.009)
    >3 0.742 (0.009)
    >4 0.742 (0.009)
    >5 0.742 (0.009)
    >6 0.742 (0.009)
    >7 0.742 (0.009)
    >8 0.742 (0.009)
    >9 0.742 (0.009)

    What could be the reason ?
    Maybe it is not working since it is part of a Pipeline?

    Thanks

    • Jason Brownlee August 14, 2020 at 6:13 am #

      Thanks!

      Perhaps there is a bug in your logistic regression example?
      Perhaps varying the features does not impact model skill (unlikely)?

  20. Tushar yadav August 21, 2020 at 3:11 am #

    Hi Jason,

    Your articles are a great source of information. Have one question, if I understand it correctly, for using RFE, we need to at first normalize or standardize the available data in order to get the correct features according to the importance in the model. Else, the features with smaller values will have a higher coefficient associated and vice versa. This is my understanding. Please correct me if I am understanding it wrong.

    Thanks

    • Jason Brownlee August 21, 2020 at 6:34 am #

      Thank you!

      It depends on the model you are using. If you are using a tree within RFE, then no. If you are using a logistic regression, then probably yes.

  21. tushar yadav August 21, 2020 at 3:15 am #

    Hi Jason,

    Your articles are a great source of information

  22. Diego August 26, 2020 at 4:06 am #

    Hi Jason,

    When tuning the best of number of features to be selected by rfe, shouldn’t we drop duplicates before running the model ?

    i.e. if you keep only 2 variables, you will probably have more duplicated rows than if you use 5.
    This can happen with categorical features or a mixture between categorical / numerical

    Thanks a lot 🙂

  23. Noob_data_scientist September 5, 2020 at 7:37 am #

    can we plot RMSE(root mean squared error) using the RFE algorithm?
    as may not be as it is float
    how can i plot then ?

  24. shaheen September 9, 2020 at 2:07 am #

    what algorithms that can be used in the core RFE for regression and how to calculate accuracy for these algorithms thank you.

  25. shaheen September 10, 2020 at 1:43 am #

    when try to use algorithms that can be used in the core RF for regression problem not classification i get this error

    ValueError: Unknown label type: ‘continuous’

    what can I change in the code to avoid this error

    • Jason Brownlee September 10, 2020 at 6:33 am #

      Sorry to hear that you’re having problems, perhaps start with the regression example above and adapt it for your project?

  26. Ale September 17, 2020 at 6:05 am #

    Hi Jason is there a way to see which features were selected during the cross validation instead of fitting in all the data to get these?

    It seems to me that the accuracy you obtained in the section “Automatic Select the Number of Features” was not based on the features you obtained in the section “”Which features were selected?”. The first section uses CV and the second fits in all the data.

    • Jason Brownlee September 17, 2020 at 6:54 am #

      Yes, for each fold if you enumerate manually and print the features selected by the object.

      But why do you need to know? Do you care what coefficients a linear regression model chooses each fold? Never!

      • Ale September 20, 2020 at 1:50 am #

        I mean how can we extract the subset selected that outputs that cross-validation score.

        • Ale September 20, 2020 at 1:51 am #

          sorry , that was a question.

        • Jason Brownlee September 20, 2020 at 6:50 am #

          You can do it by manually enumeration the cv folds and evaluate your model, then access the RFE within the pipeline and print the reported features for that fold.

          But I am asking why do you want this information? It is irrelevant!

          • Ale September 20, 2020 at 9:09 am #

            I thinking of it for interpretation. To know which 10 features were found as the most important ones.

          • Jason Brownlee September 20, 2020 at 1:31 pm #

            You can run the model once in a standalone manner to discover what features might be important.

          • Ale September 20, 2020 at 9:11 am #

            But please let me know if there is a better way. Thanks for your help

          • Jason Brownlee September 20, 2020 at 1:32 pm #

            Yes, you can run the procedure on a train/test split of the data to learn more about the dataset.

            This is a separate procedure from evaluating modeling pipelines where knowing what features were selected does not change anything.

  27. Masoud October 6, 2020 at 4:49 am #

    Hi Jason,
    Thank you for the useful blog. Can you please tell me that, for a regression problem, if I can use the “DecisionTreeRegressor” as the estimator inside the RFE and “Deep Neural Network” as the model? If not, do you have any recommendation for the estimator inside the RFE when I want to select the best subset of data for a DNN algorithm?
    Thanks in advance,
    Masoud

    • Jason Brownlee October 6, 2020 at 7:00 am #

      Sure.

      Generally, it is a good idea to use the same algorithm in both cases, but it is not a rule.

      • Masoud October 7, 2020 at 4:03 am #

        Thank you dear Jason.

  28. Mac October 7, 2020 at 12:47 pm #

    Thank you, Jason, for the very informative blog post.

    I have a few questions to discuss.

    In the conventional method that the statistician uses to fit the regression model.
    They would do a single feature test for p-value and select only those variable which tends to have significant, such as p<0.15, for first-round filtering features.
    Then, they would use multiple regression and using forward or backward elimination for the final feature selection.

    Do you have information on which one gave a better result? Comparing single RFE alone, or perform the single feature filter first before RFE, or RFE vs forward/backward elimination.

  29. Graham October 10, 2020 at 8:03 pm #

    Is it worthwhile doing RFE when using more complex models, such as XGBoost?

  30. Marlon October 14, 2020 at 5:48 am #

    Hello Jason,

    first of all thanks for all your amazing work! I gained so much knowledge through your website. I can’t put into words how much I thank you for that.

    I implemented your pipeline on an own dataset. When I want to check on the different feature importances all 47 features are equally important. Looks like I got some leakage, doesn’t it?

    • Jason Brownlee October 14, 2020 at 6:26 am #

      You’re welcome.

      Maybe, or maybe the technique cannot tell the difference between your features – e.g. your problem might not be predictable.

  31. Matthew Avaylon October 16, 2020 at 5:18 am #

    Hi Jason

    I have a question. Say we are using RFE but we haven’t chosen a specific model to predict our output. As in we have a list of possibilities, whether that is SVM, Gradient Boost, or Random Forest etc (classification but also a different list for regression). When we perform cross-validation on RFE and set it up to automatically pick the number of features, would we have to repeat it for every model?

    Also you used DecisionTree as both the estimator and the model, can we use different models but keep the DecisionTree as the estimator? Ex) Estimator=DecisionTree Model=SVM

    Can we also use different estimators?

    • Matthew Avaylon October 16, 2020 at 5:24 am #

      Ahhh. I missed the end. Thanks for the article. I should finish reading before asking questions.

      • Matthew Avaylon October 16, 2020 at 5:50 am #

        Follow up:

        When doing feature selection and finding the best features from using RFE with cross-validation, when we test other ML algorithms for the actual modeling of the data, would we run into the issue that different models will work better with different chosen features?

        At the end of the article you compared different estimators with the same model. Say we pick the best one but later we still have to optimize the hyperparameters of the model using Gridsearch. How do we know that this is still the best model for us? How do we know that the other estimator/model combinations couldn’t be better if we optimized with grid search the hyperparameters in the model? At what point are we able to stop with that peace of mind?

        • Jason Brownlee October 16, 2020 at 6:00 am #

          Yes.

          Test everything, use what results in lowest error.

          Stop when the results are good enough, or when you run out of ideas, or when you run out of time.

      • Jason Brownlee October 16, 2020 at 6:00 am #

        No problem.

    • Jason Brownlee October 16, 2020 at 6:00 am #

      You can use different models inside and outside if you like, using the same model might be preferred.

      Each, when considering a suite of models, each should be considered in the same modeling pipeline (applying transforms to the data like rfe)

  32. Matthew Avaylon October 16, 2020 at 6:03 am #

    Would we remove highly correlated features before applying RFE?

    • Jason Brownlee October 16, 2020 at 8:08 am #

      Probably.

      It depends on the model you’re ultimately using. Also it depends if you have tons of features and are looking for ways to speed up your pipeline.

  33. Ansyl October 26, 2020 at 2:49 am #

    # define dataset
    X, y = make_classification(n_samples=1000, n_features=10, n_informative=5, n_redundant=5, random_state=1)

    # create pipeline
    rfe = RFECV(estimator=DecisionTreeClassifier())
    model = DecisionTreeClassifier()
    pipeline = Pipeline(steps=[(‘s’,rfe),(‘m’,model)])

    # evaluate model
    cv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)
    n_scores = cross_val_score(pipeline, X, y, scoring=’accuracy’, cv=cv, n_jobs=-1, error_score=’raise’)

    # report performance
    print(‘Accuracy: %.3f (%.3f)’ % (mean(n_scores), std(n_scores)))

    I want to know how many features by RFECV but since it is in pipeline object I am not able to get the support and rank attribute. Please help!!

    • Jason Brownlee October 26, 2020 at 6:51 am #

      If you run RFE on a train set only you can see the features that were selected – as we do in the above tutorial.

      When used as part of a pipeline, we don’t care what was selected, just like we don’t care what the specific decision tree looks like – only that it performs well.

  34. Akshay November 20, 2020 at 7:00 am #

    Hey,

    Thanks for such a nice tutorial.

    I am not able to understand one thing. We use RFE inside pipeline and then use gridsearchCV to find out optimal number of features let’s say [2,5,10].

    So my understanding is, gridsearchCV will split the data into k folds. Use k-1 subsets for training and apply RFE on it to select best performing features. Finally evaluate it on remaining subsets to evaluate it. Repeat the same step k times to find out the average model performance. Follow the same procedure for each value for RFE features.

    My question is, does RFE select same features in each fold or they could be different. If the later is true, why gridsearchCV returns only one list of selected features for best performing parameters/model.

    Thank you in advance.

    • Jason Brownlee November 20, 2020 at 7:38 am #

      RFECV will select the number of features for you, no need to grid search as well.

  35. Hou November 22, 2020 at 8:15 pm #

    Thank you, Jason, for the very informative blog post.

    I have a question.When I use RFECV, why I get different result for each run.Sometime return 1 feature to select, sometime return 15 features.Thank you so much.

Leave a Reply