How to Build an Ensemble Of Machine Learning Algorithms in R

Ensembles can give you a boost in accuracy on your dataset.

In this post you will discover how you can create three of the most powerful types of ensembles in R.

This case study will step you through Boosting, Bagging and Stacking and show you how you can continue to ratchet up the accuracy of the models on your own datasets.

Kick-start your project with my new book Machine Learning Mastery With R, including step-by-step tutorials and the R source code files for all examples.

Let’s get started.

Build an Ensemble Of Machine Learning Algorithms in R

Build an Ensemble Of Machine Learning Algorithms in R
Photo by Barbara Hobbs, some rights reserved.

Increase The Accuracy Of Your Models

It can take time to find well performing machine learning algorithms for your dataset. This is because of the trial and error nature of applied machine learning.

Once you have a shortlist of accurate models, you can use algorithm tuning to get the most from each algorithm.

Another approach that you can use to increase accuracy on your dataset is to combine the predictions of multiple different models together.

This is called an ensemble prediction.

Combine Model Predictions Into Ensemble Predictions

The three most popular methods for combining the predictions from different models are:

  • Bagging. Building multiple models (typically of the same type) from different subsamples of the training dataset.
  • Boosting. Building multiple models (typically of the same type) each of which learns to fix the prediction errors of a prior model in the chain.
  • Stacking. Building multiple models (typically of differing types) and supervisor model that learns how to best combine the predictions of the primary models.

This post will not explain each of these methods. It assumes you are generally familiar with machine learning algorithms and ensemble methods and that you are looking for information on how to create ensembles with R.

Need more Help with R for Machine Learning?

Take my free 14-day email course and discover how to use R on your project (with sample code).

Click to sign-up and also get a free PDF Ebook version of the course.

Ensemble Machine Learning in R

You can create ensembles of machine learning algorithms in R.

There are three main techniques that you can create an ensemble of machine learning algorithms in R: Boosting, Bagging and Stacking. In this section, we will look at each in turn.

Before we start building ensembles, let’s define our test set-up.

Test Dataset

All of the examples of ensemble predictions in this case study will use the ionosphere dataset.

This is a dataset available from the UCI Machine Learning Repository. This dataset describes high-frequency antenna returns from high energy particles in the atmosphere and whether the return shows structure or not. The problem is a binary classification that contains 351 instances and 35 numerical attributes.

Let’s load the libraries and the dataset.

Note that the first attribute was a factor (0,1) and has been transformed to be numeric for consistency with all of the other numeric attributes. Also note that the second attribute is a constant and has been removed.

Here is a sneak-peek at the first few rows of the ionosphere dataset.

For more information, see the description of the Ionosphere dataset on the UCI Machine Learning Repository.

See this summary of published world-class results on the dataset.

1. Boosting Algorithms

We can look at two of the most popular boosting machine learning algorithms:

  • C5.0
  • Stochastic Gradient Boosting

Below is an example of the C5.0 and Stochastic Gradient Boosting (using the Gradient Boosting Modeling implementation) algorithms in R. Both algorithms include parameters that are not tuned in this example.

We can see that the C5.0 algorithm produces a more accurate model with an accuracy of 94.58%.

Boosting Machine Learning Algorithms in R

Boosting Machine Learning Algorithms in R

Learn more about caret boosting models tree: Boosting Models.

2. Bagging Algorithms

Let’s look at two of the most popular bagging machine learning algorithms:

  • Bagged CART
  • Random Forest

Below is an example of the Bagged CART and Random Forest algorithms in R. Both algorithms include parameters that are not tuned in this example.

We can see that random forest produces a more accurate model with an accuracy of 93.25%.

Bagging Machine Learning Algorithms in R

Bagging Machine Learning Algorithms in R

Learn more about caret bagging model here: Bagging Models.

3. Stacking Algorithms

You can combine the predictions of multiple caret models using the caretEnsemble package.

Given a list of caret models, the caretStack() function can be used to specify a higher-order model to learn how to best combine the predictions of sub-models together.

Let’s first look at creating 5 sub-models for the ionosphere dataset, specifically:

  • Linear Discriminate Analysis (LDA)
  • Classification and Regression Trees (CART)
  • Logistic Regression (via Generalized Linear Model or GLM)
  • k-Nearest Neighbors (kNN)
  • Support Vector Machine with a Radial Basis Kernel Function (SVM)

Below is an example that creates these 5 sub-models. Note the new helpful caretList() function provided by the caretEnsemble package for creating a list of standard caret models.

We can see that the SVM creates the most accurate model with an accuracy of 94.66%.

Comparison of Sub-Models for Stacking Ensemble in R

Comparison of Sub-Models for Stacking Ensemble in R

When we combine the predictions of different models using stacking, it is desirable that the predictions made by the sub-models have low correlation. This would suggest that the models are skillful but in different ways, allowing a new classifier to figure out how to get the best from each model for an improved score.

If the predictions for the sub-models were highly corrected (>0.75) then they would be making the same or very similar predictions most of the time reducing the benefit of combining the predictions.

We can see that all pairs of predictions have generally low correlation. The two methods with the highest correlation between their predictions are Logistic Regression (GLM) and kNN at 0.517 correlation which is not considered high (>0.75).

Correlations Between Predictions Made By Sub-Models in Stacking Ensemble

Correlations Between Predictions Made By Sub-Models in Stacking Ensemble

Let’s combine the predictions of the classifiers using a simple linear model.

We can see that we have lifted the accuracy to 94.99% which is a small improvement over using SVM alone. This is also an improvement over using random forest alone on the dataset, as observed above.

We can also use more sophisticated algorithms to combine predictions in an effort to tease out when best to use the different methods. In this case, we can use the random forest algorithm to combine the predictions.

We can see that this has lifted the accuracy to 96.26% an impressive improvement on SVM alone.

You Can Build Ensembles in R

You do not need to be an R programmer. You can copy and paste the sample code from this blog post to get started. Study the functions used in the examples using the built-in help in R.

You do not need to be a machine learning expert. Creating ensembles can be very complex if you are doing it from scratch. The caret and the caretEnsemble package allow you start creating and experimenting with ensembles even if you don’t have a deep understanding of how they work. Read-up on each type of ensemble to get more out of them at a later time.

You do not need to collect your own data. The data used in this case study was from the mlbench package. You can use standard machine learning dataset like this to learn, use and experiment with machine learning algorithms.

You do not need to write your own ensemble code. Some of the most powerful algorithms for creating ensembles is provided by R, ready to run. Use the examples in this post to get started right now. You can always adapt it to your specific cases or try out new ideas with custom code at a later time.

Summary

In this post you discovered that you can use ensembles of machine learning algorithms to improve the accuracy of your models.

You discovered three types of ensembles of machine learning algorithms that you can build in R:

  • Boosting
  • Bagging
  • Stacking

You can use the code in this case study as a template on your current or next machine learning project in R.

Next Step

Did you work through the case study?

  1. Start your R interactive environment.
  2. Type or copy-paste all of the code in this case study.
  3. Take the time to understand each part of the case study using the help for R functions.

Do you have any questions about this case study or using ensembles in R? Leave a comment and ask and I will do my best to answer.

Discover Faster Machine Learning in R!

Master Machine Learning With R

Develop Your Own Models in Minutes

...with just a few lines of R code

Discover how in my new Ebook:
Machine Learning Mastery With R

Covers self-study tutorials and end-to-end projects like:
Loading data, visualization, build models, tuning, and much more...

Finally Bring Machine Learning To Your Own Projects

Skip the Academics. Just Results.

See What's Inside

172 Responses to How to Build an Ensemble Of Machine Learning Algorithms in R

  1. Avatar
    Robert April 8, 2016 at 12:24 am #

    Thanks for posting Jason, very helpful.

    One question: In the stacked random forest and GLM ensemble models, how are the hyperparameters specified for each model (e.g., which value of k for k Nearest Neighbor, how many trees in the random forest)? I’m assuming default values are used, but can these values be tweaked within the caretEnsemble package?

    • Avatar
      Jason Brownlee April 8, 2016 at 1:36 pm #

      The algorithms use “sensible” defaults. I think 100 trees for RF, I don’t recall the value for K in KNN. The caret docs will specify the default or you can print the model after training.

      • Avatar
        Gaurav November 11, 2017 at 5:39 pm #

        How to make predict function work after building models
        Also can these methods be used for Multi class classification beause while doing caretstack I get error for multiclass classification

    • Avatar
      Ben May 19, 2018 at 8:25 pm #

      Hi,

      thanks for this tutorial. I have a question regarding the validation of the models. Should I always use the same seeds (folds) for the whole evaluation process (i.e. tuning the individual models, stacking the models etc.). I think it´s really hard to get valid numbers for the hyperparameters for the individual models, good performance of the individual models and uncorrelated models at the same time. For example, if I use 10 * 10 cross-validation with the same seeds (folds) for all models their performance is quite stable but the intercorrelation of their results tends to be quiet high.

      • Avatar
        Jason Brownlee May 20, 2018 at 6:37 am #

        No, I think it is better to evaluate the model including the variance in the data and model (e.g. average over many runs).

  2. Avatar
    Sreenath April 19, 2016 at 11:19 pm #

    Hi Jason,

    I started following your posts recently and so far this is one of the best post I came across, not just in this website but among all the ML websites I have visited. Really appreciate your efforts to explain such a useful topic in a simple & clear manner. I will definitely try out this technique soon. Thanks again..

  3. Avatar
    pradnya May 6, 2016 at 5:51 am #

    One of best articles I came accross. Superhelpful

    Thanks!!

  4. Avatar
    Ahmed Mohy June 3, 2016 at 6:19 pm #

    one question : how to stack models of different subsets of a dataset into one model i tried caretStack but i got an error

    • Avatar
      Jason Brownlee June 15, 2016 at 5:54 am #

      Sorry to hear that. What error did you get?

      Did the above example work for you?

  5. Avatar
    Ramasubramaniam June 14, 2016 at 8:51 pm #

    Fantastic tutorial!!. One of the most simple and elegant ways of introducing ensembles. As someone mentioned in earlier comment, this is one of the cutest resource for introducing ensembles.

  6. Avatar
    Mudit June 18, 2016 at 5:07 am #

    models <- caretList(Loan_Status~., data=train1, trControl=control, methodList=algorithmList)

    Throws a error

    Error in { : task 1 failed – "argument is not interpretable as logical"
    In addition: There were 32 warnings (use warnings() to see them)

    any suggestion ?

    • Avatar
      Jason Brownlee June 18, 2016 at 6:39 am #

      It looks like you are adapting the example to your own problem nice work!

      Were you able to reproduce the example in the tutorial first? This will confirm your environment is working correctly.

      I have not seen this error before, but I suspect it is related to your dataset. Confirm the data is in a form to make it easy to model, such as all numeric attributes and a factor being predicted.

      You can ignore the warnings for now.

      • Avatar
        Mudit June 19, 2016 at 2:19 am #

        Hi Jason,

        The environment is fine as i have tried the other codes of your to as well and those run fine……

        Can I send you the dataset i m using on your mail id and you can help me with the code and let me know what mistake i m doing….just to give you a heads up i have also converted all the variables to as.numeric before applying the algorithm but i m still facing the same issue.

        Please if you can help me with your mail id to send the dataset.

        Regards
        Mudit

  7. Avatar
    Mudit June 22, 2016 at 1:41 am #

    Hi Jason,

    I totally respect your time constraint……..its just when I’m stuck with code it get hard to debug…seeing the same code working on other dataset fine….

    Regards
    Mudit

    • Avatar
      Rajagopalan Kannan July 1, 2016 at 2:49 am #

      Hi Mudit,
      I see you are participating in AV Loan default prediction competition. I am also participating in the same in the learning mode and my score (so far) is same as yours. At present i am working on stacking using caretEnsemble, I have not come across the error that you are seeing. Would you be able to provide more info (or) share code.

      Thanks,
      Raj

  8. Avatar
    Komal Sinha July 1, 2016 at 5:27 am #

    I need to do combine 6 different predictive models , this is a 7 class problem.
    Each model predicts output probability for being in all the seven classes (lib SVMs using -b 1 option) and I need to combine them to get a better model. I just have to use the predictions of these models to train the ensemble.
    Can you please tell me how to do the ensemble learning for multiclass problem and how this can be done in matlab ? Please reply as soon as possible. I shall be thankful. Thanks !

    • Avatar
      Jason Brownlee July 1, 2016 at 5:44 am #

      Sorry, I don’t have any examples in Matlab at the moment.

  9. Avatar
    Rajagopalan Kannan July 1, 2016 at 6:28 am #

    Hi Jason,
    I have built many models with different pre-processing done on each model and the model itself is unique tuned by its own tune parameters and i want to stack them now. how do i get them into caretStack, because caretStack requires caretList to be its input. but i have generated models using caret train function. hope you got what i am saying. Appreciate your response.

    Cheers,
    Raj

    • Avatar
      Rajagopalan Kannan July 1, 2016 at 6:44 am #

      Jason, no worries. i figured it out. thanks.

      • Avatar
        Ernest September 13, 2016 at 6:37 am #

        How did you do it? I am currently working on one such problem. Could you advise me on how to go about it. Using a caretlist with different preprocessing, and different variables for each model to build a stacked ensemble.

  10. Avatar
    Seun August 3, 2016 at 3:46 am #

    Hi Jason,
    I am currently trying to construct an ensembles of classifiers using the stacking technique. I am currently using the Naive Bayes and j48 classifier as the base leaarners with Random Forest as the meta learner.Where can I find the correlation values for predictions in Weka?
    Thanks

    • Avatar
      Jason Brownlee August 3, 2016 at 8:21 am #

      Hi Seun, if you are looking to investigate the correlation between prediction errors in Weka, you will have to do this manually. I don’t believe there is facility.

      You will have to make predictions on a test dataset for each classier, save them to CSV files and then calculate the PEARSON() correlation in MS Excel or similar.

      I hope that helps.

      • Avatar
        Seun August 10, 2016 at 10:43 am #

        Hi Jason,Thanks. Thats sounds like a lot of work.

  11. Avatar
    babi September 27, 2016 at 5:48 pm #

    what changes shd be made if i deal with a categorical featurespace with large no of levels. on execution if this code shows an error “Error in train.default(x, y, weights = w, …) : Stopping
    In addition: There were 33 warnings (use warnings() to see them)”..

    • Avatar
      babi September 27, 2016 at 5:49 pm #

      i was talking about executing stacking

    • Avatar
      Jason Brownlee September 28, 2016 at 7:40 am #

      A hard question babi. Try converting the categorical features to binary features and see if that makes a difference.

  12. Avatar
    Narendra Prasad K October 18, 2016 at 5:48 am #

    Thanks a lot Jason.. This article is very helpful to Ensemble the model.

    I tried one of our regression Problem, the RMSE values is increased..

    is there any article to Merge the models other than caret Packages, If it is ok…

    Thanks,
    Narendra Prasad K

    • Avatar
      Jason Brownlee October 18, 2016 at 5:56 am #

      You can merge the predictions from models manually, that is how we had to do it before caretEnsemble.

  13. Avatar
    Siddhesh November 6, 2016 at 6:48 am #

    Can i merge classification model like logistic regression and linear regression? If so which method is to be used?

    • Avatar
      Jason Brownlee November 7, 2016 at 7:11 am #

      Hi Siddhesh,

      Yes, you can create an ensemble by combining the predictions from these models. You can use voting (mode), take the average or use another model to learn how to best combine the predictions (called stacked generalization or stacking).

  14. Avatar
    Ajas November 19, 2016 at 8:37 am #

    Hey Jason,

    The tutorial is awesome, but i am not able to install caretEnsemble pkg for R version 3.3.1, is there any other pkgs available for same task or any workaround.

    Thanks for the help..!..

    • Avatar
      Jason Brownlee November 19, 2016 at 8:55 am #

      I’m sorry to hear that.

      Perhaps you could update to 3.3.2 and try again?

      Perhaps check stack overflow to see if anyone else has your error?

      You may even want to try to R mailing list, if you’re brave.

  15. Avatar
    Surya November 21, 2016 at 6:59 pm #

    models <- caretList(Item_Outlet_Sales~., data=BigMartimp, trControl=trainControl, methodList=algorithmList)
    results <- resamples(models)
    summary(results)
    dotplot(results)
    modelCor(results)
    splom(results)

    From above we found that Decision tree is having low correlation with all other models used in the method list previously.

    So while creating stackedensemble model do we need to remove the rpart and train the model again before doing stacking?

    stack.rf <- caretStack(models, method="rf", metric="RMSE", trControl=stackControl)

    • Avatar
      Jason Brownlee November 22, 2016 at 7:01 am #

      Good question Surya,

      You could create the stacked model yourself. I believe a limitation of caretStack is that it expects to train the submodels itself.

      • Avatar
        Abhi February 21, 2017 at 4:17 pm #

        Do you know where I can find code to implement Stacked Models in R? I am interested in tuning my xgboost models and stacking them with other optimized models. Any pointers towards examples/implementations would be awesome!

        • Avatar
          Jason Brownlee February 22, 2017 at 9:57 am #

          Does the stacking example in this post not help?

          XGBoost is available in R and caret, and can be used from caretEnsemble.

  16. Avatar
    Surya November 21, 2016 at 8:34 pm #

    Let’s say you have got predictions where there are two models which are highly correlated (>0.75). Should we apply the same code as below for the mentioned example in the article?

    # stack using random forest
    set.seed(seed)
    stack.rf <- caretStack(models, method="rf", metric="Accuracy", trControl=stackControl)
    print(stack.rf)

    • Avatar
      Jason Brownlee November 22, 2016 at 7:05 am #

      I would suggest try and see if you can lift the performance by using a stacked model. Even just a few better predictions can help model performance.

    • Avatar
      Surya November 22, 2016 at 6:21 pm #

      Sorry, if you can elaborate on this will be great. How stacking algorithm has to be used when One model is having low correlation with all other models.

      • Avatar
        Jason Brownlee November 23, 2016 at 8:55 am #

        I don’t remember the specifics of the paper, but it was along the lines of better performance when predictions between weak learners are uncorrelated (or maybe it was the errors being uncorrelated). This may have been in PAC learning theory – it has been a while, sorry.

  17. Avatar
    Gunwoo Nam November 30, 2016 at 1:12 pm #

    Hi, thanks for nice introduction to ensemble techniques.

    I have one question. Can I ensemble 1 method repeatedly?
    It seems CaretEnsemble mix different methods.
    What if I want to ensemble a single method many times?(like boosting)

    Function “train” seems to help resampling data and apply method repeatedly. But does it mean bagging or boosting? Or it just helps to pick a best parameter for single method?

    Thanks in advance.

    • Avatar
      Jason Brownlee December 1, 2016 at 7:23 am #

      You can, but may not get any gain Gunwoo.

      You could create 10 neural nets and take the mean prediction – this would probably outperform a stacking approach.

      You can put other methods inside bagging (I don’t remember the package name off hand), but bagging works so much better with high variance methods – like unpruned trees or low-k in KNN with subsampled training data, etc.

      At the end of the day, try different methods and see what gives the best results on your problem.

  18. Avatar
    Anh Bui December 6, 2016 at 1:25 am #

    Thanks for your writting!
    Can I ask you a question?
    What is the maximum model can combine in this package?

    • Avatar
      Jason Brownlee December 6, 2016 at 8:27 am #

      I don’t know Anh. Perhaps it is limited by memory and CPU power.

      • Avatar
        Anh Bui December 6, 2016 at 8:58 pm #

        I got it. Thank you so much

  19. Avatar
    Jay Hyunwoo Jeong December 19, 2016 at 1:53 am #

    This post is the most useful job that I’ve seen about ensemble modeling!!
    I have a question.

    below sentences seem to show correlations between resampled accuracy measures of each models,
    not correlations between predictions.

    # correlation between results
    modelCor(results)
    splom(results)

    correlations between accuracy ratios can be interpreted as them between predictions
    by each model ?

  20. Avatar
    Aadi January 5, 2017 at 8:54 pm #

    Hello Jason, first l would like to thank you for such a nice article of ensemble methods. I have a question that may be naive but i m confused,
    My question is that since ensemble methods are used on a base classifer model (naive base classifier, SVM etc.) to improve accuracy of the base model but how to choose on which classifier we should apply ensemble methods.
    How do we know that wihch classifier is best for applying ensemble methods?

    • Avatar
      Jason Brownlee January 6, 2017 at 9:09 am #

      Great question Aadi.

      Generally, we cannot know before hand. Use trial and error.

      Perhaps we can ensemble a suite of well-performing models in the case of stacked generalization.

      Perhaps we can ensemble a suite of well performing high variance models in the case of bagging.

      It is really problem specific. This post will shed more light on the open problem of algorithm selection:
      https://machinelearningmastery.com/a-data-driven-approach-to-machine-learning/

  21. Avatar
    Carl Turner January 26, 2017 at 7:58 am #

    Best explanation of ensembles I’ve seen. Thanks for posting this

  22. Avatar
    shikha March 23, 2017 at 4:00 pm #

    we need to combine c5.0 and random forest for our dataset can we do it and how ?

    • Avatar
      Jason Brownlee March 24, 2017 at 7:52 am #

      Yes, the tutorial above should help. Sorry I cannot write the code for you.

  23. Avatar
    Zhying April 16, 2017 at 10:43 pm #

    Hi Jason, really inspired by your post. However, I have one question regarding the output of the stacking model.
    “A rf ensemble of 2 base models: lda, rpart, glm, knn, svmRadial”, why there are only 2 base models? 2 is selected by the accuracy?

    • Avatar
      Jason Brownlee April 17, 2017 at 5:14 am #

      Good question.

      I think that is a typo in the R code for the ensemble package. I would suggest ignoring it.

  24. Avatar
    Prakash April 18, 2017 at 1:22 am #

    Can we have a deep learning system with (n-1) neural net layers and a final layer with a different algorithm, say SVM?

  25. Avatar
    olufemi April 20, 2017 at 5:38 am #

    Hi,

    Is there any advantage in converting the first column to a numeric value instead of keeping it as a factor?

    Thanks

    • Avatar
      Jason Brownlee April 20, 2017 at 9:33 am #

      This really depends on the specific data and the algorithms being used.

      Most algorithms expect to work with numeric data instead of factors.

  26. Avatar
    Luis April 27, 2017 at 2:16 am #

    Hi, Jason.

    what would be your recommended library to do something like this in python?

    Thanks.

  27. Avatar
    Shankar Rao Pandala May 10, 2017 at 1:45 pm #

    This is definitely one of the best tutorial for ensemble learning using R for participants in competitions.

    Can we have an tutorial for multi level ensemble models which are winning the competitions ?

  28. Avatar
    Sergio June 10, 2017 at 10:52 am #

    Thanks Jason, it is a really clear and sound example. I really appreciated. I am starting in this arena, and it is pretty impressive the method.

    I found references to a package named SuperLearner in R. I wonder if this package also work for this sort of examples or it has restrictions. I have a quick look at the documentation and I see that mentioned parametric models. Do you anything about SuperLearner?

    • Avatar
      Jason Brownlee June 11, 2017 at 8:20 am #

      Thanks Sergio.

      Sorry, I am not familiar with that package.

  29. Avatar
    Pat June 15, 2017 at 11:01 pm #

    You are the best and most unselfish ML expert out there. Reproducible, simple, and well explained.

    • Avatar
      Jason Brownlee June 16, 2017 at 7:59 am #

      Thanks Pat, that is very kind of you to say.

      I’m very happy to hear you found the example useful!

  30. Avatar
    Ellakkia June 27, 2017 at 2:24 am #

    Thank You Mr.Jason, very nice explanation. Can you please upload the same tutorial using some hydrological data?

  31. Avatar
    Eric June 28, 2017 at 6:55 am #

    One thing that would improve the tutorial is to show how to use the stacked models on the testing dataset or on new data for making predictions.

  32. Avatar
    Gary August 16, 2017 at 7:39 pm #

    Just wow Jason. Thank you so much for this very useful tutorial on ensemble methods. I have bought many a book on Machine Learning in R over the last 5 years and I think this is the best summary of how you can use multiple machine learning methods together to enable you to select the best option and the method which is most fit for purpose.

    Again, you have proverbially saved my bacon here – so thank you!

  33. Avatar
    veera krishna August 17, 2017 at 3:38 pm #

    can i make an ensemble to predict unknown data.
    like :
    https://archive.ics.uci.edu/ml/datasets/diabetes
    dataset
    consisting of
    (1) Date in MM-DD-YYYY format
    (2) Time in XX:YY format
    (3) Code
    (4) Value

    number of codes are 20

    33 = Regular insulin dose
    34 = NPH insulin dose
    35 = UltraLente insulin dose
    48 = Unspecified blood glucose measurement
    57 = Unspecified blood glucose measurement
    58 = Pre-breakfast blood glucose measurement
    59 = Post-breakfast blood glucose measurement
    60 = Pre-lunch blood glucose measurement
    61 = Post-lunch blood glucose measurement
    62 = Pre-supper blood glucose measurement
    63 = Post-supper blood glucose measurement
    64 = Pre-snack blood glucose measurement
    65 = Hypoglycemic symptoms
    66 = Typical meal ingestion
    67 = More-than-usual meal ingestion
    68 = Less-than-usual meal ingestion
    69 = Typical exercise activity
    70 = More-than-usual exercise activity
    71 = Less-than-usual exercise activity
    72 = Unspecified special event

    value represents for the respective code of a record

    i have understood the dataset and transformed it to

    input codes and output codes

    i.e input codes means code values for glucose value
    output codes as insulin values

    as output vector codes: 33,34,35.
    input codes vector: 48,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72.

    sample file contents of data01
    date time code value
    1 04-21-1991 9:09 58 100
    2 04-21-1991 9:09 33 9
    3 04-21-1991 9:09 34 13
    4 04-21-1991 17:08 62 119
    5 04-21-1991 17:08 33 7
    after transformation
    the sample table:
    c48 c57 c58 c59 c60 c61 c62 c63 c64 c65 c66 c67 c68 c69 c70 c71 c72 c33 c34 c35 c33x c34x c35x
    1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    2 0 0 100 0 0 0 0 0 0 0 0 0 0 0 0 0 0 9 13 0 0 0 0
    3 0 0 0 0 0 0 119 0 0 0 0 0 0 0 0 0 0 7 0 0 0 0 0
    4 123 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    5 0 0 216 0 0 0 0 0 0 0 0 0 0 0 0 0 0 10 13 0 2 0 0

    i used knn
    and neural net to predict by giving any one code value and in response i got respective values which i trained the machine, my question is is it possible to predict the unknown data on which it has not trained
    for example
    upto glucose values i trained the machine for 350
    if i have given 600 then why it is not giving the relative output it is showng the last i.e nearest neighbour value as a predicted value in neural networks..

  34. Avatar
    Stephen-Fashoto October 18, 2017 at 8:32 pm #

    Hi Jason, Thanks for this great work.

  35. Avatar
    Bhavesh November 9, 2017 at 5:21 am #

    This one is really superb Jason. I used this for public kernel for titanic competition in Kaggle(with due credit of this article). https://www.kaggle.com/bhavesh09/stacking-example

    However my score is just around 40%. I am not able to figure out why?

    • Avatar
      Jason Brownlee November 9, 2017 at 10:04 am #

      You will need to tune it to your specific dataset.

  36. Avatar
    Mohammad December 8, 2017 at 6:57 am #

    Hi jason, thanks for this great work.
    i have a question, if i have a data set contains of android code and i need to apply ensemble method on this data set to improve the accuracy how i can apply ensemble techniques can i use weka tool or if u have another way to apply this work.

    thank you

  37. Avatar
    YiLi March 2, 2018 at 10:41 am #

    Hi Jason,

    why do you think C5.0 is a boosting algorithm? Based on my understanding, C5.0 is just a simple decision tree algorithm.

    • Avatar
      Jason Brownlee March 2, 2018 at 3:23 pm #

      From Page 392 of “Applied predictive modeling”:

      C5.0 is a more advanced version of Quinlan’s C4.5 classification model that has additional features, such as boosting and unequal costs for different types of errors.

  38. Avatar
    Jagan April 1, 2018 at 2:24 am #

    Hi Jason,

    I am planning to stack different algorithms like Artificial Neural Network, SVM, KNN, C5.0, Random Forest in my research work for predicting diabetes disease…Whether this is feasible?? or whether the library above works only with certain algorithms?

    • Avatar
      Jason Brownlee April 1, 2018 at 5:50 am #

      Sure, try it and see how you go on your specific dataset.

      • Avatar
        Jagan April 2, 2018 at 12:15 am #

        Any suggestions before implementation of my model? Or could you please share your previous experience on doing so?

  39. Avatar
    Rajat April 8, 2018 at 5:00 pm #

    Hello Sir,
    I am trying to stacking using neural network for my research problem. I want to know whether it will be apt to use NN in case of stacking?

    • Avatar
      Jason Brownlee April 9, 2018 at 6:07 am #

      Perhaps try and discover what works best on your specific dataset.

  40. Avatar
    chen May 6, 2018 at 12:14 pm #

    don’t you split a trainingdata and a testdata?

    • Avatar
      chen May 6, 2018 at 12:16 pm #

      how do you define a test set and a training set?

        • Avatar
          chen May 11, 2018 at 5:05 pm #

          thank you for your advice! And I tried to use stacking,but the following error occurred

          Error: wrong model type for regression
          In addition: Warning messages:
          1: In trControlCheck(x = trControl, y = target) :
          x$savePredictions == TRUE is depreciated. Setting to ‘final’ instead.
          2: In trControlCheck(x = trControl, y = target) :
          indexes not defined in trControl. Attempting to set them ourselves, so each model in the ensemble will have the same resampling indexes.

          can you give me some advice,please

          • Avatar
            Jason Brownlee May 12, 2018 at 6:27 am #

            Sorry, I have not seen this error, perhaps post to stackoverflow?

        • Avatar
          chen May 20, 2018 at 7:05 pm #

          I‘ve solved the previous problem,another questions
          1.is there any way to add the xgboost models?
          2.can change KAPPA to AUC to validate model?
          thank you!

          • Avatar
            Jason Brownlee May 21, 2018 at 6:29 am #

            Sure, you can use xgboost in R. Using them as part of another ensemble such as voting or stacking might not add a lot of value.

            I believe you can calculate any metrics you wish. Sorry, I don’t have a worked example for AUC.

    • Avatar
      Jason Brownlee May 7, 2018 at 6:48 am #

      Yes, that can be a good way to evaluate the skill of machine learning algorithms on new data.

      Here, we use repeated k-fold cross-validation.

  41. Avatar
    Rahayu June 7, 2018 at 6:06 pm #

    Good evening mr. Jason.
    I want to asking , what ensemble learning bagging-boosting-random forest for logistic regression and linear discriminant. Any specific syntac bagging-boosting-random forest for that classifier?

    • Avatar
      Jason Brownlee June 8, 2018 at 6:07 am #

      Bagging can be used with each algorithm, it is better suited to algorithms with high variance like unpruned trees.

  42. Avatar
    chen June 10, 2018 at 3:58 pm #

    Hello, I’d like to ask why the relevance of the model below 0.75 is feasible?why is not 0.7 or 0.8? Is there any papers to support that? Thank you

    • Avatar
      Jason Brownlee June 11, 2018 at 6:07 am #

      It is just an arbitrary cut-off for high correlation. You can choose another value if you wish.

  43. Avatar
    Julien July 21, 2018 at 4:42 am #

    When you say that the “predictions ” must not be too correlated, do you mean the posterior probabilities (not the predicted values which are two possible values)?

    • Avatar
      Jason Brownlee July 21, 2018 at 6:39 am #

      I mean the samples of predictions from the two methods.

      • Avatar
        Julien July 21, 2018 at 4:32 pm #

        But in most cases, the predictions are the same for every method here, since it’s a binary classification problem. How can the correlations be the so small ?

        Other question : the caretList function does not seem to support missing values. How do you deal with that ?

        • Avatar
          Jason Brownlee July 22, 2018 at 6:21 am #

          I’m not sure caret supports imputing, perhaps prepare the data before hand.

  44. Avatar
    Tarek September 1, 2018 at 4:03 am #

    Could you help me to find solution for that error (Value Error: could not convert string to float: ‘ [-5.67604346 -0.37017465 1.41012661 2.30903599 2.85538861] ‘)?

    • Avatar
      Jason Brownlee September 1, 2018 at 6:22 am #

      Sorry, I don’t know the cause of your error. Perhaps try posting all code and data to stackoverflow?

  45. Avatar
    Steve September 11, 2018 at 2:04 pm #

    Hi Jason,

    I am trying different algorithms on classifying cancer as either Malignant or Benign. When I compare these models (LDA, NN, KNN, RF, NB…) using modelCor function in caret package, I get negative correlation coefficient form some models. Is this something possible and how can they (negatively correlated) models be interpreted?

    Thanks,

    • Avatar
      Jason Brownlee September 11, 2018 at 2:18 pm #

      Yes, negative correlation means when the value of one goes up, the value of the other goes down.

  46. Avatar
    arunima ghosh December 16, 2018 at 11:14 pm #

    Thanks for the tutorial. It was very much helpful. Can you please suggest how can I plot the ROC curve for a stacked model? Following your example, I have generated stack.svm, now please tell me how can I draw a ROC curve for it?

    • Avatar
      Jason Brownlee December 17, 2018 at 6:22 am #

      Sorry, I don’t have an example of plotting ROC curves in R.

  47. Avatar
    Nandhini January 9, 2019 at 5:20 pm #

    HI i have few questions,

    1. What is the reason behind choosing 5 models as base models. On what basis 5 models are chosen.

    2. what is the maximum number of base models we can have.

    3. Are the base models treated as uncorrelated decision trees by random forest algorithm? if not how the models are combined using random forest?

    • Avatar
      Jason Brownlee January 10, 2019 at 7:47 am #

      I chose 5 base models arbitrarily.

      You can have as many or fewer models as you like. Perhaps test to discover what works best on your problem.

      Ideally, the base models should make predictions (or make errors) that are weakly correlated.

  48. Avatar
    nandhini January 10, 2019 at 12:03 pm #

    2. what is the maximum number of base models we can have.

    3. Are the base models treated as uncorrelated decision trees by random forest algorithm? if not how the models are combined using random forest?

    • Avatar
      nandhini January 10, 2019 at 12:05 pm #

      how models are combined using random forest

      • Avatar
        Jason Brownlee January 11, 2019 at 7:38 am #

        Fitting unpruned trees where split points operate on a random subset of features.

    • Avatar
      Jason Brownlee January 11, 2019 at 7:38 am #

      No limit.

      In all methods, it helps if the models are weakly correlated. Random forest is an extension of bagging.

  49. Avatar
    Cory Garms January 17, 2019 at 9:30 am #

    Great Tutorial! Thank you Jason. Was able to easily adapt your script for my PhD Remote Sensing research at Oregon State.

  50. Avatar
    Beato Lau February 14, 2019 at 12:08 am #

    Hi Jason,

    When I ran my codes, it witnessed an error.

    > models <- caretList(Stroke~., data=d11, trControl=control, methodList=algorithmList)
    Error: At least one of the class levels is not a valid R variable name; This will cause errors when class probabilities are generated because the variables names will be converted to X0, X1 . Please use factor levels that can be used as valid R variable names (see ?make.names for help).
    In addition: Warning messages:
    1: In trControlCheck(x = trControl, y = target) :
    x$savePredictions == TRUE is depreciated. Setting to 'final' instead.
    2: In trControlCheck(x = trControl, y = target) :
    indexes not defined in trControl. Attempting to set them ourselves, so each model in the ensemble will have the same resampling indexes.

    any suggestion?
    Thanks for your time.

  51. Avatar
    Sophi March 18, 2019 at 8:43 pm #

    Hello Jason! your post helps me so much I really so thankful to you

    How many layers have you used in stacking example?
    if two, which classifiers exist or reside in the bottom layer and which classifiers reside in the top layer?

    in the following line of code does “glm” in the top layer?
    stack.glm <- caretStack(models, method="glm", metric="Accuracy", trControl=stackControl)

    based on what condition I should choose the top layer's model in order to best combine the predictions of all models?

    • Avatar
      Jason Brownlee March 19, 2019 at 8:56 am #

      This is explained in the post, I recommend re-reading that section.

      I recommend testing a suite of different models and discover what works best, ideally they should be somewhat uncorrelated in their predictions/error.

      • Avatar
        Sophi March 20, 2019 at 12:14 am #

        sorry, Jason, I read the section many times but I didn’t figure out how many layers in the stacking ensemble in your example? Maybe you refer to it implicitly throughout the article but I am a beginner and I didn’t get it?

        • Avatar
          Jason Brownlee March 20, 2019 at 8:32 am #

          Sorry, one level of base models and one level for the ensemble model.

          • Avatar
            Sophi March 20, 2019 at 5:48 pm #

            Thank you Jason that is very kind of you

  52. Avatar
    Andrew March 19, 2019 at 6:15 pm #

    Hi Jason
    your explanation is so clear and easy to grasp

    I have a question about modelCor() function, it computes the correlation according to the Pearson or Spearman correlation method?

    I would like to read more about the working of modelCor()function because I want to print the prediction’s matrix for each model in R. the models that are used as inputs to the modelCor function in order to see the matrix used in correlation

    is it possible to provide me a link or any information about modelCor()function because I read about this function in the help of R but the information was so limited?

  53. Avatar
    Rose August 1, 2019 at 1:35 pm #

    Hi Jason,
    Thank you very much for your instruction.
    I have 2 questions, hoping you will help me explain.
    1. In the part of Stacking Algorithms, I see that you used 5 sub-model, How I can find another model, Have any target?
    2. Why you choose Random forest and glm? How about the other?
    Thank you very much

    • Avatar
      Jason Brownlee August 1, 2019 at 2:14 pm #

      The answer to these questions can only be found via systematic experimentation – there are no theories of mapping algorithms to problems or algorithm configuration based on problem type.

      Test different numbers of sub-models on your problem to see what works best.

      Test different model types, to see what works best for your specific dataset.

  54. Avatar
    shahla August 31, 2019 at 9:15 am #

    hi Jason,
    thanks for the tutorial. it was great.
    does the stacking algorithm only work for binary classification?
    do you know how can I change the stacking code in order to use for Multi-class classification?

    • Avatar
      Jason Brownlee September 1, 2019 at 5:33 am #

      No, I believe you can use it directly for multi-class classification.

  55. Avatar
    Ankita September 15, 2019 at 10:47 pm #

    very useful example and nice tutorial..Pls tell me which coorelation function used by modelCOr() and why? Any refrence that i can use for modelCor()

    • Avatar
      Jason Brownlee September 16, 2019 at 6:36 am #

      Thanks.

      I would guess Pearson’s correlation coefficient for linear correlation, but you can check the documentation for the function to be sure.

  56. Avatar
    mohie eldin December 1, 2019 at 8:55 pm #

    This technique in ensemble is very easy but actually it gave me less accuracy than this technique :

    can any one tell me way ?

    • Avatar
      Jason Brownlee December 2, 2019 at 6:02 am #

      Yes, this can happen.

      It is important to test a suite of algorithms on your dataaset to discover what works best.

  57. Avatar
    parvin January 15, 2020 at 5:47 pm #

    Hi
    Thanks for your awesome post. I have some questions. What about tuning parameters in ensemble? And how we measure the accuracy?

    • Avatar
      parvin January 15, 2020 at 5:48 pm #

      Sorry instead of accuracy I meant variance.

      • Avatar
        Jason Brownlee January 16, 2020 at 6:10 am #

        You can collect the sample of accuracy scores across repeated k-fold cross validation and report the mean and standard deviation.

    • Avatar
      Jason Brownlee January 16, 2020 at 6:10 am #

      Yes, you can tune hyperparameters in the ensemble.

      You can measure accuracy as you do for any other model.

  58. Avatar
    Sithabile February 17, 2020 at 8:23 pm #

    Dear Jason

    Thank you very much for such a wonderful post which was a pleasure to follow. I have a set of 10 GCM predictions for which I would like to have the best prediction out of them (i.e. the best ensemble). I have clearly understood your tutorial, but is it possible to come up with a time series data file for the best ensemble. I would like to use the time series for further investigation.

    • Avatar
      Jason Brownlee February 18, 2020 at 6:17 am #

      You’re welcome.

      Perhaps. I don’t have an example, sorry.

  59. Avatar
    Neeraj Kumar March 18, 2020 at 11:09 pm #

    Hi Jason
    How to ensemble random forest and neural network in R.

    • Avatar
      Jason Brownlee March 19, 2020 at 6:27 am #

      Perhaps try a voting ensemble or a stacking ensemble.

      • Avatar
        Neeraj Kumar March 20, 2020 at 9:09 pm #

        Thank you. I need some help here, can you please state with some example that how to perform.

        • Avatar
          Jason Brownlee March 21, 2020 at 8:22 am #

          Sorry, I don’t have the capacity to prepare a custom example for you.

  60. Avatar
    Hunter Ross April 27, 2020 at 10:09 am #

    Hey Jason – really love this example & all of your explanations…each one is very succinct & easy to understand.

    I just have one question: In theory, should caretStack only be applied to base learners like a GLM & SVM or could you also use caretStack to combine boosting & bagging algorithms like C5.0, GBM, and RF so long as these models have little correlation to each other?

  61. Avatar
    Carlos June 15, 2020 at 1:16 am #

    Dear Jason,

    The “Bagging Models” does not work.

    Thanks for your job.

    • Avatar
      Jason Brownlee June 15, 2020 at 6:06 am #

      Sorry to hear that, what problem do you get exactly?

  62. Avatar
    Fernanda Alves June 19, 2020 at 9:36 am #

    Dear Jason,

    Thanks for the tutorial. I’ve tried to use modelCor() to present the correlation of some models in order to verify if an ensemble of the methods will improve the results. However, some correlation resulted in “NA”. Did you know some feature to correct this?

    Thanks in advance.

    Best regards.

    • Avatar
      Jason Brownlee June 19, 2020 at 1:11 pm #

      Do you mean the diagonal, if so, no need to calculate correlation of a sample with itself.

      • Avatar
        Fernanda Alves June 19, 2020 at 9:46 pm #

        No, other values of the matrix. Example: I consider four methods: rf, svmRadialWeights, gbm and rpart. The correlation of svmRadialWeights and rf is 0.5722874, the correlation of gbm and rf is -0.1273161, but when I calculate the correlation of rpart with any method the results are NA.

        When I run the method rpart individually it works fine.

        • Avatar
          Jason Brownlee June 20, 2020 at 6:13 am #

          If some of your values are na, then the result will also be na, you will need to remove them from the values.

          • Avatar
            Fernanda Alves June 20, 2020 at 7:44 am #

            I don’t have any NA values in my data, that’s why I didn’t understand the results of ensemble when using method rpart…

          • Avatar
            Jason Brownlee June 21, 2020 at 5:58 am #

            Perhaps it is worth posting your code, data, and error to stackoverflow.

  63. Avatar
    Cliff September 19, 2020 at 12:50 am #

    Hi Jason,

    Thanks for the tutorial. Just a quick update, as of September 2020, my results using exactly the same code are slightly different. Not a big deal, just gbm provides more accurate results than c5.0 in the Boosting Algorithms section now. I tested both in R 4.0 and 3.6.3, I got the same results.

  64. Avatar
    الکسیس چت November 29, 2020 at 6:56 pm #

    We know things related to the above, or we know things because we have inherited some of the knowledge with our gene

    • Avatar
      Jason Brownlee November 30, 2020 at 6:36 am #

      Sorry, I don’t understand. Can you please elaborate?

  65. Avatar
    ayse December 10, 2020 at 6:26 am #

    hello Jason,
    Thanks for the explanation! I was wondering if I can use these techniques for non binary numerical variables as the response variable. Or are there other types of algorithms to use for that? Ex: to predict the imdb score of a movie, using machine learning techniques.

    • Avatar
      Jason Brownlee December 10, 2020 at 6:32 am #

      You’re welcome.

      Yes, that is called regression. Many of the algorithms can be used for regression also.

  66. Avatar
    A.V March 15, 2021 at 9:57 pm #

    Hi! Very helpfull article!

    One question. If i have qualitative predictors like

    Q11={less than one time per week, 1-2 times per week, 3 or more times per week}

    and i want to predict

    Q15={very bad ,bad,good,very good}

    do i have to make them numeric ? or if turn them in to factors and feed them to the model it will be ok ?

    • Avatar
      Jason Brownlee March 16, 2021 at 4:47 am #

      Some algorithms can work with them as factors directly, some may require you to encode them first.

  67. Avatar
    Viji March 17, 2021 at 1:00 pm #

    Indeed it’s good article.

    Just a quick question.

    I have only two variables X and Y and it’s a regression problem. I used power function to predict Y and getting 47% accuracy.

    If I try boosting or bagging using linear regression, can I expect significant improvement in R-square.

    How to get the prediction function and formula for the same.

    • Avatar
      Jason Brownlee March 18, 2021 at 5:15 am #

      Thanks!

      Accuracy cannot be measured for regression, instead we must use error.

      Some algorithms will give you a formula (such as linear algorithms), most do not. You may need to check the documentation for the algorithm that you are using.

  68. Avatar
    Fan August 4, 2021 at 1:11 am #

    Hi Jason,

    I got this error “Error: wrong model type for regression”.
    How do I correct it?
    In order words, I apply this code for a regression problem, what should I change in the code?

    I tried to change: metric <- "RMSE", but it is not work.

    • Avatar
      Jason Brownlee August 4, 2021 at 5:15 am #

      Looks like you are trying to use a classification algorithm for a regression problem.

Leave a Reply