Last Updated on

It is important to both present the expected skill of a machine learning model a well as confidence intervals for that model skill.

Confidence intervals provide a range of model skills and a likelihood that the model skill will fall between the ranges when making predictions on new data. For example, a 95% likelihood of classification accuracy between 70% and 75%.

A robust way to calculate confidence intervals for machine learning algorithms is to use the bootstrap. This is a general technique for estimating statistics that can be used to calculate empirical confidence intervals, regardless of the distribution of skill scores (e.g. non-Gaussian)

In this post, you will discover how to use the bootstrap to calculate confidence intervals for the performance of your machine learning algorithms.

After reading this post, you will know:

- How to estimate confidence intervals of a statistic using the bootstrap.
- How to apply this method to evaluate machine learning algorithms.
- How to implement the bootstrap method for estimating confidence intervals in Python.

Discover statistical hypothesis testing, resampling methods, estimation statistics and nonparametric methods in my new book, with 29 step-by-step tutorials and full source code.

Let’s get started.

**Update June/2017**: Fixed a bug where the wrong values were provided to numpy.percentile(). Thanks Elie Kawerk.**Update March/2018**: Updated link to dataset file.

## Bootstrap Confidence Intervals

Calculating confidence intervals with the bootstrap involves two steps:

- Calculate a Population of Statistics
- Calculate Confidence Intervals

### Need help with Statistics for Machine Learning?

Take my free 7-day email crash course now (with sample code).

Click to sign-up and also get a free PDF Ebook version of the course.

### 1. Calculate a Population of Statistics

The first step is to use the bootstrap procedure to resample the original data a number of times and calculate the statistic of interest.

The dataset is sampled with replacement. This means that each time an item is selected from the original dataset, it is not removed, allowing that item to possibly be selected again for the sample.

The statistic is calculated on the sample and is stored so that we build up a population of the statistic of interest.

The number of bootstrap repeats defines the variance of the estimate, and more is better, often hundreds or thousands.

We can demonstrate this step with the following pseudocode.

1 2 3 4 5 |
statistics = [] for i in bootstraps: sample = select_sample_with_replacement(data) stat = calculate_statistic(sample) statistics.append(stat) |

### 2. Calculate Confidence Interval

Now that we have a population of the statistics of interest, we can calculate the confidence intervals.

This is done by first ordering the statistics, then selecting values at the chosen percentile for the confidence interval. The chosen percentile in this case is called alpha.

For example, if we were interested in a confidence interval of 95%, then alpha would be 0.95 and we would select the value at the 2.5% percentile as the lower bound and the 97.5% percentile as the upper bound on the statistic of interest.

For example, if we calculated 1,000 statistics from 1,000 bootstrap samples, then the lower bound would be the 25th value and the upper bound would be the 975th value, assuming the list of statistics was ordered.

In this, we are calculating a non-parametric confidence interval that does not make any assumption about the functional form of the distribution of the statistic. This confidence interval is often called the empirical confidence interval.

We can demonstrate this with pseudocode below.

1 2 3 |
ordered = sort(statistics) lower = percentile(ordered, (1-alpha)/2) upper = percentile(ordered, alpha+((1-alpha)/2)) |

## Bootstrap Model Performance

The bootstrap can be used to evaluate the performance of machine learning algorithms.

The size of the sample taken each iteration may be limited to 60% or 80% of the available data. This will mean that there will be some samples that are not included in the sample. These are called out of bag (OOB) samples.

A model can then be trained on the data sample each bootstrap iteration and evaluated on the out of bag samples to give a performance statistic, which can be collected and from which confidence intervals may be calculated.

We can demonstrate this process with the following pseudocode.

1 2 3 4 5 6 |
statistics = [] for i in bootstraps: train, test = select_sample_with_replacement(data, size) model = train_model(train) stat = evaluate_model(test) statistics.append(stat) |

## Calculate Classification Accuracy Confidence Interval

This section demonstrates how to use the bootstrap to calculate an empirical confidence interval for a machine learning algorithm on a real-world dataset using the Python machine learning library scikit-learn.

This section assumes you have Pandas, NumPy, and Matplotlib installed. If you need help setting up your environment, see the tutorial:

First, download the Pima Indians dataset and place it in your current working directory with the filename “pima*–*indians*-diabetes.data.csv*” (update: download here).

We will load the dataset using Pandas.

1 2 3 |
# load dataset data = read_csv('pima-indians-diabetes.data.csv', header=None) values = data.values |

Next, we will configure the bootstrap. We will use 1,000 bootstrap iterations and select a sample that is 50% the size of the dataset.

1 2 3 |
# configure bootstrap n_iterations = 1000 n_size = int(len(data) * 0.50) |

Next, we will iterate over the bootstrap.

The sample will be selected with replacement using the resample() function from sklearn. Any rows that were not included in the sample are retrieved and used as the test dataset. Next, a decision tree classifier is fit on the sample and evaluated on the test set, a classification score calculated, and added to a list of scores collected across all the bootstraps.

1 2 3 4 5 6 7 8 9 10 11 12 |
# run bootstrap stats = list() for i in range(n_iterations): # prepare train and test sets train = resample(values, n_samples=n_size) test = numpy.array([x for x in values if x.tolist() not in train.tolist()]) # fit model model = DecisionTreeClassifier() model.fit(train[:,:-1], train[:,-1]) # evaluate model predictions = model.predict(test[:,:-1]) score = accuracy_score(test[:,-1], predictions) |

Once the scores are collected, a histogram is created to give an idea of the distribution of scores. We would generally expect this distribution to be Gaussian, perhaps with a skew with a symmetrical variance around the mean.

Finally, we can calculate the empirical confidence intervals using the percentile() NumPy function. A 95% confidence interval is used, so the values at the 2.5 and 97.5 percentiles are selected.

Putting this all together, the complete example is listed below.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 |
import numpy from pandas import read_csv from sklearn.utils import resample from sklearn.tree import DecisionTreeClassifier from sklearn.metrics import accuracy_score from matplotlib import pyplot # load dataset data = read_csv('pima-indians-diabetes.data.csv', header=None) values = data.values # configure bootstrap n_iterations = 1000 n_size = int(len(data) * 0.50) # run bootstrap stats = list() for i in range(n_iterations): # prepare train and test sets train = resample(values, n_samples=n_size) test = numpy.array([x for x in values if x.tolist() not in train.tolist()]) # fit model model = DecisionTreeClassifier() model.fit(train[:,:-1], train[:,-1]) # evaluate model predictions = model.predict(test[:,:-1]) score = accuracy_score(test[:,-1], predictions) print(score) stats.append(score) # plot scores pyplot.hist(stats) pyplot.show() # confidence intervals alpha = 0.95 p = ((1.0-alpha)/2.0) * 100 lower = max(0.0, numpy.percentile(stats, p)) p = (alpha+((1.0-alpha)/2.0)) * 100 upper = min(1.0, numpy.percentile(stats, p)) print('%.1f confidence interval %.1f%% and %.1f%%' % (alpha*100, lower*100, upper*100)) |

Running the example prints the classification accuracy each bootstrap iteration.

A histogram of the 1,000 accuracy scores is created showing a Gaussian-like distribution.

Finally, the confidence intervals are reported, showing that there is a 95% likelihood that the confidence interval 64.4% and 73.0% covers the true skill of the model.

1 2 3 4 5 6 7 |
... 0.646288209607 0.682203389831 0.668085106383 0.673728813559 0.686021505376 95.0 confidence interval 64.4% and 73.0% |

This same method can be used to calculate confidence intervals of any other errors scores, such as root mean squared error for regression algorithms.

## Further Reading

This section provides additional resources on the bootstrap and bootstrap confidence intervals.

- An Introduction to the Bootstrap, 1996
- Bootstrap Confidence Intervals, Statistical Science, 1996
- Section 5.2.3, Bootstrap Confidence Intervals, Empirical Methods for Artificial Intelligence
- Bootstrapping on Wikipedia
- Section 4.4 Resampling Techniques, Applied Predictive Modeling

## Summary

In this post, you discovered how to use the bootstrap to calculate confidence intervals for machine learning algorithms.

Specifically, you learned:

- How to calculate the bootstrap estimate of confidence intervals of a statistic from a dataset.
- How to apply the bootstrap to evaluate machine learning algorithms.
- How to calculate bootstrap confidence intervals for machine learning algorithms in Python.

Do you have any questions about confidence intervals?

Ask your questions in the comments below.

Thank you Jason

I’m glad you found the post useful.

Where can i get your book “Mastering machine learning with python”

Here:

https://machinelearningmastery.com/machine-learning-with-python/

Sir ,how we can apply bootstrap resampling for software effort interval prediction and what is the procedure and initial steps that are to be followed.

Collect your data, load it in Python and apply the procedure to estimate the desired quantity.

Which part are you struggling with exactly?

Thank you Jason,

Are you considering to include the posts on confidence intervals in a previous (or new) book?

Regards

Elie

Hi Elie,

Not especially. I have a soft idea of “statistical methods for machine learning” or something to that effect that could cover topics like sampling theory and confidence intervals. Posts like this are a test to see if there is interest.

Why do you ask? Is this a topic for which you’re looking for more help?

Yes,

I actually find posts like these very useful for reporting statistically meaningful results for machine learning. Here’s a nice free book link on the subject: https://www.otexts.org/book/sfml

Regards

Jason,

I have another question. How do we report confidence intervals on an evaluation done on a hold-out set? We cannot apply the bootstrap with training here since this would contaminate our results.

Regards

Generally, you can repeat the holdout process many times with different random samples and use the outcomes as your population of results.

I agree, it’s a great book!

Wouldn’t it be suitable to apply the bootstrap by sampling from one hold out set for a certain number of iterations without any training involved?

As in only resampling the set used to evaluate the skill of the model?

No, I don’t think this would be valid.

Jason,

Something seems not to make sense in the confidence interval you obtain. The one you calculate is [61.5, 63.9]. However the mean (that I obtain) is about 69% (similar to the one you get from the graph) and visually one can inspect and estimate the 95% CI to be [64, 74]. I think there is something wrong.

Please correct me if I am wrong.

Regards

Elie

You are correct Elie, thank you so much.

The values provided to numpy.percentile() must be in [0,100] and I was providing them in [0,1].

I have fixed the example and made it clearer how the percentile values are calculated (so they can be debugged by the skeptical developers we should be).

I really appreciate you finding this and the comment. It makes the example better for everyone!

np.quantile(q) = np.percentile(p) for q=p/100

I love the piece of cake notes about ML.

Thanks Dawit.

Hi Jason, thanks for this post. Is the concept of fitting your classifier using a training set and calculating it’s accuracy on unseen data (a holdout set) now outdated? It seems to me that CV is testing how good your algorithm (with optimized parameters) is at building a model, in that the model is re-fitted in each fold prior to testing it on the held-out fold. It doesn’t seem to predict how a trained model will work once it is put into production and starts having to classify completely new instances. For that, I can’t see any option other than training on the full training set and testing against the hold out set – am I missing something ?

Cross Validation (CV) is doing the same thing, just repeated 10 times, for 10-fold CV.

CV and train/test splits are both resampling methods intended to estimate the skill of the model on unseen data. Perhaps this post will clear things up for you:

https://machinelearningmastery.com/train-final-machine-learning-model/

On many problems where we have millions of examples (e.g. deep learning) we often do not have the resources for CV.

Thanks for the link – that clarifies a lot of things ! In summary, is it correct to say where you have the processing power and time to perform it, k-fold validation is always going to be the superior option as it provides means, standard deviations and confidence intervals ?

CV will give a less biased estimate of model skill, in general. This is the key objective of the method.

I often use a bootstrap to then present the final confidence interval for the chosen configuration.

Hi, thanks for this article. I’m after calculating confidence intervals for sensitivity and specificity. I’ve used your code, but changed the prep step to use the ‘train_test_split’ (with a random seed), to create samples of the data. Then, once per iteration, I’ve fitted a model and made predictions, then created a confusion matrix, and then worked out the sensitivity and specificity. Do you think that sounds reasonable? The results look sensible.

Thanks.

I think the bootstrap approach would work for other accuracy related measures.

Nice post (:

Might be useful to mention the bootsrapped package in case people are feeling lazy 😉

https://pypi.python.org/pypi/bootstrapped/0.0.1

Thanks for the link Ian.

Great post Jason. I was wondering if you think it might be feasible to use bootstrap method for neural nets to estimate confidence level?

Absolutely!

How can I estimate a 95% likelihood of classification accuracy for a particular unseen data point?

Suppose, I have three new data points x1, x2, x3. How can I estimate that my method has a 95% likelihood of classification accuracy between 80% and 85% for predicting, say, x2, but between 60% and 65% for predicting x1, and between 90% and 95% for predicting x3?

It sounds like you might be asking about a prediction interval instead of a confidence interval.

I cover this in my book on stats and I have a post on the topic scheduled for soon.

Until then see this article:

https://en.wikipedia.org/wiki/Prediction_interval

Thanks for the link!

Dear Dr Jason,

While I understand the general concept presented above, I am lost on one fine detail on

lines 21 and 23. It is about selection array [:,:-1] and [:,-1].

It is about

and

In other words, when there is an array, what is the difference between the selection methods.

and

Thank you,

Anthony of Sydney

Yes, we are selecting columns.

You can learn more about slicing arrays here:

https://machinelearningmastery.com/index-slice-reshape-numpy-arrays-machine-learning-python/

Dear Dr Jason,

Thank you for that reference. I admit I was having difficulty then. Today I understand.

Regards

Anthony of Sydney

Glad to hear it.

Hello,

can you please point to a reference for the expressions in line 32 and 34 of the complete example. frame?

p = ((1.0-alpha)/2.0) * 100

p = (alpha+((1.0-alpha)/2.0)) * 100

Thanks

Perhaps see the references at the end of the post.

Please ignore my previous comment. I just observed that alpha was set to .95.

They now make sense.

No problem.

Hi! Thanks very much for your post!

However I don’t understand your confidence intervals. If you wanted a 95% confidence interval shouldn’t you be setting your alpha to 0.05?

Cheers,

Santiago

Perhaps re-read the post. Remember that we want both sides, e.g. 2.5% gap on both sides of the mean.

Perhaps this will help:

https://machinelearningmastery.com/confidence-intervals-for-machine-learning/

it’s very intuitive. Thanks for writing it

Thanks.

Jason,

Thank you for writing this article!!

During my PhD in Physics, we had to fit unwieldy (highly) nonlinear functions to real data and it was essential for us (as “budding” scientists) to at least attempt to report on both the standard and systematic errors of our results.

While learning about and using propagation of error methods throughout school, it surprised me to learn so late in my studies that the bootstrap method was a (kosher) go-to method for (a simulated) estimate of standard error of parameters from such complicated functions.

When nature’s constituent parts at various scales appear to often exhibit non-linear interaction dynamics, one would think our engineer and science teaching forefathers (e.g. for engineers, physicists, chemists, biologists, etc.) would have emphasized merit and further study of this method, particularly in an era now heavily (re)focused more on phenomenological studies and less so on pure analytical theories.

To end on a high note, I’m still thankful for the advances in DS and ML for making such phenomenological studies, and basic predictive analytics accessible to the masses, and namely a laymen like me.

Thank you again for another great article!!

Thanks for sharing, I could not agree with you more.

In fact, more simulation/monte carlo methods should be taught in general.

Hi Jason,

Great article on this. It’s helped me immensely. I’ve been trying to apply this to getting CIs for AUC and after visualizing the sampling distribution it looks like it is a truncated normal distribution. Have you worked with these type of distributions when calculating CIs? Any advice would be much appreciated.

I have not sorry.

Perhaps try repeating the experiment 100+ times and then review the distribution, e.g. histogram and perhaps a statistical test.

I find this article very useful. I have been working on a data set where I am using RandomForestRegressor to predict 2 outputs (multi output ) and there are 17 predictors. I made the model and prediction but I want prediction intervals from them, I read papers jackknife and infinitesimal jackknife and quantile regression, and found out about foresci package which makes interval based on jackknife paper, but none of them explained the multi output case like how to find the interval for multivariate case. It will be helpful if you could give me some advice.

Thanks for the suggestion, I may cover the topic in the future.

Hi Jason, thank you very much, I love this article, is super helpful and clear.

I have one question: I’m currently interested in just the confidence intervals, I’ve noticed that varying the size of the sample gives me different intervals. I came across the Seaborn package that has a bootstrap function that takes samples that are 100% the size of the dataset [1].

Is it Ok to sample with a size equal to 100% of the dataset? Why do the intervals vary?

Thanks for any guide you could provide me.

[1] https://github.com/mwaskom/seaborn/blob/b9551aff1e2b020542a5fb610fec468b69b87c6e/seaborn/algorithms.py#L86

Yes, I believe using bootstrap samples that are 100% is a standard practice.

Hello!

Really helpful article, thanks a lot!

I want to ask: if we made not bootstrap, but some shufflesplit CV. And have a list of some metric values after it. Can we test the normality of this values (with shapiro test for example) and if we don’t reject null hypothesis, build a confidence interval for mean over this values with unknown standard deviation (x.mean +- t*s/sqrt(n))? I was trying to do something like that, but I don’t really confident in correctness of this computations.

Thanks for any advice you could give me!

Might be a bit messy, e.g. k mean values for m repeats with different splits.

Not sure about the implications sorry. Perhaps check in with a statistician?

Very nice article! Can you please suggest how can we use Bootstrapping with logistic regression? I want to change the C value in every iteration. How this can be done?

Thank you!

Sounds straightforward, what problem are you having exactly?

I am not sure how can I send series of C for logistic regression and which value will give me best solution. Can you provide me the syntax of doing this? Currently I am using LogReg.fit to do the logistic regression but not sure how can I allocate different C. Also, if C can be optimized?

Thank you!

Perhaps try a range of values for C on a log scale?

Hi, is there are reason why you don’t use replacement? And what is the ideal percentage of your original dataset that we should keep on each bootstrap? Is it just random?

Thanks

The bootstrap does use replacement.

The bootstrap size should match the original dataset size, that is a good heuristic.

Thanks

And should we care about stratification when resampling?

Yes, probably.

Thanks Jason! Great article, as always.

The mlxtend Python package has a nice implementation of the bootstrap, as well as the .632 and .632+ methods (bootstrap_point632.py). My only complaint is that it only takes one scoring function at a time as of now, but I’m working on a PR to make it more efficient for multiple scores. 🙂

Thanks for sharing.

Hello Dr. Jason Bronwlee,

It is very nice stuff. I want to calculate the Confidence intervals for XGBoost of H2o. I do not understand a few steps. I am applying data to XGBoost by splitting into train, valid and test assuming time series case. How can we put it into iterations? I would appreciate your guidance.

import numpy

from pandas import read_csv

from sklearn.utils import resample

from sklearn.metrics import accuracy_score

from matplotlib import pyplot

from xgboost import XGBRegressor

# load dataset

#data = read_csv(‘pima-indians-diabetes.data.csv’, header=None)

#values = data.values

# configure bootstrap

n_iterations = 100

n_size = int(len(hf_train) * 0.50)

# run bootstrap

stats = list()

for i in range(n_iterations):

# prepare train and test sets

#train = resample(values, n_samples=n_size)

#test = numpy.array([x for x in values if x.tolist() not in train.tolist()])

# fit model

import h2o

from h2o.estimators.xgboost import H2OXGBoostEstimator

h2o.init()

param = {

“ntrees” : 10000

, “nfolds” : 0

,”stopping_rounds” : 40

, “max_depth” : 25

,”stopping_tolerance” : 0.005

, “learn_rate” : 0.05

, “eta” : 0.3

, “sample_rate” : 0.8

, “subsample” : 1

, “col_sample_rate” : 1

, “col_sample_rate_per_tree” : 1

, “colsample_bylevel” : 1

, “min_rows” : 0.06

,”min_child_weight”: 1

, “sample_type” : “uniform”

, “normalize_type” : “tree”

,”nthread” : -1

, “max_bins” : 256

, “max_leaves” : 0

, “seed”: 1601

, “score_tree_interval”: 0

,”reg_lambda” : 0.1

,”reg_alpha” : 1/1

,”grow_policy” : “depthwise”

, “booster” : “gbtree”

}

from h2o.estimators import H2OXGBoostEstimator

model = H2OXGBoostEstimator(**param)

model.train(x = X, y = y, training_frame = hf_train, validation_frame = hf_valid)

# evaluate model

predictions = model.predict(hf_test)

score = model.model_performance(test_data= hf_test)

print(score)

stats.append(score)

# plot scores

pyplot.hist(stats)

pyplot.show()

# confidence intervals

alpha = 0.95

p = ((1.0-alpha)/2.0) * 100

lower = max(0.0, numpy.percentile(stats, p))

p = (alpha+((1.0-alpha)/2.0)) * 100

upper = min(1.0, numpy.percentile(stats, p))

print(‘%.1f confidence interval %.1f%% and %.1f%%’ % (alpha*100, lower*100, upper*100))

This is a common question that I answer here:

https://machinelearningmastery.com/faq/single-faq/can-you-read-review-or-debug-my-code

Thanks: That,s make sense.

You’re welcome.

Hello Dr. Jason,

I followed the step given the link above by “cutting the problem” into a small level. I have now been able to apply to the XGBoost H2o. I need still your guidance for a few things.

first: I want to divide the data into train,valid and test. The models are trained using the training data and scored using the validation data and obtain a final score of the model using test data. How can we change it.?

train = resample(values, n_samples=n_size)

test = numpy.array([x for x in values if x.tolist() not in train.tolist()])

Second: I do not understand why lower and upper confidence interval are same i.e.(4.4% and 4.4%). I am doing it for regression .For score I am using:

score = model.rmse()

You can use this function to split data into subsets:

https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html

Thank a lot: I have now posted my second question at Stackoverflow with reproducible codes.

https://stackoverflow.com/questions/60956655/bootstrap-confidence-intervals-for-xgboost-h2o-regression-python

Well done.

Hello Dr Jason: I am still looking for the answer my second question (in comments 31March).

In your example. When we print(score), we will the score equal to the number of n_iterations.This is how we can plot the histogram.

score = accuracy_score(test[:,-1], predictions)

print(score)

IN my case: XGBoost h2o. (whole codes can be seen in Stackoverflow). I get the single value for the score when we do n_iterations. This is the problem. How can we solve it?

predictions = model.predict(test_hf1)

score = model.rmse()

print(score)

Sorry, I don’t know about the h20 api, not sure I can give you advice. Perhaps contact their support?

Thanks a lot for the always prompt response. I have contacted with H2O. I am also working on it.

I’m happy to hear that, good luck!

I checked it with using LightGBM library it seem to have the same problem for the regression as I am getting with XGBoost H2o. It seems to me the codes does not work for the regression. It works well with the XGBoost classifier. Also it means that the problem is not pertain to specific API such H2o rather to applying to regression or classification. It works fine with classification.

I shall be very thankful to you if you can check it for regression.

Yes, the code example calculates accuracy, you must change this for regression to an error score like MSE.

Probably I could not explain my question. I did change to MSE/RMSE but the problem is not solved. It works perfectly only for classification case.

score = np.sqrt(mean_squared_error(yt, y_pred))

print(score)

stats.append(score)

# plot scores

pyplot.hist(stats)

pyplot.show()

# confidence intervals

alpha = 0.95

p = ((1.0-alpha)/2.0) * 100

lower = max(0.0, numpy.percentile(stats, p))

p = (alpha+((1.0-alpha)/2.0)) * 100

upper = min(1.0, numpy.percentile(stats, p))

print(‘%.1f confidence interval %.1f%% and %.1f%%’ % (alpha*100, lower*100, upper*100))