It is important to compare the performance of multiple different machine learning algorithms consistently.

In this post you will discover how you can create a test harness to compare multiple different machine learning algorithms in Python with scikit-learn.

You can use this test harness as a template on your own machine learning problems and add more and different algorithms to compare.

Let’s get started.

**Update March/2018**: Added alternate link to download the dataset as the original appears to have been taken down.

## Choose The Best Machine Learning Model

How do you choose the best model for your problem?

When you work on a machine learning project, you often end up with multiple good models to choose from. Each model will have different performance characteristics.

Using resampling methods like cross validation, you can get an estimate for how accurate each model may be on unseen data. You need to be able to use these estimates to choose one or two best models from the suite of models that you have created.

### Compare Machine Learning Models Carefully

When you have a new dataset, it is a good idea to visualize the data using different techniques in order to look at the data from different perspectives.

The same idea applies to model selection. You should use a number of different ways of looking at the estimated accuracy of your machine learning algorithms in order to choose the one or two to finalize.

A way to do this is to use different visualization methods to show the average accuracy, variance and other properties of the distribution of model accuracies.

In the next section you will discover exactly how you can do that in Python with scikit-learn.

### Need help with Machine Learning in Python?

Take my free 2-week email course and discover data prep, algorithms and more (with code).

Click to sign-up now and also get a free PDF Ebook version of the course.

## Compare Machine Learning Algorithms Consistently

The key to a fair comparison of machine learning algorithms is ensuring that each algorithm is evaluated in the same way on the same data.

You can achieve this by forcing each algorithm to be evaluated on a consistent test harness.

In the example below 6 different algorithms are compared:

- Logistic Regression
- Linear Discriminant Analysis
- K-Nearest Neighbors
- Classification and Regression Trees
- Naive Bayes
- Support Vector Machines

The problem is a standard binary classification dataset from the UCI machine learning repository called the Pima Indians onset of diabetes problem (update: download from here). The problem has two classes and eight numeric input variables of varying scales.

The 10-fold cross validation procedure is used to evaluate each algorithm, importantly configured with the same random seed to ensure that the same splits to the training data are performed and that each algorithms is evaluated in precisely the same way.

Each algorithm is given a short name, useful for summarizing results afterward.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 |
# Compare Algorithms import pandas import matplotlib.pyplot as plt from sklearn import model_selection from sklearn.linear_model import LogisticRegression from sklearn.tree import DecisionTreeClassifier from sklearn.neighbors import KNeighborsClassifier from sklearn.discriminant_analysis import LinearDiscriminantAnalysis from sklearn.naive_bayes import GaussianNB from sklearn.svm import SVC # load dataset url = "https://raw.githubusercontent.com/jbrownlee/Datasets/master/pima-indians-diabetes.data.csv" names = ['preg', 'plas', 'pres', 'skin', 'test', 'mass', 'pedi', 'age', 'class'] dataframe = pandas.read_csv(url, names=names) array = dataframe.values X = array[:,0:8] Y = array[:,8] # prepare configuration for cross validation test harness seed = 7 # prepare models models = [] models.append(('LR', LogisticRegression())) models.append(('LDA', LinearDiscriminantAnalysis())) models.append(('KNN', KNeighborsClassifier())) models.append(('CART', DecisionTreeClassifier())) models.append(('NB', GaussianNB())) models.append(('SVM', SVC())) # evaluate each model in turn results = [] names = [] scoring = 'accuracy' for name, model in models: kfold = model_selection.KFold(n_splits=10, random_state=seed) cv_results = model_selection.cross_val_score(model, X, Y, cv=kfold, scoring=scoring) results.append(cv_results) names.append(name) msg = "%s: %f (%f)" % (name, cv_results.mean(), cv_results.std()) print(msg) # boxplot algorithm comparison fig = plt.figure() fig.suptitle('Algorithm Comparison') ax = fig.add_subplot(111) plt.boxplot(results) ax.set_xticklabels(names) plt.show() |

Running the example provides a list of each algorithm short name, the mean accuracy and the standard deviation accuracy.

1 2 3 4 5 6 |
LR: 0.769515 (0.048411) LDA: 0.773462 (0.051592) KNN: 0.726555 (0.061821) CART: 0.695232 (0.062517) NB: 0.755178 (0.042766) SVM: 0.651025 (0.072141) |

The example also provides a box and whisker plot showing the spread of the accuracy scores across each cross validation fold for each algorithm.

From these results, it would suggest that both logistic regression and linear discriminate analysis are perhaps worthy of further study on this problem.

## Summary

In this post you discovered how to evaluate multiple different machine learning algorithms on a dataset in Python with scikit-learn.

You learned how to both use the same test harness to evaluate the algorithms and how to summarize the results both numerically and using a box and whisker plot.

You can use this recipe as a template for evaluating multiple algorithms on your own problems.

Do you have any questions about evaluating machine learning algorithms in Python or about this post? Ask your questions in the comments below and I will do my best to answer them.

Great article! Just a quick question, what do you think is the best method? First optimise hyper-parameters and then compare the algorithms or vice versa. Thanks.

I recommend first spot checking algorithms and comparing them, followed by tuning.

Thank you for your recommendation…

You should not just rely on this. Accuracy is only one portion of a model’s accuracy. Depending on the investigation desired, you should look at Precision and Recall because accuracy may be only a tiny portion. Also, why have you not taken an approach with ANOVA, or the Wilcoxon Test, major tests within the realm of data science and widely accepted? Additionally, 5×2 cross-validation should be done, not 10-fold (this is widely accepted). Last, what I find completely missing in this is that you have not discussed how to actually arrive at a statistically-significant decision. This is not a good representation.

Eric, nobody cares about your phd, whatever it is you did it in. Also, stop calling it ANOVA when all you’re doing is a regression, it doesn’t make you any smarter. And lastly, nobody cares about your phd and your academic research, this is a machine learning article for Data Scientists.

There’s always room for improvement and we have to start/stop somewhere in an introductory piece.

Hi Dr Brownlee,

Big fan of your tutorials. I have a question regarding the compare first then tune approach. When we plot them on a box plot and select the best, this is all based on the default model setting right? But once we have tune the different settings in a given model, would the predictive performance be different?

So the not so good models might even outperform the best model given in the first glance boxplot, if we have trained them more properly. So in this sense, wouldn’t it better to train every module separately, and then say, connect all of them and plot their ROC to see which performs best?

Sure, if you have the resources.

Accuracy is easily readable but, in my opinion, it should be replaced by AUC: AUC is “consistent” and “more discriminating” than accuracy (Ling et al. 2003).

AUC can be examined on an ROC or Precision vs Recall curve. What should happen is weights based on misclassification, in a confusion matrix. In this case, you can tune a model to avoid certain misclassifications, as some may be more valuable to avoid. If you have zero care about which misclassification occurs, ROC is a decent metric for how you should tune parameters. ROC should be examined for hyperparameter decisions.

So go write your own article

In the code, “seed = 7” is hard coded. Shouldn’t we have a different seed for each fold?

To answer my own question, it appears that each model is trained and tested for all folds before moving on to the next model. The seed applies to the initial state so for the above, the 10 folds will all be different from one another, but the same data split for each of the 10 folds will be presented to each algorithm.

Yes Tom, the seed ensures we have the same sequence of random numbers. The random numbers ensure we have a random split of the data into the k folds.

Thank you for sharing.

I had to tweak the code a little to make it work with scikit-learn 0.18.

The cross_validation module is deprecated. It’s replaced by model_selection.

The KFold parameters have changed too:

0.17: cross_validation.KFold(n, n_folds=3, shuffle=False, random_state=None)

0.18: model_selection.KFold(n_splits=3, shuffle=False, random_state=None)

I have a question: is it ok to train the classifier before adding it to the list? Like:

lr = LogisticRegression()

lr.fit(X_train, y_train)

models.append((‘LR’,lr))

Thanks Guillaume, I will look at updating the example. I have recently updated all of my books to support the new sklearn.

No, the structure of the example fits and evaluates each model in turn. Your example essentially unrolls the for loop.

What a great article! I learned so much from your writing 🙂

I also read your other article comparing different algorithms in R, and I noticed that you used a lot more techniques in that article:

• Table Summary

• Box and Whisker Plots

• Density Plots

• Dot Plots

• Parallel Plots

• Scatterplot Matrix

• Pairwise xyPlots

• Statistical Significance Tests

I was wondering why you did not provide the same techniques in this Python article? Is it because these functions are more readily available in R?

Thanks so much!

Great question.

These capabilities are available in Python, but are spread through the scipy and statsmodels libs rather than directly available in sklearn.

R is a more technical platform for more technical types, I tend to go into more detail in those examples.

Is this something you would like to see more of Angela?

Hi Jason. Thank you for these great articles. I also read this article of yours (https://goo.gl/v71GPT). What I wonder is the proper validation method. Should we conduct k-fold or repeated n*k-fold cross validation? I recently read a journal article where researchers compare around 50 models under 5*2-folds setting, suggesting it is more robust. How should we proceed while comparing models?

Hi Suleyman,

Using k-fold cross validation is a gold standard. The specific configuration is problem specific, but common configurations of 3,5, 10 do well on many datasets.

On very large datasets, a train-test split may be sufficient. For complex or small datasets, if you have the resources, repeated k-fold cross validation is preferred. Often, we would like to use repeated k-fold cross validation, but the computational expense is too high.

There is no “best”, just lots of options to tune for your given problem.

How do you choose?

Balance your constraints (amount of data, resources, time, ..) against your requirements (robustness of result).

Thank you Jason. That is what I have been doing.

Do stick with 5×2. Nested Cross Validation is widely accepted, and way ahead of regular k-fold.

Hi Jason,

Thanks a lot for this good article.

Could you please give some interpretations of the standard deviation values?

Especially regarding overfitting.

I thought that in case we have a small standard deviation of the cv results, we will have more overfitting, but I am not sure about that.

Thanks

Hi Othmane, great question.

So standard deviation summarizes the spread of the distribution, assuming it is Gaussian.

A tight spread may suggest overfitting or it may not, but we can only be sure by evaluating the model on a hold out dataset.

One use of the stdev is to specify a confidence interval for the result. For example, the performance of the model is x% on unseen data, with the performance in the range of 2 standard deviations of that score (95th percentile).

This might help:

https://en.wikipedia.org/wiki/68%E2%80%9395%E2%80%9399.7_rule

Thanks for the prompt, this topic of interpreting results is not discussed enough. I plan to write more about it.

Following Othmane’s question, shouldn’t we work with the standard error of the mean instead of the standard deviation? Basically, divide the standard deviations by sqrt(10). This is because “The standard error (SE) of a statistic (most commonly the mean) is the standard deviation of its sampling distribution”. https://en.wikipedia.org/wiki/Standard_error

You can, I have an example of calculating standard error here:

http://machinelearningmastery.com/estimate-number-experiment-repeats-stochastic-machine-learning-algorithms/

I have examples of calculating confidence intervals here:

http://machinelearningmastery.com/report-classifier-performance-confidence-intervals/

and here:

http://machinelearningmastery.com/calculate-bootstrap-confidence-intervals-machine-learning-results-python/

Hi, from the boxplot, we get LR and LDA to have higher accuracy, so we select them as our models.

So now, can I apply train_test_split to check the RMSE and the accuracy for the testing data using both these models. Whichever gives the best result, I will make that my final model?

Hi Dhrubajit,

There are many ways to choose a final model. Often we prefer a model with better average performance rather than better absolute performance, this is because of the natural variance in the estimation of performance of the models on unseen data.

Once you choose a final model, train it on all available data and you can start to use it to make predictions.

Great article! Could you please explain me why this program doesn’t work when Y is float?

Hi Peter,

Classification problems assume the outcome is a label.

great post! thanks for sharing

Thanks Edmond.

Hi Jason,

I have started learning and implementing Machine learning algorithms.

One question – the above blog will tell us which Machine learning algorithm to go with. However, Should we ever check that if we are using Regression, how well the regression fits the data by checking

Autocorrelation, Multicollinearity and normality.

What I have learnt from reading blogs and articles that we all calculate score by using cross validation methodology, and then find out which would fit best. have not seen anyone following traditional ways such as checking Autocorrelation, Multicollinearity and normality. I might be wrong. Please throw some light on the same.

THanks Nitesh

Yes, on time series, an understanding of the autocorrelation is practically required.

When using a linear method, an idea of multicollinearity can be helpful.

I would suggest this type of analysis before investigating models to get a better idea of the structure of your problem.

Hi Jason! First of all thanks for all your blog posts, they are really helping me to better understand how to work with datasets and machine learning algorithms.

I’ve a question related to the scoring method. Before discovering the method you are using here, I was using the .score() method in this way (assume I already have splitted the dataset 80/20 and tranformed the data):

from sklearn.svm import LinearSVC

lin_svc = LinearSVC()

lin_svc.fit(train_set_scaled, train_set_labels)

lin_svc.score(test_set_scaled, test_set_labels)

getting a score that was similar but different to the one I obtain with the method explained in this post. What’s the difference between using score() and using cross_val_score() ?

Thanks!

Good question.

The cross_val_score() function uses the more robust cross validation method to evaluate model skill:

http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_val_score.html#sklearn.model_selection.cross_val_score

The score() function on a model evaluate the skill of the model on the provided dataset.

http://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVC.html#sklearn.svm.LinearSVC.score

Does that help?

While comparing algorithms by using same code which is mentioned above i got one error ‘could not covert string to float’.

Can you please tell me how to tackle it.

Confirm that you copied the code exactly and that you are using the same data file. Also confirm that all of your Python libraries are up to date.

Thanks Jason.

Actually I am using different dataset. My dataset is about stock related. So what can I do for it while comparing the algorithms.

Hi,

Thank you so much for this tutorial. It really helps one using Machine Learning in sklearn.

One questio: I’mm trying to use this code with my dataset but I have features which are strings and not numbers, as in your dataset.

What can I do to change the code in order for it to work? (I’m getting an error saying “could not convert string to float”

If they are labels, you can use a label encoder.

If you are working with string input, see NLP:

https://machinelearningmastery.com/start-here/#nlp

Hello, Dr. Jason! Thank you so much for this wonderful article. I have a question for you. I noticed you have not mentioned feature selection and feature engineering in the Python mini course. So, my question is that if we were to implement both of these tasks, what should be the order with respect to this present stage of spot checking and comparison of machine learning algorithms? Should we first select one or two best performing models after comparison and then implement feature selection and feature engineering or first implement them and then perform spot checking and model comparison?

Thank you in advanced.

Generally before spot checking.

Thank you for answering! But, that “generally” is really not helping. Can you please explain in which cases it is suggested to do that and in which cases not? I think it’s really important for me to learn.

No, it depends on the data and the project. I have to speak in generalities because I do not have the capacity to get involved in everyones project. Sorry.

Oh, okay. I hope to learn it as I do more projects. Thank you!

Thank you for the brilliant tutorial! If we have split our data in to train and test sets and wanted to know the accuracy of the trained model on the held out test data, could we do:

cv_results = model_selection.cross_val_score(model, X_test, Y_test, cv=kfold, scoring=scoring)

Or would we need to use .fit on the training data before testing on the test data? I hope this makes sense! Thank you.

Generally, we would fit the model on all of the training set, make predictions for the test set, then evaluate the skill of the predictions on the test set.

Ahh OK, so the above method is really just for comparing models? And then we train/ test using e.g. a 0.2 split? Would you ever actually train your model using cross-validation? Thanks so much.

No, you train the model on all data before making predictions. See this post:

https://machinelearningmastery.com/train-final-machine-learning-model/

Thanks for a great tutorial! I can follow the logic, but it seems UCI has taken down the pima indians dataset.

Thanks, I have updated the link to the dataset.

Thank you for the post Jason. I am curious however, how which scores are normally reported when comparing ML models fitted with slightly different features. E.g there is a set of features both models A and B share and A has been fitted a single unique feature and B has one as well, which A is not trained with. When one would evaluate such models for comparing, should we report and compare the scores on the test set or use cross_val_score with all the data?

Interesting.

Perhaps you could pick a measure that is relevant to the general domain, it could be something generic such as model accuracy or prediction error.

Thank you for the response, I was thinking of either using MAE or MSE since mine is a regression problem and see, which model achieves the lowest score.

However I don’t know, what is the general practice, should we compare the two models like:

A: Using cross_val_score to report MSE with the regressor and X, y where X and y is the entire data (no train/test split)

or

B: Doing a train/test split, fitting the model with train data and then making predictions using the test data and compare the MSE scores of the models

The choice and configuration of the test harness is a big part of the challenge of applied machine learning.

It must be tailored to your specific problem.

Hi Dr. Jason,

Thank you for this wonderful post. I have a question about complexity. As we usually use Accuracy to score the performance of Learning Algorithms. Is there any provision made for measuring the time taken(i.e. speed ) for the algorithm to complete the specific task given. I added a timeit function in one of your code i used from your post as follows

%timeit results = cross_val_score(bcancermodelb, X, Y, cv=kfold,scoring=scoring)

print(“Accuracy: %.3f%% (%.3f%%)” % (results.mean()*100.0, results.std()*100.0))

and i end up with the following result

19.2 ms ± 1.08 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

Accuracy: 93.420% (3.623%).

Can this be a good measure for the time taken for a particular model?

Generally no. If this is an important consideration in your model, then you can take it into account.

Hi Jason,

I am little confused about the box plot I am getting. I had compared 9 algorithms accuracy on my datasets with more than 90 features. The results I am getting is bit confusing as I only LDA showing more than 90% accuracy than other models, which ranges from 20% h 40%. Is the plot is valid and I can consider LDA as best model?

Perhaps inspect the raw data to confirm the finding.

hi i get this error , why would it be?

ValueError: Input contains NaN, infinity or a value too large for dtype(‘float64’).

Perhaps check that your input data was loaded correctly?

cv_results = model_selection.cross_val_score(model, X, Y, cv=kfold,scoring=scoring)

i probably have error in this line

Ensure you have copied the code exactly, preserving white space.

If I use the code above, I receive the following error message. (Python 3.6 – Spyder)

for name, model in models:

File “”, line 1

for name, model in models:

^

SyntaxError: unexpected EOF while parsing

Looks like you added a “,”.

Be sure to copy the code exactly.

I found the mistake.

Glad to hear it.

explain this: “preg’, ‘plas’, ‘pres’, ‘skin’, ‘test’, ‘mass’, ‘pedi’, ‘age’, ‘class'”

They are explained here:

https://github.com/jbrownlee/Datasets/blob/master/pima-indians-diabetes.names

Hi Jason

how can I get confusion matrix’s per algorithm from Your example? Also TP FP TN and FN to calculate prediction and recall per algorithm? Thanks!

This post explains the confusion matrix and shows how to calculate it:

https://machinelearningmastery.com/confusion-matrix-machine-learning/

Thanks Jason.

Hi, problem is that I know actual value (Y) but cannot find how to gain/print predicted values for each algorithm in this example?

This post explains how to make predictions with sklearn:

https://machinelearningmastery.com/make-predictions-scikit-learn/

Thank you so much for your all posts. They are extremely helpful.

I’m happy they help!