We cannot know which algorithm will be best for a given problem.

Therefore, we need to design a test harness that we can use to evaluate different machine learning algorithms.

In this tutorial, you will discover how to develop a machine learning algorithm test harness from scratch in Python.

After completing this tutorial, you will know:

- How to implement a train-test algorithm test harness.
- How to implement a k-fold cross-validation algorithm test harness.

Let’s get started.

**Update Jan/2017**: Changed the calculation of fold_size in cross_validation_split() to always be an integer. Fixes issues with Python 3.**Update Mar/2018**: Added alternate link to download the dataset as the original appears to have been taken down.**Update Aug/2018**: Tested and updated to work with Python 3.6.

## Description

A test harness provides a consistent way to evaluate machine learning algorithms on a dataset.

It involves 3 elements:

- The resampling method to split-up the dataset.
- The machine learning algorithm to evaluate.
- The performance measure by which to evaluate predictions.

The loading and preparation of a dataset is a prerequisite step that must have been completed prior to using the test harness.

The test harness must allow for different machine learning algorithms to be evaluated, whilst the dataset, resampling method and performance measures are kept constant.

In this tutorial, we are going to demonstrate the test harnesses with a real dataset.

The dataset used is the Pima Indians diabetes dataset. It contains 768 rows and 9 columns. All of the values in the file are numeric, specifically floating point values. You can learn more about this dataset on the UCI Machine Learning Repository (update: download from here).

The Zero Rule algorithm will be evaluated as part of the tutorial. The Zero Rule algorithm always predicts the class that has the most observations in the training dataset.

## Tutorial

This tutorial is broken down into two main sections:

- Train-Test Algorithm Test Harness.
- Cross-Validation Algorithm Test Harness.

These test harnesses will give you the foundation that you need to evaluate a suite of machine learning algorithms on a given predictive modeling problem.

### 1. Train-Test Algorithm Test Harness

The train-test split is a simple resampling method that can be used to evaluate a machine learning algorithm.

As such, it is a good starting point for developing a test harness.

We can assume the prior development of a function to split a dataset into train and test sets and a function to evaluate the accuracy of a set of predictions.

We need a function that can take a dataset and an algorithm and return a performance score.

Below is a function named **evaluate_algorithm()** that achieves this. It takes 3 fixed arguments including the dataset, the algorithm function and the split percentage for the train-test split.

First, the dataset is split into train and test elements. Next, a copy of the test set is made and each output value is cleared by setting it to the **None** value, to prevent the algorithm from cheating accidentally.

The algorithm provided as a parameter is a function that expects the train and test datasets on which to prepare and then make predictions. The algorithm may require additional configuration parameters. This is handled by using the variable arguments ***args** in the **evaluate_algorithm()** function and passing them on to the algorithm function.

The algorithm function is expected to return a list of predictions, one for each row in the training dataset. These are compared to the actual output values from the unmodified test dataset by the **accuracy_metric()** function.

Finally, the accuracy is returned.

1 2 3 4 5 6 7 8 9 10 11 12 |
# Evaluate an algorithm using a train/test split def evaluate_algorithm(dataset, algorithm, split, *args): train, test = train_test_split(dataset, split) test_set = list() for row in test: row_copy = list(row) row_copy[-1] = None test_set.append(row_copy) predicted = algorithm(train, test_set, *args) actual = [row[-1] for row in test] accuracy = accuracy_metric(actual, predicted) return accuracy |

The evaluation function does make some strong assumptions, but they can easily be changed if needed.

Specifically, it assumes that the last row in the dataset is always the output value. A different column could be used. The use of the **accuracy_metric()** assumes that the problem is a classification problem, but this could be changed to mean squared error for regression problems.

Let’s piece this together with a worked example.

We will use the Pima Indians diabetes dataset and evaluate the Zero Rule algorithm. You can download and learn more about the diabetes dataset on the UCI Machine Learning Repository (update: download from here).

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 |
# Train-Test Test Harness from random import seed from random import randrange from csv import reader # Load a CSV file def load_csv(filename): file = open(filename, "rb") lines = reader(file) dataset = list(lines) return dataset # Convert string column to float def str_column_to_float(dataset, column): for row in dataset: row[column] = float(row[column].strip()) # Split a dataset into a train and test set def train_test_split(dataset, split): train = list() train_size = split * len(dataset) dataset_copy = list(dataset) while len(train) < train_size: index = randrange(len(dataset_copy)) train.append(dataset_copy.pop(index)) return train, dataset_copy # Calculate accuracy percentage def accuracy_metric(actual, predicted): correct = 0 for i in range(len(actual)): if actual[i] == predicted[i]: correct += 1 return correct / float(len(actual)) * 100.0 # Evaluate an algorithm using a train/test split def evaluate_algorithm(dataset, algorithm, split, *args): train, test = train_test_split(dataset, split) test_set = list() for row in test: row_copy = list(row) row_copy[-1] = None test_set.append(row_copy) predicted = algorithm(train, test_set, *args) actual = [row[-1] for row in test] accuracy = accuracy_metric(actual, predicted) return accuracy # zero rule algorithm for classification def zero_rule_algorithm_classification(train, test): output_values = [row[-1] for row in train] prediction = max(set(output_values), key=output_values.count) predicted = [prediction for i in range(len(test))] return predicted # Test the zero rule algorithm on the diabetes dataset seed(1) # load and prepare data filename = 'pima-indians-diabetes.csv' dataset = load_csv(filename) for i in range(len(dataset[0])): str_column_to_float(dataset, i) # evaluate algorithm split = 0.6 accuracy = evaluate_algorithm(dataset, zero_rule_algorithm_classification, split) print('Accuracy: %.3f%%' % (accuracy)) |

The dataset was split into 60% for training the model and 40% for evaluating it.

Notice how the name of the Zero Rule algorithm **zero_rule_algorithm_classification** was passed as an argument to the **evaluate_algorithm()** function. You can see how this test harness may be used again and again with different algorithms.

Running the example above prints out the accuracy of the model.

1 |
Accuracy: 67.427% |

### 2. Cross-Validation Algorithm Test Harness

Cross-validation is a resampling technique that provides more reliable estimates of algorithm performance on unseen data.

It requires the creation and evaluation of k models on different subsets of your data, and such is more computationally expensive. Nevertheless, it is the gold standard for evaluating machine learning algorithms.

As in the previous section, we need to create a function that ties together the resampling method, the evaluation of the algorithm on the dataset and the performance calculation method.

Unlike above, the algorithm must be evaluated on different subsets of the dataset many times. This means we need additional loops within our **evaluate_algorithm()** function.

Below is a function that implements algorithm evaluation with cross-validation.

First, the dataset is split into **n_folds** groups called folds.

Next, we loop giving each fold an opportunity to be held out of training and used to evaluate the algorithm. A copy of the list of folds is created and the held out fold is removed from this list. Then the list of folds is flattened into one long list of rows to match the algorithms expectation of a training dataset. This is done using the **sum()** function.

Once the training dataset is prepared the rest of the function within this loop is as above. A copy of the test dataset (the fold) is made and the output values are cleared to avoid accidental cheating by algorithms. The algorithm is prepared on the train dataset and makes predictions on the test dataset. The predictions are evaluated and stored in a list.

Unlike the train-test algorithm test harness, a list of scores is returned, one for each cross-validation fold.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
# Evaluate an algorithm using a cross validation split def evaluate_algorithm(dataset, algorithm, n_folds, *args): folds = cross_validation_split(dataset, n_folds) scores = list() for fold in folds: train_set = list(folds) train_set.remove(fold) train_set = sum(train_set, []) test_set = list() for row in fold: row_copy = list(row) test_set.append(row_copy) row_copy[-1] = None predicted = algorithm(train_set, test_set, *args) actual = [row[-1] for row in fold] accuracy = accuracy_metric(actual, predicted) scores.append(accuracy) return scores |

Although slightly more complex in code and slower to run, this function provides a more robust estimate of algorithm performance.

We can tie all of this together with a complete example on the diabetes dataset with the Zero Rule algorithm.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 |
# Cross Validation Test Harness from random import seed from random import randrange from csv import reader # Load a CSV file def load_csv(filename): file = open(filename, "rb") lines = reader(file) dataset = list(lines) return dataset # Convert string column to float def str_column_to_float(dataset, column): for row in dataset: row[column] = float(row[column].strip()) # Split a dataset into k folds def cross_validation_split(dataset, n_folds): dataset_split = list() dataset_copy = list(dataset) fold_size = int(len(dataset) / n_folds) for i in range(n_folds): fold = list() while len(fold) < fold_size: index = randrange(len(dataset_copy)) fold.append(dataset_copy.pop(index)) dataset_split.append(fold) return dataset_split # Calculate accuracy percentage def accuracy_metric(actual, predicted): correct = 0 for i in range(len(actual)): if actual[i] == predicted[i]: correct += 1 return correct / float(len(actual)) * 100.0 # Evaluate an algorithm using a cross validation split def evaluate_algorithm(dataset, algorithm, n_folds, *args): folds = cross_validation_split(dataset, n_folds) scores = list() for fold in folds: train_set = list(folds) train_set.remove(fold) train_set = sum(train_set, []) test_set = list() for row in fold: row_copy = list(row) test_set.append(row_copy) row_copy[-1] = None predicted = algorithm(train_set, test_set, *args) actual = [row[-1] for row in fold] accuracy = accuracy_metric(actual, predicted) scores.append(accuracy) return scores # zero rule algorithm for classification def zero_rule_algorithm_classification(train, test): output_values = [row[-1] for row in train] prediction = max(set(output_values), key=output_values.count) predicted = [prediction for i in range(len(test))] return predicted # Test the zero rule algorithm on the diabetes dataset seed(1) # load and prepare data filename = 'pima-indians-diabetes.csv' dataset = load_csv(filename) for i in range(len(dataset[0])): str_column_to_float(dataset, i) # evaluate algorithm n_folds = 5 scores = evaluate_algorithm(dataset, zero_rule_algorithm_classification, n_folds) print('Scores: %s' % scores) print('Mean Accuracy: %.3f%%' % (sum(scores)/len(scores))) |

A total of 5 cross validation folds were used to evaluate the Zero Rule Algorithm. As such, 5 scores were returned from the **evaluate_algorithm()** algorithm.

Running this example both prints these list of scores calculated and prints the mean score.

1 2 |
Scores: [62.091503267973856, 64.70588235294117, 64.70588235294117, 64.70588235294117, 69.28104575163398] Mean Accuracy: 65.098% |

You now have two different test harnesses that you can use to evaluate your own machine learning algorithms.

## Extensions

This section lists extensions to this tutorial that you may wish to consider.

**Parameterized Evaluation**. Pass in the function used to evaluate predictions, allowing you to seamlessly work with regression problems.**Parameterized Resampling**. Pass in the function used to calculate resampling splits, allowing you to easily switch between the train-test and cross-validation methods.**Standard Deviation Scores**. Calculate the standard deviation to get an idea of the spread of scores when evaluating algorithms using cross-validation.

**Did you try any of these extensions?**

Share your experiences in the comments below.

## Review

In this tutorial, you discovered how to create a test harness from scratch to evaluate your machine learning algorithms.

Specifically, you now know:

- How to implement and use a train-test algorithm test harness.
- How to implement and use a cross-validation algorithm test harness.

**Do you have any questions?**

Ask your questions in the comments below and I will do my best to answer.

Thank you do for this will definitely get my start in machine learning.

I’m glad to hear it Timothy.

What’s the difference between this & using the built-in cross_val_score in Python sklearn?

Great question.

Use sklearn in practice.

If you want to learn how all these methods work from first principles, try implementing them yourself.

How about using the weka tool to harness the algorithm? As you said above, is it the same to that of using built-in function in sklearn? which stops at practicing and not to learn principles.

Can you elaborate on what you mean Anand?

In my implementatin get the following

line 163, in k_cross_validate

train_set.remove(fold)

ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

Sorry, I’ve not seen this error before. Are you using Python 2?