fbpx

How to Manually Optimize Machine Learning Model Hyperparameters

Last Updated on March 29, 2021

Machine learning algorithms have hyperparameters that allow the algorithms to be tailored to specific datasets.

Although the impact of hyperparameters may be understood generally, their specific effect on a dataset and their interactions during learning may not be known. Therefore, it is important to tune the values of algorithm hyperparameters as part of a machine learning project.

It is common to use naive optimization algorithms to tune hyperparameters, such as a grid search and a random search. An alternate approach is to use a stochastic optimization algorithm, like a stochastic hill climbing algorithm.

In this tutorial, you will discover how to manually optimize the hyperparameters of machine learning algorithms.

After completing this tutorial, you will know:

  • Stochastic optimization algorithms can be used instead of grid and random search for hyperparameter optimization.
  • How to use a stochastic hill climbing algorithm to tune the hyperparameters of the Perceptron algorithm.
  • How to manually optimize the hyperparameters of the XGBoost gradient boosting algorithm.

Let’s get started.

How to Manually Optimize Machine Learning Model Hyperparameters

How to Manually Optimize Machine Learning Model Hyperparameters
Photo by john farrell macdonald, some rights reserved.

Tutorial Overview

This tutorial is divided into three parts; they are:

  1. Manual Hyperparameter Optimization
  2. Perceptron Hyperparameter Optimization
  3. XGBoost Hyperparameter Optimization

Manual Hyperparameter Optimization

Machine learning models have hyperparameters that you must set in order to customize the model to your dataset.

Often, the general effects of hyperparameters on a model are known, but how to best set a hyperparameter and combinations of interacting hyperparameters for a given dataset is challenging.

A better approach is to objectively search different values for model hyperparameters and choose a subset that results in a model that achieves the best performance on a given dataset. This is called hyperparameter optimization, or hyperparameter tuning.

A range of different optimization algorithms may be used, although two of the simplest and most common methods are random search and grid search.

  • Random Search. Define a search space as a bounded domain of hyperparameter values and randomly sample points in that domain.
  • Grid Search. Define a search space as a grid of hyperparameter values and evaluate every position in the grid.

Grid search is great for spot-checking combinations that are known to perform well generally. Random search is great for discovery and getting hyperparameter combinations that you would not have guessed intuitively, although it often requires more time to execute.

For more on grid and random search for hyperparameter tuning, see the tutorial:

Grid and random search are primitive optimization algorithms, and it is possible to use any optimization we like to tune the performance of a machine learning algorithm. For example, it is possible to use stochastic optimization algorithms. This might be desirable when good or great performance is required and there are sufficient resources available to tune the model.

Next, let’s look at how we might use a stochastic hill climbing algorithm to tune the performance of the Perceptron algorithm.

Perceptron Hyperparameter Optimization

The Perceptron algorithm is the simplest type of artificial neural network.

It is a model of a single neuron that can be used for two-class classification problems and provides the foundation for later developing much larger networks.

In this section, we will explore how to manually optimize the hyperparameters of the Perceptron model.

First, let’s define a synthetic binary classification problem that we can use as the focus of optimizing the model.

We can use the make_classification() function to define a binary classification problem with 1,000 rows and five input variables.

The example below creates the dataset and summarizes the shape of the data.

Running the example prints the shape of the created dataset, confirming our expectations.

The scikit-learn provides an implementation of the Perceptron model via the Perceptron class.

Before we tune the hyperparameters of the model, we can establish a baseline in performance using the default hyperparameters.

We will evaluate the model using good practices of repeated stratified k-fold cross-validation via the RepeatedStratifiedKFold class.

The complete example of evaluating the Perceptron model with default hyperparameters on our synthetic binary classification dataset is listed below.

Running the example reports evaluates the model and reports the mean and standard deviation of the classification accuracy.

Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.

In this case, we can see that the model with default hyperparameters achieved a classification accuracy of about 78.5 percent.

We would hope that we can achieve better performance than this with optimized hyperparameters.

Next, we can optimize the hyperparameters of the Perceptron model using a stochastic hill climbing algorithm.

There are many hyperparameters that we could optimize, although we will focus on two that perhaps have the most impact on the learning behavior of the model; they are:

  • Learning Rate (eta0).
  • Regularization (alpha).

The learning rate controls the amount the model is updated based on prediction errors and controls the speed of learning. The default value of eta is 1.0. reasonable values are larger than zero (e.g. larger than 1e-8 or 1e-10) and probably less than 1.0

By default, the Perceptron does not use any regularization, but we will enable “elastic net” regularization which applies both L1 and L2 regularization during learning. This will encourage the model to seek small model weights and, in turn, often better performance.

We will tune the “alpha” hyperparameter that controls the weighting of the regularization, e.g. the amount it impacts the learning. If set to 0.0, it is as though no regularization is being used. Reasonable values are between 0.0 and 1.0.

First, we need to define the objective function for the optimization algorithm. We will evaluate a configuration using mean classification accuracy with repeated stratified k-fold cross-validation. We will seek to maximize accuracy in the configurations.

The objective() function below implements this, taking the dataset and a list of config values. The config values (learning rate and regularization weighting) are unpacked, used to configure the model, which is then evaluated, and the mean accuracy is returned.

Next, we need a function to take a step in the search space.

The search space is defined by two variables (eta and alpha). A step in the search space must have some relationship to the previous values and must be bound to sensible values (e.g. between 0 and 1).

We will use a “step size” hyperparameter that controls how far the algorithm is allowed to move from the existing configuration. A new configuration will be chosen probabilistically using a Gaussian distribution with the current value as the mean of the distribution and the step size as the standard deviation of the distribution.

We can use the randn() NumPy function to generate random numbers with a Gaussian distribution.

The step() function below implements this and will take a step in the search space and generate a new configuration using an existing configuration.

Next, we need to implement the stochastic hill climbing algorithm that will call our objective() function to evaluate candidate solutions and our step() function to take a step in the search space.

The search first generates a random initial solution, in this case with eta and alpha values in the range 0 and 1. The initial solution is then evaluated and is taken as the current best working solution.

Next, the algorithm iterates for a fixed number of iterations provided as a hyperparameter to the search. Each iteration involves taking a step and evaluating the new candidate solution.

If the new solution is better than the current working solution, it is taken as the new current working solution.

At the end of the search, the best solution and its performance are then returned.

Tying this together, the hillclimbing() function below implements the stochastic hill climbing algorithm for tuning the Perceptron algorithm, taking the dataset, objective function, number of iterations, and step size as arguments.

We can then call the algorithm and report the results of the search.

In this case, we will run the algorithm for 100 iterations and use a step size of 0.1, chosen after a little trial and error.

Tying this together, the complete example of manually tuning the Perceptron algorithm is listed below.

Running the example reports the configuration and result each time an improvement is seen during the search. At the end of the run, the best configuration and result are reported.

Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.

In this case, we can see that the best result involved using a learning rate slightly above 1 at 1.004 and a regularization weight of about 0.002 achieving a mean accuracy of about 79.1 percent, better than the default configuration that achieved an accuracy of about 78.5 percent.

Can you get a better result?
Let me know in the comments below.

Now that we are familiar with how to use a stochastic hill climbing algorithm to tune the hyperparameters of a simple machine learning algorithm, let’s look at tuning a more advanced algorithm, such as XGBoost.

XGBoost Hyperparameter Optimization

XGBoost is short for Extreme Gradient Boosting and is an efficient implementation of the stochastic gradient boosting machine learning algorithm.

The stochastic gradient boosting algorithm, also called gradient boosting machines or tree boosting, is a powerful machine learning technique that performs well or even best on a wide range of challenging machine learning problems.

First, the XGBoost library must be installed.

You can install it using pip, as follows:

Once installed, you can confirm that it was installed successfully and that you are using a modern version by running the following code:

Running the code, you should see the following version number or higher.

Although the XGBoost library has its own Python API, we can use XGBoost models with the scikit-learn API via the XGBClassifier wrapper class.

An instance of the model can be instantiated and used just like any other scikit-learn class for model evaluation. For example:

Before we tune the hyperparameters of XGBoost, we can establish a baseline in performance using the default hyperparameters.

We will use the same synthetic binary classification dataset from the previous section and the same test harness of repeated stratified k-fold cross-validation.

The complete example of evaluating the performance of XGBoost with default hyperparameters is listed below.

Running the example evaluates the model and reports the mean and standard deviation of the classification accuracy.

Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.

In this case, we can see that the model with default hyperparameters achieved a classification accuracy of about 84.9 percent.

We would hope that we can achieve better performance than this with optimized hyperparameters.

Next, we can adapt the stochastic hill climbing optimization algorithm to tune the hyperparameters of the XGBoost model.

There are many hyperparameters that we may want to optimize for the XGBoost model.

For an overview of how to tune the XGBoost model, see the tutorial:

We will focus on four key hyperparameters; they are:

  • Learning Rate (learning_rate)
  • Number of Trees (n_estimators)
  • Subsample Percentage (subsample)
  • Tree Depth (max_depth)

The learning rate controls the contribution of each tree to the ensemble. Sensible values are less than 1.0 and slightly above 0.0 (e.g. 1e-8).

The number of trees controls the size of the ensemble, and often, more trees is better to a point of diminishing returns. Sensible values are between 1 tree and hundreds or thousands of trees.

The subsample percentages define the random sample size used to train each tree, defined as a percentage of the size of the original dataset. Values are between a value slightly above 0.0 (e.g. 1e-8) and 1.0

The tree depth is the number of levels in each tree. Deeper trees are more specific to the training dataset and perhaps overfit. Shorter trees often generalize better. Sensible values are between 1 and 10 or 20.

First, we must update the objective() function to unpack the hyperparameters of the XGBoost model, configure it, and then evaluate the mean classification accuracy.

Next, we need to define the step() function used to take a step in the search space.

Each hyperparameter is quite a different range, therefore, we will define the step size (standard deviation of the distribution) separately for each hyperparameter. We will also define the step sizes in line rather than as arguments to the function, to keep things simple.

The number of trees and the depth are integers, so the stepped values are rounded.

The step sizes chosen are arbitrary, chosen after a little trial and error.

The updated step function is listed below.

Finally, the hillclimbing() algorithm must be updated to define an initial solution with appropriate values.

In this case, we will define the initial solution with sensible defaults, matching the default hyperparameters, or close to them.

Tying this together, the complete example of manually tuning the hyperparameters of the XGBoost algorithm using a stochastic hill climbing algorithm is listed below.

Running the example reports the configuration and result each time an improvement is seen during the search. At the end of the run, the best configuration and result are reported.

Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.

In this case, we can see that the best result involved using a learning rate of about 0.02, 52 trees, a subsample rate of about 50 percent, and a large depth of 53 levels.

This configuration resulted in a mean accuracy of about 87.3 percent, better than the default configuration that achieved an accuracy of about 84.9 percent.

Can you get a better result?
Let me know in the comments below.

Further Reading

This section provides more resources on the topic if you are looking to go deeper.

Tutorials

APIs

Articles

Summary

In this tutorial, you discovered how to manually optimize the hyperparameters of machine learning algorithms.

Specifically, you learned:

  • Stochastic optimization algorithms can be used instead of grid and random search for hyperparameter optimization.
  • How to use a stochastic hill climbing algorithm to tune the hyperparameters of the Perceptron algorithm.
  • How to manually optimize the hyperparameters of the XGBoost gradient boosting algorithm.

Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.

11 Responses to How to Manually Optimize Machine Learning Model Hyperparameters

  1. Slava Kostin March 24, 2021 at 10:46 am #

    Thanks again – very easy to understand and experiment.
    I tried to reduce step size in the loop. For some reason hill climbing got me the best result with a very low etha = 0.009 :
    cfg=[0.00916649745512498, 0.012304975044232886]: Mean Accuracy: 0.844000

    • Slava Kostin March 25, 2021 at 12:43 am #

      For the XGBClassifier – whatever I was doing it couldn’t get better than :
      cfg=[0.00779, 42, 0.5165, 174]: Mean Accuracy: 0.875667

    • Jason Brownlee March 25, 2021 at 4:39 am #

      Well done!

  2. Jack March 24, 2021 at 12:15 pm #

    Dear Dr.Jason
    How to Manually Optimize Machine Learning Model Hyperparameters by applying regression?

    • Jason Brownlee March 25, 2021 at 4:40 am #

      You can adapt the above example directly for regression problems.

  3. William Smith March 27, 2021 at 8:59 am #

    Thanks Jason.

    Minor typo in variable name : candidte_eval => candidate_eval

    Question : when we have multiple dimensions, I know it’s theoretically possible to get caught in a local maximum. How common is this in ML doing this kind of hyperparameter optimization? It’s what always made me scared to code my own optimizer like this, I figure the inbuilt libraries must handle this. E.g. if you start with many randomized starting points, how often do they converge at a single point, and how often do you get different maxima? E.g. we would call hillclimbing() multiple times.

    Thanks
    William

    • Jason Brownlee March 29, 2021 at 6:00 am #

      Thanks, fixed.

      It is very common. It’s a good idea to run the search multiple times if possible.

    • Rafael M. Martins April 18, 2021 at 4:17 pm #

      This kind of black-box hill climbing is a very simple optimization algorithm, with few guarantees other than “probably better than nothing”. Remember that there is no gradient here, not even a rough estimate considering the subsequent runs of the algorithm itself. The hope is that it will fall into some obvious slope and improve a little bit, but in truth it’s hard to know if it actually improves anything over grid or random search. There is a great paper by Bergstra and Bengio that diacusses this a little bit: https://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf

  4. snahta July 23, 2021 at 4:48 pm #

    I’m excited to uncover this page. I need to thank you for your time for this, particularly fantastic read!! I definitely really liked every part of it and I also have you saved to fav to look at new information in your site.

Leave a Reply