Machine learning algorithms have hyperparameters that allow the algorithms to be tailored to specific datasets.
Although the impact of hyperparameters may be understood generally, their specific effect on a dataset and their interactions during learning may not be known. Therefore, it is important to tune the values of algorithm hyperparameters as part of a machine learning project.
It is common to use naive optimization algorithms to tune hyperparameters, such as a grid search and a random search. An alternate approach is to use a stochastic optimization algorithm, like a stochastic hill climbing algorithm.
In this tutorial, you will discover how to manually optimize the hyperparameters of machine learning algorithms.
After completing this tutorial, you will know:
Stochastic optimization algorithms can be used instead of grid and random search for hyperparameter optimization.
How to use a stochastic hill climbing algorithm to tune the hyperparameters of the Perceptron algorithm.
How to manually optimize the hyperparameters of the XGBoost gradient boosting algorithm.
Kick-start your project with my new book Optimization for Machine Learning, including step-by-step tutorials and the Python source code files for all examples.
Let’s get started.
How to Manually Optimize Machine Learning Model Hyperparameters Photo by john farrell macdonald, some rights reserved.
Tutorial Overview
This tutorial is divided into three parts; they are:
Manual Hyperparameter Optimization
Perceptron Hyperparameter Optimization
XGBoost Hyperparameter Optimization
Manual Hyperparameter Optimization
Machine learning models have hyperparameters that you must set in order to customize the model to your dataset.
Often, the general effects of hyperparameters on a model are known, but how to best set a hyperparameter and combinations of interacting hyperparameters for a given dataset is challenging.
A better approach is to objectively search different values for model hyperparameters and choose a subset that results in a model that achieves the best performance on a given dataset. This is called hyperparameter optimization, or hyperparameter tuning.
A range of different optimization algorithms may be used, although two of the simplest and most common methods are random search and grid search.
Random Search. Define a search space as a bounded domain of hyperparameter values and randomly sample points in that domain.
Grid Search. Define a search space as a grid of hyperparameter values and evaluate every position in the grid.
Grid search is great for spot-checking combinations that are known to perform well generally. Random search is great for discovery and getting hyperparameter combinations that you would not have guessed intuitively, although it often requires more time to execute.
For more on grid and random search for hyperparameter tuning, see the tutorial:
Grid and random search are primitive optimization algorithms, and it is possible to use any optimization we like to tune the performance of a machine learning algorithm. For example, it is possible to use stochastic optimization algorithms. This might be desirable when good or great performance is required and there are sufficient resources available to tune the model.
Next, let’s look at how we might use a stochastic hill climbing algorithm to tune the performance of the Perceptron algorithm.
Want to Get Started With Optimization Algorithms?
Take my free 7-day email crash course now (with sample code).
Click to sign-up and also get a free PDF Ebook version of the course.
It is a model of a single neuron that can be used for two-class classification problems and provides the foundation for later developing much larger networks.
In this section, we will explore how to manually optimize the hyperparameters of the Perceptron model.
First, let’s define a synthetic binary classification problem that we can use as the focus of optimizing the model.
We can use the make_classification() function to define a binary classification problem with 1,000 rows and five input variables.
The example below creates the dataset and summarizes the shape of the data.
Running the example reports evaluates the model and reports the mean and standard deviation of the classification accuracy.
Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.
In this case, we can see that the model with default hyperparameters achieved a classification accuracy of about 78.5 percent.
We would hope that we can achieve better performance than this with optimized hyperparameters.
1
Mean Accuracy: 0.786 (0.069)
Next, we can optimize the hyperparameters of the Perceptron model using a stochastic hill climbing algorithm.
There are many hyperparameters that we could optimize, although we will focus on two that perhaps have the most impact on the learning behavior of the model; they are:
Learning Rate (eta0).
Regularization (alpha).
The learning rate controls the amount the model is updated based on prediction errors and controls the speed of learning. The default value of eta is 1.0. reasonable values are larger than zero (e.g. larger than 1e-8 or 1e-10) and probably less than 1.0
By default, the Perceptron does not use any regularization, but we will enable “elastic net” regularization which applies both L1 and L2 regularization during learning. This will encourage the model to seek small model weights and, in turn, often better performance.
We will tune the “alpha” hyperparameter that controls the weighting of the regularization, e.g. the amount it impacts the learning. If set to 0.0, it is as though no regularization is being used. Reasonable values are between 0.0 and 1.0.
First, we need to define the objective function for the optimization algorithm. We will evaluate a configuration using mean classification accuracy with repeated stratified k-fold cross-validation. We will seek to maximize accuracy in the configurations.
The objective() function below implements this, taking the dataset and a list of config values. The config values (learning rate and regularization weighting) are unpacked, used to configure the model, which is then evaluated, and the mean accuracy is returned.
Next, we need a function to take a step in the search space.
The search space is defined by two variables (eta and alpha). A step in the search space must have some relationship to the previous values and must be bound to sensible values (e.g. between 0 and 1).
We will use a “step size” hyperparameter that controls how far the algorithm is allowed to move from the existing configuration. A new configuration will be chosen probabilistically using a Gaussian distribution with the current value as the mean of the distribution and the step size as the standard deviation of the distribution.
We can use the randn() NumPy function to generate random numbers with a Gaussian distribution.
The step() function below implements this and will take a step in the search space and generate a new configuration using an existing configuration.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
# take a step in the search space
def step(cfg,step_size):
# unpack the configuration
eta,alpha=cfg
# step eta
new_eta=eta+randn()*step_size
# check the bounds of eta
ifnew_eta<=0.0:
new_eta=1e-8
# step alpha
new_alpha=alpha+randn()*step_size
# check the bounds of alpha
ifnew_alpha<0.0:
new_alpha=0.0
# return the new configuration
return[new_eta,new_alpha]
Next, we need to implement the stochastic hill climbing algorithm that will call our objective() function to evaluate candidate solutions and our step() function to take a step in the search space.
The search first generates a random initial solution, in this case with eta and alpha values in the range 0 and 1. The initial solution is then evaluated and is taken as the current best working solution.
1
2
3
4
5
...
# starting point for the search
solution=[rand(),rand()]
# evaluate the initial point
solution_eval=objective(X,y,solution)
Next, the algorithm iterates for a fixed number of iterations provided as a hyperparameter to the search. Each iteration involves taking a step and evaluating the new candidate solution.
1
2
3
4
5
...
# take a step
candidate=step(solution,step_size)
# evaluate candidate point
candidate_eval=objective(X,y,candidate)
If the new solution is better than the current working solution, it is taken as the new current working solution.
At the end of the search, the best solution and its performance are then returned.
Tying this together, the hillclimbing() function below implements the stochastic hill climbing algorithm for tuning the Perceptron algorithm, taking the dataset, objective function, number of iterations, and step size as arguments.
Running the example reports the configuration and result each time an improvement is seen during the search. At the end of the run, the best configuration and result are reported.
Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.
In this case, we can see that the best result involved using a learning rate slightly above 1 at 1.004 and a regularization weight of about 0.002 achieving a mean accuracy of about 79.1 percent, better than the default configuration that achieved an accuracy of about 78.5 percent.
Can you get a better result?
Let me know in the comments below.
cfg=[1.0048613895650054, 0.002162219228449132]: Mean Accuracy: 0.791333
Now that we are familiar with how to use a stochastic hill climbing algorithm to tune the hyperparameters of a simple machine learning algorithm, let’s look at tuning a more advanced algorithm, such as XGBoost.
XGBoost Hyperparameter Optimization
XGBoost is short for Extreme Gradient Boosting and is an efficient implementation of the stochastic gradient boosting machine learning algorithm.
The stochastic gradient boosting algorithm, also called gradient boosting machines or tree boosting, is a powerful machine learning technique that performs well or even best on a wide range of challenging machine learning problems.
First, the XGBoost library must be installed.
You can install it using pip, as follows:
1
sudo pip install xgboost
Once installed, you can confirm that it was installed successfully and that you are using a modern version by running the following code:
1
2
3
# xgboost
import xgboost
print("xgboost",xgboost.__version__)
Running the code, you should see the following version number or higher.
1
xgboost 1.0.1
Although the XGBoost library has its own Python API, we can use XGBoost models with the scikit-learn API via the XGBClassifier wrapper class.
An instance of the model can be instantiated and used just like any other scikit-learn class for model evaluation. For example:
1
2
3
...
# define model
model=XGBClassifier()
Before we tune the hyperparameters of XGBoost, we can establish a baseline in performance using the default hyperparameters.
We will use the same synthetic binary classification dataset from the previous section and the same test harness of repeated stratified k-fold cross-validation.
The complete example of evaluating the performance of XGBoost with default hyperparameters is listed below.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# xgboost with default hyperparameters for binary classification
from numpy import mean
from numpy import std
from sklearn.datasets import make_classification
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import RepeatedStratifiedKFold
Running the example evaluates the model and reports the mean and standard deviation of the classification accuracy.
Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.
In this case, we can see that the model with default hyperparameters achieved a classification accuracy of about 84.9 percent.
We would hope that we can achieve better performance than this with optimized hyperparameters.
1
Mean Accuracy: 0.849 (0.040)
Next, we can adapt the stochastic hill climbing optimization algorithm to tune the hyperparameters of the XGBoost model.
There are many hyperparameters that we may want to optimize for the XGBoost model.
For an overview of how to tune the XGBoost model, see the tutorial:
We will focus on four key hyperparameters; they are:
Learning Rate (learning_rate)
Number of Trees (n_estimators)
Subsample Percentage (subsample)
Tree Depth (max_depth)
The learning rate controls the contribution of each tree to the ensemble. Sensible values are less than 1.0 and slightly above 0.0 (e.g. 1e-8).
The number of trees controls the size of the ensemble, and often, more trees is better to a point of diminishing returns. Sensible values are between 1 tree and hundreds or thousands of trees.
The subsample percentages define the random sample size used to train each tree, defined as a percentage of the size of the original dataset. Values are between a value slightly above 0.0 (e.g. 1e-8) and 1.0
The tree depth is the number of levels in each tree. Deeper trees are more specific to the training dataset and perhaps overfit. Shorter trees often generalize better. Sensible values are between 1 and 10 or 20.
First, we must update the objective() function to unpack the hyperparameters of the XGBoost model, configure it, and then evaluate the mean classification accuracy.
Next, we need to define the step() function used to take a step in the search space.
Each hyperparameter is quite a different range, therefore, we will define the step size (standard deviation of the distribution) separately for each hyperparameter. We will also define the step sizes in line rather than as arguments to the function, to keep things simple.
The number of trees and the depth are integers, so the stepped values are rounded.
The step sizes chosen are arbitrary, chosen after a little trial and error.
The updated step function is listed below.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
# take a step in the search space
def step(cfg):
# unpack config
lrate,n_tree,subsam,depth=cfg
# learning rate
lrate=lrate+randn()*0.01
iflrate<=0.0:
lrate=1e-8
iflrate>1:
lrate=1.0
# number of trees
n_tree=round(n_tree+randn()*50)
ifn_tree<=0.0:
n_tree=1
# subsample percentage
subsam=subsam+randn()*0.1
ifsubsam<=0.0:
subsam=1e-8
ifsubsam>1:
subsam=1.0
# max tree depth
depth=round(depth+randn()*7)
ifdepth<=1:
depth=1
# return new config
return[lrate,n_tree,subsam,depth]
Finally, the hillclimbing() algorithm must be updated to define an initial solution with appropriate values.
In this case, we will define the initial solution with sensible defaults, matching the default hyperparameters, or close to them.
1
2
3
...
# starting point for the search
solution=step([0.1,100,1.0,7])
Tying this together, the complete example of manually tuning the hyperparameters of the XGBoost algorithm using a stochastic hill climbing algorithm is listed below.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
# xgboost manual hyperparameter optimization for binary classification
from numpy import mean
from numpy.random import randn
from numpy.random import rand
from numpy.random import randint
from sklearn.datasets import make_classification
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import RepeatedStratifiedKFold
Running the example reports the configuration and result each time an improvement is seen during the search. At the end of the run, the best configuration and result are reported.
Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision. Consider running the example a few times and compare the average outcome.
In this case, we can see that the best result involved using a learning rate of about 0.02, 52 trees, a subsample rate of about 50 percent, and a large depth of 53 levels.
This configuration resulted in a mean accuracy of about 87.3 percent, better than the default configuration that achieved an accuracy of about 84.9 percent.
Can you get a better result?
Let me know in the comments below.
It provides self-study tutorials with full working code on: Gradient Descent, Genetic Algorithms, Hill Climbing, Curve Fitting, RMSProp, Adam,
and much more...
Bring Modern Optimization Algorithms to Your Machine Learning Projects
Thanks again – very easy to understand and experiment.
I tried to reduce step size in the loop. For some reason hill climbing got me the best result with a very low etha = 0.009 :
cfg=[0.00916649745512498, 0.012304975044232886]: Mean Accuracy: 0.844000
Minor typo in variable name : candidte_eval => candidate_eval
Question : when we have multiple dimensions, I know it’s theoretically possible to get caught in a local maximum. How common is this in ML doing this kind of hyperparameter optimization? It’s what always made me scared to code my own optimizer like this, I figure the inbuilt libraries must handle this. E.g. if you start with many randomized starting points, how often do they converge at a single point, and how often do you get different maxima? E.g. we would call hillclimbing() multiple times.
This kind of black-box hill climbing is a very simple optimization algorithm, with few guarantees other than “probably better than nothing”. Remember that there is no gradient here, not even a rough estimate considering the subsequent runs of the algorithm itself. The hope is that it will fall into some obvious slope and improve a little bit, but in truth it’s hard to know if it actually improves anything over grid or random search. There is a great paper by Bergstra and Bengio that diacusses this a little bit: https://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf
I’m excited to uncover this page. I need to thank you for your time for this, particularly fantastic read!! I definitely really liked every part of it and I also have you saved to fav to look at new information in your site.
Thanks again – very easy to understand and experiment.
I tried to reduce step size in the loop. For some reason hill climbing got me the best result with a very low etha = 0.009 :
cfg=[0.00916649745512498, 0.012304975044232886]: Mean Accuracy: 0.844000
For the XGBClassifier – whatever I was doing it couldn’t get better than :
cfg=[0.00779, 42, 0.5165, 174]: Mean Accuracy: 0.875667
Very cool!
Well done!
Dear Dr.Jason
How to Manually Optimize Machine Learning Model Hyperparameters by applying regression?
You can adapt the above example directly for regression problems.
Thanks Jason.
Minor typo in variable name : candidte_eval => candidate_eval
Question : when we have multiple dimensions, I know it’s theoretically possible to get caught in a local maximum. How common is this in ML doing this kind of hyperparameter optimization? It’s what always made me scared to code my own optimizer like this, I figure the inbuilt libraries must handle this. E.g. if you start with many randomized starting points, how often do they converge at a single point, and how often do you get different maxima? E.g. we would call hillclimbing() multiple times.
Thanks
William
Thanks, fixed.
It is very common. It’s a good idea to run the search multiple times if possible.
This kind of black-box hill climbing is a very simple optimization algorithm, with few guarantees other than “probably better than nothing”. Remember that there is no gradient here, not even a rough estimate considering the subsequent runs of the algorithm itself. The hope is that it will fall into some obvious slope and improve a little bit, but in truth it’s hard to know if it actually improves anything over grid or random search. There is a great paper by Bergstra and Bengio that diacusses this a little bit: https://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf
I’m excited to uncover this page. I need to thank you for your time for this, particularly fantastic read!! I definitely really liked every part of it and I also have you saved to fav to look at new information in your site.
Thanks.