# Comprehensive Guide on Grid Search

*chevron_right*

*schedule*Mar 9, 2022

*toc*Table of Contents

*expand_more*

# What is grid search?

Grid search is a brute-force technique to find the optimal hyper-parameters for model building. Finding the optimal hyper-parameters is extremely important in machine learning because the final performance of a model will depend largely on the hyper-parameters. Grid search simply trains and evaluates a model based on the chosen values of hyper-parameters, and then selects the model that performs the best.

As an example, consider a Random Forest classifier. The two hyper-parameters are as follows:

`max_depth`

: the maximum-depth that a decision tree can go down to.`max_features`

: the maximum number of features that can be selected at random at each split.

For grid search, we supply values for these hyper-parameters that we want to test. For instance, suppose we wanted to test out the following values:

```
max_depth: [2,3]max_features: [1,2,3]
```

Grid search will then select every combination of the hyper-parameters (like a grid) and build the model for each combination using cross validation to obtain its performance. In this case, grid search will test out the following 6 combinations of hyper-parameters:

max_depth | max_features |
---|---|

2 | 1 |

2 | 2 |

2 | 3 |

3 | 1 |

3 | 2 |

3 | 3 |

For each model built, cross validation will return a performance metric, and therefore grid search will return the combination of hyper-parameters with the best performance.

# Using Python's sklearn to implement grid search

Suppose we wanted to classify the type of an iris given four features (e.g. sepal length) using a Random Forest classifier. As explained above, we can use grid search to tune the two hyper-parameters: `max_depth`

and `max_features`

.

We begin by importing the relevant modules:

```
from sklearn.ensemble import RandomForestClassifierfrom sklearn.model_selection import train_test_splitfrom sklearn.model_selection import GridSearchCVfrom sklearn.metrics import classification_reportfrom sklearn import datasetsimport pandas as pdimport numpy as np
```

We then read the Iris dataset and convert the data-type to Pandas' DataFrame:

```
sepal length (cm) sepal width (cm) petal length (cm) petal width (cm) target0 5.1 3.5 1.4 0.2 0.01 4.9 3.0 1.4 0.2 0.02 4.7 3.2 1.3 0.2 0.03 4.6 3.1 1.5 0.2 0.04 5.0 3.6 1.4 0.2 0.0
```

We then split the data into training and testing sets:

```
# Break into X (features) and y (target)
```

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=2000)
Number of rows of X_train: 120Number of rows of y_train: 120Number of rows of X_test: 30Number of rows of y_test: 30

We then use the training set to perform grid search:

```
param_grid = { "max_depth":[2,3], "max_features": [1,2,3]}
```

# random_state is like the seed - this is for reproducible resultsmodel = RandomForestClassifier(random_state=42)grid_search = GridSearchCV(model, param_grid, cv=5, scoring="accuracy")grid_search.fit(X_train, y_train)
Best hyper-parameters:{'max_depth': 2, 'max_features': 1}Best score:0.9833333333333334

Here, note the following:

Just like in our previous example, we are testing out 6 different combinations of hyper-parameters. Make sure that the keys of the

`param_grid`

match the keyword argument of the model - in this case,`RandomForestClassifier`

takes in as argument the keyword arguments`max_depth`

and`max_features`

.Since

`GridSearchCV`

uses cross validation to obtain the performance metric, we need to specify the number of folds with`cv`

. For a guide on cross validation, click here.The results of the grid search tell us that the best combination of hyper-parameters is

`max_depth=2`

and`max_features=1`

. With these hyper-parameters, the classification accuracy over 0.98.

Now that we have obtained the optimal hyper-parameters, we can build our model using the entire training set and get our final performance metric using the testing set:

```
model_optimal = RandomForestClassifier(max_depth=2, max_features=1, random_state=42)model_optimal.fit(X_train, y_train)y_test_predicted = model_optimal.predict(X_test)
precision recall f1-score support 0.0 1.00 1.00 1.00 8 1.0 0.88 0.70 0.78 10 2.0 0.79 0.92 0.85 12 accuracy 0.87 30 macro avg 0.89 0.87 0.87 30weighted avg 0.87 0.87 0.86 30
```

We see that the classification accuracy based on the training set is 0.87.

It is generally recommended that only the training set is used for hyper-parameter tuning and model selection.