Feature engineering is an important step in the machine learning pipeline. It is the process of transforming data in its native format into meaningful features to help the machine learning model learn better from the data.
If done right, feature engineering can significantly enhance the performance of machine learning algorithms. Beyond the basics of understanding your data and preprocessing, effective feature engineering involves creating interaction terms, generating indicator variables, and binning features into buckets.
These techniques help extract relevant information from the data and help build robust machine learning solutions. In this guide, we’ll explore these feature engineering techniques by spinning up a sample housing dataset.
Note: You can code along to this tutorial in your preferred Jupyter notebook environment. You can also follow along with the Google Colab notebook for this tutorial.
1. Understand Your Data
Before jumping into feature engineering, you should first thoroughly understand your data. This includes:
Exploring and visualizing your dataset to get an idea of the distribution and relationships between variables
Know the types of features you have (categorical, numerical, datetime objects, and more) and understand their significance in your analysis
Try to use domain knowledge to understand what each feature represents and how it might interact with other features. This insight can guide you in creating meaningful new features
Let’s create a sample housing dataset to work with:
In addition to getting basic information on the dataset, you can generate distribution plots and count plots for numeric and categorical variables, respectively. The following code snippets show basic exploratory data analysis on the dataset.
First, we get some basic information on the dataframe:
1
2
3
4
# Basic data exploration on the entire dataset
print(df.head())
print(df.info())
print(df.describe())
You can try to visualize the distribution of numeric features ‘size’ and ‘income’ in the dataset:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
import matplotlib.pyplot asplt
import seaborn assns
# Visualize distributions using distplot for 'size' and 'income'
plt.figure(figsize=(8,6))
sns.histplot(df['size'],kde=True)
plt.title('Distribution of House Sizes')
plt.xlabel('Size')
plt.ylabel('Frequency')
plt.show()
plt.figure(figsize=(8,6))
sns.histplot(df['income'],kde=True)
plt.title('Distribution of Household Income')
plt.xlabel('Income')
plt.ylabel('Frequency')
plt.show()
For categorical variables, count plot can help understand how the different values are distributed:
By understanding your data, you can identify key features and relationships between features that will inform the subsequent feature engineering steps. This step ensures that your preprocessing and feature creation efforts are grounded in a thorough understanding of the dataset.
2. Preprocess the Data Effectively
Effective preprocessing involves handling missing values and outliers, scaling numerical features, and encoding categorical variables. The choice of preprocessing techniques also depend on the data and the requirements of the machine learning algorithms.
We don’t have any missing values in the example dataframe. For most real-world datasets, you can handle missing values using suitable imputation strategies.
Before you go ahead with preprocessing, split the dataset into train and test sets:
1
2
3
4
5
6
7
8
from sklearn.model_selection import train_test_split
To bring numeric features all to the same scale, you can use minmax or standard scaling. Here’s a generic code snippet to impute missing values and scale numeric features:
1
2
3
4
5
6
7
8
9
10
11
12
13
from sklearn.model_selection import train_test_split
Replace ‘features_to_impute’ and ‘features_to_scale’ with the specific features you’d like to impute. We’ll also look at creating more representative features from the existing features in the next sections.
In summary, effective preprocessing prepares your data for all downstream tasks by ensuring consistency and addressing any issues with the raw data. This step is essential for getting accurate and reliable results from your machine learning models.
3. Create Interaction Terms
Creating interaction terms involves generating new features that capture the interactions between existing features.
For our example dataset, we’ll generate interaction terms for ‘size’ and ‘num_rooms’ using PolynomialFeatures from scikit-learn:
1
2
3
4
5
6
7
8
9
10
11
12
13
from sklearn.preprocessing import PolynomialFeatures
# Creating polynomial and interaction features on training set
Creating interaction terms can improve your model by capturing supposedly complex relationships between features.
4. Create Indicator Variables
You can create indicator variables to flag certain conditions or mark thresholds in your data. These variables take on values of 0 or 1, indicating the absence or presence of a particular value.
For example, suppose you have a dataset to predict loan default with a large number of defaults on student loans. It can be helpful to create an ‘is_student’ feature from the ‘professions’ categorical column.
In the housing dataset, we can create an indicator variable to denote if the houses are over 30 years old and create a count plot on the indicator variable ‘age_indicator’:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
import seaborn assns
import matplotlib.pyplot asplt
# Creating an indicator variable for houses older than 30 years
plt.title('Count of Houses Based on Age Indicator (>30 years)')
plt.xlabel('Age Indicator')
plt.ylabel('Count')
plt.show()
You can create indicator variable from the number of rooms, the ‘num_rooms’ column, as well. As seen, creating indicator variables can help encode additional information for machine learning models.
5. Create More Representative Features with Binning
Binning features into buckets involves grouping continuous variables into discrete intervals. Sometimes grouping features like age and income into bins can help find patterns that are hard to identify within continuous data.
For the example housing dataset, let’s bin the age of the house and income of the household into different bins with descriptive labels. You can use the cut() function in pandas to bin features into equal-width intervals like so:
Hi Bala!
Great article! Thank you so much for sharing…
In your point 5. Create More Representative Features with Binning:
where is written:
# Creating income bins
X_train[‘age_bin’] = pd.cut(X_train[‘age’], bins=3, labels=[‘new’, ‘moderate’, ‘old’])
X_test[‘age_bin’] = pd.cut(X_test[‘age’], bins=3, labels=[‘new’, ‘moderate’, ‘old’])
I believe there is a typo, as you meant: “# Creating age bins”.
Also, your instruction: “X_train[‘age_bin’] = pd.cut(X_train[‘age’], bins=3, labels=[‘new’, ‘moderate’, ‘old’])” raised an error in Pandas: “putmask: first argument must be an array”.
Thank you again.
Jose
Hi Jose…You’re correct that the comment should be about creating “age bins” rather than “income bins.” Thank you for pointing that out. The error you’re encountering might be due to the use of non-ASCII characters for the quotes in the code. Pandas expects standard ASCII quotes for specifying column names.
This should work without raising any errors. Make sure you’re using straight single quotes (') instead of any stylized or curly quotes (‘’), as this can cause issues in Python.
If you continue to experience issues, double-check that the age column exists in your X_train and X_test DataFrames and that the data type of the age column is numeric.
Hi Bala!
Great article! Thank you so much for sharing…
In your point 5. Create More Representative Features with Binning:
where is written:
# Creating income bins
X_train[‘age_bin’] = pd.cut(X_train[‘age’], bins=3, labels=[‘new’, ‘moderate’, ‘old’])
X_test[‘age_bin’] = pd.cut(X_test[‘age’], bins=3, labels=[‘new’, ‘moderate’, ‘old’])
I believe there is a typo, as you meant: “# Creating age bins”.
Also, your instruction: “X_train[‘age_bin’] = pd.cut(X_train[‘age’], bins=3, labels=[‘new’, ‘moderate’, ‘old’])” raised an error in Pandas: “putmask: first argument must be an array”.
Thank you again.
Jose
Hi Jose…You’re correct that the comment should be about creating “age bins” rather than “income bins.” Thank you for pointing that out. The error you’re encountering might be due to the use of non-ASCII characters for the quotes in the code. Pandas expects standard ASCII quotes for specifying column names.
Here’s the corrected code:
python
# Creating age bins
X_train['age_bin'] = pd.cut(X_train['age'], bins=3, labels=['new', 'moderate', 'old'])
X_test['age_bin'] = pd.cut(X_test['age'], bins=3, labels=['new', 'moderate', 'old'])
This should work without raising any errors. Make sure you’re using straight single quotes (
') instead of any stylized or curly quotes (‘’), as this can cause issues in Python.If you continue to experience issues, double-check that the
agecolumn exists in yourX_trainandX_testDataFrames and that the data type of theagecolumn is numeric.