The real estate market is a complex ecosystem driven by numerous variables such as location, property features, market trends, and economic indicators. One dataset that offers a deep dive into this complexity is the Ames Housing dataset. Originating from Ames, Iowa, this dataset comprises various properties and their characteristics, ranging from the type of alley access to the overall condition of the property.
In this post, you aim to take a closer look at this dataset using data science techniques. Specifically, you’ll focus on how to identify categorical and numerical variables, as understanding these variables is crucial for any data-driven decision-making process.
Let’s get started.
Exploring Dictionaries, Classifying Variables, and Imputing Data in the Ames Dataset Photo by Brigitte Tohm. Some rights reserved.
Overview
This post is divided into three parts; they are:
The Importance of a Data Dictionary
Identifying Categorical and Numerical Variables
Missing Data Imputation
The Importance of a Data Dictionary
A crucial first step in analyzing the Ames Housing dataset is utilizing its data dictionary. This version does more than list and define the features; it categorizes them into nominal, ordinal, discrete, and continuous types, guiding our analysis approach.
Nominal Variables are categories without an order like ‘Neighborhood’. They help in identifying segments for grouping analysis.
Ordinal Variables have a clear order (e.g ‘KitchenQual’). They allow for ranking and order-based analysis but don’t imply equal spacing between categories.
Discrete Variables are countable numbers, like ‘Bedroom’. They are integral to analyses that sum or compare quantities.
Continuous Variables measure on a continuous scale, like ‘Lot Area’. They enable a wide range of statistical analyses that depend on granular detail.
Understanding these variable types also guides the selection of appropriate visualization techniques. Nominal and ordinal variables are well-suited to bar charts, which can effectively highlight categorical differences and rankings. In contrast, discrete and continuous variables are best represented through histograms, scatter plots, and line charts, which illustrate distributions, relationships, and trends within the data.
Building on our understanding of the data dictionary, let’s delve into how we can practically distinguish between categorical and numerical variables within the Ames dataset using Python’s pandas library. This step is crucial for informing our subsequent data processing and analysis strategies.
1
2
3
4
5
6
# Load and obtain the data types from the Ames dataset
import pandas aspd
Ames=pd.read_csv('Ames.csv')
print(Ames.dtypes)
print(Ames.dtypes.value_counts())
Executing the above code will yield the following output, categorizing each feature by its data type:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
PID int64
GrLivArea int64
SalePrice int64
MSSubClass int64
MSZoning object
...
SaleCondition object
GeoRefNo float64
Prop_Addr object
Latitude float64
Longitude float64
Length:85,dtype:object
object44
int6427
float6414
dtype:int64
This output reveals that the dataset comprises object (44 variables), int64 (27 variables), and float64 (14 variables) data types. Here, object typically indicates nominal variables, which are categorical data without an inherent order. Meanwhile, int64 and float64 suggest numerical data, which could be either discrete (int64 for countable numbers) or continuous (float64 for measurable quantities on a continuous scale).
Now we can leverage pandas’ select_dtypes() method to explicitly separate numerical and categorical features within the Ames dataset.
The numerical_features captures variables stored as int64 and float64, indicative of countable and measurable quantities, respectively. Conversely, the categorical_features comprises variables of type object, typically representing nominal or ordinal data without a quantitative value:
Notably, some variables, like ‘MSSubClass’ despite being encoded numerically, actually serve as categorical data, underscoring the importance of referring back to our data dictionary for accurate classification. Similarly, features like ‘MoSold’ (Month Sold) and ‘YrSold’ (Year Sold) are numerical in nature, but they can often be treated as categorical variables, especially when there is no interest in performing mathematical operations on them. We can use the astype() method in pandas to convert these to categorical features.
After performing this conversion, the count of columns with the object data type has increased to 47 (from the previous 44), while int64 has dropped to 24 (from 27).
1
2
3
4
object47
int6424
float6414
dtype:int64
A careful assessment of the data dictionary, the nature of the dataset, and domain expertise can contribute to properly reclassifying data types.
Missing Data Imputation
Dealing with missing data is a challenge that every data scientist faces. Ignoring missing values or handling them inadequately can lead to skewed analysis and incorrect conclusions. The choice of imputation technique often depends on the nature of the data—categorical or numerical. In addition, information in the data dictionary will be useful (such as the case for Pool Quality) where a missing value (“NA”) has a meaning, namely the absence of this feature for a particular property.
Data Imputation For Categorical Features with Missing Values
You can identify categorical data types and rank them in the order in which they are most affected by missing data.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
# Calculating the percentage of missing values for each column
missing_data=Ames.isnull().sum()
missing_percentage=(missing_data/len(Ames))*100
data_type=Ames.dtypes
# Combining the counts and percentages into a DataFrame for better visualization
The data dictionary indicates that missing values for the entire list of categorical features above indicate the absence of that feature for a given property, except for “Electrical”. With this insight, we can impute with the “mode” for the 1 missing data point for the electrical system and impute all others using "None" (with quotations to make it a Python string).
This confirms that there are now no more missing values for categorical features:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
PoolQC0
MiscFeature0
Alley0
Fence0
FireplaceQu0
GarageCond0
GarageQual0
GarageFinish0
GarageType0
BsmtExposure0
BsmtFinType20
BsmtFinType10
BsmtQual0
BsmtCond0
Prop_Addr0
MasVnrType0
Electrical0
Data Imputation For Numerical Features with Missing Values
We can apply the same technique demonstrated above to identify numerical data types and rank them in the order in which they are most affected by missing data.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
# Build on the above blocks of code
# Import Numpy
import numpy asnp
# Calculating the percentage of missing values for each column
missing_data=Ames.isnull().sum()
missing_percentage=(missing_data/len(Ames))*100
data_type=Ames.dtypes
# Combining the counts and percentages into a DataFrame for better visualization
The above illustrates that there are fewer instances of missing numerical data versus missing categorical data. However, the data dictionary is not as useful for a straightforward imputation. Whether or not to impute missing data in data science largely depends on the goal of the analysis. Often, a data scientist may generate multiple imputations to account for the uncertainty in the imputation process. Common multiple imputation methods include (but are not limited to) mean, median, and regression imputation. As a baseline, we will illustrate how to employ mean imputation here, but may refer to other techniques depending on the task at hand.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
# Build on the above blocks of code
# Initialize a DataFrame to store the concise information
concise_info=pd.DataFrame(columns=['Feature','Missing Values After Imputation',
'Mean Value Used to Impute'])
# Identify and impute missing numerical values, and store the related concise information
# Append the concise information to the concise_info DataFrame
concise_info.loc[len(concise_info)]=pd.Series({
'Feature':item,
'Missing Values After Imputation':Ames[item].isnull().sum(),
# This should be 0 as we are imputing all missing values
'Mean Value Used to Impute':mean_value
})
# Display the concise_info DataFrame
print(concise_info)
This prints:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
Feature Missing Values After Imputation Mean Value Used toImpute
0LotFrontage06.851063e+01
1GarageYrBlt01.976997e+03
2Longitude0-9.364254e+01
3Latitude04.203456e+01
4GeoRefNo07.136762e+08
5MasVnrArea09.934698e+01
6BsmtFullBath04.353900e-01
7BsmtHalfBath06.208770e-02
8BsmtFinSF205.325950e+01
9GarageArea04.668646e+02
10BsmtFinSF104.442851e+02
11BsmtUnfSF05.391947e+02
12TotalBsmtSF01.036739e+03
13GarageCars01.747867e+00
At times, we may also opt to leave the missing value without any imputation to retain the authenticity of the original dataset and remove the observations that do not have complete and accurate data if required. Alternatively, you may also try to build a machine learning model to guess the missing value based on some other data in the same rows, which is the principle behind imputation by regression. As a final step of the above baseline imputation, let us cross-check if there are any missing values.
1
2
3
# Build on the above blocks of code
missing_values_count=Ames.isnull().sum().sum()
print(f'The DataFrame has a total of {missing_values_count} missing values.')
You should see:
1
The DataFrame hasatotal of0missing values.
Congratulations! We have successfully imputed every missing value in the Ames dataset using baseline operations. It’s important to note that numerous other techniques exist for imputing missing data. As a data scientist, exploring various options and determining the most appropriate method for the given context is crucial to producing reliable and meaningful results.
Want to Get Started With Beginner's Guide to Data Science?
Take my free email crash course now (with sample code).
Click to sign-up and also get a free PDF Ebook version of the course.
In this tutorial, we explored the Ames Housing dataset through the lens of data science techniques. We discussed the importance of a data dictionary in understanding the dataset’s variables and dove into Python code snippets that help identify and handle these variables effectively.
Understanding the nature of the variables you’re working with is crucial for any data-driven decision-making process. As we’ve seen, the Ames data dictionary serves as a valuable guide in this respect. Coupled with Python’s powerful data manipulation libraries, navigating complex datasets like the Ames Housing dataset becomes a much more manageable task.
Specifically, you learned:
The importance of a data dictionary when assessing data types and imputation strategies.
Identification and reclassification methods for numerical and categorical features.
How to impute missing categorical and numerical features using the pandas library.
Do you have any questions? Please ask your questions in the comments below, and I will do my best to answer.
Get Started on The Beginner's Guide to Data Science!
Learn the mindset to become successful in data science projects
...using only minimal math and statistics, acquire your skill through short examples in Python
It provides self-study tutorials with all working code in Python to turn you from a novice to an expert. It shows you how to find outliers, confirm the normality of data, find correlated features, handle skewness, check hypotheses, and much more...all to support you in creating a narrative from a dataset.
Kick-start your data science journey with hands-on exercises
No comments yet.