Description
About The Data
Our goal for this lab is construct a model that can take a certain set of housing features and give us
back a price estimate. Since price is a continuous variable, linear regression may be a good place to
start from.
The dataset that we’ll be using for this task comes from kaggle.com and contains the following
attributes:
‘Avg. Area Income’: Avg. income of residents of the city house is located in.
‘Avg. Area House Age’: Avg age of houses in same city
‘Avg. Area Number of Rooms’: Avg number of rooms for houses in same city
‘Avg. Area Number of Bedrooms’: Avg number of bedrooms for houses in same city
‘Area Population’: Population of city house is located in
‘Price’: Price that the house sold at (target)
‘Address’: Address for the house
Exploratory Data Analysis
Let’s begin by importing some necessary libraries that we’ll be using to explore the data.
Our first step is to load the data into a pandas DataFrame
Avg. Area
Income
Avg.
Area
House
Age
Avg.
Area
Number
of
Rooms
Avg. Area
Number
of
Bedrooms
Area
Population
Price Address
0 79545.458574 5.682861 7.009188 4.09 23086.800503 1.059034e+06
208 Michael Ferry Apt.
674\nLaurabury, NE
3701…
1 79248.642455 6.002900 6.730821 3.09 40173.072174 1.505891e+06
188 Johnson Views
Suite 079\nLake
Kathleen, CA…
2 61287.067179 5.865890 8.512727 5.13 36882.159400 1.058988e+06
9127 Elizabeth
Stravenue\nDanieltown,
WI 06482…
3 63345.240046 7.188236 5.586729 3.26 34310.242831 1.260617e+06 USS Barnett\nFPO AP
44820
4 59982.197226 5.040555 7.839388 4.23 26354.109472 6.309435e+05 USNS Raymond\nFPO
AE 09386
From here, it’s always a good step to use describe() and info() to get a better sense of the data and
see if we have any missing values.
Avg. Area
Income
Avg. Area
House Age
Avg. Area
Number of
Rooms
Avg. Area
Number of
Bedrooms
Area
Population
Price
count 5000.000000 5000.000000 5000.000000 5000.000000 5000.000000 5.000000e+03
mean 68583.108984 5.977222 6.987792 3.981330 36163.516039 1.232073e+06
std 10657.991214 0.991456 1.005833 1.234137 9925.650114 3.531176e+05
min 17796.631190 2.644304 3.236194 2.000000 172.610686 1.593866e+04
25% 61480.562388 5.322283 6.299250 3.140000 29403.928702 9.975771e+05
50% 68804.286404 5.970429 7.002902 4.050000 36199.406689 1.232669e+06
75% 75783.338666 6.650808 7.665871 4.490000 42861.290769 1.471210e+06
max 107701.748378 9.519088 10.759588 6.500000 69621.713378 2.469066e+06
The info below lets us know that we have 5,000 entries and 5,000 non‑null values in each
feature/column. Therefore, there are no missing values in this dataset.
<class
’pandas.core.frame.DataFrame’>
RangeIndex:
5000
entries,
0
to
4999
Data
columns
(total
7
columns):
#
Column
NonNull
Count
Dtype
0
Avg.
Area
Income
5000
nonnull
float64
1
Avg.
Area
House
Age
5000
nonnull
float64
2
Avg.
Area
Number
of
Rooms
5000
nonnull
float64
3
Avg.
Area
Number
of
Bedrooms
5000
nonnull
float64
4
Area
Population
5000
nonnull
float64
5
Price
5000
nonnull
float64
6
Address
5000
nonnull
object
dtypes:
float64(6),
object(1)
memory
usage:
273.6+
KB
A quick pairplot lets us get an idea of the distributions and relationships in our dataset. From here, we
could choose any interesting features that we’d like to later explore in greater depth. Warning: The
more features in our dataset, the harder our pairplot will be to interpret.
Taking a closer look at price, we see that it’s normally distributed with a peak around 1.232073e+06,
and 75% of houses sold were at a price of 1.471210e+06 or lower.
count
5.000000e+03
mean
1.232073e+06
std
3.531176e+05
min
1.593866e+04
25%
9.975771e+05
50%
1.232669e+06
75%
1.471210e+06
max
2.469066e+06
Name:
Price,
dtype:
float64
A scatterplot of Price vs. Avg. Area Income shows a strong positive linear relationship between the
two.
Creating a boxplot of Avg. Area Number of Bedrooms lets us see that the median average area number
of bedrooms is around 4, with a minimum of 2 and max of around 6.5. We can also so that there are no
outliers present.
Try plotting some of the other features for yourself to see if you can discover some interesting
findings. Refer back to the matplotlib lab if you’re having trouble creating any graphs.
Another important thing to look for while we’re exploring our data is multicollinearity. Multicollinearity
means that several variables are essentially measuring the same thing. Not only is there no point to
having more than one measure of the same thing in a model, but doing so can actually cause our
model results to fluctuate. Luckily, checking for multicollinearity can be done easily with the help of a
heatmap. Note: Depending on the situation, it may not be a problem for your model if only slight
or moderate collinearity issue occur. However, it is strongly advised to solve the issue if severe
collinearity issue exists(e.g. correlation >0.8 between 2 variables)
This dataset is quite clean, and so there’s no severe collinearity issues. We’ll later dive into some
messier datasets which will require some type of feature engineering or PCA to resolve.
Creating Our Linear Model
We’re now ready to begin creating and training our model. We first need to split our data into training
and testing sets. This can be done using sklearn’s train_test_split(X, y, test_size) function. This
function takes in your features (X), the target variable (y), and the test_size you’d like (Generally a test
size of around 0.3 is good enough). It will then return a tuple of X_train, X_test, y_train, y_test sets for
us. We will train our model on the training set and then use the test set to evaluate the model.
We’ll now import sklearn’s LinearRegression model and begin training it using the fit(train_data,
train_data_labels) method. In a nutshell, fitting is equal to training. Then, after it is trained, the model
can be used to make predictions, usually with a predict(test_data) method call. You can think of fit as
the step that finds the coefficients for the equation.
LinearRegression()
Model Evaluation
Now that we’ve finished training, we can make predictions off of the test data and evaluate our
model’s performance using the corresponding test data labels (y_test).
To get a rough idea of how well the model is predicting, we can make a scatterplot with the true test
labels (y_test) on the x‑axis, and our predictions on the y‑axis. Ideally, we’d like a 45 degree line. The
straighter the line, the better our predictions are.
Something that you may recall from MATH 3339 is that we’d like to see the residuals be normally
distributed in regression analysis. We can exam this as follows:
Here are the most common evaluation metrics for regression problems:
Mean Absolute Error (MAE) is the mean of the absolute value of the errors:
Mean Squared Error (MSE) is the mean of the squared errors:
Root Mean Squared Error (RMSE) is the square root of the mean of the squared errors:
Comparing these metrics:
MAE is the easiest to understand, because it’s the average error.
MSE is more popular than MAE, because MSE “punishes” larger errors, which tends to be useful in
the real world.
RMSE is even more popular than MSE, because RMSE is interpretable in the “y” units.
All of these are loss functions, because we want to minimize them.
Luckily, sklearn can calculate all of these metrics for us. All we need to do is pass the true labels
(y_test) and our predictions to the functions below. What’s more important is that we understand what
each of these means. Root Mean Square Error (RMSE) is what we’ll most commonly use, which is the
standard deviation of the residuals (prediction errors). Residuals are a measure of how far from the
regression line data points are; RMSE is a measure of how spread out these residuals are. In other
words, it tells us how concentrated the data is around the line of best fit. Determining a good RMSE
depends on your data. You can find a great example here, or refer back to the power points.
MAE:
83410.59496702514
MSE:
10608579825.136667
RMSE:
102997.96029600133
Something we also like to look at is the coefficient of determination ( ), which is the percentage of
variation in y explained by all the x variables together. Usually an of .70 is considered good.
R2
Score:
0.9150208174786678
Finally, let’s see how we can interpret our model’s coefficients. We can access the coefficients by
calling coef_ on our linear model (lm in this case). We’ll use this and put it in a nice pandas DataFrame
for visual purposes. Note: You can also call intercept_ if you’d like to get the intercept.
Coefficient
Avg. Area Income 21.564645
Avg. Area House Age 166102.423648
Avg. Area Number of Rooms 122398.915857
Avg. Area Number of Bedrooms 887.665746
Area Population 15.309706
What these coefficients mean:
Holding all other features fixed, a 1 unit increase in Avg. Area Income is associated with an
increase of $21.564645 .
Holding all other features fixed, a 1 unit increase in Avg. Area House Age is associated with an
increase of $166102.423648 .
Holding all other features fixed, a 1 unit increase in Avg. Area Number of Rooms is associated
with an increase of $122398.915857 .
Holding all other features fixed, a 1 unit increase in Avg. Area Number of Bedrooms is associated
with an increase of $887.665746 .
Holding all other features fixed, a 1 unit increase in Area Population is associated with an
increase of $15.309706 .
Congratulations! You now know how to create and evaluate linear models using sklearn. As extra
practice, I’d recommend now trying to find a used car or similar housing dataset on kaggle.com and
use this notebook as a guide.
In
[1]:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
In
[2]:
from matplotlib import rcParams
rcParams[‘figure.figsize’] = 15, 5
sns.set_style(‘darkgrid’)
In
[3]:
housing_data = pd.read_csv(‘USA_Housing.csv’)
housing_data.head()
Out[3]: In
[4]:
housing_data.describe()
Out[4]: In
[5]:
housing_data.info()
In
[6]: sns.pairplot(housing_data)
plt.show()
In
[7]: sns.histplot(housing_data[‘Price’])
plt.show()
print(housing_data[‘Price’].describe())
In
[8]: sns.scatterplot(x=’Price’, y=’Avg.
Area
Income’, data=housing_data)
plt.show()
In
[9]: sns.boxplot(x=’Avg.
Area
Number
of
Bedrooms’, data=housing_data)
plt.show()
In
[10]: sns.heatmap(housing_data.corr(), annot=True)
plt.show()
In
[11]:
from sklearn.model_selection import train_test_split
X = housing_data[[‘Avg.
Area
Income’, ‘Avg.
Area
House
Age’, ‘Avg.
Area
Number
of
Room
‘Avg.
Area
Number
of
Bedrooms’, ‘Area
Population’]]
y = housing_data[‘Price’]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)
In
[12]:
from sklearn.linear_model import LinearRegression
lm = LinearRegression()
lm.fit(X_train,y_train)
Out[12]: In
[13]:
predictions = lm.predict(X_test)
plt.scatter(y_test,predictions)
plt.show()
In
[14]:
residuals = y_test predictions
sns.histplot(residuals)
plt.show()
n∑
i=1
|yi − y^i
|
1
n
n∑
i=1
(yi − y^i)
2 1
n
⎷
n∑
i=1
(yi − y^i)
2 1
n
In
[15]:
from sklearn import metrics
print(‘MAE:’, metrics.mean_absolute_error(y_test, predictions))
print(‘MSE:’, metrics.mean_squared_error(y_test, predictions))
print(‘RMSE:’, np.sqrt(metrics.mean_squared_error(y_test, predictions)))
R2
R2
In
[16]:
from sklearn.metrics import r2_score
print(‘R2
Score:
’, r2_score(y_test, predictions))
In
[17]:
coeff_df = pd.DataFrame(lm.coef_,X.columns,columns=[‘Coefficient’])
coeff_df
Out[17]:
COSC 3337
