Description
(Decision Trees and Random Forest)
This lab will walk you through how you can use decision tree and random forest in sklearn on your own datasets. We will also be
comparing the two methods. To begin, let’s quickly review some of the decision tree intuition that should sound familiar if you’ve
attended the corresponding decision tree lecture.
Intuition
The Decision Tree Algorithm
This is a supervised learning algorithm, but unlike other supervised learning algorithms, the decision tree algorithm can be used
for solving both regression and classification problems.
The goal of a Decision Tree is to create a model that can predict the class or value of the target variable by learning simple
decision rules inferred from the training data.
Decision trees classify the examples by sorting them down the tree from the root to some leaf/terminal node, with the leaf/terminal
node providing the classification of the example. Each node in the tree acts as a test case for some attribute, and each edge
descending from the node corresponds to the possible answers to the test case. This process is recursive in nature and is
repeated for every subtree rooted at the new node.
The primary challenge in the decision tree implementation is to identify which attributes do we need to consider as the root node
at each level. For solving this attribute selection problem, researchers have devised some of the following attribute selection
measures:
Entropy,
Information gain,
Gini index,
Gain Ratio,
Reduction in Variance
Chi‑Square
These criterias will calculate values for every attribute. The values are sorted, and attributes are placed in the tree by following the
order i.e, the attribute with the highest value(in case of information gain) is placed at the root.
Note: The most popular attribute selection methods that we’ll use in this course are information gain and gini index.
Potential Problems
Overfitting is a practical problem while building a Decision‑Tree model. The problem of overfitting is considered when the
algorithm continues to go deeper and deeper to reduce the training‑set error but results with an increased test‑set error. So,
accuracy of prediction for our model goes down. It generally happens when we build many branches due to outliers and
irregularities in data.
To avoid overfitting, we can use the following:
Pre‑Pruning: Stop the tree construction a bit early. We prefer not to split a node if its goodness measure is below a threshold
value, but it is difficult to choose an appropriate stopping point.
Post‑Pruning: First generate the decision tree and then remove non‑significant branches. Post‑pruning a decision tree implies
that we begin by generating the (complete) tree and then adjust it with the aim of improving the accuracy on unseen
instances.
About The Data
We’ll be using the Car Evaluation Data Set from the UCI Machine Learning Repository for this lab, but feel free to follow along with
your own dataset. The dataset contains the following attributes:
buying (v‑high, high, med, low)
maint (v‑high, high, med, low)
doors (2, 3, 4, 5‑more)
persons (2, 4, more)
lug_boot (small, med, big)
safety (low, med, high)
class (unacc, acc, good, vgood)
Quick Exploratory Data Analysis
Let’s begin by importing some necessary libraries that we’ll be using to explore the data.
Our first step is to load the data into a pandas DataFrame. For some reason, this dataset did not come with a header/column
names, so we will specify that when loading the data and manually add the column names ourselves.
buying maint doors persons lug_boot safety class
0 vhigh vhigh 2 2 small low unacc
1 vhigh vhigh 2 2 small med unacc
2 vhigh vhigh 2 2 small high unacc
3 vhigh vhigh 2 2 med low unacc
4 vhigh vhigh 2 2 med med unacc
After checking .info() we can see that there are no missing values. Our dataset contains 1728 entries, and each of our columns
contains 1728 non‑null values.
<class
’pandas.core.frame.DataFrame’>
RangeIndex:
1728
entries,
0
to
1727
Data
columns
(total
7
columns):
#
Column
NonNull
Count
Dtype
0
buying
1728
nonnull
object
1
maint
1728
nonnull
object
2
doors
1728
nonnull
object
3
persons
1728
nonnull
object
4
lug_boot
1728
nonnull
object
5
safety
1728
nonnull
object
6
class
1728
nonnull
object
dtypes:
object(7)
memory
usage:
94.6+
KB
If we create countplots of each attribute we can see that there seems to be an equal balance of each unique type in each column.
Let’s also take a look at our target/class variable
unacc
1210
acc
384
good
69
vgood
65
Name:
class,
dtype:
int64
Majority of our dataset consists of the unacc and acc, with very few vgood and good records.
Data Preprocessing
Let’s now prepare our data for training. Notice that all of our variables are ordinal categorical variables. When dealing with ordinal
categorical variables, you want to make sure to preserve the order when encoding them, so we can use sklearn’s ordinal encoder,
or manually map each unique value in each column to some number [0, n_classes‑1].
Note: We can’t use one hot encoding / get_dummies here because that won’t preserve the order.
buying maint doors persons lug_boot safety class
0 vhigh vhigh 2 2 small low unacc
1 vhigh vhigh 2 2 small med unacc
2 vhigh vhigh 2 2 small high unacc
3 vhigh vhigh 2 2 med low unacc
4 vhigh vhigh 2 2 med med unacc
One way to do our encoding is by first creating mappings that preserve the order in each column. For example, low category maps
to 0, and vhigh category maps with 3.
We can then pass use pandas .map(dictionary) to apply our mappings to the necessary columns.
displaying our DataFrame again we can see that it’s now ready for training.
buying maint doors persons lug_boot safety class
0 3 3 2 2 0 0 0
1 3 3 2 2 0 1 0
2 3 3 2 2 0 2 0
3 3 3 2 2 1 0 0
4 3 3 2 2 1 1 0
Creating Our Tree Models
We’re now ready to begin creating and training our model. We first need to split our data into training and testing sets. This can be
done using sklearn’s train_test_split(X, y, test_size) function. This function takes in your features (X), the target variable (y), and
the test_size you’d like (Generally a test size of around 0.3 is good enough). It will then return a tuple of X_train, X_test, y_train,
y_test sets for us. We will train our model on the training set and then use the test set to evaluate the model.
We’ll now import sklearn’s DecisionTreeClassifier model and begin training it using the fit(train_data, train_data_labels) method.
In a nutshell, fitting is equal to training. Then, after it is trained, the model can be used to make predictions, usually with a
predict(test_data) method call.
DecisionTreeClassifier(max_depth=3,
random_state=0)
Model Evaluation
Now that we’ve finished training, we can make predictions off of the test data and evaluate our model’s performance using the
corresponding test data labels (y_test).
we’ll import sklearn’s accuracy_score to evaluate our model. This will take the true values and predictions as input.
Model
accuracy
score
with
criterion
gini
index:
0.7803
Let’s also compare the train‑set and test‑set accuracy and check for overfitting.
Training
set
score:
0.7965
Test
set
score:
0.7803
Here, the training‑set accuracy score is 0.7965 while the test‑set accuracy is 0.7803. These two values are quite comparable, so
there is no sign of overfitting.
Visualize decision‑trees
Note: Try running the 3 lines of code that are commented out. If the arrows don’t appear then you’ll have to run the uncommented
code to manually fix the arrows. This is a jupyter notebook issue some people face when using sklearn’s tree visualizer.
Awesome! As a bonus exercise, try creating a Decision Tree Classifier with criterion entropy instead.
Random Forests
Now let’s compare the decision tree model to a random forest. This is fairly quick to do using sklearn.
RandomForestClassifier()
Model
accuracy
score:
0.9653
Much stronger performance! Why do you think the random forest performed better? Refer back to lecture powerpoints if you’re
In
[1]:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
In
[2]:
from matplotlib import rcParams
rcParams[‘figure.figsize’] = 15, 5
sns.set_style(‘darkgrid’)
In
[3]:
car_data = pd.read_csv(‘car_evaluation.csv’, header=None)
car_data.columns = [‘buying’, ‘maint’, ‘doors’, ‘persons’, ‘lug_boot’, ‘safety’, ‘class’]
car_data.head()
Out[3]: In
[4]:
car_data.info()
In
[5]:
fig, axes = plt.subplots(nrows=3, ncols=2, sharey=True, figsize=(14, 10))
sns.countplot(x=car_data[‘buying’], ax=axes[0][0])
sns.countplot(x=car_data[‘maint’], ax=axes[0][1])
sns.countplot(x=car_data[‘doors’], ax=axes[1][0])
sns.countplot(x=car_data[‘persons’], ax=axes[1][1])
sns.countplot(x=car_data[‘lug_boot’], ax=axes[2][0])
sns.countplot(x=car_data[‘safety’], ax=axes[2][1])
plt.show()
In
[6]: sns.countplot(x=car_data[‘class’])
plt.show()
car_data[‘class’].value_counts()
Out[6]: In
[7]:
car_data.head()
Out[7]: In
[8]:
buying_mappings = {‘low’:0, ‘med’:1, ‘high’:2, ‘vhigh’:3}
maint_mappings = {‘low’:0, ‘med’:1, ‘high’:2, ‘vhigh’:3}
door_mappings = {‘2’:2, ‘3’:3, ‘4’:4, ‘5more’:5}
persons_mappings = {‘2’:2, ‘4’:4, ‘more’:5}
lug_boot_mappings = {‘small’:0, ‘med’:1, ‘big’:2}
safety_mappings = {‘low’:0, ‘med’:1, ‘high’:2}
class_mappings = {‘unacc’:0, ‘acc’:1, ‘good’:2, ‘vgood’:3}
In
[9]:
car_data[‘buying’] = car_data[‘buying’].map(buying_mappings)
car_data[‘maint’] = car_data[‘maint’].map(maint_mappings)
car_data[‘doors’] = car_data[‘doors’].map(door_mappings)
car_data[‘persons’] = car_data[‘persons’].map(persons_mappings)
car_data[‘lug_boot’] = car_data[‘lug_boot’].map(lug_boot_mappings)
car_data[‘safety’] = car_data[‘safety’].map(safety_mappings)
car_data[‘class’] = car_data[‘class’].map(class_mappings)
In
[10]:
car_data.head()
Out[10]: In
[11]:
from sklearn.model_selection import train_test_split
X = car_data[[‘buying’, ‘maint’, ‘doors’, ‘persons’, ‘lug_boot’, ‘safety’]]
y = car_data[‘class’]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)
In
[12]:
from sklearn.tree import DecisionTreeClassifier
#
instantiate
the
DecisionTreeClassifier
model
with
criterion
gini
index
clf_gini = DecisionTreeClassifier(criterion=’gini’, max_depth=3, random_state=0)
#
fit
the
model
clf_gini.fit(X_train, y_train)
Out[12]: In
[13]:
#
predict
the
test
set
results
with
criterion
gini
index
y_pred_gini = clf_gini.predict(X_test)
In
[14]:
from sklearn.metrics import accuracy_score
print(‘Model
accuracy
score
with
criterion
gini
index:
{0:0.4f}’.format(accuracy_score(y_test, y_pred_gini))
In
[15]:
y_pred_train_gini = clf_gini.predict(X_train)
print(‘Training
set
score:
{:.4f}’.format(clf_gini.score(X_train, y_train)))
print(‘Test
set
score:
{:.4f}’.format(clf_gini.score(X_test, y_test)))
In
[23]:
#
from
sklearn
import
tree
#
tree.plot_tree﴾clf_gini﴿
#
plt.show﴾﴿
fig, ax = plt.subplots(figsize=(10,10))
out = tree.plot_tree(clf_gini, filled=True, rounded=True,
feature_names=[‘buying’, ‘maint’, ‘doors’, ‘persons’, ‘lug_boot’, ‘safety’])
for o in out:
arrow = o.arrow_patch
if arrow is not None:
arrow.set_edgecolor(‘black’)
arrow.set_linewidth(3)
In
[24]:
from sklearn.ensemble import RandomForestClassifier
rfc = RandomForestClassifier(n_estimators=100)
rfc.fit(X_train, y_train)
Out[24]: In
[28]:
rfc_pred = rfc.predict(X_test)
print(‘Model
accuracy
score:
{0:0.4f}’.format(accuracy_score(y_test, rfc_pred)))
unsure.
Great job! You now know how to use decision tree and random forest in sklearn. Try using decision trees and or random forest on
your own dataset and refer back to this lecture if you get stuck.
COSC 3337
