Description
1 Objectives
The objectives of this programming assignment are:
• To practise some data importing and preprocessing skills by using the pandas library.
• To gain a better understanding of supervised learning methods by using scikit-learn.
• To evaluate the performance of several supervised learning methods by conducting empirical study on a real-world dataset.
2 Dataset
You will use a water quality dataset provided as a ZIP file (data.zip). There are two csv data
files in it. The table below summarizes the attributes of the data in each csv file:
File # of records Has label? # of columns
train.csv 7,496 yes 21
test.csv 800 yes 21
The first 20 columns represent the features and the last column named ‘is safe’ indicates whether
or not the water is safe.
3 Major Tasks
The assignment consists of four parts and a written report:
Part 1: Use pandas for data importing and preprocessing.
Part 2: Use the linear regression model and feedforward neural networks for regression.
Part 3: Use the logistic regression model and feedforward neural networks for classification.
Part 4: Use scikit-learn to tune the hyperparameters.
Written Report: Report the results and answer some questions.
More details will be provided in the following sections. Note that [Qn] refers to a specific
question (the nth question) that you need to answer in the written report. All the coding work
should be done using Python 3.
1
4 Part 1: Data Preprocessing
In this part, you are required to preprocess the data and visualize the basic properties of the
dataset. To be specific, if applicable, you need to remove the duplicates and fill in the missing
values with their mean value. Note that we are not supposed to have the test data ready when
performing data preprocessing on the training data. In the later questions, you are required to
split the data in train.csv into training and validation sets. You need to perform data preprocessing first before splitting the data. After you have finished handling the above cases, visualize
the correlation between every two of all the features in the training data with a heatmap.
[Q1] Visualize the correlation between every two of all the features with a heatmap.
5 Part 2: Regression
5.1 Linear Regression
In this task, you will build six linear regression models in the first step, where each model
uses only one feature to find whether it is correlated with the attribute ‘chloramine’. The six
features, which should not include ‘is safe’, are the ones of highest correlation to the attribute
‘chloramine’. After this step, you will obtain six models for the plotting part of [Q3]. Then,
in the second step, you will build another linear regression model to explore the relationship
between linear combinations of the six selected features and ‘chloramine’.
You are required to use the train test split submodule in scikit-learn to split the data in
train.csv, with 80% for training and 20% for validation. You should set random state = 4211
for reproducibility.
[Q2] After training the seven models described above using the training set, use them to make
predictions on the validation set. Report the validation R2
score of each of the models to evaluate
the relationship between different features and ‘chloramine’.
[Q3] Plot the regression line and the data points of the validation set for each of the first
six models. For illustration, the figure below shows a plot of ‘chloramine’ versus the feature
‘perchlorate’ as well as the regression line.
[Q4] Report the mean squared error of each of the seven models in the validation set and compare
them based on the two performance metrics, i.e., R2
score and mean squared error.
[Q5] Build a linear regression model that uses the attribute (from among the first 20 columns
2
of the dataset, except the attribute ‘chloramine’) that has the least correlation to the attribute
‘chloramine’. Plot the regression line and the data points of the validation set similar to [Q3].
Discuss the difference of the figures plotted in [Q3] and [Q5] that have the attributes of highest
and lowest correlations, respectively.
5.2 Feedforward Neural Networks
In this part, you are asked to use the six features selected in Section 5.1 to train feedforward
neural networks.
You need to try different number of hidden units H ∈ {1, 4, 16, 64, 128} to build different threehidden-layer neural networks. The hyperparameter early stopping can be set to ‘True’ to
avoid overfitting (default is ‘False’). The other hyperparameters may just take their default
values. For each hidden layer in a specific neural network model, the number of hidden units
should be kept the same for simplicity. During training, you are expected to record the training
time of each model. After training, evaluate your models by reporting the R2
scores on the
validation set. You have to report the R2
score for each value of H by plotting them using
matplotlib.
[Q6] Report the model setting, training time, and performance of the neural network model
for each value of H. You are also expected to repeat each setting three times for the same
hyperparameter setting and report the mean and standard deviation of the training time and
R2
score for each setting.
[Q7] Compare the training time and R2
score of the linear regression model and the best neural
network model.
6 Part 3: Classification
In this task, you will build a logistic regression model as well as neural network classifiers to
predict whether or not the water quality is safe.
You are also required to use the train test split submodule in scikit-learn to split the data,
with 80% for training and 20% for validation. As before, we ask that you set random state =
4211 for reproducibility.
6.1 Feature Selection
To reduce the computational cost and remove the irrelevant features, we would like to choose a
subset of features for classification. You can use the feature selection module in scikit-learn.
Use the mutual information statistics as the score for feature selection and then drop the five
least important features.
[Q8] Report the score for each of the features.
6.2 Logistic Regression
Learning of the logistic regression model should use a gradient-descent algorithm by minimizing
the cross-entropy loss. It requires that the step size parameter η be specified. Try out a few values
(<1) and choose one that leads to stable convergence. You may also decrease η gradually during
the learning process to enhance convergence. This can be done automatically in scikit-learn
when set properly.
3
Use the features selected in Section 6.1 to train the model. During training, record the training
time for the logistic regression model. After training, you are required to evaluate your model
using accuracy and the F1 score on the validation set.
[Q9] Report the model setting, training time, and performance of the logistic regression model.
Since the solution found may depend on the initial weight values, you are expected to repeat each
setting three times and report the corresponding mean and standard deviation of the training
time, accuracy, and the F1 score for each setting.
[Q10] Calculate the confusion matrix on the validation set with the model in [Q9] and report
them. Give one reason why we need to examine the confusion matrix as well.
[Q11] Train the model on the training set and report the accuracy and F1 score on the validation set using different learning rate scheduling types (including ‘sag’, ‘lbfgs’, and ‘sgd’).
The random state hyperparameter is again set to 4211. Other settings are the same as those
mentioned in [Q9].
6.3 Feedforward Neural Networks
Neural network classifiers generalize logistic regression by introducing one or more hidden layers.
The learning algorithm for them is similar to that for logistic regression as described above.
Remember to standardize the features before training and validation.
You need to try different number of hidden units H ∈ {1, 4, 16, 64, 128} to build different threehidden-layer neural networks. The hyperparameter early stopping can be set to ‘True’ to avoid
overfitting (default is ‘False’). The other hyperparameters may just take their default values.
During training, you are expected to record the training time of each model. After training,
evaluate your models using accuracy and the F1 score on the validation set. You have to report
the accuracy and F1 score for each value of H by plotting them using matplotlib.
[Q12] Report the model setting, training time, and performance of the neural networks for each
value of H. You are also expected to repeat each setting three times and report the mean and
standard deviation of the training time, accuracy, and the F1 score for each setting.
[Q13] Plot the accuracy and F1 score for each value of H. Suggest a possible reason for the gap
between the accuracy and F1 score.
[Q14] Compare the training time, accuracy and the F1 score of the logistic regression model
and the best neural network model.
[Q15] Do you notice any trend when you increase the hidden layer size from 1 to 128? If so,
please describe what the trend is and suggest a reason for your observation.
7 Part 4: Performance Enhancement
7.1 Hyperparameter Tuning
In this task, you need to use grid search to tune the hyperparameters of a three-hidden-layer
feedforward neural network model to predict whether or not the water quality is safe. For your
reference, all the hyperparameters defined in the MLPClassifier class in scikit-learn except
the number of hidden layers can be tuned. Use the features selected in Section 6.1 for training
and testing.
4
This time, you are required to evaluate your model on the test set (test.csv) provided. You
need to import test.csv as a dataframe and standardize the features using the statistics you
used for the training data. We assume that the test set comes from the same distribution as the
training set.
You are required to use the model selection submodule in scikit-learn to facilitate performing grid search cross validation for hyperparameter tuning. This is done by randomly sampling
80% of the training instances to train a classifier and then validating it on the remaining 20%.
Five such random data splits are performed and the average over these five trials is used to
estimate the generalization performance. You are expected to try at least 10 combinations of
the hyperparameter setting. Set the random state hyperparameter of the neural network model
to 4211 for reproducibility and early stopping to ‘True’ to avoid overfitting.
[Q16] Report six combinations of the hyperparameter setting.
[Q17] Report the three best hyperparameter settings in terms of accuracy as well as the mean
and standard deviation of the validation accuracy of the five random data splits for each hyperparameter setting.
[Q18] Use the best model in terms of accuracy to predict the instances in the test set. Report
the accuracy, F1 score and visualize the confusion matrix of the predictions on the test set.
7.2 Oversampling
By counting the number of records for different labels, you will notice that the training set is
imbalanced. In this task, you will apply the oversampling strategy to tackle this problem. To be
specific, you have to randomly oversample from the minority class and add these examples to the
training set so that the two classes will become balanced in size. Set the random state hyperparameter of the neural networks and oversampling to 4211 for reproducibility and early stopping
to ‘True’ to avoid overfitting. Another method to tackle this problem is to assign different weights
to different classes. This can be done in scikit-learn.
[Q19] Train two logistic regression models using the two methods above (i.e., oversampling and
different weights) to predict the instances in the test set (test.csv). Report the accuracy, F1
score and the confusion matrix of the predictions on the test set.
[Q20] Compare the accuracy, F1 score and the confusion matrix with those in Section 7.1 and
discuss your findings.
8 Bonus: Comparison with Ensemble Methods (Optional)
The questions in this part are optional and they will not be counted towards your grade for this
assignment. As mentioned in class, students who do reasonably well for the bonus questions will
be entitled for one day late in the submission of problem set or project later.
In the previous questions, you solved the classification problem using two different methods,
logistic regression and feedforward neural networks. In this part, you will need to compare
them with ensemble methods. The goal of ensemble methods is to combine the predictions of
several base estimators built with a given learning algorithm in order to improve generalizability
and robustness over a single estimator. There are two families of ensemble methods: averaging
methods and boosting methods. In averaging methods, the driving principle is to build sev5
eral estimators independently and then to average their predictions. By contrast, in boosting
methods, base estimators are built sequentially and one tries to reduce the bias of the combined
estimator. You can use the ensemble module in scikit-learn.
[Q21] Try to use the bagging classifier which is one of the averaging methods for the classification
task. You can use logistic regression models or feedforward neural networks as the estimators.
You also need to report the training time and the accuracy and F1 score on the test set. You are
expected to repeat each setting three times and report the corresponding mean and standard
deviation of the training time, accuracy, and the F1 score for each setting.
[Q22] Try to use the gradient boosting classifier which is one of the boosting methods for the
classification task. You also need to report the training time and the accuracy and F1 score on
the test set.
[Q23] Compare the performance of the two ensemble models with logistic regression and feedforward neural networks.
9 Report Writing
Answer [Q1] to [Q20] ([Q1] to [Q23] if you do the bonus part as well) in the report.
10 Some Programming Tips
As is always the case, good programming practices should be applied when coding your program.
Below are some common ones but they are by no means complete:
• Using functions to structure your code clearly
• Using meaningful variable and function names to improve readability
• Using consistent styles
• Including concise but informative comments
You are recommended to take full advantage of the built-in classes of scikit-learn to keep
your code both short and efficient. Proper use of implementation tricks often leads to speedup
by orders of magnitude. Also, please be careful to choose the built-in models that are suitable
for your tasks, e.g., sklearn.linear model.LogisticRegression is not a correct choice for
our logistic regression model since it does not use gradient descent.
11 Assignment Submission
Assignment submission should only be done electronically on the Canvas course site.
There should be two files in your submission with the following naming convention required:
1. Report (with filename report.pdf): in PDF form.
2. Source code (with filename code.zip): all necessary code, written in one or more Jupyter
notebooks, compressed into a single ZIP file. The data should not be submitted to keep
the file size small.
When multiple versions with the same filename are submitted, only the latest version according
to the timestamp will be used for grading. Files not adhering to the naming convention above
6
will be ignored.
12 Grading Scheme
This programming assignment will be counted towards 10% of your final course grade. Note
that the plus sign (+) in the last column of the table below indicates that reporting without
providing the corresponding code will get zero point. The maximum scores for different tasks
are shown below:
7
Grading scheme Code
(60)
Report
(+40)
Part 1
- [Q1] 2 +1
Part 2
- Build the linear regression model 3
- Compute the R2
scores of the seven linear regression models + [Q2] 2 +2
- Make prediction on the validation set + [Q3] 3 +2
- Compute the MSE scores of the seven linear regression models + [Q4] 2 +2
- Build the model and make prediction on the validation set + [Q5] 2 +1
- Build the feedforward neural network model 2
- Compute the training time and R2
score for each value of H in the
feedforward neural network model + [Q6] 3 +2
- Plot the R2
score with different values of H for the feedforward
neural network model + [Q7] 3 +2
Part 3
- Select the features according to mutual information + [Q8] 3 +2
- Build the logistic regression model by adopting the gradient descent
optimization algorithm 2
- Compute the training time, accuracy, and F1 score of the logistic
regression model + [Q9] 3 +2
- Calculate the confusion matrix on the validation set + [Q10] 3 +2
- Logistic regression with different learning rate scheduling types + [Q11] 3 +2
- Build the feedforward neural network model 2
- Compute the training time, accuracy, and F1 score for each value of H
in the feedforward neural network model + [Q12] 3 +3
- Plot the accuracy and F1 score with different values of H for the
feedforward neural network model + [Q13] 3 +3
- [Q14] +2
- [Q15] +2
Part 4
- Grid search on the feedforward neural network model for at least
six combinations + [Q16] 4 +2
- Report the three best hyperparameter settings and the validation
accuracy (both mean and standard deviation) for each setting + [Q17] 4 +2
- Report the accuracy and F1 score on the test set and visualize the
confusion matrix + [Q18] 4 +2
- Use the two methods to tackle imbalanced dataset + [Q20] 4 +2
- [Q21] +2
Bonus
- Build the averaging model + [Q21]
- Build the boosting model + [Q22]
- [Q23]
Late submission will be accepted but with penalty.
The late penalty is deduction of one point (out of a maximum of 100 points) for every minute
late after 11:59pm. Being late for a fraction of a minute is considered a full minute. For example,
8
two points will be deducted if the submission time is 00:00:34.
13 Academic Integrity
Please refer to the regulations for student conduct and academic integrity on this webpage:
https://registry.hkust.edu.hk/resource-library/academic-standards.
While you may discuss with your classmates on general ideas about the assignment, your submission should be based on your own independent effort. In case you seek help from any person
or reference source, you should state it clearly in your submission. Failure to do so is considered
plagiarism which will lead to appropriate disciplinary actions.
9
