Description
Regression, Probability Theory, and Optimization
Questions:
1. [60 marks, 30 marks for Grads] This Assignment requires you to write a Python script file called sol1.py.
You are not allowed to use any Python public libraries related to regression and metrics. You can compare the
results of your program sklearn, numpy, or scipy linear models, but the whole exercise is to write the
algorithm yourself.
1 This dataset contains house sale prices for King County, which includes Seattle. It includes homes sold between May 2014
and May 2015.
1.1. [5 marks, 3 marks for Grads] Write a Python script file called sol1_1.py to load the House sales
dataset from the houses.csv file and place them in a data-frame df. (Hint: You can use
pandas.read_csv function. New in pandas? Click here ). Then generate and show various statistic
summary using pandas.DataFrame.describe method. What is pandas dataframe? Using
pandas.DataFrame methods split the dataset into target value Y (price) and feature matrix X (all
feature columns). In addition, extract the sqft_living column into a feature vector name X_1.
1.2. [15 marks, 7 marks for Grads] Write a function named linear_regression to implement Linear
Regression without using public libraries related to regression. The inputs of this function should be
predictor values (X or X_1), a target value (Y), a learning rate (lr), and the number of iterations
(repetition). The function must build a linear model using gradient descent and output the model
(params) and loss values per iteration (loss). Set the iteration to 10000 and calculate and show the
mean squared error (MSE) for the models obtained from both X and X_1 predictors (hint: you might
write another function named predict to predict the values based on X or X_1 and params) and plot
the learning curve (loss) for both models in one figure (hint: use log scaling plot). Try different
learning rates (10, 1, 0.1, 0.01, and 0.001) and compare and show the results.
1.3. [10 marks, 5 marks for Grads] Visualize the best-obtained model for X_1 using a scatter plot to show
price vs area and plot the linear model. Then, visualize the best-obtained model for all features (X) using
a scatter plot to show the predicted vs actual target values. Does the scatter plot create a linear line?
Why?
1.4. [10 marks, 5 marks for Grads] Modify the linear_regression function in a way that applies
Ridge regression and LASSO, and name them linear_regression_Ridge and
linear_regression_LASSO respectively. Then repeat the assignments 1.2 and 1.3. You can
thereby use a fixed learning rate that you find appropriate, but you should try different values for the
regularization penalty alpha.
1.5. [10 marks, 5 marks for Grads] Use linear_regression_LASSO and write a function named
linear_regression_LASSO_momentum in which you add a momentum term. Try different
momenta (0.1, 0.5, 0.7, and 0.9) and plot the learning curves with and without momentum for a fixed
learning rate.
1.6. [10 marks, 5 marks for Grads] Modify the linear_regression_LASSO_momentum function in
a way that it fits the feature vector X_1 with a polynomial of order 2 and name it as
polynomial_regression_optimized. Calculate the MSE, plot the learning curve and show
the quadratic model on the scatter plot (price vs area).
2. [40 marks, 25 marks for Grads] This Assignment requires you to write a Python script file called sol2.py.
2.1. [10 marks, 5 marks for Grads] Write a program that rolls two dice (randomly selects a number
between 1 and 6 inclusive for each die). Repeat rolling dice 20 times. For each trial, you should add the
two numbers that appear on each die and save it in a vector. Plot a histogram of the values you have
gathered in the vector. Based on this histogram, what are the estimates for the probabilities of each
number?
2.2. [10 marks, 5 marks for Grads] Run your code for 1000 times. What is your estimation now?
2.3. [20 marks, 10 marks for Grads] Change your program and assume that you have one fake die and one
correct die. The fake die has 0 instead of 2. This means when you roll this defect die, a random number
should be chosen among (1,0,3,4,5, and 6). Plot histogram of the values you have gathered in the vector.
Calculate the probability of getting a 7 as the sum of the numbers appeared on the dice. What is the
probability of getting 3?
2.4. Graduate students only [5 marks]: Try the same problem with seven dice and plot the distribution
together with a Gaussian fit. Report the mean and variance.
3. Graduate students only [25 marks]: (From Thrun, Burgard and Fox, Probabilistic Robotics) A robot uses a
sensor that can measure ranges from 0m to 3m. For simplicity, assume that the actual ranges are distributed
uniformly in this interval. Unfortunately, the sensors can be faulty. When the sensor is faulty it constantly
outputs a range below 1m, regardless of the actual range in the sensor’s measurement cone. We know that the
prior probability for a sensor to be faulty is p = 0.01. Suppose the robot queries its sensors N times, and every
single time the measurement value is below 1m. What is the posterior probability of a sensor fault, for N = 1,
2, …, 10? Formulate the corresponding probabilistic model.
4. Graduate students only [20 marks]: Theoretical limit of Gaussian classification example. We followed an
example of classifications with two Gaussian classes in section 2.4. In this assignment, you should calculate
analytically the theoretical limit of the optimal accuracy for the parameters used in the printed program.
Provide your answer with a brief outline of the calculation.
