Description
1. (15 marks) Ridge regression v.s. Least squares Given data (yi , xi)i=1,…,n, yi = β0+β1xi+i , where i i.i.d ∼ N(0, σ2 ), (xi)i=1,…,n is known and fixed. Least square estimate (βˆLS 0 , βˆLS 1 ) = argmin β0,β1 Pn i=1(yi − β0 − β1xi) 2 . Ridge regression (βˆRidge 0 , βˆRidge 1 ) = argmin β0,β1 Pn i=1(yi − β0 − β1xi) 2 + λβ2 1 . (a) Show that the least square estimate is unbiased by showing E(βˆLS 0 ) = β0 and E(βˆLS 1 ) = β1. (b) Show that the ridge regression estimate is biased by calculating E(βˆRidge 0 ) and E(βˆRidge 1 ). Hint: You may directly use some derivations in the lecture. 2. (20 marks) Invariant of linear regression to scaling, but not ridge regression without standardization (a) Consider the following data set: y = (2.2, 3.3, 3.8), x = (1, 2, 3). Fit y = β0 + β1x. 1 i. Calculate least square parameter estimates βˆLS 0 , βˆLS 1 and ridge regression parameter estimates βˆRidge 0 , βˆRidge 1 with λ = 1. ii. Calculate ˆy LS and ˆy Ridge for x. (b) Consider the data set in (a) with x 0 = 10x, i.e. x 0 = (10, 20, 30). i. Calculate least square parameter estimates βˆL 0 , βˆL 1 and ridge regression parameter estimates βˆR 0 , βˆR 0 with λ = 1. ii. Compare βˆR 0 and βˆR 1 in 2b(i) with βˆRidge 0 and βˆRidge 1 10 in 2a(i) which are without scaling, also compare βˆL 0 and βˆL 1 2b(i) with βˆLS 0 and βˆLS 1 10 in 2a(i) for least square. iii. Calculate ˆy L and ˆy R for x 0 , and compare with 2a(ii). (Note: You will see that scaling x will have an effect on yˆ for ridge regression, but not in least square) Hint: You may directly use some derivations in the lecture. 3. (15 marks) Cyclic coordinate descent for LASSO Given f(βj ) = aβ2 j − 2bβj + λ|βj |, where a > 0, λ > 0. Show that when b < − λ 2 < 0, βˆ j = 2b+λ 2a minimizes f(βj ). 4. (35 marks) Variance and bias for Linear regression v.s. Ridge regression Fit the data with model yi = β0 + β1xi + i , where i i.i.d ∼ N(0, σ2 ). Least square parameter estimtates: βˆLS 0 = ¯y − βˆLS 1 x¯, and βˆLS 1 = Pn i=1 P (xi−x¯)yi n i=1(xi−x¯) 2 Ridge regression parameter estimtates: βˆRidge 0 = ¯y−βˆRidge 1 x¯, and βˆRidge 1 = Pn i=1 P (xi−x¯)yi n i=1(xi−x¯) 2+λ For a new point x0, calculate bias and variance for (a) Linear regression (b) Ridge regression Where bias2 = [β0 + β1×0 − E(ˆy0)]2 and variance = E[ˆy0 − E(ˆy0)]2 . (Note: You will see that compared with linear regression, the bias2 for ridge is larger, but the variance is smaller. At high dimensional setting, ridge regression (and lasso) will be better because of the smaller variance.) Hint: You may directly use some derivations in the lecture. V ar(A + B) = V ar(A) + V ar(B) + 2cov(A, B) V ar(αA) = α 2V ar(A), where α is a scalar. 5. (15 marks) R code exercise 2 (a) Use the rnorm() function to generate a predictor X of length n = 100, µX = 0, σX = 1, as well as a noise vector of length n = 100, µ = 0, σ = 0.1. (b) Generate a response vector Y of length n = 100 according to the model Y = 1 + X + X2 + X3 + . (c) Fit a lasso model to the simulated data, using X, X2 , …, X10 as predictors. Use cross-validation to select the optimal value of λ. Create plots of the crossvalidation error (i.e. Mean-Square error v.s. log(λ)) as a function of λ. Report the resulting coefficient estimates. (d) Now re-generate a response vector Y according to the new model Y = 1 + X7 +. Again, re-fit a lasso model using X, X2 , …, X10 as predictors. Use cross-validation to select the optimal value of λ. Create plots of the cross-validation error (i.e. Mean-Square error v.s. log(λ)) as a function of λ. Report the resulting coefficient estimates. (Note: You will see that when the true data-generating model is sparser, cross-validation tends to select a sparser model.) Hint: You may refer to the tutorial notes ‘Tutorial05’. – End – 3
