## Description

## 1 Written Problems (6 points)

1. MLE minimizes KL divergence to the empirical distribution (Exercise

2.15 of Murphy’s book) (1 point)

2. Centering and ridge regression (Exercise 7.3 of Murphy’s book) (1

point)

3. Symmetric version of `2 regularized multinomial logistic regression (Exercise 8.5 of Murphy’s book) (1 point)

4. Elementary properties of `2 regularized logistic regression (Exercise 8.6

of Murphy’s book) (1 point)

5. Given the following denominator layout derivatives, (2 points)

• Differentiation of a scalar function w.r.t. a vector: If f(w) is a

scalar function of d variables, w is a d×1 vector, then differentiation

of f(w) w.r.t. w results in a d × 1 vector

df(w)

dw

=

∂f

∂w1

.

.

.

∂f

∂wd

• Differentiation of a vector function w.r.t. a vector: If f(w) is a

vector function of size h×1 and w is a d×1 vector, then differentiation

of f(w) w.r.t. w results in a d × h vector

df(w)

dw

=

∂f1

∂x1

. . .

∂fh

∂w1

.

.

.

.

.

.

.

.

.

∂f1

∂wd

. . .

∂fh

∂wd

1

Please prove the following derivatives, and X and y are not functions of

w:

d(X>w)

dw

= X,

d(y

>Xw)

dw

= X>y

d(w>Xw)

dw

= (X + X>)w

## 2 Programming (5 points)

Given Xw = y where X =

1 2

4 3

5 6

3 8

9 10

and y =

−1

0

1

0

0

which constitute an exemplary problem. Write a Python routine to find the least squares solution

for w given arbitrary X ∈ R5×2 and y ∈ R5×1

. Submit your Python codes

as a function routine (“def A1 MatricNumber(X,y)”) that takes in X and y as

inputs and generate (X>X)

−1 and w as outputs in a single file with filename

“A1 StudentMatriculationNumber.py”. Your Python routine should return the

least squares solution vector w (as numpy array) and two matrices X> and

(X>X)

−1

. Hint: you will need “import numpy as np” and its matrix manipulation functions.

Please use the python template provided to you. Remember to rename both

“A1 StudentMatriculationNumber.py” and “A1 MatricNumber” using your student matriculation number. For example, if your matriculation ID is 123456789,

then you should submit “A1 123456789.py” that contains the function “A1 123456789”.

The way we would run your code might be something like this:

import A1 123456789 as grading

w, XT, InvXTX = grading.A1 123456789(X,y)

Marks allocation is based on the three outputs: w (3 points), XT (1 point),

InvXTX (1 point)

NOTE: Please do NOT zip/compress your file. Please make sure you replace

“StudentMatriculationNumber” and “MatricNumber” with your matriculation

number! Because of the large class size, points will be deducted if instructions are not followed.

2