Description
Theme: Neural Networks & Backpropagation
Question 1 (Based on Example 7.1 from LFD)
Consider the following neural network.
1
x tanh Σ tanh
x
(2)
Σ
Σ tanh
1
s
(2) 1
0.2
−3
0.1
0.2
0.3
0.4
s
(1)
2
s
(1)
1
x
(1)
2
x
(1)
1
tanh
s
(3)
Σ
1
1
2 x
(3)
s
(1) x
(1)
Letting x = 2 and y = 1, we would like to calculate the gradients as in Example 7.1 from LFD.
Clearly show the intermediate steps in calculations (note these steps are not presented in Example
7.1) and perform one round of forward and backward propagation, and use the results to calculate
the gradients. Specifically, fill-in all the missing intermediate steps in the computation of
x
(1) = (x
(1)
1
, x
(1)
2
), s(2), x(2), s(3), x(3), δ(1), δ(2), δ(3)
,
∂e
∂W(2) ,
∂e
∂W(3) .
For any missing problem specifications/parameters, assume those provided in Example 7.1.
After you have worked out this problem, it may be useful to repeat the steps for Exercise 7.8,
where the final (output) tanh is replaced with the identity (note that this part is not considered for
grading).
1
Homework Problems – Tutorial #6
Question 2 (Problem 3
x
(0)
1
x
(0)
2 x
(0)
3
θ θ θ
Σ Σ
w w w
v v
x
(1)
1
yˆ = x
(2)
3
x
(1)
3
x
(1)
2
x
(2)
2
Given an input (x, y), where x = (x
(0)
1
, x
(0)
2
, x
(0)
3
) ∈ R
3
, the above network computes ˆy as shown.
More specifically, the intermediate computations are
x
(1)
1 = θ(w · x
(0)
1
)
x
(1)
2 = θ(w · x
(0)
2
)
x
(1)
3 = θ(w · x
(0)
3
)
x
(2)
2 = x
(1)
2 + v · x
(1)
1
x
(2)
3 = x
(1)
3 + v · x
(2)
2
yˆ = x
(2)
3
.
(1)
Note that v and w are shared weights on the horizontal edges and vertical edges as shown in the
figure. Assume that θ(·) is some arbitrary activation function with derivative denoted by θ
0
(·). For
the input (x, y) and model parameter Ω = (w, v) of the network, assume that the loss is given by
e(Ω) = (ˆy − y)
2
. (2)
(a) Find an expression for de
dv . Express your answer in terms of x
(1)
1
, v, x
(2)
2
and ∆ = ˆy − y.
(b) Find the expressions for de
dx(2)
2
,
de
dx(1)
1
,
de
dx(1)
2
and de
dx(1)
3
. Express your answer in the simplest
possible form (with as few variables as possible). (Hint: at-least one of your answer should
be in terms of ∆ and v.)
(c) Using parts (a) and (b) find an expression for de
dw .
