Description
Convolutional Neural Networks
1. Let X and W be arrays,
X =
0 0 0 0 0
0 3 3 3 0
0 3 3 3 0
0 3 2 3 0
0 3 2 3 0
0 0 0 0 0
, W =
1 −1
1 −1
.
Let Z be the 2D convolution (without reversal):
Z[i, j] = X
k1,k2
W[k1, k2]X[i + k1, j + k2]. (1)
Assume that the arrays are indexed starting at (0, 0).
(a) What are the limits of the summations over k1 and k2 in (1)?
(b) What is the size of the output Z[i, j] if the convolution is computed only on the valid
pixels (i.e. the pixel locations (i, j) where the summation in (1) does not exceed the
boundaries of W or X).
(c) What is the largest positive value of Z[i, j] and state one pixel location (i, j) where that
value occurs.
(d) What is the largest negative value of Z[i, j] and state one pixel location (i, j) where that
value occurs.
(e) Find one pixel location where Z[i, j] = 0.
2. Suppose that a convolutional layer of a neural network has an input tensor X[i, j, k] and
computes an output via a convolution and ReLU activation,
Z[i, j, m] = X
k1
X
k2
X
n
W[k1, k2, n, m]X[i + k1, j + k2, n] + b[m],
U[i, j, m] = max{0, Z[i, j, m]}.
for some weight kernel W[k1, k2, n, m] and bias b[m]. Suppose that X has shape (48,64,10)
and W has shape (3,3,10,20). Assume the convolution is computed on the valid pixels.
1
(a) What are the shapes of Z and U?
(b) What are the number of input channels and output channels?
(c) How many multiplications must be performed to compute the convolution in that layer?
(d) If W and b are to be learned, what are the total number of trainable parameters in the
layer?
3. Suppose that a convolutional layer in some neural network is described as a linear convolution
followed by a sigmoid activation,
Z[i, j, m] = X
k1
X
k2
X
n
W[k1, k2, n, m]X[i + k1, j + k2, n] + b[m],
U[i, j, m] = 1/(1 + exp(−Z[i, j, m])).
where X[i, j, n] is the input of the layer and U[i, j, m] is the output. Suppose that during
back-propagation, we have computed the gradient ∂J/∂U for some loss function J. That is,
we have computed the components ∂J/∂U[i, j, m].
Show how to compute the following:
(a) The gradient components ∂J/∂Z[i, j, m].
(b) The gradient components ∂J/∂W[k1, k2, n, m].
(c) The gradient components ∂J/∂X[i, j, n].
4. In the previous problem, we considered a single sample. Suppose there were a mini-batch of
samples.
(a) How would you represent Z and U for the mini-batch case?
(b) Re-write the equations for Z and U.
(c) Re-compute the gradients.
2
