Description
ECE-GY 6143 Homework 5
1. (10 points) Consider a one-hidden layer neural network (without biases for simplicity) with
sigmoid activations trained with squared-error loss. Draw the computational graph and derive
the forward and backward passes used in backpropagation for this network.
yˆ = W2σ(W1x), L = kyˆ − yk
2
2
Qualitatively compare the computational complexity of the forward and backward passes.
Which pass is more expensive and by roughly how much?
2. (10 points) Suppose that a convolutional layer of a neural network has an input tensor X[i, j, k]
and computes an output as follows:
Z[i, j, m] = X
k1
X
k2
X
n
W[k1, k2, n, m]X[i + k1, j + k2, n] + b[m]
Y [i, j, m] = ReLU(Z[i, j, m])
for some kernel W and bias b.
Suppose X and W have shapes (48, 64, 10) and (3, 3, 10, 20)
respectively.
a. What are the shapes of Z and Y ?
b. What are the number of input and output channels?
c. How many multiply- and add- operations are required to perform a forward pass through
this layer? Rough calculations are OK.
d. What are the total number of trainable parameters in this layer?
3. (10 points) The use of `2 regularization for training multi-layer neural networks has a special
name: weight decay. Assume an arbitrary dataset {(xi
, yi)}
n
i=1 and a loss function L(w) where
w represents the trainable weights (and biases).
a. Write down the `2 regularized loss, using a weighting parameter λ for the regularizer.
b. Derive the gradient descent update rules for this loss.
c. Conclude that in each update, the weights are “shrunk” or “decayed” by a multiplicative
factor before applying the descent update.
d. What does increasing λ achieve algorithmically, and how should the learning rate be
chosen to make the updates stable?
4. (20 points) In this exercise, we will fine-tune a pre-trained deep network (ResNet34) for a
particular two-class dataset which can be downloaded from the attached zip file. Code for
pre-processing the dataset, and for loading ResNet34, can be found below. Since ResNet34
is for 1000 output classes, you will need to modify the last layer to reduce two classes. Train
for 20 epochs and plot train and validation loss curves. Report your final train and validation
accuracies.
import torchvision
from torchvision import datasets, models, transforms
transforms = transforms.Compose([
transforms.Resize(256),
transforms.CenterCrop(224),
transforms.ToTensor(),
transforms.Normalize([0.485, 0.456, 0.406],
[0.229, 0.224, 0.225])
])
train_set = datasets.ImageFolder(“data/train”,transforms)
val_set = datasets.ImageFolder(“data/val”,transforms)
model = models.resnet34(pretrained=True)
2
