EE5904/ME5404 Neural Networks Homework 3 solved

Description

Q1. Function Approximation with RBFN (10 Marks)

Consider using RBFN to approximate the following function:
𝑦𝑦 = 1.2 sin(𝜋𝜋𝜋𝜋) − cos(2.4𝜋𝜋𝜋𝜋) , 𝑓𝑓𝑓𝑓𝑓𝑓 𝑥𝑥 ∈ [−1, 1]
The training set is constructed by dividing the range [−1, 1] using a uniform step
length 0.05, while the test set is constructed by dividing the range [−1, 1] using a
uniform step length 0.01. Assume that the observed outputs in the training set are
corrupted by random noise as follows.

𝑦𝑦(𝑖𝑖) = 1.2 sin�𝜋𝜋𝜋𝜋(𝑖𝑖)� − cos�2.4𝜋𝜋𝜋𝜋(𝑖𝑖)� + 0.3𝑛𝑛(𝑖𝑖)
where the random noise 𝑛𝑛(𝑖𝑖) is Gaussian noise with zero mean and stand deviation of
one, which can be generated by MATLAB command randn. Note that the test set is not
corrupted by noises. Perform the following computer experiments:
a) Use the exact interpolation method (as described on pages 16-21 in the slides of
lecture five) and determine the weights of the RBFN. Assume the RBF is Gaussian
function with standard deviation of 0.1. Evaluate the approximation performance of
the resulting RBFN using the test set.
(3 Marks)

b) Follow the strategy of “Fixed Centers Selected at Random” (as described on page 37
in the slides of lecture five), randomly select 20 centers among the sampling points.
Determine the weights of the RBFN. Evaluate the approximation performance of the
resulting RBFN using test set. Compare it to the result of part a).
(4 Marks)
c) Use the same centers and widths as those determined in part a) and apply the
regularization method as described on pages 42-45 in the slides for lecture five. Vary
the value of the regularization factor and study its effect on the performance of RBFN.
(3 Marks)
2 EE5904/ME5404

Q2. Handwritten Character Classification using RBFN (20 Marks)

In this task, you will build a handwritten character classifier using RBFN. The training
data is provided in characters10.mat which contains 3,500 grayscale images (of size
28×28) over 10 classes as listed below.
Character N E U R A L T W O K
Example
Label 0 1 2 3 4 5 6 7 8 9

Specifically, each class possesses 300 images for training and 50 images for test. Please
select two classes according to the last two different digits of your matric number (e.g.
A0642311, choose classes 3 and 1; A1234567, choose classes 6 and 7). Make sure you
have selected the correct two classes for both training and testing. There will be some
mark deduction for wrong selections. Please state the class of your selected handwritten
characters for both training and testing.

In MATLAB, the following code can be used to load the training and test data:

——————————————————————————————————-
After loading the data, you may view them using the code below:
——————————————————————————————————-
imshow(reshape(train_data(column_no, :), [28,28]));
——————————————————————————————————-
To select a few classes for training, you may refer to the following code:
——————————————————————————————————-
trainIdx = find(train_label==0 | train_label==1 | train_label==2); % select classes 0, 1, 2
trainY = train_label(trainIdx);
trainX = train_data(trainIdx,:);
——————————————————————————————————-

Please use the following code to evaluate:

——————————————————————————————————-
TrAcc = zeros(1,1000);
TeAcc = zeros(1,1000);
thr = zeros(1,1000);
TrN = length(TrLabel);
TeN = length(TeLabel);
for i = 1:1000
t = (max(TrPred)-min(TrPred)) * (i-1)/1000 + min(TrPred);
thr(i) = t;

3 EE5904/ME5404
TrAcc(i) = (sum(TrLabel(TrPred<t)==0) + sum(TrLabel(TrPred>=t)==1)) / TrN;
TeAcc(i) = (sum(TeLabel(TePred<t)==0) + sum(TeLabel(TePred>=t)==1)) / TeN;
end
plot(thr,TrAcc,’.- ‘,thr,TeAcc,’^-‘);legend(‘tr’,’te’);

——————————————————————————————————-
TrPred and TePred record respectively the predicted labels of training data and test
data; and are determined by TrPred(j) = ∑ 𝑤𝑤𝑖𝑖𝜑𝜑𝑖𝑖(TrData(j, : )) 𝑁𝑁
𝑖𝑖=0 and TePred(j) =
∑ 𝑤𝑤𝑖𝑖𝜑𝜑𝑖𝑖(TeData(j, : )) 𝑁𝑁
𝑖𝑖=0 where 𝑁𝑁 is the number of hidden neurons. TrData and
TeData are the training and test data selected based on your matric number. TrLabel
and TeLabel record respectively the ground-truth labels (convert to {0,1} before use!)
of training data and test data.

You are required to complete the following tasks:

a) Use the “Exact Interpolation” method (as described in pages 16-29 of lecture five)
and apply regularization (as described in pages 42-45 of lecture five). Assume the RBF
is Gaussian function with standard deviation of 100. Firstly, determine the weights of
RBFN without regularization and evaluate its performance; then vary the value of
regularization factor and study its effect on the resulting RBFNs’ performance.
(6 Marks)

b) Follow the strategy of “Fixed Centers Selected at Random” (as described in page 37
of lecture five). Randomly select 100 centers among the training samples. Firstly,
determine the weights of RBFN with widths fixed at an appropriate size and compare
its performance to the result of a); then vary the value of width from 0.1 to 10000 and
study its effect on the resulting RBFNs’ performance.
(8 Marks)

c) Try classical “K-Mean Clustering” (as described in pages 38-39 of lecture five) with
2 centers. Firstly, determine the weights of RBFN and evaluate its performance; then
visualize the obtained centers and compare them to the mean of training images of each
class. State your findings.
(6 Marks)
4 EE5904/ME5404

Q3. Self-Organizing Map (SOM) (20 Marks)

a) Write your own code to implement a SOM that maps a 1-dimensional output layer
of 36 neurons to a “sinusoid curve”. Display the trained weights of each output neuron
as points in a 2D plane, and plot lines to connect every topological adjacent neurons
(e.g. the 2nd neuron is connected to the 1st and 3rd neuron by lines). The training points
sampled from the “sinusoid curve” can be obtained by the following code:
——————————————————————————————————-
x = linspace(-pi,pi,400);
trainX = [x; 2*sin(x)];  2×400 matrix
plot(trainX(1,:),trainX(2,:),’+r’); axis equal
——————————————————————————————————-

(3 Marks)
b) Write your own code to implement a SOM that maps a 2-dimensional output layer
of 36 (i.e. 6×6) neurons to a “circle”. Display the trained weights of each output neuron
as a point in the 2D plane, and plot lines to connect every topological adjacent neurons
(e.g. neuron (2,2) is connected to neuron (1,2) (2,3) (3,2) (2,1) by lines). The training
points sampled from the “circle” can be obtained by the following code:
——————————————————————————————————-
X = randn(800,2);
s2 = sum(trainX.^2,2);
trainX = (X.*repmat(1*(gammainc(s2/2,1).^(1/2))./sqrt(s2),1,2))’;  2×800 matrix
plot(trainX(1,:),trainX(2,:),’+r’); axis equal
——————————————————————————————————-

(4 Marks)
c) Write your own code to implement a SOM that clusters and classifies handwritten
characters. The training data is provided in characters10.mat (as introduced in Q2).
Please omit two classes according to the last two different digits of your matric number
(e.g. A0642311, ignore classes 3 and 1; A1234567, ignore classes 6 and 7.) Make sure
you have selected the correct eight classes for both training and testing. There will be
some mark deduction for wrong classes selected. Please state the class of your selected
handwritten characters for both training and testing.

After loading the data, complete the following tasks:

c-1) Print out corresponding conceptual/semantic map of the trained SOM (as
described in page 24 of lecture six) and visualize the trained weights of each output
neuron on a 10×10 map (a simple way could be to reshape the weights of a neuron
into a 28×28 matrix and display it as an image). Make comments on them, if any.
(8 Marks)
c-2) Apply the trained SOM to classify the test images (in test_data). The
classification can be done in the following fashion: input a test image to SOM and
find out the winner neuron; then label the test image with the winner neuron’s label
(note: labels of all the output neurons have already been determined in c-1)).
5 EE5904/ME5404

Calculate the classification accuracy on the whole test set and discuss your
findings.
(5 Marks)
The recommended values of design parameters are:
1. The size of SOM is 1×36 for a), 6×6 for b), and 10×10 for c).
2. The total iteration number N is set to be 600 for a) & b) and 1000 for c). Only
the first phase (self-organizing) of learning is used in this experiment.
3. The learning rate 𝜂𝜂(𝑛𝑛) is set as:
𝜂𝜂(𝑛𝑛) = 𝜂𝜂0 exp �− 𝑛𝑛
𝜏𝜏2
� , 𝑛𝑛 = 0,1,2, …

where 𝜂𝜂0 is the initial learning rate and is set to be 0.1, 𝜏𝜏2 is the time constant
and is set to be N.
4. The time-varying neighborhood function is:
ℎ𝑗𝑗,𝑖𝑖(𝑥𝑥)(𝑛𝑛) = exp �− 𝑑𝑑𝑗𝑗,𝑖𝑖
2
2𝜎𝜎(𝑛𝑛)2� , 𝑛𝑛 = 0,1,2, …
where 𝑑𝑑𝑗𝑗,𝑖𝑖 is the distance between neuron j and winner I; 𝜎𝜎(𝑛𝑛) is the effective
width and satisfies:
𝜎𝜎(𝑛𝑛) = 𝜎𝜎0 exp �− 𝑛𝑛
𝜏𝜏1
� , 𝑛𝑛 = 0,1,2, …

where 𝜎𝜎0 is the initial effective width and is set according to the size of output
layer’s lattice, 𝜏𝜏1 is the time constant and is chosen as 𝜏𝜏𝑖𝑖 = 𝑁𝑁
log(𝜎𝜎0)
.
Again, please feel free to experiment with other design parameters which may be
different from the given ones.