Description
Question 1 (50%)
In this exercise, you will train many multilayer perceptrons (MLP) to approximate the class
label posterior, using maximum likelihood parameter estimation (equivalently, with minimum average cross-entropy loss).
Pick your own C value (number of classes) from this set: {3,4,5}. Pick two activation functions for the perceptrons; one sigmoid, one soft-ReLu type differentiable function. Your MLP
structure will consist of 2 fully-connected layers of adaptive weights, followed by a softmax layer
to ensure your model output conforms to the requirements of a probability distribution over discrete class labels. The dataset you will classify consists of 2-dimensional real vectors. The number
of perceptrons in the first layer of the MLP will be determined using cross-validation procedures
(e.g., from the set {1,…,6}).
Using the Matlab function generateMultiringDataset.m generate two sets of data. Dtrain will
have 100 iid samples and their class labels. Dtest will have 10000 iid samples and their class labels.
Do NOT use Dtest in making any training or model selection choices in the process. You will use
it only for final performance evaluation.
Using 10-fold cross-validation with Dtrain, and minimum classification error probability as the
objective, select the best combination of number of perceptrons in the first layer and the activation
function.
When you determine the best combination of model order and activation nonlinearity, train
an MLP with these specifications using Dtrain. This is your final trained MLP model for classlabel posteriors. Throughout this process, make sure that you train the MLS from multiple initial
conditions and select the best solution encountered, to avoid the possibility of relying on training
attempts that may get trapped in a local minimum.
Using MAP classification rule and your final trained MLP model for class label posteriors,
classify the samples in Dtest and estimate the probability of error your classifier would achieve.
Report/explain the entire process you went through including your modeling and algorithm
design choices, present your results and discuss any insights/observations you made.
Repeat the entire process with new training datasets that have 500 and 1000 training samples,
respectively. Use the same test dataset for final performance evaluation. Discuss the impact of
number of training samples on your final modeling choices and test performance.
Note: You may use software packages for MLP weight optimization with a given training set.
However, you must implement your own 10-fold cross-validation procedure.
1
Question 2 (50%)
In this excercise, you will train an alternative approximate MAP classifier for the same datasets
used in the previous question, but this time using Gaussian Mixture Models for each class conditional pdf. You will compare results with those obtained by the MLP models.
Pick one of your training datasets (e.g., the one with 100 samples). From the labels estimate
the class prior probabilities. Then, for each class, using only the samples with that class label, train
a GMM for the class-conditional pdf of that class.
For each class, to select the number of Gaussian components for the associated GMM, use
10-fold cross validation with maximum validation-log-likelihood as the objective.
Once you determine the best model order (number of components), use all the data from the
training data for the class label under consideration to fit one final GMM using the entire training
dataset.
Use the EM algorithm for all GMM-optimization procedures (and make sure to use multiple
random initializations for each attempt and select the best training-log-likelihood solution to help
mitigate the impact of local minima that may trap some of the EM training attempts).
Once you have these final trained GMMs for each class label, construct an approximate MAP
classifier using these GMMs and estimated class priors, and apply this decision rule to the test
dataset to assess the probability of error achieved by this classifier.
Repeat the process for all three training sets and report/explain your entire process of modeling
this dataset towards approximating the MAP classifier using GMMs for class conditional models.
Discuss your results, including the impact of the number of training samples on your final modeling
choices and test performance.
Note: You may use software packages for EM-based GMM optimization with a given training
set. However, you must implement your own 10-fold cross-validation procedure.
2
