Description
For Assignment 2, you will implement an image stitcher that uses image warping
and homographies to automatically create an image mosaic. We will focus on the
case where we have two input images that should form the mosaic, where we warp
one image into the plane of the second image and display the combined views.
This
problem will give some practice manipulating homogeneous coordinates, computing
homography matrices, and performing image warps. For simplicity, we’ll specify
corresponding pairs of points manually using mouse clicks.
Note:
There are some built-in Matlab functions that could do much of the work for
this project. However, to get practice with the workings of the algorithms, we want
you to write your own code. Specifically, you may not use any of these functions in
your implementation: cp2tform, imtransform, tformarray, tformfwd, tforminv,
maketform.
Provided files:
Two image files that can be used for the mosaic.
What You Have to Do
Task 1 (15 points) Getting correspondences:
Write code to get manually identified corresponding points from two views. Look at
Matlab’s ginput function for an easy way to collect mouse click positions. The
results will be sensitive to the accuracy of the corresponding points; when providing
clicks, choose distinctive points in the image that appear in both views.
Task 2 (20 points) Computing the homography parameters:
Write a function that takes a set of corresponding image points and computes the
associated 3 x 3 homography matrix H. This matrix transforms any point pi in one
view to its corresponding homogeneous coordinates in the second view, pi’, such
that
λ* pi = H * pi’
Note that pi and pi’ are both vectors with 3 elements. The function should take a list
of pairs of corresponding points (what is the minimum number of points you
need?) from the two views, where each point is specified with its 2d image
coordinates.
Useful Matlab functions: ‘\’ operator (help mldivide), reshape
Verify that the homography matrix your function computes is correct by mapping
the clicked image points from one view to the other, and displaying them on top of
each respective image. Be sure to handle homogenous and non-homogenous
coordinates correctly.
Task 3 (20 points) Warping between image planes:
Write a function that can take the recovered homography matrix and an image, and
return a new image that is the warp of the input image using H . Since the
transformed coordinates will typically be sub-pixel values, you will need to sample
the pixel values from nearby pixels. Feel free to use the sampleBilinear.m you
developed for the first assignment. For color images, warp each RGB channel
separately and then stack together to form the output. To avoid holes in the output,
use inverse warp rather than direct mapping.
To compute the bounding box of the destination image, you will need to warp the
points from the source image into the reference frame of the destination. Then
sample all points in that destination bounding box from the proper coordinates in
the source image. Note that transforming all the points will generate an image of a
different shape / dimensions than the original input. It is ok to have some areas of
the new image be black (0).
Useful Matlab functions: round, interp2, meshgrid, isnan.
Task 4 (20 points) Create the output mosaic:
Once we have the source image warped into the destination image’s frame of
reference, we can create a merged image showing the mosaic. Create a new image
large enough to hold both (registered) views; overlay one view onto the other,
simply leaving it black wherever no data is available. Don’t worry about artifacts
that result at the boundaries.
You are free to use a method/convention of your own choosing for the overlap
areas.
Task 5 (25 points) After writing and debugging your system:
1. [5 pts] Apply your system to the provided pair of images, and display the output
mosaic.
2. [10 pts] Show two additional examples of mosaics you created using images that
you have taken. You can make a mosaic from two or more images of a broad scene
that requires a wide angle view to see well. Or, make a mosaic using two images
from the same room where the same person appears in both.
3. [10 pts] Warp one image into a “frame” region in the second image. To do this, let
the points from the one view be the corners of the image you want to insert in the
frame, and the let the corresponding points in the second view be the clicked points
of the frame (rectangle) into which the first image should be warped. Use this idea
to replace one surface in an image with an image of something else.
For example —
overwrite a billboard with a picture of your dog, or project a drawing from one
image onto the street in another image, or replace a portrait on the wall with
someone else’s face, or paste a Powerpoint slide onto a movie screen, …
For all examples, play around a bit with the choice of points for the correspondence
pairs until you get a reasonable alignment.
[OPTIONAL] Extra credit
Implement RANSAC for robustly estimating the homography matrix from noisy
correspondences. Your function should be able to handle more than 4 point
correspondences. Show with an example where it successfully gives good results
even when there are outlier (bad) correspondences given as input. Compare the
robust output to the original (non-RANSAC, only 4 point correspondences)
implementation where all correspondences are used.
Note:
feel free to attempt using David Lowe’s SIFT Matlab demo
(http://www.cs.ubc.ca/~lowe/keypoints/siftDemoV4.zip). Warning: it only works
in Linux and Windows, it does not work on Mac.
The result of calling the sift.m function will be 2 vectors, one of features and one
of keypoints. Make sure you understand what is saved in these two variables so that
you can potentially use them for the RANSAC implementation. If you can’t get SIFT
to work, you will need to select make than 4 pairs of points using the function
developed in Task 1.
Submitting the assignment:
Make sure each script or function file is well commented and it includes a block
comment with your name, course number, assignment number and instructor name.
Zip all the .m and image files together and submit the resulting .zip file through
Moodle as Assignment 2 by Friday, September 30th, by 11:55pm.
Acknowledgement: project description and test images courtesy of Kristen
Grauman, University of Texas at Austin.
Tips:
• It can be useful when debugging to plot the corners and clicked points from
one view on top of the second view after transforming them via H. Use
axis([minx, maxx, miny, maxy]); to adjust the viewing window so that you
can see all points.
• You will need the inverse of the homography matrix to transform
“backwards”.
• Be aware that Matlab’s image (matrix) indices are specified in (row,col)
order, i.e., (y,x), whereas the plot and ginput functions use (col,row)
order, i.e., (x,y).
• Check the order of the clicked corresponding points, to make sure your code
uses the intended corresponding point pairs.
• As usual, be careful with how images are cast for computations and display
(double vs. uint8). In particular, for your sampleBilinear.m function, be
sure to pass a matrix of doubles for the image input.
• When collecting your own images, be sure to either maintain the same center
of projection (hold the camera at one location, but rotate between views), or
else take shots of a scene with a large planar component (a building, maybe).
In either case, use a static scene. Textured images that have distinctive points
you can click on are good. Also ensure that there is an adequate overlap
between the two views.
