Description
In this assignment, you will be implementing an AR application step by step using planar
homographies. Before we step into the implementation, we will walk you through the theory
of planar homographies. In the programming section, you will first learn to find point
correspondences between two images and use these to estimate the homography between
them. Using this homography you will then warp images and finally implement your own
AR applications.
Instructions
1. Integrity and collaboration: Students are encouraged to work in groups but each
student must submit their own work. If you work as a group, include the names of
your collaborators in your write up. Code should NOT be shared or copied. Please
DO NOT use external code unless permitted. Plagiarism is strongly prohibited and
may lead to failure of this course.
2. Start early! This is a much bigger assignment than homework 1.
3. Questions: If you have any questions, please look at Piazza first. Other students may
have encountered the same problem, and it may be solved already. If not, post your
question on the discussion board. Teaching staff will respond as soon as possible.
4. Write-up: Your write-up should mainly consist of three parts, your answers to theory
questions, resulting images of each step, and the discussions for experiments. Please
note that we DO NOT accept handwritten scans for your write-up in this assignment. Please type your answers to theory questions and discussions for experiments
electronically.
5. Please stick to the function prototypes mentioned in the handout. This makes verifying
code easier for the TAs.
6. Submission: Create a zip file,
Python implementations (including helper functions) and results, and the implementations and results for extra credits (optional). Please make sure to remove the data/
folder, and any other temporary files you’ve generated. Your final upload should have
the files arranged in this layout:
1
•
–
∗
∗ python/
· ar.py
· briefRotTest.py
· HarryPotterize.py
· helper.py
· loadVideo.py
· matchPics.py
· opts.py
· planarH.py
· q2 1 4.py
· yourHelperFunctions.py (optional)
∗ result/
· ar.avi
∗ ec/ (optional for extra credit)
· ar ec.py
· loadVideo.py
· panorama.py
· the images required for generating the results.
· yourHelperFunctions.py (optional)
Please make sure you do follow the submission rules mentioned above before
uploading your zip file to Canvas. Assignments that violate this submission rule
will be penalized by up to 10% of the total score.
7. File paths: Please make sure that any file paths that you use are relative and not
absolute. Not cv2.imread(’/name/Documents/subdirectory/hw2/data/xyz.jpg’)
but cv2.imread(’../data/xyz.jpg’).
1 Homographies
Planar Homographies as a Warp
Recall that a planar homography is an warp operation (which is a mapping from pixel
coordinates from one camera frame to another) that makes a fundamental assumption of the
points lying on a plane in the real world. Under this particular assumption, pixel coordinates
in one view of the points on the plane can be directly mapped to pixel coordinates in another
camera view of the same points.
2
Figure 1: A homography H links all points xπ lying in plane π between two
camera views x and x
0
in cameras C and C
0
respectively such that x
0 = Hx.
[From Hartley and Zisserman]
Q1.1 Homography (5 points)
Prove that there exists a homography H that satisfies equation 1 given two 3×4 camera
projection matrices P1 and P2 corresponding to the two cameras and a plane Π. You
do not need to produce an actual algebraic expression for H. All we are asking for is
a proof of the existence of H.
x1 ≡ Hx2 (1)
The ≡ symbol stands for identical to. The points x1 and x2 are in homogenous
coordinates, which means they have an additional dimension. If x1 is a 3D vector
xi yi zi
T
, it represents the 2D point
xi
zi
yi
zi
T
(called inhomogenous coordinates).
This additional dimension is a mathematical convenience to represent transformations
(like translation, rotation, scaling, etc) in a concise matrix form. The ≡ means that
the equation is correct to a scaling factor.
Note: A degenerate case happens when the plane Π contains both cameras’ centers,
in which case there are infinite choices of H satisfying equation 1. You can ignore this
special case in your answer.
The Direct Linear Transform
A very common problem in projective geometry is often of the form x ≡ Ay, where x and y
are known vectors, and A is a matrix which contains unknowns to be solved. Given matching
points in two images, our homography relationship clearly is an instance of such a problem.
Note that the equality holds only up to scale (which means that the set of equations are of
the form x = λHx0
), which is why we cannot use an ordinary least squares solution such as
what you may have used in the past to solve simultaneous equations. A standard approach
to solve these kinds of problems is called the Direct Linear Transform, where we rewrite
the equation as proper homogeneous equations which are then solved in the standard least
3
