## Description

Use the Dynamic Mode Decomposition method to take a video clip containing a foreground and

background object and separate the video stream to both the foreground video and a background.

Make several test videos to try the algorithm on.

You can use your smart phone to generate appropriate videos.

The DMD spectrum of frequencies can be used to subtract background modes. Specifically,

assume that ωp, where p ∈ {1, 2, . . . , `}, satisfies ∥ωp∥ ≈ 0, and that ∥ωj∥ ∀ j ≠ p is bounded away

from zero.

Thus,

XDMD = bpϕp

e

ωpt

´ ¹¹¹¹¹¹¹¹¸ ¹¹¹¹¹¹¹¹¶

Background Video

+ ∑

j≠p

bjϕj

e

ωj t

´ ¹¹¹¹¹¹¹¹¹¹¹¹¹¸ ¹¹¹¹¹¹¹¹¹¹¹¹¹¶

Foreground Video

(1)

Assuming that X ∈ R

n×m, then a proper DMD reconstruction should also produce XDMD ∈ R

n×m.

However, each term of the DMD reconstruction is complex: bjϕj

exp (ωjt) ∈ C

n×m ∀j, though they

sum to a real-valued matrix.

This poses a problem when separating the DMD terms into approximate

low-rank and sparse reconstructions because real-valued outputs are desired and knowing how to

handle the complex elements can make a significant difference in the accuracy of the results. Consider

calculating the DMD’s approximate low-rank reconstruction according to

XLow-Rank

DMD = bpϕp

e

ωpt

.

Since it should be true that

X = XLow-Rank

DMD + X

Sparse

DMD ,

then the DMD’s approximate sparse reconstruction,

X

Sparse

DMD = ∑

j≠p

bjϕj

e

ωj t

,

can be calculated with real-valued elements only as follows. . .

X

Sparse

DMD = X − ∣XLow-Rank

DMD ∣,

where ∣ ⋅ ∣ yields the modulus of each element within the matrix. However, this may result in X

Sparse

DMD

having negative values in some of its elements, which would not make sense in terms of having

negative pixel intensities. These residual negative values can be put into a n × m matrix R and then

be added back into XLow-Rank

DMD as follows:

XLow-Rank

DMD ← R + ∣XLow-Rank

DMD ∣

X

Sparse

DMD ← X

Sparse

DMD − R

This way the magnitudes of the complex values from the DMD reconstruction are accounted for,

while maintaining the important constraints that

X = XLow-Rank

DMD + X

Sparse

DMD ,

so that none of the pixel intensities are below zero, and ensuring that the approximate low-rank and

sparse DMD reconstructions are real-valued. This method seems to work well empirically.

NOTE: it is pretty easy to produce a video on your smart phone.