Dynamic Mode Decomposition for Real-Time Background/Foreground Separation in Video

Abstract: This paper introduces the method of dynamic mode decomposition (DMD) for robustly separating video frames into background (low-rank) and foreground (sparse) components in real-time. The method is a novel application of a technique used for characterizing nonlinear dynamical systems in an equation-free manner by decomposing the state of the system into low-rank terms whose Fourier components in time are known. DMD terms with Fourier frequencies near the origin (zero-modes) are interpreted as background (low-rank) portions of the given video frames, and the terms with Fourier frequencies bounded away from the origin are their sparse counterparts. An approximate low-rank/sparse separation is achieved at the computational cost of just one singular value decomposition and one linear equation solve, thus producing results orders of magnitude faster than a leading separation method, namely robust principal component analysis (RPCA). The DMD method that is developed here is demonstrated to work robustly in real-time with personal laptop-class computing power and without any parameter tuning, which is a transformative improvement in performance that is ideal for video surveillance and recognition applications.