Background Subtraction Methods Overview

• Background subtraction is a reference-based signal isolation paradigm that models both static and dynamic backgrounds to extract meaningful foreground elements.
• It leverages statistical models, local descriptors, and deep learning to adapt to challenging conditions such as illumination changes, dynamic motion, and noisy environments.
• Applications span video surveillance, object tracking, astronomical signal processing, and even black-hole thermodynamics, demonstrating its broad versatility.

Searching arXiv for recent and foundational papers on background subtraction to ground the article in cited literature. arxiv_search(query="background subtraction method computer vision CNN U-Net CDnet 2014", max_results=10) arxiv_search(query="(Zeng et al., 2018) background subtraction convolutional neural network", max_results=5) arxiv_search(query="background subtraction dynamic background weakly supervised arXiv 2023", max_results=10) Background subtraction method denotes a family of procedures that estimate a background or reference state and isolate the deviation from that reference as the quantity of interest. In computer vision, the term most commonly refers to foreground detection for fixed-camera video: moving or newly appeared objects are separated from scene background so that downstream tasks such as tracking, recognition, and activity analysis can operate reliably (Zeng et al., 2018). In other fields, the same phrase is used for subtracting common detector fluctuations in radio-camera time streams, assigning event-wise background probabilities in unbinned particle-physics analyses, or regularizing Euclidean actions by subtracting a reference spacetime in black-hole thermodynamics (Mandal et al., 22 Aug 2025, Wang et al., 2014, Xiao et al., 10 Nov 2025).

1. Problem definition and conceptual scope

In the classical surveillance setting, background subtraction addresses foreground detection under conditions in which the background is only approximately static. The practical difficulty is that real scenes exhibit illumination change, shadows, dynamic backgrounds such as water or foliage, camera jitter, intermittent object motion, thermal imagery, low frame rate, night scenes, pan-tilt-zoom motion, and turbulence; no single method is consistently robust across all such conditions (Zeng et al., 2018). This immediately separates the problem from naive frame differencing: the target is not merely temporal change, but change relative to an evolving or modeled reference.

The same logic appears in application-specific variants. In stationary-incident detection, the relevant signal is not motion but temporary stationarity, so the background model must be manipulated to reveal objects that have stopped and would otherwise be absorbed into the background (Ghimire et al., 17 Jun 2025). In detector-array astronomy, the goal is to remove atmospheric and instrumental common-mode fluctuations while preserving compact sources (Mandal et al., 22 Aug 2025). In covariant black-hole thermodynamics, the objective is to obtain a finite Euclidean action by subtracting a matched reference background, typically a horizonless AdS geometry (Xiao et al., 10 Nov 2025). This suggests that “background subtraction” is best understood as a reference-based signal-isolation paradigm rather than a single algorithmic template.

2. Classical formulations in video analysis

Representative computer-vision formulations span pixel-wise statistical models, local structural descriptors, spatio-temporal dynamical systems, and low-rank subspace models. Pixel-wise Gaussian Mixture Model, ViBe, KDE, and related methods remain standard reference points in comparative studies and deployment-oriented systems (Chen et al., 2018, Makantasis et al., 2016). These methods are attractive because they are local and online, but they do not directly encode broader spatial structure.

A different line models each location by local shape. One example represents each pixel with an 80-dimensional Local Self-Similarity descriptor derived from comparisons of a $5 \times 5$ patch inside a $41 \times 41$ neighborhood, encoded in 20 angular bins and 4 radial intervals. Foreground is then detected by comparing the current descriptor and the stored background descriptor with Euclidean distance and thresholding at 30; the method tends to produce complete foregrounds but often includes shadows, and it explicitly does not handle dynamicBackground, intermittentObjectMotion, or cameraJitter in its current form (Jodoin et al., 2012).

Another formulation treats the background as a local dynamic process over short video volumes. In the dynamic spatio-temporal model, each video brick $v_i$ is written as

$$v_i=\mathbf{C}_n z_i+\omega_i,\qquad z_{i+1}=A_n z_i+B_n\epsilon_i,$$

so appearance consistency and temporal coherence are modeled jointly through a low-dimensional subspace and ARMA-style state dynamics (Lin et al., 2015). This local spatio-temporal view is explicitly designed for dynamic backgrounds, illumination variation, and indistinct foreground objects.

Low-rank methods formulate the background as a dominant subspace of the frame matrix. Fast robust matrix completion estimates a low-rank background $\mathbf{B}$ by solving

$$\min \|\mathbf{B}-\mathbf{V}\|_F^2 \quad \text{s.t.} \quad \|\mathbf{B}\|_\star<\delta,$$

then interprets $\mathbf{V}-\mathbf{B}$ as foreground; it is reported to be at least twice as fast as IALM-based RPCA and low-rank robust matrix completion while maintaining comparable accuracy on BMC and SABS (Rezaei et al., 2017). Adaptive SVD maintains a background subspace online and projects each new frame $J$ onto that subspace,

$\hat{J}_{bg}=U_\ell(U_\ell^T J),$

with the residual interpreted as new information; block-wise updates and re-initialization control complexity and allow robust adaptation (Reitberger et al., 2019). Online mixture-of-Gaussians matrix factorization goes further by combining a low-rank background $\mathbf{U}\mathbf{v}$ with a frame-specific residual mixture model,

$41 \times 41$0

and, with sub-sampling, is reported to execute more than 250 frames per second on average (Yong et al., 2017).

3. Learned, hybrid, and weakly supervised models

Deep models often enter background subtraction as learned fusion or scene-specific residual suppression rather than as generic image segmentation alone. A prominent hybrid example treats fusion itself as an image-to-image problem: the binary masks produced by SuBSENSE, FTSG, and CwisarDH are resized to $41 \times 41$1, concatenated into a $41 \times 41$2 tensor, and fed to a VGG16-based U-Net-like encoder-decoder. On CDnet 2014, this CNN-SFC reaches Recall 0.7845, Precision 0.8969, and F-Measure 0.8243, compared with 0.7745 for majority vote and 0.7694 for IUTIS-3 using the same three inputs; the paper explicitly interprets the gain as better learned disagreement resolution and false-positive suppression rather than a large recall increase (Zeng et al., 2018).

Weakly supervised scene-specific learning replaces pixelwise manual masks with object-free training footage. One such framework trains a fully connected autoencoder to reconstruct the static background $41 \times 41$3, thresholds the residual on object-free frames to obtain pseudo dynamic-background labels, and trains a U-Net to predict the dynamic background $41 \times 41$4. The foreground is then defined by

$41 \times 41$5

On CDnet 2014 Dynamic Background, the reported average F-measure is 0.91, on Bad Weather 0.89, and the implementation is reported to run at 107 frames per second on a GeForce GTX 1080 Ti GPU (Bahri et al., 2023).

DBSGen is an unsupervised generative alternative for dynamic backgrounds. It combines a generative multi-resolution convolutional network for motion removal and a generative fully connected network for background generation, uses

$41 \times 41$6

and then thresholds with a dynamic entropy map. On the CDnet 2014 Dynamic Background category it reports average F-measure 0.86; removing the motion-estimation component lowers the average from 0.86 to 0.72 when post-processing is retained, which identifies motion compensation as the central mechanism rather than a peripheral refinement (Bahri et al., 2022).

4. Modalities, sensing constraints, and application-specific variants

Background subtraction changes substantially when the sensing modality changes. For thermal video, one method models each pixel as a Gaussian mixture with an unknown effective number of components estimated by variational inference with Dirichlet and Gaussian-Gamma priors. It uses about 100 frames for initialization, starts from $41 \times 41$7, prunes components with weight below $41 \times 41$8, and updates each incoming sample online. On the thermal subset of CDB-2014 it reports Precision 0.718, Recall 0.868, F1 0.732, and an FPGA implementation achieves 69.88 fps at $41 \times 41$9 on Virtex-7 using 16 background-subtraction cores (Makantasis et al., 2016).

For depth-only video, BGSNet-D addresses the fact that depth value 0 denotes an absent measurement rather than zero distance. Its preprocessing uses

$v_i$0

with $v_i$1, and the background image averages only valid nonzero depth observations. The CNN then classifies $v_i$2 patches formed from current depth and background depth. On SBM-RGBD, BGSNet-D reports Recall 0.7827, Precision 0.8671, and F-Measure 0.8138, outperforming the depth-only baselines listed in the paper (Wang et al., 2019).

Resource-constrained deployment motivates compression-oriented variants. One study runs ViBe and GMM on compressed low-resolution grayscale frames, then resizes the resulting mask back to the original size. Across three surveillance-style datasets, precision remains almost unchanged over a broad range of compression, while recall and F-measure decrease slowly at first and then drop dramatically around a compression ratio of 60%; the practical conclusion is that ratios from 0% to 60% offer a favorable speed–accuracy trade-off (Chen et al., 2018).

Application-specific systems may alter the very definition of foreground. A block-based vehicle-detection pipeline builds a Static Reference Background Image from adjacent frames using Absolute Difference, Image Entropy, XOR, or DCT block comparison; the reported vehicle-detection accuracies are 82%, 89%, 93%, and 96%, respectively, with DCT performing best (Shehata et al., 2019). For temporarily stationary objects such as illegally parked vehicles, a dual-background design maintains fast and slow adaptive backgrounds $v_i$3 and $v_i$4, then detects stopped objects from their difference before long-term monitoring with NCC; for ROI size $v_i$5, the single-background method runs at 33.26 fps and the dual-background method at 20.13 fps, while the dual-background method is reported as better and more stable on the evaluation dataset (Ghimire et al., 17 Jun 2025).

5. Evaluation protocols and persistent failure modes

The dominant benchmark vocabulary in computer-vision background subtraction is exemplified by CDnet 2014, which provides dense human ground truth and categories including baseline, camera jitter, dynamic background, intermittent object motion, shadow, thermal, bad weather, low framerate, night videos, pan-tilt-zoom, and turbulence (Zeng et al., 2018). The standard metrics include Recall, Specificity, False Positive Rate, False Negative Rate, Percentage of Wrong Classifications, Precision, and F-Measure, with

$v_i$6

Several papers explicitly treat F-measure as the most informative overall summary because it balances recall and precision (Zeng et al., 2018, Bahri et al., 2023).

Failure modes remain highly structured by scene type. Learned fusion over classical masks is strong on baseline, dynamic background, shadow, and bad weather, but remains weaker on night videos, PTZ, and intermittent object motion, where the constituent algorithms fail in more correlated ways (Zeng et al., 2018). Local-self-similarity methods yield high recall and complete object extraction but often include shadows, and their static background assumption excludes dynamicBackground and intermittentObjectMotion (Jodoin et al., 2012). Weakly supervised dynamic-background suppression improves Dynamic Background and several other categories, yet performs worse on Baseline, Camera Jitter, and Night Videos than one comparator (Bahri et al., 2023). DBSGen is explicitly presented as a dynamic-background method, and its future work is directed toward illumination changes and shadows rather than claiming those problems solved (Bahri et al., 2022). A plausible implication is that benchmark success in background subtraction is less about a single universal model than about the match between scene assumptions, modality, and update mechanism.

6. Extensions beyond video foreground detection

Outside video surveillance, the term “background subtraction method” names related but distinct procedures that still subtract a matched estimate of nuisance structure. In partial wave analysis at BESIII, background subtraction via probabilistic event weights assigns each event a signal probability

$v_i$7

then uses

$v_i$8

in place of sideband subtraction. On the test problem reported in the paper, the reconstructed background agrees much better with MC truth than ordinary sidebands, with $v_i$9 improving, for example, from 13.07 to 1.50 on $$v_i=\mathbf{C}_n z_i+\omega_i,\qquad z_{i+1}=A_n z_i+B_n\epsilon_i,$$0 (Wang et al., 2014).

In radio-camera data analysis, ChunkedPCA removes common atmospheric and instrumental fluctuations from detector time streams by dividing the observation into temporal chunks, excluding detectors that see source-contaminated samples in a chunk, and performing PCA only on the remaining off-source detectors. On the simulated dataset in the paper, the average RMS values are 0.1112 for polynomial fitting, 0.1088 for conventional PCA, and 0.1065 for ChunkedPCA; on the real Mars observation, the combined-map RMS falls from 0.0050 with conventional PCA to 0.0036 with ChunkedPCA (Mandal et al., 22 Aug 2025).

In Euclidean black-hole thermodynamics, background subtraction regularizes the action by comparing the black-hole geometry with a matched reference background,

$$v_i=\mathbf{C}_n z_i+\omega_i,\qquad z_{i+1}=A_n z_i+B_n\epsilon_i,$$1

so that

$$v_i=\mathbf{C}_n z_i+\omega_i,\qquad z_{i+1}=A_n z_i+B_n\epsilon_i,$$2

or, in the charged case, $$v_i=\mathbf{C}_n z_i+\omega_i,\qquad z_{i+1}=A_n z_i+B_n\epsilon_i,$$3, becomes finite (Xiao et al., 10 Nov 2025). The same paper argues that the equivalence between Euclidean background subtraction and the Iyer–Wald formalism persists in matter-coupled theories when formulated carefully (Xiao et al., 10 Nov 2025). A related study on 5-dimensional Kerr-AdS with curvature-squared corrections reports that the corrected Gibbs free energy obtained by background subtraction is in exact agreement with a previous holographic-renormalization result, thereby supporting the applicability of the method in that higher-derivative setting (Chen et al., 25 Aug 2025). Across these domains, background subtraction retains the same structural purpose: isolate the physically relevant deviation by subtracting a reference that captures the common or divergent background contribution.