Foreground-Background Memorization (FB-Mem)

• FB-Mem is a framework that segments memorization into distinct regions (verbatim, foreground, background, not memorized) to analyze local and global copying.
• It quantifies memorization using metrics like MS-SSIM and adaptive segmentation protocols, revealing that foreground memorization often persists even after mitigation.
• The topic integrates insights from diffusion models, self-supervised video learning, and layered GANs, highlighting challenges like information leakage and background bias.

Foreground Background Memorization (FB-Mem) denotes a family of phenomena in which model behavior differs systematically across foreground and background, either because memorized content is spatially localized or because foreground and background are encoded, separated, or exploited asymmetrically. In the strict sense introduced for diffusion models, FB-Mem is a segmentation-based memorization framework that compares generated and training images at full-image, foreground-region, and background-region levels, and classifies pairs as Verbatim Memorization (VM), Foreground Memorization (FM), Background Memorization (BM), or Not Memorized (NM) (Di et al., 16 Aug 2025). Closely related work studies analogous effects under other names—notably “déjà vu memorization,” “background bias,” “information leakage,” and “foreground-background entanglement”—across representation learning, video self-supervision, layered generation, anomaly detection, navigation, collaborative perception, and reconstruction-based background modeling (Kokhlikyan et al., 8 Apr 2025).

1. Definitions, scope, and lineage

The most explicit definition of FB-Mem appears in diffusion-model evaluation. There, a generated image $x_g$ and a training image $x_t$ are segmented into foreground and background, and memorization is assessed separately for the whole image, the foreground region, and the background region. The framework is not a single scalar score; it is an algorithmic classification procedure that prioritizes VM over FM over BM over NM (Di et al., 16 Aug 2025).

Across the broader literature, related papers do not usually adopt the term FB-Mem directly, but they isolate structurally similar phenomena. “Déjà vu memorization” asks whether the foreground label can be recovered from a background-only crop beyond what is explained by dataset-level foreground-background correlation (Kokhlikyan et al., 8 Apr 2025). “Background bias” in contrastive video learning describes the tendency to solve representation learning through static scene context rather than motion (Ding et al., 2021). “Information leakage” in layered GANs denotes semantic and visual confusion between foreground and background layers (Zou et al., 2022). “Foreground-background entanglement” in CLIP-based anomaly detection refers to simultaneous activation in object and irrelevant background regions (Hu et al., 20 Mar 2026).

Formulation	Paper	Relation to FB-Mem
VM/FM/BM/NM	(Di et al., 16 Aug 2025)	Direct definition
Déjà vu memorization	(Kokhlikyan et al., 8 Apr 2025)	Correlation-aware foreground-from-background test
Background bias	(Ding et al., 2021)	Background shortcut in video SSL
Information leakage	(Zou et al., 2022)	Cross-layer foreground/background dependence
Foreground-background entanglement	(Hu et al., 20 Mar 2026)	Background interference in anomaly localization

The lineage of the topic predates the term itself. An early masked-RBM model already separated latent foreground appearance and shape from a separately modeled background, under the assumption that foreground and background are independent and that the background is occluded by the foreground (Heess et al., 2011). Later scene-specific background subtraction systems treated background as a learned reference image reconstructed from current frames and used deviations from that reference to infer foreground (Zhao et al., 2017). Autoencoder-based background reconstruction then made this logic fully unsupervised and scene-specific, with a compact latent bottleneck acting as an implicit background memory store and a noise head estimating expected background residuals (Sauvalle et al., 2021).

2. Formal criteria and measurement protocols

In the diffusion setting, FB-Mem uses the notation $x_g$ for a generated image, $x_t$ for a training image, $M(\cdot,\cdot)$ for an image similarity metric, $S_f(\cdot)$ and $S_b(\cdot)$ for foreground and background masks, $T$ for the memorization threshold, and $\beta$ for the segmentation proportion threshold. The full-image similarity is

$M_{\text{full}} = M(x_g, x_t).$

The classification rule is priority-ordered: $x_t$0 Operationally, the algorithm declares VM if $x_t$1, else FM if $x_t$2, else BM if $x_t$3, else NM. The experiments use $x_t$4, and the adaptive region-comparison scheme switches masking behavior depending on whether the generated foreground occupies less than $x_t$5, more than $x_t$6, or an intermediate fraction of the image; in experiments, $x_t$7 (Di et al., 16 Aug 2025).

The same paper shows why full-image duplicate detection is insufficient for local copying. On a manually labeled benchmark, MS-SSIM achieved AUROC $x_t$8, F1 $x_t$9, and Accuracy $x_g$0 for VM vs NM; for TM vs NM it reported AUROC $x_g$1 and TP@1%FP $x_g$2; and for VM vs TM it reported AUROC $x_g$3, F1 $x_g$4, and Accuracy $x_g$5. It also reports that comparing one generated image against 500 ground-truth images takes over 5 minutes with SSCD but 24 seconds with MS-SSIM on an NVIDIA A6000 GPU. The evaluation uses 500 memorized LAION prompts for Stable Diffusion v1.4, with 498 retrievable memorized images due to two broken URLs, generates $x_g$6 images per prompt, and performs $x_g$7 pairwise comparisons. An ablation with $x_g$8 generations per prompt shows that one prompt can map to as many as 17 matched training images (Di et al., 16 Aug 2025).

A different but complementary formalization appears in efficient déjà vu measurement for representation models. Let $x_g$9, where $x_t$0 is a background-only crop and $x_t$1 is the foreground label. The paper defines memorization as the existence of a predictor $x_t$2 such that $x_t$3 while $x_t$4. This makes memorization explicitly relative to dataset-level correlation rather than raw background predictiveness. It also defines stability-based memorization through a leave-one-out criterion using $x_t$5 and $x_t$6, and, in the vision case, uses

$x_t$7

as the per-example correctness indicator (Kokhlikyan et al., 8 Apr 2025).

That paper’s main methodological contribution is to replace the original two-model protocol with a one-model approximation of the correlation baseline. For ImageNet representation models, the reference model can be either a ResNet50 trained on background crops or a Naive Bayes classifier over detected crop objects from Grounded-SAM. The Naive Bayes baseline estimates

$x_t$8

and then computes

$x_t$9

It also defines a sample-level heuristic

$M(\cdot,\cdot)$0

intended to rank likely memorized examples (Kokhlikyan et al., 8 Apr 2025).

3. Mechanisms that generate foreground-background asymmetry

In diffusion models, the central mechanism is local memorization that is spatially concentrated and not adequately captured by full-image duplicate retrieval. FB-Mem shows that memorization can be one-prompt-to-many-training-images rather than one-prompt-to-one-image, and that existing mitigation methods optimized for prompt-wise or verbatim memorization suppress VM well while leaving FM behind. This shifts the question from whether a model duplicates whole images to whether it reproduces semantically central subregions, especially in the foreground (Di et al., 16 Aug 2025).

In self-supervised video learning, the mechanism is positive-pair construction. Standard contrastive learning pulls together clips from the same source video; these clips often share the static background while differing in instantaneous motion. The result is “background bias,” “background shortcut,” or “background cheating”: the model can align positives by keying on stable scene context rather than motion. FAME counters this by estimating a foreground mask from frame difference and color statistics, then constructing

$M(\cdot,\cdot)$1

so that background consistency across positives is broken while moving foreground is preserved (Ding et al., 2021).

In layered GANs, the mechanism is interlayer dependence. ILSGAN argues that foreground and background generators gradually acquire semantic information about each other, producing semantic confusion and visual similarity across layers. It formalizes this as “information leakage” and minimizes mutual information between visible foreground and visible background, and between invisible foreground and visible background, using a variational CLUB upper bound with a Laplace approximation. The resulting independence loss is

$M(\cdot,\cdot)$2

and lower measured MI correlates with higher IoU in ablations (Zou et al., 2022).

A simpler predictive mechanism appears in scene recognition. There, three VGG-16 branches pretrained on ImageNet, Places, and a Hybrid dataset are interpreted as foreground, background, and hybrid feature extractors. The paper reports that background features outperform foreground and hybrid features individually on both MIT-67 and SUN-397, with the qualitative ranking

$M(\cdot,\cdot)$3

This does not establish memorization by itself, but it demonstrates that background/context can be the strongest standalone signal, which is precisely the condition under which shortcutting or memorization pressure becomes plausible (Sitaula et al., 2020).

4. Architectures for separation, control, and deliberate use

A large branch of related work treats foreground-background asymmetry through explicit decomposition. V3GAN factorizes video generation into a static background branch $M(\cdot,\cdot)$4, a foreground branch $M(\cdot,\cdot)$5, a temporal branch $M(\cdot,\cdot)$6, and a feature-level masking branch $M(\cdot,\cdot)$7, with compositing

$M(\cdot,\cdot)$8

Its relevance to FB-Mem is methodological rather than diagnostic: branchwise outputs and masks make leakage inspectable, but the paper does not directly test whether backgrounds are memorized independently of foregrounds (Keshari et al., 2022).

Several systems instead model background as an explicit reference or memory. A two-stage CNN for foreground segmentation first reconstructs a clean background image with a convolutional encoder-decoder and then concatenates the reconstructed background with the current frame in a six-channel MCFCN. The overall loss

$M(\cdot,\cdot)$9

couples background reconstruction and foreground segmentation, and the paper interprets the first stage as encoding rich knowledge of background scenes in a scene-specific latent representation (Zhao et al., 2017). Autoencoder-based background subtraction pushes this logic further by reconstructing a dynamic background manifold and predicting a pixelwise background-noise map $S_f(\cdot)$0, leading to the adaptive threshold

$S_f(\cdot)$1

so that foreground is what exceeds expected reconstruction noise rather than raw residual alone (Sauvalle et al., 2021).

In CLIP-based zero-shot anomaly detection, FB-CLIP tackles foreground-background entanglement and residual background interference through a stacked design: Multi-Strategy Text Feature Fusion, Multi-View Foreground-Background Enhancement, Background Suppression, and Semantic Consistency Regularization. The visual side constructs a soft foreground mask $S_f(\cdot)$2, enhances token features along identity, semantic, and spatial views, builds a background prototype

$S_f(\cdot)$3

and suppresses tokens similar to that prototype before visual-text alignment (Hu et al., 20 Mar 2026).

Other domains convert the same theme into routing or curriculum. In vision-language navigation, COFA uses semantically enhanced landmark identification, EVF-SAM masking, and a two-stage consensus-driven online augmentation procedure to choose per viewpoint among original, foreground, and background features. The final choice is a switch-like rule over $S_f(\cdot)$4, rather than continuous fusion (Xu et al., 1 Oct 2025). In collaborative perception, FadeLead uses Foreground Context Attention, Curricular Background Pruning, and Foreground Amplification Fusion; during training it shares enriched foreground plus a shrinking subset of informative background, with the pruning rule

$S_f(\cdot)$5

and at inference it transmits only enriched foreground. The paper explicitly describes this as learning to encapsulate background context into compact foreground features during training and forcing the model to internalize context into foreground representations without transmitting background itself (Wu et al., 22 Oct 2025).

Animal re-identification offers a different form of explicit control. Images are segmented with MegaDetector and Segment Anything, and foreground appearance is modeled separately from background-derived priors. The combined posterior is

$S_f(\cdot)$6

where $S_f(\cdot)$7 is the foreground classifier and $S_f(\cdot)$8 is a home-location, migrating-location, or time-decay prior. Per-Instance Temperature Scaling predicts a sample-specific temperature $S_f(\cdot)$9 for the foreground logits,

$S_b(\cdot)$0

so that appearance evidence can be fused with contextual priors in calibrated form (Picek et al., 2024).

5. Empirical regularities and failure modes

The most direct empirical finding in the strict FB-Mem literature is that local memorization is more pervasive than verbatim duplication. In Stable Diffusion v1.4, one prompt can correspond to multiple memorized training images; with $S_b(\cdot)$1 generations per prompt, one-to-many correspondence can reach 17 matched images. Under mitigation, all tested methods remove over 90% of VM and BM is also largely reduced, but FM persists. The transition-based mitigation scores averaged over 2,500 images are 0.74 for NeMo, 0.67 for DetectMem, 0.79 for Wanda, and 0.83 for NeMo-C, indicating that cluster-wise neuron deactivation better addresses concept-level memorization than prompt-wise deactivation (Di et al., 16 Aug 2025).

Efficient measurement produces a complementary aggregate picture. On ImageNet subset-trained models, the one-model ResNet-based déjà vu score closely matches the original two-model estimate for VICReg and Barlow Twins, while DINO is lower in both setups. The paper reports approximate pairs $S_b(\cdot)$2 vs $S_b(\cdot)$3 for VICReg, $S_b(\cdot)$4 vs $S_b(\cdot)$5 for Barlow Twins, and $S_b(\cdot)$6 vs $S_b(\cdot)$7 for DINO. It also finds that open-source VICReg, Barlow Twins, and DINO typically have lower aggregate memorization than similar models trained on a 300k subset of ImageNet. At the same time, sample-level agreement across correlation estimators is only about 40%, so the protocol is strongest as a population-level estimator rather than a definitive per-example oracle (Kokhlikyan et al., 8 Apr 2025).

Animal re-identification makes the distinction between useful context and brittle memorization unusually explicit. For lynx, background-only classification reaches 27.1% overall but only 3.5% on new locations; whole-image CE obtains 49.7% overall and 13.9% on new locations; foreground-only PITS reaches 52.5% overall and 19.1% on new locations; and foreground plus PITS plus the migrating-location prior reaches 60.9% overall and 19.6% on new locations. For sea turtles, whole-image CE is 16.5%, foreground-only PITS is 21.7%, and foreground plus PITS plus the time-decay prior reaches 25.7%. These results show that background can be highly predictive in-distribution and highly unstable under background drift, while explicit prior modeling can still improve over appearance-only recognition (Picek et al., 2024).

Background reconstruction methods exhibit a different failure mode: persistent or dominant foreground can be absorbed into the background model itself. The unsupervised autoencoder-based method reports failures on CDnet sequences “office,” “library,” and “canoe,” and on BMC video 007, where a large train occupies much of the frame; in such cases the autoencoder reconstructs the foreground object as part of the background. The paper describes this as overfitting caused by excessive capacity or persistent foreground, which is the canonical failure mode for reconstruction-based background memory (Sauvalle et al., 2021).

FadeLead supplies a final empirical pattern that is closely aligned with the broader FB-Mem interpretation. At a 1% selection ratio, it reports 95.81 / 95.05 / 88.10 on OPV2V, 92.21 / 90.88 / 81.23 on V2X-R, and 83.27 / 79.10 / 64.45 on DAIR-V2X, and gains only marginally when the selection ratio rises from 1% to 10%. The paper interprets this as evidence that essential background context is already encapsulated into the minimal set of shared foreground regions. It also reports that GT-BG only can outperform GT-FG only in the motivating study, and that removing Curricular Background Pruning causes notable degradation or even early collapse when training strictly foreground-only sharing (Wu et al., 22 Oct 2025). This suggests a learned transfer of useful context into foreground channels, although the paper does not formalize that transfer as a memorization metric.

6. Conceptual distinctions, controversies, and open questions

A central distinction in the literature is between memorization and correlation. Déjà vu memorization is defined only after subtracting what can be predicted from $S_b(\cdot)$8, and the one-model protocol is explicitly framed as an approximation to dataset-level correlation rather than a direct causal test (Kokhlikyan et al., 8 Apr 2025). By contrast, the diffusion FB-Mem framework is instance-level and pairwise: it localizes similarity within foreground and background regions regardless of whether that similarity arises from one training image or a cluster of near-duplicates (Di et al., 16 Aug 2025). These two perspectives are compatible but not identical.

A second distinction is between harmful shortcutting and useful context. Scene-recognition experiments show that background features can be stronger than foreground features as standalone predictors, and animal identification demonstrates that background priors can be beneficial when isolated and fused explicitly (Sitaula et al., 2020). The same papers also show why this does not imply that entangled background usage is benign: background-only lynx accuracy collapses at new locations, and whole-image turtle classification is much worse than foreground-only classification (Picek et al., 2024). This suggests that the critical issue is not whether background is informative, but whether its contribution is explicit, calibrated, and robust under shift.

A third distinction concerns what is actually being measured. V3GAN, FB-CLIP, COFA, FadeLead, reconstruction-based background subtraction, and related systems are highly informative methodologically, but several of them do not provide a direct memorization audit. V3GAN offers explicit branches and nearest-neighbor analysis but no branch-specific memorization test (Keshari et al., 2022). FB-CLIP addresses foreground-background entanglement, background interference, and residual background noise, but it does not name memorization explicitly (Hu et al., 20 Mar 2026). COFA and FadeLead show adaptive or curriculum-based control over foreground and background information flow, but the claim that background is internalized into foreground remains an architectural and empirical interpretation rather than a direct information-theoretic measurement (Xu et al., 1 Oct 2025).

The literature also has shared technical limitations. FB-Mem in diffusion models depends on segmentation quality, threshold selection, and retrieval completeness; it uses a binary foreground-background split, and its region comparisons remain threshold-based rather than null-model-based (Di et al., 16 Aug 2025). Efficient déjà vu measurement warns that alternative correlation estimators may be poor approximations to the Bayes-optimal baseline, that the reference model may itself memorize its own training set, and that sample-level agreements are much weaker than aggregate agreements (Kokhlikyan et al., 8 Apr 2025). Independent foreground/background animal modeling states directly that accurate segmentation is its main limitation (Picek et al., 2024). V3GAN assumes negligible camera motion and targets short 16-frame clips, which narrows the scope of its decomposition claims (Keshari et al., 2022).

A plausible implication is that future FB-Mem work will need to combine these strands: region-aware memorization metrics, correlation-aware baselines, explicit branchwise architectures, and drift or counterfactual evaluations that swap or perturb foreground and background independently. The papers surveyed here already indicate the core technical agenda. They show that memorization can be local rather than global, that background can be both shortcut and signal, that branch independence is not guaranteed by architecture alone, and that robust modeling often requires making foreground-background structure explicit rather than leaving it implicit in end-to-end objectives.