Incremental Sample Condensation

• Incremental sample condensation is a set of techniques that create highly informative synthetic datasets on-the-fly from streaming and non-stationary data.
• It employs methods such as gradient matching, pseudo-labeling, and contrastive learning to preserve key learning signals under tight memory constraints.
• The approach ensures efficient, adaptive memory usage in resource-constrained environments, mitigating issues like catastrophic forgetting.

Incremental sample condensation is a family of algorithmic techniques for constructing highly informative, memory-efficient synthetic datasets or representations on-the-fly from streaming, non-stationary data. Distinguished from classical condensation, incremental condensation methods are designed to operate in resource-constrained, sequential, or evolving environments—such as online continual learning, on-device adaptation, or streaming tabular inference—where only a fraction of the data can be stored, each datastream is observed once, and the model must update or adapt memory continuously to mitigate knowledge loss or catastrophic forgetting (Sangermano et al., 2022, Xu et al., 2024, He et al., 2023, Chen et al., 22 Jan 2026). These techniques are particularly pertinent in settings where ground-truth labels may be unavailable and computational restrictions preclude repeated condensation or full-buffer optimization.

1. Problem Landscape and Motivation

Incremental sample condensation algorithms address scenarios characterized by the following constraints:

• Single-pass data streams: Data arrives incrementally (rarely if ever observed more than once), often in small mini-batches or stream segments; class or task boundaries are typically unknown (Sangermano et al., 2022, Xu et al., 2024).
• Strict memory limitations: Only a small buffer (fixed memory size $M_{\max}$ or buffer $B$) is permitted for samples, which may include raw or synthetic exemplars or representations.
• Requirement for knowledge preservation: As memory fills, it is necessary to select, merge, or synthesize examples such that the condensed memory remains representative of all previously observed data, minimizing performance loss under non-stationary input distributions.
• Label-sparse or unsupervised settings: In some applications (e.g., edge devices), data arrives unlabeled and labels must be estimated or inferred, introducing additional challenges for condensation (Xu et al., 2024, Chen et al., 22 Jan 2026).
• Dynamic adaptation: Models must incrementally evolve their memory content and, in some cases, their representations to accomodate structural changes in data (e.g., addition of new features or classes) (Chen et al., 22 Jan 2026).

The principal objective of incremental sample condensation is to summarize the historical data with a minimal set of synthetic or surviving (possibly blended) samples that support robust model performance, efficient replay, or rapid adaptation during subsequent training or inference cycles.

2. Methodological Frameworks

2.1 Gradient Matching–Based Incremental Condensation

A core strategy, exemplified by OLCGM (Online Linear Combination Gradient Matching), is to iteratively condense both newly arriving and historical samples into a smaller set of synthetic exemplars. This is formalized as a constrained optimization:

Given a batch $B = \{(x_j, y_j)\}_{j=1}^m$ (new mini-batch and selected memory samples), synthetic examples $S$ are constructed as

$S = (W \circ M) \cdot B$

where $W \in \mathbb{R}^{n \times m}$ are non-negative weights, $M \in \{0,1\}^{n \times m}$ is a binary mask (typically enforcing that each synthetic blends a new and an old sample), and $n < m$ (Sangermano et al., 2022).

The optimization minimizes the expected gradient discrepancy between the synthetic set $S$ and the real set $B$:

$$\min_W\,\, \mathbb{E}_{\theta^S}\left[\sum_{c=1}^C D\left(\nabla_\theta L_c^S(\theta^S),\, \nabla_\theta L_c^B(\theta^S)\right)\right]$$

where $L^S$ and $L^B$ denote average classification losses over $S$ and $B$, and $D(\cdot,\cdot)$ is a sum of per-output-node cosine discrepancies.

Inner loops optimize network parameters $\theta$ on $S$, and outer loops update $W$ via losses back-propagated through the condensation operation. The process is periodically (every $K$ mini-batches) invoked to keep the memory buffer maximally informative under budget constraints.

2.2 Pseudo-Label–Driven and Contrastive Condensation

Condensation methods for unlabeled online streams (e.g., DECO) leverage pseudo-labeling via the current model with majority-vote filtering for robustness. Incoming data segments are pseudo-labeled, confidence-scored, and filtered to retain only major classes (with threshold $M$):

• Real and synthetic buffers are updated using a one-step gradient matching loss between new filtered real input and the current synthetic buffer, using cosine distance or MMD between gradients computed with respect to a randomly initialized model parameterization (Xu et al., 2024).
• A supervised contrastive learning term is used to improve class purity among buffer samples, where positive and negative sets for each synthetic are defined within active classes of the current segment, yielding gradients that are jointly optimized for improved representation clustering.

This estimator handles streaming, non-i.i.d., and unlabeled data and is efficient for on-device learning with minimal computation.

2.3 Information Bottleneck-Driven Architecture

In tabular domains with evolving structure (TabII), Incremental Sample Condensation (ISC) blocks implement attention-based compression over feature-augmented tabular representations at inference time (Chen et al., 22 Jan 2026):

• Each row, consisting of both original and new "incremental" features, is first embedded via a frozen feature encoder, an LLM-based prompt embedding, and a TabAdapter module. These form column tokens within each row.
• Intra-row multi-head self-attention (MSA) identifies and re-weights informative increments while discarding redundant or irrelevant variability.
• Subsequently, inter-row Interior Incremental Sample Attention (IISA) aggregates information across the batch to reinforce consistency and further suppress spurious increments.
• The system is optimized to maximize $I(Z;Y)-\beta I(X';Z)$ per information bottleneck theory, where $Z$ denotes the ISC-compressed representation.

3. Algorithmic Protocols and Pseudocode

Key procedural elements for incremental sample condensation methods are summarized below.

3.1 OLCGM Memory Condensation (Pseudocode Extract) (Sangermano et al., 2022)

LCGM_Condense(B, f, θ_init, M, T, I, η_θ, η_W): 1. Initialize W ≥ 0 2. for t = 1..T: a) S ← (W ∘ M) · B b) for each class c: - Sample B_c^B, B_c^S - Compute L_c^B, L_c^S, their gradients - Compute distill loss D(∇_θ L_c^S, ∇_θ L_c^B) c) Update W by gradient descent on distill loss; clip, normalize rows. d) Update θ: SGD on L^S for I steps, lr=η_θ 3. Return final S

Integration with continual learning is achieved by periodically calling this condensation on new+old batches and updating the buffer by replacement.

3.2 DECO Incremental Update Algorithm (Pseudocode Extract) (Xu et al., 2024)

1. Pseudo-label/filter batch → active set I_t^a, S_t^a 2. For ℓ=1…L: - Sample θ̃ ∼ init; compute gradients on X_t, X_t' - Update X_t' ← X_t' − η [∇_{X_t'}D + α∇_{X_t'}L_con] 3. Merge X_t' into buffer S (size B constant) 4. Retrain θ every β segments on S

3.3 ISC Block Procedure (Pseudocode Extract) (Chen et al., 22 Jan 2026)

function ISC_Block({r_i} for i=1…B): # 1. Intra-row MSA for i in 1…B: R_i ← MSA_columns(r_i) \bar r_i ← Pool_columns(R_i) # 2. Inter-row IISA for i in 1…B: z_i = sum_j softmax((\bar r_i W^q)•(\bar r_j W^k)/√d_out) × (\bar r_j W^v) return {z_i}

4. Theoretical Properties and Optimization Principles

Incremental sample condensation strategies are underpinned by several theoretical rationales:

• Gradient matching: By aligning the gradient updates induced by condensed and original sets, the synthetic buffer seeks to preserve the learning signal trajectory associated with the true distribution over all past data (Sangermano et al., 2022, Xu et al., 2024).
• Linear combination parameterization: Restricting synthetic construction to convex combinations of real inputs enforces a tractable search space for condensation and maintains diversity, reducing collapse risks and improving practical feasibility under tight memory budgets (Sangermano et al., 2022).
• Information bottleneck: The ISC blocks explicitly optimize the trade-off between discarding superfluous information from newly added features (lowering $I(X';Z)$) and retaining relevance to downstream prediction targets (maximizing $I(Z;Y)$) (Chen et al., 22 Jan 2026).
• Contrastive refinement: Integrating supervised contrastive loss terms during condensation (as in DECO and TabII) improves class cluster separation and buffer purity, providing resilience against noisy or corrupt pseudo-labels (Xu et al., 2024, Chen et al., 22 Jan 2026).
• Generalization and Rademacher complexity: Strategies that enforce class balance and prune by prediction error (e.g., YOCO's LBPE requirement) guarantee tighter generalization error bounds by controlling complexity and ensuring all classes are equally represented as the condensed dataset is downsampled (He et al., 2023).

5. Empirical Performance and Practical Guidelines

A comparative summary across domains and methods is provided in the following table:

Method/Setting	Core Principle	Main Empirical Findings
OLCGM (Sangermano et al., 2022)	Gradient-Matching, Linear Comb.	On SplitMNIST/Fashion/CIFAR10 under memory $|M|\le50$, improves end-of-stream ACC by 20–30% and reduces forgetting vs. replay/coreset baselines. Gains saturate as $|M|$ increases.
DECO (Xu et al., 2024)	Pseudo-label, 1-step GradMatch, Contrastive	On CIFAR-10/100, SVHN, ImageNet-10 with buffer=1 image/class, delivers 27–58% relative accuracy improvements over best baselines. Effective under extreme (1–5 per class) buffer constraints.
YOCO (He et al., 2023)	LBPE Pruning, Balanced Construction	Enables post-hoc downscaling (flexible resizing) of condensed sets without full recomputation; yields 5–15% accuracy gain over selection/pruning baselines across CIFAR/ImageNet classes per-class quotas.
TabII–ISC (Chen et al., 22 Jan 2026)	IB-Driven, Dual Attention, Contrastive	Achieves 1–3 points test accuracy gain attributable to ISC blocks; enables models to utilize new columns at inference and reach 97% of the fully supervised upper bound.

Best practices and implementation notes from reported ablations include:

• Condensation frequency $K$ controls a trade-off between buffer freshness and computational cost; values of $K=10$ (MNIST/Fashion) to $K=50$ (CIFAR10) are typical for online learning (Sangermano et al., 2022).
• Gradient-matching via linear-combination condensation (vs pixel/synthetic condensation) offers significant speedups and improved robustness at low memory (Sangermano et al., 2022).
• For pseudo-label filtering (as in DECO), a threshold $M \approx 0.4|I_t|$ efficiently balances data quality and buffer activity (Xu et al., 2024).
• Contrastive loss term weight $\alpha\approx0.1$–$0.2$ is optimal for maximizing class purity (Xu et al., 2024, Chen et al., 22 Jan 2026).
• ISC block adaptation at inference is computationally light; attention operations for $F\le100$, $B\approx64$ are practical even for large tabular data (Chen et al., 22 Jan 2026).

6. Relation to Prior Art and Extensions

Incremental sample condensation extends and differentiates from classical dataset condensation and coreset selection by enabling:

• Continual/on-device adaptation: Classical methods require repeated optimization over the full buffer or static recondensation for new target sizes or evolving feature spaces (He et al., 2023). Incremental methods operate seamlessly as new data arrives (OLCGM, DECO), or enable instant resizing by dynamic pruning rules (YOCO).
• Handling of label sparsity and noise: Recent methods combine pseudo-labeling schemes with robust filtering and buffer updates to maintain high-quality synthetic representations when labels are unavailable or unreliable (Xu et al., 2024, Chen et al., 22 Jan 2026).
• Efficient incremental update and inference mechanisms: Systems such as TabII demonstrate that ISC blocks allow models trained on fixed column sets to leverage new, unseen features in a plug-and-play manner during inference, optimizing mutual information without expensive retraining (Chen et al., 22 Jan 2026).

The ability to condense, update, and adapt sampled memory continuously underpins emerging applications in lifelong learning, adaptive edge intelligence, and data-efficient AI deployment.

7. Open Challenges and Future Directions

While incremental sample condensation methods have achieved substantial memory and computation savings, several ongoing challenges and avenues for research are evident:

• Theoretical guarantees: Formal convergence, generalization, and robustness analysis—especially for complex online/unsupervised settings—remains incomplete, though empirically promising results support their practical utility (Sangermano et al., 2022, Chen et al., 22 Jan 2026).
• Representation collapse and diversity maintenance: Safeguards such as mask constraints and attention mechanisms are used to ensure condensed memory remains representative and does not degenerate; further exploration of diversity-promoting objectives is warranted.
• Scalability to very high-dimensional or multi-modal data: While effective in vision, tabular, and basic text modalities, adapting these frameworks for large-scale, heterogeneous, or multi-source data streams is a prospective target.
• Efficient handling of missing or unreliable data: Robust adaptation to non-i.i.d., missing, or corrupted input remains a challenge, especially in real-world deployment on resource-constrained hardware.

Taken together, incremental sample condensation constitutes a rapidly advancing paradigm for sustaining information efficiency and adaptability in sequential, constrained, and dynamically evolving learning environments (Sangermano et al., 2022, Xu et al., 2024, He et al., 2023, Chen et al., 22 Jan 2026).