Streaming Dataset Distillation

• Streaming dataset distillation is a method that constructs and continuously updates a compact synthetic training memory to approximate the behavior of full-data streaming.
• Techniques like DBC and DIET optimize synthetic images or embeddings using targeted loss functions and bi-level updates to preserve performance under strict memory constraints.
• Empirical results show that distilled datasets can nearly match full-data accuracy while significantly reducing storage and computational overhead.

Searching arXiv for relevant papers on streaming dataset distillation and closely related dataset distillation work. {"query": "\"streaming dataset distillation\" arXiv", "max_results": 10} Streaming dataset distillation denotes the construction and maintenance of a compact synthetic training memory while data arrive sequentially, so that training on the distilled memory approximates training on the underlying stream under explicit memory and computational constraints. In the currently documented formulations, the distilled state is an evolving dataset rather than a static coreset: in streaming image classification it is a fixed-size reservoir of trainable synthetic images updated batch by batch, whereas in recommender systems it is a sequence of embedding–soft-label memories updated stage-wise to preserve long-term training behavior (Khatib et al., 8 Sep 2025, Zhang et al., 26 Mar 2026).

1. Conceptual scope and problem formulation

The streaming variant departs from static dataset distillation by imposing temporal order, bounded storage, and restricted access to historical data. In the image-stream setting, data arrive as a stream of image batches $B_i$; the stream is potentially very large, the algorithm sees each incoming batch once, and a fixed-size reservoir $R_d$ of distilled images of size $S$ is updated in place rather than expanded over time (Khatib et al., 8 Sep 2025). In the recommender setting, the stream is formalized as time-ordered blocks $$\mathcal{D}_{1:T} = \{\mathcal{D}_1,\dots,\mathcal{D}_T\}$$, and the objective is to learn a sequence of synthetic datasets $\mathcal{D}^{syn}_{1:T}$ such that training on these distilled sets approximates the training behavior induced by the full streaming interaction history (Zhang et al., 26 Mar 2026).

A central distinction is that streaming dataset distillation targets an evolving training memory rather than a one-shot synthetic proxy. Static dataset distillation assumes a fixed dataset and produces a single synthetic set once; streaming formulations require the distilled state to co-evolve with new data and to remain consistent with accumulated historical signals (Zhang et al., 26 Mar 2026). The corresponding fidelity target also varies by application. DBC emphasizes accurate downstream classification from a continually updated compact surrogate of the stream, while DIET emphasizes behavioral fidelity, including model performance trends and architecture ranking consistency under continual retraining scenarios (Khatib et al., 8 Sep 2025, Zhang et al., 26 Mar 2026).

This also separates streaming dataset distillation from generic continual learning. Continual learning typically updates model parameters to reduce forgetting; streaming dataset distillation instead constructs data that can later recreate the effect of streaming training, potentially for candidate architectures that were unknown at distillation time (Zhang et al., 26 Mar 2026).

2. Representations, objectives, and memory mechanisms

The distilled object is modality-dependent. DBC performs input-level distillation: it maintains a small set of synthetic or refined images in a reservoir $R_d$, and these images are trainable variables updated via backpropagation so that a CNN produces similar outputs on the distilled reservoir and on the current real batch (Khatib et al., 8 Sep 2025). DIET distills recommender data into synthetic units $(\mathbf{e}, \tilde{\mathbf{y}})$, where $\mathbf{e}$ is an embedding representation under a reference model and $\tilde{\mathbf{y}}$ is a logit vector serving as a soft label (Zhang et al., 26 Mar 2026).

Framework	Distilled state	Optimization target
DBC	Fixed-size reservoir $R_d$ of distilled images	MSE between CNN outputs on $R_d$0 and current batch $R_d$1
DIET	Stage-wise $R_d$2 of embeddings and soft labels	Bi-level objective matching real-data behavior on current block

In DBC, the distillation loss is defined classwise and summed across classes: $R_d$3 with $R_d$4 and $R_d$5. Backpropagation updates both the CNN parameters and the distilled images themselves (Khatib et al., 8 Sep 2025).

In DIET, the streaming formulation is explicitly bi-level. For stage $R_d$6,

$R_d$7

The inner optimization trains on active synthetic memory; the outer meta-loss is computed on hard real data and backpropagated through the inner trajectory using RaT-BPTT-style truncated backpropagation (Zhang et al., 26 Mar 2026).

Taken together, these formulations indicate that streaming dataset distillation is not confined to one representational regime. This suggests that the distilled state can be placed at the raw-input level when the modality admits direct synthetic optimization, or in embedding space when the original features are sparse, high-dimensional, or structurally awkward for direct synthesis.

3. Distillation Based Classification for streaming image data

DBC studies classification of streaming image data under limited memory and computation, using CIFAR-10 as a stream of training batches and a fixed reservoir $R_d$8 of distilled images of size $R_d$9 such as 100, 200, or 500 (Khatib et al., 8 Sep 2025). The method has three main components: $S$0, the primary classifier implemented as an ImageNet-pretrained ResNet-34 and fine-tuned on the distilled memory; $S$1 or $S$2, a smaller CNN such as SimpleCNN or IntermediateCNN for distillation; and the fixed-size reservoir $S$3 itself.

The procedure is interleaving. First, the initial distilled reservoir is built uniformly from the first few batches so that it contains samples of all classes. Then, for each incoming batch $S$4, the algorithm updates the distilled reservoir via the MSE distillation objective, trains the primary classifier $S$5 on the updated $S$6 for several epochs, periodically validates $S$7, and keeps the best model (Khatib et al., 8 Sep 2025). Past real batches are not retained. The streaming constraint is therefore explicit: each raw batch is used once to update the synthetic reservoir, after which only $S$8 is revisited.

The distilled images are updated in place. The paper writes the image update as

$S$9

while noting that the sign convention is informal and that implementation would use the usual gradient-descent update (Khatib et al., 8 Sep 2025). The reservoir size constraint is enforced by choosing a fixed number of distilled images per class and not increasing that number over time.

The empirical comparison is against Hoeffding Trees, Adaptive Random Forest, and Reservoir Sampling Based Classification. HT and ARF do not maintain a compact representative buffer; for image inputs they operate on ResNet-34 embeddings. HT reaches 53.2% test accuracy on CIFAR-10 streaming with 8000 seconds per epoch, while ARF reports 31.28% test accuracy for 10 trees and 38.05% for 30 trees, with training time around 12000 seconds per epoch (Khatib et al., 8 Sep 2025). RBC instead stores real images via stratified reservoir sampling and trains the same ResNet-34 on those stored examples. Its best reported test accuracy is 70.9%, with a training time of 864 seconds for the best setup (Khatib et al., 8 Sep 2025).

DBC outperforms these baselines in accuracy. With reservoir size 100 and learning rate 0.001, it reports 62.53% test accuracy; with reservoir size 200 and learning rate 0.0001, 66.18%; and with reservoir size 500, learning rate 0.0001, and 20 epochs per batch, 73.1% (Khatib et al., 8 Sep 2025). The paper also reports that IntermediateCNN outperforms SimpleCNN with similar runtime, Adam is slightly better than SGD, and smaller learning rates for the distillation model and the main model improve outcomes (Khatib et al., 8 Sep 2025). Concept drift is part of the motivation, but the experiments use standard CIFAR-10 without an explicit drift scenario.

4. DIET and continual distillation for recommender systems

DIET formulates streaming dataset distillation for recommender systems under a continual learning paradigm in which historical behavioral logs are partitioned into chronological blocks and a reference model $$\mathcal{D}_{1:T} = \{\mathcal{D}_1,\dots,\mathcal{D}_T\}$$0 is continually updated on each new block $$\mathcal{D}_{1:T} = \{\mathcal{D}_1,\dots,\mathcal{D}_T\}$$1 (Zhang et al., 26 Mar 2026). The stated motivation is that retraining deep recommender models on full historical data for every architecture comparison or iteration is prohibitively expensive, especially at industrial scales where total FLOPs can reach $$\mathcal{D}_{1:T} = \{\mathcal{D}_1,\dots,\mathcal{D}_T\}$$2 and GPU hours can reach $$\mathcal{D}_{1:T} = \{\mathcal{D}_1,\dots,\mathcal{D}_T\}$$3 on an RTX 4090 at realistic MFU (Zhang et al., 26 Mar 2026).

DIET has two phases at each stage. Phase 1 initializes a boundary memory by selecting influential real samples from the current block using task-conditioned EL2N scores, converting them into embeddings and teacher logits, and fusing them with an aligned subset of historical synthetic memory (Zhang et al., 26 Mar 2026). Samples are grouped by multi-task label vector $$\mathcal{D}_{1:T} = \{\mathcal{D}_1,\dots,\mathcal{D}_T\}$$4, EL2N is computed as

$$\mathcal{D}_{1:T} = \{\mathcal{D}_1,\dots,\mathcal{D}_T\}$$5

and the method keeps upper-middle EL2N samples within each label group rather than extreme outliers (Zhang et al., 26 Mar 2026).

Phase 2 performs influence-aware memory addressing and bi-level optimization. DIET defines an influence score

$$\mathcal{D}_{1:T} = \{\mathcal{D}_1,\dots,\mathcal{D}_T\}$$6

which is used first to identify hard real samples with low alignment deficiency and then to identify active synthetic samples with high responsibility for those hard targets (Zhang et al., 26 Mar 2026). The proxy model inner loop starts from $$\mathcal{D}_{1:T} = \{\mathcal{D}_1,\dots,\mathcal{D}_T\}$$7 for $$\mathcal{D}_{1:T} = \{\mathcal{D}_1,\dots,\mathcal{D}_T\}$$8, enforcing stage-aware initialization; the outer update refines the synthetic data via the meta-loss on hard real data. Historical synthetic samples are retained or discarded by influence-based alignment weights $$\mathcal{D}_{1:T} = \{\mathcal{D}_1,\dots,\mathcal{D}_T\}$$9, which implements controlled forgetting while keeping memory bounded (Zhang et al., 26 Mar 2026).

The empirical results emphasize compression, transfer, and ranking fidelity. On KuaiRand, Tmall, and Taobao, representative compression ratios are 1.5%, 1.39%, and 1.68%, respectively (Zhang et al., 26 Mar 2026). At these ratios, DIET yields AUC extremely close to full-data training and better than random selection, EL2N selection, and k-means subset coresets. Examples reported in the paper include KuaiRand with a DCN candidate, where full-data and DIET both achieve 0.7878 AUC; Tmall with a WuKong candidate, where full data achieves 0.7629 and DIET 0.7617; and Taobao with a DCN candidate, where full data achieves 0.7146 and DIET 0.7119 (Zhang et al., 26 Mar 2026). The paper also reports that on Tmall, using as little as 0.3% distilled data recovers $\mathcal{D}^{syn}_{1:T}$0 of full-data AUC for WuKong, and that on KuaiRand at 3% distillation ratio with a strong teacher, DIET slightly exceeds full-data performance, which the authors interpret as implicit denoising (Zhang et al., 26 Mar 2026).

A major result is cross-architecture reuse. DIET distilled with DCN is used to train DCN, DCNv2, FinalNet, FinalMLP, and WuKong, and the paper reports that performance is similar across different teachers, with model capacity mattering more than architectural match (Zhang et al., 26 Mar 2026). The claimed systems implication is a reduction in model iteration cost by up to $\mathcal{D}^{syn}_{1:T}$1, because expensive historical-data training can be amortized into a distilled stream reused across many candidate models (Zhang et al., 26 Mar 2026).

5. Comparative interpretation across domains

Across the two formulations, streaming dataset distillation has three recurring properties. First, the memory is explicit and bounded: $\mathcal{D}^{syn}_{1:T}$2 in DBC has fixed size $\mathcal{D}^{syn}_{1:T}$3, while DIET retains only a selected aligned subset of prior synthetic memory and fuses it with current-stage synthetic units (Khatib et al., 8 Sep 2025, Zhang et al., 26 Mar 2026). Second, the synthetic state is optimized rather than merely sampled. RBC stores real images selected by stratified reservoir sampling, but DBC stores synthetic images optimized by gradient descent to align CNN outputs with real batches; DIET similarly treats embeddings and logits as learnable synthetic parameters refined by meta-gradients (Khatib et al., 8 Sep 2025, Zhang et al., 26 Mar 2026). Third, the update process is coupled to the stream: DBC interleaves distillation and classifier training per batch, whereas DIET interleaves reference-model evolution, memory fusion, and stage-wise bi-level optimization per temporal block.

The relation to classical baselines clarifies what streaming distillation is not. It is not equivalent to direct incremental learning of model parameters, as in Hoeffding Trees or Adaptive Random Forest, because those methods do not maintain a trainable representative memory (Khatib et al., 8 Sep 2025). It is not equivalent to reservoir sampling, because a sampled subset of real data lacks an optimization mechanism targeted at preserving training-critical signals (Khatib et al., 8 Sep 2025). Nor is it equivalent to generic continual learning, because the output of the method is a reusable distilled dataset rather than only an updated parameter state (Zhang et al., 26 Mar 2026).

The two papers also expose different fidelity criteria. DBC measures downstream classification accuracy on a held-out CIFAR-10 test set and shows that an optimized synthetic reservoir can outperform both streaming trees and a reservoir-sampling baseline under comparable memory budgets (Khatib et al., 8 Sep 2025). DIET evaluates whether distilled data preserve full-training performance trends, candidate-model rankings, and near-full AUC at 1–2% compression, thereby treating distilled data as a reusable foundation for model iteration rather than merely a one-model surrogate (Zhang et al., 26 Mar 2026). This suggests that the evaluation of streaming dataset distillation should be task-dependent: predictive fidelity is central in some settings, while training-dynamics fidelity may be the dominant criterion in architecture-search or continual-development pipelines.

6. Limitations, unresolved questions, and research directions

The current literature remains largely empirical. DBC provides no formal convergence guarantee, no approximation bound for how well the distilled reservoir approximates the full stream, and no regret or excess-risk bound for the resulting classifier (Khatib et al., 8 Sep 2025). The only explicit theoretical component in that paper is the Hoeffding bound used to characterize HT split decisions. DIET introduces a more elaborate bi-level formulation and an influence-based addressing rule, but its contribution is likewise methodological and experimental rather than theorem-driven (Zhang et al., 26 Mar 2026).

The empirical settings also impose clear limits. DBC evaluates only standard CIFAR-10, with no explicit concept-drift scenario, no larger or higher-resolution datasets, and no study of heavily imbalanced or rare-class streams (Khatib et al., 8 Sep 2025). Its performance improves markedly with reservoir size, from 62.53% at size 100 to 73.1% at size 500, indicating a memory–accuracy trade-off that is not fully characterized (Khatib et al., 8 Sep 2025). DIET, although much closer to industrial scale, assumes stage-wise access to historical blocks and the availability of a continually trained reference model; extreme drift, cold-start items, and heavy distribution shifts are not explicitly covered (Zhang et al., 26 Mar 2026).

Computational overhead is a second recurring limitation. DBC is more expensive than RBC because it backpropagates through both the distillation CNN and the stored inputs for each batch, even though it remains much faster than HT and ARF in the reported experiments (Khatib et al., 8 Sep 2025). DIET is explicitly more expensive than ordinary training during the distillation phase because it combines influence computations, inner-loop training, and outer-loop meta-optimization; in the Taobao example, full training per epoch takes 5.3 minutes and 16.1 GB, distillation takes 10.0 minutes and 51.9 GB as a one-time cost, and training on distilled data takes 0.2 minutes and 16.1 GB (Zhang et al., 26 Mar 2026). The practical premise is amortization across many downstream trials.

Several open directions follow directly from the reported limitations. The image-stream work identifies refining image distillation, testing across various data streams, tailoring CNN architectures for streaming constraints, and improving efficiency and scalability as future directions (Khatib et al., 8 Sep 2025). The recommender work points toward improved influence approximations, scaling to foundation recommender models, and possible privacy-oriented uses of synthetic embeddings and logits, although privacy and compliance are not explored experimentally (Zhang et al., 26 Mar 2026). A plausible implication is that future streaming dataset distillation research will be shaped less by whether data are synthetic and more by how the synthetic memory is aligned with evolving training dynamics under strict resource budgets.