- The paper introduces a first-order dataset distillation method that reduces computational overhead by sampling item subsets and using trajectory-anchored parameter resets.
- It employs distributional hypothesis regularization to maintain semantic consistency in synthetic text sequences, ensuring reliable tied embedding performance.
- Empirical results across benchmarks like Amazon and Yelp show up to 25× memory reduction and competitive downstream recommendation performance with minimal synthetic data.
FOSTER: First-order Dataset Distillation for Text-based Sequential Recommendation
Problem Context and Motivation
The advent of text-based sequential recommender systems, which utilize pretrained LLMs for semantic item encoding, has enhanced the adaptability and effectiveness of recommendation algorithms, particularly in scenarios involving cold starts. However, these benefits come at a steep computational cost due to the expansion of data and the requirement for complex text encoders. Traditional strategies for dataset reduction, such as coreset selection or dataset distillation, face significant scalability and efficiency challenges when applied to text-based sequential recommendation with large, discrete item spaces. Existing distillation methods largely rely on bi-level optimization, which becomes prohibitive as catalog or token sizes increase. Moreover, the preservation of co-occurrence semantics critical for models employing tied embeddings is largely unaddressed, limiting the transferability and representativeness of synthetic data.
Methodological Contributions
FOSTER introduces a novel solution to dataset distillation for text-based sequential recommendation through three key innovations:
- Stochastic Item Subset Sampling: Rather than computing over the full item set at every optimization step, FOSTER samples small random subsets of items for both forward and backward passes. This substantially lowers computational and memory requirements, akin to negative sampling, and makes distillation tractable even in very large-scale setups.
- First-order Optimization with Trajectory-anchored Parameter Reset: FOSTER leverages a reformulation of the typical bi-level optimization as a single-level constrained problem, inspired by BOME. The method circumvents expensive computation of higher-order gradients by relying strictly on first-order methods. To prevent degenerate solutions associated with joint optimization of model and synthetic data, FOSTER periodically resets the model parameters to pre-recorded checkpoints sampled from the actual training trajectory, thereby ensuring greater generalizability and preventing overfitting of the distilled data to a single model parameterization.
- Distributional Hypothesis Regularization: To ensure the utility of tied embeddings, FOSTER incorporates a regularization term that enforces alignment between semantic similarity and conditional co-occurrence statistics in the synthetic sequences. This regularization penalizes discrepancies between the way semantically similar items behave in context and their empirical co-occurrence rates in the distilled data, directly addressing mismatches that degrade downstream learning.
Experimental Results
FOSTER's experimental evaluation is extensive, spanning three real-world benchmarks: Amazon Games, Amazon Foods, and Yelp. The key findings are as follows:
- Efficiency and Scalability: FOSTER demonstrates drastic reductions in computational footprint. For instance, it achieves up to 25× lower memory usage and an order-of-magnitude speedup per training epoch compared to previous methods such as TD3 when scaling to large vocabularies.
- Distillation Quality: With as few as 20–60 synthetic sequences, FOSTER achieves downstream recommendation metrics (Recall@K, NDCG@K) that closely approach or even occasionally surpass full-data training, especially in less tail-heavy domains such as Games and Foods. On Yelp, which presents a more challenging long-tail profile, there remains a small but consistent gap between synthetic and full training performance.
- Ablation and Transfer Studies: Systematic ablation studies validate the necessity of trajectory-anchored resets and distributional hypothesis regularization. Disabling these components leads to marked performance degradation. Transfer learning experiments, where synthetic data distilled using a lightweight backbone (TinyBERT) is used to train a large downstream LLM (Qwen3-4B), show performance that nearly matches full-data training in most settings and consistently exceeds random selection.
- Hyperparameter Sensitivity: The performance is moderately sensitive to reset intervals and the distribution regularization weight, and optimal values depend on the data domain. The number of steps per reset and items sampled per step both require practical tuning, though FOSTER's efficiency makes such sweeps tractable.
Theoretical and Practical Implications
FOSTER's first-order framework significantly reduces the prohibitive bi-level optimization costs which have so far limited dataset distillation in text-based sequential recommendation, enabling realistic deployment in industry-scale scenarios. Regularization targeting the distributional hypothesis offers a principled way to reconcile synthetic-data compression with the semantic consistency required by shared embedding architectures. Empirically, FOSTER's distilled synthetic data maintains transferability across model architectures, indicating that it preserves informative structural and semantic properties of the original dataset.
On the theoretical front, FOSTER advances understanding of the interaction between optimization-level approximations (first-order methods) and representational alignment (distributional regularization) in the context of highly discrete, large-label-space tasks.
Limitations and Prospects for Future Research
Two main limitations are acknowledged. First, hyperparameter tuning, especially for reset intervals and regularization strength, remains essential for peak performance, motivating research into adaptive or meta-learned tuning strategies. Second, while transferability to LLMs is promising, performance gaps persist on long-tailed domains and when the distillation/model architectures differ considerably, suggesting room for architectural innovations or more expressive synthetic data parameterizations.
Future research may address:
- Tightening the generalization gap in tail-heavy recommendation settings
- Integrating the distillation objective more tightly with downstream architecture selection
- Exploring further regularization schemes or dynamic curriculum distillation strategies for extreme-scale or continual learning contexts
Conclusion
FOSTER represents a substantial step forward in efficient data condensation for text-based sequential recommender systems, combining first-order optimization strategies with semantic-preserving regularization. The method achieves high data efficiency, scalability, and broad transferability, offering both theoretical insights and immediate practical value for deploying large-scale recommendation models that rely on expensive semantic encoders (2605.30772).
