Omnimodal Dataset Distillation via High-order Proxy Alignment

Abstract: Dataset distillation compresses large-scale datasets into compact synthetic sets while preserving training performance, but existing methods are largely restricted to single-modal or bimodal settings. Extending dataset distillation to scenarios involving more than two modalities, i.e., Omnimodal Dataset Distillation, remains underexplored and challenging due to increased heterogeneity and complex cross-modal interactions. In this work, we identify the key determinant that bounds the endpoint discrepancy in the omnimodal setting, which is exacerbated with an increasing number of modalities. To this end, we propose HoPA, a unified method that captures high-order cross-modal alignments via a compact proxy, which is compatible with trajectory matching as well. By abstracting omnimodal alignment with a shared similarity structure, our method avoids the combinatorial complexity of pairwise modality modeling and enables scalable joint distillation across heterogeneous modalities. Theoretical analysis from the spectral perspective reveals the rationality of our proposed method against bimodal dataset distillation techniques. Extensive experiments on various benchmarks demonstrate that the proposed method achieves superior compression-performance trade-offs compared to existing competitors. The source code will be publicly released.

• The paper presents the HoPA framework that uses high-order proxy alignment via SVD to compress omnimodal datasets while preserving key inter-modal dependencies.
• It demonstrates significantly reduced endpoint discrepancy and improved retrieval performance, maintaining over 95% of benchmark performance under high compression.
• The method offers computational efficiency, scalable distillation beyond pairwise methods, and robust transfer across diverse multimodal architectures.

Omnimodal Dataset Distillation via High-order Proxy Alignment

Problem Motivation and Setting

The paper systematically examines the limitations of dataset distillation in the context of multimodal learning, specifically extending the paradigm from unimodal and bimodal regimes to the omnimodal case involving three or more modalities. Existing dataset distillation methodologies predominantly employ pairwise trajectory or gradient matching objectives, which cannot explicitly preserve higher-order relational structures as the number of modalities grows. This pairwise strategy does not scale: both computational and representational issues arise due to an $\mathcal{O}(k^2)$ pairwise explosion and an inability to holistically model inter-modality dependencies for large $k$.

The authors define "omnimodal dataset distillation" as compressing a multimodal dataset spanning $k\geq3$ modalities into a synthetic compact set while preserving downstream performance and alignment among all modalities. As $k$ increases, the alignment challenge becomes nontrivial due to increased heterogeneity and complexity in interaction patterns among different modalities. Theoretical analysis highlights that the endpoint discrepancy between a student (trained on synthetic data) and teacher (trained on real data) model grows with the number of modalities unless explicit higher-order modeling is introduced.

High-order Proxy Alignment (HoPA) Methodology

The central contribution is HoPA, a framework that addresses dataset distillation for omnimodal data via a high-order proxy alignment mechanism. Rather than considering all pairwise similarities, HoPA constructs a proxy for the omnimodal correlation structure by leveraging the leading singular component of the Gram matrix of normalized modality embeddings, extracted via SVD.

The approach can be summarized as follows:

1. Multimodal Representation and Alignment Proxy: For a given instance with $k$ modalities, the representations are stacked into a $k\times d$ matrix, and its Gram matrix $\mathbf{G}$ is computed. The leading singular value $\sigma_1$ and its associated right singular vector $\mathbf{v}_1$ (principal direction) capture the dominant cross-modal alignment and serve as a low-rank proxy.
2. Trajectory Matching: As in previous work, HoPA uses trajectory matching: student and teacher models are rolled out from the same initialization under synthetic and real data, respectively, and the synthetic set is optimized to minimize the parameter endpoint discrepancy. HoPA extends this by directly matching training dynamics using the shared omnimodal proxy rather than modality pairs.
3. Loss Construction: The inner objective consists of (a) a modality-level loss that encourages intra-instance alignment by maximizing $\sigma_1$, driving all modalities towards a consensus direction; and (b) an instance-level weighted BCE loss acting on instancewise similarities using the proxy. This dual-objective prevents representation collapse and improves discriminability.

Immediately after computing the proxy and aligning trajectories, the Gram matrix decomposition and optimization via SVD are central components:

Figure 1: The overall framework for omnimodal distillation employing SVD on multimodal representations and utilizing the leading singular vector as a compact proxy for alignment.

Theoretical Analysis

The paper offers rigorous theoretical support for HoPA, focusing on the spectral properties of the alignment proxy and the implications for endpoint discrepancy.

• Trajectory Upper Bound: The endpoint parameter distance between student and teacher is proven to be upper-bounded by the accumulation of modality-wise gradient mismatches, with a critical dependence on the structure of the inner objective.
• Spectral Selectivity vs. Full-spectrum Losses: Pairwise objectives activate all singular modes of the Gram matrix (full-spectrum), accumulating gradient mismatches from all directions, especially those not aligned along principal semantic axes. By contrast, HoPA focuses on the leading mode: the proxy is spectrally selective and ignores the non-principal modes. This is theoretically shown to yield a strictly tighter certified upper bound on the trajectory matching error—especially relevant as $k$0 increases, where pairwise methods incur an $k$1 penalty in error accumulation.
• Gradient Decomposition: The optimization landscape induced by HoPA's objective decomposes into (i) an intra-instance centripetal alignment force—maximizing $k$2 aligns all modalities of an instance to its principal direction—and (ii) an inter-instance discriminative force—ensuring that each distilled instance remains semantically unique by penalizing proxy similarity mismatches.

Experimental Validation

Extensive empirical results are presented using three established tri-modal benchmarks: MSR-VTT (video, audio, text), VGGSound-S (video, audio, text), and DiDeMo (video, audio, text). The evaluation focuses on cross-modal retrieval performance (Recall@K) when models are trained exclusively on the synthetic set produced by each method. The main claims substantiated numerically are:

• Superiority under High Compression: HoPA consistently surpasses all considered baselines (including MTT-VL, TESLA, LoRS, RepBlend) for each data regime across datasets, especially at high-compression (e.g., with only 1-5% data retained).
• Compression–Performance Trade-off: For instance, on DiDeMo with $k$3 synthetic triplets, HoPA's average R@1 and R@5 match or exceed the best baseline using $k$4. Across compression ratios, HoPA preserves ≥95% of the performance achieved by models trained on the full real dataset.
• Computation Efficiency: Although introducing SVD-based steps, HoPA delivers at least a $k$5 speedup compared to repeated pairwise baselines due to more efficient updates on the alignment proxy, as per reported wallclock timings.
• Cross-architecture Generalization: Synthetic sets distilled by HoPA effectively transfer to different downstream architectures (e.g., LanguageBind, OmniBind), demonstrating architectural invariance.
• Ablation Analyses: Removal of either the modality-level or instance-level components yields clear degradation, supporting the necessity of both. Replacing the rank-1 proxy with a higher rank version consistently drops performance, empirically validating the spectral selectivity principle.

In-depth ablation studies further demonstrate the instability or degradation of pairwise distillation as query budget grows, whereas HoPA remains robust and scales better with increasing synthetic set size.

Visualization and Qualitative Analysis

The paper provides case visualizations of distilled omnimodal instances. Synthetic representations robustly preserve semantic cross-modal correspondence and exhibit interpretable patterns (e.g., stylized "DeepDream" video frames, meaningful audio-text pairs), even under severe data compression.

Figure 2: Visualization of original and distilled samples (video, audio, text) for representative cases in VGGSound-S. The synthetic data maintains global semantics and alignment structure.

Figure 3: Additional distilled examples exhibiting retained cross-modal coherence across video, sound, and language modalities.

Implications and Future Directions

Practical Significance

HoPA demonstrates that omnimodal dataset distillation is feasible at scale and enables aggressive compression across more than two modalities without loss of critical alignment information. This has major ramifications for multilingual, multimedia, and embodied learning systems where storage, privacy, and computation are constraining factors. Efficient distillation facilitates transfer learning, federated learning, and privacy-preserving deployment scenarios spanning video, audio, text, and potentially other sensor modalities.

Theoretical Contributions

The spectral alignment perspective formalized in this work provides a systematic account of the destructiveness of full-spectrum pairwise objectives as modality count increases. Single-mode selectivity—aligning only the leading component—is formally justified to offer the best attainable matching guarantee, thus suggesting that much of the redundancy in cross-modal interaction spaces is superfluous for downstream task retention. This framework can underpin future generalized data selection and summarization techniques in high-dimensional multimodal learning.

Prospects

Future progress may involve extending HoPA using scalable or approximate SVD methods, guaranteeing efficiency for even larger $k$6 or longer sequence-to-sequence omnimodal tasks. Additionally, integrating generative parameterizations (e.g., with diffusion models or GANs) for direct synthesis in data space, rather than embedding space, is a promising avenue for applications requiring interpretable proxy data. Finally, relaxing the alignment proxy to dynamically weight and select information from minor modes—without incurring full-spectrum costs—offers a natural refinement path.

Conclusion

HoPA provides a unified, scalable, and theoretically justified framework for omnimodal dataset distillation. By exploiting a high-order proxy derived from the principal singular mode of the data Gram matrix, it obviates the combinatorial cost and representational limitations of pairwise approaches, yielding tighter bounds on student–teacher discrepancy and superior empirical performance. The work marks a significant advance in data-efficient multimodal learning and lays a rigorous foundation for further exploration of spectral objectives in dataset distillation.