Rank-Aware Hyperbolic Alignment for Vision-Language Dataset Distillation

Abstract: Vision-language dataset distillation (VLDD) compresses a large image-text paired dataset into a small set of synthetic pairs that can efficiently train contrastive vision-LLMs under strict data and compute budgets. Most existing methods match expert trajectories or cross-modal statistics, yet still enforce full-dimensional alignment in a Euclidean embedding space. This is often overly restrictive due to rank-deficient image--text correlation, with shared semantics concentrated in a low-dimensional range and remaining variation spread across a weakly correlated residual subspace. LoRS relaxes alignment at the similarity level by low-rank factorization, but does not explicitly control dominant alignment capacity and structure in the representation space. We thus propose a rank-aware hyperbolic alignment (RAHA) that combines hierarchical geometry with explicit alignment-capacity control. RAHA lifts multimodal representations to hyperbolic space and optimizes distilled pairs with asymmetric objectives that enforce geodesic alignment in the shared range while regularizing the residual subspace to preserve modality-private diversity and improve transfer robustness. Experiments on benchmarks show that RAHA demonstrates competitive cross-modal retrieval and improved transfer indicators under fixed budgets.

• The paper introduces RAHA, which combines hyperbolic contrastive learning with adaptive range–residual alignment to preserve dominant shared image–text semantics while suppressing weakly coupled variation.
• RAHA outperforms key distribution-matching baselines at moderate and large budgets, reaching mean Recall gains such as 30.7 versus 25.9 on Flickr8k with 500 pairs and 18.6 versus 14.2 on COCO with 1,000 pairs.
• The method improves cross-architecture transfer and perturbation robustness, but costs roughly seven times more per distillation iteration than CovMatch and can underperform trajectory matching at extreme compression such as 100 pairs.

Motivation and problem setting

Vision-language dataset distillation (VLDD) compresses a large paired image–text corpus into a small synthetic set that can train contrastive retrieval models under tight data and compute budgets. The paper identifies a structural weakness shared by prior VLDD families—trajectory matching (MTT-VL, LoRS, RepBlend), generative synthesis (EDGE), and distribution-statistics matching (CovMatch): all enforce alignment largely uniformly across feature directions in Euclidean space, even though image–text correlation is effectively low-rank, with dominant semantics concentrated in a small subspace and modality-private variation spread across a weakly correlated residual (2606.29464). Under extreme compression (e.g., 100 pairs on COCO is below 0.1% of training images), uniformly aligning this residual can suppress complementary information and harm transfer.

The proposed method, RAHA (Rank-Aware Hyperbolic Alignment), addresses both concerns jointly: it lifts multimodal representations to a Lorentz hyperboloid to exploit hierarchical semantic structure, and it decomposes cross-modal correlation into an adaptive-rank "range" subspace and a "residual" complement, aligning only where coupling carries semantic content.

Method

RAHA optimizes a compact set of synthetic pairs—learnable image pixels and continuous text token embeddings (following CovMatch's differentiable text parameterization)—against frozen pretrained NFNet/BERT encoders with linear projection heads, using three loss terms.

Hyperbolic contrastive alignment (hITC). Projected embeddings are lifted onto the Lorentz hyperboloid via the exponential map with curvature $c$ and scale $s$, and logits are negative geodesic distances scaled by temperature $\tau=0.07$. This replaces Euclidean InfoNCE and provides an inductive bias for coarse-to-fine caption structure; unlike MERU-style training, no literal tree assumption is imposed.

Range–residual relevance distillation. Features are mapped back to the tangent space at the origin via the logarithmic map—a round-trip that applies a curvature-dependent radial reweighting concentrating large-norm features—and batch-level tangent-space cross-covariances $C_{\mathrm{real}}$ and $C_{\mathrm{syn}}$ are computed. The real cross-covariance is factorized by SVD, and the effective rank $k$ is chosen adaptively as the smallest rank capturing at least $\rho=0.95$ of squared singular-value energy. Range coordinates are matched from real to synthetic via row-wise relevance distributions passed through entropy-regularized optimal transport (Sinkhorn), bidirectionally averaged. Two regularizers complete the objective: a one-sided energy penalty ensuring synthetic range coupling does not collapse below real levels, and a residual compression term ($r + \max(0, r-1)$) that shrinks out-of-range coupling energy while penalizing cases where residual energy exceeds range energy. Real features are stop-gradient targets throughout, and the total loss weights range, residual, and compression terms by $\lambda_{\mathrm{range}}=0.8$, $\lambda_{\mathrm{residual}}=0.4$, $s$0.

Distillation follows the online protocol of CovMatch: per-iteration encoder reset to pretrained weights, 50 outer synthetic-update steps interleaved with 1 inner real-data model step, up to 200 iterations.

Retrieval results

On Flickr8k, Flickr30k, and COCO with budgets of 100/200/500 pairs, RAHA consistently beats coreset selection and trajectory-matching baselines at larger budgets while remaining competitive throughout. Representative mean Recall scores:

Dataset	Pairs	Best baseline	RAHA
Flickr8k	200	22.1 (LoRS)	25.3
Flickr8k	500	25.9 (CovMatch)	30.7
Flickr30k	500	31.6 (LoRS)	32.9
COCO	500	13.5 (LoRS)	13.7

At 100 pairs on Flickr30k and COCO, however, RAHA trails LoRS (20.7 vs. 27.4 and 7.2 vs. 9.4 respectively). The authors state plainly that a very small synthetic set may lack capacity to materialize the decomposed semantic modes even when the basis itself is stable, so MTT-style methods remain strong at the smallest budgets. At extended budgets of 1000 pairs, RAHA improves over CovMatch on all three datasets (e.g., Flickr8k 37.1 vs. 30.6; COCO 18.6 vs. 14.2) and over EDGE wherever EDGE numbers exist. On CC3M-595K-LLaVA, a broader and noisier source, the gap widens with budget: mean score 4.0 vs. 4.0 at 100 pairs but 8.1 vs. 5.1 at 500 pairs. Prompted zero-shot classification at 100 pairs shows gains over CovMatch on CIFAR-100 (+1.05 pp), CUB-200-2011 (+1.90 pp), and Stanford Cars (+2.17 pp), with the largest gains on fine-grained taxonomic datasets—consistent with the hierarchical inductive bias.

A reproducibility caveat bears directly on these comparisons: the released CovMatch artifacts do not match its published hyperparameters for several settings, and reproduced CovMatch numbers fall well short of published ones (e.g., Flickr30k 200-pair: 22.0 reproduced vs. 34.4 published). Under the reproduced protocol, RAHA wins at 200 and 500 pairs everywhere; against published numbers, the comparison is closer.

Transfer, robustness, and ablations

Cross-architecture transfer (retraining with swapped vision backbones or DistilBERT text encoder) is where RAHA's design intent shows most clearly. On Flickr8k, mean transfer rises from 7.2 (CovMatch) to 8.7 at 200 pairs and from 8.7 to 12.7 at 500 pairs, with consistent gains across BERT/DistilBERT targets and across NFNet/NF-ResNet/NF-RegNet/ViT-B backbones. Robustness degradation $s$1 under JPEG, bit quantization, AWGN, and PGD perturbations is also lower for RAHA at 200 and 500 pairs on both modalities. The implication is that matching rank-adaptive relevance rather than full Euclidean statistics reduces overfitting to the source encoder geometry.

Ablations isolate the components: hITC alone is a strong baseline; adding range matching gives the largest single improvement; residual matching alone is weak; the combination is best. A geometry ablation is particularly informative—an all-Euclidean variant collapses to IR/TR/RMean = 1.4/3.0/2.2, a mixed variant with Euclidean ITC reaches 19.9, and the fully hyperbolic variant reaches 20.4. The explicit subspace decomposition is thus ineffective without hyperbolic lifting, indicating the gain arises from the interaction of geometry and selective supervision rather than either alone. Rank selection is stable: $s$2 is optimal (using $s$3 degrades by absorbing the low-energy tail), and the selected rank stays within a dataset-specific bound across iterations.

Qualitatively, RAHA-distilled images show cleaner textures than CovMatch, which retains high-frequency speckling and banding artifacts consistent with satisfying second-order statistics through structured noise; radial-gap analysis further shows RAHA places matched image–text pairs at more compatible hyperbolic depths.

Cost

RAHA's per-step overhead is substantial: approximately 400 s versus 55 s per distillation iteration for CovMatch at batch size 64 on an RTX A6000 (roughly 1.3 vs. 0.14 GPU-hours for a full Flickr8k run), dominated by the SVD on the $s$4 cross-covariance ($s$5) and Sinkhorn iterations. Peak memory is identical (~9.3 GB). The authors argue the cost is a one-time offline expense amortized over downstream reuse, and that it avoids the ~18 GB trajectory-checkpoint storage of MTT-style methods—but they concede CovMatch remains preferable when wall-clock efficiency is primary.

Limitations and open questions

The paper concedes several boundaries. Performance is bounded by teacher-encoder expressivity and may degrade under domain shift or noisy captions. The hyperbolic prior assumes hierarchical structure; datasets with weak hierarchy may see limited gains, and if useful signal is distributed across many weak directions, residual suppression can discard task-relevant information. On fairness, the authors note explicitly that if harmful correlations concentrate in dominant singular directions, the range-preserving objective will retain them, and no bias-detection mechanism is provided. Open questions include whether truncated/randomized SVD and warm-started Sinkhorn can remove the compute bottleneck, and whether structure-aware distillation can be combined with explicit fairness auditing of the preserved subspace.

Conclusion

RAHA formulates VLDD as hyperbolic contrastive learning coupled with rank-adaptive range–residual relevance distillation, allocating alignment capacity selectively to dominant shared directions while regularizing modality-private residuals. It does not claim uniform dominance at extreme compression—at 100 pairs trajectory-matching baselines can still win—but it delivers the best results among distribution-matching methods as budget grows, along with measurably better cross-architecture transfer and perturbation robustness, at the cost of a roughly sevenfold increase in distillation-step wall-clock time relative to CovMatch.