Joint affinity matching as an explanation for the superiority of Tri-Affinity Distillation

Establish whether matching all three image–image, text–text, and image–text affinity relations in Tri-Affinity Distillation is necessary because matching only a subset allows the unmatched relations to absorb task-driven drift and distort the joint image–text geometry.

Background

Tri-Affinity Distillation (TAD) combines image–image, text–text, and cross-modal affinity losses. Ablation results show that removing any one component substantially reduces performance, while the full objective performs best.

The paper offers an explicitly labeled conjectural explanation: constraining only some affinity relations may leave the unconstrained relations free to absorb adaptation-induced drift, thereby damaging the overall multimodal geometry. This explanation is presented as a conjecture rather than a proved result.

References

We conjecture that matching only a subset of the affinity relations pins part of the CLIP geometry and lets the unmatched relations absorb the task-driven drift, which distorts the joint image–text structure more than leaving all relations free.

— COMET: Decoupled Distillation, Routing, and Capacity Control for Task-Agnostic Continual Vision--Language Learning  (2610.22907 - Woo et al., 16 Oct 2026) in Appendix G, “TAD Sub-Component Decomposition,” p. 21