Generalization of mechanism evidence beyond contrastive scorers

Determine whether the residual-geometry and modality-gap explanation for Orthogonal Matching Pursuit’s behavior persists when frame selection uses image-text matching heads, multimodal-language-model attention, or subtitle-aware scorers rather than frozen contrastive image-text embeddings.

Background

The paper’s explanation of OMP’s behavior is based primarily on residual geometry and Gram statistics computed from frozen LongCLIP stem embeddings, with a limited selector-ordering check using SigLIP. Both encoders use a contrastive image-text objective, so the observed modality gap may be specific to that scorer family.

The paper explicitly identifies image-text matching heads, multimodal-language-model attention, and subtitle-aware selection as untested alternatives. Establishing whether the same mechanism applies across these scoring paradigms would determine whether the explanation is a property of OMP or an artifact of the embedding space.

References

The selector ordering survives a swap to SigLIP (\S\ref{sec:scorer}), but both encoders share a contrastive image--text objective; BLIP-ITM heads, MLLM-attention scorers, and subtitle-aware selection remain untested, and the modality-gap explanation in \S\ref{sec:mechanism} is a property of this embedding space rather than of pursuit selection in general.

— Select, Compress, Reinvest: A Controlled Study of Visual-Token Allocation in Long-Video MLLMs  (2609.03820 - Khatri, 3 Sep 2026) in Section 6, “Limitations,” paragraph “Mechanism evidence is encoder-specific”