Explain the model-size-dependent word-class alignment gap

Explain why DINOv3-H exhibits substantially stronger alignment between rendered-word directions and corresponding visual class directions than the larger DINOv3-7B model, despite DINOv3-H having been distilled from DINOv3-7B.

Background

The paper reports that DINOv3-H outperforms DINOv3-7B on matched cosine similarity and word-to-class and class-to-word retrieval, even though DINOv3-7B is larger and DINOv3-H was distilled from it. Other large models, including Web-DINO-1B and DINOv2-G, also lack comparable word-class alignment, indicating that parameter count alone does not explain the behavior.

The mechanism responsible for this discrepancy is explicitly unresolved. Establishing its cause could clarify how model scale, distillation, architecture, training data, or optimization affect the emergence of semantic alignment in vision transformers.

References

We cannot explain this gap. Web-DINO-1B and DINOv2-G, at a billion parameters or more, show no word-class alignments, so size alone does not determine model behavior.

— Vision Transformers Learn Word Meaning from Images Alone  (2610.13252 - Gjølbye et al., 9 Oct 2026) in Section 5, subsection “Model size”