Determine whether the robustness deficit in attention transfer is feature-borne

Determine whether the robustness deficit of vision-transformer students trained by attention-map distillation arises from the student learning its own features rather than from a failure to transfer the teacher's attention routing.

Background

The paper studies ViT-S students trained either from scratch, by weight transfer and fine-tuning, or by distilling only a self-supervised teacher's attention maps. It finds that attention transfer reproduces the teacher's visible attention structure with high fidelity, while robustness under distribution shift can remain lower than in fine-tuned students at rule-governed endpoints.

These observations support, but do not directly establish, the earlier conjecture that the missing robustness is carried by learned features rather than by attention routing. The authors explicitly characterize their evidence as elimination plus intervention and acknowledge that the deficient features themselves have not been exhibited, leaving the feature-borne account unresolved beyond the measured regime.