Determine how token reduction should adapt to distribution shift

Determine whether and how token reduction in vision transformers should differ under distribution shift, beyond schedules tuned for clean, in-distribution accuracy.

Background

The paper identifies a gap in the study of robustness for token-reduced vision transformers. Existing efficiency methods are primarily tuned for clean-data accuracy, while their behavior and appropriate design under distribution shift remain insufficiently characterized. The paper addresses this gap specifically through depth allocation: it shows that a late-concentrated token-removal schedule improves out-of-distribution accuracy relative to the conventional flat schedule. The broader question of how token reduction should differ under shift is nevertheless presented as open, encompassing possible changes beyond the static schedule studied here.

References

The robustness of token-reduced ViTs, however, is largely unexamined: efficiency methods are tuned for clean accuracy, and whether, and how, token reduction should differ under shift is open.

— Later Is Better: Token Reduction for ViTs Under Distribution Shift  (2610.07758 - Cha et al., 6 Oct 2026) in Section 2, Related Work, paragraph “Robustness to distribution shift”