REDI: Corpus Aware Patch Ranking for DINOv3 Token Reduction

Abstract: Most token reduction methods for Vision Transformers seek favorable tradeoffs between accuracy and efficiency by pruning, merging, or pooling patch tokens. REDI (Relevance for DINOv3 Token Reduction) studies this question through a controlled supervised reference: how should a fixed token budget be allocated across patches for image classification? REDI quantizes final block DINOv3 patch representations into a visual vocabulary and derives class conditioned corpus scores using supervised TF-IDF over visual words. For each validation image, the ground truth class selects a row of the TF-IDF table, and four transformed views produce a TF-IDF map aligned to a reference center crop. A separate dense pass on the same crop provides an attention map. After independent min max normalization, their elementwise product defines the REDI score. A fixed keep, merge, and compress operator then uses score rank to assign patch roles and score magnitude to weight merging and compression. With precomputed REDI scores, a frozen DINOv3 ViT-B/16 backbone, and the same linear classifier used for dense evaluation, the operator reduces the sequence length from 201 to 107 tokens, a 46.8% sequence reduction. The REDI variant based on incoming attention mass achieves 84.706% Top-1 accuracy on ImageNet-1K, compared with 83.514% for the dense baseline, 82.634% for incoming attention mass alone, and 81.796% for supervised TF-IDF alone. The same corpus term also improves reduced classification for three alternative attention formulations relative to their attention only counterparts. Together, these controlled comparisons indicate that class specific corpus statistics and image specific attention provide complementary signals for patch ranking in this setting.

• The paper introduces REDI, which fuses supervised class-conditioned TF-IDF with per-image attention to rank and reduce ViT patch tokens.
• It achieves a 46.8% token reduction and 71.7% decrease in quadratic attention compute, boosting ImageNet-1K Top-1 accuracy to 84.706%.
• The work emphasizes the complementary role of corpus-level statistics and instance-level cues for smarter token allocation in vision transformers.

Corpus-Aware Patch Ranking for DINOv3 Token Reduction: An Expert Analysis of REDI

Introduction

The computational complexity of Vision Transformers (ViTs) is tightly coupled to their input sequence length, motivating the field of token reduction: selecting, merging, or compressing patch tokens to reduce inference costs while maintaining classification performance. Existing methods for token reduction often leverage attention maps or dynamic sparsification modules, but the allocation of a fixed token budget across patches remains underexplored from a corpus-driven, supervised relevance perspective. REDI (Relevance for DINOv3 Token Reduction) addresses this challenge by introducing a hybrid patch scoring methodology that fuses class-conditioned corpus-level statistics with per-image attention, enabling efficient token reduction under controlled settings on DINOv3 backbones.

REDI: Methodology and Scoring Construction

REDI constructs patch relevance scores through a two-stage pipeline: (1) offline computation of supervised class-conditioned TF-IDF scores over a quantized visual vocabulary derived from the final block DINOv3 features, and (2) integration with image-specific attention maps for ranking and aggregating patches.

A visual vocabulary is established via spherical $k$-means clustering ($K=512$) of unit-normalized final-block patch features across the ImageNet-1K training set. Each patch representation is assigned to its nearest centroid, enabling the abstraction of each patch as a "visual word." Class-specific TF-IDF statistics are computed, where the relevance of a visual word for a class is driven by within-class prevalence and corpus rarity, formulated as a product of a clipped log-odds contrast and inverse document frequency.

For a target validation image, four transformed views are mapped to TF-IDF scores by leveraging the ground truth class to select the corresponding row in the class–visual word table, and mapping each patch according to its assigned visual word. These maps are aligned and aggregated onto the patch grid of the reference crop. In parallel, a separate dense pass yields incoming attention mass for each patch: the mean of its attention matrix column across all query positions and heads in the final block.

Both the TF-IDF map and the attention map undergo independent min–max normalization. The REDI score is computed as the elementwise product of these normalized maps, normalized again post-multiplication. This hybrid score reflects both class-conditioned corpus-level saliency and image-specific attention.

Figure 1: Overview of the REDI pipeline, illustrating the creation and alignment of class-conditioned TF-IDF and attention maps, score normalization, and patch role assignment for the reduction operator.

Patch roles are allocated by ordering the REDI scores: the top 98 patches are explicitly kept, the next 49 are merged into keep tokens weighted by their score, and the final 49 are compressed into four spatial summaries. The reduction operator thus yields a 107-token sequence (down from 201) that is processed by the frozen DINOv3 ViT-B/16 backbone and the original classifier.

Visual Vocabulary Analysis

The patch-to-visual-word assignments exhibit heavy-tailed, Zipf-like distributions, as seen in the rank-frequency plots. Crucially, the distribution of accumulated REDI score mass over visual words deviates significantly from raw assignment frequency. This decoupling underscores that the REDI score’s discriminative power arises not merely from frequent vocabulary members but also from rare, class-informative words identified through corpus-level analysis.

Figure 2: Left: Visual word assignment frequencies across selected DINOv3 blocks. Right: Accumulated REDI score mass across the same assignments reveals the class-discriminative focus distinct from raw frequency.

Experimental Results

Strong numerical improvements are reported: With the fixed token budget (107 tokens), the REDI variant using incoming attention mass achieves 84.706% Top-1 accuracy on ImageNet-1K, which exceeds the dense-path baseline (83.514%) and substantially outperforms both attention-alone (82.634%) and TF-IDF-alone (81.796%) conditions. The performance gains when combining $\mathrm{TFIDF}_{\mathrm{gt}}$ with alternative attention formulations are consistent, with the hybrid REDI signal always surpassing the dense and attention-only baselines. The operator reduces total sequence tokens by 46.8% and quadratic attention compute by 71.7% per block, yielding substantial throughput improvements.

Detailed comparative results demonstrate that the corpus-level class statistics and image-conditioned attention signals encode complementary, non-redundant information, with their combination consistently providing synergistic gains under a controlled, fixed reduction operator.

Implications and Theoretical Insights

The REDI framework introduces a rigorous methodology for corpus-aware, class-conditional token allocation within frozen, pretrained ViTs. By isolating the contribution of the patch ranking signal from that of the reducer and classifier, this analysis provides clear evidence that corpus-level class context—captured by supervised TF-IDF over quantized features—can guide token selection more effectively than per-image attention alone.

The findings challenge the prevalent focus on dynamic, attention-derived or entirely unsupervised criteria for ViT token reduction by demonstrating the untapped effectiveness of supervised corpus statistics in fixed-budget, class-informed scenarios. Practically, this has implications for efficient deployment of ViTs in resource-constrained environments, where minimizing activation footprint without accuracy loss is critical.

Theoretically, the work draws a parallel between established text retrieval techniques and visual representation learning, suggesting further exploration of corpus-level weighting and indexing paradigms in the vision domain. It also raises questions about the optimal fusion of corpus- and instance-level cues, and underlines the importance of explicit supervision and semantic structure in patch ranking.

Limitations and Future Directions

REDI is evaluated as an offline, supervised reference—dependent on ground truth class and dense path attention—rather than as a directly deployable acceleration scheme. As such, the measured gains over dense evaluation should not be interpreted as universal for shorter sequences or for arbitrary reduction operators. The analytic compute reductions do not account for score construction overheads or hardware-dependent effects.

Recommended future work includes: (1) replacing dependence on ground truth class and dense attention with lightweight, predictive or proxy signals for real-time deployment, (2) extending the approach to broader architectures, tasks, and reduction strategies, and (3) exploring end-to-end, potentially differentiable integration of corpus-informed signals into ViT frameworks.

Conclusion

REDI exemplifies a theoretically grounded, corpus-aware approach to token reduction in ViTs, leveraging supervised class statistics and image-conditioned attention for patch ranking within a fixed reduction protocol. The observed consistent improvements across attention formulations underscore the distinct and complementary value of corpus-level statistics for token allocation—motivating broader adoption and further research into corpus-driven token reduction strategies in vision models.