DINO-QPM: Adapting Visual Foundation Models for Globally Interpretable Image Classification

Abstract: Although visual foundation models like DINOv2 provide state-of-the-art performance as feature extractors, their complex, high-dimensional representations create substantial hurdles for interpretability. This work proposes DINO-QPM, which converts these powerful but entangled features into contrastive, class-independent representations that are interpretable by humans. DINO-QPM is a lightweight interpretability adapter that pursues globally interpretable image classification, adapting the Quadratic Programming Enhanced Model (QPM) to operate on strictly frozen DINO backbones. While classification with visual foundation models typically relies on the \texttt{CLS} token, we deliberately diverge from this standard. By leveraging average-pooling, we directly connect the patch embeddings to the model's features and therefore enable spatial localisation of DINO-QPM's globally interpretable features within the input space. Furthermore, we apply a sparsity loss to minimise spatial scatter and background noise, ensuring that explanations are grounded in relevant object parts. With DINO-QPM we make the level of interpretability of QPM available as an adapter while exceeding the accuracy of DINOv2 linear probe. Evaluated through an introduced Plausibility metric and other interpretability metrics, extensive experiments demonstrate that DINO-QPM is superior to other applicable methods for frozen visual foundation models in both classification accuracy and explanation quality.

• The paper proposes DINO-QPM, a lightweight adapter that transforms frozen DINOv2 features into globally interpretable representations using strict sparsity and quantised patch embeddings.
• It employs a structured pipeline with average pooling, a trainable MLP, and a sparse BLDD layer to ensure spatially localized and contrastive class features.
• Experimental results on datasets like CUB-200 and Stanford Cars show improved accuracy (88.3%) and interpretability, making decision processes more transparent.

DINO-QPM: Globally Interpretable Image Classification with Frozen Visual Foundation Models

Introduction and Motivation

The paper "DINO-QPM: Adapting Visual Foundation Models for Globally Interpretable Image Classification" (2604.07166) addresses the critical challenge of global interpretability in state-of-the-art visual foundation models (VFMs), particularly self-supervised models such as DINOv2. While VFMs demonstrate impressive feature extraction capabilities, their high-dimensional latent spaces and reliance on complex aggregation tokens (e.g., CLS tokens in Vision Transformers) severely hinder both local and global interpretability—an essential requirement for real-world, high-stakes domains.

To resolve this, the work presents DINO-QPM, a lightweight interpretability adapter leveraging strict sparsity, class-independent features, and average-pooled patch representations. Rather than requiring prohibitively expensive end-to-end retraining or external supervision (e.g., language-based concept banks), DINO-QPM operates entirely on frozen foundation backbones. By structurally coupling the final features to spatially local input evidence and enforcing quantised, compact, and globally contrastive representations, DINO-QPM provides faithful, human-interpretable explanations—both locally for individual predictions and globally for the entire class structure—while achieving strong accuracy.

Methodology: Sparse, Quantised, and Localised Interpretability

The paper adapts the Quadratic Programming Enhanced Model (QPM) framework (Norrenbrock et al., 27 Feb 2025) to transform the entangled representations of the frozen DINOv2 backbone into interpretable, spatially localised features via a specialized adapter:

Figure 1: The DINO-QPM architecture: frozen DINOv2 patch embeddings feed a trainable MLP, followed by a sparse BLDD layer for compact, interpretable image classification.

Pipeline Overview:

• Patch Embeddings: Input images are processed through the frozen DINOv2, producing a set of patch-level feature vectors.
• Interpretability Adapter (MLP): These patch embeddings are projected into a problem-specific feature space.
• Averaging and Localisation: Features are aggregated via simple average pooling (not global CLS token aggregation), retaining spatial mapping to input patches and enabling high-fidelity feature saliency maps.
• Sparse Assignment (BLDD/QPM): The BLDD layer enforces a strict assignment of a small, fixed number of features per class (typically 5), promoting a globally sparse, binary, and low-dimensional class representation. This structure is derived explicitly through discrete quadratic programming, selecting general and contrastive features that are broadly shared across classes.
• Auxiliary Losses: Feature diversity, L1 sparsity, and Plausibility losses are introduced to ensure the features are diverse, exclusive, and aligned with human-segmented object regions.

  Figure 2: QPM’s three-stage training pipeline: diversity training, quadratic program optimization for sparse feature selection, and retraining under the enforced assignment.

Novelty: In contrast to prior approaches that either use dense linear probes or prototype-based heads (which lack true global interpretability or require fine-tuning), DINO-QPM’s architectural choices—particularly the average pooling on patch features and strict sparsity at the decision layer—enable both high accuracy and a transparent, globally consistent semantic decomposition of class evidence.

Quantitative and Qualitative Results

The model is validated on fine-grained benchmarks (CUB-200-2011, Stanford Cars) using a broad suite of interpretability metrics: Plausibility, SID@5 (feature spatial diversity), Class-Independence, Contrastiveness, and standard accuracy.

Figure 3: DINO-QPM surpasses both the non-frozen QPM and baselines in accuracy and all interpretability dimensions, achieving over 98% of the achievable aggregate score.

Key Results:

• Accuracy: On CUB-200-2011, DINO-QPM achieves 88.3% accuracy, exceeding both dense (CLS token) baselines and non-finetuned post-hoc concept bottlenecks.
• Plausibility: Average Plausibility (fraction of feature activation in annotated object regions) rises to 95%, compared to 43% for a linear probe, verifying substantial spatial grounding.
• Feature Diversity and Generality: SID@5 and Class-Independence metrics confirm that the extracted features are both spatially diverse and class-independent, supporting robust global explanations.
• Contrastiveness: By enforcing mutual exclusivity in feature assignments, DINO-QPM achieves perfect class contrastiveness in feature activations.

  Figure 4: The Plausibility metric—DINO-QPM’s feature activations are tightly constrained within human-annotated object masks, a property lacking in dense baselines.

Qualitative Interpretability

The model produces global and local explanations enabling transparent dissection of decision rules. For visually similar classes, DINO-QPM’s features consistently localise on semantically meaningful parts directly tied to human-recognizable distinctions.

Figure 5: Two visually similar blackbird species are differentiated by the DINO-QPM feature corresponding to beak morphology, reflecting ornithological expertise.

Figure 6: DINO-QPM autonomously discovers the five features distinguishing Hooded Oriole and Hooded Warbler, highlighting throat region differences—no external supervision required.

Figure 7: For Stanford Cars, DINO-QPM localises distinctive headlights, enabling class differentiation through explicit, part-level feature discovery.

Additionally, model ablations demonstrate the critical role of using patch embeddings and structured sparsity for both accuracy and faithfulness, and that the adapter generalizes across different DINOv2 backbone variants.

Comparison to Existing Interpretable and Post-Hoc Methods

While post-hoc methods such as GradCAM, ViT-Shapley, or CBMs can offer superficial attribution maps and partial semantic bottlenecks, they either operate on dense, entangled feature spaces or require external concept supervision and re-training. DINO-QPM uniquely enables:

• End-to-end spatial grounding of globally defined features—each class is represented by exactly the same fixed, low-dimensional, and non-overlapping feature subset across the dataset.
• Full compatibility with frozen large-scale self-supervised backbones, drastically reducing computational overhead.
• Interpretability not just at the level of individual predictions (local), but globally across the learned class structure.

Implications and Future Directions

Practical Implications: DINO-QPM makes inherently faithful, globally interpretable image classification accessible for foundation model pipelines without expensive fine-tuning or domain-specific supervision. The clear structure, compact explanations, and high Plausibility enable robust auditability and trust, especially in safety-critical or regulated settings.

Theoretical Insights: The work highlights the efficacy of explicit discrete optimization for feature selection in neural networks—outperforming more heuristic prototype selection or post-hoc attribution—and provides quantitative evidence that direct patch-feature coupling in ViT backbones is essential for robust local grounding.

Future Work: Directions include:

• Extending the structurally sparse interpretability paradigms to segmentation, detection, or video classification (where global explanations over spatiotemporal features are needed).
• Integrating DINO-QPM-like adapters with multimodal or text-image aligned VFMs for transparent cross-modal reasoning.
• Exploring differentiable relaxations of the BLDD layer for gradient-based neural architecture search.

Conclusion

DINO-QPM demonstrates that interpretable global classification is achievable in large-scale, frozen VFMs by structurally decoupling decision features from entangled token representations and enforcing sparse, global, and localizable features via QPM optimization. The model substantially advances both accuracy and interpretability, establishing a rigorous foundation for future research into faithful, efficient, and globally transparent AI vision systems.