SPAR: Single-Pass Any-Resolution ViT for Open-vocabulary Segmentation

Abstract: Foundational Vision Transformers (ViTs) have limited effectiveness in tasks requiring fine-grained spatial understanding, due to their fixed pre-training resolution and inherently coarse patch-level representations. These challenges are especially pronounced in dense prediction scenarios, such as open-vocabulary segmentation with ViT-based vision-LLMs, where high-resolution inputs are essential for accurate pixel-level reasoning. Existing approaches typically process large-resolution images using a sliding-window strategy at the pre-training resolution. While this improves accuracy through finer strides, it comes at a significant computational cost. We introduce SPAR: Single-Pass Any-Resolution ViT, a resolution-agnostic dense feature extractor designed for efficient high-resolution inference. We distill the spatial reasoning capabilities of a finely-strided, sliding-window teacher into a single-pass student using a feature regression loss, without requiring architectural changes or pixel-level supervision. Applied to open-vocabulary segmentation, SPAR improves single-pass baselines by up to 10.5 mIoU and even surpasses the teacher, demonstrating effectiveness in efficient, high-resolution reasoning. Code: https://github.com/naomikombol/SPAR

• The paper introduces SPAR, which distills spatial reasoning from a high-fidelity sliding-window teacher to enable efficient, any-resolution segmentation.
• SPAR achieves an average improvement of +10.5 mIoU on multiple benchmarks and delivers a 52× speedup compared to traditional sliding-window inference.
• The framework scales ViT accuracy with resolution by fine-tuning only the last transformer blocks, ensuring robust performance for dense prediction tasks.

SPAR: Single-Pass Any-Resolution ViT for Open-vocabulary Segmentation

Motivation and Background

Resolution mismatch and patch-size-induced spatial aliasing remain fundamental issues when deploying ViT-based Vision-LLMs (VLMs) for dense prediction tasks, particularly Open-Vocabulary Segmentation (OVS). ViTs, typically pre-trained at fixed, low resolutions, perform poorly on high-resolution inputs due to the coarse spatial granularity of patch embeddings and the inability to generalize positional encodings. Existing solutions—interpolating positional encodings or using sliding-window inference over overlapping image windows—entail either severe accuracy degradation or computational intractability, especially in the high-resolution regime required for OVS.

Methodology: The SPAR Framework

The proposed SPAR framework addresses these challenges by distilling the spatial reasoning ability of a high-fidelity, sliding-window teacher into a student ViT that supports efficient, single-pass inference at arbitrary image resolutions and aspect ratios without modifying the ViT backbone or requiring pixel-level supervision.

SPAR operates via teacher-student feature-level distillation:

1. Teacher Construction: Given a frozen ViT-based VLM, the teacher generates dense features via a finely-strided sliding-window routine over high-resolution images. Overlapping window outputs are spatially stitched into a coherent feature map aligned with the original image grid.
2. Student Training: The student, initialized from the same pre-trained weights, learns to regress to the teacher’s stitched feature map using a feature-wise L2 loss, allowing for backpropagation only in selected upper network layers for computational efficiency and to regularize resolution tolerance.
3. Resolution and Aspect Generalization: During training, images are sampled at diverse resolutions and aspect ratios to anchor robust any-shape generalization. Empirical ablations show that fine-tuning only the last two transformer blocks suffices for strong performance, while tuning the full model grants additional gains for extreme resolution generalization.

At inference, the student performs a single pass over the native-resolution image, enabling a $52\times$ speedup over sliding-window baselines with no requirement for custom upsampling heads or architectural changes.

Figure 2: Overview of SPAR's teacher-student paradigm leveraging sliding-window feature distillation for efficient, resolution-agnostic dense extraction.

Experimental Evaluation

Datasets and Benchmarks

SPAR is systematically evaluated on six OVS benchmarks derived from Pascal VOC, Pascal Context, ADE20K, and Cityscapes, spanning a wide range of resolutions and class cardinalities, using SigLIP2, OpenCLIP, and DINOv3 ViT backbones.

Accuracy/Speed Trade-off

SPAR attains substantial gains compared to native single-pass inference:

• On SigLIP2 ViT-B-16, SPAR yields an average improvement of +10.5 mIoU over the single-pass baseline and outperforms the high-accuracy sliding-window teacher at a fraction (up to $1/52$) of the computational cost.

Figure 1: SPAR exceeds sliding-window accuracy at the cost/inference time of a single pass, closing the performance gap without high compute.

Additional integration with label propagation (LPOSS) and learned upsamplers (AnyUp) demonstrates orthogonal and complementary gains, further raising mIoU.

Scaling with Resolution

SPAR uniquely enables ViT models to benefit from higher image resolutions. While standard single-pass or NaFlex models fail to leverage increased resolution, SPAR-trained models scale in accuracy in line with sliding-window at low computational cost.

Figure 3: mIoU performance as a function of resolution for SPAR, sliding-window, and single-pass models; SPAR enables accuracy growth with resolution without exponential compute growth.

Ablation Studies

Ablations confirm:

• Fine-tuning depth: Training only the last two transformer blocks provides nearly optimal generalization/efficiency trade-off.
• Data/label independence: SPAR’s gains are robust to distillation set choice and size; even 10k-25k unlabeled images suffice.
• Teacher configuration: Stride not divisible by patch-size in the teacher effectively exposes sub-patch regions to diverse contexts, boosting model robustness.

Transfer to Vision-only Dense Tasks

SPAR-trained vision encoders, although optimized for OVS, yield enhanced descriptors for dense prediction tasks such as linear probe segmentation, Hummingbird kNN segmentation, and oracle panoptic segmentation, indicating improved visual feature quality and spatial consistency.

Qualitative Analysis

Visual comparisons show that SPAR outperforms native single-pass and even sliding-window ViTs in terms of prediction smoothness, boundary quality, and intra-object consistency, while better separating distinct semantic regions. Principal Component Analysis (PCA) projections of feature maps further corroborate improved class separation and homogeneous spatial structure.

Figure 5: Qualitative results and PCA analysis: SPAR yields spatially denoised and semantically coherent segmentation maps and features, further enhanced by integration with label propagation.

Figure 4: Additional qualitative results show improved delineation, reduced noise, and structure preservation with SPAR over teacher and baseline methods.

Figure 6: Feature PCA: SPAR demonstrates improved inter-object boundaries and intra-object smoothness compared to the teacher and baseline models.

Theoretical and Practical Implications

SPAR demonstrates that post-hoc, label-free distillation from the intra-contextual representations of ViT backbones enables robust, architecture-agnostic resolution scaling in dense prediction tasks. This property is achieved with minimal modification and user intervention; no pixel-supervision, no domain narrowing, and no custom heads are required. The purely feature-space regression phase generalizes both to vision-language and vision-only dense prediction, laying groundwork for scalable, flexible, and efficient deployment of foundation models in real-world high-res computer vision settings.

Future research directions include extending this approach to multi-modal or temporal scenarios, integrating feature-level distillation into self-supervised or prompt-adaptive settings, and leveraging architectural invariances for further compression or edge deployment.

Conclusion

SPAR establishes a new regime for efficient, resolution-agnostic segmentation with ViTs. Via simple teacher-student distillation at the feature level, SPAR equips foundation vision backbones with the ability to match and surpass high-fidelity sliding-window teachers in OVS, but at single-pass cost. This work provides a foundation for future advances in architectural robustness, efficient input scaling, and general-purpose dense perceptual reasoning (2604.02252).