RaysUp: Ultra-light Universal Feature Upsampling via Geometry-Aware Ray Representation

Abstract: Pre-trained Vision Foundation Models (VFMs) have become central to modern computer vision due to their powerful semantic representations and strong generalization ability. However, their patchified or pooled outputs are inherently low-resolution, limiting their effectiveness in tasks requiring fine-grained, pixel-level reasoning. Existing feature upsampling approaches either degrade semantic fidelity or rely on VFM-specific retraining and heavy architectures, hindering efficiency and scalability. To address these challenges, we propose RaysUp, an ultra-lightweight, task-agnostic, and VFM-agnostic feature upsampling framework that reconstructs high-resolution feature maps at arbitrary resolutions. Unlike conventional 2D interpolation or attention-based schemes, RaysUp lifts feature reconstruction into a geometry-aware ray domain. Specifically, we introduce a Spatially Decoupled Guidance Encoder for direction-aware guidance encoding, an Any-Resolution Cross-Attention mechanism for resolution-flexible reconstruction, and a novel Ray Positional Encoding (RayPE) that injects implicit 3D geometric priors via 6D Plucker ray coordinates. Finally, a Geometry-Aware Neighborhood Attention module further ensures content-adaptive bilateral aggregation while preserving geometric consistency. Extensive experiments across diverse dense prediction tasks demonstrate that RaysUp achieves state-of-the-art performance while using only 16% of the parameters of AnyUp and delivering approximately 7x faster inference. These results highlight a substantially improved accuracy-efficiency trade-off and establish RaysUp as a practical and scalable solution for universal feature upsampling. Code is available at https://github.com/MAP-RaysUp/RaysUp.

• The paper introduces RaysUp, a universal feature upsampling method leveraging ray positional encoding to enhance 3D geometric consistency.
• It employs an efficient spatially decoupled guidance encoder and any-resolution cross-attention, reducing parameters by 69% compared to similar methods.
• Experimental results show improved semantic segmentation and depth estimation with sharper boundaries and up to 7× higher throughput.

RaysUp: Ultra-light Universal Feature Upsampling via Geometry-Aware Ray Representation

Introduction

The performance of modern Vision Foundation Models (VFMs) is largely underpinned by their ability to extract semantically rich, transferable low-resolution feature maps via large-scale pretraining. However, downstream dense prediction tasks—such as depth estimation, semantic segmentation, and 3D scene reconstruction—demand high-resolution, geometrically consistent feature representations. Current upsampling solutions often degrade semantic fidelity, impose excessive computational cost, or lack generalization across backbone architectures. The paper "RaysUp: Ultra-light Universal Feature Upsampling via Geometry-Aware Ray Representation" (2606.22749) addresses these limitations by introducing RaysUp, a task-agnostic, encoder-agnostic, and ultra-lightweight upsampling framework. The core novelty is the unification of adaptive, efficient cross-resolution attention with explicit injection of 3D geometric priors via Ray Positional Encoding. This yields practical scalability and robust performance across a broad spectrum of dense prediction tasks.

Figure 1: RaysUp enables ultra-lightweight, task-agnostic, and VFM-agnostic feature upsampling to arbitrary resolutions with high semantic fidelity and geometric consistency.

Methodology

Architectural Overview

RaysUp decomposes the upsampling problem into four principal modules:

1. Spatially Decoupled Guidance Encoder: Learns direction-aware spatial features from the RGB input, decomposed into branch-wise representations (center, horizontal, vertical, diagonal) with a parameterization substantially more efficient than typical $3\times3$ convolutions.
2. Any-Resolution Cross-Attention: Upsampled (query) and backbone (key) features are adaptively pooled from the guidance signal, allowing flexible mapping to arbitrary target resolutions.
3. Ray Positional Encoding (RayPE): Queries and keys are modulated by explicit 3D ray descriptors (6D Plücker coordinates—ray origins and directions in camera space) and multi-band harmonic encodings, binding feature aggregation to 3D geometry rather than mere pixel proximity.
4. Geometry-Aware Neighborhood Cross-Attention: Feature reconstruction aggregates values from learned local neighborhoods, with attention scores computed in RayPE space and topological alignment enforced by dilated correspondence windows.

   Figure 2: The RaysUp pipeline: Given an image and backbone features, direction-aware query/key features are mapped into 3D ray geometry and modulated for cross-attention-based upsampling.

This approach generalizes Joint Bilateral Upsampling (JBU), which uses RGB-based, fixed-neighborhood guidance, into a trainable, geometry-consistent attention mechanism that preserves structural and semantic details even at large upsampling ratios.

Detailed Modules

• Spatial Decoupling: Empirical evidence (Appendix) demonstrates that convolutional kernels in prior works (e.g., JAFAR) assign lower center weights, propagating artifacts and holes in upsampled features. The proposed encoder increases central mixing while partitioning other spatial patterns to directional branches, with 69% fewer parameters.

  Figure 3: Visualization of learned convolutional kernel weights. The spatially decoupled encoder promotes center-centric mixing critical for consistent upsampling.

• RayPE Geometry Injection: In contrast to RoPE or standard sinusoids, RayPE encodes viewing rays with origin and normalized direction, mapping attention from the 2D pixel grid to the unit viewing sphere. This design ensures that upsampling is sensitive to true 3D structure rather than simple image-plane proximity.

  Figure 4: RayPE attention biases facilitate aggregation along rays with similar geometry, supporting equivariance under viewpoint and scale changes.

• Neighborhood Cross-Attention: To avoid the inefficiency of global attention, only local dilated windows are attended, establishing correspondence in the backbone resolution and propagating features to high-resolution queries. This reduces complexity from $$\mathcal{O}(H^\text{out}W^\text{out}H^\text{in}W^\text{in})$$ to $\mathcal{O}(H^\text{out}W^\text{out}k^2)$.

Experimental Results

Task-Agnostic and VFM-Agnostic Generalization

Extensive experiments spanned semantic segmentation (COCO-Stuff, Pascal-VOC, ADE20K, Cityscapes), depth and surface normal estimation (NYUv2), and video object segmentation (DAVIS). Evaluation was conducted using principal VFM backbones—DINOv2, DINOv3, SigLIP2, PE Spatial—across multiple scales and architectural variants.

Key findings:

• On semantic segmentation, RaysUp matches or surpasses the SoTA (e.g., 84.64/96.34 mIoU/Acc on Pascal-VOC), even outperforming encoder-agnostic alternatives (AnyUp) while using only 16% of their parameters and delivering ~7$\times$ higher throughput.
• In geometric prediction, RaysUp provides the lowest RMSEs and highest angular consistency for depth and normals, indicating effective preservation of underlying 3D structure.
• RaysUp outperforms AnyUp, JAFAR, and FeatUp in dense label propagation tasks, with sharper mask boundaries and higher regional accuracy in both static and video data.

  Figure 5: RaysUp delivers higher geometric and semantic consistency than attention-based and traditional upsamplers across diverse image tasks.

• VFM generalization: RaysUp maintains superior upsampling performance when trained on DINOv2-S but evaluated on disparate VFMs, indicating robust, backbone-independent design.

Efficiency and Scalability

RaysUp deploys with 0.14M parameters (vs. 0.87M for AnyUp), and inference memory and runtime are favorable even at 2K$\times$2K resolutions. Competing methods either crash or degrade to unusable speed above 0.5Mpx spatial scales.

Figure 6: PCA projections of upsampled features. RaysUp, unlike most baselines, sharpens high-dimensional structure, confirming precise feature space alignment.

Qualitative Evaluation

• Surface & Depth Estimation: Visually, RaysUp predicts sharper edges and improved depth transitions, outperforming all baselines (Figure 7).
• Semantic Segmentation: Segmentation predictions demonstrate boundary fidelity and often surpass annotation sharpness (Figure 8).
• Video Segmentation: RaysUp yields robust and temporally consistent masks, mitigating typical artifacts seen in prior works (Figure 9).

  Figure 7: Depth estimation samples: RaysUp produces sharper, more accurate object boundaries than all tested baselines.

  

  Figure 8: Semantic segmentation: RaysUp generates sharper and sometimes more detailed mask boundaries than the ground truth.

Ablation Study

Systematic ablations confirm that the spatially decoupled guidance encoder provides both accuracy gains and major parameter reduction. RayPE (vs. RoPE/SinRays or no positional encoding) emerges as the principal contributor to geometric consistency and boundary reasoning. Incorporation of additional camera pose does improve scores but with a tradeoff in complexity; the default identity configuration is selected for efficiency.

Implications and Future Directions

RaysUp's demonstration of universal, ultra-lightweight, geometry-consistent upsampling contributes new architectural and representational principles for dense visual reasoning. Practically, its VFM-agnosticism and computational footprint enable scalable deployment in memory- or speed-critical environments (mobile robotics, AR/VR, large-scale 3D reconstruction). Theoretically, explicit ray-domain modeling establishes a pathway for bridging 2D feature encoding and 3D perception—key for robust representations in open-world vision.

Given the separability of content and geometry, future work may pursue: joint feature-depth upsampling, integration with neural rendering pipelines, or direct use in online 3D scene understanding from monocular video, extending beyond frozen backbone settings towards fully self-supervised vision-language training with explicit spatial priors.

Conclusion

RaysUp advances the state-of-the-art in universal feature upsampling by coupling geometry-aware representations and highly efficient cross-attention. The combined ablation and benchmarking evidence demonstrates strong advantages in accuracy, scalability, and speed relative to all comparable frameworks. RaysUp is a substantive step toward practical, universal, and geometrically robust upsampling in vision-centric AI systems.