---
title: 'RaysUp: Geometry-Aware Feature Upsampling'
url: https://www.emergentmind.com/topics/raysup
type: topic
---

# RaysUp: Geometry-Aware Feature Upsampling

Searching arXiv for RaysUp and closely related feature upsampling work to ground the article in the current literature.
arXiv search query: "RaysUp feature upsampling Vision Foundation Models AnyUp FeatUp JAFAR LoftUp"
RaysUp is a geometry-aware feature upsampling framework for Vision Foundation Models (VFMs) that reconstructs high-resolution feature maps from low-resolution backbone outputs at arbitrary target resolutions while remaining task-agnostic and VFM-agnostic. Its central premise is that dense feature reconstruction should not be performed solely on the 2D image lattice; instead, it should be lifted into a ray domain in which correspondence is conditioned by viewing directions and implicit geometric structure. In the reported formulation, RaysUp combines a Spatially Decoupled Guidance Encoder, Any-Resolution Cross-Attention, Ray Positional Encoding (RayPE), and Geometry-Aware Neighborhood Attention. The method is described as ultra-light, with **0.14M parameters**, and is reported to use only **16% of the parameters of AnyUp** while delivering approximately **7x faster inference** and state-of-the-art results across several dense prediction tasks [2606.22749].

## 1. Problem formulation and scope

RaysUp addresses a structural limitation of modern VFMs: their outputs are spatially coarse because transformer-based encoders patchify the image into large tokens, and some architectures further apply pooling or aggressive spatial reduction. The paper frames this as a bottleneck for dense prediction tasks requiring localized reasoning, including semantic segmentation, depth estimation, surface normal estimation, video object segmentation, and open-vocabulary segmentation [2606.22749].

The method takes an input image \(I\) and a low-resolution feature map \(F^{lr}\) produced by a frozen VFM, and reconstructs a high-resolution feature map \(F^{hr}\) at an arbitrary target size \((H_{any}, W_{any})\), preserving the VFM feature dimensionality \(D_f\). The reported formulation is:
\[
I \in \mathbb{R}^{3 \times H_{in} \times W_{in}}, \qquad
F^{lr} \in \mathbb{R}^{D_f \times H_{lr} \times W_{lr}},
\]
with output
\[
F^{hr} \in \mathbb{R}^{D_f \times H_{any} \times W_{any}}.
\]

The paper identifies three limitations in prior feature upsampling approaches. First, classical interpolation and some filtering methods produce semantic degradation, especially near boundaries and discontinuities. Second, some approaches require VFM-specific retraining or image-specific optimization at test time, reducing scalability. Third, AnyUp removes the need for backbone-specific retraining and supports arbitrary resolutions, but is characterized as computationally heavier. RaysUp is proposed as a joint response to these constraints: preserve semantic fidelity, generalize across backbones and tasks, and remain efficient at high resolution [2606.22749].

The term “universal” in the paper refers to any-resolution reconstruction; “task-agnostic” refers to training with feature reconstruction losses rather than task-specific losses; and “VFM-agnostic” refers to training on one VFM and transferring to others without retraining. This suggests that RaysUp is intended as a generic post-encoder feature operator rather than a task head specialized to a single downstream objective.

## 2. Architecture and geometric representation

The architecture comprises four modules: a Spatially Decoupled Guidance Encoder, Any-Resolution Cross-Attention, Ray Positional Encoding, and Geometry-Aware Neighborhood Attention [2606.22749].

The guidance encoder maps the RGB image to a dense guidance tensor
\[
F_g(I) \in \mathbb{R}^{D_g \times H_{in} \times W_{in}}.
\]
The default guidance dimension is **\(D_g = 256\)**. Instead of a single isotropic convolution, the encoder uses four directional branches:

- center-like branch: \(1\times1\)
- horizontal branch: \(1\times3\)
- vertical branch: \(3\times1\)
- diagonal/corner branch: \(2\times2\) dilated convolution with dilation 2

The branch outputs are given as
\[
\begin{aligned}
F^c_g &= \phi\!\left(\mathrm{GN}\!\left(\mathrm{Conv}_{1\times1}^{D_g/4}(I)\right)\right),\\
F^h_g &= \phi\!\left(\mathrm{GN}\!\left(\mathrm{Conv}_{1\times3}^{D_g/4}(I)\right)\right),\\
F^v_g &= \phi\!\left(\mathrm{GN}\!\left(\mathrm{Conv}_{3\times1}^{D_g/4}(I)\right)\right),\\
F^{diag}_g &= \phi\!\left(\mathrm{GN}\!\left(\mathrm{Conv}_{2\times2,d=2}^{D_g/4}(I)\right)\right),
\end{aligned}
\]
where \(\mathrm{GN}\) is GroupNorm and \(\phi\) is SiLU. Each branch then uses a shallow residual block,
\[
\tilde{F}^b_g = X^b + \phi\!\left(\mathrm{GN}\!\left(\mathrm{Conv}_b(X^b)\right)\right), \quad b \in \{c,h,v,diag\},
\]
and the final guidance tensor is
\[
F_g(I)=\mathrm{Concat}\!\left(\tilde{F}^c_g,\tilde{F}^h_g,\tilde{F}^v_g,\tilde{F}^{diag}_g\right).
\]

The purpose of this decoupled design is to encode central, horizontal, vertical, and diagonal structure separately. The paper reports that a standard \(3\times3\) convolution requires \(27D_g\) parameters, whereas the decoupled design requires \(8.25D_g\), corresponding to about **69.4% fewer parameters**.

Any-Resolution Cross-Attention decouples query resolution from key/value resolution. From the same guidance tensor, RaysUp forms
\[
Q_g = \mathrm{AdaptiveAvgPool}(F_g(I), H_{any}, W_{any}) \in \mathbb{R}^{D_g \times H_{any} \times W_{any}},
\]
\[
K_g = \mathrm{AdaptiveAvgPool}(F_g(I), H_{lr}, W_{lr}) \in \mathbb{R}^{D_g \times H_{lr} \times W_{lr}},
\]
and uses the low-resolution VFM features directly as values:
\[
V = F^{lr} \in \mathbb{R}^{D_f \times H_{lr} \times W_{lr}}.
\]
Because the query grid is formed independently from the key/value grid, the target size need not be tied to a fixed integer upsampling ratio.

RayPE is the module that differentiates RaysUp from purely image-grid-based upsamplers. For a pixel coordinate
\[
x = [i,j,1]^T,
\]
the ray origin and direction are defined as
\[
o = t, \qquad d(x) = \frac{R M^{-1} x}{\|R M^{-1} x\|_2},
\]
and the ray descriptor is
\[
r = \mathrm{Concat}(o, d(x)) \in \mathbb{R}^{6}.
\]
The paper describes this as using “6D Plücker ray coordinates,” while the explicit formula is the concatenation of origin and normalized direction. This distinction is relevant because the operational representation in the method is the 6D vector \([o;d(x)]\), which is then encoded harmonically and injected into attention.

## 3. Mathematical formulation and training procedure

RayPE applies a multi-band harmonic phase encoding to the 6D ray descriptor. For each ray component \(r_d\), \(d=1,\dots,6\), and frequency band \(f=0,\dots,N-1\),
\[
\Theta_{f,d} = \omega_f \, r_d,
\]
followed by
\[
\theta = \mathrm{Flatten}(\Theta) \in \mathbb{R}^{6N}.
\]
The implementation details report **\(N=5\)** frequency bands. The paper also specifies wavelength ranges used for the origin and direction components: for ray origin, \(\lambda_{\max}=4.0\) and \(\lambda_{\min}=4.0/81\); for ray direction, \(\lambda_{\max}=8.0\) and \(\lambda_{\min}=8.0/81\) [2606.22749].

For each attention-head feature tensor \(F_h \in \{Q_g, K_g\}\), the head is split into two halves,
\[
F_h = [F_1, F_2],
\]
and a RoPE-style rotation is applied:
\[
\begin{bmatrix} \tilde{F}_1 \\ \tilde{F}_2 \end{bmatrix}
=
\begin{bmatrix}
\cos\theta & -\sin\theta \\
\sin\theta & \cos\theta
\end{bmatrix}
\begin{bmatrix} F_1 \\ F_2 \end{bmatrix},
\qquad
\tilde{F}_h=\mathrm{Concat}(\tilde{F}_1,\tilde{F}_2).
\]
The resulting tensors are denoted \(Q_{ray}\) and \(K_{ray}\). The dimensionality constraint given in the paper is
\[
6N \le \lfloor d_{\text{head}}/2 \rfloor.
\]

Reconstruction is performed by local geometry-aware cross-attention rather than global attention. For each target position \((i,j)\),
\[
F^{hr}_{i,j}= \sum_{(u,v)\in\mathcal{N}_{i,j}}
\mathrm{Softmax}
\left(
\frac{(Q_{ray})_{i,j}^{\top}(K_{ray})_{u,v}}{\sqrt{d}}
\right)
V_{u,v},
\]
where \(\mathcal{N}_{i,j}\) is a \(k\times k\) neighborhood in low-resolution space. The supplementary gives the default neighborhood size as **\(k=6\)**. To accommodate arbitrary source-target ratios, RaysUp defines dilation factors
\[
d_h = \max(1,\lfloor H_{any}/H_{lr} \rfloor), \qquad
d_w = \max(1,\lfloor W_{any}/W_{lr} \rfloor).
\]
The paper interprets this as local, geometry-consistent sampling on the ray manifold.

The computational argument is explicit: global attention would cost
\[
\mathcal{O}(H_{any}W_{any}\cdot H_{lr}W_{lr}),
\]
whereas neighborhood attention costs
\[
\mathcal{O}(H_{any}W_{any}\cdot k^2).
\]
This is a principal source of the method’s efficiency.

Training uses a frozen encoder \(e(\cdot)\). Given a high-resolution image \(I_{hr}\) and a downsampled image \(I_{lr}\),
\[
F_{tgt}=e(I_{hr}), \qquad F^{lr}=e(I_{lr}),
\]
and RaysUp predicts
\[
\hat{F}^{hr}=f(I_{hr},F^{lr}).
\]
The training loss is
\[
\mathcal{L} = \mathcal{L}_{cos}(\hat{F}^{hr},F_{tgt}) + \mathcal{L}_{L2}(\hat{F}^{hr},F_{tgt}).
\]
A crop-based variant is also reported:
\[
\mathcal{L}(\hat{F}^{hr}_{crop},F_{tgt}) = 1-\cos(\hat{F}^{hr}_{crop},F_{tgt}) + \|\hat{F}^{hr}_{crop}-F_{tgt}\|_2^2.
\]

The reported training setup is **ImageNet**, **AdamW**, **100,000** iterations, batch size **4**, initial learning rate **\(2\times10^{-4}\)**, target high-resolution images resized to **\(448\times448\)**, guidance input downsampled to **\(224\times224\)**, and training time of about **1 hour on a single NVIDIA A100**. The default pose setting is **Identity** [2606.22749].

## 4. Empirical performance and efficiency profile

RaysUp is evaluated on semantic segmentation, depth estimation, surface normal estimation, video object segmentation, and open-vocabulary segmentation. The reported backbones include **DINOv2** (ViT-S, ViT-M, ViT-L), **DINOv3** (ViT-S, ViT-M, ViT-L), **SigLIP2** (ViT-B, ViT-L), and **PE Spatial** (ViT-B, ViT-L), with **DINOv2-S** used as the default training backbone [2606.22749].

At **\(224\times224\)**, RaysUp uses **0.14M parameters**, **10.17 GFLOPs**, **1.26 GB GPU memory**, and runs at **55 FPS**. The comparison given for AnyUp at the same resolution is **0.87M parameters**, **84.21 GFLOPs**, **2.19 GB**, and **11 FPS**. The reported scaling points are:

| Resolution | RaysUp runtime profile |
|---|---|
| \(224\times224\) | 10.17 GFLOPs, 1.26 GB, 55 FPS |
| \(448\times448\) | 40.67 GFLOPs, 2.69 GB, 27 FPS |
| \(896\times896\) | 162.68 GFLOPs, 8.47 GB, 8 FPS |
| \(2K\times2K\) | still runnable at 1 FPS |

The paper states that several baselines become very slow or OOM at high resolutions. This supports the characterization of RaysUp as an any-resolution method whose efficiency does not collapse immediately at large target sizes.

For semantic segmentation, the reported task-agnostic results are:

| Dataset | RaysUp result |
|---|---|
| COCO-Stuff | **62.32 mIoU / 81.47 Acc** |
| Pascal-VOC | **84.64 / 96.34** |
| ADE20K | **42.34 / 75.81** |
| Cityscapes | **61.88 / 93.64** |

The corresponding AnyUp values reported in the paper are **62.14 / 81.38**, **84.18 / 96.20**, **42.15 / 75.71**, and **60.62 / 93.26**, respectively, so RaysUp is reported to outperform AnyUp on all four benchmarks [2606.22749].

On **NYUv2**, RaysUp reports:

- **Normals RMSE**: **27.69**
- **\(11.25^\circ\)**: **0.4986**
- **\(22.5^\circ\)**: **0.7030**
- **\(30^\circ\)**: **0.7775**
- **Depth abs RMSE**: **0.4658**
- **\(\delta_1\)**: **0.8103**
- **Depth rel RMSE**: **0.3195**
- **\(\delta_1\)**: **0.9309**

The paper characterizes these as generally the best results in the table, especially on geometric tasks, which is consistent with the method’s geometry-aware design.

On **DAVIS 2017** video object segmentation, RaysUp reports:

- \(\mathcal{J}\): **68.14**
- \(\mathcal{F}\): **74.81**
- \(\mathcal{J}\mathcal{F}\): **71.47**

This is reported as the best result among the compared methods.

For open-vocabulary segmentation, RaysUp is described as competitive but not always best. The reported values are:

- COCO-Stuff: **27.11 mIoU / 48.57 Acc**
- Pascal-VOC: **63.28 / 87.34**
- ADE20K: **20.54 / 42.71**
- Cityscapes: **37.24 / 59.68**

The paper notes that **LoftUp** and **AnyUp** slightly surpass RaysUp on some open-vocabulary metrics and attributes some of LoftUp’s gains to **SAM-based auxiliary supervision**, which RaysUp does not use. This is an important qualification: the geometry-aware formulation appears particularly effective for dense geometric and video tasks, but does not dominate uniformly across all semantic evaluation settings.

## 5. Transfer behavior, ablations, and design rationale

A major claim of RaysUp is that it generalizes across different VFMs without retraining. The reported cross-backbone results are consistent with that claim [2606.22749]. Examples given in the paper include:

- **DINOv2 ViT-S**: AnyUp **84.18** mIoU, depth RMSE **0.478 / 0.324**; RaysUp **84.64**, **0.465 / 0.319**
- **DINOv3 ViT-L**: AnyUp **87.48**, **0.402 / 0.257**; RaysUp **88.07**, **0.398 / 0.255**
- **SigLIP2 ViT-L**: AnyUp **79.16**, **0.726 / 0.474**; RaysUp **80.10**, **0.725 / 0.472**
- **PE Spatial ViT-L**: AnyUp **82.23**, **0.635 / 0.404**; RaysUp **82.75**, **0.613 / 0.388**

The any-to-any resolution evaluation on Pascal-VOC includes source-target pairs \(16 \rightarrow 112\), \(16 \rightarrow 448\), \(16 \rightarrow 896\), \(32 \rightarrow 112\), \(32 \rightarrow 224\), and \(32 \rightarrow 896\). RaysUp is reported to be best or second-best across all settings, and the paper emphasizes that it remains stable at the highest scales without OOM. Two reported examples are **\(32 \rightarrow 224\): 84.81 mIoU / 96.37 Acc** and **\(32 \rightarrow 896\): 84.67 / 96.35**.

The ablation studies clarify which design choices matter most:

| Ablation axis | Reported outcome |
|---|---|
| Guidance encoder design | **Decoupled-Branch**: **82.17**, **0.14M** params |
| Guidance dimension | **\(D_g=256\)**: **82.17**, **0.14M** params |
| Number of conv blocks | **\(L=1\)**: **82.17**, **0.14M** params |
| Positional encoding | **RayPE**: **82.17** avg |
| Image pose | Identity **82.17**; DA3-Base **82.44** |

The encoder-design ablation compares **Single-Branch** (**81.87**, **0.266M params**), **Dual-Branch** (**82.03**, **0.66M**), **Multi-Branch** (**82.09**, **0.268M**), and **Decoupled-Branch** (**82.17**, **0.14M**). The preferred guidance dimension is **\(D_g=256\)**, as larger values yield only marginal gains with higher parameter counts. The default **\(L=1\)** block depth sacrifices a small amount of accuracy relative to **\(L=2,3,4\)** but preserves the method’s light footprint.

The positional encoding ablation is especially consequential. The reported averages are:

- None: **81.05**
- RoPE: **81.92**
- SinRays: **81.59**
- **RayPE**: **82.17**

This indicates that positional encoding is not a marginal detail but a substantive contributor to performance. The image-pose ablation further shows that stronger pose estimates via **DA3** improve results slightly, with **DA3-Base** reaching **82.44**, but the default remains **Identity** for efficiency. A plausible implication is that RaysUp can exploit stronger geometry cues when they are available, without requiring them for its standard operating point.

The supplementary also reports that training with crops improves semantic segmentation further; for example, on Cityscapes, **63.04** mIoU is reported versus **61.88** without crop-based training, at the cost of about **4 hours** of training.

## 6. Interpretation, limitations, and position within feature upsampling research

RaysUp’s principal conceptual contribution is the redefinition of feature upsampling as transport from a coarse set of rays to a denser set of rays, rather than interpolation on the image lattice [2606.22749]. In the paper’s framing, ordinary image-space proximity is an unreliable guide to correspondence under perspective projection: adjacent pixels can lie on different surfaces at depth discontinuities, while spatially separated pixels can belong to the same physical structure. RayPE and geometry-aware local attention are intended to compensate for this mismatch by biasing aggregation toward angular and structural consistency.

This positioning explains the method’s particularly strong results on depth, surface normals, and video object segmentation. The paper’s qualitative claims are that RaysUp produces fewer holes than JAFAR, AnyUp, and FeatUp; sharper boundaries than LoftUp; better geometric consistency in depth and normal estimation; more robust inter-frame consistency in video segmentation; and clearer PCA projections preserving the underlying feature space. These observations are consistent with the claim that geometry-aware correspondence is especially beneficial when the downstream task is sensitive to discontinuities, shape, or multiview-consistent structure.

Several limitations are also explicit. First, the geometry is only **implicit**: RaysUp does not perform a full 3D reconstruction or rendering process. Second, the paper frequently uses **identity pose** and simplified camera assumptions; although stronger camera information improves results slightly, it is not central to the default setup. Third, RaysUp is not uniformly best across all task categories; in some open-vocabulary settings, methods such as LoftUp remain slightly stronger, which the authors partly attribute to auxiliary supervision not used by RaysUp. Fourth, the paper notes that some mathematical expressions are imperfectly typeset, especially the frequency formula for RayPE and some dimensions, although the intended method is presented clearly enough to reconstruct the implementation.

A common misconception would be to interpret the “ray” terminology as implying explicit volumetric reasoning or full projective scene reconstruction. The paper does not support that reading. RaysUp uses rays as a positional and correspondence prior inside an efficient local cross-attention operator. Its geometry is therefore representational rather than physically simulated. This suggests that its primary significance lies in introducing geometric bias into universal feature upsampling without losing the efficiency required for large-scale dense prediction.

In the literature position established by the paper, RaysUp is best understood as a lightweight universal upsampler that improves the accuracy-efficiency trade-off relative to AnyUp and other task-agnostic upsamplers while retaining arbitrary-resolution support and cross-backbone transfer. Its strongest documented attributes are **0.14M parameters**, **10.17 GFLOPs** at \(224\times224\), **55 FPS**, operation up to **\(2K\times2K\)** at **1 FPS**, and performance gains across segmentation, geometry, and video benchmarks. Within that scope, RaysUp functions as a general-purpose dense feature reconstruction layer for frozen VFMs rather than a task-specific decoder or a full geometric vision system [2606.22749].

Source: https://www.emergentmind.com/topics/raysup