RaysUp: Geometry-Aware Feature Upsampling

• RaysUp is a geometry-aware feature upsampling method that reconstructs high-resolution maps from low-res VFM outputs using ray-domain encoding.
• It employs a decoupled guidance encoder, any-resolution cross-attention, and RayPE to capture geometric structure and boost dense prediction performance.
• RaysUp achieves state-of-the-art efficiency with 0.14M parameters and 7x faster inference compared to similar methods, making it highly scalable.

Searching arXiv for RaysUp and closely related feature upsampling work to ground the article in the current literature. arXiv search query: "RaysUp feature upsampling Vision Foundation Models AnyUp FeatUp JAFAR LoftUp" RaysUp is a geometry-aware feature upsampling framework for Vision Foundation Models (VFMs) that reconstructs high-resolution feature maps from low-resolution backbone outputs at arbitrary target resolutions while remaining task-agnostic and VFM-agnostic. Its central premise is that dense feature reconstruction should not be performed solely on the 2D image lattice; instead, it should be lifted into a ray domain in which correspondence is conditioned by viewing directions and implicit geometric structure. In the reported formulation, RaysUp combines a Spatially Decoupled Guidance Encoder, Any-Resolution Cross-Attention, Ray Positional Encoding (RayPE), and Geometry-Aware Neighborhood Attention. The method is described as ultra-light, with 0.14M parameters, and is reported to use only 16% of the parameters of AnyUp while delivering approximately 7x faster inference and state-of-the-art results across several dense prediction tasks (Ding et al., 22 Jun 2026).

1. Problem formulation and scope

RaysUp addresses a structural limitation of modern VFMs: their outputs are spatially coarse because transformer-based encoders patchify the image into large tokens, and some architectures further apply pooling or aggressive spatial reduction. The paper frames this as a bottleneck for dense prediction tasks requiring localized reasoning, including semantic segmentation, depth estimation, surface normal estimation, video object segmentation, and open-vocabulary segmentation (Ding et al., 22 Jun 2026).

The method takes an input image $I$ and a low-resolution feature map $F^{lr}$ produced by a frozen VFM, and reconstructs a high-resolution feature map $F^{hr}$ at an arbitrary target size $(H_{any}, W_{any})$, preserving the VFM feature dimensionality $D_f$. The reported formulation is: $$I \in \mathbb{R}^{3 \times H_{in} \times W_{in}}, \qquad F^{lr} \in \mathbb{R}^{D_f \times H_{lr} \times W_{lr}},$$ with output

$$F^{hr} \in \mathbb{R}^{D_f \times H_{any} \times W_{any}}.$$

The paper identifies three limitations in prior feature upsampling approaches. First, classical interpolation and some filtering methods produce semantic degradation, especially near boundaries and discontinuities. Second, some approaches require VFM-specific retraining or image-specific optimization at test time, reducing scalability. Third, AnyUp removes the need for backbone-specific retraining and supports arbitrary resolutions, but is characterized as computationally heavier. RaysUp is proposed as a joint response to these constraints: preserve semantic fidelity, generalize across backbones and tasks, and remain efficient at high resolution (Ding et al., 22 Jun 2026).

The term “universal” in the paper refers to any-resolution reconstruction; “task-agnostic” refers to training with feature reconstruction losses rather than task-specific losses; and “VFM-agnostic” refers to training on one VFM and transferring to others without retraining. This suggests that RaysUp is intended as a generic post-encoder feature operator rather than a task head specialized to a single downstream objective.

2. Architecture and geometric representation

The architecture comprises four modules: a Spatially Decoupled Guidance Encoder, Any-Resolution Cross-Attention, Ray Positional Encoding, and Geometry-Aware Neighborhood Attention (Ding et al., 22 Jun 2026).

The guidance encoder maps the RGB image to a dense guidance tensor

$$F_g(I) \in \mathbb{R}^{D_g \times H_{in} \times W_{in}}.$$

The default guidance dimension is $D_g = 256$. Instead of a single isotropic convolution, the encoder uses four directional branches:

• center-like branch: $1\times1$
• horizontal branch: $F^{lr}$0
• vertical branch: $F^{lr}$1
• diagonal/corner branch: $F^{lr}$2 dilated convolution with dilation 2

The branch outputs are given as

$F^{lr}$3

where $F^{lr}$4 is GroupNorm and $F^{lr}$5 is SiLU. Each branch then uses a shallow residual block,

$F^{lr}$6

and the final guidance tensor is

$F^{lr}$7

The purpose of this decoupled design is to encode central, horizontal, vertical, and diagonal structure separately. The paper reports that a standard $F^{lr}$8 convolution requires $F^{lr}$9 parameters, whereas the decoupled design requires $F^{hr}$0, corresponding to about 69.4% fewer parameters.

Any-Resolution Cross-Attention decouples query resolution from key/value resolution. From the same guidance tensor, RaysUp forms

$F^{hr}$1

$F^{hr}$2

and uses the low-resolution VFM features directly as values: $F^{hr}$3 Because the query grid is formed independently from the key/value grid, the target size need not be tied to a fixed integer upsampling ratio.

RayPE is the module that differentiates RaysUp from purely image-grid-based upsamplers. For a pixel coordinate

$F^{hr}$4

the ray origin and direction are defined as

$F^{hr}$5

and the ray descriptor is

$F^{hr}$6

The paper describes this as using “6D Plücker ray coordinates,” while the explicit formula is the concatenation of origin and normalized direction. This distinction is relevant because the operational representation in the method is the 6D vector $F^{hr}$7, which is then encoded harmonically and injected into attention.

3. Mathematical formulation and training procedure

RayPE applies a multi-band harmonic phase encoding to the 6D ray descriptor. For each ray component $F^{hr}$8, $F^{hr}$9, and frequency band $(H_{any}, W_{any})$0,

$(H_{any}, W_{any})$1

followed by

$(H_{any}, W_{any})$2

The implementation details report $(H_{any}, W_{any})$3 frequency bands. The paper also specifies wavelength ranges used for the origin and direction components: for ray origin, $(H_{any}, W_{any})$4 and $(H_{any}, W_{any})$5; for ray direction, $(H_{any}, W_{any})$6 and $(H_{any}, W_{any})$7 (Ding et al., 22 Jun 2026).

For each attention-head feature tensor $(H_{any}, W_{any})$8, the head is split into two halves,

$(H_{any}, W_{any})$9

and a RoPE-style rotation is applied: $D_f$0 The resulting tensors are denoted $D_f$1 and $D_f$2. The dimensionality constraint given in the paper is

$D_f$3

Reconstruction is performed by local geometry-aware cross-attention rather than global attention. For each target position $D_f$4,

$D_f$5

where $D_f$6 is a $D_f$7 neighborhood in low-resolution space. The supplementary gives the default neighborhood size as $D_f$8. To accommodate arbitrary source-target ratios, RaysUp defines dilation factors

$D_f$9

The paper interprets this as local, geometry-consistent sampling on the ray manifold.

The computational argument is explicit: global attention would cost

$$I \in \mathbb{R}^{3 \times H_{in} \times W_{in}}, \qquad F^{lr} \in \mathbb{R}^{D_f \times H_{lr} \times W_{lr}},$$0

whereas neighborhood attention costs

$$I \in \mathbb{R}^{3 \times H_{in} \times W_{in}}, \qquad F^{lr} \in \mathbb{R}^{D_f \times H_{lr} \times W_{lr}},$$1

This is a principal source of the method’s efficiency.

Training uses a frozen encoder $$I \in \mathbb{R}^{3 \times H_{in} \times W_{in}}, \qquad F^{lr} \in \mathbb{R}^{D_f \times H_{lr} \times W_{lr}},$$2. Given a high-resolution image $$I \in \mathbb{R}^{3 \times H_{in} \times W_{in}}, \qquad F^{lr} \in \mathbb{R}^{D_f \times H_{lr} \times W_{lr}},$$3 and a downsampled image $$I \in \mathbb{R}^{3 \times H_{in} \times W_{in}}, \qquad F^{lr} \in \mathbb{R}^{D_f \times H_{lr} \times W_{lr}},$$4,

$$I \in \mathbb{R}^{3 \times H_{in} \times W_{in}}, \qquad F^{lr} \in \mathbb{R}^{D_f \times H_{lr} \times W_{lr}},$$5

and RaysUp predicts

$$I \in \mathbb{R}^{3 \times H_{in} \times W_{in}}, \qquad F^{lr} \in \mathbb{R}^{D_f \times H_{lr} \times W_{lr}},$$6

The training loss is

$$I \in \mathbb{R}^{3 \times H_{in} \times W_{in}}, \qquad F^{lr} \in \mathbb{R}^{D_f \times H_{lr} \times W_{lr}},$$7

A crop-based variant is also reported: $$I \in \mathbb{R}^{3 \times H_{in} \times W_{in}}, \qquad F^{lr} \in \mathbb{R}^{D_f \times H_{lr} \times W_{lr}},$$8

The reported training setup is ImageNet, AdamW, 100,000 iterations, batch size 4, initial learning rate $$I \in \mathbb{R}^{3 \times H_{in} \times W_{in}}, \qquad F^{lr} \in \mathbb{R}^{D_f \times H_{lr} \times W_{lr}},$$9, target high-resolution images resized to $$F^{hr} \in \mathbb{R}^{D_f \times H_{any} \times W_{any}}.$$0, guidance input downsampled to $$F^{hr} \in \mathbb{R}^{D_f \times H_{any} \times W_{any}}.$$1, and training time of about 1 hour on a single NVIDIA A100. The default pose setting is Identity (Ding et al., 22 Jun 2026).

4. Empirical performance and efficiency profile

RaysUp is evaluated on semantic segmentation, depth estimation, surface normal estimation, video object segmentation, and open-vocabulary segmentation. The reported backbones include DINOv2 (ViT-S, ViT-M, ViT-L), DINOv3 (ViT-S, ViT-M, ViT-L), SigLIP2 (ViT-B, ViT-L), and PE Spatial (ViT-B, ViT-L), with DINOv2-S used as the default training backbone (Ding et al., 22 Jun 2026).

At $$F^{hr} \in \mathbb{R}^{D_f \times H_{any} \times W_{any}}.$$2, RaysUp uses 0.14M parameters, 10.17 GFLOPs, 1.26 GB GPU memory, and runs at 55 FPS. The comparison given for AnyUp at the same resolution is 0.87M parameters, 84.21 GFLOPs, 2.19 GB, and 11 FPS. The reported scaling points are:

Resolution	RaysUp runtime profile
$$F^{hr} \in \mathbb{R}^{D_f \times H_{any} \times W_{any}}.$$3	10.17 GFLOPs, 1.26 GB, 55 FPS
$$F^{hr} \in \mathbb{R}^{D_f \times H_{any} \times W_{any}}.$$4	40.67 GFLOPs, 2.69 GB, 27 FPS
$$F^{hr} \in \mathbb{R}^{D_f \times H_{any} \times W_{any}}.$$5	162.68 GFLOPs, 8.47 GB, 8 FPS
$$F^{hr} \in \mathbb{R}^{D_f \times H_{any} \times W_{any}}.$$6	still runnable at 1 FPS

The paper states that several baselines become very slow or OOM at high resolutions. This supports the characterization of RaysUp as an any-resolution method whose efficiency does not collapse immediately at large target sizes.

For semantic segmentation, the reported task-agnostic results are:

Dataset	RaysUp result
COCO-Stuff	62.32 mIoU / 81.47 Acc
Pascal-VOC	84.64 / 96.34
ADE20K	42.34 / 75.81
Cityscapes	61.88 / 93.64

The corresponding AnyUp values reported in the paper are 62.14 / 81.38, 84.18 / 96.20, 42.15 / 75.71, and 60.62 / 93.26, respectively, so RaysUp is reported to outperform AnyUp on all four benchmarks (Ding et al., 22 Jun 2026).

On NYUv2, RaysUp reports:

• Normals RMSE: 27.69
• $$F^{hr} \in \mathbb{R}^{D_f \times H_{any} \times W_{any}}.$$7: 0.4986
• $$F^{hr} \in \mathbb{R}^{D_f \times H_{any} \times W_{any}}.$$8: 0.7030
• $$F^{hr} \in \mathbb{R}^{D_f \times H_{any} \times W_{any}}.$$9: 0.7775
• Depth abs RMSE: 0.4658
• $$F_g(I) \in \mathbb{R}^{D_g \times H_{in} \times W_{in}}.$$0: 0.8103
• Depth rel RMSE: 0.3195
• $$F_g(I) \in \mathbb{R}^{D_g \times H_{in} \times W_{in}}.$$1: 0.9309

The paper characterizes these as generally the best results in the table, especially on geometric tasks, which is consistent with the method’s geometry-aware design.

On DAVIS 2017 video object segmentation, RaysUp reports:

• $$F_g(I) \in \mathbb{R}^{D_g \times H_{in} \times W_{in}}.$$2: 68.14
• $$F_g(I) \in \mathbb{R}^{D_g \times H_{in} \times W_{in}}.$$3: 74.81
• $$F_g(I) \in \mathbb{R}^{D_g \times H_{in} \times W_{in}}.$$4: 71.47

This is reported as the best result among the compared methods.

For open-vocabulary segmentation, RaysUp is described as competitive but not always best. The reported values are:

• COCO-Stuff: 27.11 mIoU / 48.57 Acc
• Pascal-VOC: 63.28 / 87.34
• ADE20K: 20.54 / 42.71
• Cityscapes: 37.24 / 59.68

The paper notes that LoftUp and AnyUp slightly surpass RaysUp on some open-vocabulary metrics and attributes some of LoftUp’s gains to SAM-based auxiliary supervision, which RaysUp does not use. This is an important qualification: the geometry-aware formulation appears particularly effective for dense geometric and video tasks, but does not dominate uniformly across all semantic evaluation settings.

5. Transfer behavior, ablations, and design rationale

A major claim of RaysUp is that it generalizes across different VFMs without retraining. The reported cross-backbone results are consistent with that claim (Ding et al., 22 Jun 2026). Examples given in the paper include:

• DINOv2 ViT-S: AnyUp 84.18 mIoU, depth RMSE 0.478 / 0.324; RaysUp 84.64, 0.465 / 0.319
• DINOv3 ViT-L: AnyUp 87.48, 0.402 / 0.257; RaysUp 88.07, 0.398 / 0.255
• SigLIP2 ViT-L: AnyUp 79.16, 0.726 / 0.474; RaysUp 80.10, 0.725 / 0.472
• PE Spatial ViT-L: AnyUp 82.23, 0.635 / 0.404; RaysUp 82.75, 0.613 / 0.388

The any-to-any resolution evaluation on Pascal-VOC includes source-target pairs $$F_g(I) \in \mathbb{R}^{D_g \times H_{in} \times W_{in}}.$$5, $$F_g(I) \in \mathbb{R}^{D_g \times H_{in} \times W_{in}}.$$6, $$F_g(I) \in \mathbb{R}^{D_g \times H_{in} \times W_{in}}.$$7, $$F_g(I) \in \mathbb{R}^{D_g \times H_{in} \times W_{in}}.$$8, $$F_g(I) \in \mathbb{R}^{D_g \times H_{in} \times W_{in}}.$$9, and $D_g = 256$0. RaysUp is reported to be best or second-best across all settings, and the paper emphasizes that it remains stable at the highest scales without OOM. Two reported examples are $D_g = 256$1: 84.81 mIoU / 96.37 Acc and $D_g = 256$2: 84.67 / 96.35.

The ablation studies clarify which design choices matter most:

Ablation axis	Reported outcome
Guidance encoder design	Decoupled-Branch: 82.17, 0.14M params
Guidance dimension	$D_g = 256$3: 82.17, 0.14M params
Number of conv blocks	$D_g = 256$4: 82.17, 0.14M params
Positional encoding	RayPE: 82.17 avg
Image pose	Identity 82.17; DA3-Base 82.44

The encoder-design ablation compares Single-Branch (81.87, 0.266M params), Dual-Branch (82.03, 0.66M), Multi-Branch (82.09, 0.268M), and Decoupled-Branch (82.17, 0.14M). The preferred guidance dimension is $D_g = 256$5, as larger values yield only marginal gains with higher parameter counts. The default $D_g = 256$6 block depth sacrifices a small amount of accuracy relative to $D_g = 256$7 but preserves the method’s light footprint.

The positional encoding ablation is especially consequential. The reported averages are:

• None: 81.05
• RoPE: 81.92
• SinRays: 81.59
• RayPE: 82.17

This indicates that positional encoding is not a marginal detail but a substantive contributor to performance. The image-pose ablation further shows that stronger pose estimates via DA3 improve results slightly, with DA3-Base reaching 82.44, but the default remains Identity for efficiency. A plausible implication is that RaysUp can exploit stronger geometry cues when they are available, without requiring them for its standard operating point.

The supplementary also reports that training with crops improves semantic segmentation further; for example, on Cityscapes, 63.04 mIoU is reported versus 61.88 without crop-based training, at the cost of about 4 hours of training.

6. Interpretation, limitations, and position within feature upsampling research

RaysUp’s principal conceptual contribution is the redefinition of feature upsampling as transport from a coarse set of rays to a denser set of rays, rather than interpolation on the image lattice (Ding et al., 22 Jun 2026). In the paper’s framing, ordinary image-space proximity is an unreliable guide to correspondence under perspective projection: adjacent pixels can lie on different surfaces at depth discontinuities, while spatially separated pixels can belong to the same physical structure. RayPE and geometry-aware local attention are intended to compensate for this mismatch by biasing aggregation toward angular and structural consistency.

This positioning explains the method’s particularly strong results on depth, surface normals, and video object segmentation. The paper’s qualitative claims are that RaysUp produces fewer holes than JAFAR, AnyUp, and FeatUp; sharper boundaries than LoftUp; better geometric consistency in depth and normal estimation; more robust inter-frame consistency in video segmentation; and clearer PCA projections preserving the underlying feature space. These observations are consistent with the claim that geometry-aware correspondence is especially beneficial when the downstream task is sensitive to discontinuities, shape, or multiview-consistent structure.

Several limitations are also explicit. First, the geometry is only implicit: RaysUp does not perform a full 3D reconstruction or rendering process. Second, the paper frequently uses identity pose and simplified camera assumptions; although stronger camera information improves results slightly, it is not central to the default setup. Third, RaysUp is not uniformly best across all task categories; in some open-vocabulary settings, methods such as LoftUp remain slightly stronger, which the authors partly attribute to auxiliary supervision not used by RaysUp. Fourth, the paper notes that some mathematical expressions are imperfectly typeset, especially the frequency formula for RayPE and some dimensions, although the intended method is presented clearly enough to reconstruct the implementation.

A common misconception would be to interpret the “ray” terminology as implying explicit volumetric reasoning or full projective scene reconstruction. The paper does not support that reading. RaysUp uses rays as a positional and correspondence prior inside an efficient local cross-attention operator. Its geometry is therefore representational rather than physically simulated. This suggests that its primary significance lies in introducing geometric bias into universal feature upsampling without losing the efficiency required for large-scale dense prediction.

In the literature position established by the paper, RaysUp is best understood as a lightweight universal upsampler that improves the accuracy-efficiency trade-off relative to AnyUp and other task-agnostic upsamplers while retaining arbitrary-resolution support and cross-backbone transfer. Its strongest documented attributes are 0.14M parameters, 10.17 GFLOPs at $D_g = 256$8, 55 FPS, operation up to $D_g = 256$9 at 1 FPS, and performance gains across segmentation, geometry, and video benchmarks. Within that scope, RaysUp functions as a general-purpose dense feature reconstruction layer for frozen VFMs rather than a task-specific decoder or a full geometric vision system (Ding et al., 22 Jun 2026).