Upsample Anything: A Simple and Hard to Beat Baseline for Feature Upsampling (2511.16301v1)

Abstract: We present \textbf{Upsample Anything}, a lightweight test-time optimization (TTO) framework that restores low-resolution features to high-resolution, pixel-wise outputs without any training. Although Vision Foundation Models demonstrate strong generalization across diverse downstream tasks, their representations are typically downsampled by 14x/16x (e.g., ViT), which limits their direct use in pixel-level applications. Existing feature upsampling approaches depend on dataset-specific retraining or heavy implicit optimization, restricting scalability and generalization. Upsample Anything addresses these issues through a simple per-image optimization that learns an anisotropic Gaussian kernel combining spatial and range cues, effectively bridging Gaussian Splatting and Joint Bilateral Upsampling. The learned kernel acts as a universal, edge-aware operator that transfers seamlessly across architectures and modalities, enabling precise high-resolution reconstruction of features, depth, or probability maps. It runs in only $\approx0.419 \text{s}$ per 224x224 image and achieves state-of-the-art performance on semantic segmentation, depth estimation, and both depth and probability map upsampling.

• The paper introduces a test-time optimization framework that leverages anisotropic Gaussian splatting for edge-preserving feature upsampling from vision foundation models without retraining.
• It achieves state-of-the-art performance across segmentation, depth, and 3D upsampling tasks while scaling linearly in time and memory with output resolution.
• The method generalizes across diverse modalities and architectures, offering a plug-and-play, efficient solution for feature enhancement in modern vision pipelines.

Unified and Efficient Feature Upsampling via Per-Image Test-Time Optimization

Introduction

The proliferation of Vision Foundation Models (VFMs) such as DINO, CLIP, and their derivatives has transformed encoder design for vision tasks—but these models inherently produce highly downsampled feature maps (often 14× or 16× smaller than original input), posing a challenge for recovering spatially precise, pixel-level predictions. Canonical approaches rely on heavy, decoupled decoder architectures or retrained upsamplers which are computationally intensive, memory-inefficient, and domain-specific. The paper "Upsample Anything: A Simple and Hard to Beat Baseline for Feature Upsampling" (2511.16301) introduces a test-time optimization (TTO) framework to upsample low-resolution features from any pretrained VFM, requiring no dataset-level retraining and generalizing robustly to disparate modalities, including feature, depth, segmentation, and 3D data.

The method leverages a lightweight, per-image optimization that fits spatially and photometrically adaptive Gaussian splatting kernels, unifying ideas from Joint Bilateral Upsampling (JBU) and modern Gaussian Splatting for efficient, edge-aware interpolation. The approach is simple, fast (0.419 s for a 224×224 image), and resolution-agnostic, while outperforming all prior dataset-level and test-time feature upsamplers across multiple tasks.

Figure 1: Comparative overview—dataset-level upsamplers require paired data/training and support only 2D features, whereas Upsample Anything requires only one HR image and generalizes to feature, depth, segmentation, and 3D signals via TTO.

Methodology: Upsample Anything

Core Algorithmic Pipeline

The Upsample Anything pipeline consists of two sequential phases: (1) test-time optimization of per-pixel anisotropic Gaussian parameters by reconstructing a high-resolution guidance signal (typically RGB), and (2) application of the learned splatting kernels to transfer low-resolution feature, depth, or probability maps to high resolution. The upsampling rule does not synthesize new content but only mixes neighboring values using kernels tuned to fit both spatial and range (appearance) coherence within each image instance.

Figure 2: Method overview—anisotropic per-pixel Gaussian splats are optimized on RGB reconstruction, then transferred to upsample foundation features.

Theoretical Foundation

Upsample Anything unifies the expressiveness of Gaussian Splatting with the edge-preserving properties of JBU. Compared to standard JBU with fixed, isotropic, global kernels, Upsample Anything learns pixelwise anisotropic covariances ($\sigma_x$, $\sigma_y$, $\theta$, $\sigma_r$), enabling orientation- and content-adaptive kernels. This continuous generalization not only improves expressivity but allows effective transferability between image, feature, and segmentation domains.

Unlike dataset-level upsamplers which require heavy retraining for novel backbones or resolutions, the entire framework is model-agnostic and stateless, fully amortized via TTO per image.

Implementation and Optimization

Initialization uses heuristics for stability, and Adam optimizer is employed for a fixed number (50) of iterations without batch processing. Time and memory grow only linearly with spatial resolution, contrasting with AnyUp, whose memory and compute scale quadratically.

Empirical Evaluation

Quantitative Results Across Benchmarks

Extensive evaluation demonstrates state-of-the-art performance for Upsample Anything on standard semantic segmentation datasets (COCO, PASCAL-VOC, ADE20K) and depth estimation (NYUv2, Middlebury), outperforming both dataset-level (FeatUp, LoftUp, JAFAR, AnyUp) and prior TTO-based methods, especially for geometry-centric tasks (depth and surface normal estimation).

Robustness to Resolution Degradation

Upsample Anything retains structural sharpness, coherent boundaries, and fine details even when input feature maps are at extremely low spatial resolutions (e.g., 7×7 or 4×4), while previous SOTA upsamplers exhibit over-smoothed outputs. This is visualized in comparative qualitative evaluations.

Figure 3: Visual comparison—Upsample Anything maintains edge sharpness and structure at all tested input resolutions, in contrast to previous SOTA methods.

Architecture-Generalization

By design, Upsample Anything is agnostic to feature extractor architecture. Qualitative experiments across ConvNeXt, CLIP, DINOv1, DINOv2, and DINOv3 confirm consistently higher edge fidelity and more meaningful feature clustering for all types of backbone outputs.

Figure 4: Visual comparison across architectures—sharp boundaries and rich texture from Upsample Anything across ConvNeXt, CLIP, and DINO variants.

Feature Consistency Analysis

Investigations of cross-image feature similarity demonstrate sharper, semantically precise similarity maps after upsampling with Upsample Anything, indicating better suitability for downstream retrieval or few-shot segmentation.

Figure 5: Cosine similarity analysis highlights localized, object-aware feature alignment achieved by Upsample Anything.

Computational and Memory Efficiency

AnyUp incurs quadratic memory and compute overhead with increasing output resolution, leading to out-of-memory failures beyond $512\times 512$. In contrast, Upsample Anything scales linearly, remaining efficient even for $1024\times 1024$ and beyond.

3D Feature Volume Upsampling and Multi-Modal Generalization

As the kernel learning is independent of signal type, Upsample Anything is leveraged for 3D feature upsampling—mapping sparse 2D features into dense 3D volumes, preserving depth continuity and segmentation-relevant structure without any retraining or additional supervision.

Figure 6: 3D feature upsampling—recovered feature volumes preserve edge and geometric continuity across depth layers.

Visualization and Introspection

Parameter visualizations reveal that the learned anisotropic Gaussian blobs track local scene structure, encoding coherent orientations and magnitudes.

Figure 7: Example Gaussian blob overlay—learned kernels adapt spatially to underlying image structure.

Probabilistic-upsampled segmentation maps exhibit superior boundary sharpness and semantic consistency, motivating a segment-then-upsample paradigm.

Figure 8: Segment-then-upsample pipeline yields sharp object boundaries and preserves semantic alignment.

Limitations

TTO-based approaches inherently overfit to noisy or corrupted inputs, faithfully reconstructing noise present in the guidance image and propagating this into the upsampled features, whereas direct upsampling of backbone features (e.g., AnyUp) shows greater robustness under high perturbation or low-SNR scenarios.

Figure 9: Robustness analysis—AnyUp is stable to noise, while TTO-based Upsample Anything overfits to corruptions on noisy inputs.

Implications and Future Directions

Upsample Anything supports plug-and-play insertion between VFMs and shallow decoders, without requiring architecture-specific retraining or dataset-level annotation. This universal, edge-preserving, and efficient upsampler is highly attractive for new architectures, resource-constrained platforms, and rapidly evolving foundation model ecosystems. The demonstrated generality—from 2D feature and probability maps to 3D volumetric upsampling—suggests seamless adaptability across tasks and modalities.

Future directions include robustifying the TTO process to improve performance in the presence of noise or domain artifacts and extending the methodology toward self-supervised or multi-modal upsampling in novel domains.

Conclusion

The Upsample Anything framework (2511.16301) establishes a new, simple, and highly effective baseline for universal feature upsampling. By harmonizing anisotropic Gaussian splatting and joint bilateral filtering under efficient TTO, it achieves resolution- and model-agnostic, edge-preserving upsampling within sub-second runtime per image. The results robustly challenge the necessity of dataset-level upsampler retraining and open the path to flexible, efficient, and generalizable feature upsampling in modern vision pipelines.