Adaptive Guided Upsampling (AGU)

• Adaptive Guided Upsampling (AGU) is a class of methods that reconstructs high-resolution outputs from sparse or noisy inputs using auxiliary guidance signals such as RGB images and semantic labels.
• It employs techniques like deformable sampling, spatially variant kernels, and attention-based aggregation to optimize interpolation and preserve boundary fidelity.
• Empirical results in depth upsampling, semantic segmentation, and low-light enhancement demonstrate AGU’s efficiency and enhanced reconstruction quality over traditional methods.

Adaptive Guided Upsampling (AGU) is best understood as a family of guidance-conditioned upsampling and densification methods rather than as a single canonical algorithm. Across the cited literature, the common objective is to reconstruct a dense, high-resolution, or semantically enriched output from a low-resolution, sparse, noisy, or otherwise impoverished input by making interpolation, aggregation, or regularization depend on auxiliary structure. That guidance may be a registered RGB image, structured edges, semantic labels, high-resolution shallow features, query features, flow-aligned frames, low-light guidance images, or curvature proxies, while the adaptive component may take the form of geodesic weights, spatially variant kernels, deformable sampling offsets, query-key attention, class-conditioned coefficients, or curvature-biased feature expansion (1608.00753, Kim et al., 2019, Du et al., 2024).

1. Terminological scope and historical development

The exact term “Adaptive Guided Upsampling” is not used uniformly. Some papers describe “guided depth upsampling,” “semantically guided” interpolation, “Guided Upsampling Module,” “Guided Linear Upsampling,” “Guided Attentive Interpolation,” or “Adaptive Query-guided Upsampling,” while nonetheless instantiating the same underlying pattern: a target signal is upsampled under the control of another signal whose structure is assumed to be informative for the reconstruction (1608.00753, Mazzini, 2018, Song et al., 2023, Cheng et al., 3 Jan 2026).

In early depth-oriented formulations, the problem is posed as estimating a dense high-resolution depth map from sparse or low-resolution depth together with a registered high-resolution image. “Robust High Quality Image Guided Depth Upsampling” (Liu et al., 2015) addresses noisy Time-of-Flight depth by combining a robust pixel-by-patch data term, a color-guided smoothness term, and a data-driven adaptive bandwidth. “Semantically Guided Depth Upsampling” (1608.00753) extends guidance beyond intensity cues by incorporating structured edge detection and semantic scene labeling into a geodesic distance measure, together with a locally planar scene model and a global energy minimization framework.

Later work recasts AGU in explicitly learned operators. “Guided Upsampling Network for Real-Time Semantic Segmentation” (Mazzini, 2018) learns a dense offset field that steers the final upsampling grid in semantic segmentation. “Deformable kernel networks for guided depth map upsampling” (Kim et al., 2019) predicts sparse, spatially variant kernels and deformable sampling locations. “CAD-PU: A Curvature-Adaptive Deep Learning Solution for Point Set Upsampling” (Lin et al., 2020) shifts the same logic into 3D geometry, where local curvature becomes the guidance field that controls point allocation. “Guided Linear Upsampling” (Song et al., 2023) formulates each high-resolution pixel as a linear interpolation of two low-resolution pixels with optimized indices and weights. “LDA-AQU: Adaptive Query-guided Upsampling via Local Deformable Attention” (Du et al., 2024) and “Cross-Layer Attentive Feature Upsampling for Low-latency Semantic Segmentation” (Cheng et al., 3 Jan 2026) reinterpret upsampling as dynamic feature reassembly driven by attention. Recent task-specific variants include guided flow refinement in video frame interpolation and class-conditioned guided filtering for low-light enhancement (Wu et al., 20 Nov 2025, Dcosta et al., 2 Oct 2025).

2. Formal structure of AGU operators

A broad mathematical template appears in the learned guided filtering literature: $$\hat f_{\mathbf p} = \sum_{j=1}^{K} w_j(\mathbf p; f,g)\, f\!\left(\phi_j(\mathbf p; f,g)\right),$$ where $f$ is the target signal to be upsampled, $g$ is the guidance signal, $w_j$ are content-dependent weights, and $\phi_j$ are sampling locations or supports (Kim et al., 2019). This formulation is sufficiently general to cover classical guided filtering, deformable sparse kernels, and several attention-based reassembly schemes.

In linear-model AGU, the support is minimal and the adaptivity lies in the choice of neighbors and interpolation coefficient. Guided Linear Upsampling writes

$$\hat{T}_p = \omega_{ab}\, T'_a + (1-\omega_{ab})\, T'_b,\quad a,b \in \Omega_p^{\downarrow},$$

with $\Theta_p=\{a,b,\omega_{ab}\}$ optimized from the high-resolution guidance image by minimizing self-upsampling error (Song et al., 2023). In low-light AGU, the guided linear form is extended to brightness and sharpness compensation: $$O_p = A_k\,(G_p + \tau_p + \xi_p) + B_k, \qquad O_f = I(A_k)\,\bigl(G_f + \tau_p + \xi_p\bigr) + I(B_k),$$ where $G_f$ is the high-resolution low-light guidance image, $\tau_p$ is a class-based brightness correction, and $f$0 is a class-based sharpness correction (Dcosta et al., 2 Oct 2025).

In attention-based AGU, upsampling becomes adaptive aggregation of value features under query-key affinities. Guided Attentive Interpolation uses

$f$1

with the query constructed from concatenated high-resolution and coarsely upsampled low-resolution features, and keys and values derived from the low-resolution semantic branch (Cheng et al., 3 Jan 2026). LDA-AQU writes the final aggregation as

$f$2

but crucially samples $f$3 and $f$4 at deformed, query-guided neighbor coordinates rather than on a fixed grid (Du et al., 2024).

A residual-refinement form appears in flow upsampling: $f$5 where bilinearly upsampled low-resolution flow is corrected by a refinement network conditioned on high-resolution image features (Wu et al., 20 Nov 2025). This residual form preserves the stability of a simple base upsampler while allowing content-aware correction near edges and fine motion boundaries.

3. Guidance signals and adaptation mechanisms

The guidance signal in AGU is not restricted to raw intensity. In semantically guided depth upsampling, three signals are used jointly: intensity or color, structured edge detection, and semantic scene labeling. These are combined within a geodesic distance measure so that pixels that are separated by strong edges or semantic boundaries become geodesically far even if they are spatially close, thereby enabling boundary-preserving interpolation and support propagation within coherent regions (1608.00753).

Depth-oriented robust variational AGU additionally adapts the shape of the loss. In “Robust High Quality Image Guided Depth Upsampling” (Liu et al., 2015), the exponential error norm is modulated by a spatially varying bandwidth $f$6, estimated data-adaptively. Large $f$7 in flat regions supports denoising, while small $f$8 near discontinuities suppresses smoothing across edges. This makes the method guided not only by color but also by the evolving depth structure itself.

Learned kernel AGU uses the guidance signal to determine both which samples to use and how strongly to use them. DKN employs two streams, one for the high-resolution RGB image and one for the bicubic-upsampled depth, and combines them multiplicatively to predict per-pixel kernel weights and 2D offsets (Kim et al., 2019). The adaptive component is therefore dual: weights are spatially variant, and the neighborhood itself is deformable.

Segmentation-oriented AGU often uses high-resolution shallow features as the guidance signal because they retain fine spatial detail lost in deep encoders. GUM predicts a dense 2D Guidance Offset Table from either large receptive-field features, high-resolution early activations, or a fusion of both, then uses those offsets to steer nearest-neighbor or bilinear upsampling of low-resolution logits (Mazzini, 2018). GAI sharpens this idea by explicitly treating high-resolution features as queries and low-resolution semantic features as keys and values, so that spatial detail guides the interpolation of semantic content (Cheng et al., 3 Jan 2026).

In LDA-AQU, the guidance signal is the query feature at the projected high-resolution location itself. The method uses that query both to predict deformed neighbor coordinates and to compute attention weights over those neighbors (Du et al., 2024). A plausible implication is that AGU can be organized into two separable subproblems: adaptive support selection and adaptive aggregation.

Task-specific AGU can replace semantic or photometric guidance with other structural fields. CAD-PU uses discrete curvature proxies to bias feature expansion toward high-curvature surface regions, which is the 3D analogue of allocating interpolation capacity to structurally informative areas (Lin et al., 2020). Low-light AGU introduces Laplacian-of-Gaussian edge classes and brightness-difference classes to drive class-wise sharpness and brightness compensation (Dcosta et al., 2 Oct 2025). VTinker’s Guided Flow Upsampling uses high-resolution input frames as guidance to refine bilinearly upsampled flows, emphasizing that AGU is especially effective when the guidance and the upsampled field share the same coordinate system and visible boundaries (Wu et al., 20 Nov 2025).

4. Major methodological families

A first family is variational and model-based AGU. “Robust High Quality Image Guided Depth Upsampling” (Liu et al., 2015) formulates upsampling as minimization of a data term and a smoothness term, both built from a robust exponential error norm. The data term is pixel-by-patch rather than pixel-by-pixel, the smoothness term is color-guided but also depth-aware through the current estimate, and the bandwidth field $f$9 is learned by steepest gradient descent. “Semantically Guided Depth Upsampling” (1608.00753) similarly uses a global optimization framework, but models depth by locally planar elements,

$g$0

and couples planes through geodesic distances derived from image edges and semantic labels. In both cases, AGU is expressed as adaptive regularization rather than as a feed-forward kernel.

A second family is explicit kernel and interpolation AGU. DKN and FDKN directly parameterize the weighted averaging process with learned sparse, deformable, spatially variant kernels. Their core aggregation with residual connection,

$g$1

learns both the coefficients and the fractional sampling positions (Kim et al., 2019). GLU makes a more constrained but analytically transparent choice: two support pixels per high-resolution location, selected and weighted under a color-line model and, optionally, jointly optimized downsampling (Song et al., 2023). These methods are AGU in the narrow, operator-design sense: the upsampler itself is the central learned or optimized object.

A third family is offset- and resampling-based AGU. GUM introduces a learned 2D offset field $g$2 at the target resolution and replaces fixed-grid bilinear or nearest-neighbor upsampling with content-guided sampling (Mazzini, 2018). VTinker’s Guided Flow Upsampling does not predict interpolation kernels; instead, it refines bilinearly upsampled flow with a guidance CNN and a residual correction (Wu et al., 20 Nov 2025). In both cases, adaptivity is primarily geometric: the model changes where information is taken from.

A fourth family is attention-based AGU. GAI uses Criss-Cross Attention to reduce the cost of cross-resolution content-aware interpolation while keeping long-range dependencies along rows and columns (Cheng et al., 3 Jan 2026). LDA-AQU interprets upsampling as local self-attention and then adds query-guided deformation of neighbor positions, presenting many prior upsamplers as special cases of a more general local-attention formulation (Du et al., 2024). These methods replace fixed interpolation kernels with dynamic, query-conditioned affinity operators.

A fifth family is task-conditioned AGU beyond 2D depth or logits. CAD-PU allocates more output points to high-curvature surface regions through curvature-adaptive feature expansion and curvature-oriented adversarial training (Lin et al., 2020). Low-light AGU extends guided filtering with brightness-aware and scale-aware corrections so that the guidance image can be darker, noisier, and of different resolution from the target (Dcosta et al., 2 Oct 2025). This suggests that AGU is not tied to a particular modality; it is a reusable principle for guidance-conditioned reconstruction.

5. Empirical profile and application domains

The reported empirical record is heterogeneous because AGU is applied to different tasks, datasets, and output representations, but the recurring pattern is improved boundary fidelity, reduced leakage across structural boundaries, and better use of sparse or low-resolution information.

Setting	Reported result	Source
GAI-based semantic segmentation	78.8 mIoU with 22.3 FPS on Cityscapes; 80.6 mIoU with 64.5 FPS on CamVid	(Cheng et al., 3 Jan 2026)
Guided depth upsampling with DKN/FDKN	NYU v2, scale $g$3: DJFR 5.86 cm RMSE, DKN 3.26 cm, FDKN 3.58 cm; FDKN runs about 17 times faster and takes 0.01 seconds for a HR image of size 640 x 480	(Kim et al., 2019)
Query-guided deformable attention upsampling	Performance enhancements of 1.7 AP, 1.5 AP, 2.0 PQ, and 2.5 mIoU compared to the baseline models	(Du et al., 2024)
Guided flow upsampling in VFI edge regions	Edge PSNR 22.64 to 25.07, edge SSIM 0.7575 to 0.8143, edge IoU 0.2343 to 0.3926 on Xiph-2K	(Wu et al., 20 Nov 2025)
Low-light AGU runtime	For base 848×480 → 1920×1080: AGU ≈ 15.8 ms; low-light UNet ≈ 15.9 ms; total pipeline stays within 66 ms/frame	(Dcosta et al., 2 Oct 2025)
Guided Upsampling Module in real-time segmentation	69.64% mIoU, 33.3 FPS on Cityscapes val; 70.4% mIoU on Cityscapes test	(Mazzini, 2018)

Depth upsampling papers repeatedly report that guidance quality matters most at boundaries and under severe sparsity. “Semantically Guided Depth Upsampling” (1608.00753) reports superior performance over the state-of-the-art even for very sparse measurements and emphasizes sharper depth discontinuities aligned with object contours. “Robust High Quality Image Guided Depth Upsampling” (Liu et al., 2015) reports encouraging performance even in the case of large upsampling, e.g. $g$4 and $g$5, while reducing texture copy artifacts and blur of depth discontinuity.

In semantic segmentation, AGU improves the last-mile decoding step without requiring a heavy full-resolution decoder. GUM improves trimap IoU near boundaries and yields a gain of roughly two mIoU points over a bilinear baseline at real-time speed (Mazzini, 2018). GAI and LDA-AQU report larger gains by making the interpolation kernel itself content-dependent, with LDA-AQU showing improvements across object detection, instance segmentation, panoptic segmentation, and semantic segmentation (Du et al., 2024), and GAI reporting new state-of-the-art results of low-latency semantic segmentation (Cheng et al., 3 Jan 2026).

Outside depth and segmentation, the same empirical pattern persists. VTinker attributes ghosting and discontinuities in high-resolution video frame interpolation to inaccurate high-resolution flows obtained from low-resolution estimates, and uses guided flow refinement to make flow edges clearer before block-based texture mapping (Wu et al., 20 Nov 2025). CAD-PU shows that curvature-adaptive allocation improves surface reconstruction quality over uniform deep upsampling baselines (Lin et al., 2020). Low-light AGU shows that guided upsampling can be used not only for spatial enlargement but also for the joint optimization of denoising, sharpening, and brightness alignment under real-time constraints (Dcosta et al., 2 Oct 2025).

6. Limitations, misconceptions, and open problems

A common misconception is that AGU denotes a single named method. The literature instead uses multiple task-specific names—“semantically guided,” “Guided Upsampling Module,” “Guided Linear Upsampling,” “Guided Attentive Interpolation,” “Adaptive Query-guided Upsampling,” and explicit “Adaptive Guided Upsampling” in low-light enhancement—while sharing a common principle of guidance-conditioned interpolation or regularization (1608.00753, Mazzini, 2018, Song et al., 2023, Du et al., 2024, Dcosta et al., 2 Oct 2025, Cheng et al., 3 Jan 2026).

A second misconception is that AGU is equivalent to bilinear upsampling followed by generic sharpening. Several papers show that the crucial degrees of freedom are more structured: sparse deformable sampling in DKN, geodesic support and planar priors in semantically guided depth upsampling, learned sampling-grid offsets in GUM, query-conditioned attention in GAI and LDA-AQU, or class-wise brightness and sharpness correction in low-light AGU (1608.00753, Mazzini, 2018, Kim et al., 2019, Du et al., 2024, Dcosta et al., 2 Oct 2025, Cheng et al., 3 Jan 2026).

The limitations are correspondingly method-specific. Color-guided depth upsampling can suffer texture-copy artifacts or blur of depth discontinuity when guidance is used too literally, which is precisely the failure mode targeted by robust variational AGU (Liu et al., 2015). Guided Linear Upsampling assumes shared local affinities between source and target and may fail when the operator introduces new edges not present in the guidance, or when source and target affinities diverge strongly (Song et al., 2023). Low-light AGU is intended mainly for about $g$6 operation and is sensitive to outliers in the small curated training set used to estimate class-wise parameters (Dcosta et al., 2 Oct 2025). CAD-PU shows that pure curvature sampling degrades; the best reported behavior occurs with a hybrid strategy at $g$7, indicating that guidance fields can be noisy and should not necessarily dominate the allocation rule (Lin et al., 2020). LDA-AQU requires tuning of deformation range $g$8, offset groups, and channel reduction factor, while GAI trades modest extra latency for richer semantic interpolation (Du et al., 2024, Cheng et al., 3 Jan 2026). VTinker notes that extremely large motions outside the low-resolution receptive field, texture-less regions, and complex occlusion patterns remain challenging even with guided flow upsampling (Wu et al., 20 Nov 2025).

The broader open question is how AGU should balance guidance strength against model flexibility. Some papers favor explicit geometric priors and robust energies; others favor deformable kernels, attention, or learned class-wise transforms. This suggests that AGU is most coherently viewed as a design space rather than a single architecture class: a structurally informative guide determines where information should come from, how much it should contribute, and where smoothing or propagation should stop.