GeomPrompt: Geometric Prompting for RGB-D Segmentation

• GeomPrompt is a lightweight cross-modal adaptation framework that synthesizes task-specific geometric prompts from RGB or degraded depth inputs to enhance segmentation performance.
• It reframes depth compensation as prompt learning by leveraging a frozen RGB-D segmentation model and optimizing solely with semantic segmentation labels, bypassing direct depth reconstruction.
• Evaluations on SUN RGB-D show GeomPrompt achieves notable mIoU gains over RGB-only inputs while offering superior computational efficiency compared to traditional monocular depth estimators.

GeomPrompt is a lightweight cross-modal adaptation framework for RGB-D semantic segmentation under missing and degraded depth. It treats the depth stream of a pretrained RGB-D segmenter not as a signal that must be metrically reconstructed, but as a geometric slot that can be populated with a task-driven prompt learned solely through downstream segmentation supervision. In its missing-depth form, GeomPrompt synthesizes this geometric input from RGB alone; in its degraded-depth form, GeomPrompt-Recovery predicts a bounded correction to corrupted depth rather than replacing it wholesale. The method is evaluated with frozen RGB-D segmentation backbones and is presented as an efficient mechanism for recovering the geometric prior useful for segmentation without any depth supervision (Jaganathan et al., 13 Apr 2026).

1. Problem setting and conceptual reframing

GeomPrompt addresses a deployment failure mode of RGB-D semantic segmentation systems: many models are trained and evaluated assuming a clean, always-available depth channel, whereas deployed robots and embodied agents may encounter depth that is absent, spatially incomplete, quantized, noisy, or corrupted by reflective or transparent materials, range limits, and sensor failures (Jaganathan et al., 13 Apr 2026). In this setting, depth is not merely an auxiliary modality. It acts as a geometric prior that helps sharpen boundaries, separate similarly colored objects, and improve layout understanding. When that prior is removed or statistically mismatched, a strong RGB-D segmenter may fail because its learned fusion machinery expects meaningful geometric input.

The central claim of GeomPrompt is that, for downstream semantic segmentation, explicit monocular depth estimation is not necessarily the best compensation strategy. A monocular depth estimator is optimized for metric or relative geometry, often with its own supervision or self-supervision, whereas the frozen segmentation backbone only requires the geometry-like signal that best supports its internal computations (Jaganathan et al., 13 Apr 2026). GeomPrompt therefore reframes compensation as prompt learning in the geometric input space expected by the pretrained segmenter. This means the learned signal should be interpreted as a task-relevant geometric prior rather than a depth estimate.

This distinction is important because GeomPrompt is not a generic RGB-to-depth front end. It is a cross-modal adapter optimized only by semantic labels. A common misconception is that the method aims to recover true depth. The paper explicitly argues the opposite: the synthesized fourth-channel signal need not be metrically faithful; it needs only to be useful for the frozen segmenter’s downstream semantic parsing (Jaganathan et al., 13 Apr 2026).

2. Formal setup and architectural design

The setup assumes a pretrained RGB-D semantic segmentation model $S$ that remains frozen during adaptation. Under missing depth, only RGB $x$ is available and a prompt generator $G$ predicts a task-driven prompt $p^*$. Under degraded depth, RGB $x$ and corrupted depth $\tilde d$ are available and a recovery network $G_R$ predicts a corrected prompt $p^*$. In both cases, prediction is written as

$\hat y = S(x, p^*),$

with

$p^* = G(x)$

for missing depth and

$x$0

for degraded depth (Jaganathan et al., 13 Apr 2026).

The “fourth channel” denotes the geometric slot expected by the RGB-D backbone, not necessarily a literal single tensor channel throughout implementation. GeomPrompt standardizes the interface by producing a prompt in image-like form, replicating it to three channels before normalization and adaptation, and then applying the preprocessing expected by the frozen segmenter. If the segmenter expects three-channel geometric input, it uses it directly; if it expects one channel, standard preprocessing reduces it in the usual way (Jaganathan et al., 13 Apr 2026).

Architecturally, GeomPrompt begins from an ImageNet-normalized RGB image $x$1. A ViT-S/16 encoder extracts patch tokens; prefix tokens such as a class token are discarded and the remaining tokens are reshaped into a 2D spatial grid. A lightweight CNN decoder then applies a $x$2 bilinear upsampling stage, two $x$3 Conv-BN-ReLU blocks, and a $x$4 projection to a single-channel low-resolution residual map $x$5 at spatial scale $x$6 (Jaganathan et al., 13 Apr 2026).

The prompt parameterization is

$x$7

Here $x$8 is a fixed anti-aliased upsampler, $x$9 is normalization, $G$0 is a lightweight residual PromptAdapter, and $G$1 is a hard low-pass projection (Jaganathan et al., 13 Apr 2026). The low-resolution design is deliberate: GeomPrompt predicts a coarse structural residual and then upsamples it, biasing the learned geometry toward smooth, stable, low-frequency scene structure rather than dense high-frequency pseudo-depth.

The PromptAdapter operates in 3-channel prompt space with a $G$2 projection from 3 to 16 channels, a $G$3 convolution from 16 to 16, and a final $G$4 projection from 16 back to 3. Its last layer is zero-initialized, so it begins as an identity map and learns only segmenter-specific refinements. After this stage, the hard low-pass projection applies average-pooling downsampling followed by bilinear upsampling by factor 2, suppressing high-frequency prompt artifacts (Jaganathan et al., 13 Apr 2026).

GeomPrompt-Recovery reuses the same philosophy when depth is present but corrupted. It adds a lightweight depth-condition encoder: the degraded depth $G$5 is replicated to 3 channels and processed by a four-layer stride-2 CNN with channel progression $G$6, producing features at the same $G$7 token-grid scale as the RGB ViT features. The RGB and degraded-depth features are concatenated channel-wise, fused by a $G$8 convolution, and decoded as in GeomPrompt. Instead of synthesizing depth from scratch, it predicts a bounded additive correction

$G$9

The correction head is zero-initialized so that training begins near identity with respect to the supplied degraded depth (Jaganathan et al., 13 Apr 2026).

3. Training objective and optimization regime

A defining property of GeomPrompt is that the segmentation backbone $p^*$0 is entirely frozen. Only the prompt-generation module is trainable, and the only task supervision is semantic segmentation labels $p^*$1. The method uses no depth supervision, no metric-depth loss, no relative-depth loss, no photometric self-supervision, no pseudo-depth teacher, and no distillation from a depth estimator (Jaganathan et al., 13 Apr 2026).

The training loss is

$p^*$2

The paper states that $p^*$3 is an Online Hard Example Mining cross-entropy loss. The TV term acts on the raw prompt to encourage smoothness, and the $p^*$4 term penalizes the low-resolution residual magnitude. The regularization weights are $p^*$5 and $p^*$6 (Jaganathan et al., 13 Apr 2026).

Optimization is staged through a residual-scale curriculum. The residual bound $p^*$7 is linearly ramped from 15 to 80 over 300 training epochs, so training begins near a safe constant prompt and gradually allows richer structure. This curriculum is empirically important: removing it and using a constant maximal scale $p^*$8 from the start reduces mIoU by 1.4 (Jaganathan et al., 13 Apr 2026).

Training on SUN RGB-D uses random resize, random crop to $p^*$9, AdamW, a poly learning-rate schedule with power 0.9, 10 warmup epochs, and an effective batch size of 32 on 8 A40 GPUs. The ViT encoder uses learning rate $x$0, while the decoder uses $x$1. Prompt outputs $x$2 are normalized using DFormer depth statistics $x$3 and $x$4, applied consistently across backbones (Jaganathan et al., 13 Apr 2026).

4. Missing-depth semantic segmentation performance

Experiments are conducted on SUN RGB-D with the standard test split and evaluate two representative frozen RGB-D segmenters: DFormer and GeminiFusion. DFormer uses a dedicated hierarchical RGB-D encoder with geometry-aware RGB-D blocks, while GeminiFusion uses lightweight pixel-wise fusion over aligned multimodal features in a shared four-stage transformer backbone (Jaganathan et al., 13 Apr 2026). The primary metric is mean Intersection over Union (mIoU), with pixel accuracy (PA) as secondary.

For missing depth, the comparison is controlled within each segmenter family: RGB preprocessing, backbone, and inference configuration remain fixed, and only the source of the geometric input changes. Baselines include ground-truth depth, RGB-only fallback implemented by zeroing all depth pixels, and three zero-shot monocular depth estimators: Metric3Dv2 (ViT-S), Depth Anything 2 standard checkpoint, and Depth Anything 2 Hypersim checkpoint (Jaganathan et al., 13 Apr 2026).

Backbone	Setting	mIoU / PA
DFormer	Ground-truth depth	51.2 / 83.4
DFormer	RGB-only	41.7 / 78.3
DFormer	GeomPrompt	47.8 / 81.6
GeminiFusion	Ground-truth depth	52.7 / 82.8
GeminiFusion	RGB-only	43.4 / 78.8
GeminiFusion	GeomPrompt	46.4 / 80.3

On DFormer, GeomPrompt achieves 47.8 mIoU / 81.6 PA, improving over RGB-only by $x$5 mIoU. It beats Depth Anything 2 standard by 3.8 points, Metric3Dv2 by 1.2 points, and slightly surpasses Depth Anything 2 Hypersim by 0.3 points. On GeminiFusion, GeomPrompt reaches 46.4 / 80.3, a $x$6 mIoU gain over RGB-only; it is essentially tied with Metric3Dv2, outperforms Depth Anything 2 Hypersim, but trails the best monocular baseline, Depth Anything 2 standard, by 1.3 mIoU (Jaganathan et al., 13 Apr 2026).

These results support a narrower claim than universal superiority over monocular depth estimators. The paper’s strongest supported conclusion is that GeomPrompt is consistently better than RGB-only fallback, often competitive with strong monocular depth baselines, and substantially cheaper (Jaganathan et al., 13 Apr 2026). The paper also notes suggestive but not fully harmonized comparisons to missing-modality methods that retrain the segmenter itself, such as M3L at 41.31 mIoU and OS-MD at 43.64 mIoU on SUN RGB-D, versus 47.8 with DFormer and 46.4 with GeminiFusion for GeomPrompt (Jaganathan et al., 13 Apr 2026).

5. Degraded-depth recovery, efficiency, and ablations

GeomPrompt-Recovery is trained using synthetic corruption of ground-truth depth. For each training image, depth is left clean with probability 0.2; otherwise exactly one corruption is sampled uniformly from {quantize, hole, dropout, noise, blur, banding, scale shift}, with severity sampled uniformly from $x$7. Evaluation focuses on quantization, dropout, and additive Gaussian noise (Jaganathan et al., 13 Apr 2026).

Under these degradations, GeomPrompt-Recovery consistently improves over feeding raw degraded depth into DFormer. The average and high-severity mIoU gains are $x$8 and $x$9 for quantization, $\tilde d$0 and $\tilde d$1 for dropout, and $\tilde d$2 and $\tilde d$3 for noise. The strongest effect appears under severe noise, where even with strong saturation artifacts the bounded residual correction recovers up to 3.6 mIoU (Jaganathan et al., 13 Apr 2026). This suggests that GeomPrompt-Recovery functions as a robustness buffer rather than as a replacement sensor model.

Efficiency is a major empirical result. On a single A40 GPU with batch size 1 and fp16 autocast, GeomPrompt runs in 7.8 ms, versus 38.3 ms for Depth Anything 2 and 71.9 ms for Metric3Dv2. FLOPs are 44.0G for GeomPrompt, compared to 280.7G and 315.2G, and parameter counts are 23.4M, 97.5M, and 37.5M respectively (Jaganathan et al., 13 Apr 2026). This reported setup makes GeomPrompt roughly 5× faster than Depth Anything 2 and over 9× faster than Metric3Dv2.

The ablation studies clarify what drives the gains. Handcrafted pseudo-depth controls—grayscale luminance, Laplacian edges, Scharr-based structural cue, and a Canny-distance blend—do not consistently beat RGB-only. On DFormer, all underperform RGB-only, with luminance falling to 25.4 mIoU and the best handcrafted cue reaching only 40.3 versus 41.7 for RGB-only; on GeminiFusion they remain near the RGB-only level, from 42.4 to 43.5 versus 43.4 (Jaganathan et al., 13 Apr 2026). This indicates that the learned prompt is not reducible to a generic edge map or intensity prior.

Among learned components, freezing the ViT encoder inside GeomPrompt causes the largest drop, 3.5 mIoU, indicating that the prompt generator itself must adapt end-to-end. Removing the low-pass projection costs only 0.2 mIoU, and removing TV and magnitude regularization costs 0.3 mIoU. Disabling the PromptAdapter causes a modest drop, which the paper interprets as evidence that input-statistics alignment to the frozen segmenter matters but is not the dominant source of improvement (Jaganathan et al., 13 Apr 2026).

6. Interpretation, limitations, and placement within geometric prompting

Qualitatively, GeomPrompt’s outputs differ visibly from metric depth maps. The prompts emphasize boundaries, planar surfaces, and semantically coherent regions more than exact geometry, and GeomPrompt-Recovery smooths high-frequency corruption artifacts while restoring object boundaries under severe sensor failures (Jaganathan et al., 13 Apr 2026). This supports the interpretation that the method learns a geometry-like representation optimized for semantic parsing rather than for geometric reconstruction.

The method also has clear assumptions. It requires access to a pretrained RGB-D segmentation model and segmentation labels for adaptation; it is not a zero-training solution. It optimizes for semantic segmentation, not for metrically correct depth, so the learned prompt should not be treated as a general-purpose geometric estimate for SLAM or manipulation planning without further validation. Reported experiments are limited to SUN RGB-D, an indoor dataset, and the missing-depth gains are stronger on DFormer than on GeminiFusion, indicating some backbone dependence (Jaganathan et al., 13 Apr 2026).

Within the broader prompt-learning literature, GeomPrompt belongs to a family of methods that manipulate geometry-bearing inputs or geometry-conditioned side channels while keeping large pretrained models largely fixed. In graph learning, the ProG benchmark formalizes prompt learning as graph transformation under a frozen pretrained GNN and organizes methods into prompts as graphs and prompts as tokens (Zi et al., 2024). In point-prompted segmentation with SAM, another work titled “GeomPrompt” generates ridge-aligned point prompts from multiscale differential geometry to target thin curvilinear structures without retraining SAM (Ball et al., 27 May 2025). In 3D point-cloud PEFT, GAPrompt adds learnable input-space point prompts, a Point Shift Prompter, and Prompt Propagation to geometry-aware adaptation of pretrained 3D transformers (Ai et al., 7 May 2025). These works operate in different modalities, but they share a common design principle: prompt learning can be framed as task-conditioned manipulation of the geometry or geometric interface expected by a pretrained model.

In that sense, GeomPrompt for RGB-D segmentation is a modality-specific instance of a broader technical pattern. Its distinctive contribution is the combination of a frozen RGB-D segmenter, a lightweight front-end adapter, and pure segmentation supervision, which together shift the objective from recovering depth signals to recovering the geometric prior that the segmenter actually uses (Jaganathan et al., 13 Apr 2026).