PanoAffordanceNet: 360° Indoor Affordance Grounding

• The paper introduces a holistic affordance grounding framework that localizes interaction regions in equirectangular panoramic images using semantically aligned soft heatmaps.
• It employs dual encoders with DINOv2 and CLIP, enhanced by a Distortion-Aware Spectral Modulator and an Omni-Spherical Densification Head to address ERP distortion and sparse supervision.
• Experimental results on the 360-AGD dataset demonstrate competitive performance with significant improvements in KLD, SIM, and NSS metrics across both easy and hard splits.

Searching arXiv for the cited paper and closely related affordance-grounding work. PanoAffordanceNet is an end-to-end one-shot affordance grounding framework for Holistic Affordance Grounding in 360° Indoor Environments, a vision-language grounding task introduced for embodied perception in panoramic indoor scenes (Zhu et al., 10 Mar 2026). The task requires identifying, in a full equirectangular panoramic image, all regions that support a queried affordance such as sitting, leaning, grasping, or resting an arm, rather than assigning an affordance to a single visible object in a perspective view. In this formulation, affordances are grounded at the scene level across the entire $360^\circ$ environment, with outputs represented as soft heatmaps over panoramic regions. PanoAffordanceNet is designed around three difficulties emphasized by the task setting: severe geometric distortion from equirectangular projection (ERP), semantic dispersion over wide scenes, and cross-scale alignment under sparse supervision (Zhu et al., 10 Mar 2026).

1. Task formulation and problem setting

The task addressed by PanoAffordanceNet takes as input a panoramic ERP indoor image together with an affordance query represented through text embeddings over $C$ affordance classes (Zhu et al., 10 Mar 2026). The visual encoder produces patch-level features

$\mathbf{F}_v \in \mathbb{R}^{B \times L \times D},$

and the language encoder produces text embeddings

$\mathbf{F}_t \in \mathbb{R}^{B \times C \times D}.$

The model predicts an initial affordance map

$\mathbf{A}_{\text{init} \in \mathbb{R}^{B \times C \times L},$

which is then densified into a refined map

$\mathbf{A}_{\text{refined}.$

During training and evaluation, these maps correspond to soft heatmaps over ERP image regions (Zhu et al., 10 Mar 2026).

This task differs from prior affordance grounding in three respects stated explicitly in the source paper: earlier methods are largely object-centric, perspective-view based, and typically evaluated in simplified settings where global layout reasoning is less demanding (Zhu et al., 10 Mar 2026). By contrast, PanoAffordanceNet operates on equirectangular panoramic images, performs scene-level grounding, and must account for the fact that panoramic geometry is altered by projection distortion, non-uniform sampling, and cross-boundary continuity issues.

The target of prediction is not a single connected segment. A single panorama may contain multiple valid regions for the same affordance, often spatially disjoint and distributed around the scene (Zhu et al., 10 Mar 2026). As a result, the objective is to localize all valid regions corresponding to an affordance concept with a heatmap that is semantically aligned with the text query, spatially coherent despite ERP distortion, topologically continuous on the spherical scene manifold, and robust under sparse keypoint-derived supervision.

The formulation is motivated by embodied agents that act in a 360° action space. The stated rationale is that interaction opportunities may exist behind or beside the current viewing direction, scene-level affordances depend on spatial organization across the room, and action planning benefits from a holistic map of functional regions (Zhu et al., 10 Mar 2026). This suggests a shift from local object snapshots toward room-scale functional perception.

2. Architectural design and core modules

PanoAffordanceNet is organized into four stated stages: dual-encoder feature extraction, the Distortion-Aware Spectral Modulator (DASM), a Spherical-Aware Hierarchical Decoder with Omni-Spherical Densification Head (OSDH), and multi-level training objectives (Zhu et al., 10 Mar 2026). The paper describes the framework as end-to-end and one-shot.

The visual backbone is DINOv2 ViT-B/14, adapted with LoRA on attention layers for efficient adaptation under one-shot supervision (Zhu et al., 10 Mar 2026). The language branch uses a pre-trained CLIP text encoder (ViT-B/16) together with a CoOp prompt learner, producing context-aware affordance embeddings

$\mathbf{F}_t \in \mathbb{R}^{B \times C \times D}.$

These class-level text features serve as semantic queries for grounding.

Cross-modal semantic injection is performed before distortion-aware modulation. The paper defines

$$\mathbf{F}'_v = \text{Softmax}\big( (\mathbf{F}_v \mathbf{W}_Q)(\mathbf{F}_t \mathbf{W}_K)^\top / \sqrt{d} \big) (\mathbf{F}_t \mathbf{W}_V),$$

where $\mathbf{W}_Q,\mathbf{W}_K,\mathbf{W}_V$ are learnable projections and $d$ is the attention scaling dimension (Zhu et al., 10 Mar 2026). This step conditions visual activations on affordance semantics.

The downstream decoder is described as a Spherical-Aware Hierarchical Decoder, but the paper does not specify a conventional FPN or an explicit multi-resolution pyramid (Zhu et al., 10 Mar 2026). What is specified is a lightweight transformer decoder using text embeddings as queries, cross-attention from text to refined visual features, generation of initial affordance maps, and subsequent densification via OSDH. In that sense, the hierarchy is semantic rather than an explicitly enumerated feature pyramid.

Initial grounding is produced as

$C$0

followed by densification to obtain

$C$1

The paper frames the final prediction as a panoramic heatmap per affordance class (Zhu et al., 10 Mar 2026).

3. Distortion-aware processing in ERP panoramas

The central geometry-specific component is the Distortion-Aware Spectral Modulator (DASM), inserted after backbone feature extraction and before decoding (Zhu et al., 10 Mar 2026). Its stated purpose is to mitigate latitude-dependent geometric distortion and semantic dispersion due to clutter and projection artifacts.

DASM first applies the cross-modal semantic injection described above, yielding semantically activated visual features $C$2. These features are then decomposed into high- and low-frequency components: $C$3 where $C$4 is a Laplacian operator for high-frequency content and $C$5 is a Gaussian smoothing kernel for low-frequency structure (Zhu et al., 10 Mar 2026). The implementation is therefore frequency-aware but realized in the spatial domain, not through FFTs, DCTs, or explicit Fourier transforms.

The paper attributes a specific imbalance to ERP geometry: near the equator, edges are relatively preserved, whereas near the poles, structures become stretched (Zhu et al., 10 Mar 2026). To address this, DASM uses a High-Frequency Enhancement Module (HFEM) and a Low-Frequency Stabilization Module (LFSM). HFEM is described as sharpening interaction boundaries in equatorial regions while suppressing polar artifacts, whereas LFSM preserves large-scale structure near the poles and reduces semantic fragmentation. However, the paper does not provide explicit equations for the internals of HFEM or LFSM, nor explicit formulas for latitude as a variable or for row-wise spherical priors (Zhu et al., 10 Mar 2026).

Fusion is defined through a gated residual modulation: $C$6 where $C$7 are learnable scalar weights, $C$8 is a language-driven channel gate, and $C$9 is a self-adaptive spatial gate (Zhu et al., 10 Mar 2026). Exact gate-generation formulas are not specified. Contextual re-aggregation is then performed with MHSA and FFN, producing refined features $\mathbf{F}_v \in \mathbb{R}^{B \times L \times D},$0.

The intended effect of DASM is to separate boundaries from global structure, calibrate their contribution according to ERP distortion characteristics, and restore global context after modulation (Zhu et al., 10 Mar 2026). A plausible implication is that this module functions as the architecture’s principal mechanism for compensating the mismatch between planar feature processing and spherical observation geometry.

4. Omni-Spherical Densification Head and topological continuity

The Omni-Spherical Densification Head (OSDH) is introduced to address a second failure mode of panoramic affordance grounding: sparse, fragmented activations in the initial heatmap (Zhu et al., 10 Mar 2026). The paper links this phenomenon to uneven ERP sampling, sparse keypoint-based supervision, and topological discontinuity introduced by panoramic projection.

OSDH begins from the initial affordance map

$\mathbf{F}_v \in \mathbb{R}^{B \times L \times D},$1

The refined visual features are projected onto the unit hypersphere, and pairwise cosine similarity is used to construct a symmetric affinity matrix

$\mathbf{F}_v \in \mathbb{R}^{B \times L \times D},$2

This matrix provides a self-similarity prior over panorama locations (Zhu et al., 10 Mar 2026).

A high-confidence seed set $\mathbf{F}_v \in \mathbb{R}^{B \times L \times D},$3 is selected using top-$\mathbf{F}_v \in \mathbb{R}^{B \times L \times D},$4 ranking from the initial activations. The paper does not specify in further detail whether top-$\mathbf{F}_v \in \mathbb{R}^{B \times L \times D},$5 is chosen globally per class or per sample (Zhu et al., 10 Mar 2026). A confidence map is then computed as

$\mathbf{F}_v \in \mathbb{R}^{B \times L \times D},$6

where $\mathbf{F}_v \in \mathbb{R}^{B \times L \times D},$7 and $\mathbf{F}_v \in \mathbb{R}^{B \times L \times D},$8 are the mean and standard deviation of the initial map, and $\mathbf{F}_v \in \mathbb{R}^{B \times L \times D},$9 is a temperature parameter (Zhu et al., 10 Mar 2026). This is used to suppress spurious noise and retain reliable regions.

The final densified map is given by

$\mathbf{F}_t \in \mathbb{R}^{B \times C \times D}.$0

where $\mathbf{F}_t \in \mathbb{R}^{B \times C \times D}.$1 is a learnable residual scalar (Zhu et al., 10 Mar 2026). Propagation therefore transfers activation from high-confidence seed locations to visually similar regions.

The paper interprets this procedure as restoring topological continuity on the spherical manifold, though it also states that the mechanism is based on feature affinity rather than an explicit geodesic spherical graph (Zhu et al., 10 Mar 2026). No explicit neighborhood graph from spherical coordinates, no geodesic interpolation formula, and no dedicated continuity regularizer unique to OSDH are provided. This is important for accurate characterization: OSDH is “spherical” in its intended role and in the use of panorama-wide affinity, but not in the sense of an explicitly parameterized spherical graph algorithm.

5. Supervision, objectives, and the 360-AGD dataset

Training supervision is derived from keypoint-based affordance annotation rather than dense masks (Zhu et al., 10 Mar 2026). For each affordance class in a panorama, annotators place multiple keypoints inside all valid visible interaction regions; heavily occluded regions with ambiguous boundaries are ignored; each keypoint is converted into a continuous signal by a Gaussian kernel; and the result is a probability heatmap per affordance class. The ground-truth heatmap is denoted $\mathbf{F}_t \in \mathbb{R}^{B \times C \times D}.$2, and the predicted map is $\mathbf{F}_t \in \mathbb{R}^{B \times C \times D}.$3.

The training objective combines pixel-wise, distribution-level, and region-text semantic supervision: $\mathbf{F}_t \in \mathbb{R}^{B \times C \times D}.$4 The paper states that $\mathbf{F}_t \in \mathbb{R}^{B \times C \times D}.$5 is a Binary Cross-Entropy loss, but does not provide its explicit formula (Zhu et al., 10 Mar 2026). Distribution-level topology consistency is enforced with

$\mathbf{F}_t \in \mathbb{R}^{B \times C \times D}.$6

and region-text alignment is imposed through an InfoNCE-style Region–Text Contrastive loss. For class $\mathbf{F}_t \in \mathbb{R}^{B \times C \times D}.$7, region pooling is defined as

$\mathbf{F}_t \in \mathbb{R}^{B \times C \times D}.$8

followed by

$\mathbf{F}_t \in \mathbb{R}^{B \times C \times D}.$9

The paper does not specify the numerical values of $\mathbf{A}_{\text{init} \in \mathbb{R}^{B \times C \times L},$0, $\mathbf{A}_{\text{init} \in \mathbb{R}^{B \times C \times L},$1, or $\mathbf{A}_{\text{init} \in \mathbb{R}^{B \times C \times L},$2 (Zhu et al., 10 Mar 2026).

The accompanying dataset, 360-AGD, is described as the first high-quality panoramic affordance grounding dataset for indoor environments (Zhu et al., 10 Mar 2026). It contains 19 affordance classes and is organized into an Easy Split and a Hard Split. The Easy Split is built from 360-Indoor and Gibson with original resolutions approximately

$\mathbf{A}_{\text{init} \in \mathbb{R}^{B \times C \times L},$3

while the Hard Split is built from PanoContext and Sun360 with resolutions up to

$\mathbf{A}_{\text{init} \in \mathbb{R}^{B \times C \times L},$4

The annotation protocol follows the keypoint-to-heatmap procedure above. The paper does not specify the exact number of panoramas, scenes, annotations, or train/validation/test split sizes (Zhu et al., 10 Mar 2026).

This dataset design encodes an important property of the task: a single affordance may occupy multiple spatially disjoint regions in one panorama. In that respect, 360-AGD differs from object-centric affordance datasets and from conventional referring-expression benchmarks, where targets are more often single objects or connected regions.

6. Experimental results, implementation details, and broader context

Evaluation on 360-AGD uses KLD, SIM, and NSS (Zhu et al., 10 Mar 2026). On the Easy Split, the reported results are: OOAL with KLD 2.868, SIM 0.117, NSS 1.267; OS-AGDO with KLD 2.853, SIM 0.124, NSS 1.299; and PanoAffordanceNet with KLD 1.270, SIM 0.506, NSS 4.490 (Zhu et al., 10 Mar 2026). On the Hard Split, OOAL obtains KLD 3.067, SIM 0.097, NSS 1.484; OS-AGDO obtains KLD 2.965, SIM 0.115, NSS 1.484; and PanoAffordanceNet obtains KLD 1.306, SIM 0.474, NSS 4.398 (Zhu et al., 10 Mar 2026).

On the perspective-view AGD20K benchmark, the paper reports that PanoAffordanceNet remains competitive rather than uniformly dominant (Zhu et al., 10 Mar 2026). On the Seen split, OOAL reports KLD 0.740, SIM 0.577, NSS 1.745, while PanoAffordanceNet reports KLD 0.739, SIM 0.616, NSS 1.750. On the Unseen split, OOAL reports KLD 1.070, SIM 0.461, NSS 1.503, whereas PanoAffordanceNet reports KLD 1.185, SIM 0.475, NSS 1.419. The appropriate summary stated in the source is therefore that the model is competitive on perspective data but does not dominate all metrics on the unseen split.

Ablation results on the 360-AGD Hard Split attribute gains to all three principal additions: LoRA, DASM, and OSDH (Zhu et al., 10 Mar 2026). The full model reaches KLD 1.306, SIM 0.474, NSS 4.398, compared with 1.475, 0.416, 4.196 for the variant with none of those components. Loss ablation shows that BCE alone is insufficient, and that combining $\mathbf{A}_{\text{init} \in \mathbb{R}^{B \times C \times L},$5, $\mathbf{A}_{\text{init} \in \mathbb{R}^{B \times C \times L},$6, and $\mathbf{A}_{\text{init} \in \mathbb{R}^{B \times C \times L},$7 yields the best overall performance, although the $\mathbf{A}_{\text{init} \in \mathbb{R}^{B \times C \times L},$8 configuration without BCE reaches slightly higher SIM than the full objective (Zhu et al., 10 Mar 2026). Hyperparameter analysis identifies the best reported LoRA rank as $\mathbf{A}_{\text{init} \in \mathbb{R}^{B \times C \times L},$9, and for OSDH the reported best overall setting is top-$\mathbf{A}_{\text{refined}.$0, while the method is described as robust across a broad range of seed counts (Zhu et al., 10 Mar 2026).

Implementation details provided for reproduction include: DINOv2-Base as visual encoder, CLIP as text encoder, CoOp, LoRA, AdamW, an initial learning rate of $\mathbf{A}_{\text{refined}.$1, cosine annealing, 20k iterations, batch size 4, and training on two NVIDIA A6000 GPUs (Zhu et al., 10 Mar 2026). Input panoramas are resized to

$\mathbf{A}_{\text{refined}.$2

Augmentation includes random flipping, color jittering, panorama-specific random rotations of $\mathbf{A}_{\text{refined}.$3, random scaling by $\mathbf{A}_{\text{refined}.$4, and horizontal wraparound shifts (Zhu et al., 10 Mar 2026). The exact CoOp prompt template, loss coefficients, and temperature values are not specified.

The paper also reports qualitative real-world experiments with a wearable Insta360 X4 head-mounted panoramic camera in office and home scenes, where the model grounds affordances such as sit and display under varying illumination and distortion; no quantitative field metrics are reported (Zhu et al., 10 Mar 2026).

In the broader affordance-grounding landscape, PanoAffordanceNet is positioned against perspective-view affordance grounding, referring-expression-style grounding, panoramic scene understanding, and embodied perception (Zhu et al., 10 Mar 2026). A related but complementary line is 3DAffordSplat, which studies language-guided affordance reasoning on 3D Gaussian Splatting representations and emphasizes local 3D structural fidelity rather than panorama-wide scene coverage (Wei et al., 15 Apr 2025). The contrast is informative: PanoAffordanceNet targets scene-level functional grounding in ERP panoramas, whereas 3DAffordSplat targets object-centric 3D affordance reasoning. This suggests a broader research division between global functional awareness in $\mathbf{A}_{\text{refined}.$5 observation spaces and high-fidelity local geometric reasoning in explicit 3D representations.

7. Limitations, interpretation, and research significance

The source paper does not include an extensive standalone limitations section, but it indicates several open directions through its conclusion and discussion (Zhu et al., 10 Mar 2026). The current framework focuses on static images, does not address dynamic scene understanding, and leaves integration with 3D spatial representations as future work. Specifically, the authors point to temporal reasoning for dynamic scenes and cross-modal synergy with 3D spatial representations as targets for further research.

Several architectural details are also intentionally left underspecified in the paper. DASM is described as latitude-dependent, but no explicit latitude variable or coordinate-dependent weighting function is given (Zhu et al., 10 Mar 2026). OSDH is said to restore continuity on the spherical manifold, but it relies on feature affinity rather than an explicit spherical graph or geodesic operator. The dataset is introduced as a benchmark, yet key scale statistics such as the number of panoramas and exact split sizes are not reported in the provided technical summary. These omissions do not negate the contribution, but they delimit what can be reconstructed directly from the published description.

Within the affordance-grounding literature, the significance of PanoAffordanceNet lies in treating panoramic affordance grounding as a distinct problem formulation rather than an incremental widening of perspective affordance localization (Zhu et al., 10 Mar 2026). The model’s specialized modules map closely onto the failure modes of ERP observation: DASM addresses distortion and semantic dispersion, OSDH addresses fragmentation and incomplete region recovery, and the multi-level objective addresses semantic drift under low supervision. This suggests a methodological template for future $\mathbf{A}_{\text{refined}.$6 embodied perception systems: global scene coverage, geometry-aware calibration, and query-conditioned densification are all treated as first-class requirements rather than post hoc adjustments.

A plausible implication is that PanoAffordanceNet establishes a baseline for panoramic functional perception analogous to how earlier affordance-grounding models established baselines for object-centric perspective images. In that interpretation, its main contribution is less a universal affordance architecture than a problem-specific reframing of affordance grounding around whole-environment interaction opportunity maps in $\mathbf{A}_{\text{refined}.$7 indoor spaces (Zhu et al., 10 Mar 2026).