---
title: 'PanoAffordanceNet: 360° Indoor Affordance Grounding'
url: https://www.emergentmind.com/topics/panoaffordancenet
type: topic
---

# PanoAffordanceNet: 360° Indoor Affordance Grounding

Searching arXiv for the cited paper and closely related affordance-grounding work.
PanoAffordanceNet is an end-to-end one-shot affordance grounding framework for **Holistic Affordance Grounding in 360° Indoor Environments**, a vision-language grounding task introduced for embodied perception in panoramic indoor scenes [2603.09760]. The task requires identifying, in a full equirectangular panoramic image, **all regions that support a queried affordance** such as sitting, leaning, grasping, or resting an arm, rather than assigning an affordance to a single visible object in a perspective view. In this formulation, affordances are grounded at the **scene level** across the entire \(360^\circ\) environment, with outputs represented as soft heatmaps over panoramic regions. PanoAffordanceNet is designed around three difficulties emphasized by the task setting: severe geometric distortion from equirectangular projection (ERP), semantic dispersion over wide scenes, and cross-scale alignment under sparse supervision [2603.09760].

## 1. Task formulation and problem setting

The task addressed by PanoAffordanceNet takes as input a **panoramic ERP indoor image** together with an **affordance query** represented through text embeddings over \(C\) affordance classes [2603.09760]. The visual encoder produces patch-level features
\[
\mathbf{F}_v \in \mathbb{R}^{B \times L \times D},
\]
and the language encoder produces text embeddings
\[
\mathbf{F}_t \in \mathbb{R}^{B \times C \times D}.
\]
The model predicts an initial affordance map
\[
\mathbf{A}_{\text{init} \in \mathbb{R}^{B \times C \times L},
\]
which is then densified into a refined map
\[
\mathbf{A}_{\text{refined}.
\]
During training and evaluation, these maps correspond to **soft heatmaps** over ERP image regions [2603.09760].

This task differs from prior affordance grounding in three respects stated explicitly in the source paper: earlier methods are largely **object-centric**, **perspective-view based**, and typically evaluated in simplified settings where global layout reasoning is less demanding [2603.09760]. By contrast, PanoAffordanceNet operates on **equirectangular panoramic images**, performs **scene-level grounding**, and must account for the fact that panoramic geometry is altered by projection distortion, non-uniform sampling, and cross-boundary continuity issues.

The target of prediction is not a single connected segment. A single panorama may contain **multiple valid regions for the same affordance**, often spatially disjoint and distributed around the scene [2603.09760]. As a result, the objective is to localize all valid regions corresponding to an affordance concept with a heatmap that is semantically aligned with the text query, spatially coherent despite ERP distortion, topologically continuous on the spherical scene manifold, and robust under sparse keypoint-derived supervision.

The formulation is motivated by embodied agents that act in a **360° action space**. The stated rationale is that interaction opportunities may exist behind or beside the current viewing direction, scene-level affordances depend on spatial organization across the room, and action planning benefits from a holistic map of functional regions [2603.09760]. This suggests a shift from local object snapshots toward room-scale functional perception.

## 2. Architectural design and core modules

PanoAffordanceNet is organized into four stated stages: **dual-encoder feature extraction**, the **Distortion-Aware Spectral Modulator (DASM)**, a **Spherical-Aware Hierarchical Decoder with Omni-Spherical Densification Head (OSDH)**, and **multi-level training objectives** [2603.09760]. The paper describes the framework as end-to-end and one-shot.

The visual backbone is **DINOv2 ViT-B/14**, adapted with **LoRA** on attention layers for efficient adaptation under one-shot supervision [2603.09760]. The language branch uses a **pre-trained CLIP text encoder (ViT-B/16)** together with a **CoOp prompt learner**, producing context-aware affordance embeddings
\[
\mathbf{F}_t \in \mathbb{R}^{B \times C \times D}.
\]
These class-level text features serve as semantic queries for grounding.

Cross-modal semantic injection is performed before distortion-aware modulation. The paper defines
\[
\mathbf{F}'_v = \text{Softmax}\big( (\mathbf{F}_v \mathbf{W}_Q)(\mathbf{F}_t \mathbf{W}_K)^\top / \sqrt{d} \big) (\mathbf{F}_t \mathbf{W}_V),
\]
where \(\mathbf{W}_Q,\mathbf{W}_K,\mathbf{W}_V\) are learnable projections and \(d\) is the attention scaling dimension [2603.09760]. This step conditions visual activations on affordance semantics.

The downstream decoder is described as a **Spherical-Aware Hierarchical Decoder**, but the paper does **not specify a conventional FPN or an explicit multi-resolution pyramid** [2603.09760]. What is specified is a **lightweight transformer decoder** using text embeddings as queries, cross-attention from text to refined visual features, generation of initial affordance maps, and subsequent densification via OSDH. In that sense, the hierarchy is semantic rather than an explicitly enumerated feature pyramid.

Initial grounding is produced as
\[
\mathbf{A}_{\text{init} = (\mathbf{F}_t \cdot {\mathbf{F}''_v}^{\top}) \cdot D^{-0.5},
\]
followed by densification to obtain
\[
\mathbf{A}_{\text{refined}.
\]
The paper frames the final prediction as a panoramic heatmap per affordance class [2603.09760].

## 3. Distortion-aware processing in ERP panoramas

The central geometry-specific component is the **Distortion-Aware Spectral Modulator (DASM)**, inserted after backbone feature extraction and before decoding [2603.09760]. Its stated purpose is to mitigate **latitude-dependent geometric distortion** and **semantic dispersion due to clutter and projection artifacts**.

DASM first applies the cross-modal semantic injection described above, yielding semantically activated visual features \(\mathbf{F}'_v\). These features are then decomposed into high- and low-frequency components:
\[
\mathbf{F}_h = \nabla^2 * \mathbf{F}'_v, \quad \mathbf{F}_l = \mathcal{K}_{\sigma} * \mathbf{F}'_v,
\]
where \(\nabla^2\) is a Laplacian operator for high-frequency content and \(\mathcal{K}_{\sigma}\) is a Gaussian smoothing kernel for low-frequency structure [2603.09760]. The implementation is therefore frequency-aware but realized in the **spatial domain**, not through FFTs, DCTs, or explicit Fourier transforms.

The paper attributes a specific imbalance to ERP geometry: near the equator, edges are relatively preserved, whereas near the poles, structures become stretched [2603.09760]. To address this, DASM uses a **High-Frequency Enhancement Module (HFEM)** and a **Low-Frequency Stabilization Module (LFSM)**. HFEM is described as sharpening interaction boundaries in equatorial regions while suppressing polar artifacts, whereas LFSM preserves large-scale structure near the poles and reduces semantic fragmentation. However, the paper does **not provide explicit equations** for the internals of HFEM or LFSM, nor explicit formulas for latitude as a variable or for row-wise spherical priors [2603.09760].

Fusion is defined through a gated residual modulation:
\[
\mathbf{F}_{\text{freq} = \mathbf{F}'_v + \sum_{k \in \{h,l\} \lambda_k \, \big(\mathbf{g}_{\text{ch} \odot \mathbf{g}_{\text{sp} \odot \mathbf{F}_k\big),
\]
where \(\lambda_k\) are learnable scalar weights, \(\mathbf{g}_{\text{ch}\) is a **language-driven channel gate**, and \(\mathbf{g}_{\text{sp}\) is a **self-adaptive spatial gate** [2603.09760]. Exact gate-generation formulas are not specified. Contextual re-aggregation is then performed with **MHSA and FFN**, producing refined features \(\mathbf{F}''_v\).

The intended effect of DASM is to separate boundaries from global structure, calibrate their contribution according to ERP distortion characteristics, and restore global context after modulation [2603.09760]. A plausible implication is that this module functions as the architecture’s principal mechanism for compensating the mismatch between planar feature processing and spherical observation geometry.

## 4. Omni-Spherical Densification Head and topological continuity

The **Omni-Spherical Densification Head (OSDH)** is introduced to address a second failure mode of panoramic affordance grounding: **sparse, fragmented activations** in the initial heatmap [2603.09760]. The paper links this phenomenon to uneven ERP sampling, sparse keypoint-based supervision, and topological discontinuity introduced by panoramic projection.

OSDH begins from the initial affordance map
\[
\mathbf{A}_{\text{init} = (\mathbf{F}_t \cdot {\mathbf{F}''_v}^{\top}) \cdot D^{-0.5}.
\]
The refined visual features are projected onto the unit hypersphere, and pairwise cosine similarity is used to construct a symmetric affinity matrix
\[
\mathcal{S}_{ij} = \frac{\mathbf{f}''_{v,i} \cdot \mathbf{f}''_{v,j}}{\|\mathbf{f}''_{v,i}\| \, \|\mathbf{f}''_{v,j}\|}.
\]
This matrix provides a self-similarity prior over panorama locations [2603.09760].

A high-confidence seed set \(\mathcal{K}\) is selected using **top-\(k\) ranking** from the initial activations. The paper does not specify in further detail whether top-\(k\) is chosen globally per class or per sample [2603.09760]. A confidence map is then computed as
\[
\mathcal{C} = \text{Sigmoid}\left( \frac{\mathbf{A}_{\text{init} - \mu_{\mathbf{A}}}{\sigma_{\mathbf{A}} / T} \right),
\]
where \(\mu_{\mathbf{A}\) and \(\sigma_{\mathbf{A}\) are the mean and standard deviation of the initial map, and \(T\) is a temperature parameter [2603.09760]. This is used to suppress spurious noise and retain reliable regions.

The final densified map is given by
\[
\mathbf{A}_{\text{refined} = \mathbf{A}_{\text{init} + \alpha \cdot \max_{j \in \mathcal{K}} \big( \mathcal{S}_{ij} \cdot \mathcal{C}_j \big),
\]
where \(\alpha\) is a learnable residual scalar [2603.09760]. Propagation therefore transfers activation from high-confidence seed locations to visually similar regions.

The paper interprets this procedure as restoring **topological continuity on the spherical manifold**, though it also states that the mechanism is based on feature affinity rather than an explicit geodesic spherical graph [2603.09760]. No explicit neighborhood graph from spherical coordinates, no geodesic interpolation formula, and no dedicated continuity regularizer unique to OSDH are provided. This is important for accurate characterization: OSDH is “spherical” in its intended role and in the use of panorama-wide affinity, but not in the sense of an explicitly parameterized spherical graph algorithm.

## 5. Supervision, objectives, and the 360-AGD dataset

Training supervision is derived from **keypoint-based affordance annotation** rather than dense masks [2603.09760]. For each affordance class in a panorama, annotators place **multiple keypoints** inside all valid visible interaction regions; heavily occluded regions with ambiguous boundaries are ignored; each keypoint is converted into a continuous signal by a Gaussian kernel; and the result is a **probability heatmap** per affordance class. The ground-truth heatmap is denoted \(\hat{\mathcal{M}\), and the predicted map is \(\mathcal{M}\).

The training objective combines **pixel-wise**, **distribution-level**, and **region-text semantic** supervision:
\[
\mathcal{L}_{total} = \lambda_1 \mathcal{L}_{BCE} + \lambda_2 \mathcal{L}_{KL} + \lambda_3 \mathcal{L}_{RTC}.
\]
The paper states that \(\mathcal{L}_{BCE}\) is a **Binary Cross-Entropy loss**, but does not provide its explicit formula [2603.09760]. Distribution-level topology consistency is enforced with
\[
\mathcal{L}_{KL} = \sum_{i,j} \hat{\mathcal{M}_{i,j} \log \left( \frac{\hat{\mathcal{M}_{i,j} + \epsilon}{\mathcal{M}_{i,j} + \epsilon} \right),
\]
and region-text alignment is imposed through an InfoNCE-style Region–Text Contrastive loss. For class \(c\), region pooling is defined as
\[
\mathbf{v}_c = \left( \sum_{l=1}^{L} \hat{M}_{c,l} \mathbf{f}''_{v,l} \right) \bigg/ \left( \sum_{k} \hat{M}_{c,k} + \epsilon \right),
\]
followed by
\[
\mathcal{L}_{RTC} = -\frac{1}{C} \sum_{c=1}^{C} \log \frac{ \exp(\text{sim}(\mathbf{v}_c,\mathbf{t}_c)/\tau) }{ \sum_{k=1}^{C} \exp(\text{sim}(\mathbf{v}_c,\mathbf{t}_k)/\tau) }.
\]
The paper does **not specify** the numerical values of \(\lambda_1,\lambda_2,\lambda_3\), \(T\), or \(\tau\) [2603.09760].

The accompanying dataset, **360-AGD**, is described as the **first high-quality panoramic affordance grounding dataset** for indoor environments [2603.09760]. It contains **19 affordance classes** and is organized into an **Easy Split** and a **Hard Split**. The Easy Split is built from **360-Indoor** and **Gibson** with original resolutions approximately
\[
512 \times 1024,
\]
while the Hard Split is built from **PanoContext** and **Sun360** with resolutions up to
\[
4552 \times 9104.
\]
The annotation protocol follows the keypoint-to-heatmap procedure above. The paper does **not specify** the exact number of panoramas, scenes, annotations, or train/validation/test split sizes [2603.09760].

This dataset design encodes an important property of the task: a single affordance may occupy **multiple spatially disjoint regions** in one panorama. In that respect, 360-AGD differs from object-centric affordance datasets and from conventional referring-expression benchmarks, where targets are more often single objects or connected regions.

## 6. Experimental results, implementation details, and broader context

Evaluation on 360-AGD uses **KLD**, **SIM**, and **NSS** [2603.09760]. On the **Easy Split**, the reported results are: **OOAL** with KLD 2.868, SIM 0.117, NSS 1.267; **OS-AGDO** with KLD 2.853, SIM 0.124, NSS 1.299; and **PanoAffordanceNet** with **KLD 1.270, SIM 0.506, NSS 4.490** [2603.09760]. On the **Hard Split**, **OOAL** obtains KLD 3.067, SIM 0.097, NSS 1.484; **OS-AGDO** obtains KLD 2.965, SIM 0.115, NSS 1.484; and **PanoAffordanceNet** obtains **KLD 1.306, SIM 0.474, NSS 4.398** [2603.09760].

On the perspective-view **AGD20K** benchmark, the paper reports that PanoAffordanceNet remains competitive rather than uniformly dominant [2603.09760]. On the **Seen** split, **OOAL** reports KLD 0.740, SIM 0.577, NSS 1.745, while **PanoAffordanceNet** reports **KLD 0.739, SIM 0.616, NSS 1.750**. On the **Unseen** split, **OOAL** reports KLD 1.070, SIM 0.461, NSS 1.503, whereas **PanoAffordanceNet** reports KLD 1.185, SIM 0.475, NSS 1.419. The appropriate summary stated in the source is therefore that the model is **competitive** on perspective data but does not dominate all metrics on the unseen split.

Ablation results on the **360-AGD Hard Split** attribute gains to all three principal additions: **LoRA**, **DASM**, and **OSDH** [2603.09760]. The full model reaches **KLD 1.306, SIM 0.474, NSS 4.398**, compared with **1.475, 0.416, 4.196** for the variant with none of those components. Loss ablation shows that **BCE alone is insufficient**, and that combining \(\mathcal{L}_{BCE}\), \(\mathcal{L}_{KL}\), and \(\mathcal{L}_{RTC}\) yields the best overall performance, although the \(\mathcal{L}_{KL}+\mathcal{L}_{RTC}\) configuration without BCE reaches slightly higher SIM than the full objective [2603.09760]. Hyperparameter analysis identifies the best reported **LoRA rank** as \(r=16\), and for OSDH the reported best overall setting is **top-\(k = 10\)**, while the method is described as robust across a broad range of seed counts [2603.09760].

Implementation details provided for reproduction include: **DINOv2-Base** as visual encoder, **CLIP** as text encoder, **CoOp**, **LoRA**, **AdamW**, an initial learning rate of \(\mathbf{1\times10^{-5}}\), **cosine annealing**, **20k iterations**, **batch size 4**, and training on **two NVIDIA A6000 GPUs** [2603.09760]. Input panoramas are resized to
\[
560 \times 1120.
\]
Augmentation includes random flipping, color jittering, panorama-specific random rotations of \(\pm 3^\circ\), random scaling by \(\pm 5\%\), and horizontal wraparound shifts [2603.09760]. The exact CoOp prompt template, loss coefficients, and temperature values are not specified.

The paper also reports qualitative real-world experiments with a **wearable Insta360 X4 head-mounted panoramic camera** in office and home scenes, where the model grounds affordances such as **sit** and **display** under varying illumination and distortion; no quantitative field metrics are reported [2603.09760].

In the broader affordance-grounding landscape, PanoAffordanceNet is positioned against perspective-view affordance grounding, referring-expression-style grounding, panoramic scene understanding, and embodied perception [2603.09760]. A related but complementary line is **3DAffordSplat**, which studies language-guided affordance reasoning on **3D Gaussian Splatting** representations and emphasizes local 3D structural fidelity rather than panorama-wide scene coverage [2504.11218]. The contrast is informative: PanoAffordanceNet targets **scene-level functional grounding** in ERP panoramas, whereas 3DAffordSplat targets **object-centric 3D affordance reasoning**. This suggests a broader research division between global functional awareness in \(360^\circ\) observation spaces and high-fidelity local geometric reasoning in explicit 3D representations.

## 7. Limitations, interpretation, and research significance

The source paper does not include an extensive standalone limitations section, but it indicates several open directions through its conclusion and discussion [2603.09760]. The current framework focuses on **static images**, does not address dynamic scene understanding, and leaves integration with **3D spatial representations** as future work. Specifically, the authors point to **temporal reasoning for dynamic scenes** and **cross-modal synergy with 3D spatial representations** as targets for further research.

Several architectural details are also intentionally left underspecified in the paper. DASM is described as **latitude-dependent**, but no explicit latitude variable or coordinate-dependent weighting function is given [2603.09760]. OSDH is said to restore continuity on the spherical manifold, but it relies on feature affinity rather than an explicit spherical graph or geodesic operator. The dataset is introduced as a benchmark, yet key scale statistics such as the number of panoramas and exact split sizes are not reported in the provided technical summary. These omissions do not negate the contribution, but they delimit what can be reconstructed directly from the published description.

Within the affordance-grounding literature, the significance of PanoAffordanceNet lies in treating panoramic affordance grounding as a **distinct problem formulation** rather than an incremental widening of perspective affordance localization [2603.09760]. The model’s specialized modules map closely onto the failure modes of ERP observation: DASM addresses distortion and semantic dispersion, OSDH addresses fragmentation and incomplete region recovery, and the multi-level objective addresses semantic drift under low supervision. This suggests a methodological template for future \(360^\circ\) embodied perception systems: global scene coverage, geometry-aware calibration, and query-conditioned densification are all treated as first-class requirements rather than post hoc adjustments.

A plausible implication is that PanoAffordanceNet establishes a baseline for panoramic functional perception analogous to how earlier affordance-grounding models established baselines for object-centric perspective images. In that interpretation, its main contribution is less a universal affordance architecture than a problem-specific reframing of affordance grounding around **whole-environment interaction opportunity maps** in \(360^\circ\) indoor spaces [2603.09760].

Source: https://www.emergentmind.com/topics/panoaffordancenet