360-AGD: Panoramic Affordance Grounding Benchmark

• 360-AGD is a pioneering dataset and benchmark that operationalizes holistic affordance grounding by localizing functional regions in 360° indoor panoramas.
• It features two splits, Easy and Hard, to evaluate models under varying levels of scene complexity and distortions intrinsic to equirectangular projections.
• The benchmark employs keypoint-derived soft heatmaps along with innovations like DASM and OSDH to tackle ERP distortion, semantic dispersion, and cross-scale alignment difficulties.

Searching arXiv for papers related to “360-AGD” and its primary source paper. 360-AGD is the dataset and benchmark introduced in "PanoAffordanceNet: Towards Holistic Affordance Grounding in 360° Indoor Environments" (Zhu et al., 10 Mar 2026). It operationalizes holistic affordance grounding in 360° indoor environments by asking a model to localize functional regions corresponding to affordance queries directly in panoramic scenes. In contrast to conventional affordance grounding, which is typically object-centric and restricted to perspective views, 360-AGD is scene-level and omnidirectional: the target affordance may occur in multiple spatially disjoint regions across a single panorama, and the input geometry is represented in equirectangular projection rather than an ordinary perspective image.

1. Problem definition and motivation

360-AGD was created to address a specific mismatch between existing affordance grounding benchmarks and the perceptual setting of embodied agents. The motivating claim is that a robot does not physically operate only within a cropped frontal view, so perspective-only affordance understanding mismatches the true embodied interaction space. Existing datasets such as AGD20K address exocentric perspective affordance grounding, while 3D scene affordance datasets use different annotation regimes, impose different representational burdens, and incur much higher acquisition and supervision costs (Zhu et al., 10 Mar 2026).

The benchmark therefore targets interactive affordance grounding in panoramic indoor scenes. Given a panoramic indoor image and an affordance query, or an affordance-conditioned text representation, the model must localize the corresponding functional region or regions in the scene. This differs sharply from traditional formulations that assume a limited camera field of view, center a salient object, and ask for a functional part of that object. In 360-AGD, the affordance may be distributed across a cluttered room, tied to long-range spatial dependencies, and expressed over a full $360^\circ$ action space rather than a narrow field of view.

The paper identifies several task-defining difficulties. Equirectangular Projection (ERP) distortion is latitude-dependent: structures near the poles are stretched, whereas equatorial regions preserve sharper local boundaries. Semantic dispersion arises because affordance-relevant evidence is spread over an entire room with multiple candidate objects and a high-entropy background. Cross-scale alignment difficulty appears because affordance semantics are abstract, while the visual cues supporting them may be local and small even though the relevant context is global and scene-wide. The paper further emphasizes non-uniform sampling in panoramas, topological discontinuity, fragmented predictions, and semantic drift under low supervision, especially when one object supports multiple affordances such as grasp and sit on different parts of a sofa.

2. Dataset composition and split design

360-AGD is described as the first high-quality panoramic affordance grounding dataset and the first public benchmark built specifically for grounding affordances directly in panoramic indoor scenes (Zhu et al., 10 Mar 2026). It is a multi-source dataset organized into Easy and Hard splits.

Split	Primary sources	Original resolution
Easy Split	360-Indoor, Gibson	around $512 \times 1024$
Hard Split	PanoContext, Sun360	up to $4552 \times 9104$

The Easy Split is intended to test core affordance grounding ability in relatively clean, simple indoor scenes. The Hard Split is designed to probe generalization under more challenging conditions, with more visually complex and higher-fidelity panoramas. The paper states that the statistics of the two splits reveal both the semantic diversity of indoor scenes and the distinct data distributions between Easy and Hard.

A notable limitation of the published description is that it does not provide exact counts for the number of panoramas, environments, or scenes in each split, and it does not specify train/validation/test cardinalities in the provided text. It also does not report exact percentages of split composition. The precise summary is therefore that 360-AGD is a multi-source panoramic indoor dataset organized into Easy and Hard splits, with source datasets and approximate source resolutions explicitly described, but without total image counts in the provided text.

3. Annotation design and supervision regime

The annotation protocol is central to the benchmark’s identity. 360-AGD targets 19 affordance classes in complex indoor panoramas, including examples such as sit, lie, rest_arm, and display (Zhu et al., 10 Mar 2026). A defining issue in panoramic annotation is that one affordance class may correspond to multiple spatially disjoint regions in the same panorama. Annotators are therefore instructed to mark all visible instances of each affordance class.

The protocol excludes heavily occluded interaction regions whose boundaries are unclear. Instead of dense segmentation masks, the dataset adopts a keypoint-based supervision strategy. Annotators place multiple keypoints inside the boundaries of all valid, non-occluded interaction areas, and these keypoints are transformed into a continuous per-affordance probability heatmap by applying a Gaussian kernel to each point. The resulting supervision is therefore neither bounding boxes nor pixel-perfect masks, but keypoint-derived soft heatmaps.

This annotation design supports the benchmark’s formulation as a one-shot affordance grounding task in panoramas. The model receives a $360^\circ$ indoor image together with a text-conditioned affordance representation and outputs a dense heatmap over the panorama indicating where the queried affordance holds. The paper notes that “a Gaussian blur is then applied to each binary annotation mask to obtain the final soft supervision heatmaps,” which is consistent with the keypoint-to-probability-map formulation.

The figure-based descriptive analyses mentioned in the paper include a word cloud of object categories, a word cloud of affordance categories, and a distribution of affordances across the Easy and Hard splits. The text states that these reveal semantic diversity and split-specific distributional differences, but the exact frequency counts are not provided in the supplied description.

4. Benchmark protocol and reference performance

Evaluation on 360-AGD follows saliency-style heatmap comparison using three metrics: KLD, SIM, and NSS (Zhu et al., 10 Mar 2026). Kullback-Leibler Divergence (KLD) measures the distributional divergence between predicted and ground-truth heatmaps, with lower values better. Similarity (SIM) measures histogram intersection, with higher values better. Normalized Scanpath Saliency (NSS) evaluates normalized prediction responses at ground-truth keypoint locations, with higher values better.

The paper establishes the first benchmark numbers on 360-AGD by comparing PanoAffordanceNet against two one-shot affordance grounding methods adapted to equirectangular panoramas: OOAL and OS-AGDO.

Split / Method	KLD	SIM	NSS
Easy / OOAL	2.868	0.117	1.267
Easy / OS-AGDO	2.853	0.124	1.299
Easy / PanoAffordanceNet	1.270	0.506	4.490
Hard / OOAL	3.067	0.097	1.484
Hard / OS-AGDO	2.965	0.115	1.484
Hard / PanoAffordanceNet	1.306	0.474	4.398

These results are presented as evidence that straightforward adaptation of perspective affordance grounding methods to ERP panoramas is insufficient. Qualitatively, the perspective-based baselines are described as suffering a “catastrophic breakdown of spatial reasoning” on 360° ERP images, with predictions that are erratic, fragmented, and semantically drifted. By contrast, PanoAffordanceNet is reported to produce cleaner, better localized, and more structurally coherent heatmaps.

The paper also highlights a fine-grained qualitative distinction: for different affordances on the same object, such as “lean_back” versus “rest_arm”, competing methods produce similar and confused maps, whereas the proposed model better differentiates them through stronger cross-modal alignment.

5. PanoAffordanceNet as the first strong baseline

The baseline and main model established for 360-AGD is PanoAffordanceNet, an end-to-end framework purpose-built for panoramic affordance grounding (Zhu et al., 10 Mar 2026). Its architecture has four major stages: a dual-encoder multimodal backbone, the Distortion-Aware Spectral Modulator (DASM), a Spherical-Aware Hierarchical Decoder with the Omni-Spherical Densification Head (OSDH), and a multi-level training objective.

The visual encoder is DINOv2 ViT-B/14, adapted using LoRA for parameter-efficient fine-tuning under sparse supervision. The language branch uses the CLIP text encoder (ViT-B/16) together with a CoOp prompt learner to produce task-adaptive affordance embeddings. DASM is designed to address ERP distortion through dual-frequency spectral distillation after cross-modal semantic injection. Its two specialized submodules are the High-Frequency Enhancement Module (HFEM), which sharpens equatorial interaction boundaries while suppressing pole-amplified artifacts, and the Low-Frequency Stabilization Module (LFSM), which maintains structural consistency near the poles to counter stretching-induced fragmentation.

OSDH addresses sparse and fragmented affordance responses. Initial class-conditioned affordance maps are first produced by the decoder and are then densified using a spherical self-similarity prior. The intended effect is to propagate sparse evidence into topologically continuous regions on the spherical manifold, which is especially important when an affordance appears in multiple disconnected places or when ERP disrupts local continuity.

The training objective combines Binary Cross-Entropy loss $\mathcal{L}_{BCE}$, KL loss $\mathcal{L}_{KL}$, and Region–Text Contrastive loss $\mathcal{L}_{RTC}$: $$\mathcal{L}_{total} = \lambda_1 \mathcal{L}_{BCE} + \lambda_2 \mathcal{L}_{KL} + \lambda_3 \mathcal{L}_{RTC}.$$ The paper’s interpretation is that BCE alone cannot recover holistic functional shapes from sparse supervision, KL adds distribution-level structural consistency, and RTC improves semantic-sensitive metrics by enforcing stronger region-text correspondence.

Ablations on the 360-AGD Hard Split quantify the contributions of the main components. The baseline without LoRA, DASM, or OSDH gives KLD 1.475, SIM 0.416, NSS 4.196. Adding LoRA improves this to 1.421 / 0.429 / 4.257. Adding OSDH on top of LoRA gives 1.380 / 0.450 / 4.317. Adding DASM on top of LoRA gives 1.359 / 0.448 / 4.339. The full model reaches 1.306 / 0.474 / 4.398. The loss ablation is similarly explicit: using only $\mathcal{L}_{BCE}$ gives 1.596 / 0.395 / 3.891, while the full objective yields 1.306 / 0.474 / 4.398.

The hyperparameter analysis further reports that for LoRA rank $r$, the best value is $512 \times 1024$0 with 1.306 / 0.474 / 4.398, and that OSDH is stable over a wide range of seed-selection values $512 \times 1024$1, including $512 \times 1024$2, $512 \times 1024$3, $512 \times 1024$4, $512 \times 1024$5, and $512 \times 1024$6 with only modest variation in KLD, SIM, and NSS.

6. Significance, limitations, and terminological context

The principal novelty of 360-AGD is not merely that it contains panoramic images, but that it defines a new holistic affordance grounding regime (Zhu et al., 10 Mar 2026). The benchmark focuses on scene-level functional reasoning over full indoor panoramas, supports multi-instance, spatially disjoint affordance grounding, and uses a supervision scheme that is feasible for large-scale annotation: multiple keypoints per affordance region transformed into soft probability heatmaps. This makes it particularly relevant for embodied AI, where agents must reason globally about where actions can occur in a room rather than only identify a functional patch on a centered object. It is also relevant for 360° perception research, because it exposes the compounded effects of ERP distortion, semantic dispersion, weak supervision, and topology disruption on a task that depends jointly on geometry and semantics.

The benchmark’s limitations are also explicit or strongly implied. Its supervision is weak rather than pixel-perfect; heavily occluded regions are omitted; and it appears limited to indoor panoramas. The paper does not present temporal information, dynamic scenes, or multimodal 3D grounding. The conclusion identifies temporal reasoning for dynamic scenes and cross-modal synergy with 3D spatial representations as future directions. A plausible implication is that the scene distribution of 360-AGD is shaped by the design choices of its source datasets and by the Easy/Hard split construction.

The term “360-AGD” is specific to this panoramic affordance-grounding benchmark and should be distinguished from other uses of AGD in the arXiv literature. Autonomous Gaussian Decomposition is an automated method for decomposing 1D spectra into Gaussian components and is unrelated to panoramic affordance grounding (Lindner et al., 2014). The paper "On generalizations of the pentagram map: discretizations of AGD flows" uses the notation $512 \times 1024$7-AGD for Adler–Gel'fand–Dickey flows and explicitly notes that it does not use the exact term “360-AGD” (Beffa, 2011). Likewise, the dynamic reconstruction method 4DGS360 is relevant to 360° Gaussian reconstruction, but the paper explicitly states that it does not use the name “360-AGD” (Jang et al., 23 Mar 2026). Within current arXiv usage, 360-AGD therefore denotes the benchmark introduced for one-shot holistic affordance grounding in 360° indoor panoramas, and not a general label for AGD methods in spectroscopy, integrable systems, or dynamic Gaussian reconstruction.