---
title: '360-AGD: Panoramic Affordance Grounding Benchmark'
url: https://www.emergentmind.com/topics/360-agd
type: topic
---

# 360-AGD: Panoramic Affordance Grounding Benchmark

Searching arXiv for recent papers related to “360-AGD” and its primary source paper.
360-AGD is the dataset and benchmark introduced in "PanoAffordanceNet: Towards Holistic Affordance Grounding in 360° Indoor Environments" [2603.09760]. It operationalizes **holistic affordance grounding in 360° indoor environments** by asking a model to localize functional regions corresponding to affordance queries directly in panoramic scenes. In contrast to conventional affordance grounding, which is typically object-centric and restricted to perspective views, 360-AGD is **scene-level and omnidirectional**: the target affordance may occur in multiple spatially disjoint regions across a single panorama, and the input geometry is represented in equirectangular projection rather than an ordinary perspective image.

## 1. Problem definition and motivation

360-AGD was created to address a specific mismatch between existing affordance grounding benchmarks and the perceptual setting of embodied agents. The motivating claim is that a robot does not physically operate only within a cropped frontal view, so perspective-only affordance understanding mismatches the true embodied interaction space. Existing datasets such as AGD20K address exocentric perspective affordance grounding, while 3D scene affordance datasets use different annotation regimes, impose different representational burdens, and incur much higher acquisition and supervision costs [2603.09760].

The benchmark therefore targets **interactive affordance grounding in panoramic indoor scenes**. Given a panoramic indoor image and an affordance query, or an affordance-conditioned text representation, the model must localize the corresponding functional region or regions in the scene. This differs sharply from traditional formulations that assume a limited camera field of view, center a salient object, and ask for a functional part of that object. In 360-AGD, the affordance may be distributed across a cluttered room, tied to long-range spatial dependencies, and expressed over a full \(360^\circ\) action space rather than a narrow field of view.

The paper identifies several task-defining difficulties. **Equirectangular Projection (ERP) distortion** is latitude-dependent: structures near the poles are stretched, whereas equatorial regions preserve sharper local boundaries. **Semantic dispersion** arises because affordance-relevant evidence is spread over an entire room with multiple candidate objects and a high-entropy background. **Cross-scale alignment difficulty** appears because affordance semantics are abstract, while the visual cues supporting them may be local and small even though the relevant context is global and scene-wide. The paper further emphasizes **non-uniform sampling in panoramas**, **topological discontinuity**, fragmented predictions, and **semantic drift under low supervision**, especially when one object supports multiple affordances such as *grasp* and *sit* on different parts of a sofa.

## 2. Dataset composition and split design

360-AGD is described as the **first high-quality panoramic affordance grounding dataset** and the **first public benchmark** built specifically for grounding affordances directly in panoramic indoor scenes [2603.09760]. It is a multi-source dataset organized into **Easy** and **Hard** splits.

| Split | Primary sources | Original resolution |
|---|---|---|
| Easy Split | 360-Indoor, Gibson | around \(512 \times 1024\) |
| Hard Split | PanoContext, Sun360 | up to \(4552 \times 9104\) |

The **Easy Split** is intended to test core affordance grounding ability in relatively clean, simple indoor scenes. The **Hard Split** is designed to probe generalization under more challenging conditions, with more visually complex and higher-fidelity panoramas. The paper states that the statistics of the two splits reveal both the **semantic diversity** of indoor scenes and the **distinct data distributions** between Easy and Hard.

A notable limitation of the published description is that it does **not** provide exact counts for the number of panoramas, environments, or scenes in each split, and it does **not** specify train/validation/test cardinalities in the provided text. It also does not report exact percentages of split composition. The precise summary is therefore that 360-AGD is a **multi-source panoramic indoor dataset organized into Easy and Hard splits**, with source datasets and approximate source resolutions explicitly described, but without total image counts in the provided text.

## 3. Annotation design and supervision regime

The annotation protocol is central to the benchmark’s identity. 360-AGD targets **19 affordance classes** in complex indoor panoramas, including examples such as *sit*, *lie*, *rest_arm*, and *display* [2603.09760]. A defining issue in panoramic annotation is that one affordance class may correspond to **multiple spatially disjoint regions** in the same panorama. Annotators are therefore instructed to mark **all visible instances** of each affordance class.

The protocol excludes **heavily occluded interaction regions whose boundaries are unclear**. Instead of dense segmentation masks, the dataset adopts a **keypoint-based supervision strategy**. Annotators place **multiple keypoints inside the boundaries of all valid, non-occluded interaction areas**, and these keypoints are transformed into a **continuous per-affordance probability heatmap** by applying a **Gaussian kernel** to each point. The resulting supervision is therefore neither bounding boxes nor pixel-perfect masks, but **keypoint-derived soft heatmaps**.

This annotation design supports the benchmark’s formulation as a **one-shot affordance grounding** task in panoramas. The model receives a \(360^\circ\) indoor image together with a text-conditioned affordance representation and outputs a **dense heatmap over the panorama** indicating where the queried affordance holds. The paper notes that “a Gaussian blur is then applied to each binary annotation mask to obtain the final soft supervision heatmaps,” which is consistent with the keypoint-to-probability-map formulation.

The figure-based descriptive analyses mentioned in the paper include a **word cloud of object categories**, a **word cloud of affordance categories**, and a **distribution of affordances across the Easy and Hard splits**. The text states that these reveal semantic diversity and split-specific distributional differences, but the exact frequency counts are not provided in the supplied description.

## 4. Benchmark protocol and reference performance

Evaluation on 360-AGD follows saliency-style heatmap comparison using three metrics: **KLD**, **SIM**, and **NSS** [2603.09760]. **Kullback-Leibler Divergence (KLD)** measures the distributional divergence between predicted and ground-truth heatmaps, with lower values better. **Similarity (SIM)** measures histogram intersection, with higher values better. **Normalized Scanpath Saliency (NSS)** evaluates normalized prediction responses at ground-truth keypoint locations, with higher values better.

The paper establishes the first benchmark numbers on 360-AGD by comparing **PanoAffordanceNet** against two one-shot affordance grounding methods adapted to equirectangular panoramas: **OOAL** and **OS-AGDO**.

| Split / Method | KLD | SIM | NSS |
|---|---:|---:|---:|
| Easy / OOAL | 2.868 | 0.117 | 1.267 |
| Easy / OS-AGDO | 2.853 | 0.124 | 1.299 |
| Easy / PanoAffordanceNet | 1.270 | 0.506 | 4.490 |
| Hard / OOAL | 3.067 | 0.097 | 1.484 |
| Hard / OS-AGDO | 2.965 | 0.115 | 1.484 |
| Hard / PanoAffordanceNet | 1.306 | 0.474 | 4.398 |

These results are presented as evidence that straightforward adaptation of perspective affordance grounding methods to ERP panoramas is insufficient. Qualitatively, the perspective-based baselines are described as suffering a **“catastrophic breakdown of spatial reasoning”** on 360° ERP images, with predictions that are **erratic, fragmented, and semantically drifted**. By contrast, PanoAffordanceNet is reported to produce cleaner, better localized, and more structurally coherent heatmaps.

The paper also highlights a fine-grained qualitative distinction: for different affordances on the same object, such as **“lean_back”** versus **“rest_arm”**, competing methods produce similar and confused maps, whereas the proposed model better differentiates them through stronger cross-modal alignment.

## 5. PanoAffordanceNet as the first strong baseline

The baseline and main model established for 360-AGD is **PanoAffordanceNet**, an end-to-end framework purpose-built for panoramic affordance grounding [2603.09760]. Its architecture has four major stages: a dual-encoder multimodal backbone, the **Distortion-Aware Spectral Modulator (DASM)**, a **Spherical-Aware Hierarchical Decoder** with the **Omni-Spherical Densification Head (OSDH)**, and a **multi-level training objective**.

The visual encoder is **DINOv2 ViT-B/14**, adapted using **LoRA** for parameter-efficient fine-tuning under sparse supervision. The language branch uses the **CLIP text encoder (ViT-B/16)** together with a **CoOp prompt learner** to produce task-adaptive affordance embeddings. DASM is designed to address ERP distortion through **dual-frequency spectral distillation** after cross-modal semantic injection. Its two specialized submodules are the **High-Frequency Enhancement Module (HFEM)**, which sharpens equatorial interaction boundaries while suppressing pole-amplified artifacts, and the **Low-Frequency Stabilization Module (LFSM)**, which maintains structural consistency near the poles to counter stretching-induced fragmentation.

OSDH addresses sparse and fragmented affordance responses. Initial class-conditioned affordance maps are first produced by the decoder and are then densified using a spherical self-similarity prior. The intended effect is to propagate sparse evidence into topologically continuous regions on the spherical manifold, which is especially important when an affordance appears in multiple disconnected places or when ERP disrupts local continuity.

The training objective combines **Binary Cross-Entropy loss** \(\mathcal{L}_{BCE}\), **KL loss** \(\mathcal{L}_{KL}\), and **Region–Text Contrastive loss** \(\mathcal{L}_{RTC}\):
\[
\mathcal{L}_{total} = \lambda_1 \mathcal{L}_{BCE} + \lambda_2 \mathcal{L}_{KL} + \lambda_3 \mathcal{L}_{RTC}.
\]
The paper’s interpretation is that BCE alone cannot recover holistic functional shapes from sparse supervision, KL adds distribution-level structural consistency, and RTC improves semantic-sensitive metrics by enforcing stronger region-text correspondence.

Ablations on the **360-AGD Hard Split** quantify the contributions of the main components. The baseline without LoRA, DASM, or OSDH gives **KLD 1.475, SIM 0.416, NSS 4.196**. Adding **LoRA** improves this to **1.421 / 0.429 / 4.257**. Adding **OSDH** on top of LoRA gives **1.380 / 0.450 / 4.317**. Adding **DASM** on top of LoRA gives **1.359 / 0.448 / 4.339**. The full model reaches **1.306 / 0.474 / 4.398**. The loss ablation is similarly explicit: using only \(\mathcal{L}_{BCE}\) gives **1.596 / 0.395 / 3.891**, while the full objective yields **1.306 / 0.474 / 4.398**.

The hyperparameter analysis further reports that for LoRA rank \(r\), the best value is **\(r=16\)** with **1.306 / 0.474 / 4.398**, and that OSDH is stable over a wide range of seed-selection values \(k\), including **\(k=1\)**, **\(k=5\)**, **\(k=10\)**, **\(k=20\)**, and **\(k=30\)** with only modest variation in KLD, SIM, and NSS.

## 6. Significance, limitations, and terminological context

The principal novelty of 360-AGD is not merely that it contains panoramic images, but that it defines a new **holistic affordance grounding regime** [2603.09760]. The benchmark focuses on **scene-level functional reasoning over full indoor panoramas**, supports **multi-instance, spatially disjoint affordance grounding**, and uses a supervision scheme that is feasible for large-scale annotation: **multiple keypoints per affordance region transformed into soft probability heatmaps**. This makes it particularly relevant for **embodied AI**, where agents must reason globally about where actions can occur in a room rather than only identify a functional patch on a centered object. It is also relevant for **360° perception research**, because it exposes the compounded effects of ERP distortion, semantic dispersion, weak supervision, and topology disruption on a task that depends jointly on geometry and semantics.

The benchmark’s limitations are also explicit or strongly implied. Its supervision is **weak rather than pixel-perfect**; **heavily occluded regions are omitted**; and it appears limited to **indoor panoramas**. The paper does not present **temporal information**, **dynamic scenes**, or **multimodal 3D grounding**. The conclusion identifies **temporal reasoning for dynamic scenes** and **cross-modal synergy with 3D spatial representations** as future directions. A plausible implication is that the scene distribution of 360-AGD is shaped by the design choices of its source datasets and by the Easy/Hard split construction.

The term **“360-AGD”** is specific to this panoramic affordance-grounding benchmark and should be distinguished from other uses of **AGD** in the arXiv literature. **Autonomous Gaussian Decomposition** is an automated method for decomposing 1D spectra into Gaussian components and is unrelated to panoramic affordance grounding [1409.2840]. The paper "On generalizations of the pentagram map: discretizations of AGD flows" uses the notation **\(k\)-AGD** for Adler–Gel'fand–Dickey flows and explicitly notes that it does **not** use the exact term **“360-AGD”** [1103.5047]. Likewise, the dynamic reconstruction method **4DGS360** is relevant to 360° Gaussian reconstruction, but the paper explicitly states that it does **not** use the name **“360-AGD”** [2603.21618]. Within current arXiv usage, 360-AGD therefore denotes the benchmark introduced for **one-shot holistic affordance grounding in 360° indoor panoramas**, and not a general label for AGD methods in spectroscopy, integrable systems, or dynamic Gaussian reconstruction.

Source: https://www.emergentmind.com/topics/360-agd