Omni-Spherical Densification Head (OSDH)

• OSDH is a head module that converts sparse, fragmented affordance activations in ERP images into continuous maps using a unit-hypersphere projection and global cosine affinity.
• It builds a fully connected similarity graph over all visual tokens, bypassing explicit spherical transforms and handling distortions like polar stretching and seam discontinuities.
• The method enhances spatial integrity and improves performance metrics (e.g., SIM, NSS) by propagating high-confidence seeds through confidence-gated max propagation.

Omni-Spherical Densification Head (OSDH) is a head module in PanoAffordanceNet for holistic affordance grounding in 360° indoor environments. It is introduced to convert sparse, fragmented initial affordance activations in equirectangular projection (ERP) images into continuous, geometrically coherent regions by exploiting global self-similarity on a feature-space hypersphere rather than explicit ERP-to-3D spherical coordinate transforms or spherical convolutions. Within the framework, OSDH operates after distortion-aware feature calibration by the Distortion-Aware Spectral Modulator (DASM) and after initial map generation by a lightweight transformer decoder, taking refined visual features and initial affordance maps as input and producing densified affordance maps before loss computation (Zhu et al., 10 Mar 2026).

1. Problem setting and functional role

OSDH is motivated by a specific failure mode of panoramic affordance grounding: in ERP images, initial activations are sparse and fragmented because of non-uniform sampling and severe geometric distortion, especially near the poles. The module is designed to restore topological continuity from these sparse activations and to mitigate cross-boundary discontinuities, including the left–right ERP seam and polar stretching, by operating on a global affinity graph that connects visually similar tokens irrespective of ERP location (Zhu et al., 10 Mar 2026).

In PanoAffordanceNet, DASM first produces distortion-robust, affordance-aware visual features $F_v''$ by calibrating latitude-dependent ERP distortions and balancing high- and low-frequency content. A lightweight transformer decoder then uses text queries to derive initial affordance maps $A_{\text{init}}$, but these remain sparse and fragmented. OSDH takes $F_v''$ and $A_{\text{init}}$, builds an omni-spherical affinity structure, and propagates confident seed activations to densify $A_{\text{init}}$ into continuous maps $A_{\text{refined}}$. The division of labor is explicit: DASM prepares geometry-aware features, whereas OSDH repairs continuity by exploiting spherical self-similarity.

This placement makes OSDH a decoder-side refinement mechanism rather than a backbone operator. It is applied after the decoder has produced $A_{\text{init}}$ from $F_v''$, and before losses are computed. There is no OSDH-specific loss; the module is optimized end-to-end through the broader training objective.

2. Architectural formulation

OSDH receives visual features $F_v'' \in \mathbb{R}^{B \times L \times D}$ from DASM and initial affordance maps $$A_{\text{init}} \in \mathbb{R}^{B \times C \times L}$$ from the transformer decoder. The output is $A_{\text{init}}$0, intended to preserve global continuity and coherent extents (Zhu et al., 10 Mar 2026).

For the reported configuration, the backbone is ViT-B/14 with LoRA. With input resolution $A_{\text{init}}$1 and patch size $A_{\text{init}}$2, the token count is approximately

$A_{\text{init}}$3

This tokenization scale is central to the design, because OSDH constructs a global $A_{\text{init}}$4 affinity matrix over all tokens.

The module is parameter-light. Aside from a learnable residual scalar $A_{\text{init}}$5, it relies on non-parametric computations: L2 feature normalization, cosine similarity, top-$A_{\text{init}}$6 selection, sigmoid gating, and elementwise max propagation. No attention or deformable sampling is used inside OSDH, and no explicit convolutions, strides, paddings, or normalizations beyond L2 norm and sigmoid are used.

A concise summary of the interface is as follows:

Component	Tensor / operation	Role
Input features	$A_{\text{init}}$7	Distortion-robust visual tokens from DASM
Initial maps	$A_{\text{init}}$8	Sparse class-wise affordance activations
Output maps	$A_{\text{init}}$9	Densified, continuity-aware activations

The paper’s terminology “omni-spherical” does not denote explicit spherical coordinates. Instead, it refers to projecting tokens to a unit hypersphere in feature space and defining a fully connected similarity graph over all tokens. This distinction is important because the method models spherical continuity implicitly through global affinities rather than through spherical kernels or seam-specific indexing.

3. Computational mechanism

The forward pass begins by projecting refined visual features onto the unit hypersphere. For each token $F_v''$0, the feature is normalized to unit length, which facilitates cosine-affinity computation. A global symmetric similarity matrix $F_v''$1 is then constructed over all token pairs: $F_v''$2 for tokens $F_v''$3 (Zhu et al., 10 Mar 2026).

The initial affordance maps are produced upstream by scaled dot-product similarity between text queries and visual features: $F_v''$4 where $F_v''$5 are text queries and $F_v''$6 are visual features.

For each affordance class $F_v''$7, OSDH selects high-confidence seed indices $F_v''$8 via top-$F_v''$9 ranking on $A_{\text{init}}$0. It then computes a normalized confidence map using sigmoid: $A_{\text{init}}$1 where $A_{\text{init}}$2 and $A_{\text{init}}$3 are the mean and standard deviation over the $A_{\text{init}}$4 tokens, and $A_{\text{init}}$5 is a temperature. The confidence term suppresses spurious seeds before propagation.

Densification is performed by max propagation through the affinity graph: $A_{\text{init}}$6 where $A_{\text{init}}$7 is the learnable residual scalar. This update preserves the original activation and adds an affinity-driven response from confident seeds. Because the neighborhood is defined globally by cosine similarity across all tokens rather than by local pixel adjacency, OSDH uses self-similarity on the token graph rather than local spatial diffusion.

The reported forward procedure is therefore: input $A_{\text{init}}$8 and $A_{\text{init}}$9; normalize features; build $A_{\text{init}}$0; for each class, select top-$A_{\text{init}}$1 seeds, compute confidence, propagate via the max over seed affinities, and form $A_{\text{init}}$2.

4. Topology handling in ERP imagery

A central property of OSDH is that it addresses ERP boundary and distortion artifacts without explicit seam-wrapping logic. Because the affinity matrix spans all tokens globally, regions separated by the left–right ERP seam can still be linked if their features are similar. The same mechanism applies to pole regions: stretched areas near the poles can propagate from visually similar equatorial tokens, which are less affected by local geometric deformation (Zhu et al., 10 Mar 2026).

This design implies that topological continuity emerges from affinity-driven propagation rather than from hand-crafted spherical operators. The method does not perform explicit ERP-to-3D sphere coordinate transforms, does not specify longitude–latitude equations for OSDH, and does not employ spherical convolutions. The spherical aspect of the module is therefore feature-geometric rather than coordinate-geometric.

The upstream role of DASM is consequential here. The paper states that DASM further stabilizes features across latitudes, which strengthens OSDH’s affinity structure. A plausible implication is that OSDH’s propagation quality depends not only on the refinement rule itself but also on the degree to which upstream features remain comparable across ERP distortions. This is consistent with the reported limitation that, without robust upstream calibration, affinities may become noisier near the poles.

A common misconception would be to interpret OSDH as a spherical convolutional layer. The paper explicitly distinguishes it from such methods: it leverages global token affinities, uses no learned offsets, and recovers continuity through self-similarity and confidence-gated propagation.

5. Optimization, complexity, and implementation

OSDH is trained end-to-end under the same multi-level objectives as the rest of PanoAffordanceNet. These are pixel-wise Binary Cross-Entropy loss $A_{\text{init}}$3, distributional KL divergence $A_{\text{init}}$4, and Region–Text Contrastive loss $A_{\text{init}}$5. The KL term is described as enforcing shape continuity at the heatmap level, while the contrastive term aligns densified regions semantically to language and reduces drift among affordances (Zhu et al., 10 Mar 2026).

The region-text formulation uses a region feature

$A_{\text{init}}$6

and an InfoNCE objective over similarities between $A_{\text{init}}$7 and text embeddings $A_{\text{init}}$8. The total objective is a weighted combination of $A_{\text{init}}$9, $A_{\text{refined}}$0, and $A_{\text{refined}}$1.

From a systems perspective, the dominant cost is affinity construction. The global similarity matrix requires $A_{\text{refined}}$2 time and memory per image. At $A_{\text{refined}}$3, the affinity matrix contains approximately 10 million entries per image. By contrast, seed selection and max propagation scale as $A_{\text{refined}}$4, and the paper notes that with small $A_{\text{refined}}$5, for example $A_{\text{refined}}$6–$A_{\text{refined}}$7, this stage is light relative to building $A_{\text{refined}}$8.

The implementation is described as using standard tensor operations on PyTorch: feature L2 normalization, batched cosine similarity, top-$A_{\text{refined}}$9 selection, sigmoid gating, and elementwise max. No custom CUDA kernels are described. Inference speed is dominated by forming $A_{\text{init}}$0, while OSDH adds negligible parameters and minor compute beyond that matrix construction.

The training setup reported for the overall framework is AdamW, initial learning rate $A_{\text{init}}$1, cosine annealing, batch size $A_{\text{init}}$2, $A_{\text{init}}$3 iterations, and inputs resized to $A_{\text{init}}$4. Augmentations include random flips, color jitter, $A_{\text{init}}$5 rotations, $A_{\text{init}}$6 scaling, and horizontal wraparound shifts tailored to 360° topology. Gaussian blur is applied to binary masks to form soft supervision heatmaps.

6. Empirical behavior, limitations, and usage guidance

On the 360-AGD Hard Split, module ablation shows measurable gains from OSDH. The reported values are: LoRA only, KLD $A_{\text{init}}$7, SIM $A_{\text{init}}$8, NSS $A_{\text{init}}$9; LoRA + OSDH, KLD $F_v''$0, SIM $F_v''$1, NSS $F_v''$2; Full (LoRA + DASM + OSDH), KLD $F_v''$3, SIM $F_v''$4, NSS $F_v''$5. The paper states that OSDH yields clear gains in SIM $F_v''$6 and NSS $F_v''$7 over LoRA-only, indicating improved spatial integrity and stronger responses at annotated keypoints. In the full system, OSDH contributes to the best overall performance (Zhu et al., 10 Mar 2026).

Hyperparameter analysis further reports robustness to seed count. Across $F_v''$8, KLD varies narrowly from $F_v''$9 to $F_v'' \in \mathbb{R}^{B \times L \times D}$0, SIM remains approximately $F_v'' \in \mathbb{R}^{B \times L \times D}$1–$F_v'' \in \mathbb{R}^{B \times L \times D}$2, and NSS remains approximately $F_v'' \in \mathbb{R}^{B \times L \times D}$3–$F_v'' \in \mathbb{R}^{B \times L \times D}$4. The paper recommends top-$F_v'' \in \mathbb{R}^{B \times L \times D}$5 in the range $F_v'' \in \mathbb{R}^{B \times L \times D}$6–$F_v'' \in \mathbb{R}^{B \times L \times D}$7, with $F_v'' \in \mathbb{R}^{B \times L \times D}$8 as a good default, and suggests starting with $F_v'' \in \mathbb{R}^{B \times L \times D}$9 for confidence normalization and tuning lightly.

Qualitative results are described as showing cleaner, continuous heatmaps with reduced semantic drift, particularly in cluttered, multi-affordance scenes. This suggests that the densification mechanism is not merely smoothing activations but selectively extending them along semantically coherent affinity paths.

The stated limitations are also specific. First, the $$A_{\text{init}} \in \mathbb{R}^{B \times C \times L}$$0 affinity matrix can be memory-intensive at very high resolutions or with smaller patches. Second, when visual self-similarity is weak, such as under highly ambiguous textures, affinity-based propagation may underperform, either retaining sparsity or propagating to semantically similar but incorrect regions. Third, the method relies on DASM to stabilize features across latitudes; without robust upstream calibration, affinities may be noisier near the poles.

For integration into panoramic or spherical pipelines, the paper’s guidance is to use a robust backbone, such as a ViT with geometry-aware modulation like DASM, compute global cosine affinities on normalized tokens, and densify class-wise maps via top-$$A_{\text{init}} \in \mathbb{R}^{B \times C \times L}$$1 seeds and confidence-gated max propagation. It further recommends maintaining ERP resolution consistency, including wraparound augmentation to encourage seam invariance, and keeping patch size large enough to control $$A_{\text{init}} \in \mathbb{R}^{B \times C \times L}$$2 for feasible $$A_{\text{init}} \in \mathbb{R}^{B \times C \times L}$$3 affinity.

7. Position within panoramic affordance grounding

OSDH is presented in the “Spherical-Aware Hierarchical Decoder” section of the paper, with its architecture illustrated by spherical projection of tokens, cosine affinity matrix construction, top-$$A_{\text{init}} \in \mathbb{R}^{B \times C \times L}$$4 seed selection, confidence gating, and max propagation with scalar $$A_{\text{init}} \in \mathbb{R}^{B \times C \times L}$$5. Within the broader PanoAffordanceNet design, it is paired with DASM and trained under pixel-wise, distributional, and region-text contrastive constraints to suppress semantic drift under low supervision (Zhu et al., 10 Mar 2026).

Relative to spherical or geometry-aware convolutions and to deformable sampling, OSDH is characterized by three properties explicitly stated in the paper: it does not require explicit spherical kernels or coordinate transforms, it leverages global token affinities to recover continuity across seams, poles, and dispersed regions, and it avoids learned offsets. Continuity emerges from global self-similarity and confidence-gated propagation.

In this sense, OSDH occupies a specific methodological niche within 360° scene-level perception. It is not a general-purpose geometric operator for all panoramic vision tasks, but a lightweight densification head tailored to the topology and sparsity issues of panoramic affordance grounding. The paper’s summary describes it as turning sparse, fragmented affordance activations into continuous, geometrically coherent regions in 360° ERP imagery through global cosine affinities over unit-normalized features, confidence-gated top-$$A_{\text{init}} \in \mathbb{R}^{B \times C \times L}$$6 seed propagation, and a residual update.