SphereViT: Spherical Attention for Panoramic Vision

• SphereViT is a module that integrates spherical geometry into transformer models to accurately represent 360° panoramic images.
• It replaces conventional 2D positional encodings with fixed spherical embeddings using a cross-attention mechanism to directly inject geometric priors.
• Integrated within the DA² framework, SphereViT improves zero-shot panoramic depth estimation performance by 38% in AbsRel while streamlining computational efficiency.

SphereViT is a module designed for panoramic vision tasks that explicitly incorporates spherical geometry into transformer-based modeling of 360° panoramic images. It forms a key architectural component within the DA² ("Depth Anything in Any Direction") framework for end-to-end panoramic depth estimation. SphereViT directly addresses the inadequacy of conventional 2D positional encoding in standard Vision Transformers (ViTs) for equirectangular panoramas, which are characterized by nonuniform distortions introduced in the mapping from the spherical domain to the 2D image plane.

1. Motivation and Challenges in Panoramic Representation

Panoramic images, especially those with a 360°×180° field of view, provide comprehensive scene coverage but exhibit severe geometric distortions, particularly near the poles, when represented in standard equirectangular projection. Traditional ViTs, which rely on 2D pixel-based positional embeddings, fail to account for the underlying spherical geometry, resulting in representations that do not reflect the true spatial relationships of points on the sphere. This limitation is further exacerbated by the scarcity of panoramic data, which has historically hindered the generalization capacity of models in zero-shot settings. Approaches that attempt to circumvent distortions through perspective splitting (e.g., cubemap transforms and fusion) introduce inefficiencies and architectural complexity.

SphereViT is proposed to resolve these issues by encoding explicit spherical coordinates into the transformer framework and allowing features to attend directly to this geometric prior via a dedicated attention mechanism.

2. Spherical Positional Encoding Construction

SphereViT replaces standard 2D positional embeddings with embeddings that represent the intrinsic spherical position of each patch. For a panoramic image of dimensions (W, H), every pixel (u, v) is associated with a pair of spherical angles:

• Azimuth (longitude): $\varphi = 2\pi \cdot \frac{u}{W}$
• Polar (colatitude/latitude): $\theta = \pi \cdot \frac{v}{H}$

This deterministic mapping ensures that each patch or pixel is assigned its true geometric location on the unit sphere.

To encode these parameters for transformer input, a two-channel angle field $A \in \mathbb{R}^{H\times W \times 2}$ is constructed, with each element storing ($\varphi$, $\theta$). This field is averaged or resized according to the patch grid (patch size $P$), resulting in $A' \in \mathbb{R}^{(H'\times W') \times 2}$ where $H' = H/P$, $W' = W/P$.

For each spatial unit in $A'$, a high-dimensional embedding is computed over a set of frequency coefficients:

• Frequency exponent for dimension $n$ ($n = 1, ..., D'$; $D'$ typically set as $D/4$ for embedding dimension $D$):

$d_n = \frac{(n-1) \cdot \log_2(H')}{D'}$

• The spherical embedding $E_{\rm sphere}(i,j)$ for patch $(i,j)$ is then:

$$E_{\rm sphere}(i,j) = [\sin(2^{d_1} \varphi_i), \cos(2^{d_1} \varphi_i), ..., \sin(2^{d_{D'}} \theta_j), \cos(2^{d_{D'}} \theta_j)]$$

By stacking these sine-cosine functions over both angles and multiple frequencies, the resulting embedding provides a rich, distortion-aware geometric prior for each spatial location.

3. Cross-Attention Mechanism for Geometric Injection

SphereViT employs a cross-attention module to inject spherical awareness into image features, departing from the conventional practice of summing positional embeddings with image features before self-attention. Specifically, the image features $Z \in \mathbb{R}^{(H' \times W') \times D}$, extracted by a standard ViT backbone, serve as queries. The fixed spherical embedding $E_{\rm sphere}$ furnishes both keys and values.

The cross-attention computation is as follows:

$$\text{CrossAttn}(Z, E_{\rm sphere}) = \text{SoftMax}\left( \frac{Z W_Q (E_{\rm sphere} W_K)^T}{\sqrt{D_k}} \right) \cdot (E_{\rm sphere} W_V)$$

where $W_Q, W_K, W_V \in \mathbb{R}^{D \times D_k}$ are trainable projection matrices, and $\sqrt{D_k}$ is the standard transformer scaling factor.

Because the full panoramic field of view is consistent across images, $E_{\rm sphere}$ is fixed and not updated during training, acting as a constant geometric prior throughout the network.

4. Comparison with Previous and Alternative Approaches

Conventional panoramic processing methods, including perspective splitting and cubemap fusion, attempt to mitigate distortion by dividing the panorama into multiple views for separate processing, followed by feature fusion. Such strategies incur additional computational and design overhead, often yielding suboptimal efficiency. Flat 2D positional encodings, commonly employed in existing transformers, cannot account for the stretching and compression inherent in the equirectangular projection, particularly towards the poles.

SphereViT obviates the need for these multi-projection or fusion-based techniques by enabling the model to process the entire panorama in one pass, attending directly to the true geometric structure and thus maintaining both efficiency and fidelity.

5. Impact on Panoramic Depth Estimation Performance

Within the DA² framework, SphereViT yields tangible improvements in several aspects:

• Spherical geometric consistency: By embedding explicit spherical structure, the network is distortion-aware, permitting accurate depth estimation even under severe projection-induced nonuniformities, which is critical for high-fidelity 3D scene reconstruction.
• Zero-shot generalization: Direct encoding of geometric relationships enables the model to generalize robustly to previously unseen panoramic images—empirically, DA² with SphereViT demonstrates an average 38% improvement in AbsRel over the strongest zero-shot baseline and outperforms prior in-domain methods, signifying substantially improved out-of-distribution robustness.
• Computational and architectural efficiency: As an end-to-end system, DA² with SphereViT avoids the need for multi-view splitting or post-processing, facilitating a streamlined training and inference workflow.

6. Architectural Innovations and Broader Relevance

SphereViT introduces a distinct architectural paradigm by supplanting standard transformer self-attention with a cross-attention mechanism in which image feature queries attend to a fixed, spherical positional prior. This grants the network intrinsic awareness of manifold geometry without requiring dynamic positional embedding updates or complex post hoc fusion procedures. The spherical embedding remains constant across all full-FoV panoramas, simplifying design and training.

A plausible implication is that this approach may generalize to other vision tasks on non-Euclidean manifolds where domain geometry is critical for accurate modeling. By integrating geometric priors directly into attention architectures, SphereViT exemplifies a method of enhancing transformer adaptability to domains beyond standard perspective imagery.