GASAM: Geometry-Aware Attention

• GASAM is a technique that incorporates 3D geometric priors into attention computations, addressing limitations of classical self-attention in spatial tasks.
• It selectively modulates attention scores using biasing and gating mechanisms based on spatial relevance and learned geometric features.
• This approach improves geometric consistency and task performance in applications like multi-view synthesis, 3D scene editing, and robotic manipulation.

Geometry-Aware Selective Attention Modulation (GASAM) refers to a class of techniques in deep neural architectures—primarily attention-based models—that introduce explicit 3D geometric structure or priors into the computation of attention maps. Unlike classical self-attention, which is agnostic to spatial or geometric relationships among tokens, GASAM modulates attention distributions so that geometric constraints or relevance guide where and how information is integrated. This approach improves geometric consistency, spatial reasoning, and task fidelity in domains such as multi-view synthesis, 3D scene editing, robotic manipulation, and spatially grounded language modeling (Wen et al., 7 Jul 2025, Tian et al., 29 Apr 2026, Li et al., 5 Feb 2026, Miyato et al., 2023).

1. Motivation and Foundational Principles

Geometry is a critical inductive bias in spatially structured data. Traditional attention mechanisms ignore 3D spatial relationships, resulting in ambiguous correspondences, redundancy, or semantic-geometry misalignment in tasks that require precise spatial reasoning. GASAM directly addresses these limitations by:

• Explicitly incorporating geometric priors or metrics—such as 3D positions, depths, camera extrinsics, or group-theoretic transforms—into attention computation.
• Enabling selective modulation—scaling, gating, or biasing of attention scores or value vectors—conditioned on geometric relevance.
• Targeting task-specific geometric dependencies—e.g., attending more strongly to tokens spatially close to a given query in 3D, or relevant for a downstream control or reasoning objective.

Early work in geometry-aware attention used hand-crafted positional encodings or projected spatial relationships (Miyato et al., 2023); more recent GASAM approaches directly learn geometric gating or bias, often with auxiliary prediction networks (“geometry experts”) and group-consistent transformations (Wen et al., 7 Jul 2025, Tian et al., 29 Apr 2026, Li et al., 5 Feb 2026).

2. Mathematical Formulation and Mechanisms

GASAM generalizes standard attention by supplementing or modulating the per-token (or per-pair) attention computation with geometric information.

Standard Attention

Given queries $\mathbf{Q}\in\mathbb{R}^{n\times d}$, keys $\mathbf{K}\in\mathbb{R}^{n\times d}$, values $\mathbf{V}\in\mathbb{R}^{n\times d}$, standard dot-product attention is

$$\operatorname{Attention}(\mathbf{Q}, \mathbf{K}, \mathbf{V}) = \operatorname{softmax}\left(\frac{\mathbf{Q} \mathbf{K}^\top}{\sqrt{d}}\right) \mathbf{V}.$$

Geometry-Aware Modulation (Illustrative Variants)

• Score biasing via geometric weights: Add an explicit bias based on 3D distance or semantic-geometry affinity:

$$\operatorname{Attn}_{ij} = \operatorname{softmax}\left(\frac{\mathbf{Q}_i^\top \mathbf{K}_j}{\sqrt{d}} + \lambda \log(w_{ij}+\epsilon)\right),$$

where $w_{ij}$ encodes the geometric proximity or relevance between tokens $i$ and $j$ (Tian et al., 29 Apr 2026, Li et al., 5 Feb 2026).

• Transforming token representations: Apply group-theoretic or coordinate frame transforms to $K$, $V$ based on the relative geometry between tokens:

$\mathbf{K}\in\mathbb{R}^{n\times d}$0

before aggregating and mapping the output into the query's frame (Miyato et al., 2023).

• Frame-strict or region-strict cross-attention: Compute attention only within spatially or semantically aligned groups or restrict cross-attention to frame-paired tokens (Li et al., 5 Feb 2026).
• Learned importance gating: Apply a learned, query-conditioned gate:

$\mathbf{K}\in\mathbb{R}^{n\times d}$1

and use $\mathbf{K}\in\mathbb{R}^{n\times d}$2 as an additive attention logit (Li et al., 5 Feb 2026).

Across implementations, the principle is to ensure that attention is “selectively” guided by geometry—emphasizing spatially or semantically relevant relationships and de-emphasizing irrelevant or misaligned ones.

3. Architectures and Integration Strategies

GASAM instantiations vary across domains and architectures, including:

• Multimodal Transformers for Robotic Control: In STARRY, a dedicated Geometry Expert predicts future scene depth and end-effector 3D positions. Per-token attention weights are computed as a function $\mathbf{K}\in\mathbb{R}^{n\times d}$3 and injected into the cross-attention from action to video tokens as a log-bias (Tian et al., 29 Apr 2026).
• Spatial Reasoning in MLLMs: GeoThinker interleaves Spatial-Grounded Fusion (SGF) layers within a multimodal LLM. At each selected transformer layer, semantic image tokens (SH_img) selectively query frame-aligned geometry tokens (ST_G) via cross-attention, further modulated by an importance-gated logit addition based on learned relevance (Li et al., 5 Feb 2026).
• 3D Scene Editing with 3D Gaussian Splatting: InterGSEdit constructs a 3D Geometry-Consistent Attention Prior (GAP³ᴰ) by unprojecting 2D attention maps across reference views into the 3D Gaussian Splatting domain, weighted for semantic consistency. During diffusion, an Attention Fusion Network dynamically blends 3D and 2D attention by a schedule $\mathbf{K}\in\mathbb{R}^{n\times d}$4, prioritizing geometric consistency early and high-frequency details late (Wen et al., 7 Jul 2025).
• Geometry-Transformed Attention in Multi-View Transformers: GTA extends attention by transforming $\mathbf{K}\in\mathbb{R}^{n\times d}$5 and $\mathbf{K}\in\mathbb{R}^{n\times d}$6 for each key $\mathbf{K}\in\mathbb{R}^{n\times d}$7 to the query’s coordinate frame using a block-diagonal $\mathbf{K}\in\mathbb{R}^{n\times d}$8, supporting geometric equivariance and the ability to modulate attention by learned gates or analytic functions applied to relative geometric features (Miyato et al., 2023).

4. Typical Pipelines and Implementation Details

A generalized GASAM-enabled model involves:

1. Geometry Acquisition: Obtain 3D information (depth maps, camera extrinsics, 3D keypoints, or group-theoretic features) per token or frame from sensor data or geometry prediction modules.
2. Geometric Weight Computation: For each query-key pair, compute a geometric relevance score—either analytically (e.g., distance functions, transformation norms) or via a learned network.
3. Attention Modulation: Modify attention maps:
   • By biasing pre-softmax logits with geometric scores.
   • By gating values or soft-attention outputs post-aggregation.
   • By preprocessing tokens via transformations aligning local frames (Miyato et al., 2023).
4. Selective Application: Restrict GASAM layers to a subset of the transformer’s depth to preserve non-geometric reasoning capacity (e.g., fusion ratio $\mathbf{K}\in\mathbb{R}^{n\times d}$9 in GeoThinker (Li et al., 5 Feb 2026)) or target only action-conditional branches (Tian et al., 29 Apr 2026).
5. Training and Hyperparameters: Geometry experts and gating functions are trained either independently (with metric losses for depth/pose prediction) or end-to-end within the global objective. Key hyperparameters control modulation strength, numerical stability offsets, and fusion schedules (e.g., $\mathbf{V}\in\mathbb{R}^{n\times d}$0, $\mathbf{V}\in\mathbb{R}^{n\times d}$1, $\mathbf{V}\in\mathbb{R}^{n\times d}$2) (Tian et al., 29 Apr 2026, Li et al., 5 Feb 2026, Wen et al., 7 Jul 2025).

5. Empirical Evidence and Impact

Across domains, GASAM-enabled architectures demonstrate state-of-the-art or strongly competitive performance in spatially grounded tasks:

Model/Task	Metric(s)	Gain Attributed to GASAM
STARRY: Robotic Manipulation (RoboTwin)	Success Rate	+10.9% (Act-only), +4.5% (Full ST), +28% real
GeoThinker: Spatial Reasoning (VSI-Bench)	Avg Score	+17–18 points over passive fusion
InterGSEdit: 3DGS Editing (IN2N scenes)	CLIP/CTIDS/CDC	Outperforms all baselines; ↑ consistency

Ablative studies repeatedly show that disabling the geometry-aware modulation degrades performance most substantially on tasks requiring precise geometric alignment or 3D consistency—such as non-rigid 3D editing, manipulation “handover” events, or long-range spatial reasoning (Wen et al., 7 Jul 2025, Tian et al., 29 Apr 2026, Li et al., 5 Feb 2026).

Further, selective attention modulation is most effective when geometric signals are actively fused where needed rather than uniformly mixed, avoiding redundant or misaligned information that can arise in purely passive strategies (Li et al., 5 Feb 2026).

6. Variants and Extensions

• Mode of Modulation: GASAM allows modulation at the attention-score (logit) level (additive bias), value level (multiplicative gate), or both (Miyato et al., 2023, Li et al., 5 Feb 2026).
• Integration Granularity: Some systems realize frame-strict, mask-strict, or region-strict application; others generalize to full cross-view or spatial aggregation.
• Gating Functions: Both fixed analytic mappings (e.g., RBF on distance) and trainable MLP gates (condensed geometric descriptors) are used to compute per-pair modulation (Miyato et al., 2023).
• Temporal Adaptivity: In diffusion frameworks for image/video or 3D editing, blending schedules such as $\mathbf{V}\in\mathbb{R}^{n\times d}$3 allow dynamic prioritization, coupling early geometric consistency with late-stage appearance recovery (Wen et al., 7 Jul 2025).

7. Relation to Prior Art and Future Directions

GASAM generalizes and supersedes classic positional encoding and geometric feature fusion. Its fundamental insight—that spatial priors must condition and bias, not merely accompany, information integration—has motivated advances in:

• Multimodal world-modeling for embodied intelligence (Tian et al., 29 Apr 2026)
• Spatially-aware reasoning in LLMs (Li et al., 5 Feb 2026)
• Geometrically consistent cross-view editing and reconstruction (Wen et al., 7 Jul 2025, Miyato et al., 2023)

A plausible implication is that further generalizations—such as hierarchical or graph-based geometric attention, non-Euclidean priors, or domain-adaptive gating—will become standard for spatial and embodied AI, as spatially passive attention fusion approaches reach their empirical limits.

References:

• "InterGSEdit: Interactive 3D Gaussian Splatting Editing with 3D Geometry-Consistent Attention Prior" (Wen et al., 7 Jul 2025)
• "STARRY: Spatial-Temporal Action-Centric World Modeling for Robotic Manipulation" (Tian et al., 29 Apr 2026)
• "Thinking with Geometry: Active Geometry Integration for Spatial Reasoning" (Li et al., 5 Feb 2026)
• "GTA: A Geometry-Aware Attention Mechanism for Multi-View Transformers" (Miyato et al., 2023)