Geometry-Aware Attention Guidance

• Geometry Aware Attention Guidance is a principle that integrates explicit geometric information into attention computations, aligning features with spatial and structural cues.
• It employs diverse formulations—ranging from additive biasing in logits to query-key concatenation and residual corrections—to reduce ambiguity in feature aggregation.
• Empirical results in tasks like CT reconstruction and stereo inpainting demonstrate improved metrics such as PSNR and SSIM by embedding geometry into the attention process.

Searching arXiv for papers and closely related formulations of “Geometry Aware Attention Guidance”. Geometry Aware Attention Guidance (GAG) denotes a family of mechanisms that use explicit geometric information to steer attention, cross-attention, or attention-like guidance so that feature aggregation follows the spatial, projective, or structural organization of the underlying problem rather than treating tokens as geometry-free elements. In the current literature, the term is not standardized. Some works use the name directly for a specific diffusion-time guidance rule in attention space (Kim, 3 Mar 2026), while others realize the same principle under different names, including Geometry-Aware Attention, Attention-Guided Projection, Geometric Transform Attention, Geometry-Enhanced Multi-scale Attention, and masked geometry-aware joint attention (Li et al., 2022, Miyato et al., 2023, Tang et al., 12 Jul 2025, Lin et al., 3 Oct 2025, Kim et al., 18 Jun 2026). Across these formulations, the common objective is to reduce ambiguity, suppress geometrically implausible interactions, and preserve structure in settings where appearance-only attention is insufficient.

1. Terminology and conceptual scope

The label “Geometry Aware Attention Guidance” covers several related but non-identical constructions. In some papers the acronym is explicit; in others it is only a useful conceptual mapping. The literature therefore presents GAG less as a single algorithm than as a recurring design principle: geometry is encoded, projected, or inferred, and then used to bias the way attention weights are formed or how attended values are aggregated.

Formulation	Paper	Realization of geometry guidance
GAG as a named method	(Kim, 3 Mar 2026)	Decomposes sparse-vs-dense attention residuals into parallel and orthogonal components
“GAA + ICG” as GAG mapping	(Li et al., 2022)	Epipolar geometry-aware cross-view attention plus iterative cross guidance
Geometry-aware lifting in CT reconstruction	(Kim et al., 18 Jun 2026)	Differentiable back-projection and AGP-3D
Geometric Transform Attention	(Miyato et al., 2023)	Relative transformations applied directly to $Q$, $K$, and $V$
Geometry-conditioned diffusion completion	(Lin et al., 3 Oct 2025)	Projected point-cloud tokens with masked joint attention

This diversity is substantive rather than terminological. In stereo inpainting, geometry guidance is defined by rectified epipolar structure and disparity-wise correspondence search (Li et al., 2022). In pediatric skull CT reconstruction, it is tied to acquisition matrices, differentiable back-projection, and voxel-wise 2D-to-3D attention (Kim et al., 18 Jun 2026). In multi-view transformers, geometry guidance means expressing token relations through relative elements of $SE(3)\times SO(2)\times SO(2)$ rather than through additive positional codes (Miyato et al., 2023). In diffusion guidance, geometry awareness refers to preserving the component of the attention-space extrapolation aligned with the retrieval direction while suppressing orthogonal components (Kim, 3 Mar 2026).

A recurrent misconception is that all geometry-aware attention methods impose hard geometric constraints. The surveyed systems do not support that view. Several methods use soft biases, gates, masks, or coordinate-frame transforms rather than exact constraint satisfaction. This suggests that, in practice, GAG is often an interpolation between geometric prior and learned correspondence rather than a replacement for learned attention.

2. Geometric signals used to guide attention

The geometry supplied to attention varies sharply by domain, but the dominant pattern is explicit encoding of spatial relations that would otherwise have to be inferred implicitly from features.

In reconstruction from sparse biplanar radiographs, PSCT-Net begins with a differentiable back-projection operator

$\mathrm{BP}(I_{\text{in}, M) := \begin{cases} \dfrac{I_{\text{in}}}{|L| \cdot \Delta p}, & \text{if } M y \in L,\ 0, & \text{otherwise}, \end{cases}$

where $M$ is the projection matrix encoding the view geometry, $L$ is the set of sampled points along the X-ray beam path, and $y$ is a voxel coordinate. This establishes a volumetric prior aligned with acquisition geometry before AGP-3D learns voxel-wise correspondences (Kim et al., 18 Jun 2026). The paper explicitly states that geometry-agnostic lifting spreads 2D features uniformly across depth and degrades osseous boundaries; the back-projected prior is introduced to mitigate that ambiguity.

In rectified stereo inpainting, geometry guidance is based on the epipolar constraint and its horizontal-disparity simplification. IGGNet assumes rectified stereo pairs, constructs a 4D cost volume by shifting the reference feature map across discrete disparities, and computes attention over the disparity dimension after 3D convolutions and a Softmax. Correspondences therefore remain confined to the same image row, and the guidance is explicitly geometric rather than globally content-based (Li et al., 2022).

In point-cloud analysis, the geometric signal is local and differential. GAM uses relative vectors $v_{ij}=p_j-p_i$, their norm $d_{ij}$, and the zenith and azimuth angles

$K$0

to form an angle-based gradient proxy

$K$1

This replaces distance-only aggregation with geometry-sensitive gating, which the paper reports as more robust to abnormal points and sharper at boundaries (Hu et al., 2023).

In PDE solvers for Helmholtz problems, geometry is encoded through a signed distance vector $K$2, a masking matrix $K$3 that suppresses out-of-domain interactions, and trunk features of boundary proximity $K$4. Attention is thus domain-aware at the level of admissible token interaction, not merely spatially aware in the Euclidean sense (Versano et al., 2024). In multi-view video generation, Phys4View predicts per-view depth, lifts pixels to 3D, aligns them to a common world frame, and derives a geometry prior from cross-view 3D distances and a depth-confidence modulation term (Wang et al., 19 Mar 2026). RaysUp similarly replaces image-grid positional cues with a ray-domain encoding derived from a 6D ray descriptor $K$5 and rotary modulation over log-spaced frequencies (Ding et al., 22 Jun 2026).

A further class of systems uses projected scene geometry as an auxiliary token stream. GeoComplete projects a multi-view point cloud into the target frame, rasterizes it into a point map, and uses that map as a geometry branch in a dual-branch diffusion U-Net (Lin et al., 3 Oct 2025). Geo-RepNet uses raw depth maps to derive geometry priors, including relative positional encodings and a geometry bias mask $K$6, which later enter a geometry-enhanced attention block (Tang et al., 12 Jul 2025).

3. Attention parameterizations

The central technical question in GAG is not whether geometry is available, but where it enters the attention computation. The literature exhibits four main insertion points: query-key construction, additive logit biasing, value-space transformation, and residual guidance.

One widely used formulation injects geometry as an additive term in the attention logits. In Phys4View, the geometry-enhanced cross-view attention uses

$K$7

where $K$8, with $K$9 derived from world-space 3D distances and $V$0 from depth-confidence modulation (Wang et al., 19 Mar 2026). Geo-RepNet uses an analogous pattern, with

$V$1

where $V$2 is the geometry bias from DGPG (Tang et al., 12 Jul 2025). Earlier image-captioning work already implemented a related mechanism, adding a learnable relative-geometry term $V$3 to the logits over object regions (Guo et al., 2020).

A second formulation places geometry directly in query-key construction. GAT concatenates appearance and geometry embeddings inside the query and key,

$V$4

so that the dot product itself becomes geometry-sensitive (Wang et al., 2021). PSCT-Net’s AGP-3D uses a different but related approach: 3D grid locations define learnable queries, 2D features become keys and values, and the lifted output is

$V$5

Here, the query geometry is tied to the target voxel grid, while additional geometry enters through repeated back-projection conditioning streams (Kim et al., 18 Jun 2026).

A third formulation changes the coordinate frame in which attention is executed. GTA defines a per-token representation $V$6 from camera pose and image position, then computes

$V$7

This means that dot products and value aggregation occur after geometrically aligning tokens into a shared local frame (Miyato et al., 2023). Rather than learning to infer relative camera transformations from positional embeddings, the model receives the transformation structure explicitly.

A fourth formulation treats guidance as a residual correction in attention space. In the named GAG method for diffusion models, dense attention $V$8 and sparse attention $V$9 are combined through a residual

$SE(3)\times SO(2)\times SO(2)$0

which is decomposed into parallel and orthogonal components relative to $SE(3)\times SO(2)\times SO(2)$1. The filtered update is

$SE(3)\times SO(2)\times SO(2)$2

with

$SE(3)\times SO(2)\times SO(2)$3

The recommended setting is $SE(3)\times SO(2)\times SO(2)$4, so orthogonal residuals are removed and only the component aligned with the retrieval direction is amplified (Kim, 3 Mar 2026). This is geometrically aware in a different sense: “geometry” refers to the internal geometry of the attention update direction, not to external scene geometry.

4. Representative realizations across domains

In inverse problems and reconstruction, GAG addresses severe underdetermination. PSCT-Net reconstructs pediatric skull CT from sparse frontal and lateral X-rays using differentiable back-projection, encoder-side BP-C, decoder-side MV3D-C, AGP-3D lifting, and a BiM-3D bottleneck (Kim et al., 18 Jun 2026). The key role of guidance is to constrain depth early and repeatedly, so that subsequent attention does not propagate 2D evidence uniformly along rays. In Helmholtz solvers, geometry-aware masked self-attention provides a learned preconditioner that respects arbitrary domains, including cracks, holes, and obstacles, by suppressing out-of-domain interactions at the attention level (Versano et al., 2024). GeoTransolver adopts a related philosophy on irregular CAE domains: GALE cross-attends every slice to a shared multi-scale geometry, global, and boundary-condition context, with a learned gate blending self-attention and context attention in each block (Adams et al., 23 Dec 2025).

In correspondence and view-transfer problems, geometry guidance is organized around epipolar, pose, or ray structure. IGGNet uses disparity-wise geometry-aware attention across epipolar lines and alternates target and reference roles through Iterative Cross Guidance, progressively shrinking unreliable masks (Li et al., 2022). GTA aligns token interactions through relative transformations in $SE(3)\times SO(2)\times SO(2)$5 and replaces additive positional encodings with explicit feature transforms (Miyato et al., 2023). RaysUp uses a Spatially Decoupled Guidance Encoder, Any-Resolution Cross-Attention, Ray Positional Encoding, and geometry-aware neighborhood attention so that feature upsampling obeys viewing-ray consistency instead of 2D interpolation heuristics (Ding et al., 22 Jun 2026).

In generative modeling, GAG frequently appears as a geometry-conditioned auxiliary branch. GeoComplete conditions a dual-branch diffusion U-Net on both masked image latents and projected point-map latents, then uses a sparse joint attention mask that allows an image token to access only the geometry token at the same spatial index across branches (Lin et al., 3 Oct 2025). Harmonizing Attention modifies self-attention layers in latent diffusion so that keys and values can be concatenated across geometry, source, and target branches, thereby aligning geometry during inversion and preserving it during generation without retraining (Ikuta et al., 2024). PhysVideo implements geometry-enhanced cross-view attention in Phys4View and then hands the generated foreground motion to VideoSyn as flow-guided conditioning for full-video synthesis (Wang et al., 19 Mar 2026).

Other domains instantiate GAG at different scales. GAM inserts geometry-aware gating into local neighborhood aggregation for point clouds (Hu et al., 2023). Geo-RepNet injects depth-derived geometry priors into a RepVGG-based classifier for surgical phase recognition (Tang et al., 12 Jul 2025). Geometry-aware captioning models integrate object-box geometry into self-attention or cross-attention, either as pairwise biases (Guo et al., 2020) or as concatenated geometry embeddings plus GLU refinement (Wang et al., 2021). These systems differ in architecture, but all use geometry to alter attention pathways that would otherwise depend only on appearance or token order.

5. Empirical profile

The empirical record reported in the cited papers is heterogeneous but consistently framed around gains in structural fidelity, correspondence quality, or convergence behavior.

Paper and domain	Reported result	Citation
PSCT-Net, pediatric skull CT	PedSkull-CT: PSNR 31.49, PSNR3D 28.03, SSIM 0.882, LPIPS 0.100	(Kim et al., 18 Jun 2026)
IGGNet, stereo inpainting	KITTI2015, mask 20–40%: PSNR 24.20, SSIM 0.881, FID 8.07	(Li et al., 2022)
GAM, point clouds	S3DIS: mIoU/OA/mAcc of 74.4%/90.6%/83.2%	(Hu et al., 2023)
GTA, novel view synthesis	CLEVR-TR PSNR 39.63; MSN-Hard with SRT+GTA 25.72 / 0.289 / 0.798	(Miyato et al., 2023)
RaysUp, universal upsampling	0.14M params; 55 FPS vs 11 FPS for AnyUp at 224×224	(Ding et al., 22 Jun 2026)
GeoComplete, reference-driven completion	RealBench: PSNR 17.32 vs 14.78, SSIM 0.578 vs 0.424, LPIPS 0.197 vs 0.431	(Lin et al., 3 Oct 2025)
GAG for diffusion guidance	SDXL+CFG+GAG: GE 0.605, CS 27.25, IR 0.811, PS 22.39, HPSv2 0.271	(Kim, 3 Mar 2026)

Ablations reinforce the view that geometry and attention are complementary rather than interchangeable. In PSCT-Net, the baseline reports PSNR=26.03 and SSIM=0.645; adding BP-I, BP-C, AGP, and BiM culminates in 27.18 and 0.671, while BiM alone yields the largest single-module gain and AGP alone lifts PSNR to 26.92 (Kim et al., 18 Jun 2026). IGGNet reports that simple concatenation, disparity argmax, and full GAA produce 25.17/0.863/10.73, 25.68/0.875/9.04, and 26.42/0.897/7.86 on KITTI2015 with mask 0–60% and $SE(3)\times SO(2)\times SO(2)$6, respectively, indicating that geometry-aware attention is stronger than fusion without explicit epipolar aggregation (Li et al., 2022). In GAM, both angle-derived gradient information and Euclidean distance help, but their combination performs best in classification and segmentation ablations (Hu et al., 2023).

The diffusion-guidance variant provides a particularly explicit decomposition of what kind of guidance is useful. On MS-COCO with SDXL+CFG, the full residual gives CS 26.54, IR 0.732, and PS 22.10, orthogonal-only guidance drops to CS 25.25, IR 0.415, and PS 21.79, while parallel-only guidance reaches CS 27.25, IR 0.811, and PS 22.39 (Kim, 3 Mar 2026). This supports the paper’s claim that orthogonal residuals act as destabilizing components and that geometry-aware filtering should preserve only the aligned part of the extrapolation.

6. Limitations, misconceptions, and open directions

The first limitation is conceptual: GAG is not a single canonical method. The same label can refer to epipolar attention in stereo, ray-domain positional encoding in upsampling, signed-distance masking in PDE solvers, or parallel-residual filtering in diffusion attention space. Treating these as equivalent algorithms would be misleading. A more accurate statement is that they instantiate a shared principle: geometry is used to regulate token interaction.

The second limitation is that many papers explicitly leave parts of the geometry model unspecified. PSCT-Net states that it does not provide an explicit pinhole camera model, weighting schemes, gradient expressions for back-projection, attention $SE(3)\times SO(2)\times SO(2)$7 formulas, Bi-SSM internals, or detailed training schedules (Kim et al., 18 Jun 2026). PhysVideo does not impose an explicit epipolar loss, even though the relevant matrices are derivable from camera parameters (Wang et al., 19 Mar 2026). GeoComplete does not add an explicit geometric consistency loss; the consistency is meant to emerge from target-aware masking and aligned joint attention (Lin et al., 3 Oct 2025). This suggests that many current GAG systems remain hybrid: explicit geometry enters, but some of the most important constraints are still left to learned inference.

The third limitation is sensitivity to the quality of the geometric signal. IGGNet assumes rectified stereo, and significant mis-rectification breaks the horizontal-epipolar assumption (Li et al., 2022). GAM notes instability when $SE(3)\times SO(2)\times SO(2)$8 is very small or neighbors are collinear (Hu et al., 2023). PhysVideo states that inaccurate camera parameters or depth errors near occlusions weaken the geometry prior (Wang et al., 19 Mar 2026). RaysUp notes that non-pinhole cameras or severe lens distortions can misalign the ray model (Ding et al., 22 Jun 2026). GeoTransolver reports sensitivity to geometry-token coverage, neighborhood settings, and the balance between query and geometry tokens (Adams et al., 23 Dec 2025).

A related misconception is that geometry-aware attention necessarily dominates appearance-based modeling. The surveyed evidence argues for a more qualified view. In smooth Helmholtz domains, vanilla or non-masked variants can match or outperform masked geometry-aware attention (Versano et al., 2024). In Harmonizing Attention, using only source or only target keys and values is weaker than combining them, which indicates that geometry transfer and texture continuity must be co-optimized rather than separated completely (Ikuta et al., 2024). In Geo-RepNet, DGPG, geometry-aware attention logits, and multi-scale EMA all contribute complementary gains; none of the components alone reproduces the full model’s 85.02% AC, 81.74% F1, and 93.10% AUC (Tang et al., 12 Jul 2025).

The main open directions named in the literature are increasingly fine-grained and domain-specific. PSCT-Net identifies patch-based refinement, implicit neural representations such as NeRF, explicit 3D Gaussian primitives, and reader studies on craniosynostosis as future work (Kim et al., 18 Jun 2026). PhysVideo suggests adding epipolar proximity bias or loss to complement its depth-based prior (Wang et al., 19 Mar 2026). GeoTransolver points toward physics-constrained training and coupled multi-physics (Adams et al., 23 Dec 2025). Geo-RepNet suggests temporal consistency and cross-frame geometry aggregation (Tang et al., 12 Jul 2025). The diffusion-guidance formulation suggests adaptive schedules for $SE(3)\times SO(2)\times SO(2)$9, $\mathrm{BP}(I_{\text{in}, M) := \begin{cases} \dfrac{I_{\text{in}}}{|L| \cdot \Delta p}, & \text{if } M y \in L,\ 0, & \text{otherwise}, \end{cases}$0, $\mathrm{BP}(I_{\text{in}, M) := \begin{cases} \dfrac{I_{\text{in}}}{|L| \cdot \Delta p}, & \text{if } M y \in L,\ 0, & \text{otherwise}, \end{cases}$1, and $\mathrm{BP}(I_{\text{in}, M) := \begin{cases} \dfrac{I_{\text{in}}}{|L| \cdot \Delta p}, & \text{if } M y \in L,\ 0, & \text{otherwise}, \end{cases}$2, as well as true Anderson-style memory beyond the $\mathrm{BP}(I_{\text{in}, M) := \begin{cases} \dfrac{I_{\text{in}}}{|L| \cdot \Delta p}, & \text{if } M y \in L,\ 0, & \text{otherwise}, \end{cases}$3 case (Kim, 3 Mar 2026).

Taken together, these works indicate that GAG is best understood as a research program rather than a fixed recipe. Its defining move is to relocate geometry from a peripheral preprocessing artifact into the internal mechanics of attention, where it can bias logits, transform coordinates, gate aggregation, or stabilize guidance updates. The specific form of that relocation remains task-dependent, and the current literature shows that its effectiveness depends less on the slogan of “geometry awareness” than on the exact way geometry is encoded, aligned, and coupled to attention.