Vision-to-Geometry Mapping

• Vision-to-Geometry Mapping is a field that transforms visual data into explicit 3D representations, enabling clear scene reconstruction using both classical and learned methods.
• The topic covers explicit approaches using camera calibration and projective geometry as well as implicit models with deep neural networks that infer continuous occupancy or SDFs.
• Applications span autonomous driving, robotics, and navigation, where accurate 3D geometry underpins decision-making, manipulation, and environment mapping.

Vision-to-Geometry Mapping

Vision-to-geometry mapping encompasses the methodologies that transform raw visual input—images, videos, or streams from multi-view sensors—into explicit or implicit 3D geometric representations. This mapping is foundational to computer vision, robotics, autonomous driving, and scene understanding, directly enabling perception-driven decision-making, planning, and control in real-world environments. The field includes a spectrum of approaches from strict geometric estimators to learned implicit geometry models, all aiming to recover, reconstruct, or reason over metric 3D structure from one or more visual observations.

1. Formal Problem Scope and Taxonomy

Vision-to-geometry mapping targets the problem of inferring a scene’s geometric structure, typically represented as metric point clouds, occupancy grids, signed distance functions, or parametric scene graphs, from visual sensor data. This process can be formalized as learning or computing a mapping

$f : \mathcal{V} \rightarrow \mathcal{G}$

where $\mathcal{V}$ is visual input space (image(s), video, or raw pixel arrays) and $\mathcal{G}$ is the geometric representation space (such as $\mathbb{R}^{n \times 3}$ for point clouds, or functions $f(\mathbf{p}) \rightarrow [0,1]$ for occupancy/SDF). Principal subclasses include:

• Explicit geometric mapping: Hand-coded or differentiable geometric algorithms convert calibrated 2D detections or semantic maps into 3D metric geometry via homographies, triangulation, and projection/inverse-projection (e.g., IPM for parking lots (Nandi et al., 9 Mar 2026)).
• Implicit geometric models: Deep neural networks infer continuous functions (e.g., occupancy or SDF) from image inputs, often using transformer-based cross-view or patch-wise fusion, with or without explicit calibration (e.g., ViGT (Shirokov et al., 5 Feb 2026), IVGT (Wu et al., 15 May 2026), DVGT (Zuo et al., 18 Dec 2025)).
• Latent geometry embedding: Models encode visual input into geometry-consistent latent features, typically grounded by dense self-supervised or multi-modal losses (e.g., VGGT as a foundation for robotic manipulation (Song et al., 14 Apr 2026), geometry-aware VLA models (Abouzeid et al., 17 Sep 2025), eVGGT (Vuong et al., 19 Sep 2025)).
• Canonical geometric programming from diagrams: Vision models learn to encode and translate diagrams of geometric constructs into symbolic or programmatic representations, often with ViT-based backbones and vision–LLMs (GeoCLIP/GeoDANO (Cho et al., 17 Feb 2025)).
• Self-supervised and domain-agnostic mapping: Calib-free, LiDAR-pair-supervised, or pose-free models learn geometry entirely from data statistics and weak supervision (ViGT (Shirokov et al., 5 Feb 2026), IVGT (Wu et al., 15 May 2026), DVGT (Zuo et al., 18 Dec 2025)).

2. Classical and Differentiable Geometric Mapping

Traditional pipelines rely on explicit geometric models and camera calibration. These core elements include:

• Camera calibration and homography: Intrinsic and extrinsic matrices ($K, [R|t]$) determine the mapping from world and image coordinates. In IPM-based parking and mapping systems, a planar ground assumption enables a 3x3 homography $H$, estimated from landmark correspondences via DLT and RQ decomposition, defining the ground/image mapping

$(u, v, 1)^T \sim H (X, Y, 1)^T$

and inverted for IPM as $(X, Y)^T = H^{-1}(u, v, 1)^T / w$ (Nandi et al., 9 Mar 2026).

• Perspective transformation layers: In CNNs and DNNs, differentiable layers parameterize and learn the homographies $H$ directly as layer weights, enabling multi-view geometric warping and viewpoint invariance, with fully analytic gradients and integration anywhere in a deep vision model (Khatri et al., 2022).
• Bird’s-eye-view mapping and projection: Projection modules transform egocentric images and depth into BEV feature grids via patch-wise or continuous projective mapping (see Trans4Map, which projects front-view RGB/D with $\mathcal{V}$0 into world space, bins to a grid, and performs feature scattering and memory aggregation (Chen et al., 2022); BEV scene graphs integrate multi-view context via cross-attention and explicit spatial pooling for VLN (Liu et al., 2023)).
• Differentiable geometry-aware mapping for exploration: Pipeline decompositions such as floor segmentation $\mathcal{V}$1 geometric planar homography $\mathcal{V}$2 accumulation and egocentric map update (warping and addition), with all components differentiable for end-to-end RL/vision learning (Zhi et al., 2019).

3. Implicit and Learned Geometric Foundations

Transformer-based implicit geometric models represent the dominant paradigm for scalable, data-driven vision-to-geometry mapping:

• Implicit geometry fields: ViGT represents 3D scenes as a continuous function $\mathcal{V}$3 giving the occupancy probability for an arbitrary $\mathcal{V}$4, implemented by querying a BEV feature grid and decoding with a small MLP, without ever instantiating a discrete voxel or point cloud (Shirokov et al., 5 Feb 2026). IVGT extends this to a continuous SDF and appearance field, supporting spatial queries and mesh extraction via Marching Cubes (Wu et al., 15 May 2026).
• Cross-view attention and calibration-free fusion: ViGT and DVGT process tokenized multi-view inputs with alternating intra-view, cross-view, and temporal attention layers, learning to reconstitute metric 3D geometry and vehicle pose without explicit calibration. BEV features are inferred by learning cross-attention projections (from BEV queries to image patch tokens) with no access to known intrinsics/extrinsics, partitioning BEV space according to learned camera frusta (Shirokov et al., 5 Feb 2026, Zuo et al., 18 Dec 2025).
• Self-supervised metric association: Supervision exclusively from synchronized image–LiDAR pairs allows fully self-supervised, calibration-free training and robust generalization across rigs and datasets. For each LiDAR ray, points are labeled as free/occupied by stratified sampling; a binary cross-entropy loss supervises the continuous field, and mapping is directly metric by regressing world-frame points without further alignment (Shirokov et al., 5 Feb 2026).
• Joint representation and multi-task learning: Multi-head transformer architectures (DVGT, VGGT) decode both geometry (dense point maps, camera poses) and auxiliary tasks (semantic segmentation, depth, multi-view tracking) from a shared latent, supporting multi-task transfer and action grounding (Zuo et al., 18 Dec 2025, Abouzeid et al., 17 Sep 2025).
• Implicit encoder regularization: Losses incorporate uncertainty-weighted regression, eikonal and smoothness constraints for SDF fields, auxiliary camera head regression, and photometric and silhouette rendering consistency losses for robust, dataset-agnostic geometry recovery (Wu et al., 15 May 2026).

4. Latent, Symbolic, and Relational Vision-to-Geometry Pipelines

Beyond strictly metric mapping, several paradigms focus on geometric structures embedded in learned latent or symbolic domains:

• Latent geometry backbones for robotic control: VGGT serves as a 3D geometry backbone for manipulation, providing viewpoint-invariant 3D tokens and depth maps as inputs to action heads (e.g., Vision-Geometry-Action models), outperforming vision-language-action or 2D-centric video backbones by directly grounding action in a metric latent space (Song et al., 14 Apr 2026, Vuong et al., 19 Sep 2025). eVGGT distills geometric expertise with heavy-tailed distillation and strong speedup for real-world control (Vuong et al., 19 Sep 2025).
• Visual configuration manifolds in robotics: Visual generalized coordinates map dense sets of robot images to a topological manifold homeomorphic to joint-space, enabling visual roadmaps for inversion and planning, without ever regressing explicit joint or link parameters, and capable of collision avoidance and visual IK via local tangent interpolation and feature-space edge checking (Ramaiah et al., 2015).
• Symbolic geometry from diagrams and language: GeoDANO and GeoCLIP learn to map plane geometry diagram images into token embeddings that a LLM can use to infer primitives (e.g., points, lines, orthogonality), relations (concyclicity, angle measures), and formal symbolic programs, outperforming general-purpose VLMs in retrieval and expression of geometry (Cho et al., 17 Feb 2025).
• Relational 3D spatial memory for reasoning: Systems such as 3DSPMR build global spatial memory from egocentric RGB-D input, synthesizing TSDF volumetric maps, scene graphs, and keyframes, and fusing them into prompts to augment multi-modal LLMs for geometric reasoning, question answering, and navigation, fully leveraging geometric structure in high-level tasks (Cai et al., 2 Dec 2025).

5. Applications, Benchmarks, and Empirical Findings

Vision-to-geometry mapping underpins critical advances in autonomous driving, robotics, exploration, scene graph generation, and vision-language navigation:

• Autonomous driving: ViGT achieves SOTA metrics on NuScenes Occ3D (F1=0.7115, IoU=0.5658) and leading average rank in point-map estimation across five diverse driving datasets without camera calibration or manual 3D annotation (Shirokov et al., 5 Feb 2026). DVGT generalizes across arbitrary multi-view rigs, attaining $\mathcal{V}$5 accuracy on nuScenes, and 3D completeness/accuracy below 0.5 m (Zuo et al., 18 Dec 2025).
• Real-time actionable geometry for manipulation: eVGGT delivers >6% improvement in imitation learning success rate versus 2D vision baselines while reducing model size 5x and inference time 9x, and surpasses point-cloud-only alternatives for bimanual/monomanual manipulation (Vuong et al., 19 Sep 2025). VGA outperforms top VLA/GeoVLA baselines and demonstrates +6% cross-view generalization on real hardware (Song et al., 14 Apr 2026).
• Parking and urban mapping: Calibrated multi-view IPM-fused pipelines (YOLO+homography) enable accurate, cost-effective parking slot detection and 3D spatial allocation, with full code-level modularity around invertible projective mappings and 3D visualization (Nandi et al., 9 Mar 2026).
• Localization and navigation: CNNs trained purely on lean geometric renderings (edges/faces/depth) achieve meter-level accuracy in urban geo-localization, revealing the sufficiency of geometry for pose recovery and the distinctiveness of certain geometric city layouts (Kadosh et al., 2019). BEV-based navigation with geometric scene graph representations provides decisive gains (3–5%+) in instruction following and object-oriented navigation (Liu et al., 2023).
• Pose estimation under extreme views: Virtual Correspondence methods, leveraging human 3D shape priors, restore epipolar constraints and enable relative pose and downstream scene reconstruction even with zero co-visible features, outperforming SOTA across >90° view baselines (Ma et al., 2022).

6. Limitations, Assumptions, and Open Directions

Vision-to-geometry mapping approaches make a range of explicit and implicit assumptions:

• Visual distinguishability: Image-to-pose or configuration mappings are only bijective if the scene/robot is visually distinguishable in every configuration. This sometimes requires active texture placement or synchronized multi-camera capture to break ambiguities (Ramaiah et al., 2015).
• Sampling and coverage: Visual manifold and VRM methods require sufficiently dense sampling to ensure completeness and path feasibility. Geometry-aware recurrent approaches demand egomotion and view selection policies targeting maximal uncertainty reduction (Ramaiah et al., 2015, Cheng et al., 2018).
• Planar and homogeneous assumptions: IPM and homography-based layers typically assume ground- or scene-planarity, and are not directly applicable to arbitrary 3D surface topologies (Khatri et al., 2022, Nandi et al., 9 Mar 2026).
• Sensor calibration and temporal consistency: Explicit geometry pipelines assume accurate calibration and stable sensor synchronization (e.g., Cam1→Cam2 extrinsics in LED mapping, or Know camera poses in BEV mapping); fully implicit pipelines must overcome domain-induced confusion via strong LiDAR or depth anchoring, or extensive cross-domain pretraining (Huang et al., 2022, Ramaiah et al., 2015).
• Representation limitations: Homography-based PT layers can only model projective deformations of a plane, not arbitrary scene geometry; point-map or voxel models may struggle with fine geometry at low density, while continuous SDFs/ONets require large compute at high spatial resolution (Khatri et al., 2022, Wu et al., 15 May 2026).

Ongoing directions include improving pose-free and self-supervised geometric learning, deploying foundation models with stronger scene-generalization, developing efficient multi-resolution continuous representations, integrating geometric mapping into language/action pipelines, and addressing real-time, compute-constrained requirements for practical robotics and navigation deployments.