Lifted Geometric Pre-Training

• Lifted geometric pre-training is a method that converts dense visual data into sparse, interpretable geometric representations such as keypoints, Gaussians, and occupancy fields.
• It refines raw features into structured forms through geometry-aware objectives and intermediate transformations, supporting tasks from 3D reconstruction to multimodal reasoning.
• It improves data efficiency, robustness, and transferability in applications like robotics, autonomous driving, and 3D scene understanding.

Lifted geometric pre-training denotes a family of pre-training paradigms in which learning first passes through a geometry-grounded intermediate space—such as image-space keypoints, bird’s-eye-view tensors, 3D Gaussians, continuous occupancy fields, SE(3)-aware graph embeddings, or formal geometric programs—before downstream prediction. Across recent work, the “lift” may convert dense visual features into sparse coordinates for control, map multi-view RGB into explicit or implicit 3D structure, distill 2D foundation-model supervision into 3D renderable variables, or reformulate optimization itself in a lifted auxiliary-variable space. What unifies these variants is that geometry is treated as an explicit interface for learning, transfer, and reasoning rather than as a by-product of semantic pre-training (Buoso et al., 15 May 2026, Xu et al., 2024, Pengmei et al., 2024, Ljungbergh et al., 19 Mar 2025, Zhang et al., 17 Mar 2026).

1. Conceptual foundations

Lifted geometric pre-training is best understood as a shift in where inductive bias enters the pipeline. In standard representation learning, a model is often pretrained on appearance-dominant objectives and only later adapted to geometry-sensitive tasks. Lifted methods instead impose geometric structure before or during adaptation. In visuomotor control, this means replacing a high-dimensional frozen feature map with a small set of object-centric coordinates. In autonomous driving, it means learning BEV, Gaussian, or occupancy representations that already live in a physically meaningful frame. In molecular learning, it means learning SE(3)-aware all-atom descriptors through coordinate denoising rather than task-specific labels. In multimodal reasoning, it can mean forcing a model to predict metric 3D points or formal geometric relations before optimizing for text-heavy downstream objectives (Buoso et al., 15 May 2026, Huang et al., 2023, Pengmei et al., 2024, Zhang et al., 17 Mar 2026).

The term “lifted” is used in at least three technically distinct senses. First, there is representational lifting, where raw observations are mapped into a structured geometric state, as in 2D anchors, 3D Gaussians, pointmaps, or occupancy fields. Second, there is supervisory lifting, where inexpensive or external signals—simulator masks, LiDAR, differentiable rendering, SAM or CLIP pseudo-labels, pseudo-depth, or formal captions—are transferred into the target model through geometry-aware objectives. Third, there is optimization lifting, where nested network composition is rewritten as a higher-dimensional problem over auxiliary variables, with Bregman or Fenchel geometry shaping the training landscape and enabling stable proximal updates (Cao et al., 27 Feb 2025, Xia et al., 2024, Wang et al., 10 Oct 2025).

A recurring misconception is that lifted geometric pre-training is synonymous with explicit 3D reconstruction. The recent literature is broader. GAP for manipulation lifts frozen VFM features into 2D keypoints rather than 3D points; GeoX lifts diagram images into formal captions and executable programs; lifted Bregman training lifts activations into auxiliary optimization variables rather than scene geometry. What remains constant is the use of an intermediate geometric or structure-preserving representation that constrains downstream learning (Buoso et al., 15 May 2026, Xia et al., 2024, Wang et al., 10 Oct 2025).

2. Representational forms and lifting operators

One influential form is the coordinate interface. GAP introduces a lightweight spatial adapter on top of a frozen VFM; a $3\times 3$ convolution plus $1\times 1$ convolution produce $K$ heatmaps, and a spatial softmax returns $K$ 2D keypoints. For the $k$-th heatmap, the keypoint is computed as

$p_{k,t}=\sum_{x=1}^{w}\sum_{y=1}^{h} \begin{bmatrix}x\y\end{bmatrix} \frac{\exp(\Phi_{t,k,x,y})}{\sum_{x',y'}\exp(\Phi_{t,k,x',y'})}.$

These coordinates, with $K=16$ per camera in all experiments, form the policy state directly. The lift is therefore from dense image features $\Phi_t$ to sparse coordinates $P_t$, turning policy learning into state-based learning in a small coordinate space (Buoso et al., 15 May 2026).

A second form is the explicit 3D scene representation. GaussianPretrain conceptualizes 3D Gaussian anchors as volumetric LiDAR points and learns position, covariance, rotation, scale, opacity, and color from lifted voxel features. LIFT-GS likewise predicts 3D Gaussians from point clouds, but uses them as differentiable, renderable carriers of masks and feature channels for 3D vision-language grounding. GAP-MLLM adopts sparse pointmaps rather than full explicit point clouds: prompted pixels are required to decode to metric $(x,y,z)$ coordinates in the first-frame camera coordinate system. These approaches differ in granularity, but all replace raw 2D evidence with a renderable or queryable geometric substrate (Xu et al., 2024, Cao et al., 27 Feb 2025, Zhang et al., 17 Mar 2026).

A third form is the continuous field. GASP defines a general occupancy field $1\times 1$0, an ego-occupancy field $1\times 1$1, and a semantic feature field $1\times 1$2, all queried at arbitrary spacetime points. GeoPT uses a similar $1\times 1$3-conditioned interface, but with synthetic per-point velocities and trajectory targets derived from geometry alone; the lift is from static geometry to geometry–dynamics trajectories. In both cases, the representation is not a discrete token list but a continuous function over space or spacetime (Ljungbergh et al., 19 Mar 2025, Wu et al., 23 Feb 2026).

Other domains instantiate the same pattern with different geometric objects. All-atom Geom-GNN pre-training learns transferable descriptors from radius graphs with SE(3)-equivariant or invariant processing, so the lifted space is an all-atom latent embedding rather than a visual coordinate frame. GeoX lifts diagram perception into formal primitive–relation captions and formal solver programs. In lifted Bregman training, the lifted objects are pre-activations and activations $1\times 1$4, and geometry is supplied by convex generators, Bregman distances, and proximal operators rather than physical space (Pengmei et al., 2024, Xia et al., 2024, Wang et al., 10 Oct 2025).

3. Supervision and objective design

Lifted geometric pre-training is defined as much by its objectives as by its representations. A common strategy is task-decoupled proxy supervision. GAP pre-trains only the spatial adapter on simulated trajectories with free object masks and no actions, using centroid alignment, geometric spread, and keypoint diversity losses. The objective is explicitly geometric: $1\times 1$5 with $1\times 1$6, $1\times 1$7, $1\times 1$8. Notably, the reported method uses no separate entropy term and no explicit temporal consistency loss; repeatability is described as emerging implicitly from training over moving scenes under mask-based geometric constraints (Buoso et al., 15 May 2026).

A second strategy is self-supervised geometry reconstruction or denoising. In all-atom Geom-GNN pre-training, coordinates are corrupted and the model predicts the added noise,

$1\times 1$9

producing transferable SE(3)-aware descriptors. GASP similarly uses binary occupancy, ego-path occupancy, and semantic feature regression at sampled spacetime points. MeSa inserts an intermediate geometric stage based on projective reprojection and geometry consistency after masked pre-training, specifically because masked reconstruction alone leaves later layers suboptimal for depth estimation (Pengmei et al., 2024, Ljungbergh et al., 19 Mar 2025, Khan et al., 2023).

A third strategy is render-supervised distillation. GaussianPretrain reconstructs RGB, depth, and occupancy only on LiDAR-filtered masked patches, using 3D Gaussian splatting rather than NeRF-style fields. LIFT-GS renders 3D Gaussian masks and features to posed 2D views and supervises them with SAM masks, CLIP features, and correspondence losses; this transfers 2D foundation-model supervision into a 3D grounding model without any 3D annotations. GLaD distills a frozen geometry-aware transformer into the LLM hidden states corresponding to image tokens, while GAP-MLLM uses a visual-prompted joint task in which the model predicts both semantic labels and sparse metric pointmaps for prompted pixels (Xu et al., 2024, Cao et al., 27 Feb 2025, Guo et al., 10 Dec 2025, Zhang et al., 17 Mar 2026).

A fourth strategy is formal geometry–language alignment. GeoX pre-trains a diagram encoder with masked autoencoding on more than 120K geometric diagrams, a symbol decoder on a 100M-token geometry corpus, and then aligns the two with contrastive loss, geometry–text matching, caption generation, and a sparsity-regularized semantics-guided sampler. The resulting instruction-tuned model emits executable formal programs that can be checked by a symbolic solver, so the lifted representation is tied directly to verifiability rather than only to perceptual similarity (Xia et al., 2024).

4. Major application families

In robotics and action-conditioned control, lifted geometry is used as a control interface. GAP keeps the VFM frozen and regularizes only a $K$0M-parameter adapter, obtaining stable anchors that are concatenated and fed into a Diffusion Policy. GLaD targets Vision-Language-Action models and moves geometry into the multimodal latent space itself: instead of distilling geometry only into the vision encoder, it aligns the LLM hidden states of visual tokens to VGGT features, so that geometric priors influence the representations that drive action prediction (Buoso et al., 15 May 2026, Guo et al., 10 Dec 2025).

In autonomous driving and 3D scene understanding, several variants appear. GAPretrain uses a LiDAR teacher and a unified BEV representation to pretrain both the image backbone and the view transformer, explicitly addressing the backbone–transformer misalignment left by depth-only pretraining. GaussianPretrain replaces NeRF-style decoders with explicit 3D Gaussian splatting and jointly supervises geometry and texture through RGB, depth, and occupancy reconstruction. GASP lifts past LiDAR into continuous 4D occupancy and semantic fields for forecasting, mapping, and trajectory prediction. GAP-MLLM extends the same alignment idea to RGB-only multimodal large models by activating geometric perception through sparse pointmap prediction before downstream grounding or captioning (Huang et al., 2023, Xu et al., 2024, Ljungbergh et al., 19 Mar 2025, Zhang et al., 17 Mar 2026).

In 3D vision-language grounding, LIFT-GS addresses the six-order-of-magnitude data gap between 2D and 3D by predicting renderable 3D Gaussians from point clouds and supervising them entirely through 2D pseudo-labels. The key geometric object is not a latent feature tensor but a differentiable scene representation that can be observed from multiple views and optimized end-to-end through rendered masks, features, and photometric outputs (Cao et al., 27 Feb 2025).

In molecular modeling and scientific simulation, the lift often couples geometry to symmetry or dynamics. The all-atom Geom-GNN work studies pre-training, scaling, and zero-shot transfer with ET and ViSNet architectures under coordinate denoising, then aggregates atom-level embeddings into residue-level tokens for VAMP or fold classification. GeoPT argues that geometry-only pretraining on static 3D shapes can cause negative transfer for physics tasks, and therefore augments static geometry with synthetic velocity fields and geometry-feature trajectories; the pre-training space is lifted from $K$1 to $K$2 so that the learned correlations are dynamics-aware rather than purely shape-based (Pengmei et al., 2024, Wu et al., 23 Feb 2026).

In medical imaging, lifting often replaces appearance-based similarity with topology-aware or geometry-aware supervision. GVSL learns inter-image similarity through a geometric matching head that predicts an affine transform and a local deformation field, with local NCC and smoothness enforcing topology-preserving correspondence across 3D volumes. MeSa inserts geometric and supervised depth-specific pre-training stages after masked pre-training so that later layers become aligned with projective geometry, surfaces, and edges. PrimGeoSeg pre-trains 3D segmentation models entirely on synthetic primitive volumes, where contour images $K$3 are mapped to filled masks $K$4, emphasizing 3D shape composition and boundaries rather than texture (He et al., 2023, Khan et al., 2023, Tadokoro et al., 2024).

In generative modeling and symbolic reasoning, the same principle appears in different form. LiftedGAN distills a frozen 2D StyleGAN2 into a 3D-aware generator with disentangled shape, albedo, viewpoint, and lighting, using differentiable rendering and teacher-generated proxy views rather than 3D supervision. GeoX, by contrast, lifts plane geometry into a formal symbolic regime in which diagrams are encoded, aligned to formal captions, and decoded into solver-executable programs. These two cases sit at opposite ends of the spectrum—continuous 3D rendering versus symbolic proof generation—but both exemplify pre-training through an explicitly structured intermediate geometry (Shi et al., 2020, Xia et al., 2024).

5. Empirical behavior, efficiency, and scaling

Across application domains, lifted geometric pre-training is most consistently associated with improvements in low-data transfer, robustness to perturbation, or better use of frozen high-capacity backbones.

Setting	Reported outcome	Source
RoboMimic Can, 15 demos	GAP $K$5 vs AFA $K$6	(Buoso et al., 15 May 2026)
nuScenes 3D detection	$K$7 NDS over ImageNet pretraining on UVTR	(Xu et al., 2024)
BEVFormer on nuScenes val	$K$8 mAP and $K$9 NDS, with gains of $K$0 mAP and $K$1 NDS	(Huang et al., 2023)
ScanNet200 open-vocabulary instance segmentation	$K$2 mAP vs $K$3 prior SOTA	(Cao et al., 27 Feb 2025)
LIBERO average success	$K$4 vs UniVLA $K$5	(Guo et al., 10 Dec 2025)
Physics simulation benchmarks	labeled-data reductions of $K$6–$K$7 and $K$8 convergence acceleration	(Wu et al., 23 Feb 2026)

The reported gains are not uniform in origin. In manipulation, the advantage comes from regularizing the bottleneck between frozen visual features and policy learning. In autonomous driving, gains often come from aligning the lifting operator itself—BEV transformation, Gaussian decoding, or occupancy forecasting—rather than only enriching the image backbone. In 3D VLG and MLLM settings, the benefit is frequently attributed to moving geometry into the multimodal latent space where grounding decisions are formed, rather than concatenating geometric features only at the encoder boundary.

Scaling behavior is also heterogeneous. The all-atom Geom-GNN study explicitly reports that denoising loss deviates from the standard power-law form $K$9, with early saturation beyond approximately $k$0 parameters and architecture-dependent expressivity differences, especially ViSNet $k$1 ET. By contrast, GASP reports predictable logarithmic scaling on large unlabeled driving corpora and no saturation within the tested range. This suggests that “more geometry” is not a universal scaling law; empirical behavior depends on whether the lifted target is well matched to the downstream task, the symmetry class, and the model family (Pengmei et al., 2024, Ljungbergh et al., 19 Mar 2025).

6. Limitations, debates, and future directions

A central limitation is that geometry-only pre-training is not automatically beneficial. GeoPT shows that native geometry pre-training on static shapes can degrade aerodynamics accuracy versus training from scratch, because downstream physics lives in the richer joint space $k$2 rather than geometry alone. GAPretrain similarly argues that depth-pretrained image backbones remain spatially misaligned with randomly initialized view transformers, and MeSa shows that masked pre-training alone leaves later depth-network layers poorly aligned with 3D reasoning. The recurring lesson is that lifted targets must be aligned with the actual downstream operator—control, view transformation, dynamics, or reasoning—not merely with generic geometric plausibility (Wu et al., 23 Feb 2026, Huang et al., 2023, Khan et al., 2023).

A second limitation concerns supervision and infrastructure. Several methods rely on simulator masks, LiDAR-derived occupancy, accurate camera poses, or expert-produced formal annotations. GAP assumes access to object masks in simulation during proxy pre-training; GaussianPretrain depends on calibration and LiDAR coverage; GeoX requires 6,232 real diagram–formal-caption pairs annotated by 10 experts over roughly 200 hours; LIFT-GS inherits biases from SAM and CLIP pseudo-labels and from posed-frame coverage. These requirements are often cheaper than downstream human annotation, but they remain nontrivial and can determine transfer quality (Buoso et al., 15 May 2026, Xu et al., 2024, Xia et al., 2024, Cao et al., 27 Feb 2025).

A third limitation lies in representational fidelity. GAP uses 2D image-space anchors and leaves 3D lifting as future work. GAP-MLLM uses sparse pointmap supervision and reports coarser boundaries than dense explicit-3D systems. The Geom-GNN work reports over-smoothing and limited global relationship modeling without token mixers. GLaD improves robustness to object appearance perturbations, but position and layout perturbations remain difficult across LIBERO-PRO suites. These observations indicate that lifting often improves structural bias without fully solving occlusion reasoning, depth ambiguity, or long-range spatial composition (Buoso et al., 15 May 2026, Zhang et al., 17 Mar 2026, Pengmei et al., 2024, Guo et al., 10 Dec 2025).

Future directions recur across the literature. Several works call for integrating semantics and geometry more tightly: GAP proposes combining semantic tokens with anchors and extending to text-conditioned policies; GASP suggests richer teachers such as CLIP, SAM, or SAL and broader sensor-agnostic encoders; GeoX identifies theorem-prover integration and richer primitive–relation vocabularies; GeoPT points toward more general input schemas spanning wider physics families. A plausible implication is that the next stage of lifted geometric pre-training will not be defined by a single representation type, but by architectures that can move flexibly between sparse coordinates, dense fields, renderable structure, and formal symbolic objects while preserving a task-aligned geometric interface (Ljungbergh et al., 19 Mar 2025, Xia et al., 2024, Wu et al., 23 Feb 2026).