SPARGen: Unifying Spatial Perception and Reasoning through Native Multimodal Generation

Abstract: Spatial perception and reasoning from visual observations require recovering geometric structure, establishing correspondences, and understanding spatial relations. Existing approaches typically address these capabilities separately using task-specific architectures or external geometric modules, limiting knowledge transfer among complementary representations of the same physical scene. We introduce SPARGen, a unified multimodal framework that casts 3D reconstruction, dense correspondence, and spatial reasoning as instruction-conditioned generation tasks. SPARGen serializes compact structured and linguistic outputs as token sequences while generating dense geometric fields in image-aligned forms, enabling spatial supervision to jointly shape shared representations within a native multimodal generative model. Experiments across benchmarks for 3D reconstruction, correspondence, and spatial reasoning show that SPARGen achieves competitive performance across heterogeneous spatial tasks within a single native multimodal generative framework.

• The paper introduces a shared multimodal generative framework that produces spatial answers as autoregressive tokens and depth, point maps, and optical flow as VAE-compatible dense fields.
• The model achieves leading spatial-reasoning results across four benchmarks, including a 76.04 average on MindCube and 66.60 on SPAR, while remaining competitive with specialized geometry systems.
• The paper shows cross-task transfer from joint geometry, flow, and reasoning supervision, although VAE compression, relative-scale outputs, and semantic–flow capacity competition limit precision and metric reconstruction.

Problem formulation and central contribution

SPARGen addresses the fragmentation of spatial intelligence across geometric perception, correspondence estimation, and language-based spatial reasoning. Existing systems commonly specialize in one of these capabilities: feed-forward geometry models predict depth, point maps, camera poses, or tracks; correspondence models estimate optical flow or dense matches; and multimodal LLMs answer spatial questions. The paper’s central claim is that these tasks provide complementary supervision over a shared physical scene and should therefore be learned through a common multimodal generative interface rather than through independent task-specific heads.

The proposed framework, “SPARGen: Unifying Spatial Perception and Reasoning through Native Multimodal Generation” (2608.14138), instantiates this idea using a pretrained Bagel-style Mixture-of-Transformer-Experts backbone. Given an RGB image sequence and a natural-language instruction, the model generates either a token sequence or an image-aligned dense field. Token sequences represent textual answers, sparse queried geometric quantities, and camera poses. Dense fields represent depth, point maps, and optical flow. The system consequently treats 3D reconstruction, dense correspondence, and spatial question answering as instruction-conditioned generation tasks.

This formulation is important because it avoids introducing separate geometry encoders, regression heads, or external geometric modules. Instead, SPARGen uses the two generative pathways already present in its multimodal backbone: autoregressive generation for discrete sequences and rectified-flow generation for continuous image-like fields. The resulting architecture does not make all spatial targets homogeneous; rather, it preserves the representational structure of each target while allowing their supervision to update a shared multimodal representation.

Native representations for heterogeneous spatial outputs

SPARGen divides its targets into sequence-generation and field-generation tasks. For sequence tasks, a deterministic serializer maps the target into a canonical vocabulary sequence. Textual answers are emitted directly. Sparse geometric quantities, such as depths at queried points or numerical spatial attributes, are serialized in the order specified by the instruction, optionally followed by a textual answer. Camera poses are represented using a rotation quaternion, a translation direction, and a translation magnitude. Scalar components are quantized at resolution $10^{-3}$ and converted into dedicated numerical tokens.

This representation makes camera-pose estimation compatible with ordinary autoregressive decoding, but it also imposes a quantization scheme and a fixed serialization order. The model therefore predicts a discretized parameterization rather than a continuous pose distribution. That design is computationally convenient and consistent with the backbone’s language interface, although its effect on fine-grained pose accuracy is not isolated experimentally.

Dense outputs are converted into image-aligned three-channel fields before being encoded by the frozen VAE. Relative depth is normalized per image and replicated across the three channels. Point maps are expressed in a coordinate frame centered at the first camera and normalized using a sequence-level center and scale, preserving relative geometry between views. Optical flow is normalized by image width and height and transformed with a signed square-root mapping, which allocates more representational resolution to small displacements while retaining the sign of each component.

The use of image-aligned fields is a direct architectural adaptation of the pretrained image-generation pathway. It allows geometric quantities to be represented in a format compatible with the VAE and rectified-flow machinery, but it also means that the spatial precision of the dense pathway is constrained by the VAE’s latent compression. The paper explicitly acknowledges that this may impair geometric edges and high-precision physical quantities.

Figure 1: SPARGen maps spatial targets either to autoregressive token sequences or to VAE-compatible image-aligned fields generated through rectified flow.

Shared MoT architecture and training objective

The backbone contains an understanding expert and a generation expert. Visual inputs are processed through a ViT encoder, while text instructions are embedded as language tokens. The two streams use modality-specific projections and feed-forward layers but interact through joint multimodal self-attention. For sequence generation, the conditioning context consists of text and ViT representations. For dense-field generation, it additionally includes VAE representations of the input images.

Sequence targets are optimized with standard autoregressive cross-entropy, evaluated only at target-token positions. Dense targets are encoded by the frozen VAE into clean latents. A rectified-flow path interpolates each clean latent with Gaussian noise, and the generation expert predicts the corresponding velocity field. At inference, sampling begins from Gaussian noise and integrates the learned velocity field toward the target latent before VAE decoding.

The overall objective activates either the sequence or field loss for each training example, with the sequence loss weighted by $\lambda=0.25$. All non-VAE parameters are fine-tuned for 100K iterations using AdamW on 64 NVIDIA H100 GPUs. The training mixture combines three supervision groups:

• spatial reasoning data from MindCube, OmniSpatial, OST-Bench, SPAR-7M, and LLaVA-OneVision;
• visual-geometry data spanning relative depth, camera pose, and multiview point-map reconstruction;
• optical-flow data from synthetic and real-world motion datasets, including TartanAir, AutoFlow, FlyingChairs, FlyingThings3D, Kubric-4D, ParallelDomain-4D, and Spring.

Because many source datasets contain incomplete or sparse geometry annotations, the authors additionally use MoGe-generated dense pseudo-labels (Mohamed et al., 6 May 2025). The targets are normalized into relative-scale coordinates, and evaluation follows scale-aligned protocols. Consequently, SPARGen does not recover metric scale, a limitation shared with the compared VGGT and G$^2$VLM systems but material for applications requiring absolute scene dimensions.

Visual geometry and camera-pose results

SPARGen is evaluated on depth estimation, point-map reconstruction, and camera-pose estimation. Its results are strongest relative to the spatially unified baseline G$^2$VLM, while specialized geometry systems such as VGGT remain competitive or superior on several reconstruction metrics.

Task and benchmark	SPARGen	Strong specialized comparator	Interpretation
Depth, Sintel AbsRel	0.235	VGGT: 0.265	SPARGen is better
Depth, Sintel $\delta_1$	0.725	VGGT: 0.676	SPARGen is better
Depth, NYU-v2 AbsRel	0.071	VGGT: 0.065	VGGT remains better
Point maps, 7Scenes Acc.	0.034	VGGT: 0.022	Specialized model remains better
Point maps, 7Scenes Comp.	0.028	VGGT: 0.032	SPARGen is better
Camera pose, CO3Dv2 AUC@30	74.32	VGGT: 89.78	Specialized model remains substantially better

On Sintel, SPARGen obtains an AbsRel of 0.235 and $\delta_1$ of 0.725, improving over G$^2$VLM at 0.257 and 0.674, respectively. On NYU-v2, it reduces AbsRel relative to G$^2$VLM from 0.079 to 0.071 while matching its $\delta_1$ score at 0.935. The implication is that incorporating dense geometry into a native multimodal generator does not necessarily prevent competitive monocular depth estimation, although it does not uniformly match specialized models.

For point-map reconstruction, SPARGen improves over G$^2$VLM on 7Scenes, with accuracy error decreasing from 0.062 to 0.034 and completeness error from 0.031 to 0.028. On ETH3D, however, its accuracy error is 0.393 compared with G$\lambda=0.25$0VLM’s 0.539, while its completeness error is 0.445 compared with 0.355. This asymmetric behavior indicates that the unified formulation can improve some aspects of geometric reconstruction without providing consistently better coverage or reconstruction fidelity.

Camera-pose estimation exposes a larger gap with specialized geometry models. SPARGen achieves RRA@30 of 96.84, RTA@30 of 94.33, and AUC@30 of 74.32 on CO3Dv2. These values improve substantially over G$\lambda=0.25$1VLM’s 96.69, 92.22, and 56.85, respectively, but remain below VGGT’s 98.79, 96.89, and 89.78. Thus, the paper’s claim is not that native multimodal generation dominates dedicated geometric architectures; rather, it demonstrates that one model can provide broad spatial functionality while retaining reasonable geometric competence.

Dense correspondence through rectified-flow generation

Optical flow is generated through the same dense-field pathway as depth and point maps. The paper further introduces a predict–warp–predict refinement procedure. The initial flow estimate warps the second image toward the first, after which the model predicts a residual flow between the reference image and the warped image. The final estimate is the sum of the initial flow and residual.

On zero-shot KITTI evaluation, SPARGen obtains an EPE of 4.09 and F1-all of 13.34. It slightly improves on FlowFormer’s EPE of 4.10 and clearly improves on its F1-all score of 14.51. It also outperforms RAFT and GMFlow on both reported metrics.

Model	KITTI EPE $\lambda=0.25$2	KITTI F1-all $\lambda=0.25$3
RAFT	5.03	17.45
GMFlow	7.77	23.40
FlowFormer	4.10	14.51
SPARGen without refinement	5.26	21.82
SPARGen	4.09	13.34

The ablation is particularly informative: refinement reduces EPE from 5.26 to 4.09 and F1-all from 21.82 to 13.34. This shows that the native generative field prediction is not sufficient by itself to explain the final optical-flow result; the iterative warping procedure contributes materially to performance. The zero-shot result nevertheless supports the feasibility of using an image-generation pathway for dense correspondence estimation, provided that the output representation and inference procedure are carefully designed.

Figure 2: Qualitative predictions for point maps, optical flow before and after refinement, and depth fields.

Spatial reasoning performance

SPARGen produces its strongest empirical results on spatial reasoning. It is evaluated on MindCube, OmniSpatial, OST-Bench, and SPAR-Bench against general-purpose VLMs, spatial specialists, and unified spatial models.

Benchmark	SPARGen average	Improvement over strongest competing result
MindCube	76.04	9.85 points
OmniSpatial	54.00	1.97 points
OST	56.02	4.99 points
SPAR	66.60	24.71 points

SPARGen ranks first in 13 of the 15 reported non-proprietary categories. Its MindCube average of 76.04 exceeds Spatial-MLLM’s 66.19, while its SPAR average of 66.60 substantially exceeds VLM3R’s 41.89. The largest gains occur on the medium and high difficulty subsets of SPAR, where SPARGen reaches 74.04 and 79.25, respectively, compared with VLM3R’s 28.38 and 51.92.

The comparison with Qwen2.5-VL-72B is especially notable. SPARGen-7B surpasses the 72B model on the average scores of all four benchmarks shown in the paper. This result supports the authors’ contention that the gains are not attributable solely to parameter count. However, the evaluation does not eliminate differences in pretraining data, instruction tuning, benchmark exposure, or inference configuration. The result should therefore be interpreted as evidence for the value of spatially targeted joint supervision, not as a controlled scaling comparison.

The performance pattern also distinguishes SPARGen from models that attach a geometry encoder to an otherwise language-centered system. The paper argues that dense geometric supervision directly shaping the shared MoT representations improves the model’s ability to answer spatial questions. The benchmark gains are consistent with this hypothesis, although the experiments do not measure whether the model’s textual answers are explicitly grounded in its predicted geometric fields at inference time.

Cross-task transfer and supervision interactions

The ablation study removes one supervision category at a time: geometry, optical flow, or spatial reasoning. The results provide evidence for cross-task transfer, but also reveal a task-dependent trade-off.

Removing geometry supervision degrades KITTI optical flow from EPE 4.09 and F1-all 13.34 to 4.29 and 14.73, while the SPAR average falls from 66.60 to 62.41. This supports the claim that 3D structure provides transferable information for both dense correspondence and language-based spatial reasoning.

Removing optical-flow supervision increases 7Scenes point-map errors from 0.034 accuracy and 0.028 completeness to 0.040 and 0.038. The SPAR average also decreases to 65.92. The implication is that correspondence supervision contributes to cross-view geometric consistency and can benefit spatial reasoning despite not being itself a language task.

Removing reasoning supervision produces a smaller but still measurable reconstruction improvement: 7Scenes accuracy and completeness become 0.037 and 0.031, compared with 0.034 and 0.028 for the full model, so the full model remains better. The paper interprets this as evidence that semantic supervision can assist geometric representation learning. Optical flow, however, improves slightly without reasoning supervision, reaching EPE 4.06 and F1-all 13.01. The authors attribute this contradiction to capacity competition within the shared MoT backbone: autoregressive semantic generation may compete with dynamic dense-field prediction, even while benefiting static 3D structure.

These ablations are consistent with complementary supervision, but they do not establish causal mechanisms independently of changes in data volume or optimization dynamics. Each ablation removes an entire supervision family, so the measured effect combines representation transfer, task balancing, and altered training distributions.

Limitations and open questions

The principal architectural limitation is the frozen VAE. Although freezing it permits direct reuse of the pretrained image-generation pathway, its spatial compression can discard thin structures, sharp geometric boundaries, and high-frequency physical detail. This limitation is particularly relevant for optical-flow discontinuities, depth boundaries, and metric quantities.

SPARGen also operates in relative-scale geometry. Its normalized depth and point-map targets cannot recover metric scale, and the camera-pose parameterization separates translation direction from magnitude without resolving the global scale ambiguity. Results therefore do not establish metric reconstruction capability.

The dense-field pathway relies on target-specific encodings and, for some incomplete annotations, pseudo-labels generated by MoGe. The extent to which pseudo-label noise affects cross-task transfer is not isolated. Similarly, optical-flow performance includes a task-specific refinement procedure, although the broader architectural claim emphasizes the absence of task-specific prediction modules. The distinction between a native generation pathway and task-specific inference refinement is therefore relevant when assessing the degree of architectural unification.

Finally, the evaluation demonstrates correlation between joint supervision and improved benchmark performance but leaves open whether the model internally uses predicted geometry to answer spatial questions. A direct intervention study—such as corrupting, masking, or replacing dense geometric outputs while holding visual inputs fixed—would be needed to distinguish genuine geometric grounding from shared feature learning.

Conclusion

SPARGen presents a unified formulation in which dense geometry, optical flow, camera poses, and spatial answers are generated through the native modalities of a shared multimodal model. Its image-aligned field representation and autoregressive serialization scheme accommodate heterogeneous spatial targets without geometry-specific encoders or regression heads. Empirically, the 7B model achieves strong spatial-reasoning performance, competitive depth and reconstruction results, and zero-shot optical-flow performance comparable to specialized methods. The ablations indicate positive transfer among geometry, correspondence, and reasoning supervision, alongside a measurable trade-off between semantic generation and dynamic flow estimation. The paper therefore establishes native multimodal generation as a viable interface for jointly learning spatial perception and spatial reasoning, while leaving metric scale recovery, VAE-limited precision, and the causal grounding of language answers in geometric predictions unresolved.