SPARGen: Unified Spatial Perception and Reasoning

• SPARGen is an instruction-conditioned multimodal framework that treats 3D reconstruction, optical flow, camera poses, and spatial questions as either token sequences or image-aligned fields generated by one shared model.
• Its shared Mixture-of-Transformer-Experts backbone transfers supervision across geometry, correspondence, and language, achieving strong spatial-reasoning results and refined KITTI optical-flow performance of 4.09 EPE and 13.34 F1-all.
• SPARGen simplifies multimodal system design by avoiding task-specific prediction heads, but normalized geometry prevents absolute metric-scale recovery and frozen-VAE compression can reduce fine boundary accuracy.

SPARGen (“Spatial Perception and Reasoning Generation”) is a unified multimodal framework that formulates 3D reconstruction, dense correspondence estimation, and spatial reasoning as instruction-conditioned generation tasks. Introduced in “SPARGen: Unifying Spatial Perception and Reasoning through Native Multimodal Generation” (Quan et al., 14 Aug 2026), it uses a pretrained multimodal generative model to produce compact structured or linguistic outputs as autoregressive token sequences and dense geometric outputs as image-aligned three-channel fields. Its architecture is initialized from Bagel and uses a shared Mixture-of-Transformer-Experts (MoT) backbone, allowing geometric, correspondence, and language supervision to update shared multimodal representations.

1. Conceptual framework and objectives

SPARGen addresses spatial intelligence as a combination of complementary capabilities. Reconstruction estimates scene geometry and camera motion; correspondence identifies image locations representing the same scene content across observations; and spatial reasoning combines geometric and semantic information to answer questions about relative position, distance, orientation, and other spatial relations.

Conventional systems often separate these functions into specialized reconstruction networks, optical-flow architectures, vision-LLMs, or multimodal systems augmented with geometry encoders, regression heads, point transformers, or external reconstruction modules. SPARGen instead treats the tasks as different generation behaviors conditioned on RGB observations and a natural-language instruction.

The framework distinguishes two output classes:

• Sequence-generation tasks: textual answers, sparse numerical quantities, and structured camera poses are serialized into autoregressive token sequences.
• Dense-field tasks: depth maps, multi-view point maps, and optical-flow fields are transformed into image-aligned three-channel fields and generated through the model’s native image-generation pathway.

In this setting, native multimodal generation denotes production of different modalities through the pretrained model’s own generation mechanisms rather than through task-specific prediction modules. Language and structured numerical outputs use the native autoregressive decoder, while dense fields use the native image-generation and rectified-flow pathway. SPARGen therefore does not add a separate depth regressor, optical-flow head, camera-pose head, geometry encoder, or external geometric solver.

The underlying hypothesis is that the tasks provide mutually informative supervision. Depth and point maps encode scene structure, optical flow supplies cross-view or temporal consistency, and language reasoning encourages semantic abstraction and task-conditioned use of geometry. The ablation results support cross-task transfer, although the benefits are not uniform for every metric.

2. Output representations

Let the visual input be a sequence of $N$ RGB images,

$$\mathcal{I}=\{I_i\}_{i=1}^{N}, \qquad I_i\in\mathbb{R}^{H\times W\times 3},$$

and let $p$ denote the tokenized natural-language instruction. For a spatial task $\tau$ with target $Y_\tau$, SPARGen applies a task-dependent representation operator:

$$\mathcal{R}_{\tau}(Y_\tau)= \begin{cases} \mathcal{S}_{\tau}(Y_\tau)\in\mathcal{V}^{L_\tau}, & \tau\in\mathcal{T}_{\mathrm{seq}},\ \Phi_{\tau}(Y_\tau)\in\mathbb{R}^{M_\tau\times H\times W\times 3}, & \tau\in\mathcal{T}_{\mathrm{field}}. \end{cases}$$

Here, $\mathcal{S}_{\tau}$ is a deterministic serializer for compact targets, while $\Phi_{\tau}$ converts dense geometric targets into image-aligned three-channel fields. The unified conditional model is

$$p_{\theta}\!\left(\mathcal{R}_{\tau}(Y_\tau)\mid\mathcal{I},p\right).$$

Sequence outputs

Sequence outputs include textual answers, sparse geometric quantities, numerical spatial attributes, and structured camera poses. If a target is serialized as

$$\mathcal{S}_{\tau}(Y_\tau)=(s_1,\ldots,s_{L_\tau}),$$

SPARGen models it autoregressively:

$$\mathcal{I}=\{I_i\}_{i=1}^{N}, \qquad I_i\in\mathbb{R}^{H\times W\times 3},$$0

At inference, tokens are generated until an end-of-sequence token is produced, after which a deterministic deserializer converts numerical or structured outputs into the requested state.

Camera poses are represented by a rotation $$\mathcal{I}=\{I_i\}_{i=1}^{N}, \qquad I_i\in\mathbb{R}^{H\times W\times 3},$$1 and translation $$\mathcal{I}=\{I_i\}_{i=1}^{N}, \qquad I_i\in\mathbb{R}^{H\times W\times 3},$$2. The translation is parameterized by its direction and magnitude:

$$\mathcal{I}=\{I_i\}_{i=1}^{N}, \qquad I_i\in\mathbb{R}^{H\times W\times 3},$$3

The scalar components are quantized at resolution $$\mathcal{I}=\{I_i\}_{i=1}^{N}, \qquad I_i\in\mathbb{R}^{H\times W\times 3},$$4, mapped to dedicated numerical tokens, and emitted in a fixed canonical order. Because translation is normalized during training, recovered translation is not metric-scale calibrated.

Dense image-aligned fields

Dense targets are represented as image-like tensors whose channels retain correspondence with image pixels. Before VAE encoding, fields are rescaled to the VAE input range.

For a depth map $$\mathcal{I}=\{I_i\}_{i=1}^{N}, \qquad I_i\in\mathbb{R}^{H\times W\times 3},$$5, SPARGen uses relative min-max normalization:

$$\mathcal{I}=\{I_i\}_{i=1}^{N}, \qquad I_i\in\mathbb{R}^{H\times W\times 3},$$6

The normalized scalar map is replicated across three channels. This representation preserves relative near-to-far structure but discards absolute metric scale.

For point maps, $$\mathcal{I}=\{I_i\}_{i=1}^{N}, \qquad I_i\in\mathbb{R}^{H\times W\times 3},$$7 denotes the 3D point associated with pixel $$\mathcal{I}=\{I_i\}_{i=1}^{N}, \qquad I_i\in\mathbb{R}^{H\times W\times 3},$$8 in image $$\mathcal{I}=\{I_i\}_{i=1}^{N}, \qquad I_i\in\mathbb{R}^{H\times W\times 3},$$9. All point maps are transformed into a common coordinate system whose origin is the first camera. A sequence-wide center and shared scale are computed as

$p$0

and

$p$1

The normalized point map is

$p$2

Its channels represent normalized Cartesian coordinates $p$3, preserving both per-pixel geometry and relative cross-view scene arrangement.

For optical flow $p$4, horizontal and vertical displacements are normalized by image width and height and transformed using a signed square root:

$p$5

The first two channels encode horizontal and vertical flow, while the third encodes flow magnitude.

3. Architecture and multimodal pathways

SPARGen is built on Bagel’s shared MoT architecture, which contains two interacting expert streams:

• an understanding expert for visual and textual context;
• a generation expert for autoregressive language generation and image-like latent generation.

The streams retain modality-specific projections and feed-forward networks but interact through joint multimodal self-attention. This arrangement provides distinct mechanisms for different output modalities while maintaining a shared representation space.

Input images are processed by a ViT image encoder to produce visual-understanding tokens. Dense-field generation additionally uses a frozen VAE encoder to produce image-generation tokens. Text instructions are embedded by the text embedding module. The approximate contexts are

$p$6

for sequence generation, and

$p$7

for dense-field generation.

Thus, dense-field tasks use both semantic and image-generation features. The VAE encoder and decoder remain frozen during training, while the other model parameters are fine-tuned.

Rectified-flow dense generation

For a dense task, the target field is encoded by the frozen VAE:

$p$8

A noisy latent is sampled along a straight interpolation between the target latent and Gaussian noise:

$p$9

The target velocity is

$\tau$0

The generation expert predicts $\tau$1 and is trained to match $\tau$2. At inference, generation begins from $\tau$3, and the learned velocity field is numerically integrated from $\tau$4 to $\tau$5. The decoded latent is then converted deterministically into depth, point maps, or optical flow.

For optical flow, SPARGen uses predict–warp–predict refinement. Given an initial flow estimate $\tau$6, the second image is warped according to

$\tau$7

The model predicts a residual flow $\tau$8 between the first image and the warped second image, producing

$\tau$9

This refinement focuses prediction on residual misalignment.

4. Training procedure and objectives

SPARGen is trained on a mixture of sequence-generation and dense-field examples. Sequence targets use average autoregressive cross-entropy:

$Y_\tau$0

Dense-field targets use rectified-flow matching:

$Y_\tau$1

where $Y_\tau$2 is the number of scalar elements in the target latent.

The per-task objective is

$Y_\tau$3

with sequence-loss weight $Y_\tau$4. The overall objective is the expectation of $Y_\tau$5 over the mixture of training data and task instructions. There is no separate reconstruction loss, correspondence loss, or pose-regression loss beyond the representation-specific generative objectives.

The training mixture includes spatial reasoning data from MindCube, OmniSpatial, OST-Bench, SPAR-7M, and LLaVA-OneVision VQA data. Visual geometry supervision is aggregated from ASE, BlendedMVS, CO3D, DeMoN, DL3DV, Hypersim, IRS, MegaSynth, MVS-Synth, Objaverse, OmniObject3D, ScanNet v2, ScanNet++, SceneNet RGB-D, Taskonomy, and WildRGB-D. Optical-flow training uses TartanAir, AutoFlow, FlyingChairs, FlyingChairs2, FlyingThings3D, Monkaa, Kubric-4D, ParallelDomain-4D, and Spring. MoGe supplies dense image-aligned pseudo-labels for incomplete or sparse geometric annotations.

SPARGen is initialized from pretrained Bagel weights. The VAE encoder and decoder are frozen, while other parameters are fine-tuned for 100K iterations using AdamW on 64 NVIDIA H100 GPUs with learning rate $Y_\tau$6.

5. Empirical evaluation

SPARGen is evaluated on visual geometry, optical flow and dense correspondence, and spatial reasoning.

Visual geometry

Depth is evaluated on Sintel and NYU-v2; point-map reconstruction on 7Scenes and ETH3D; and camera pose on CO3D v2. The reported measures include depth AbsRel and $Y_\tau$7, reconstruction accuracy and completeness error, relative rotation accuracy $Y_\tau$8, relative translation accuracy $Y_\tau$9, and pose AUC@30.

Model	Sintel AbsRel	NYU AbsRel	7Scenes Acc.	CO3D AUC
FLARE	.409	.164	.035	77.99
DUSt3R	.362	.134	.026	79.33
VGGT	.265	.065	.022	89.78
$$\mathcal{R}_{\tau}(Y_\tau)= \begin{cases} \mathcal{S}_{\tau}(Y_\tau)\in\mathcal{V}^{L_\tau}, & \tau\in\mathcal{T}_{\mathrm{seq}},\ \Phi_{\tau}(Y_\tau)\in\mathbb{R}^{M_\tau\times H\times W\times 3}, & \tau\in\mathcal{T}_{\mathrm{field}}. \end{cases}$$0VLM	.257	.079	.062	56.85
SPARGen	.235	.071	.034	74.32

SPARGen reports the strongest Sintel AbsRel and improves substantially over $$\mathcal{R}_{\tau}(Y_\tau)= \begin{cases} \mathcal{S}_{\tau}(Y_\tau)\in\mathcal{V}^{L_\tau}, & \tau\in\mathcal{T}_{\mathrm{seq}},\ \Phi_{\tau}(Y_\tau)\in\mathbb{R}^{M_\tau\times H\times W\times 3}, & \tau\in\mathcal{T}_{\mathrm{field}}. \end{cases}$$1VLM on most visual-geometry measures, particularly Sintel depth, 7Scenes reconstruction, and the reported CO3D pose metrics relative to that unified baseline. VGGT remains stronger on several specialized reconstruction measures, especially ETH3D and CO3D. SPARGen predicts normalized geometry, so depth and reconstruction evaluation uses scale alignment.

Optical flow

Zero-shot evaluation is conducted on KITTI without benchmark fine-tuning.

Model	EPE	F1-all
RAFT	5.03	17.45
GMFlow	7.77	23.40
FlowFormer	4.10	14.51
SPARGen without refinement	5.26	21.82
SPARGen with refinement	4.09	13.34

Refinement lowers EPE from $$\mathcal{R}_{\tau}(Y_\tau)= \begin{cases} \mathcal{S}_{\tau}(Y_\tau)\in\mathcal{V}^{L_\tau}, & \tau\in\mathcal{T}_{\mathrm{seq}},\ \Phi_{\tau}(Y_\tau)\in\mathbb{R}^{M_\tau\times H\times W\times 3}, & \tau\in\mathcal{T}_{\mathrm{field}}. \end{cases}$$2 to $$\mathcal{R}_{\tau}(Y_\tau)= \begin{cases} \mathcal{S}_{\tau}(Y_\tau)\in\mathcal{V}^{L_\tau}, & \tau\in\mathcal{T}_{\mathrm{seq}},\ \Phi_{\tau}(Y_\tau)\in\mathbb{R}^{M_\tau\times H\times W\times 3}, & \tau\in\mathcal{T}_{\mathrm{field}}. \end{cases}$$3 and F1-all from $$\mathcal{R}_{\tau}(Y_\tau)= \begin{cases} \mathcal{S}_{\tau}(Y_\tau)\in\mathcal{V}^{L_\tau}, & \tau\in\mathcal{T}_{\mathrm{seq}},\ \Phi_{\tau}(Y_\tau)\in\mathbb{R}^{M_\tau\times H\times W\times 3}, & \tau\in\mathcal{T}_{\mathrm{field}}. \end{cases}$$4 to $$\mathcal{R}_{\tau}(Y_\tau)= \begin{cases} \mathcal{S}_{\tau}(Y_\tau)\in\mathcal{V}^{L_\tau}, & \tau\in\mathcal{T}_{\mathrm{seq}},\ \Phi_{\tau}(Y_\tau)\in\mathbb{R}^{M_\tau\times H\times W\times 3}, & \tau\in\mathcal{T}_{\mathrm{field}}. \end{cases}$$5, making the refined model slightly better than FlowFormer on both metrics.

Spatial reasoning

SPARGen is evaluated on MindCube, OmniSpatial, OST-Bench, and SPAR-Bench. Its reported average scores are $$\mathcal{R}_{\tau}(Y_\tau)= \begin{cases} \mathcal{S}_{\tau}(Y_\tau)\in\mathcal{V}^{L_\tau}, & \tau\in\mathcal{T}_{\mathrm{seq}},\ \Phi_{\tau}(Y_\tau)\in\mathbb{R}^{M_\tau\times H\times W\times 3}, & \tau\in\mathcal{T}_{\mathrm{field}}. \end{cases}$$6 on MindCube, $$\mathcal{R}_{\tau}(Y_\tau)= \begin{cases} \mathcal{S}_{\tau}(Y_\tau)\in\mathcal{V}^{L_\tau}, & \tau\in\mathcal{T}_{\mathrm{seq}},\ \Phi_{\tau}(Y_\tau)\in\mathbb{R}^{M_\tau\times H\times W\times 3}, & \tau\in\mathcal{T}_{\mathrm{field}}. \end{cases}$$7 on OmniSpatial, $$\mathcal{R}_{\tau}(Y_\tau)= \begin{cases} \mathcal{S}_{\tau}(Y_\tau)\in\mathcal{V}^{L_\tau}, & \tau\in\mathcal{T}_{\mathrm{seq}},\ \Phi_{\tau}(Y_\tau)\in\mathbb{R}^{M_\tau\times H\times W\times 3}, & \tau\in\mathcal{T}_{\mathrm{field}}. \end{cases}$$8 on OST, and $$\mathcal{R}_{\tau}(Y_\tau)= \begin{cases} \mathcal{S}_{\tau}(Y_\tau)\in\mathcal{V}^{L_\tau}, & \tau\in\mathcal{T}_{\mathrm{seq}},\ \Phi_{\tau}(Y_\tau)\in\mathbb{R}^{M_\tau\times H\times W\times 3}, & \tau\in\mathcal{T}_{\mathrm{field}}. \end{cases}$$9 on SPAR. It achieves the best average among the compared models on all four benchmarks and ranks first in 13 of 15 reported categories among non-proprietary models.

Relative to the strongest competing result for each benchmark, the average improvements are 9.85 points on MindCube, 1.97 points on OmniSpatial, 4.99 points on OST, and 24.71 points on SPAR. The largest gains occur on the medium- and high-difficulty subsets of SPAR. The comparison includes GPT-4o, Claude Sonnet 4.6, Qwen2.5-VL-7B and Qwen2.5-VL-72B, LLaVA-Video-7B, LLaVA-OneVision-7B, Bagel-7B, VLM3R-7B, Spatial-MLLM-7B, and $\mathcal{S}_{\tau}$0VLM-2B.

6. Ablations, transfer, and limitations

Ablation studies remove geometry, optical-flow, or reasoning supervision. Without geometry supervision, KITTI EPE increases to 4.29, F1-all to 14.73, and SPAR average performance falls to 62.41. Without flow supervision, 7Scenes accuracy and completeness become 0.040 and 0.038, while SPAR reaches 65.92. Without reasoning supervision, 7Scenes accuracy and completeness become 0.037 and 0.031, and KITTI EPE and F1-all become 4.06 and 13.01. The complete model obtains 7Scenes accuracy 0.034, completeness 0.028, KITTI EPE 4.09, F1-all 13.34, and SPAR average 66.60.

These results suggest that explicit 3D supervision benefits correspondence and spatial reasoning, while optical-flow supervision improves cross-view consistency and spatial reasoning. The slight improvement in flow after removing reasoning supervision suggests capacity competition within the shared MoT backbone: autoregressive semantic generation may compete mildly with dynamic dense-field prediction.

The principal architectural limitation is spatial compression by the frozen VAE, which can blur fine geometric boundaries and limit high-precision physical quantities. Normalized depth and point-map representations also prevent recovery of absolute metric scale. Consequently, depth and reconstruction require scale alignment during evaluation.

Other limitations follow from the representation and training design. Dense-field outputs depend on the VAE’s ability to preserve pixel-aligned geometry, while sequence outputs depend on deterministic serialization, numerical quantization, and decoding. Multitask learning provides an overall balance rather than uniformly improving every metric: removing reasoning supervision slightly improves flow, whereas removing geometry or flow supervision harms other task families.

7. Relation to prior frameworks and scope

SPARGen differs from specialized geometry systems such as DUSt3R and VGGT, optical-flow systems such as RAFT, GMFlow, and FlowFormer, and geometry-aware multimodal systems such as $\mathcal{S}_{\tau}$1VLM. Specialized models may achieve higher performance on particular reconstruction or flow measures, while SPARGen provides a single instruction-conditioned interface for depth, point maps, camera poses, optical flow, and spatial answers.

Its defining methodological property is not merely multitask training but the use of representations compatible with native multimodal generation: tokens for compact structured information and image-aligned fields for dense spatial information. Shared attention layers permit supervision from depth, point maps, optical flow, camera poses, and language answers to influence common representations.

The name SPARGen is specific to the 2026 multimodal framework (Quan et al., 14 Aug 2026). Sparse Partitioning (Speed et al., 2011) and the spatial generalized linear mixed model using INLA-SPDE (Guillot, 2012) are distinct methods and do not introduce SPARGen. Any analogy to sparse grouping, spatial association analysis, or genotype–environment modeling would be conceptual rather than a relationship established by those papers. SPARGen is appropriately characterized as an instruction-conditioned native multimodal generation framework for unified spatial perception and reasoning.