SpatialEdit-16B: Diffusion Spatial Editor

• SpatialEdit-16B is a cascaded diffusion-based editor designed for fine-grained control, allowing explicit object placement and camera viewpoint adjustments.
• It leverages a pretrained multimodal encoder, an MM-DiT U-Net, and a synthetic SpatialEdit-500K dataset to achieve precise spatial manipulations.
• Benchmark results and ablation studies show marked improvements in moving, rotation, and framing errors compared to previous open-source methods.

SpatialEdit-16B is a two-stage cascaded diffusion-based editor specifically architected for fine-grained image spatial editing, enabling explicit control over object placement and camera viewpoint adjustments. Developed atop a pretrained multimodal encoder and DiT-style transformer decoder, it is benchmarked with geometry- and plausibility-aware metrics and trained on a large-scale, systematically generated synthetic dataset. It achieves competitive general editing performance and outperforms prior approaches on spatial manipulation benchmarks, establishing a new standard for open-source controllable spatial image editing (Xiao et al., 6 Apr 2026).

1. Model Architecture

SpatialEdit-16B employs a cascaded architecture integrating the following components:

• Input Conditioning
  • A reference image $I_0$ is passed through a frozen VAE encoder, producing latent variables $z_0$.
  • Textual instructions $t$ and $I_0$ are jointly processed by a vision–LLM (Qwen3-VL), outputting a global embedding $e_t$.
• Denoising Backbone & Conditioning
  • An MM-DiT U-Net, derived from StableDiffusion3, is used for denoising. At each diffusion timestep $s$, it receives:
  • Noisy latent $z_s$ and text embedding $e_t$.
  • For object-centric edits: bounding-box tokens $B = \{b_1, ..., b_k\}$, discrete viewpoint tokens $V$ (one-hot, 8 orientations).
  • For camera-centric edits: continuous pose tokens $z_0$0.
  • Spatial tokens are projected and injected into cross-attention layers to directly steer spatial geometry.
• Output, Training, and Fine-tuning
  • The network predicts noise $z_0$1 to match the diffusion-injected noise $z_0$2 using a standard DDPM objective. The final denoised latent $z_0$3 is decoded by the VAE into the edited image $z_0$4.
  • Initial pretraining involves general image editing with approximately 1.5M GPT-generated edits. The DiT decoder is then frozen, and only rank-16 low-rank adapters (LoRA) are trained with the SpatialEdit-500K dataset for spatial editing specialization.

2. SpatialEdit-500K: Synthetic Dataset Design

SpatialEdit-500K is constructed with a controllable Blender pipeline, yielding precise ground truth for both object- and camera-centric edits across diverse backgrounds.

• Object-Centric Pipeline
  • Assets: 8 categories from TexVerse (e.g., chairs, cars, mugs).
  • Canonical front views rendered; verified with Gemini 2.5 VLM. Side-heavy models are filtered out.
  • 8 discrete viewpoints at 45° increments; segmentation by SAM3, with poor masks discarded.
  • For each view, apply random translations $z_0$5 (relative to image width) and scale $z_0$6; retain instances that remain wholly visible.
  • Foreground composited onto semantically compatible backgrounds using Nano-Pro. Precise projected 2D bounding boxes and discrete rotation labels are recorded.
• Camera-Centric Pipeline
  • ~10,000 high-fidelity static scenes (indoor, outdoor).
  • Salient objects identified per scene; pose sampled uniformly in yaw $z_0$7, pitch $z_0$8, and distance $z_0$9.
  • YOLOv10 ensures focus object is visible, QwenVL-30B filters unnatural/penetrating meshes.
  • Paired samples $t$0 with derived relational pose and corresponding language instructions.
  • Dataset maintains balanced splits across camera subtasks and uniform category distribution.

3. Training Objectives and Losses

SpatialEdit-16B employs a standard diffusion reconstruction objective with regularization to stabilize LoRA training:

$t$1

Low-rank adapter training is subject to additional regularization:

$t$2

Geometry-specific tokens condition the sampling process, obviating the need for explicit geometric-fidelity penalization. Rank=16 adapters with the 500K instance scale achieve the optimal trade-off between spatial precision and background preservation; lower ranks underfit spatial transformations, while higher ranks induce background drift.

4. SpatialEdit-Bench: Geometry-Aware Evaluation Metrics

SpatialEdit-Bench is designed for objective measurement of spatial editing quality with paired ground truth and geometry-aware analysis:

• Object-Level Metrics
  • Moving Score (MS): $t$3, blending IoU of detected/predicted bboxes (via YOLO) and VLM-based “object consistency”.
  • Rotation Score (RS): $t$4, encoding VLM-based “view correctness” and “appearance consistency.”
  • Object overall: $t$5.
• Camera-Level Metrics
  • Viewpoint Error (VE) via VGGT pose recon: Combines translation and rotation error:

  $t$6 - Framing Error (FE): Angular error of object rays plus a zoom-direction classification indicator:

  $t$7

  $t$8 - Lower VE and FE imply superior geometric accuracy.

A correlation study demonstrates Spearman coefficients of 0.932 (VE) and 0.659 (FE), far exceeding VLM-only approaches (GPT4.1 = 0.445), thus validating these geometry-anchored metrics.

5. Results, Benchmarks, and Comparative Analysis

On SpatialEdit-Bench, SpatialEdit-16B achieves substantial improvements over previous state-of-the-art in fine-grained spatial editing. Key results from Table 1 (Xiao et al., 6 Apr 2026):

Method	Move ↑	Rot ↑	ViewErr ↓	FramErr ↓	ObjOv ↑	CamOvErr ↓
LongCatImage-Edit	0.373	0.505	0.802	0.684	0.439	0.743
SpatialEdit (ours)	0.673	0.632	0.243	0.527	0.653	0.385

Object moving precision increases from 0.373 to 0.673, rotation accuracy from 0.505 to 0.632, viewpoint error drops from 0.802 to 0.243, and framing error from 0.684 to 0.527. On the general-editing GEdit-Bench, SpatialEdit-16B remains competitive among open-source models (overall score: 7.52), while yielding marked spatial fidelity improvements.

Qualitative failure analysis indicates prior models exhibit object drift, background hallucination, mis-scaled zoom, and canonical face errors during rotation. SpatialEdit-16B sharply mitigates these issues.

6. Ablation Studies and Reliability Insights

Ablations confirm that joint training on all three spatial sub-tasks (moving, rotation, camera) yields the best aggregate performance:

• Moving only: MS = 0.653
• Rotation only: RS = 0.628
• Camera only: CamErr = 0.395
• Moving + Rotation: (0.657, 0.632)
• Moving + Camera: MS = 0.665, CamErr = 0.402
• All tasks: MS = 0.673, RS = 0.632, CamErr = 0.385 (best overall)

This indicates strong positive inter-task transfer when jointly optimizing for multiple spatial manipulation types.

Metric reliability analysis shows that the geometry-anchored VGGT pose metrics offer greater trustworthiness for camera tasks versus text/image-only baselines, with Spearman correlations of 0.932 for VE, 0.659 for FE, and 0.445 for GPT4.1.

Data scale and LoRA ablations reveal that a 500K image–caption corpus with rank=16 adapters offers the optimal balance between spatial controllability and visual fidelity.

7. Significance and Future Research Directions

SpatialEdit-16B demonstrates, for the first time in the open-source domain, the ability to reliably execute semantically precise instructions—such as “rotate the camera 90°” or “move the chair to that red box”—with explicit metric verification. The model’s architecture, paired dataset, and benchmark establish strong baselines and evaluation methodology for future research on geometry-controllable image editing. A plausible implication is that the foundation established by this work will facilitate advances in both scalable spatial editing models and geometry-grounded evaluation frameworks adapted to more complex and varied real-world scenarios (Xiao et al., 6 Apr 2026).