Vidarc: Diffusion Models in 4D Reconstruction & Control

• Vidarc is a dual-framework approach that applies diffusion models for both high-fidelity 4D scene reconstruction and real-time robotic control.
• ViDAR utilizes monocular video diffusion with pseudo-multi-view supervision to improve dynamic novel view synthesis, achieving over 1 dB improvement in PSNR-D.
• Vidarc’s autoregressive diffusion model with action masking enables closed-loop control, reducing inference latency by 91% and ensuring robust error recovery.

Vidarc refers to two distinct but state-of-the-art frameworks in vision-based artificial intelligence: (1) Video Diffusion-Aware 4D Reconstruction (ViDAR), a monocular video-based 4D geometry and appearance reconstruction methodology, and (2) Vidarc: Embodied Video Diffusion Model for Closed-loop Control, an autoregressive video diffusion model for real-time robotic manipulation. Both approaches leverage diffusion models, but target different application domains—scene reconstruction and robot control, respectively (Nazarczuk et al., 23 Jun 2025, Feng et al., 19 Dec 2025).

1. Monocular Video Diffusion-Aware 4D Reconstruction (ViDAR)

ViDAR addresses the ill-posed nature of dynamic novel-view synthesis from single-camera videos, where both structure-from-motion ambiguity and data scarcity limit faithful 4D reconstructions. The approach introduces a pipeline that synthesizes pseudo-multi-view supervision using a scene-personalized video diffusion model, then integrates these views in a Gaussian splatting-based 4D reconstruction with explicit spatio-temporal consistency terms (Nazarczuk et al., 23 Jun 2025).

Key challenges in this domain include disentangling depth from motion in monocular inputs and reconstructing motion-rich areas with minimal direct supervision. ViDAR generates high-fidelity pseudo-observations using a DreamBooth-fine-tuned Stable Diffusion XL model, providing novel, scene-specific views for supervising dynamic region reconstructions.

2. Embodied Video Diffusion for Closed-loop Robot Control (Vidarc)

In the context of robotics, Vidarc formulates closed-loop manipulation as autoregressive video diffusion, focusing on predicting actions and future observations grounded on robot embodiment. The main architectural innovation is the combination of a video diffusion generator with a masked inverse dynamics module, yielding fast, action-centric prediction and control (Feng et al., 19 Dec 2025).

At each timestep, the system observes an image, predicts both the next frame (via the video diffusion model $G_\theta$) and the action (via the masked inverse dynamics model $I_\phi$), executes this action in the environment, and receives real feedback for subsequent steps. A learned action-relevant mask, produced by a U-Net-style network, ensures that only robot-relevant visual regions supervise the action regressor. Real-time operation is achieved by key–value caching and chunk-wise autoregressive frame prediction, with environment "re-prefill" allowing efficient error recovery during long sequences.

3. Technical Foundations and Pipeline Architecture

ViDAR's Monocular-to-Gaussian Pipeline

ViDAR's processing follows three principal stages:

• A. Monocular Baseline: Run a monocular 4D reconstruction (e.g., MoSca) to estimate initial time-varying 3D Gaussian parameters $G$ and camera trajectories $C_{inp}$.
• B. Diffusion Enhancement: For each timestep $t$, sample novel views $R_{m,t}$ and enhance them via scene-personalized Stable Diffusion XL, producing pseudo-ground-truths $E_{m,t}$.
• C. Diffusion-aware Reconstruction: Retrain and refine $G$, $C_{inp}$, and the set of sampled camera poses $C_{sample}$ using a composite loss that includes photometric, diffusion-based, and regularization terms. Diffusion supervision is restricted to dynamic image regions, leveraging masks derived from motion tracking.

The principal loss terms are:

$I_\phi$0

where $I_\phi$1 enforces consistency with input views, $I_\phi$2 supervises with diffusion-enhanced pseudo-ground-truths (partitioned into dynamic and camera pose alignment subterms), and $I_\phi$3 regularizes Gaussian parameters and motion smoothness.

Vidarc's Action-centric Autoregressive Diffusion

Vidarc's methodology maintains closed-loop control via the following steps:

1. Framewise Action Masking: For observation $I_\phi$4, generate mask $I_\phi$5 to localize action-informative pixels.
2. Action Regression: Predict action $I_\phi$6.
3. Masked Embodiment-aware Loss: Apply an L2 loss on diffusion model outputs, reweighted by action-mask magnitude.
4. KV Caching & Real-time Feedback: Sequentially predict future frames and actions for a "chunk," execute in environment, then re-initialize cache using ground-truth robot observations to prevent error drift and support rapid inference.

Pre-training leverages one million cross-embodiment episodes, subsequently fine-tuned on specific platforms such as RoboTwin and Aloha, with both simulation and real-world data.

4. Experimental Evaluation and Benchmarks

ViDAR on Dynamic Novel View Synthesis

ViDAR is evaluated on the DyCheck iPhone benchmark, using metrics standard for novel-view synthesis plus new measures (PSNR-D, SSIM-D, LPIPS-D) targeting dynamic regions:

Metric	Static (PSNR-m)	Dynamic (PSNR-D)	Static (SSIM-m)	Dynamic (SSIM-D)	Static (LPIPS-m)	Dynamic (LPIPS-D)
ViDAR	19.69 dB	16.46 dB	0.7126	0.8850	0.2231	0.2793
MoSca	19.32 dB	15.63 dB	0.7060	0.8755	0.2640	0.2904

Dynamic regions such as spinning toys and waving flags demonstrate sharper textures and temporally stable geometry without motion "floaters" or flicker. The one-stage monocular-to-Gaussian pipeline, combined with personalized diffusion-based multi-view augmentation, provides >1 dB PSNR-D improvement over previous methods (Nazarczuk et al., 23 Jun 2025).

Vidarc for Robotic Manipulation

Vidarc is benchmarked on simulated RoboTwin and real-world Aloha tasks:

Method	Sim Success (Avg)	Real Success (Avg)	Inference Latency (s)
Pi0.5	52.9%	41.0%	0.482
Vidar	71.1%	39.0%	34.3
Vidarc	80.7%	56.0%	3.03

Vidarc achieves at least a 15% absolute improvement over Vidar and a 91% reduction in end-to-end inference latency. Re-prefilling prevents compounding prediction errors and enables error-recovery in dynamically changing robot contexts (Feng et al., 19 Dec 2025).

5. Limitations and Prospective Directions

Both frameworks acknowledge fundamental dependencies and open technical questions:

• ViDAR relies on the quality of the initial monocular baseline; diffusion cannot correct severe geometry errors, and DreamBooth-based personalization imposes substantial computational cost. Future work includes joint end-to-end optimization with the diffusion component and multi-object extension.
• Vidarc exhibits remaining computational overhead due to large diffusion models. Ongoing directions involve compression (quantization, distillation), generalization to longer horizons, higher-resolution planning, and tight integration with physical simulators and additional sensory modalities.

6. Contributions and Domain Impact

ViDAR introduces the first monocular-to-Gaussian 4D reconstruction method utilizing personalized video diffusion models for robust pseudo-multi-view learning, establishing new dynamic-region benchmarks in novel view synthesis. Vidarc extends video diffusion into real-time robot control, with unique closed-loop error-correction, mask-based action grounding, and a scalable architecture for cross-platform manipulation.

Both approaches exemplify frontier methods at the intersection of generative modeling, geometric learning, and real-world embodied AI, demonstrating how diffusion models can be specialized—and accelerated—for both visual scene synthesis and high-frequency embodied control (Nazarczuk et al., 23 Jun 2025, Feng et al., 19 Dec 2025).