OpenViGA: Modular Automotive Video Generator

• OpenViGA is an open-source, modular pipeline that integrates an image tokenizer, autoregressive world model, and 3D-CNN video decoder to generate automotive driving videos.
• It leverages pre-trained models fine-tuned on the BDD100K dataset using academic-scale GPUs, ensuring reproducibility and detailed evaluation of design choices.
• The system achieves realistic video generation at 256×256 resolution and 4 fps with one-frame algorithmic latency, validated by improved FID, SSIM, and FVD metrics.

Searching arXiv for the specified paper and closely related work to ground the article. Tool call: arxiv_search({"query":"(Möller et al., 18 Sep 2025)", "max_results": 5, "sort_by":"submittedDate"}) OpenViGA is an end-to-end, open-source video-generation pipeline tailored to automotive driving scenes. It combines three specialist modules—an image tokenizer (TOK), a world model (WM), and a video decoder (VDEC)—each initialized from publicly available pre-trained networks and then fine-tuned on the BDD100K driving dataset under academic-scale GPU budgets. At inference, a short prompt of $T=2$ historic frames is quantized into discrete tokens, the token sequence is extended by an autoregressive transformer that predicts $N=14$ future token blocks, and a lightweight 3D-CNN decoder reconstructs the predicted latents into RGB imagery at $256\times256$ pixels at $4$ fps with one-frame algorithmic latency (Möller et al., 18 Sep 2025).

1. Definition and research context

OpenViGA is presented as a response to several recurrent constraints in automotive video generation: systems that allocate tokenization, future-state prediction, and video decoding to separate models often require significant training resources, provide limited insight into design choices, and do not release code or datasets publicly (Möller et al., 18 Sep 2025). Within that setting, OpenViGA emphasizes an explicitly modular decomposition, separate quantitative and qualitative evaluation of the three subsystems, and a pipeline assembled exclusively from open-source foundations fine-tuned on public data.

The system is positioned against earlier video-generation efforts such as GAIA-1, but its primary distinction is not the rejection of modularity. Rather, it is the attempt to make modularity experimentally inspectable and reproducible. The paper states four core contributions: a deep analysis of the image tokenizer, world model, and video decoder; construction from powerful pre-trained open-source models fine-tuned on BDD100K using GPU hardware at academic scale; streamlining of interfaces across the components; and full reproducibility through public data, released code, and released models (Möller et al., 18 Sep 2025).

A common misunderstanding would be to interpret OpenViGA as a monolithic video generator. That characterization is inaccurate: its central design principle is explicit separation of tokenization, prediction, and decoding. Another possible misunderstanding is to read “one-frame algorithmic latency” as zero-latency forecasting. The reported claim is narrower: the system predicts realistic driving-scene videos frame-by-frame with only one frame of algorithmic latency at the stated resolution and frame rate (Möller et al., 18 Sep 2025).

2. System architecture and interfaces

The architecture consists of three components with fixed roles and carefully defined interfaces.

Module	Base model and scale	Role
TOK	VQGAN; encoder/decoder parameter counts reported as 59 M + 87 M	Quantizes frames into discrete image tokens
WM	Decoder-only transformer from LWM-Chat-1M; 7 B parameters, 32 layers	Autoregressively predicts future image-token indices
VDEC	Shallow 3D-CNN obtained by central inflation of the VQGAN image decoder’s 2D convolutions	Reconstructs RGB frames with temporal context

The image tokenizer operates on frames $x_t \in [-1,1]^{3 \times 256 \times 256}$. Each frame is split into $n=16\times16=256$ patches, and a convolutional encoder $\mathrm{ENC}$ maps each patch to a latent vector $z_{t,v}$ of dimension $d=64$. A learnable codebook $C \in \mathbb{R}^{K \times d}$ with $N=14$0 entries quantizes each latent to the nearest codeword $N=14$1, yielding an index $N=14$2 plus an end-of-image token $N=14$3. The result is a token sequence of length $N=14$4 for a history of $N=14$5 frames (Möller et al., 18 Sep 2025).

The world model is an autoregressive, decoder-only transformer lifted from the open-source LWM-Chat-1M, itself built on LLaMA-2. Its input comprises a fixed text prompt—“Generate a video of driving vehicles”—encoded into $N=14$6 textual tokens $N=14$7, together with the sequence of image-token indices produced by TOK. Using causal attention, the WM predicts the next $N=14$8 image indices one at a time and samples from the top-$N=14$9 most probable codebook entries, with $256\times256$0 tuned in $256\times256$1 (Möller et al., 18 Sep 2025).

The video decoder is a shallow 3D-CNN with temporal context of three frames. At each step it consumes the last two real or transcoded frames and the newly predicted latent code, then produces the next RGB frame $256\times256$2. The paper describes this as a bidirectional temporal decoding design whose three-frame context improves coherence relative to frame-wise decoding (Möller et al., 18 Sep 2025).

These interfaces are central to the system’s modularity. Separating tokenization, world modeling, and decoding simplifies fine-tuning and ablation, while the use of open-source foundations avoids proprietary data and proprietary models. The specific operating point of 256 tokens per frame at 4 fps is motivated in the paper as matching LWM’s original training regime and fitting on an 80 GB GPU (Möller et al., 18 Sep 2025).

3. Fine-tuning regime and data pipeline

All data splits follow BDD100K’s $256\times256$3k/$256\times256$4k/$256\times256$5k train/val/test partition. OpenViGA fine-tunes each module separately, with distinct preprocessing and optimization schedules, and then combines them into a coherent generation pipeline (Möller et al., 18 Sep 2025).

For TOK+DEC fine-tuning, the training source is $256\times256$6, reported as approximately $256\times256$7k images sampled at $256\times256$8 fps. Preprocessing consists of downscaling from $256\times256$9 to $4$0, center-cropping to $4$1, and normalizing to $4$2. Training uses batch size $4$3 on $4$4NVIDIA H100 for $4$5k steps total, with $4$6k warm-up to learning rate $4$7 followed by $4$8k cosine decay to $4$9. Adversarial training with a patch-GAN is activated after $x_t \in [-1,1]^{3 \times 256 \times 256}$0k steps, and discriminator and generator are alternated every batch. The reported wall-clock time is approximately $x_t \in [-1,1]^{3 \times 256 \times 256}$1 hours (Möller et al., 18 Sep 2025).

For WM fine-tuning, the source is $x_t \in [-1,1]^{3 \times 256 \times 256}$2, described as $x_t \in [-1,1]^{3 \times 256 \times 256}$3k videos subsampled to $x_t \in [-1,1]^{3 \times 256 \times 256}$4 fps. The batch size is $x_t \in [-1,1]^{3 \times 256 \times 256}$5, where each sample contains tokenized representations for $x_t \in [-1,1]^{3 \times 256 \times 256}$6 initial frames and $x_t \in [-1,1]^{3 \times 256 \times 256}$7 future frames. Training runs for $x_t \in [-1,1]^{3 \times 256 \times 256}$8k steps, with $x_t \in [-1,1]^{3 \times 256 \times 256}$9 warm-up steps to learning rate $n=16\times16=256$0 and $n=16\times16=256$1k-step cosine decay to $n=16\times16=256$2. The world model uses LoRA adapters for all Linear and Embedding layers; only RMS-Norm layers are fully fine-tuned; and $n=16\times16=256$3 of weights remain frozen in bfloat16. The paper separately summarizes this as fine-tuning only $n=16\times16=256$4 of WM parameters. Total wall-clock time is approximately $n=16\times16=256$5 hours on $n=16\times16=256$6H100 (Möller et al., 18 Sep 2025).

For VDEC fine-tuning, the source is again $n=16\times16=256$7. Training uses batch size $n=16\times16=256$8 for $n=16\times16=256$9k steps, with $\mathrm{ENC}$0-step warm-up to learning rate $\mathrm{ENC}$1 and decay to $\mathrm{ENC}$2. Reconstruction plus 3D GAN loss is enabled after $\mathrm{ENC}$3k steps. The reported wall-clock time is approximately $\mathrm{ENC}$4 hours on $\mathrm{ENC}$5H100 (Möller et al., 18 Sep 2025).

This training strategy indicates a deliberate division of labor: image fidelity is addressed first through tokenizer and decoder adaptation, sequential prediction is adapted through low-rank updates to a large language-model-derived transformer, and temporal coherence is then refined in a lightweight video decoder. A plausible implication is that the architecture seeks to preserve the prior capabilities of the pre-trained bases while reducing the cost of domain transfer to automotive scenes.

4. Objectives and optimization criteria

The tokenizer-decoder stage minimizes a weighted sum of reconstruction, codebook, self-supervised distillation, and adversarial objectives:

$\mathrm{ENC}$6

The reconstruction term is

$\mathrm{ENC}$7

The codebook loss is

$\mathrm{ENC}$8

The self-supervised distillation term is

$\mathrm{ENC}$9

The generator loss is

$z_{t,v}$0

and the discriminator loss is

$z_{t,v}$1

These are the objectives reported for TOK+DEC fine-tuning (Möller et al., 18 Sep 2025).

The world model is optimized with next-token cross-entropy:

$z_{t,v}$2

This aligns the WM precisely with the discrete token interface exported by the tokenizer and conditions prediction jointly on the fixed text cue and prior image tokens (Möller et al., 18 Sep 2025).

VDEC reuses $z_{t,v}$3 and adds a 3D-GAN generator loss $z_{t,v}$4 with a matching discriminator loss. The role of this stage is not token prediction but temporally aware latent-to-RGB reconstruction. The paper’s ablations attribute a substantial fraction of video-quality gain to this decoding stage rather than to the world model alone (Möller et al., 18 Sep 2025).

5. Quantitative performance and ablations

The reported operating point is $z_{t,v}$5 pixels at $z_{t,v}$6 fps with one-frame algorithmic latency per prediction on an A100 GPU (Möller et al., 18 Sep 2025). This claim defines the deployment-level target of the system more precisely than a generic “real-time” label would.

For image tokenization and reconstruction, the best TOK+DEC setting is specified as $z_{t,v}$7, $z_{t,v}$8 with the paper’s discriminator. Under that configuration, the reported metrics on BDD100K images are PSNR $z_{t,v}$9 dB, SSIM $d=64$0, LPIPS $d=64$1, FID $d=64$2, and CMMD $d=64$3, compared with unfine-tuned FID $d=64$4 (Möller et al., 18 Sep 2025). The paper also summarizes this outcome as roughly halving the FID relative to the unfine-tuned VQGAN.

For future prediction, the top-$d=64$5 sampling ablation is conducted on $d=64$6-frame continuations. The setting $d=64$7 gives $d=64$8, $d=64$9, and $C \in \mathbb{R}^{K \times d}$0, whereas $C \in \mathbb{R}^{K \times d}$1 yields $C \in \mathbb{R}^{K \times d}$2. The paper characterizes this as a $C \in \mathbb{R}^{K \times d}$3 FVD improvement over $C \in \mathbb{R}^{K \times d}$4 (Möller et al., 18 Sep 2025). This ablation makes explicit that generation quality depends materially on the token-sampling regime, not only on the underlying transformer weights.

For video decoding, replacing the original image decoder with VDEC reduces $C \in \mathbb{R}^{K \times d}$5 from $C \in \mathbb{R}^{K \times d}$6 to $C \in \mathbb{R}^{K \times d}$7 (Möller et al., 18 Sep 2025). The paper interprets this as evidence for the benefit of three-frame 3D context. In qualitative samples, $C \in \mathbb{R}^{K \times d}$8–$C \in \mathbb{R}^{K \times d}$9 second future continuations are reported to exhibit realistic vehicle trajectories and scene consistency, with video galleries hosted in the repository (Möller et al., 18 Sep 2025).

The aggregate picture is that OpenViGA’s performance derives from cumulative gains across all three modules. A plausible implication is that the paper’s main empirical argument is not that a single component dominates, but that streamlined interfaces plus targeted fine-tuning across tokenizer, predictor, and decoder are jointly necessary for credible automotive video continuation.

6. Reproducibility, release, and significance

All training and inference code, data-preprocessing scripts, fine-tuned model checkpoints for TOK, WM-LoRA adapters, and VDEC, and evaluation tools are released under an MIT license at the project repository identified in the paper as version 1.0. Detailed READMEs enumerate exact commands, hyperparameters, and environment specifications for full reproducibility (Möller et al., 18 Sep 2025).

The paper frames reproducibility as a first-order contribution rather than an auxiliary artifact. Because the underlying models are public, the data are public, and the hyperparameters and scripts are published, the authors state that anyone can reproduce the end-to-end results on academic hardware (Möller et al., 18 Sep 2025). In the context of automotive video generation, where closed datasets and unavailable model weights have often limited scrutiny, this is a consequential methodological claim.

OpenViGA therefore occupies a specific place within the literature: it is an open, modular, and reproducible implementation of driving-scene video generation that uses pre-trained open-source models, public BDD100K data, academic-scale GPU budgets, and separate evaluation of tokenization, world modeling, and temporal decoding. This suggests that its significance lies as much in making the design space inspectable as in the particular numerical results it reports (Möller et al., 18 Sep 2025).