--- title: 'OpenViGA: Modular Automotive Video Generator' url: https://www.emergentmind.com/topics/openviga type: topic --- # OpenViGA: Modular Automotive Video Generator Searching arXiv for the specified paper and closely related work to ground the article. Tool call: arxiv_search({"query":"2509.15479", "max_results": 5, "sort_by":"submittedDate"}) OpenViGA is an end-to-end, open-source video-generation pipeline tailored to automotive driving scenes. It combines three specialist modules—an image tokenizer (TOK), a world model (WM), and a video decoder (VDEC)—each initialized from publicly available pre-trained networks and then fine-tuned on the BDD100K driving dataset under academic-scale GPU budgets. At inference, a short prompt of $T=2$ historic frames is quantized into discrete tokens, the token sequence is extended by an autoregressive transformer that predicts $N=14$ future token blocks, and a lightweight 3D-CNN decoder reconstructs the predicted latents into RGB imagery at $256\times256$ pixels at $4$ fps with one-frame algorithmic latency [2509.15479]. ## 1. Definition and research context OpenViGA is presented as a response to several recurrent constraints in automotive video generation: systems that allocate tokenization, future-state prediction, and video decoding to separate models often require significant training resources, provide limited insight into design choices, and do not release code or datasets publicly [2509.15479]. Within that setting, OpenViGA emphasizes an explicitly modular decomposition, separate quantitative and qualitative evaluation of the three subsystems, and a pipeline assembled exclusively from open-source foundations fine-tuned on public data. The system is positioned against earlier video-generation efforts such as GAIA-1, but its primary distinction is not the rejection of modularity. Rather, it is the attempt to make modularity experimentally inspectable and reproducible. The paper states four core contributions: a deep analysis of the image tokenizer, world model, and video decoder; construction from powerful pre-trained open-source models fine-tuned on BDD100K using GPU hardware at academic scale; streamlining of interfaces across the components; and full reproducibility through public data, released code, and released models [2509.15479]. A common misunderstanding would be to interpret OpenViGA as a monolithic video generator. That characterization is inaccurate: its central design principle is explicit separation of tokenization, prediction, and decoding. Another possible misunderstanding is to read “one-frame algorithmic latency” as zero-latency forecasting. The reported claim is narrower: the system predicts realistic driving-scene videos frame-by-frame with only one frame of algorithmic latency at the stated resolution and frame rate [2509.15479]. ## 2. System architecture and interfaces The architecture consists of three components with fixed roles and carefully defined interfaces. | Module | Base model and scale | Role | |---|---|---| | TOK | VQGAN; encoder/decoder parameter counts reported as 59 M + 87 M | Quantizes frames into discrete image tokens | | WM | Decoder-only transformer from LWM-Chat-1M; 7 B parameters, 32 layers | Autoregressively predicts future image-token indices | | VDEC | Shallow 3D-CNN obtained by central inflation of the VQGAN image decoder’s 2D convolutions | Reconstructs RGB frames with temporal context | The image tokenizer operates on frames $x_t \in [-1,1]^{3 \times 256 \times 256}$. Each frame is split into $n=16\times16=256$ patches, and a convolutional encoder $\mathrm{ENC}$ maps each patch to a latent vector $z_{t,v}$ of dimension $d=64$. A learnable codebook $C \in \mathbb{R}^{K \times d}$ with $K=8\,192$ entries quantizes each latent to the nearest codeword $Z_{t,v}$, yielding an index $k_{t,v}\in\{0,\dots,K-1\}$ plus an end-of-image token $k=0$. The result is a token sequence of length $T\cdot(n+1)$ for a history of $T$ frames [2509.15479]. The world model is an autoregressive, decoder-only transformer lifted from the open-source LWM-Chat-1M, itself built on LLaMA-2. Its input comprises a fixed text prompt—“Generate a video of driving vehicles”—encoded into $M$ textual tokens $c_1,\dots,c_M$, together with the sequence of image-token indices produced by TOK. Using causal attention, the WM predicts the next $N\cdot(n+1)$ image indices one at a time and samples from the top-$k$ most probable codebook entries, with $k$ tuned in $[1,1000]$ [2509.15479]. The video decoder is a shallow 3D-CNN with temporal context of three frames. At each step it consumes the last two real or transcoded frames and the newly predicted latent code, then produces the next RGB frame $\hat{y}_t$. The paper describes this as a bidirectional temporal decoding design whose three-frame context improves coherence relative to frame-wise decoding [2509.15479]. These interfaces are central to the system’s modularity. Separating tokenization, world modeling, and decoding simplifies fine-tuning and ablation, while the use of open-source foundations avoids proprietary data and proprietary models. The specific operating point of 256 tokens per frame at 4 fps is motivated in the paper as matching LWM’s original training regime and fitting on an 80 GB GPU [2509.15479]. ## 3. Fine-tuning regime and data pipeline All data splits follow BDD100K’s $70$k/$10$k/$20$k train/val/test partition. OpenViGA fine-tunes each module separately, with distinct preprocessing and optimization schedules, and then combines them into a coherent generation pipeline [2509.15479]. For TOK+DEC fine-tuning, the training source is $D^{\mathrm{train}}_{\mathrm{BDDimg}/5s}$, reported as approximately $538$k images sampled at $0.2$ fps. Preprocessing consists of downscaling from $1280\times720$ to $640\times360$, center-cropping to $256\times256$, and normalizing to $[-1,1]$. Training uses batch size $80$ on $4\times$NVIDIA H100 for $200$k steps total, with $2$k warm-up to learning rate $5\times10^{-5}$ followed by $150$k cosine decay to $5\times10^{-7}$. Adversarial training with a patch-GAN is activated after $20$k steps, and discriminator and generator are alternated every batch. The reported wall-clock time is approximately $24$ hours [2509.15479]. For WM fine-tuning, the source is $D^{\mathrm{train}\text{-}70k}_{\mathrm{BDDvid}\text{-}4fps}$, described as $70$k videos subsampled to $4$ fps. The batch size is $24$, where each sample contains tokenized representations for $2$ initial frames and $14$ future frames. Training runs for $28.3$k steps, with $250$ warm-up steps to learning rate $6\times10^{-4}$ and $15$k-step cosine decay to $6\times10^{-5}$. The world model uses LoRA adapters for all Linear and Embedding layers; only RMS-Norm layers are fully fine-tuned; and $98\%$ of weights remain frozen in bfloat16. The paper separately summarizes this as fine-tuning only $2.4\%$ of WM parameters. Total wall-clock time is approximately $65$ hours on $4\times$H100 [2509.15479]. For VDEC fine-tuning, the source is again $D^{\mathrm{train}\text{-}70k}_{\mathrm{BDDvid}\text{-}4fps}$. Training uses batch size $48$ for $100$k steps, with $100$-step warm-up to learning rate $5\times10^{-5}$ and decay to $5\times10^{-7}$. Reconstruction plus 3D GAN loss is enabled after $2$k steps. The reported wall-clock time is approximately $65$ hours on $4\times$H100 [2509.15479]. This training strategy indicates a deliberate division of labor: image fidelity is addressed first through tokenizer and decoder adaptation, sequential prediction is adapted through low-rank updates to a large language-model-derived transformer, and temporal coherence is then refined in a lightweight video decoder. A plausible implication is that the architecture seeks to preserve the prior capabilities of the pre-trained bases while reducing the cost of domain transfer to automotive scenes. ## 4. Objectives and optimization criteria The tokenizer-decoder stage minimizes a weighted sum of reconstruction, codebook, self-supervised distillation, and adversarial objectives: $$ J_{\mathrm{total}} = J_{\mathrm{rec}} + \lambda_{\mathrm{CB}} J_{\mathrm{CB}} + \lambda_{\mathrm{SSL}} J_{\mathrm{SSL}} + \lambda_G J_G. $$ The reconstruction term is $$ J_{\mathrm{rec}}(\hat{x},x)=\lambda_1\lVert \hat{x}-x\rVert_1+\lambda_2\lVert \hat{x}-x\rVert_2^2+\lambda_p\sum_\ell \lVert \phi_\ell(\hat{x})-\phi_\ell(x)\rVert_2^2. $$ The codebook loss is $$ J_{\mathrm{CB}}=\lVert \mathrm{sg}(z)-z\rVert_2^2+\beta\lVert z-\mathrm{sg}(z)\rVert_2^2,\qquad \beta=0.25. $$ The self-supervised distillation term is $$ J_{\mathrm{SSL}}(z,z^T)=1-\max\left(\frac{z\cdot z^T}{\lVert z\rVert\cdot \lVert z^T\rVert},\epsilon\right). $$ The generator loss is $$ J_G=-D(\hat{x}), $$ and the discriminator loss is $$ J_D=\frac12\left[\mathrm{ReLU}(1-D(x))+\mathrm{ReLU}(1+D(\hat{x}))\right]. $$ These are the objectives reported for TOK+DEC fine-tuning [2509.15479]. The world model is optimized with next-token cross-entropy: $$ J_{\mathrm{CE}}=-\sum_{i=1}^{(T+N)(n+1)} \log P(k_i \mid c, k_{