DreaMontage: One-Shot Video Generation

• DreaMontage is a video generation framework that synthesizes long-duration one-shot videos from arbitrary frames and clips with natural language guidance.
• It integrates multi-stage diffusion with adaptive learning and memory-efficient techniques to ensure seamless cinematic transitions.
• The framework employs shared-rotary positional embeddings and segment-wise autoregressive inference to maintain temporal coherence and high visual quality.

DreaMontage is a video generation framework that synthesizes visually coherent, long-duration one-shot videos from arbitrary user-provided frames and clips, optionally guided by natural language prompts. It addresses inherent challenges in virtual "one-shot" filmmaking—such as prohibitive costs and operational constraints of real long-take shooting—by delivering seamless cinematic transitions and robust temporal control, surpassing naïve clip concatenation approaches which typically induce discontinuities and coherence loss. DreaMontage integrates arbitrarily spaced conditioning anchors, leverages multi-stage diffusion with adaptive learning, and introduces memory-efficient generative techniques to transform fragmented visual material into unified, expressive cinematics (Liu et al., 24 Dec 2025).

1. Arbitrary-Frame-Guided Video Generation: Problem Formulation

The task is to generate a single, continuous latent video $x \in \mathbb{R}^{T \times H \times W \times 3}$ over duration $T$, given a sequence of $m$ conditioning signals $C = \{(c_1,t_1), \dots, (c_m, t_m)\}$ where $c_i$ represents static frames or clips anchored at timestamps $t_i \in [0,T]$, and optional prompt $p$. The objective is $x(t_i) \approx c_i$ and, critically, uninterrupted semantic and visual transitions between anchors. Baseline techniques concatenating video segments generated via first- and last-frame diffusion suffer from:

• Visual discontinuities: abrupt changes (cuts) at junctions,
• Temporal incoherence: flicker, color mismatches, and semantic drift,
• Motion unreality: failure to respect rational subject or camera motion across anchor transitions.

The need for a unified, arbitrarily anchored generative solution motivates the framework.

2. Model Architecture and Conditioning Mechanisms

DreaMontage architecture is constructed atop the Seedance 1.0 DiT-based video diffusion backbone, comprising:

• Base DiT: produces low-resolution 3D latent at $480p$,
• Super-resolution DiT (SR): upscales to $720p$ or $1080p$.

Intermediate-Condition Injection

At each denoising step $t$, noise prediction is parameterized by:

$$\hat{\epsilon} = \epsilon_\theta(z_t;\, p,\, \Phi(C, t))$$

where $\Phi(C, t)$ involves channel-wise concatenation of VAE latents for anchors with $t_i \leq t$, complemented by a sequence-wise Shared-RoPE (rotary positional embedding) in SR, aligning token positional encodings for conditioning fidelity. The DiT denoising update is:

$$x_{t-1} = D(x_t; \Phi(C, t)) + \sigma_t \hat{\epsilon}$$

with $\sigma_t$ the schedule noise.

Shared-RoPE Mechanism

In SR, anchor latents append to the token sequence's tail. Target frame RoPE are reused to lock positional encoding, precluding drift from base conditioning.

3. Adaptive Tuning and Training Objective

Given conventional DiT training only at start/end frames, intermediate conditioning poses latent misalignment. Adaptive tuning rectifies this via:

• One-Shot Data Filtering: Excludes multi-shot clips, maximizes first–last frame variation (low CLIP similarity), high Q-Align aesthetic, strong optical flow, definite human pose (RTMPose), with densely captioned action segments.
• Approximate Re-Encoding: Random anchor selection at action boundaries; single-image anchors encoded via 3D VAE, multi-frame anchors encode only initial frame, with subsequent noise resampling to break causality.
• Objective Function:

$$\mathcal{L}_{\text{adapt}} = \mathbb{E}_{x_0,p,\,C,\,t,\,\epsilon \sim \mathcal{N}(0,I)} \|\epsilon - \epsilon_\theta(z_t; p, \Phi(C, t))\|^2$$

Training over $\sim 30$K steps with $300$K clips enables robust respect for arbitrary intermediate anchors.

4. Cinematic SFT and Tailored DPO for Expressiveness

Adaptive tuning is supplemented with additional supervised fine-tuning (SFT) targeting cinematic expressiveness, using a $1\,000$-video, five-class dataset comprising:

• Camera shots (dolly, first-person),
• Visual effects (light trails, morphing),
• Sports (dynamic action),
• Spatial perception (depth, zoom),
• Advanced transitions (match-cuts, whip pans).

Fine-tuning with random anchor sampling over $15$K steps strengthens motion and prompt fidelity.

Direct Preference Optimization (DPO)

Deficiencies of abrupt cuts and irrational motion are resolved using DPO. The dual pipelines involve:

• Abrupt-Cut Pipeline: $10$K short clips, categorizing cut severity (5 levels via GPT-4o/human), leveraging VLM discriminator $D_{\text{cut}}$ to select best/worst candidates for optimization.
• Subject-Motion Pipeline: Defines subjects/actions, generates prompts via T2I and VLM, human evaluation of motion realism.

The DPO objective is:

$$\mathcal{L}_{\text{DPO}} = -\mathbb{E}_{(c, v_w, v_l) \sim \mathcal{D}} \log\sigma \Bigl( \beta \Bigl[ \log\frac{\pi_\theta(v_w \mid c)}{\pi_{\rm ref}(v_w \mid c)} - \log\frac{\pi_\theta(v_l \mid c)}{\pi_{\rm ref}(v_l \mid c)} \Bigr] \Bigr)$$

with $\sigma$ the sigmoid and $\beta$ the reference trust parameter. Iterative optimization ($\sim 10$K steps) ensures strong cut/motion avoidance.

5. Segment-wise Auto-Regressive Inference

To enable arbitrarily long, memory-efficient generation, a segment-wise auto-regressive (SAR) strategy is deployed, as illustrated in the following pseudocode:

Inputs: Conditions C={(c_i,t_i)}, prompt p, max segment length L_max Initialize prev_tail = none, t=0 While t < T_total: 1. next boundary t' = min{ t_i > t } or t+L_max 2. C_local = {c_i | t ≤ t_i ≤ t'} 3. prefix latent τ(prev_tail) if exists 4. s = GenerateSegment( τ(prev_tail), C_local, p ) 5. append s (overlap one frame) to output 6. prev_tail = last K frames of s 7. t = t' Decode full latent sequence via VAE decoder → RGB video

For the $n$th segment:

$$s_n = \mathcal{G}_\theta\left( \tau(s_{n-1}),\,\mathcal{C}_n \right)$$

where $\tau(\cdot)$ extracts last $K$ latent frames, and $\mathcal{C}_n$ are time-local anchors. This enables extended sequence synthesis while controlling latent drift.

6. Evaluation: Qualitative and Quantitative Results

Qualitative demonstrations encompass scenarios such as multi-keyframe match-cuts ("train interior → cyberpunk city"), progressive zooms ("eye close-up → meadow"), action transitions ("skiing → surfing"), animal extensions ("cat on bike → horse jump"), and hybrid image/video metamorphosis.

Quantitative evaluation utilizes GSB (Gain-Score-Balance), calculated as $$(\text{Wins}-\text{Losses})/(\text{Wins}+\text{Losses}+\text{Ties})$$, comparing DreaMontage against state-of-the-art baselines in two modes:

Comparative GSB Scores

Comparison	Visual Quality	Motion Effects	Prompt Following	Overall
SFT vs. Base	0.00 %	+24.58 %	+5.93 %	+20.34 %
DPO (Abrupt Cuts)	–	+12.59 %	–	+12.59 %
DPO (Subject Motion)	+13.44 %	–	–	+13.44 %
Shared-RoPE vs. SR Base	+53.55 %	–	–	+53.55 %

GSB-based mode comparison yields Multi-Keyframe: +15.79% vs Vidu Q2, +28.95% vs Pixverse V5; First-Last: motion +4.64%, prompt +4.64%, overall +3.97% vs Kling 2.5.

7. Limitations and Prospects

Limitations include residual artifacts post-latent resampling under extreme anchor semantic jumps, SAR-induced drift in sequences exceeding one minute, and computational expense for HD, long-duration outputs.

Future work may focus on:

• Integration of physics simulators for exact motion,
• Unified long-context training (e.g., temporally extended transformers),
• Real-time latent space editing for interactive neural filmmaking,
• Compression-optimized VAE architectures for reduced memory/faster inference.

DreaMontage constitutes the first end-to-end framework for arbitrary-frame, one-shot video synthesis, combining lightweight conditioning modules, adaptive data-driven tuning, cinematic supervised learning, direct preference alignment, and auto-regressive segment inference, thereby enabling seamless, expressive, and user-controllable long-take video generation (Liu et al., 24 Dec 2025).