---
title: 'Seedance 1.5 Pro: Joint AV Generation'
url: https://www.emergentmind.com/topics/seedance-1-5-pro
type: topic
---

# Seedance 1.5 Pro: Joint AV Generation

Seedance 1.5 Pro is a foundational model for native, joint audio-video generation, utilizing a dual-branch Diffusion Transformer architecture and a cross-modal joint module. It is designed to deliver high-quality, temporally-synchronized audiovisual sequences, establishing state-of-the-art results on audio-visual generation benchmarks. Key features include multilingual and dialect-specific lip-sync, dynamic cinematic camera control, and enhanced narrative coherence. Seedance 1.5 Pro is accessible through a production-ready API on Volcano Engine [2512.13507].

## 1. Model Architecture: Dual-Branch Diffusion Transformer and Cross-Modal Joint Module

Seedance 1.5 Pro features a dual-branch Diffusion Transformer, consisting of parallel video and audio branches integrated by a cross-modal joint module.

- **Video Branch:** Accepts sequences of visual tokens (e.g., patch embeddings per frame), processed through a stack of Transformer blocks adapted for the diffusion reverse process. At diffusion timestep $t$, the branch predicts denoised embeddings $x_{v_t}$.
- **Audio Branch:** Consumes mel-spectrogram tokens (80-dimensional features per audio frame), with a similarly parameterized but independently weighted Transformer stack; outputs denoised audio embedding $a_t$ at each $t$.
- **Cross-Modal Joint Module:** Interleaved at designated layers, implementing bidirectional cross-attention. Video queries attend to audio keys/values and vice versa, with modality-type embeddings ($e^{vid}$, $e^{aud}$) injected into both token and timestep embeddings to signal origin.

The architectural fusion is structured as:

| Component   | Input Tokens                 | Core Mechanism                   |
|-------------|-----------------------------|----------------------------------|
| Video Branch| Visual patches per frame    | Diffusion + Transformer layers   |
| Audio Branch| Mel-spectrogram tokens      | Diffusion + Transformer layers   |
| Joint Module| Tokens from both modalities | Bidirectional cross-attention    |

This explicit cross-modal design underpins the model's high-fidelity audio-video synchronization, enabling emergent capabilities such as accurate lip sync and dynamic audio-driven scene motion.

## 2. Diffusion Model Formulation and Loss Objectives

The model leverages a denoising diffusion probabilistic model (DDPM) backbone. While standard DDPM equations are invoked, explicit formulas are not provided beyond those common in the literature.

- **Forward process:** Gaussian noise is added via
  $$q(x_t | x_{t-1}) = \mathcal{N}(x_t; \sqrt{1 - \beta_t} x_{t-1}, \beta_t I)$$
- **Reverse denoising:** Approximated by the learned model
  $$p_\theta(x_{t-1}|x_t) = \mathcal{N}(x_{t-1}; \mu_\theta(x_t, t), \sigma_t^2 I)$$

- **Primary loss:** Mean-squared error (MSE) on the predicted noise
  $$
  \mathcal{L}_{\text{simple}} = \mathbb{E}_{x_0, \epsilon \sim \mathcal{N}(0,I),\,t} \left\| \epsilon - \epsilon_\theta\left( \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1-\bar\alpha_t}\,\epsilon,\, t \right) \right\|_2^2
  $$
- **Variational lower bound (optional):**
  $$
  \mathcal{L}_{\text{VLB}} = \sum_{t=1}^T D_{KL}( q(x_{t-1}|x_t,x_0) \| p_\theta(x_{t-1}|x_t))
  $$

- **Cross-modal alignment loss:** Auxiliary MSE on paired audio-video embedding alignment:
  $$
  \mathcal{L}_{\text{align}} = \mathbb{E}_{t} \left\| f_{\text{vid}}(x_t, t) - f_{\text{aud}}(a_t, t) \right\|_2^2
  $$

This multimodal formulation is critical to enforce synchronization and mutual information flow between the audio and visual streams [2512.13507].

## 3. Data Pipeline, Tokenization, and Training Regime

Seedance 1.5 Pro is trained on a multi-stage pipeline designed to maximize both data diversity and alignment:

1. **Stage 1:** Curation of ~100 million minutes of in-the-wild audio-video clips.
2. **Stage 2:** Automated filtering for audiovisual sync and segment quality.
3. **Stage 3:** Generation of detailed captions—comprehensively describing both visual and audio content—using an advanced captioning system.
4. **Stage 4:** Curriculum learning schedule, starting from short, simple sequences ($\leq$16s), progressing to more complex, longer clips.

**Tokenization and preprocessing** steps:

| Modality | Tokenization Method            | Preprocessing                 |
|----------|-------------------------------|-------------------------------|
| Video    | 16×16 frame patch embeddings  | Linear projection             |
| Audio    | 80-band mel-spectrogram tokens| Fixed-length segmentation     |
| Text     | Byte-Pair Encoding (BPE)      | Subword tokenization          |

This pipeline ensures high coverage of real-world data and robust prompt-conditioned generative capabilities.

## 4. Post-Training Optimization: Supervised Fine-Tuning and RLHF

After initial pretraining, Seedance 1.5 Pro undergoes post-training optimization via supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF).

- **SFT:** Performed on proprietary, professional-grade audio-visual and cinematic datasets, driving alignment on prompt-to-output correspondence and overall quality.
  - Training continues from pretrained weights, introducing text-prompt reconstruction loss on both modalities.
  - Paper reports: learning rate $1 \times 10^{-5}$, batch size 128, $\sim$100 K steps. Results in +0.4 Likert increase in prompt adherence and +0.3 in motion vividness (SeedVideoBench 1.5).
- **RLHF:** Employs a small multi-layer Transformer reward model, evaluating:
  1. Motion quality
  2. Audio fidelity
  3. AV synchronization
  4. Aesthetic appeal

  Policy optimization uses Proximal Policy Optimization (PPO) with a clip ratio of 0.2:
  $$
  \mathcal{L}_{\text{PPO}} = -\mathbb{E}\left[ \min(r_t(\theta)\,A_t,\; \mathrm{clip}(r_t(\theta),1-\epsilon,1+\epsilon)\,A_t) \right],\quad r_t(\theta)=\frac{\pi_\theta}{\pi_{\theta_{\text{old}}}}
  $$
  Targeted infrastructure optimizations yield a 3× reduction in RLHF wallclock time versus earlier pipelines.

## 5. Inference Acceleration and Production Readiness

Seedance 1.5 Pro incorporates a dedicated inference acceleration framework:

- **Multi-Stage Distillation (HyperSD-style):** A teacher model generates with a full 1,000-step DDPM sampler; a student model is distilled to 30–50 steps with consistency constraints.
- **Additional optimizations:**
  - Mixed-precision (FP16) inference
  - INT8 quantization for key matrix multiplications
  - Pipeline parallelism across GPUs and TPUs

**Results:** Over 10× reduction in Number of Function Evaluations (NFE), with $<1$\% loss in human-rated quality metrics. Example latency is $\sim$1.2s for 512×512×32-frame videos on a single A100 (30-step student), compared to $\sim$12s for the teacher model.

## 6. Quantitative Performance, Capabilities, and Deployment

**Performance:**

| Metric                        | Seedance 1.5 Pro | Veo 3.1 | Kling 2.6 |
|-------------------------------|:---------------:|:-------:|:---------:|
| AV sync: speech↔lip (±40 ms)  | 95%             | 88%     | 90%       |
| Text-to-Video (Likert 1–5)    | 4.2             | 3.7     | 3.9       |
| Image-to-Video (Likert 1–5)   | 4.1             | 3.6     | –         |
| Inference Latency (512×512×32)| 1.2s            | –       | –         |

**Notable Capabilities:**
- Native lip-sync for Mandarin, Cantonese, Sichuanese, Shanghainese, Taiwanese, with tonal accuracy.
- Cinematic camera control: automated dolly zooms, tracking and orbital shots, continuous takes; color grading during generation.
- Narrative coherence: strong long-range scene understanding and multi-shot sequencing.

**Deployment:**
- Model size: ~$4.5$ billion parameters.
- Recommended hardware: 8$\times$A100 40GB or 4$\times$H100 80GB for real-time operation.
- Accessible via Volcano Engine API, with native endpoints for text-to-video-audio (T2VA) and image-to-video-audio (I2VA) generation modes.

**Sample Code (Python):**
```python
from volcengine import Client
cli = Client(key=…, secret=…)
resp = cli.generate(
    model_id="Doubao-Seedance-1.5-pro",
    prompt="A rainy street scene in Paris, with soft piano music",
    mode="T2VA",
    length_s=8,
)
video, audio = resp["video"], resp["audio"]
```

Seedance 1.5 Pro represents a natively joint audio-video generation engine, delivering production-grade synchronous outputs with scalable, high-throughput deployment [2512.13507].

Source: https://www.emergentmind.com/topics/seedance-1-5-pro