Seedance: Unified Multimodal Video Generation
- Seedance is a family of video foundation models by ByteDance that spans from text-to-video to unified multimodal generation across audio and video.
- The models employ advanced architectures like temporally-causal VAEs and diffusion transformers to ensure strong prompt following, motion plausibility, and audiovisual synchronization.
- Benchmark evaluations position Seedance as a frontier system alongside models like Sora and Veo, while highlighting challenges in long-horizon compositionality and interactive reasoning.
Searching arXiv for papers on Seedance and closely related evaluations. to=arxiv_search 天天购彩票 娱乐开号json {"query":"Seedance", "max_results": 10, "sort_by":"submittedDate", "sort_order":"descending"}รีเมียร์assistant to=arxiv_search commentary 彩神争霸快 to=arxiv_search ปมถวายสัตย์ฯjson {"query":"Seedance 2.0 Advancing Video Generation for World Complexity", "max_results": 5, "sort_by":"relevance", "sort_order":"descending"} Searching for comparative and benchmarking papers that discuss Seedance in the 2025–2026 literature. to=arxiv_search 天天中彩票怎么json {"query":"AV-Phys Bench Seedance 2.0", "max_results": 10, "sort_by":"submittedDate", "sort_order":"descending"} Seedance is ByteDance’s family of video foundation models, spanning video-only generation in Seedance 1.0, native joint audio–video generation in Seedance 1.5 pro, and a unified multi-modal audio–video system in Seedance 2.0 that accepts text, image, audio, and video references in a single generation run (Gao et al., 10 Jun 2025, Chen et al., 15 Dec 2025, Seedance et al., 15 Apr 2026). In the 2025–2026 literature, Seedance is consistently treated as a proprietary frontier system: survey and benchmarking papers place Seedance 2.0 alongside Sora and Veo as a leading proprietary model, and open-source work uses it as a capability reference point for “omni-capable” or “unified” video generation (Hu et al., 7 Apr 2026, Pan et al., 25 Mar 2026). Across these sources, Seedance is associated with strong prompt following, motion plausibility, multi-shot coherence, multi-modal conditioning, and increasingly broad reference-and-edit workflows, while remaining only partially documented at the implementation level because later reports omit many architectural and training specifics.
1. Product line and scope
Seedance 1.0 is presented as a bilingual (Chinese/English) video foundation model that natively unifies text-to-video and image-to-video, adds true multi-shot generation, and is engineered for fast inference (Gao et al., 10 Jun 2025). Its stated target is the hard triad of prompt following, motion plausibility, and visual quality, together with multi-subject instruction adherence and shot-to-shot narrative coherence. Seedance 1.5 pro extends the line to native joint audio–video generation: it produces temporally synchronized visual motion and accompanying audio in a single, unified generation pass, from either text or image conditions (Chen et al., 15 Dec 2025). Seedance 2.0 is then described as a native, unified multi-modal audio–video model designed for “world complexity,” with direct support for text, image, audio, and video references and synchronized stereo (binaural) audio generation (Seedance et al., 15 Apr 2026).
The progression from 1.0 to 2.0 is marked by expanding modality coverage and control. Seedance 1.0 focuses on unified T2V/I2V, dense captioning, multi-shot structure, RLHF, and inference acceleration (Gao et al., 10 Jun 2025). Seedance 1.5 pro adds a native audio branch, a cross-modal joint module, multilingual and dialect lip-sync, dynamic cinematic camera control, and narrative coherence (Chen et al., 15 Dec 2025). Seedance 2.0 broadens the input space to combinations of text, image, video, and audio, supports direct generation of audio-video content with durations ranging from 4 to 15 seconds at native 480p and 720p, and on its open platform accepts up to 3 video clips, 9 images, and 3 audio clips per request (Seedance et al., 15 Apr 2026).
A recurring point in the literature is that “Seedance” is not a single static model but a family. Open evaluations therefore distinguish among Seedance 1.5, Seedance 1.5 pro, Seedance-Pro 1.0, and Seedance 2.0 depending on the task and release window (Chen et al., 15 Dec 2025, Team, 5 May 2026, Zhang et al., 21 Apr 2026). This distinction matters because results reported for anime, avatar generation, physical commonsense, or multimodal reasoning often target different members of the family rather than a single canonical system.
2. Architectural and training principles
Seedance 1.0 is the most technically disclosed member of the family. It uses a temporally-causal VAE for joint spatial-temporal compression and a diffusion transformer based on MMDiT with decoupled spatial and temporal layers and multimodal RoPE, allowing native support for multi-shot generation and joint T2V/I2V training (Gao et al., 10 Jun 2025). Its VAE maps pixels to latents with downsample ratios and , giving the compression ratio
Training uses flow matching with velocity prediction, progressively increases resolution and frame rate, and later applies fine-grained SFT, video-specific RLHF with three reward models, multi-stage distillation, and system-level optimizations. The report states that Seedance 1.0 can generate a 5-second video at 1080p resolution in 41.4 seconds on an NVIDIA L20 (Gao et al., 10 Jun 2025).
Seedance 1.5 pro introduces a dual-branch Diffusion Transformer architecture grounded in MMDiT, with an audio branch, a video branch, and a cross-modal joint module that enables audio and video streams to attend to each other for temporal alignment and semantic consistency (Chen et al., 15 Dec 2025). The paper attributes practical quality to four pillars: a unified architecture, a comprehensive audio–visual data framework, post-training with SFT and RLHF, and a multi-stage acceleration stack that yields more than 10× speedup while preserving performance. The report does not disclose tokenization details, frame rates, audio sampling rates, latent shapes, scheduler types, sequence lengths, or parameter count (Chen et al., 15 Dec 2025).
Seedance 2.0 is described only at a high level as a unified multi-modal audio–video joint generation model (Seedance et al., 15 Apr 2026). The paper emphasizes tight synchronization between generated visual events and multi-track audio, including lip-sync, action–sound alignment, and beat matching, but explicitly does not disclose whether the generator is diffusion-based or autoregressive, whether it operates in 2D, 3D, or video latent spaces, or what encoders, decoders, tokenizers, attention blocks, or explicit time-alignment modules it contains. This omission has shaped external discussion. A survey on video generative foundations argues that, given dominant contemporary practice, it is reasonable to infer that Seedance follows the diffusion-transformer paradigm in latent space with multi-modal conditioning, but the survey also states that this is an inference from model class and positioning rather than a disclosed Seedance-specific fact (Hu et al., 7 Apr 2026).
3. Conditioning, control, and generation workflows
A defining property of the family is increasingly rich conditioning. Seedance 1.0 unifies T2I, T2V, and I2V by concatenating noisy latents with clean or zero-padded frames and using binary masks to indicate which frames are conditioning inputs; in multi-shot generation, MM-RoPE encodes shot boundaries and order, while per-shot dense captions are concatenated in temporal order (Gao et al., 10 Jun 2025). The data and prompting pipeline is correspondingly dense: captions are bilingual, integrate dynamic elements such as actions and camera movements with static elements such as appearance and style, and are generated using Tarsier2 plus a Qwen2.5-14B-based prompt-engineering model aligned with DPO (Gao et al., 10 Jun 2025).
Seedance 1.5 pro generalizes this to native audio–video generation. The model supports text-to-video-audio, image-to-video-audio, and unimodal video tasks, and its data pipeline emphasizes audio–visual coherence, motion expressiveness, advanced captioning for both video and audio, and curriculum-based data scheduling (Chen et al., 15 Dec 2025). The paper highlights autonomous camera scheduling, continuous long takes, tracking shots, orbit/arc motion, dolly zoom, cinematic transitions, professional-grade color grading, and strengthened semantic understanding for cohesive storytelling. It also emphasizes precise multilingual and dialect lip-sync, with reported strengths in Sichuanese, Taiwan Mandarin, Cantonese, and Shanghainese (Chen et al., 15 Dec 2025).
Seedance 2.0 expands control into a reference-and-edit regime. Its documented R2V support includes subject reference, motion reference, style reference, visual-effects/creative reference, instruction-based editing, continuation, and extension (Seedance et al., 15 Apr 2026). The platform supports subject, motion, style, and visual-effects/creative references; instruction-based editing; continuation and extension; and combinations of these with text prompts in one run. The paper reports that it supports 20 of 22 evaluated multimodal task/input combinations and uniquely supports all three visual-effects/creative reference variants and all four continuation/extension variants listed in its task-support matrix (Seedance et al., 15 Apr 2026). A plausible implication is that Seedance 2.0 should be understood not merely as a prompt-to-video model but as a broader multimodal generation-and-editing system.
External papers repeatedly frame Seedance 2.0 as exemplifying “omni-capable” generation. “OmniWeaving: Towards Unified Video Generation with Free-form Composition and Reasoning” states that proprietary systems such as Seedance-2.0 have “largely realized next-level, ‘omni-capable’ intelligent video generation,” defined by multimodal composition and abstract reasoning over interleaved text, image, and video inputs (Pan et al., 25 Mar 2026). The same paper also cautions that it does not provide direct quantitative comparisons to Seedance and treats the underlying techniques as undisclosed, so this positioning is capability-based rather than mechanistically grounded (Pan et al., 25 Mar 2026).
4. Evaluation and comparative performance
Seedance 1.0 reports strong standing on both external and internal evaluations. On Artificial Analysis as of June 10, 2025, it ranks first on both T2V and I2V leaderboards, surpassing Veo 3, Kling 2.0/2.1, Runway Gen4, Wan 2.1, and OpenAI Sora, and in image-to-video it beats Veo 3 and Kling 2.0 by over 100 Elo points (Gao et al., 10 Jun 2025). Internally, SeedVideoBench 1.0 places Seedance among the top systems for T2V and I2V, with particular strength in instruction adherence, motion quality, and coherent shot transitions (Gao et al., 10 Jun 2025).
Seedance 1.5 pro emphasizes expert human assessment and production-aligned criteria rather than standard numeric metrics (Chen et al., 15 Dec 2025). Its evaluation is organized around SeedVideoBench 1.5. The report states that in T2V, Seedance 1.5 pro leads in instruction following, is competitive in visual aesthetics and motion dynamics, and improves over Seedance 1.0 pro in both T2V and I2V. On the audio side, it reports clear advantage over Veo 3.1 in Chinese-language dialogue, dialects, and monologues, as well as superior lip–audio alignment and sound effect timing over Veo 3.1 and Kling 2.6 (Chen et al., 15 Dec 2025).
Seedance 2.0 is documented primarily through Arena.AI Elo and SeedVideoBench 2.0 MOS. In Arena.AI crowdsourced preference, Seedance 2.0 720p ranks first on text-to-video with Elo 1450 and first on image-to-video with Elo 1449 (Seedance et al., 15 Apr 2026). In text-to-video MOS, it tops all six reported dimensions with Motion 3.75, Video prompt following 3.43, Aesthetics 3.67, Audio Quality 3.63, Audio‑Visual Sync 3.75, and Audio prompt following 3.56; the paper states improvements over Seedance 1.5 average , with on motion (Seedance et al., 15 Apr 2026). In image-to-video MOS, it ranks first on all six dimensions with Motion 3.35, Video prompt following 3.46, Image preservation 3.31, Audio Quality/Expressiveness 3.61, AV-sync 3.54, and Audio prompt following 3.70 (Seedance et al., 15 Apr 2026).
The same report also provides fine-grained R2V results. Seedance 2.0 leads all five evaluated R2V dimensions with Multimodal Task Following 2.50/3, Editing Consistency 3.54/5, Reference Alignment 3.03/5, Motion Quality 3.24/5, and Prompt Following 2.52/3 (Seedance et al., 15 Apr 2026). In motion reference, it records task following 2.60 and alignment 2.64, while Kling 3 Omni is separately noted to emphasize first-frame fidelity at 4.31 but with weaker downstream motion. In extension, Seedance 2.0 supports arbitrary uploaded videos but trails Veo 3.1 on task following and alignment (Seedance et al., 15 Apr 2026).
5. External characterizations in benchmark and systems research
Benchmarking studies present a more differentiated picture than product reports. In AV-Phys Bench, Seedance 2.0 is the strongest system among seven evaluated models on audio–visual physical commonsense, with per-dimension pass rates on physics-following prompts of V-SA 0.940, A-SA 0.933, V-PC 0.840, A-PC 0.769, and AV-PC 0.750 (Cui et al., 8 May 2026). Category-level human evaluation gives Seedance 2.0 overall SA 0.903, PC 0.660, and Both 0.653, with transition scenes—especially Event Transition—identified as hardest (Cui et al., 8 May 2026). Yet the same benchmark shows that Anti-AV-Physics prompts cause a collapse from physics-following PC 0.660 to anti-physics PC 0.208, a 68.5% drop, indicating strong physically consistent priors but limited ability to render requested physical violations (Cui et al., 8 May 2026).
CLVG-Bench, which targets zero-shot multimodal reasoning, similarly shows strength on structured generation tasks and weakness on deeper reasoning. Seedance 2.0’s human-evaluated pass rates are 61.25 on Elements Editing, 52.64 on Partial Reference, 67.65 on Script Continuation, 63.54 on Physical Simulation, 43.37 on Perception, and 21.25 on Logical Reasoning (Zhang et al., 21 Apr 2026). The paper states that state-of-the-art video models such as Seedance 2.0 fall substantially short on logically grounded and interactive generation tasks, with interactive performance effectively collapsing beyond the first turn without external assistance; VLM-assisted context organization raises Seedance 2.0 from 21.3 to 49.0 on Logical Reasoning and from 63.5 to 70.3 on Physical Simulation (Zhang et al., 21 Apr 2026).
Open-source system papers often use Seedance as a closed-source reference point rather than a fully analyzable baseline. “OmniWeaving” treats Seedance-2.0 as a proprietary leader that has “largely realized next-level, ‘omni-capable’ intelligent video generation,” but explicitly refrains from head-to-head metrics because the underlying techniques are undisclosed (Pan et al., 25 Mar 2026). “Evolution of Video Generative Foundations” similarly positions Seedance 2.0 as a milestone in “Multimodal Long Narratives,” characterizing it as strengthening control and stability through a comprehensive, omni-modal conditioning framework (Hu et al., 7 Apr 2026). These accounts suggest broad consensus on Seedance’s frontier status, but they also underscore that much of the field’s interpretation is based on observed capability rather than published internals.
Task-specific research has also used Seedance as a strong comparator while highlighting domain-specific limitations. “Avatar V: Scaling Video-Reference Avatar Video Generation” reports that Seedance 2.0 explores video-based references and “full-clip reference conditioning,” but argues that naively concatenating reference and generation tokens yields prohibitive quadratic attention cost as reference length grows and lacks explicit supervision on both static identity similarity and dynamic motion fidelity (Liang et al., 11 Jun 2026). On a cross-scene avatar benchmark, Seedance 2.0 scores Sync-C 8.86, Sync-D 6.99, Face Sim 0.823, and Q-Align 4.85, while Avatar V reports higher Sync-C, lower Sync-D, and higher Face Sim (Liang et al., 11 Jun 2026). In anime-specific evaluation, Seedance-Pro 1.0 is described as a strong closed-source baseline, but AniMatrix reports higher scores on Prompt Understanding and Artistic Motion, with Seedance-Pro 1.0 at Style Fidelity 4.15, Prompt Understanding 3.12, Artistic Motion 3.26, Structural Stability 3.84, and Anime Aesthetic 4.09 (Team, 5 May 2026).
6. Limitations, safety, and unresolved questions
The Seedance papers themselves acknowledge unresolved problems. Seedance 1.0 notes remaining challenges in very long-horizon compositionality, complex multi-agent interactions over minutes, fast non-rigid phenomena such as cloth and fluids under extreme camera motion, rare objects and events with insufficient tail data, exact camera path realism, and the absence of audio synchronization because it is video-only (Gao et al., 10 Jun 2025). Seedance 1.5 pro states that the model’s mastery of specific vocal styles within traditional Chinese opera sub-genres is “still evolving” and does not detail safety systems, watermarking, or fairness analyses (Chen et al., 15 Dec 2025). Seedance 2.0 lists edge-case motion plausibility, deformation artifacts, high-frequency visual noise, audio distortion/noise, multi-speaker lip-sync, and video extension as explicit areas for improvement (Seedance et al., 15 Apr 2026).
Independent evaluations sharpen these limitations. AV-Phys Bench shows that Seedance 2.0’s performance drops from C1 Steady State PC 0.720 to C2 Event Transition PC 0.535, and provides examples where visual actions are correct but the exact acoustic consequence is wrong, such as a xylophone sequence whose pitch fails to rise monotonically with shorter bars (Cui et al., 8 May 2026). CLVG-Bench argues that Seedance’s generator benefits substantially from an external understanding module, implying that intrinsic planning and reasoning remain bottlenecks (Zhang et al., 21 Apr 2026). “Avatar V” further argues that Seedance-style conditioning bottlenecks are insufficient for behaviorally recognizable avatars, especially in facial regions and talking-style transfer (Liang et al., 11 Jun 2026).
Safety research has exposed a distinct vulnerability in Seedance-1.5-pro. “VII: Visual Instruction Injection for Jailbreaking Image-to-Video Generation Models” evaluates Seedance-1.5-pro as one of four commercial I2V systems under a black-box setting and characterizes its behavior as “Permissive Entry, Strict Exit” (Zheng et al., 24 Feb 2026). Under the VII attack on COCO-I2VSafetyBench, Seedance’s refusal rate drops to 0.0 across categories while average attack success rate rises to 51.0 in classifier-based evaluation and 52.5 in VLM-based evaluation; on ConceptRisk, VII yields average ASR 47.5 and 44.0 with average RR 1.0 in both settings (Zheng et al., 24 Feb 2026). The paper interprets this as evidence that Seedance rarely refuses at input and relies more heavily on output-level moderation than pre-generation filtering.
A common misconception is that Seedance’s strong human preference results imply general reasoning competence or full physical grounding. The benchmark literature does not support that conclusion. Instead, it consistently reports a system that is strong at prompt following, multimodal reference handling, cinematic synthesis, and many forms of audio–visual synchronization, but still limited on event-transition physics, logically grounded generation, interactive multi-turn reasoning, and some safety failure modes (Cui et al., 8 May 2026, Zhang et al., 21 Apr 2026, Zheng et al., 24 Feb 2026). This suggests that Seedance occupies a frontier position in practical multimodal video generation while remaining, in several benchmarked senses, short of robust world modeling or true multimodal reasoning.