TurboT2VA: Fast Large-Scale Text-to-Video-Audio Generation via Score-Regularized Consistency Distillation

Abstract: Joint text-to-video-audio generation produces synchronized visual and acoustic content, but the long sampling trajectories and heterogeneous multimodal computation of large models make inference prohibitively expensive. We present TurboT2VA, a distillation and inference framework for accelerating a 19B-parameter joint video-audio model. Large-scale T2VA distillation is challenged by modality-imbalanced optimization, the difficulty of continuous-time consistency training at scale, and the quality--diversity trade-off. TurboT2VA addresses these issues with per-modality normalization and a progressive curriculum comprising discrete consistency warm-up, continuous consistency refinement, and joint consistency--distribution matching. The curriculum first establishes a stable, diverse generation trajectory and only then introduces distribution-level refinement. On LTX-2, four-step distillation reduces generator latency from 50.52s to 2.51s at the standard evaluation resolution of 512$\times$768, achieving a 20.1$\times$ speedup while maintaining strong visual quality, audio fidelity, diversity, and video-audio synchronization. We further develop an architecture-aware inference stack that combines guarded W8A8 and fused operators, padded-text compaction, and modality-aware sparse attention while preserving dense cross-modal and text-conditioning paths. Under the high-resolution deployment setting at 1024$\times$1792, the complete stack reduces generator latency from 318.74s to 5.83s on one NVIDIA H20, achieving a 54.67$\times$ generator-only speedup. Inference code and generation demos are available at https://github.com/thu-ml/TurboDiffusion/tree/main/turbot2va.

• The paper introduces a dCM → sCM → sCM+DMD curriculum that balances cross-modal consistency, perceptual quality, and generation diversity in a unified 19B-parameter text-to-video-audio model.
• TurboT2VA reduces sampling from 40 to four steps, cutting standard-resolution latency from 50.52 to 2.51 seconds while raising Javis from 0.165 to 0.196 and lowering desynchronization from 0.617 to 0.388.
• Architecture-aware optimization—including W8A8 quantization, fused kernels, sparse attention, and text-token compaction—reduces high-resolution generator latency from 318.74 to 5.83 seconds, a reported 54.67× speedup.

TurboT2VA addresses the computational cost of unified text-to-video-audio (T2VA) generation by combining large-scale consistency distillation with architecture-aware inference optimization. The target system is a 19B-parameter LTX-2 model composed of a 14B video backbone and a 5B audio backbone. The central problem is not only the number of denoising evaluations required by the teacher, but also the heterogeneous cost of each multimodal Transformer evaluation, particularly at high spatial and temporal resolutions. TurboT2VA therefore treats acceleration as a coupled modeling and systems problem: the student reduces the sampling trajectory from 40 steps to four, while the inference stack reduces the cost of each remaining step through quantization, kernel fusion, token compaction, and selective sparse attention.

Problem formulation and technical motivation

Unified T2VA generation differs materially from cascaded video-to-audio or audio-to-video pipelines. A joint model can represent cross-modal dependencies during generation, but video and audio have distinct latent dimensionalities, temporal resolutions, conditioning structures, and optimization dynamics. A naïve sum of video and audio distillation losses can consequently produce video-dominated gradients, degrading acoustic fidelity or synchronization. The paper identifies three associated difficulties: modality imbalance, instability in continuous-time consistency training, and the tension between diversity-preserving consistency objectives and realism-oriented distribution matching.

TurboT2VA builds on three distillation paradigms. Discrete consistency models, denoted dCM, impose consistency over a finite set of noise levels and are used as a comparatively stable initialization mechanism. Continuous consistency models, denoted sCM, impose trajectory consistency in continuous time and require Jacobian–vector-product (JVP) estimates. Distribution Matching Distillation (DMD) instead aligns student and teacher distributions through score discrepancies, generally improving perceptual quality but potentially inducing mode-seeking behavior. The design follows the complementary roles established in prior work on consistency models, DMD, and score-regularized continuous-time consistency (Song et al., 2023, Tan et al., 2024, Gutiérrez et al., 2024, 2608.24674).

The paper’s distinctive claim is that these objectives should not be applied symmetrically or simultaneously from the beginning. Instead, the student is trained through the ordered curriculum dCM $\rightarrow$ sCM $\rightarrow$ sCM+DMD. This sequence is intended to establish a stable denoising prior, expand it into a continuous teacher-aligned trajectory, and only then apply distribution-level refinement.

(TurboT2VA overview)

Figure 1: TurboT2VA combines a progressive distillation curriculum with modality-aware inference optimization for four-step synchronized video-audio generation.

Cross-modal joint distillation

The student is trained as a single coupled generator rather than as two independently distilled modality branches. For a text condition $c$, paired clean video and audio latents are denoted $z_0^v$ and $z_0^a$. Noise is injected using a shared base timestep, while the modality-specific noise variables remain independently sampled. The student then predicts both clean modalities in one forward pass:

$$(\hat z_{0,\theta}^v,\hat z_{0,\theta}^a) = G_\theta(z_t^v,z_t^a,t,c).$$

This construction preserves cross-modal interaction in the forward graph. Although losses are computed separately for video and audio, both gradients pass through the same multimodal Transformer. Consequently, the video trajectory can depend on audio tokens and the audio trajectory can depend on video tokens. This is important for synchronization: independent distillation would preserve modality-specific behavior but would not directly constrain the paired trajectory on which audiovisual correspondence depends.

The continuous-time consistency formulation extends this coupling to the JVP. For modality $m$, the student’s trajectory field is differentiated not only with respect to time but also with respect to both noisy modality latents. The resulting JVP follows the paired teacher direction, so cross-modal dependencies enter the local trajectory estimate itself. This is a stronger formulation than independently applying sCM to two branches and subsequently summing their losses.

The paper addresses gradient imbalance by normalizing the consistency direction separately for each modality. The sCM direction uses an L2 normalization over the latent elements of the relevant modality, while modality weights $w_m$ control the relative contribution of video and audio. This preserves a unified computational graph while preventing the larger or faster-learning video branch from overwhelming the audio branch. The implication is that modality balancing is performed at the objective level rather than by sacrificing cross-modal interaction through architectural decoupling.

The DMD component is also computed on paired student-generated video-audio samples. The student sample is perturbed, and the discrepancy between the teacher prediction and a fake-score prediction trained on student samples supplies the distribution-matching signal. DMD uses a modality-specific residual-scale normalizer rather than the L2 direction normalization used by sCM. This distinction is technically consequential: the two objectives have different gradient geometries and are not treated as interchangeable losses.

The final stage combines the two normalized objectives:

$$\mathcal L_{\mathrm{stage3}} = \lambda_{\mathrm{sCM}}\mathcal L_{\mathrm{sCM}} + \lambda_{\mathrm{DMD}}\mathcal L_{\mathrm{DMD}}.$$

sCM supplies trajectory regularization, diversity retention, and temporal structure; DMD supplies a teacher-distribution anchor for final-sample realism. Because both terms operate on the same paired samples, the distribution-matching refinement is constrained by the cross-modal structure already represented by the student.

Progressive curriculum distillation

The curriculum is the paper’s principal algorithmic contribution. In the first stage, dCM is applied over discrete noise levels. This avoids the JVP requirements of continuous-time training and supplies coarse denoising behavior, stable timestep conditioning, and an initial joint prior. The goal is not to optimize final perceptual quality at this point, but to produce a tractable initialization for the more demanding objectives.

The second stage replaces dCM with sCM. The student now learns a continuous teacher-guided trajectory, including local generation directions across noise levels. In a T2VA model, this trajectory encompasses visual motion, audio evolution, event timing, and cross-modal temporal correspondence. The paper argues that this stage is necessary because distribution matching applied before trajectory acquisition can improve local realism while narrowing the set of trajectories represented by the student.

The third stage introduces DMD while retaining sCM. This is framed as a score-regularized consistency objective: sCM constrains the student to remain trajectory-consistent, while DMD improves perceptual alignment with the teacher distribution. The paper explicitly distinguishes being trainable from being optimal. Direct sCM+DMD optimization from the base initialization is feasible and reaches reasonable quality, but it yields a weaker quality–diversity trade-off than the staged route.

The ablations support the ordering argument. After the joint stage begins, the staged curriculum overtakes direct sCM+DMD training and retains a late-training advantage through 7,500 global steps. Starting from the same dCM+sCM prefix, staged joint refinement reaches a Javis score of 0.1963 at 7,000 steps, exceeding both DMD-only refinement and sCM-only continuation.

Figure 2: Training curves show that staged dCM $\rightarrow$ sCM $\rightarrow$0 sCM+DMD optimization improves late-training performance over direct joint training and single-objective continuation.

The interpretation is consistent with the objective decomposition. sCM-only training retains broader trajectory diversity but produces weaker final quality, whereas DMD-only training improves quality but is more repetitive. The curriculum delays the mode-seeking pressure of DMD until the student has learned a sufficiently broad trajectory prior.

Quantitative generation results

The main evaluation uses a 200-prompt JavisBench split at $\rightarrow$1 resolution with 121 frames. The teacher uses 40 sampling steps, whereas TurboT2VA uses four. Generator latency falls from 50.52 seconds to 2.51 seconds per sample, corresponding to a 20.1-times speedup.

The student does not simply preserve the teacher’s metrics; it changes the quality profile. Relative to the teacher, the four-step model improves the reported visual quality score from 1.968 to 2.389, motion from 0.783 to 0.849, video–audio semantic consistency from 0.199 to 0.229, AVH from 0.195 to 0.224, and Javis from 0.165 to 0.196. Its desynchronization score also decreases from 0.617 to 0.388. These gains should be interpreted cautiously because the teacher and student may occupy different points on the benchmark’s metric trade-offs, and the student does not dominate every dimension: audio quality decreases from 4.902 to 4.486, and text–video ImageBind consistency decreases from 0.268 to 0.218.

Configuration	Steps	Latency	Javis	Desynchronization
Teacher	40	50.52 s	0.165	0.617
TurboT2VA	4	2.51 s	0.196	0.388

The comparison with open-source systems reinforces the efficiency result. TurboT2VA’s 2.51-second runtime is substantially below the reported runtimes of JavisDiT, OVI, and other cascaded systems, while its Javis score of 0.196 exceeds the listed open-source baselines. It also compares favorably with the closed-source reference values included in the benchmark, although those systems have different architectures, inference configurations, and unavailable runtime measurements. The paper therefore establishes strong benchmark competitiveness, but not a strictly controlled head-to-head comparison against all external systems.

Modality-specific fidelity and diversity

On VBench, the student improves aesthetic quality from 0.5286 to 0.5519 and imaging quality from 0.5884 to 0.6520, while maintaining subject consistency at 0.9614 versus 0.9592 for the teacher. Motion smoothness is nearly unchanged at 0.9911 versus 0.9919, and temporal consistency decreases from 0.9844 to 0.9773. The results indicate that four-step distillation preserves most video-level fidelity, but the small temporal-consistency reduction is relevant because temporal stability is a central failure mode in aggressive video distillation.

Audio metrics are similarly close but not identical. The student has slightly higher usefulness, 6.334 versus 6.307, and production quality, 6.584 versus 6.577, but lower enjoyment, 4.612 versus 4.765, production complexity, 2.964 versus 3.275, aggregate audio aesthetics, 5.115 versus 5.231, and MS-CLAP text–audio similarity, 0.347 versus 0.353. Thus, the paper’s claim is best characterized as preservation of competitive audio fidelity rather than exact teacher replication.

The diversity ablation directly motivates the staged objective. Under eight seeds per prompt, sCM-only obtains the highest video and audio diversity but the lowest Javis score. DMD-only obtains a higher Javis score than sCM-only but substantially lower diversity. The staged model reaches the highest Javis score and recovers a considerable fraction of the diversity lost by DMD-only.

Objective	Javis	Video diversity	Audio diversity	Mean audiovisual diversity
sCM-only	0.1131	0.3026	0.5037	0.4032
DMD-only	0.1812	0.1821	0.3561	0.2691
Staged sCM+DMD	0.1963	0.2322	0.4197	0.3259

The staged model does not maximize raw diversity; sCM-only remains higher on every diversity measure. Its advantage is the joint operating point: it improves Javis from 0.1812 for DMD-only to 0.1963 while increasing mean audiovisual diversity from 0.2691 to 0.3259. This supports the narrower claim that curriculum distillation offers a more favorable quality–diversity compromise, not that it eliminates the underlying trade-off.

Figure 3: Qualitative comparisons show that sCM-only varies more broadly but loses prompt fidelity, DMD-only produces more repetitive compositions, and staged training retains both fidelity and cross-seed variation.

Architecture-aware inference acceleration

Distillation reduces the number of Transformer evaluations but leaves the per-evaluation computation largely intact. TurboT2VA therefore introduces an inference stack specialized for the LTX-2 architecture. The stack applies sparse attention only to unmasked video and audio self-attention. Bidirectional video–audio attention and masked text cross-attention remain dense, preserving the paths most directly responsible for cross-modal exchange and text conditioning.

The sparse-attention dispatcher uses SageSLA with a default retention ratio $\rightarrow$2. Query-dependent block selection is applied separately to the heterogeneous modality paths, and a lower bound prevents degenerate selection on short audio sequences. This restriction is important: indiscriminate sparsification could alter cross-modal information flow or interact incorrectly with attention masks.

The stack also uses guarded W8A8 linear operators with static per-output-channel weight quantization and dynamic per-row activation quantization. INT8 products are accumulated in INT32 over the full reduction dimension before scale and bias application. Unsupported tensor shapes, dtypes, or layouts fall back to BF16 operations, making the optimization selective rather than universally approximate. Fused kernels target RMSNorm, LayerNorm, modulation, gated residual updates, rotary embeddings, and related elementwise operations. Padded text tokens are compacted only when batch-one masking permits it; valid prompt tokens are not semantically truncated.

At $\rightarrow$3 resolution with 121 frames on one NVIDIA H20, the dense 40-step teacher requires 318.74 seconds of generator-only latency. W8A8 and fused operations reduce this to 233.34 seconds. The four-step student with those optimizations requires 12.16 seconds, and adding sparse attention plus text compaction reduces latency to 5.83 seconds.

Inference configuration	Generator latency	Speedup over dense teacher
Dense 40-step teacher	318.74 s	1.00x
Quantized and fused teacher	233.34 s	1.36x
Quantized and fused four-step student	12.16 s	26.20x
Full stack, $\rightarrow$4	5.83 s	54.67x

The systems result demonstrates complementarity between algorithmic and kernel-level acceleration. The unoptimized four-step student requires 16.50 seconds, so model distillation alone yields a large reduction, while the inference stack contributes an additional 2.83-times speedup beyond step reduction. Sparse attention supplies the largest individual component gain.

Figure 4: Cumulative high-resolution profiling separates the contributions of quantization, fused operations, four-step distillation, sparse attention, and text-token compaction.

The speed–quality frontier is relatively flat across retention ratios. Reducing $\rightarrow$5 from 0.5 to 0.2 lowers latency from 6.44 to 5.57 seconds, while Javis remains between 0.1901 and 0.1948. The paper selects $\rightarrow$6 rather than the fastest configuration because it provides a stronger cross-modal alignment balance, including the best reported CAVP value among the tested settings and lower desynchronization than $\rightarrow$7. This is an empirical operating-point choice, not a universal sparsity optimum.

Sampling-step behavior and empirical interpretation

The step ablation shows monotonic improvement across most metrics as the student receives more sampling evaluations. One-step inference is substantially weaker, with a Javis score of 0.0663 and desynchronization of 0.6220. Two steps improve Javis to 0.1576 and reduce desynchronization to 0.3780. Four steps further increase Javis to 0.1963 and improve motion, audio quality, audiovisual consistency, and AVH.

The result establishes that the four-step configuration is not merely a nominal deployment choice. It occupies a meaningful point in the student’s quality–latency curve. At the same time, the experiments do not determine whether four steps are optimal under other resolutions, prompt distributions, guidance scales, or sparse-attention settings.

Figure 5: Qualitative teacher–student examples indicate preservation of subject identity, layout, temporal appearance, waveform structure, and mel-spectrogram organization under four-step sampling.

Limitations and open questions

The reported speedups are generator-only measurements. They exclude model loading, VAE decoding, audio–video muxing, disk I/O, and other end-to-end deployment costs. Consequently, the 54.67-times figure should not be interpreted as an application-level wall-clock speedup for complete media generation. The training procedure also requires approximately 21 hours on eight H20 GPUs, excluding evaluation and checkpoint-sweep overhead, and uses a 100K paired text-video-audio training set. The paper does not report the total computational cost of teacher inference during distillation.

The sparse-attention results depend on the H20 implementation, the chosen resolution, batch size, prompt distribution, and fixed retention schedule. The authors explicitly note that $\rightarrow$8 should be revalidated when resolution or prompt statistics change. Quantized kernels likewise cover only compatible shapes and dtypes; unsupported cases fall back to BF16 computation. These guards improve reliability but limit the generality of the measured acceleration.

The evaluation is primarily based on automated metrics and controlled qualitative comparisons. Audio perception, synchronization, and multimodal coherence can exhibit failure modes not captured by JavisBench, VBench, TTA-Bench, ImageBind, or MS-CLAP. Moreover, the diversity measure is embedding distance within a fixed prompt and therefore does not distinguish desirable variation from semantic drift. The staged curriculum improves this measured trade-off, but the paper does not provide a systematic sweep over modality weights, $\rightarrow$9, guidance scales, or stage lengths. It therefore leaves open how sensitive the result is to these hyperparameters.

Finally, the teacher and student are evaluated at different effective sampling trajectories, and the student improves some metrics while declining on others. The results support competitive four-step synthesis, but they do not establish that the student is uniformly equivalent to the 40-step teacher. An unresolved technical question is whether the same curriculum remains effective when the teacher architecture, latent parameterization, modality ratio, or target step count changes substantially.

Conclusion

TurboT2VA presents a coherent model-and-systems approach to accelerating joint video-audio generation. Its modeling contribution is modality-balanced, cross-modal consistency distillation combined with a staged dCM $c$0 sCM $c$1 sCM+DMD curriculum. Its systems contribution is an architecture-aware inference stack that selectively combines sparse attention, W8A8 operators, fused multimodal kernels, and text-padding compaction while retaining dense cross-modal and text-conditioning paths.

On a 19B-parameter LTX-2 system, four-step distillation reduces standard-resolution generator latency from 50.52 to 2.51 seconds, while high-resolution deployment with the complete stack reduces latency from 318.74 to 5.83 seconds on one NVIDIA H20. The central empirical result is not elimination of the quality–diversity trade-off, but a more favorable operating point: staged joint consistency and distribution matching substantially outperform sCM-only and DMD-only alternatives in combined quality and diversity, while preserving synchronized audiovisual generation under aggressive sampling reduction.