LTX-Video: Latent Diffusion Video Transformer

• LTX-Video is a latent diffusion architecture that uses a causal 3D Video-VAE and full spatiotemporal self-attention to compress videos by a 1:192 ratio.
• The design integrates a jointly trained video encoder and denoising decoder, enabling real-time generation with metrics like 5 seconds for 24 fps video on advanced GPUs.
• It supports diverse conditioning methods—from text and images to event streams and physical parameters—allowing flexible adaptations for audiovisual synthesis and scientific simulations.

LTX-Video is a transformer-based latent diffusion architecture for video generation in which the Video-VAE and denoising transformer are designed jointly rather than treated as independent subsystems. In the original formulation, patchification is relocated from the transformer input to a causal 3D VAE encoder, the video is compressed at a ratio of $1:192$ with spatiotemporal downscaling of $32\times 32\times 8$ pixels per token, full spatiotemporal self-attention is performed in latent space, and the VAE decoder carries out the final denoising step directly in pixel space (HaCohen et al., 2024). Subsequent work reuses, scales, and repurposes the same backbone family across text-to-video, image-to-video, audiovisual generation, event-based RGB reconstruction, identity-preserving synthesis, stage-aware acceleration, and differentiable physics simulation (HaCohen et al., 6 Jan 2026, Torbunov et al., 4 Dec 2025, Chen et al., 1 Jun 2026, Song et al., 31 Dec 2025, Perini et al., 22 Mar 2026).

1. Definition and lineage

The original paper, "LTX-Video: Realtime Video Latent Diffusion" (HaCohen et al., 2024), introduced a 2 billion-parameter system trained jointly for text-to-video and image-to-video generation. Its stated objective was to optimize the interaction between a highly compressed Video-VAE and a latent diffusion transformer, rather than adopting the more conventional separation in which tokenization, denoising, and final refinement are handled by loosely coupled modules. The same work reported faster-than-real-time generation: 5 seconds of 24 fps video at $768\times 512$ resolution in 2 seconds on a single NVIDIA H100.

Later papers use the name in a broader architectural sense. "FlowBlending" studies paired LTX-Video models at 2B and 13B scale for stage-aware sampling (Song et al., 31 Dec 2025). "LTX-2" identifies LTX-Video as the 14B-parameter video stream of a larger joint audiovisual foundation model (HaCohen et al., 6 Jan 2026). "Pretrained Video Models as Differentiable Physics Simulators for Urban Wind Flows" uses a 2B TI2V variant as the backbone of WinDiNet (Perini et al., 22 Mar 2026). "Spatial-Temporal Decoupled Reference Conditioning for Identity-Preserving Text-to-Video Generation" builds on the LTX-2.3 video branch (Chen et al., 1 Jun 2026). This suggests that "LTX-Video" functions in the literature as the name of an architectural lineage, not only of a single checkpoint.

Reference	LTX-Video configuration	Role
(HaCohen et al., 2024)	2B, 28 blocks, hidden size 2048	Original text-to-video and image-to-video model
(Song et al., 31 Dec 2025)	2B and 13B	Stage-aware multi-model sampling
(HaCohen et al., 6 Jan 2026)	14B video stream	Joint audiovisual generation in LTX-2
(Perini et al., 22 Mar 2026)	2B TI2V variant	Differentiable CFD surrogate in WinDiNet
(Chen et al., 1 Jun 2026)	LTX-2.3 video branch	Identity-preserving text-to-video generation

2. Latent representation and transformer design

At the core of the original model is a causal 3D Video-VAE whose encoder downsamples the input video spatially by $32$ and temporally by $8$. For an input clip of shape $T\times H\times W$, the latent grid has resolution

$$H_{\rm lat}=\frac{H}{32},\qquad W_{\rm lat}=\frac{W}{32},\qquad T_{\rm lat}=\frac{T}{8},$$

with latent channel dimension $128$. The encoder output is therefore a tensor of shape $$128\times H_{\rm lat}\times W_{\rm lat}\times T_{\rm lat}$$, and the transformer operates directly on the flattened latent grid rather than on separately patchified tokens. Because the downscaling factor is $32\times 32\times 8$, each latent token corresponds to $32\times 32\times 8$0 pixel values, and the overall compression ratio is

$32\times 32\times 8$1

This very high compression is the enabling condition for full spatiotemporal self-attention at practical cost (HaCohen et al., 2024).

The decoder is not merely a reconstruction head. It is a coarse-to-fine 3D U-Net–style "denoising decoder" that maps a noisy latent tensor and a timestep embedding directly to pixels. Timestep conditioning is applied through adaptive normalization layers, and multi-layer noise injection is used in intermediate feature maps with learned per-channel noise scales. The original paper explicitly frames this as a remedy for the fact that aggressive latent compression limits the representation of fine details; the decoder therefore combines latent-to-pixel conversion with the final denoising step (HaCohen et al., 2024).

The transformer itself performs full spatiotemporal self-attention on the latent sequence

$32\times 32\times 8$2

using standard multi-head attention with QK-normalization and rotary positional embeddings. Queries and keys are L2-normalized before the dot product, and RoPE is applied using relative fractional coordinates over space and time. The paper reports that exponential spacing for the RoPE frequencies outperforms inverse-exponential spacing (HaCohen et al., 2024). In the widely reused 2B instantiation adapted by IE2Video, the latent diffusion transformer is described as a rectified-flow DiT with 28 blocks, hidden dimension $32\times 32\times 8$3, and 32 heads operating in the VAE latent space (Torbunov et al., 4 Dec 2025).

3. Objectives, conditioning, and inference

The original LTX-Video transformer is trained under a rectified-flow velocity-prediction objective in latent space. With clean latents $32\times 32\times 8$4 and Gaussian noise $32\times 32\times 8$5, the forward interpolation is

$32\times 32\times 8$6

and the diffusion loss is

$32\times 32\times 8$7

The decoder is pretrained both as a reconstruction model and as a small-range diffusion model on noisy latents for $32\times 32\times 8$8, using pixel MSE, LPIPS, Video-DWT, and reconstruction GAN losses, together with a decoder diffusion loss in pixel space. The full training objective combines these terms with empirically chosen coefficients (HaCohen et al., 2024).

Conditioning is unified across text-to-video and image-to-video. For text-to-video, a pretrained T5 encoder such as T5-XXL produces token embeddings that are injected by cross-attention at each transformer block. For image-to-video, the conditioning frame is VAE-encoded with temporal length $32\times 32\times 8$9; during training, first-frame tokens are sometimes assigned a small random timestep $768\times 512$0, and at inference they are inserted at $768\times 512$1 while the remaining tokens start at $768\times 512$2. The model then denoises all tokens jointly. The paper stresses that no additional networks, tokens, or fine-tuning are needed for image conditioning; text- and image-conditioning share the same weights (HaCohen et al., 2024).

The training pipeline also includes aggressive data curation: public and licensed video-plus-image corpora filtered by a learned aesthetic scorer, black-bar cropping, removal of near-static clips, automatic re-captioning with an internal vision-LLM, and multi-resolution training with up to 20% stochastic token dropping to avoid padding. Reported hyperparameters for the original 2B model include 28 transformer blocks, hidden size $768\times 512$3, 4$768\times 512$4 FFN factor, 40 diffusion timesteps, and log-normal $768\times 512$5-sampling clamped to $768\times 512$6 percentiles; inference uses 20 denoising steps (HaCohen et al., 2024).

Later descendants retain the latent-video DiT structure but are not tied to a single diffusion parameterization. The NSP framework formulates the DiT as a flow-matching model over latent trajectories (Li et al., 15 Dec 2025), whereas the LTX-2 video stream follows a standard DDPM-style noise-prediction objective with linear $768\times 512$7 schedule (HaCohen et al., 6 Jan 2026). This indicates that the architectural core is compatible with both rectified-flow and DDPM-style training regimes.

4. Scaling, multimodality, and sampling efficiency

The most explicit scale-up appears in "LTX-2: Efficient Joint Audio-Visual Foundation Model" (HaCohen et al., 6 Jan 2026). There, LTX-Video is the 14B-parameter video stream in an asymmetric dual-stream transformer paired with a 5B-parameter audio stream. The video transformer stacks approximately 72 blocks at width approximately $768\times 512$8. Each block contains four stages: self-attention with 3D RoPE, text cross-attention, audio-video cross-attention, and a two-layer FFN with GELU, with RMS normalization and residual connections throughout. Cross-modality AdaLN modulates queries, keys, and values using the other modality’s diffusion timestep embedding, and bidirectional audio-video cross-attention is combined with modality-aware classifier-free guidance. In video-only mode, the cross-modal guidance term is dropped and the audio branch becomes a no-op. Reported practical properties include 24 fps generation for up to 20 seconds of continuous video and denoising of a 121-frame 720p clip in approximately 1.22 seconds per step on an NVIDIA H100.

The same paper reports strong benchmark performance for the video stream in isolation: in the November 2025 Artificial Analysis leaderboard, it ranks 3rd in Image-to-Video and 4th in Text-to-Video, and human studies are reported to place its visual fidelity and prompt adherence alongside Veo 3 and Sora 2 while delivering full audio-plus-video output (HaCohen et al., 6 Jan 2026). These claims pertain specifically to the 14B video stream as instantiated within LTX-2.

Efficiency-oriented work has also targeted inference without changing the underlying LTX dynamics. "FlowBlending: Stage-Aware Multi-Model Sampling for Fast and High-Fidelity Video Generation" (Song et al., 31 Dec 2025) studies paired LTX-Video models at 13B and 2B scale under a 40-step denoising schedule. Its LSL schedule uses the 13B model for steps $768\times 512$9 through $32$0, the 2B model for steps $32$1 through $32$2, and the 13B model again for steps $32$3 through $32$4. The switch points are chosen by two empirical criteria: a DINO-similarity "knee" for the early boundary and an FID valley for the late boundary. A complementary velocity-divergence analysis shows a U-shaped gap between large- and small-model velocity fields, with the smallest divergence in the intermediate stage. On LTX-Video, the reported result is 30.18 $32$5 0.04 seconds and 2005 TFLOPs per video for LSL, versus 49.73 $32$6 0.31 seconds and 3496 TFLOPs for the 13B-only schedule, corresponding to approximately $32$7 speedup and approximately $32$8 FLOPs saved, while maintaining or slightly improving FID, FVD, aesthetic score, background consistency, subject consistency, and motion smoothness (Song et al., 31 Dec 2025).

5. Task-specific conditioning and repurposing

The flexibility of the LTX-Video backbone is most visible in adaptation work. In "IE2Video: Adapting Pretrained Diffusion Models for Event-Based Video Reconstruction" (Torbunov et al., 4 Dec 2025), LTX is repurposed to reconstruct RGB video from a single initial frame and asynchronous event camera data. Events are binned into $32$9 temporal histograms with differential polarity, encoded by an FPN-style 3D convolutional network, and injected additively before each of the 28 transformer blocks as feature tensors $8$0 of shape $8$1. The original transformer weights are frozen and LoRA updates are inserted into every linear layer. On BS-ERGB at 32 frames, the diffusion-based LTX adaptation reports LPIPS $8$2 with LoRA rank $8$3, compared with $8$4 for frozen rank $8$5 and $8$6 for the autoregressive HyperE2VID baseline; sequence lengths are trained at 32 frames and evaluated at 32, 64, and 128 frames. The paper also reports that direct addition $8$7 is sufficient relative to cross-attention, and that 50 rectified-flow steps yield roughly 2 seconds per 5-second video at 24 fps on H100.

In "What Happens Next? Next Scene Prediction with a Unified Video Model" (Li et al., 15 Dec 2025), LTX serves as the synthesis backend of a unified reasoning-and-generation system. Qwen-VL hidden states are attended to by learnable latent queries, and the resulting features are transformed by a connector consisting of two linear layers with SiLU, RMSNorm, and a learnable scalar $8$8; the output conditions the DiT. Training proceeds in three stages—text-to-image pre-training, joint image-video pre-training, and text-to-video pre-training—followed by supervised fine-tuning on the NSP dataset and reinforcement learning via GRPO with a binary causal-consistency reward from a judge LLM. On VBench at 65 frames and $8$9, the pre-trained model reports quality $T\times H\times W$0 versus $T\times H\times W$1 for the original LTX and semantic alignment $T\times H\times W$2 versus $T\times H\times W$3. On a held-out 1k NSP test set, the causal-consistency pass rate is reported as $T\times H\times W$4, compared with $T\times H\times W$5 for the LTX baseline.

In identity-preserving generation, "Spatial-Temporal Decoupled Reference Conditioning for Identity-Preserving Text-to-Video Generation" (Chen et al., 1 Jun 2026) augments the LTX-2.3 video branch by encoding a reference image into VAE latents and concatenating it as an extra latent frame that is never decoded. To avoid trivial spatial copy-paste, reference tokens are assigned TASS-RoPE coordinates $T\times H\times W$6, placing them adjacent in time but shifted in space. Training combines appearance-invariant reference augmentation with auxiliary ArcFace-based identity and temporal identity-consistency losses, weighted by an SNR-inspired schedule. Inference uses a three-stream CFG decomposition separating unconditional, text-only, and text-plus-reference predictions. On VIP-200K test, the method reports FaceSim-Arc improvement from $T\times H\times W$7 to $T\times H\times W$8 and FaceSim-Cur improvement from $T\times H\times W$9 to $$H_{\rm lat}=\frac{H}{32},\qquad W_{\rm lat}=\frac{W}{32},\qquad T_{\rm lat}=\frac{T}{8},$$0, while also improving CLIP-Score, AQ, MS, and DD and keeping IQ competitive.

LTX-Video has also been transplanted outside video-generation benchmarks altogether. "Pretrained Video Models as Differentiable Physics Simulators for Urban Wind Flows" (Perini et al., 22 Mar 2026) fine-tunes a 2B TI2V variant on 10,000 CFD simulations and treats the resulting model as a differentiable surrogate for 2D incompressible urban wind fields. Text conditioning is dropped and replaced by scalar conditioning on inlet speed and domain size, expanded by Fourier features and mapped by a small MLP into token embeddings. The VAE is adapted with decoder fine-tuning and a physics-informed decoder loss combining reconstruction, divergence, and wall no-penetration terms. Reported fine-tuning results include VRMSE $$H_{\rm lat}=\frac{H}{32},\qquad W_{\rm lat}=\frac{W}{32},\qquad T_{\rm lat}=\frac{T}{8},$$1 for LoRA adaptation on attention weights and $$H_{\rm lat}=\frac{H}{32},\qquad W_{\rm lat}=\frac{W}{32},\qquad T_{\rm lat}=\frac{T}{8},$$2 for full DiT fine-tuning; the best VAE adaptation reduces reconstruction VRMSE from $$H_{\rm lat}=\frac{H}{32},\qquad W_{\rm lat}=\frac{W}{32},\qquad T_{\rm lat}=\frac{T}{8},$$3 to $$H_{\rm lat}=\frac{H}{32},\qquad W_{\rm lat}=\frac{W}{32},\qquad T_{\rm lat}=\frac{T}{8},$$4 and improves final surrogate VRMSE from $$H_{\rm lat}=\frac{H}{32},\qquad W_{\rm lat}=\frac{W}{32},\qquad T_{\rm lat}=\frac{T}{8},$$5 to $$H_{\rm lat}=\frac{H}{32},\qquad W_{\rm lat}=\frac{W}{32},\qquad T_{\rm lat}=\frac{T}{8},$$6. The resulting model generates 112-frame rollouts in under a second and supports gradient-based inverse design, with CFD-verified discomfort $$H_{\rm lat}=\frac{H}{32},\qquad W_{\rm lat}=\frac{W}{32},\qquad T_{\rm lat}=\frac{T}{8},$$7 m/s reduced from $$H_{\rm lat}=\frac{H}{32},\qquad W_{\rm lat}=\frac{W}{32},\qquad T_{\rm lat}=\frac{T}{8},$$8 initially to $$H_{\rm lat}=\frac{H}{32},\qquad W_{\rm lat}=\frac{W}{32},\qquad T_{\rm lat}=\frac{T}{8},$$9, and dangerous $128$0 m/s fraction reduced from $128$1 to $128$2.

These adaptations indicate that LTX-Video is not confined to prompt-conditioned video synthesis. The same latent-video transformer can absorb event streams, semantic scene priors, identity references, scalar physical parameters, and audio signals, with the specific conditioning mechanism varying from additive feature injection to cross-attention, latent concatenation, and connector-mediated semantic projection.

6. Empirical profile, limitations, and significance

A recurrent theme in the literature is that LTX-Video’s efficiency is inseparable from its compression strategy and decoder design. The original paper states that high compression inherently limits the representation of fine details, and specifically assigns the VAE decoder the dual role of latent-to-pixel conversion and final denoising to compensate for that limitation (HaCohen et al., 2024). This is a defining architectural choice rather than an implementation detail.

The conditioning literature around LTX-Video also shows that no single mechanism is universally preferred. IE2Video finds that direct additive conditioning before each transformer block is sufficient relative to cross-attention for event information (Torbunov et al., 4 Dec 2025). WinDiNet replaces text conditioning entirely with scalar embeddings and disables classifier-free guidance by setting CFG $128$3, because over-scaling the conditioning distorts the learned physical mapping (Perini et al., 22 Mar 2026). ST-DRC, by contrast, exploits latent in-context concatenation but reports that this same mechanism can invite copy-paste artifacts unless spatial-temporal decoupling and explicit identity supervision are added (Chen et al., 1 Jun 2026). The NSP work adds another limitation: a domain mismatch between dense captions used in pre-training and the concise preceding-scene prompts of next-scene prediction occasionally leads to visual artifacts (Li et al., 15 Dec 2025).

Sampling studies further complicate the usual assumption that model capacity is uniformly valuable across all timesteps. FlowBlending reports that the difference between large and small LTX-Video models is concentrated in the early and late denoising stages, while the middle stage is comparatively insensitive, as indicated by a U-shaped velocity-divergence curve (Song et al., 31 Dec 2025). A plausible implication is that LTX-Video’s denoising trajectory is structurally heterogeneous: global layout and semantics are especially capacity-sensitive early on, fine detail and coherence are capacity-sensitive late, and the middle regime is more redundant.

Taken together, the published record portrays LTX-Video as a latent video transformer family organized around three persistent ideas: very high spatiotemporal compression, full spatiotemporal attention in latent space, and flexible conditioning pathways layered onto a strong pretrained video prior. Across the cited works, these ideas have supported real-time or near-real-time video generation, multimodal audiovisual synthesis, sensory reconstruction from event streams, identity-preserving generation, accelerated multi-model sampling, and differentiable surrogate modeling for scientific simulation (HaCohen et al., 2024, HaCohen et al., 6 Jan 2026).