MarDini: Masked Autoregressive Diffusion

• MarDini is a hybrid video generative model combining masked autoregression for temporal planning with diffusion for detailed spatial rendering.
• It features an asymmetric planner–generator design where a large MAR model handles low-resolution temporal structure and a compact diffusion model manages high-resolution synthesis.
• The framework supports tasks like video interpolation, image-to-video generation, and temporal expansion through a unified frame-masking interface and a progressive training curriculum.

Searching arXiv for the provided topic and closely related papers. MarDini is a family of video generative models that combine masked autoregression (MAR) with diffusion models (DM) in a unified, asymmetric architecture. Its central design decouples temporal and spatial modelling: a large MAR planning model operates on low-resolution video latents and learns long-range temporal structure, while a smaller diffusion generation model operates on high-resolution latents and focuses on spatial detail conditioned on the planner’s signals. Because the planner is trained with arbitrary frame masking patterns, a single checkpoint can handle video interpolation, image-to-video generation, video expansion, and hierarchical autoregressive long-video generation; the model is trained from scratch on unlabeled videos without image or text pre-training (Liu et al., 2024).

1. Nomenclature and scope

In the video-generation literature, MarDini is the official name of the system introduced in “MarDini: Masked Autoregressive Diffusion for Video Generation at Scale” (Liu et al., 2024). The name refers specifically to a planner–generator architecture in which MAR handles temporal planning and DM handles high-resolution synthesis.

A separate source of ambiguity arises from mechanism-level drug–drug interaction research. In “MARD: Mirror-Augmented Reasoning Distillation for Mechanism-Level Drug-Drug Interaction Prediction,” the string “MarDini” does not appear explicitly; the paper’s official terms are MARD and MARD-7B. The accompanying details state that “MarDini” is very plausibly a colloquial or project-internal nickname in that context, but not an official term. This suggests that the name has circulated in at least two distinct settings, with only the video model using it canonically (Riyazat et al., 10 Jun 2026).

Within generative modelling, MarDini addresses a specific systems problem: standard video diffusion models must jointly model temporal structure and spatial detail in a single network, which makes full spatio-temporal attention prohibitively expensive at scale, while discrete vector-quantized autoregressive alternatives remain capacity-limited and unstable for continuous video signals. MarDini’s response is architectural asymmetry rather than a single monolithic backbone.

2. Asymmetric planner–generator architecture

MarDini has two principal components. The MAR Planning Model $\mathcal{P}$ is large, with configurations ranging from 1.3B to 3.1B parameters, and operates on low-resolution video latents. Some frames are replaced with a learnable [MASK] token, and the planner uses spatio-temporal attention across all patches and all frames to infer a per-frame planning signal $\mathbf{Z}_\text{cond}$. The Diffusion Generation Model $\mathcal{G}$ is much smaller, about 288M parameters, and operates on high-resolution video latents; masked frames are noised at each diffusion step, reference frames remain clean, and the generator predicts the velocity for masked frames while conditioning on $\mathbf{Z}_\text{cond}$ through cross-attention and MLP conditioning (Liu et al., 2024).

The latent representation follows a standard latent-diffusion pattern. A pre-trained VAE $\mathcal{D}_\text{enc}$ compresses frames into latents with 16 channels and 8× spatial downsampling, and the latents are patchified into tokens. For a video of $K$ frames,

$$\mathbf{Z}_\text{low} = \{z^\text{low}_i\}_{i=1}^K = \mathcal{D}_\text{enc}(\mathbf{X}_\text{low}),$$

and

$$\mathbf{Z}_\text{high} = \{z^\text{high}_i\}_{i=1}^K = \mathcal{D}_\text{enc}(\mathbf{X}_\text{high}),$$

with $N_\text{high} \gg N_\text{low}$. This asymmetric resolution is what makes full spatio-temporal attention affordable in the planner while preserving high-resolution synthesis in the generator.

The planner uses a transformer stack with LLaMA-style blocks, RMSNorm on block inputs, standard multi-head attention with layer norm on projected features for stability, and 2D Rotary Positional Encoding (RoPE) for 3D video. Frames are flattened into a sequence of patches, and a learnable [NEXT] token is inserted to delineate rows. The generator uses DiT-style blocks, AdaIN modulation by timestep embedding in spatial attention blocks, and temporal attention rather than full spatio-temporal attention at high resolution.

This division of labour is the defining systems insight. The planner bears most of the parameter count and the expensive global temporal modelling, but it runs only once per video and only at low resolution. The generator is invoked across diffusion steps, so it is kept lightweight and computationally efficient.

3. Masked autoregressive planning and masked diffusion synthesis

The MAR component is formulated as conditional inference over missing low-resolution frame latents given any observed subset. If $$\mathbf{Z}_\text{low} = (z^\text{low}_1, \dots, z^\text{low}_K)$$ and $\mathbf{Z}_\text{cond}$0 denotes masked frames, the planner approximates

$\mathbf{Z}_\text{cond}$1

Rather than using a strictly causal factorization, it uses bidirectional attention and a masked reconstruction objective on masked positions:

$\mathbf{Z}_\text{cond}$2

After the initial stage, the depatchification projection $\mathbf{Z}_\text{cond}$3 is no longer used, and the planner outputs planning signals rather than reconstructed latents (Liu et al., 2024).

The masking scheme is task-generic. During training, the model randomly selects $\mathbf{Z}_\text{cond}$4 frames to mask, with masked positions varying across examples. This allows a single model to support interpolation, image-to-video, future prediction, slow-motion insertion of intermediate frames, and arbitrary out-of-order conditioning. Because the planner is bidirectional, observed frames can occur at any temporal indices.

The diffusion generator is trained in a standard latent-diffusion velocity-prediction setting. For a clean high-resolution latent $\mathbf{Z}_\text{cond}$5 at timestep $\mathbf{Z}_\text{cond}$6, with $\mathbf{Z}_\text{cond}$7, the velocity is

$\mathbf{Z}_\text{cond}$8

The initial generator warm-up conditions only on an unconditional learnable planning token $\mathbf{Z}_\text{cond}$9:

$\mathcal{G}$0

After warm-up, the planner and generator are trained jointly end-to-end with

$\mathcal{G}$1

A technically important mechanism is Identity Attention. In the diffusion input sequence, [REF] tokens denote clean reference frames and [NOISE] tokens denote noisy masked frames. Identity Attention forces [REF] tokens to behave as an identity mapping rather than attending globally, while [NOISE] tokens retain global attention. This keeps reference frames fixed as conditioning information and prevents instability from mixing clean and noisy distributions at random positions.

4. Progressive training and scaling regime

MarDini is trained with a three-stage procedure. The Initial Stage warms up the MAR planner with $\mathcal{G}$2 and the DM generator with $\mathcal{G}$3 separately. The Joint-Model Stage switches to end-to-end optimization with $\mathcal{G}$4 under a simple interpolation mask regime with many references and few masked frames. The Joint-Task Stage continues end-to-end training while progressively lowering the mask ratio, thereby moving from interpolation toward image-to-video generation, and simultaneously increasing resolution and frame count. The paper characterizes this as a progressive training recipe consisting of a mask ratio curriculum and a resolution/frame-length schedule, enabling training from scratch on unlabeled videos (Liu et al., 2024).

The implementation details are substantial. Training uses 34M videos from Shutterstock, 256 NVIDIA H100 GPUs, AdamW with learning rate $\mathcal{G}$5 and a cosine schedule, plus FSDP and activation checkpointing. Batch size varies with resolution and frame number; examples given are 1024 at 256×256 with 9 frames and 720 at 512×512 with 9 frames. Inference uses DDIM as the noise solver. Classifier-free guidance (CFG) is enabled on the planning signal by replacing $\mathcal{G}$6 with $\mathcal{G}$7 with probability $\mathcal{G}$8 during training, and by combining conditional and unconditional forward passes at inference; a CFG scale of about 2.5 is used for image-to-video, whereas CFG is off for interpolation.

The architecture also evolves during training. In early diffusion pre-training, the DM uses spatio-temporal attention at low resolution because the cost is manageable and it aids convergence. After MAR is integrated and resolution increases, the DM switches to temporal attention only. This evolution is not merely an optimization convenience; it is part of the computational argument for MarDini’s asymmetry.

5. Unified task interface and inference behavior

MarDini’s operational interface is the frame mask. In video interpolation, the first and last frames are observed, frames $\mathcal{G}$9 through $\mathbf{Z}_\text{cond}$0 are masked in MAR, and the corresponding high-resolution latents are treated as [NOISE] in DM while the endpoints remain [REF]. In image-to-video generation, only the first frame is observed and all future frames are masked. In video expansion or temporal upsampling, a prefix or sparse subset of frames is retained and the remainder are generated, possibly hierarchically; the paper reports hierarchical autoregressive generation beginning from 4 frames and using a 32-frame window to generate 128 frames (32× expansion) (Liu et al., 2024).

Because the planner is trained with random masks and uses bidirectional attention, the same checkpoint also supports arbitrary anchor frames at nonconsecutive indices. The paper notes potential 3D-like view interpolation qualitatively, using two object views as first and last frames with novel middle frames inferred in between.

This unified formulation is a contrast to conventional pipelines that train separate models for interpolation and image-to-video, or that rely on explicit motion priors such as optical flow or 3D warps. A plausible implication is that MarDini’s mask-based conditioning acts as a task abstraction layer: the model family is not specialized by architecture at inference time, but by which frames are designated as known and unknown.

6. Empirical performance, efficiency, and ablations

On VIDIM-Bench video interpolation, where the task is to generate 7 intermediate frames from the first and last frame, MarDini reports state-of-the-art FVD. On DAVIS-7, the best baseline FVD is 199.32 for VIDIM, while MarDini-L/ST-256 obtains 102.87 and MarDini-L/ST-512 obtains 99.05. On UCF101-7, the best baseline is 278.00, while MarDini-L/ST-256 obtains 197.69 and MarDini-L/T-512 obtains 198.94. Reconstruction metrics such as SSIM and LPIPS are competitive, but the paper explicitly notes cases where MarDini yields sharper outputs with slightly worse SSIM or LPIPS because smoother images can score better under such metrics (Liu et al., 2024).

On VBench image-to-video evaluation, MarDini-S/T-17 at 512×512 reports 0.46 s/frame latency, I2V Subject consistency 98.76, I2V Background consistency 97.18, Video Quality (w/o D.D.) 88.17, and VBench Avg 90.62. At 768×768, it reports I2V Subj 98.77, I2V Back 96.78, VQ (w/o D.D.) 88.68, and VBench Avg 90.95; at 1024×1024, the reported VBench Avg is approximately 91.13. The paper emphasizes that these results are achieved without text conditioning, motion score guidance, advanced post-training, or image-based pre-training.

Efficiency claims are central rather than ancillary. With asymmetric resolution, MarDini can generate 13 frames at 512×512 in 6.05 s with full asymmetric attention on one A100. The symmetric configuration with planner and generator both at high resolution is reported as out-of-memory at the same setting. Figure 1(a) further shows that FVD saturates quickly with the number of diffusion steps, and the paper highlights 25 DDIM steps as sufficient for good interpolation performance.

Ablation results reinforce the planner–generator decomposition. On VIDIM interpolation FVD, MAR-1B only gives 427.66 on DAVIS and 741.80 on UCF101; MAR-3B only gives 373.03 and 701.03; DM-0.3B only gives 320.89 and 383.04; MAR-1B + DM-0.3B gives 224.07 and 258.08; and MAR-3B + DM-0.3B gives 102.87 and 197.69. The paper’s interpretation is that MAR alone in continuous space is poor for direct generation, a small DM alone lacks long-range temporal planning, and the combination yields the strongest performance.

Additional analyses support specific design choices. Replacing MAR outputs with an unconditional token leaves the DM able to generate plausible individual frames, but motion becomes inconsistent or physically implausible; buildings warp, objects move in wrong directions, and fire flows incorrectly. Training curves show that without Identity Attention, DM training fails to converge, whereas with Identity Attention it stabilizes after warm-up. During resolution scaling from 256 to 512, interpolation and image-to-video metrics improve together, which the paper treats as evidence that the joint curriculum is synergistic rather than antagonistic.

7. Limitations, future directions, and relation to adjacent work

The authors explicitly identify several limitations. MarDini does not include advanced post-training or specialized fine-tuning, even though such stages commonly improve generative models. It does not include a text encoder, motion score conditioning, or dynamic-degree guidance, which limits controllability and excludes text-to-video prompting and explicit camera-motion control. Although the system can generate at up to 1024×1024 and supports long videos via hierarchical autoregression, these regimes were not fully explored or optimized. The paper also notes failure cases in reconstruction metrics and challenging motion scenarios, as well as dependence on the large Shutterstock training corpus (Liu et al., 2024).

The proposed future directions are correspondingly concrete: add text and motion conditioning, pursue more extensive post-training, study 3D-aware generation and novel-view synthesis with explicit camera parameters, and further optimize inference through caching, static compilation, and offloading.

Within the broader field, MarDini sits between two established lines of work. Relative to standard video diffusion models, it does not begin from image pre-training and it places full spatio-temporal attention only in the low-resolution planner rather than in the high-resolution generator. Relative to masked autoregressive visual models, it avoids discrete tokenization and uses MAR only for planning rather than direct pixel prediction. The paper’s own conceptual summary is that MarDini is a planner–executor architecture: a large, low-resolution MAR transformer learns temporally coherent plans via masked reconstruction, and a compact diffusion model executes those plans as high-resolution video synthesis.

This positioning clarifies why MarDini is best understood not merely as another diffusion backbone, but as a systems-level proposal about how temporal reasoning and spatial rendering should be separated in large-scale video generation.