---
title: 'MarDini: Masked Autoregressive Diffusion'
url: https://www.emergentmind.com/topics/mardini
type: topic
---

# MarDini: Masked Autoregressive Diffusion

Searching arXiv for the provided topic and closely related papers.
MarDini is a family of video generative models that combine masked autoregression (MAR) with diffusion models (DM) in a unified, asymmetric architecture. Its central design decouples temporal and spatial modelling: a large MAR planning model operates on low-resolution video latents and learns long-range temporal structure, while a smaller diffusion generation model operates on high-resolution latents and focuses on spatial detail conditioned on the planner’s signals. Because the planner is trained with arbitrary frame masking patterns, a single checkpoint can handle video interpolation, image-to-video generation, video expansion, and hierarchical autoregressive long-video generation; the model is trained from scratch on unlabeled videos without image or text pre-training [2410.20280].

## 1. Nomenclature and scope

In the video-generation literature, **MarDini** is the official name of the system introduced in “MarDini: Masked Autoregressive Diffusion for Video Generation at Scale” [2410.20280]. The name refers specifically to a planner–generator architecture in which MAR handles temporal planning and DM handles high-resolution synthesis.

A separate source of ambiguity arises from mechanism-level drug–drug interaction research. In “MARD: Mirror-Augmented Reasoning Distillation for Mechanism-Level Drug-Drug Interaction Prediction,” the string “MarDini” does not appear explicitly; the paper’s official terms are **MARD** and **MARD-7B**. The accompanying details state that “MarDini” is very plausibly a colloquial or project-internal nickname in that context, but not an official term. This suggests that the name has circulated in at least two distinct settings, with only the video model using it canonically [2606.12578].

Within generative modelling, MarDini addresses a specific systems problem: standard video diffusion models must jointly model temporal structure and spatial detail in a single network, which makes full spatio-temporal attention prohibitively expensive at scale, while discrete vector-quantized autoregressive alternatives remain capacity-limited and unstable for continuous video signals. MarDini’s response is architectural asymmetry rather than a single monolithic backbone.

## 2. Asymmetric planner–generator architecture

MarDini has two principal components. The **MAR Planning Model** $\mathcal{P}$ is large, with configurations ranging from 1.3B to 3.1B parameters, and operates on low-resolution video latents. Some frames are replaced with a learnable `[MASK]` token, and the planner uses spatio-temporal attention across all patches and all frames to infer a per-frame planning signal $\mathbf{Z}_\text{cond}$. The **Diffusion Generation Model** $\mathcal{G}$ is much smaller, about 288M parameters, and operates on high-resolution video latents; masked frames are noised at each diffusion step, reference frames remain clean, and the generator predicts the velocity for masked frames while conditioning on $\mathbf{Z}_\text{cond}$ through cross-attention and MLP conditioning [2410.20280].

The latent representation follows a standard latent-diffusion pattern. A pre-trained VAE $\mathcal{D}_\text{enc}$ compresses frames into latents with **16 channels** and **8× spatial downsampling**, and the latents are patchified into tokens. For a video of $K$ frames,
$$
\mathbf{Z}_\text{low} = \{z^\text{low}_i\}_{i=1}^K = \mathcal{D}_\text{enc}(\mathbf{X}_\text{low}),
$$
and
$$
\mathbf{Z}_\text{high} = \{z^\text{high}_i\}_{i=1}^K = \mathcal{D}_\text{enc}(\mathbf{X}_\text{high}),
$$
with $N_\text{high} \gg N_\text{low}$. This asymmetric resolution is what makes full spatio-temporal attention affordable in the planner while preserving high-resolution synthesis in the generator.

The planner uses a transformer stack with **LLaMA-style blocks**, **RMSNorm** on block inputs, standard multi-head attention with **layer norm on projected features** for stability, and **2D Rotary Positional Encoding (RoPE)** for 3D video. Frames are flattened into a sequence of patches, and a learnable `[NEXT]` token is inserted to delineate rows. The generator uses **DiT-style blocks**, **AdaIN** modulation by timestep embedding in spatial attention blocks, and **temporal attention** rather than full spatio-temporal attention at high resolution.

This division of labour is the defining systems insight. The planner bears most of the parameter count and the expensive global temporal modelling, but it runs only once per video and only at low resolution. The generator is invoked across diffusion steps, so it is kept lightweight and computationally efficient.

## 3. Masked autoregressive planning and masked diffusion synthesis

The MAR component is formulated as conditional inference over missing low-resolution frame latents given any observed subset. If $\mathbf{Z}_\text{low} = (z^\text{low}_1, \dots, z^\text{low}_K)$ and $M \subset \{1,\dots,K\}$ denotes masked frames, the planner approximates
$$
p_\phi\big(\{z^\text{low}_i\}_{i \in M} \mid \{z^\text{low}_j\}_{j \notin M}\big).
$$
Rather than using a strictly causal factorization, it uses bidirectional attention and a masked reconstruction objective on masked positions:
$$
\mathcal{L}^{\phi, \zeta}_\text{MAR} = \big\| \mathbf{M} \cdot \mathbf{Z}_\text{low}
  - \mathbf{M} \cdot f_\zeta\big(\mathcal{P}_\phi(\mathbf{Z}_\text{low}^\text{mask})\big)
 \big\|_2^2.
$$
After the initial stage, the depatchification projection $f_\zeta$ is no longer used, and the planner outputs planning signals rather than reconstructed latents [2410.20280].

The masking scheme is task-generic. During training, the model randomly selects $K' < K$ frames to mask, with masked positions varying across examples. This allows a single model to support interpolation, image-to-video, future prediction, slow-motion insertion of intermediate frames, and arbitrary out-of-order conditioning. Because the planner is bidirectional, observed frames can occur at any temporal indices.

The diffusion generator is trained in a standard latent-diffusion velocity-prediction setting. For a clean high-resolution latent $z^\text{high}_i$ at timestep $t$, with $\epsilon \sim \mathcal{N}(0,I)$, the velocity is
$$
v_i^t = \alpha_t \epsilon - \sigma_t z^\text{high}_i.
$$
The initial generator warm-up conditions only on an unconditional learnable planning token $\mathbf{Z}_\text{uncond}$:
$$
\mathcal{L}_\text{DM}^\theta = \big\| \mathbf{M} \cdot \mathbf{V}^t
        - \mathbf{M} \cdot
          \mathcal{G}_\theta(\mathbf{Z}_\text{high}^{\text{noise}, t},
                             \mathbf{Z}_\text{uncond}, t)
   \big\|_2^2.
$$
After warm-up, the planner and generator are trained jointly end-to-end with
$$
\mathcal{L}_\text{MDiff}^{\theta,\phi} = \big\| \mathbf{M} \cdot \mathbf{V}^t
        - \mathbf{M} \cdot
          \mathcal{G}_\theta(\mathbf{Z}_\text{high}^{\text{noise}, t},
                             \mathcal{P}_\phi(\mathbf{Z}_\text{low}^\text{mask}), t)
   \big\|_2^2.
$$

A technically important mechanism is **Identity Attention**. In the diffusion input sequence, `[REF]` tokens denote clean reference frames and `[NOISE]` tokens denote noisy masked frames. Identity Attention forces `[REF]` tokens to behave as an identity mapping rather than attending globally, while `[NOISE]` tokens retain global attention. This keeps reference frames fixed as conditioning information and prevents instability from mixing clean and noisy distributions at random positions.

## 4. Progressive training and scaling regime

MarDini is trained with a three-stage procedure. The **Initial Stage** warms up the MAR planner with $\mathcal{L}_\text{MAR}$ and the DM generator with $\mathcal{L}_\text{DM}$ separately. The **Joint-Model Stage** switches to end-to-end optimization with $\mathcal{L}_\text{MDiff}$ under a simple interpolation mask regime with many references and few masked frames. The **Joint-Task Stage** continues end-to-end training while progressively lowering the mask ratio, thereby moving from interpolation toward image-to-video generation, and simultaneously increasing resolution and frame count. The paper characterizes this as a progressive training recipe consisting of a mask ratio curriculum and a resolution/frame-length schedule, enabling training from scratch on unlabeled videos [2410.20280].

The implementation details are substantial. Training uses **34M videos from Shutterstock**, **256 NVIDIA H100 GPUs**, **AdamW** with learning rate $1.4\times 10^{-4}$ and a cosine schedule, plus **FSDP** and activation checkpointing. Batch size varies with resolution and frame number; examples given are **1024** at **256×256 with 9 frames** and **720** at **512×512 with 9 frames**. Inference uses **DDIM** as the noise solver. **Classifier-free guidance (CFG)** is enabled on the planning signal by replacing $\mathbf{Z}_\text{cond}$ with $\mathbf{Z}_\text{uncond}$ with probability $1/10$ during training, and by combining conditional and unconditional forward passes at inference; a CFG scale of about **2.5** is used for image-to-video, whereas CFG is off for interpolation.

The architecture also evolves during training. In early diffusion pre-training, the DM uses **spatio-temporal attention** at low resolution because the cost is manageable and it aids convergence. After MAR is integrated and resolution increases, the DM switches to **temporal attention only**. This evolution is not merely an optimization convenience; it is part of the computational argument for MarDini’s asymmetry.

## 5. Unified task interface and inference behavior

MarDini’s operational interface is the frame mask. In **video interpolation**, the first and last frames are observed, frames $2$ through $K-1$ are masked in MAR, and the corresponding high-resolution latents are treated as `[NOISE]` in DM while the endpoints remain `[REF]`. In **image-to-video generation**, only the first frame is observed and all future frames are masked. In **video expansion** or temporal upsampling, a prefix or sparse subset of frames is retained and the remainder are generated, possibly hierarchically; the paper reports hierarchical autoregressive generation beginning from **4 frames** and using a **32-frame window** to generate **128 frames (32× expansion)** [2410.20280].

Because the planner is trained with random masks and uses bidirectional attention, the same checkpoint also supports arbitrary anchor frames at nonconsecutive indices. The paper notes potential 3D-like view interpolation qualitatively, using two object views as first and last frames with novel middle frames inferred in between.

This unified formulation is a contrast to conventional pipelines that train separate models for interpolation and image-to-video, or that rely on explicit motion priors such as optical flow or 3D warps. A plausible implication is that MarDini’s mask-based conditioning acts as a task abstraction layer: the model family is not specialized by architecture at inference time, but by which frames are designated as known and unknown.

## 6. Empirical performance, efficiency, and ablations

On **VIDIM-Bench** video interpolation, where the task is to generate **7 intermediate frames** from the first and last frame, MarDini reports state-of-the-art **FVD**. On **DAVIS-7**, the best baseline FVD is **199.32** for VIDIM, while **MarDini-L/ST-256** obtains **102.87** and **MarDini-L/ST-512** obtains **99.05**. On **UCF101-7**, the best baseline is **278.00**, while **MarDini-L/ST-256** obtains **197.69** and **MarDini-L/T-512** obtains **198.94**. Reconstruction metrics such as SSIM and LPIPS are competitive, but the paper explicitly notes cases where MarDini yields sharper outputs with slightly worse SSIM or LPIPS because smoother images can score better under such metrics [2410.20280].

On **VBench** image-to-video evaluation, **MarDini-S/T-17** at **512×512** reports **0.46 s/frame** latency, **I2V Subject consistency 98.76**, **I2V Background consistency 97.18**, **Video Quality (w/o D.D.) 88.17**, and **VBench Avg 90.62**. At **768×768**, it reports **I2V Subj 98.77**, **I2V Back 96.78**, **VQ (w/o D.D.) 88.68**, and **VBench Avg 90.95**; at **1024×1024**, the reported **VBench Avg** is approximately **91.13**. The paper emphasizes that these results are achieved without text conditioning, motion score guidance, advanced post-training, or image-based pre-training.

Efficiency claims are central rather than ancillary. With asymmetric resolution, MarDini can generate **13 frames at 512×512 in 6.05 s** with full asymmetric attention on one **A100**. The symmetric configuration with planner and generator both at high resolution is reported as **out-of-memory** at the same setting. Figure 5(a) further shows that FVD saturates quickly with the number of diffusion steps, and the paper highlights **25 DDIM steps** as sufficient for good interpolation performance.

Ablation results reinforce the planner–generator decomposition. On VIDIM interpolation FVD, **MAR-1B only** gives **427.66** on DAVIS and **741.80** on UCF101; **MAR-3B only** gives **373.03** and **701.03**; **DM-0.3B only** gives **320.89** and **383.04**; **MAR-1B + DM-0.3B** gives **224.07** and **258.08**; and **MAR-3B + DM-0.3B** gives **102.87** and **197.69**. The paper’s interpretation is that MAR alone in continuous space is poor for direct generation, a small DM alone lacks long-range temporal planning, and the combination yields the strongest performance.

Additional analyses support specific design choices. Replacing MAR outputs with an unconditional token leaves the DM able to generate plausible individual frames, but motion becomes inconsistent or physically implausible; buildings warp, objects move in wrong directions, and fire flows incorrectly. Training curves show that without Identity Attention, DM training fails to converge, whereas with Identity Attention it stabilizes after warm-up. During resolution scaling from **256** to **512**, interpolation and image-to-video metrics improve together, which the paper treats as evidence that the joint curriculum is synergistic rather than antagonistic.

## 7. Limitations, future directions, and relation to adjacent work

The authors explicitly identify several limitations. MarDini does not include advanced post-training or specialized fine-tuning, even though such stages commonly improve generative models. It does **not** include a text encoder, motion score conditioning, or dynamic-degree guidance, which limits controllability and excludes text-to-video prompting and explicit camera-motion control. Although the system can generate at up to **1024×1024** and supports long videos via hierarchical autoregression, these regimes were not fully explored or optimized. The paper also notes failure cases in reconstruction metrics and challenging motion scenarios, as well as dependence on the large **Shutterstock** training corpus [2410.20280].

The proposed future directions are correspondingly concrete: add **text and motion conditioning**, pursue more extensive **post-training**, study **3D-aware generation** and novel-view synthesis with explicit camera parameters, and further optimize inference through caching, static compilation, and offloading.

Within the broader field, MarDini sits between two established lines of work. Relative to standard video diffusion models, it does not begin from image pre-training and it places full spatio-temporal attention only in the low-resolution planner rather than in the high-resolution generator. Relative to masked autoregressive visual models, it avoids discrete tokenization and uses MAR only for planning rather than direct pixel prediction. The paper’s own conceptual summary is that MarDini is a **planner–executor architecture**: a large, low-resolution MAR transformer learns temporally coherent plans via masked reconstruction, and a compact diffusion model executes those plans as high-resolution video synthesis.

This positioning clarifies why MarDini is best understood not merely as another diffusion backbone, but as a systems-level proposal about how temporal reasoning and spatial rendering should be separated in large-scale video generation.

Source: https://www.emergentmind.com/topics/mardini