MiraMo: Diffusion Transformer for Image Animation

• The paper introduces MiraMo, a framework that synthesizes smooth, appearance-consistent videos from a single image by leveraging motion residual learning and controllable dynamics.
• It integrates a diffusion transformer backbone with linear attention and a DCT-based noise refinement strategy to mitigate appearance drift and abrupt motion transitions.
• The method addresses key challenges in image animation and demonstrates improved efficiency and performance on standard video generation benchmarks.

MiraMo is a framework for image animation that synthesizes a video from a single static image while preserving appearance consistency, producing smooth motion, and allowing explicit control over motion magnitude. It is formulated as a motion linear diffusion transformer that combines a foundational text-to-video diffusion transformer with efficient linear attention, a motion residual learning paradigm in latent space, and a DCT-based noise refinement strategy during inference (Ma et al., 10 Aug 2025). The method is positioned against three persistent difficulties in image animation—appearance drift, abrupt motion transitions, and the computational cost of vanilla Transformer self-attention—and is evaluated on standard video generation benchmarks as well as motion transfer and video editing tasks (Ma et al., 10 Aug 2025).

1. Research context and problem formulation

Image animation, in this formulation, seeks to synthesize a video from a single static image while preserving the appearance of the input across frames and producing smooth, realistic motion aligned with optional text prompts (Ma et al., 10 Aug 2025). The paper identifies three core sources of difficulty. First, appearance consistency is easily lost: models can drift in color tone and structural details, yielding identity loss, background flicker, or style drift. Second, temporal dynamics can contain sudden jumps, deformations, or “turnarounds” caused by poor temporal bias, noise mismatch, or insufficient low-frequency guidance during sampling. Third, efficiency remains a central systems constraint because video generation is intrinsically high-memory and high-compute, and naïve self-attention has $O(N^2)$ complexity in sequence length $N$ (Ma et al., 10 Aug 2025).

Within this context, MiraMo is presented as a shift away from U-Net-based image animation pipelines toward a text-to-video diffusion transformer backbone. The paper argues that Transformer-based video generation has already demonstrated strong performance in text-to-video generation, but that image animation has continued to rely heavily on U-Net-based diffusion models. MiraMo addresses this gap by introducing linear attention, residualized motion modeling, and an inference-time mechanism for suppressing abrupt temporal artifacts (Ma et al., 10 Aug 2025).

A plausible implication is that MiraMo is not merely an efficiency optimization over prior backbones; it is a reformulation of the image animation objective itself. By concentrating modeling capacity on motion rather than repeated frame reconstruction, it redefines the balance between appearance anchoring and temporal expressiveness.

2. Architectural composition

The overall pipeline takes as input a static image $x_1 \in \mathbb{R}^{C \times H \times W}$ and optionally a text prompt $p$, and generates a 16-frame clip $V = \{x_1, x_2, \dots, x_N\}$ at $512 \times 512$ resolution with $N=16$ (Ma et al., 10 Aug 2025). Each frame is encoded by a VAE encoder $\mathcal{E}$ into a latent $z \in \mathbb{R}^{c \times h \times w}$. MiraMo extends SANA’s linear diffusion transformer with linear self-attention, temporal linear attention compatible with RoPE, a Mix-FFN implemented with efficient 2D convolutional layers, and a dynamics control embedding added to the timestep embedding and injected via a single AdaIN layer uniformly across frames (Ma et al., 10 Aug 2025).

The conditioning structure is tripartite. The first-frame latent $z_1$ anchors appearance. Text conditioning is provided through standard cross-attention to a frozen or fine-tuned text encoder, using classifier-free guidance with scale $N$0 at inference. Motion magnitude is modulated by a dynamics bucket $N$1 derived from MS-SSIM-based motion degree (Ma et al., 10 Aug 2025).

The paper evaluates two temporal attention strategies. Frame-axis temporal attention, described as Latte/LaVie-style, applies attention along the frame dimension only and is reported to yield better temporal consistency and stability. Fully 3D token attention, described as CogVideoX/LTX/HunyuanVideo-style, spans all spatio-temporal tokens with 3D RoPE but underperforms the frame-axis strategy in the reported consistency metrics (Ma et al., 10 Aug 2025).

Component	Specification	Function
Input/output	Static image plus optional text; 16 frames at $N$2	Image-to-video generation
Backbone	SANA-like linear diffusion transformer	Core denoising and temporal modeling
Temporal module	Temporal linear attention compatible with RoPE	Frame-wise temporal dependency modeling
Feed-forward block	Mix-FFN with efficient 2D convolutional layers	Spatial inductive bias
Motion control	Dynamics bucket $N$3 injected via a single AdaIN layer	Control of motion magnitude
Text conditioning	Cross-attention with classifier-free guidance scale $N$4	Prompt-conditioned motion

This architecture is explicitly designed to preserve the static input appearance while allocating model capacity to motion. The paper’s formulation therefore ties architectural choice directly to the stated failure modes of image animation: appearance drift, temporal instability, and computational overhead (Ma et al., 10 Aug 2025).

3. Motion residual learning and controllable dynamics

A central design choice in MiraMo is motion residual learning in latent space. Rather than predicting full future frames, the model defines motion residuals relative to the first frame:

$N$5

The paper contrasts this with a chained residual definition such as $N$6 and reports that the chained formulation degrades rapidly because of error accumulation (Ma et al., 10 Aug 2025).

This residualization is embedded in a rectified-flow training setup. For a clean latent $N$7, flow matching uses

$N$8

with loss

$N$9

For MiraMo specifically, residuals are diffused to $x_1 \in \mathbb{R}^{C \times H \times W}$0, then added to $x_1 \in \mathbb{R}^{C \times H \times W}$1 to produce per-frame features $x_1 \in \mathbb{R}^{C \times H \times W}$2, concatenated with $x_1 \in \mathbb{R}^{C \times H \times W}$3 as frame 1 to form $x_1 \in \mathbb{R}^{C \times H \times W}$4, and optimized with

$x_1 \in \mathbb{R}^{C \times H \times W}$5

The paper states that this reduces redundancy, improves temporal consistency, and better aligns motion to descriptions without sacrificing appearance fidelity of the static input image (Ma et al., 10 Aug 2025).

Controllability is introduced through a dynamics degree control. A motion smoothness or expression score is computed as

$x_1 \in \mathbb{R}^{C \times H \times W}$6

and bucketized into $x_1 \in \mathbb{R}^{C \times H \times W}$7. To mitigate long-tail distributions, clips are sampled with random frame intervals between $x_1 \in \mathbb{R}^{C \times H \times W}$8 and $x_1 \in \mathbb{R}^{C \times H \times W}$9 during training. The bucket embedding is added to the timestep embedding and injected via a single AdaIN layer uniformly across frames (Ma et al., 10 Aug 2025).

At inference, the paper states that a low $p$0 yields small residual magnitudes and smoother motion, whereas a high $p$1 encourages larger motions and faster dynamics. An optional scaling rule can further adjust motion magnitude:

$p$2

with $p$3 linearly mapped in $p$4 (Ma et al., 10 Aug 2025). This suggests that controllability is treated as a learned conditioning variable rather than a post hoc stabilization heuristic.

4. Linear attention and DCT-based noise refinement

MiraMo’s efficiency claim is grounded in its replacement of vanilla self-attention with linear attention. Standard self-attention is written as

$p$5

which has $p$6 time and memory in sequence length $p$7 (Ma et al., 10 Aug 2025). MiraMo instead adopts a cosine-similarity-based linear formulation compatible with RoPE:

$p$8

and

$p$9

The paper highlights three shared quantities,

$V = \{x_1, x_2, \dots, x_N\}$0

which are computed once per attention layer and reused across queries, yielding $V = \{x_1, x_2, \dots, x_N\}$1 per head rather than $V = \{x_1, x_2, \dots, x_N\}$2 interactions (Ma et al., 10 Aug 2025). The denominator includes a constant $V = \{x_1, x_2, \dots, x_N\}$3 term for numerical stability, and RoPE is applied before normalization so that vector magnitudes remain unchanged.

The second major inference-time component is DCTInit, a 3D discrete cosine transform-based refinement of initialization noise. The paper argues that FFT-based low-frequency injection suffers from spectral leakage due to periodic assumption and poor energy concentration. MiraMo instead computes

$V = \{x_1, x_2, \dots, x_N\}$4

and then forms

$V = \{x_1, x_2, \dots, x_N\}$5

where $V = \{x_1, x_2, \dots, x_N\}$6 is a low-pass filter mask selecting low frequencies of the image latent along spatial and temporal axes (Ma et al., 10 Aug 2025). The refined noise $V = \{x_1, x_2, \dots, x_N\}$7 is then used to initialize sampling.

In the paper’s interpretation, DCTInit suppresses spectral leakage and prevents abrupt temporal artifacts, while the dynamics control module balances motion smoothness and expressiveness. Together with linear attention, these mechanisms define MiraMo’s claim to simultaneously improve consistency, controllability, and efficiency (Ma et al., 10 Aug 2025).

5. Training regime, benchmarks, and empirical profile

The reported training setup combines large-scale pretraining and synthetic post-training. Pretraining uses a subset of Vimeo25M of approximately $V = \{x_1, x_2, \dots, x_N\}$8 videos, with 16 frames at $V = \{x_1, x_2, \dots, x_N\}$9 and frame interval randomly sampled in $512 \times 512$0. Post-training uses approximately $512 \times 512$1 synthetic videos generated by HunyuanVideo and Wan2.1. For image animation training, each clip uses the first frame as the static input, and the model is trained to denoise residuals for the subsequent 15 frames (Ma et al., 10 Aug 2025).

Optimization uses Adam on $512 \times 512$2 NVIDIA A800 (80 GB) with total batch size 56. The schedule follows a logit-normal timestep schedule, as in SD3 rectified flow, and inference uses Flow-DPM-Solver with 25 steps. The implementation retains classifier-free guidance and the SANA-like block structure with linear self-attention, temporal linear attention compatible with RoPE, Mix-FFN with 2D convolution, a single AdaIN for $512 \times 512$3 injection, and cross-attention to text (Ma et al., 10 Aug 2025).

Evaluation is reported on UCF-101, MSR-VTT, and VBench text-to-video and image-to-video dimensions, together with a user study involving 10 raters assessing motion smoothness, motion reasonableness, subject consistency, background consistency, and dynamic degree (Ma et al., 10 Aug 2025).

Benchmark	Reported result for MiraMo	Notes
UCF-101	IS = 59.64, FID = 12.31, FVD = 201.90	IS and FID reported as best
MSR-VTT	CLIPSIM = 0.2903, FVD = 93.91	CLIPSIM reported as best
VBench T2V	Highest average score, parameter count 0.74B	Outperforms listed baselines under similar resource regimes
User study	Highest “good” proportions in four categories, fewest “bad” votes overall	Categories include motion smoothness and consistency

The ablation results are closely tied to the method’s design claims. Motion residual learning improves image-to-video average consistency over direct frame prediction. DCTInit is reported to surpass FFT-based low-frequency injection, mitigating color drift and suppressing abrupt motions. The combined MRL+DCT configuration yields the best average consistency and dynamic degree. Increasing the dynamics bucket $512 \times 512$4 visibly increases motion magnitude while preserving consistency, and synthetic post-training data improves overall VBench scores compared to comparable real-data SFT (Ma et al., 10 Aug 2025).

These results are used in the paper to support two linked conclusions: that the residual formulation improves temporal coherence without sacrificing appearance fidelity, and that the linear-attention transformer can achieve near-vanilla-attention quality with lower compute and memory (Ma et al., 10 Aug 2025).

6. Applications, limitations, and nomenclatural scope

MiraMo is presented not only as an image animation model but also as a basis for motion transfer and video editing. In motion transfer, rectified flow inversion is used to extract the initial inference noise corresponding to the residuals of a source video; the first frame can then be replaced with an edited or modified image, and the recovered noise together with the modified first frame is passed to MiraMo to transfer motion to new visual content. The paper states that this works with videos from any source, real or generated, and at various resolutions (Ma et al., 10 Aug 2025). In video editing, the first frame may be modified with an off-the-shelf image editor or text-driven image editing system, after which MiraMo preserves appearance and transfers the original motion patterns. Prompt-based motion control is also retained: textual prompts modulate type and trajectory of motion, while the bucket $512 \times 512$5 adjusts speed and magnitude (Ma et al., 10 Aug 2025).

The reported limitations are specific. Because the system uses an image-level VAE encoder-decoder rather than an explicit temporal VAE, it may induce flicker. The paper also notes that it focuses on SFT, and suggests that preference-based fine-tuning such as DPO could further improve motion plausibility and aesthetic quality. A further trade-off is that pushing dynamic degree too high may reintroduce abrupt changes. Longer sequences, higher resolutions, richer controllability via pose, flow, or trajectory, and explicit multi-axis motion controllers are identified as future directions, while data biases and rare motion types remain challenging (Ma et al., 10 Aug 2025).

The name “MiraMo” also appears in a distinct sense in work on medical retrieval-augmented generation, where it is used as a concise synthesis for MIRA’s multimodal medical RAG capability: joint encoding of images and rewritten queries, retrieval from a combined offline knowledge base and real-time online sources, and refinement through a Rethinking and Rearrangement module (Wang et al., 10 Jul 2025). That usage is unrelated to MIRMOS, the Magellan InfraRed Multi-Object Spectrograph, whose image-slicer IFU and freeform optical design belong to astronomical instrumentation rather than generative modeling (Cosens et al., 2024).