AniMatrix: An Anime Video Generation Model that Thinks in Art, Not Physics

Abstract: Video generation models internalize physical realism as their prior. Anime deliberately violates physics: smears, impact frames, chibi shifts; and its thousands of coexisting artistic conventions yield no single "physics of anime" a model can absorb. Physics-biased models therefore flatten the artistry that defines the medium or collapse under its stylistic variance. We present AniMatrix, a video generation model that targets artistic rather than physical correctness through a dual-channel conditioning mechanism and a three-step transition: redefine correctness, override the physics prior, and distinguish art from failure. First, a Production Knowledge System encodes anime as a structured taxonomy of controllable production variables (Style, Motion, Camera, VFX), and AniCaption infers these variables from pixels as directorial directives. A trainable tag encoder preserves the field-value structure of this taxonomy while a frozen T5 encoder handles free-form narrative; dual-path injection (cross-attention for fine-grained control, AdaLN modulation for global enforcement) ensures categorical directives are never diluted by open-ended text. Second, a style-motion-deformation curriculum transitions the model from near-physical motion to full anime expressiveness. Third, deformation-aware preference optimization with a domain-specific reward model separates intentional artistry from pathological collapse. On an anime-specific human evaluation with five production dimensions scored by professional animators, AniMatrix ranks first on four of five, with the largest gains over Seedance-Pro 1.0 on Prompt Understanding (+0.70, +22.4 percent) and Artistic Motion (+0.55, +16.9 percent). We will publicly release the AniMatrix model weights and inference code.

• The paper introduces a Production Knowledge System that shifts video synthesis from physical realism to artistic correctness in anime.
• It details a dual-channel conditioning method that fuses structured production tags with free-form text for precise directorial control.
• Empirical results show enhanced prompt adherence, artistic motion, and faster inference through a staged curriculum and distilled model.

AniMatrix: Shifting the Paradigm to Artistic Correctness in Anime Video Generation

Introduction

AniMatrix addresses a fundamental misalignment in video generation: conventional diffusion-based models, such as Sora and Wan2.2, rely inherently on the physical regularities governing natural video. This inductive bias toward physical realism is maladapted to anime, a domain where stylistic conventions intentionally break physics—through techniques like smears, impact frames, chibi deformations, and non-uniform timing. Given anime's immense and contradictory artistic variance, physical-prior-based models either suppress expressivity, yielding overly smoothed outputs, or collapse in the face of this diversity. AniMatrix reframes the generative objective from physical correctness to a structured, production-aware paradigm: artistic correctness.

Problem Formulation and Production Knowledge System

Unlike natural video, where every sample encodes immutable physical laws, anime encompasses myriad, coexisting artistic logics. No universal implicit prior is naturally attainable. AniMatrix tackles this heterogeneity by implementing a Production Knowledge System (PKS) that organizes the anime space into a controllable four-axis taxonomy:

$$\mathcal{T} = \mathcal{S} \times \mathcal{M} \times \mathcal{C} \times \mathcal{V}$$

where $\mathcal{S}$ is Style, $\mathcal{M}$ is Motion, $\mathcal{C}$ is Camera, and $\mathcal{V}$ is VFX. This taxonomy goes beyond conventional semantic captioning by encoding production decisions as directorial directives, converting pixel-level observations into an actionable, discrete control space.

Data curation leverages this taxonomy via domain-balanced sampling and semi-automated pipelines, resulting in a multi-tier dataset. High-quality, expert-verified S-tier clips and dynamic rebalancing (downweighting long-tail bias) ensure coverage of rare yet professionally coherent combinations (e.g., "Shinkai style" $\times$ "2D combat"). The semantic tag space is rendered actionable through AniCaption, a multimodal graph-enhanced captioning model that infers structured tags and rewrites them into a hybrid, human-readable and machine-parsable format.

Dual-Channel Creator-Language Conditioning

Standard generative models conflate directive conditioning into flat text representations, which is both architecturally lossy and semantically ambiguous. AniMatrix introduces dual-channel conditioning (Figure 1):

(Figure 1)

Figure 1: The architecture’s dual-channel creator-language conditioning: production tags are processed via a dedicated tag encoder and fused with free-form text, with injection points at both cross-attention and global AdaLN modulation.

• The tag channel is parsed via a trainable Transformer encoder operating on (field, value) pairs, preserving orthogonality across $\mathcal{T}$ and generating both per-tag and global summary embeddings.
• The text channel remains handled by a frozen umT5-XXL encoder, supporting nuanced, open-ended narrative directives.

This decoupling enables strict enforcement of categorical production directives via AdaLN at every block and flexible artistic intent through cross-attention. Robustness is injected via stochastic dropout (randomly eliminating one or both conditioning channels), partial tag drops, synonym augmentation, and explicit tag–text conflict exposure (with tags as hard constraints in the loss).

Curriculum and Alignment: Training for Artistic Correctness

The training pipeline is staged:

1. Continue-Training (CT): Adaptation from a physics-pretrained Wan2.2 initialization onto broad anime data, leveraging a blend of T2I, T2V, and I2V tasks with resolution/duration upscaling.
2. Supervised Fine-Tuning (SFT): Curriculum-guided exposure along three axes—style heterogeneity, motion amplitude, deformation intensity—transitions the model from physical-feasible to full-spectrum artistic motion, preventing collapse by incrementally bridging distribution gaps.
3. Quality Tuning (QT): Final refinement using only expert-accepted S-tier clips, focusing on critical characteristics (line stability, color, per-frame coherence) at production resolutions.
4. Deformation-Aware Direct Preference Optimization (DPO): A reward model, trained on human-annotated pairs and decomposed along structural axes (facial topology, anatomical, line continuity, and motion coherence), distinguishes intentional artistic exaggeration from structural collapse. DPO maximizes the likelihood gap between preferred and rejected outputs, employing an in-process approximation to log-likelihood differentials.

Empirical Evaluation

Human evaluation replaces automated metrics (FVD, CLIP) due to their anti-correlation with artistic dimensions in anime. Professional animation directors rate models across five axes: style fidelity, prompt understanding, artistic motion, structural stability, and anime aesthetic. AniMatrix achieves strong leadership in prompt understanding ($+$0.70, $+$22.4%) and artistic motion ($+$0.55, $\mathcal{S}$016.9%) over Seedance-Pro 1.0, with near-parity in foundational visual metrics. These quantitative results are supported by qualitative evidence:

Figure 2: On a high-momentum sakuga sequence (top), AniMatrix renders sharp energy beams and controlled pose; Wan2.2 collapses beams via motion blur, Seedance-Pro 1.0 omits VFX and fails to maintain actor presence. On complex group choreography with coupled FX (bottom), AniMatrix preserves formation, shield, and fireball timing, while baselines show VFX drift and loss of spatial precision.

The ablation study demonstrates that these gains are directly attributable to dual-channel conditioning and artistic-prior curriculum; substituting with standard flat caption conditioning yields statistically significant performance drops in prompt adherence and expressive motion.

Inference and Production Deployment

AniMatrix addresses inference latency via Distribution Matching Distillation (DMD). The 40-step teacher model is distilled to an 8-step student, with dual guidance (text+tag) compressed into the student weights. DMD not only achieves a $\mathcal{S}$1 speedup (from 577s to 57s for a $\mathcal{S}$2, 5s I2V clip) but regularizes away rare structural defects, slightly improving structural stability and line-art quality over the teacher on a held-out set.

The system demonstrates substantial uptake in production: deployed in Tencent's Workrally platform, it's adopted by 60+ studios and achieves the highest download rate among available anime-generation systems.

Implications and Future Directions

AniMatrix repositions the generative objective in stylized domains—optimizing for explicit artistic correctness, not physically plausible realism—and evidences that such priors are both learnable and deployable at production scale. The separation of production control (via symbolic schema) and narrative intent (via free-form text) matches directorial workflows, supporting nuanced creation beyond the hard-coded limits of current diffusion systems.

Remaining gaps include truly multimodal conditioning (ingesting character sheets, storyboards, and audio natively), enhanced timing and effect rendering control, and test-time directorial reasoning for shot planning. The announced AniMatrix-Uni is designed to close these gaps by fusing multimodal assets into a unified conditioning space and operationalizing shot-level planning in the optimization loop.

Conclusion

AniMatrix establishes that artistic correctness—modeled via explicit production schema and enforced through curriculum and DPO—is an independent, trainable, and impactful axis for video generation, orthogonal to the physical priors dominating previous models. Its contributions, spanning data curation, architecture, training, evaluation, and deployment, set a new operational standard for controllable, expressive anime video synthesis.

---

Reference: "AniMatrix: An Anime Video Generation Model that Thinks in Art, Not Physics" (2605.03652)