Chimera: Designing and Chinchilla-Scaling Hybrid Visual Diffusion Transformers

Abstract: Visual generation increasingly requires high-resolution images, long videos, and multimodal context, making the quadratic cost of full attention prohibitive. We introduce Chimera, a hybrid visual diffusion backbone with a principled scaling recipe. Chimera processes text, image, and video tokens in one raster-ordered stream without positional embeddings. It combines Kimi Delta Attention (KDA) for long-context state tracking with O(N) complexity, interleaved Multi-head Latent Attention (MLA) for direct global interaction, and modality-aware short convolutions for local spatiotemporal context. Sparse Mixture-of-Experts (MoE) layers expand capacity while controlling activated compute. To scale this heterogeneous architecture, we introduce HeteroP, a module-wise scheme that transfers hyperparameters across width and depth according to each tensor's functional fan-in and model depth. HeteroP yields a consistently tuned family used to fit Chinchilla-style compute-optimal laws for activated model size, training-token count, and image-video data ratio. Guided by these laws, we train an 11B-parameter Chimera with 2B activated parameters. Experiments show three results. First, measured by pretraining diffusion loss, the dense backbone is 1.7x as compute-efficient as a matched full-attention Wan-2.1 2B baseline, while the complete system reaches 7.3x. Second, without length-specific fine-tuning, Chimera extrapolates zero-shot from 5-second training clips to 30-second videos, with only 6.5% FID degradation in the last five seconds. Third, the fitted laws show that compute-optimal image pretraining divides compute nearly evenly between activated model size and training-token count, whereas video pretraining modestly favors model size at higher budgets. These results establish a foundation for designing and scaling efficient long-context diffusion architectures.

• The paper presents a hybrid visual diffusion Transformer that improves scalability with Kimi Delta Attention and modality-aware convolutions.
• It introduces HeteroP, a module-wise hyperparameter transfer method, enabling robust scaling from 59M to 19.3B parameters.
• Empirical results show up to 7.3× compute efficiency gains and strong zero-shot video generation with minimal FID degradation.

Hybrid-Linear-Global Visual Diffusion Transformers: Architecture, Scaling, and Multimodal Generation

Introduction

"Chimera: Designing and Chinchilla-Scaling Hybrid Visual Diffusion Transformers" (2607.28611) presents FutureOrange, a visual diffusion Transformer backbone specifically designed for the token-extensive regime in vision—spanning high-resolution images, long videos, and unified multimodal contexts. The work addresses architectural and methodological gaps that emerged as visual generative models increasingly require long-range context modeling, efficient scaling, and seamless integration across text, image, and video modalities.

Architectural Innovations

FutureOrange introduces a hybrid single-stream Transformer backbone with three complementary mechanisms: Kimi Delta Attention (KDA) for efficient long-context state tracking, Multi-head Latent Attention (MLA) for global bidirectional token interaction, and modality-aware short convolutions for local spatiotemporal inductive bias. These are unified in a single token sequence processed without positional embeddings, enabling flexible context extrapolation and compute efficiency. The inclusion of sparse Mixture-of-Experts (MoE) FFNs scales total parameter count while bounding per-token compute, and identity hyper-connections (iHC) with sandwich normalization empirically enhance training stability and model expressivity.

Figure 1: Overview of Chimera's architecture and scaling framework; hybrid-linear-global attention, Chinchilla-style scaling, and empirical improvements in compute and memory efficiency for long-sequence generation.

This architecture is fundamentally distinct from prior DiT-style visual diffusion Transformers, which depend on full self-attention, leading to quadratic scaling in both memory and compute. The KDA mechanism is $\mathcal{O}(N)$, supporting much longer input sequences compared to standard multi-head attention. The absence of positional embeddings is achieved via content-adaptive recurrent state tracking and short convolutions that encode local order, yielding a model with inherent temporal extrapolation and minimal inductive bias.

Figure 2: The single-stream latent diffusion Transformer architecture with KDA, MLA, modality-aware short convolution, MoE FFNs, iHC, and sandwich normalization.

The modality-aware short convolution is a notable mechanism for granular spatiotemporal modeling and is more parameter-efficient and hardware-aligned than prior multi-directional scanning approaches.

Figure 3: Conceptual comparison: modality-aware short convolution vs. conventional 1D+multi-scan locality for visual modeling.

HeteroP: Module-Wise Hyperparameter Transfer and Scaling Recipes

Typical hyperparameter transfer and scaling laws, as established in language modeling, rely on homogenous scaling (e.g., uniform width/depth increases). FutureOrange introduces HeteroP, a heterogeneity-aware module-wise scaling scheme. Each tensor’s transfer ratio is determined by its functional fan-in and model depth, solving the mismatch between model-agnostic rules and structurally diverse visual backbones with components like MoE, KDA, MLA, and convolutional layers.

This enables reliable transfer of optimizer, initialization, and residual scaling hyperparameters from a proxy model to any target scale, ensuring comparability across model size and depth. The result is a controlled FutureOrange family covering 59M to 19.3B total parameters, over which Chinchilla-style scaling laws can be robustly fit.

Figure 4: Visualization of the FutureOrange model family over width, depth, activated and total parameters.

Chinchilla-Style Scaling Laws for Vision: Empirical Results

The authors systematically fit scaling laws—training loss as a function of activated parameters and training data tokens—for both images and videos. For image pretraining, the scaling exponents for model size and data amount are near parity ($N_{\mathrm{opt}}\propto C^{0.48-0.52}$, $D_{\mathrm{opt}}\propto C^{0.48-0.52}$). For video, more compute is allocated to model capacity ($N_{\mathrm{opt}}\propto C^{0.53-0.56}$, $D_{\mathrm{opt}}\propto C^{0.44-0.47}$). This forecasts that optimal video generators at large scale should prioritize parameter count, reflecting the added modeling burden of temporal coherence and dynamics.

Figure 5: Training-loss envelope curves quantifying compute-optimal model/data allocation along the scaling frontier.

The extended scaling analysis incorporates the image–video data ratio as an explicit scaling variable; at low compute, image-heavy mixtures are optimal, but the optimal ratio approaches parity as compute budgets grow.

Figure 6: IsoFLOP surfaces tracking optimal mixture ratios and model configurations as a function of compute for image diffusion loss.

Empirical Findings: Efficiency and Extrapolation

Compute Efficiency: Under matched compute, the full FutureOrange configuration achieves a $7.3\times$ improvement in compute efficiency over a quadratic full-attention baseline (Wan 2.1) for the same training loss, and the dense backbone alone achieves $1.7\times$ the compute efficiency.

(Figure 1, A1/S2)

Figure 1 (A1/S2): Quantitative gains in training compute and efficiency with the proposed architecture and scaling approach.

Long-Range Extrapolation: FutureOrange demonstrates robust zero-shot video generation up to 30 seconds (trained only on 5-second clips), with just a 6.5% FID degradation in the extrapolated window—substantially outperforming prior models, which see >50% FID deterioration in similar settings. This result is directly attributable to the position-agnostic design of KDA-based attention and local convolutions, which avoid length-specific artifacts common to RoPE and similar position encodings.

Qualitative Results

High-resolution image and video synthesis from FutureOrange are high quality and semantically aligned, with strong prompt adherence and temporal consistency in video extrapolation. Notably, the classifier-free guidance default is used for these generations (e.g., scale 4 for images).

Figure 7: Example text-to-image generations from FutureOrange at $1344 \times 864$ resolution.

Figure 8: Example text-to-video generations, displaying temporally consistent and semantically faithful 81-frame video samples.

Implications and Future Directions

Practical Implications

• Unified Modality Modeling: The single-stream, multi-modal design allows consistent prompt, image, and video generation within a shared backbone, simplifying both architecture and scale tuning.
• Compute Efficiency: The replacement of full attention with KDA for most layers radically improves memory utilization and latency, directly impacting costs for large-scale visual synthesis and adaptation to edge or inference-constrained environments.
• Zero-Shot Adaptivity: The NoPE design enables strong length generalization without explicit tuning, relevant for deployment in domains where context sizes are unpredictable or unbounded.

Theoretical Implications

• Chinchilla-Scaling Extends to Vision: The experimental confirmation of balanced scaling laws for vision (with moderate shifts for video) substantiates the generality of compute-optimal scaling principles outside language domains.
• Systematic Hyperparameter Transfer: HeteroP provides methodological rigor for future scaling studies of heterogeneous architectures, correcting a pervasive source of bias and inefficiency in architectural scaling literature.

Speculation for Future AI Development

Emerging directions include:

• One-Stream Multimodal World Models: FutureOrange's principles may underpin future world models with arbitrarily long context, spanning textual, visual, and audio data.
• Highly Modular Scaling Recipes: The use of module-wise scaling (HeteroP) may become standard as architectures incorporate more diverse operators (e.g., SSMs, convolutions, MoE, attention).
• Simulation and Planning: Efficiently scaling to millions of tokens with low-latency computation opens the door to granular temporal reasoning, planning, and simulation in animatronics, robotics, and digital agents.

Conclusion

FutureOrange demonstrates a robust co-design of architecture and scaling methodology for visual generative models in the token-extensive regime. Key technical innovations—hybrid-linear-global attention, modality-aware short convolution, and heterogeneity-aware hyperparameter transfer—yield strong empirical gains in compute efficiency, generalization, and unified modality modeling. These findings have broad implications for the next-generation of AI systems that must reason over vast, heterogeneous, and temporally extended visual sequences.