OrbitQuant: Data-Agnostic Quantization for Image and Video Diffusion Transformers

Abstract: Diffusion transformers (DiTs) achieve state-of-the-art image and video generation, but their multi-step sampling and growing parameter count make inference expensive. Post-training quantization (PTQ) is the natural remedy, yet DiT activations shift across timesteps, prompts, and guidance branches, forcing prior methods to re-fit calibration data for every new checkpoint or modality. We present OrbitQuant, a data-agnostic weight-activation quantizer that bypasses range estimation by quantizing in a normalized, rotated basis. In this basis, a randomized permuted block-Hadamard (RPBH) rotation concentrates each coordinate around one fixed, known marginal regardless of the input, so a single Lloyd-Max codebook serves all timesteps, prompts, and layers of a given input dimension. We extend the same quantizer to weight rows offline, absorbing the rotation into the weights so that it cancels inside each linear layer and only a forward rotation on the activations remains at runtime. The same recipe transfers from image to video with no per-modality tuning. Across FLUX.1, Z-Image-Turbo, Wan 2.1, and CogVideoX, it sets the state of the art for PTQ at several low-bit settings. It also pushes PTQ of image diffusion transformers to W2A4 with usable generation quality.

• The paper introduces OrbitQuant, a framework that uses randomized permuted block-Hadamard rotation for calibration-free quantization of DiTs.
• It achieves state-of-the-art post-training quantization accuracy on both image and video models even at ultra-low precision (e.g., W4A4) without calibration data.
• The approach reduces computational overhead by efficiently quantizing weights and activations, making high-fidelity generative transformers more deployment-ready.

OrbitQuant: Data-Agnostic Quantization for Image and Video Diffusion Transformers

Introduction and Motivation

The proliferation of diffusion transformers (DiTs) has shifted the landscape of high-fidelity image and video generation. DiTs, with transformer-based denoisers, have superseded traditional convolutional architectures, delivering state-of-the-art results. However, DiTs present significant computational overhead in inference due to repeated transformer evaluations across multiple denoising timesteps and their fundamental compute-bound behavior. Post-training quantization (PTQ) of both weights and activations is an intuitive strategy for alleviating these bottlenecks. Yet, unlike in LLMs, DiT activations exhibit challenging input-dependent range drift across timesteps, prompts, and classifier-free guidance (CFG) branches. This drift fundamentally undermines conventional PTQ that relies on activation range calibration, necessitating repeated recalibration for every checkpoint and modality.

OrbitQuant introduces a data-agnostic weight-activation quantization framework that eliminates the need for calibration data. It leverages random rotations to concentrate activations in a permuted block-Hadamard (RPBH) basis where each coordinate is well-behaved and admits a fixed, analytic quantization codebook, independent of the input, prompt, or timestep. This methodology extends uniformly across image and video modalities and supports quantization down to ultra-low precision.

Figure 1: Overview of OrbitQuant. (1) DiT activations drift across timesteps and CFG branches, so calibrated scales do not transfer. (2) The RPBH rotation $\Pi_d$ maps raw activations to well-behaved coordinates. Folded into the weights, it cancels inside each layer ($\hat{W}'\hat{x}' \approx Wx$).

Methodology

Rotation-Based Quantization Framework

OrbitQuant operates by transforming both weights and activations into a shared, rotated, normalized coordinate system where the marginal coordinate distributions are fixed and analytically tractable. Specifically, the key innovation is the use of a randomized permuted block-Hadamard (RPBH) rotation, consisting of a block-diagonal Hadamard structure and a global, data-independent random permutation. This choice ensures:

• All coordinates across the rotated vector are approximately distributed as $\mathcal{N}(0, 1/d)$ regardless of input data, timestep, or prompt (as guaranteed by universal variance concentration).
• Only a single, offline MSE-optimal Lloyd–Max codebook per dimension and bit-width is necessary, eliminating the expense and risk of calibration drift.

  Figure 2: Rotated activation coordinates follow the dimension marginal $f_d$. For (a) an attention projection and (b) a feed-forward projection, the rotated distribution matches closely to $\mathcal{N}(0,1/d)$, enabling use of a fixed codebook.

Offline Weight Quantization

• Weights $\mathbf{W}$ are rotated offline by the shared RPBH transformation and split into unit directions and norms.
• Directions are quantized with the distribution-derived Lloyd–Max codebook; norms are stored as low-precision scalars.
• All weight rows for a given input dimension share the same codebook and rotation.
• After absorbing the rotation, the product with rotated activations cancels the basis transformation, reconstructing the intended semantics.

Online Activation Quantization

• Activations $\mathbf{x}$ are projected (at runtime) into the same rotated basis and normalized.
• Each normalized coordinate is quantized via lookup in the invariant codebook; the per-token activation norm is preserved as a scalar.
• Only a forward rotation per activation batch is required, resulting in negligible compute overhead, which is further minimized by the block-Hadamard structure.

RPBH Rotation and Data-Agnostic Codebook

• The RPBH is $O(d \log h)$—far more computationally efficient than a dense Haar rotation—while ensuring near-optimal matching to the target marginal.
• Theoretical analysis guarantees that the random permutation phase of RPBH effectively distributes activation variance, preventing block-concentrated outlier effects commonly encountered in DiTs.
• The codebook is therefore maximally robust, transferable across all model layers and tasks.

Experimental Results

Image and Video PTQ Performance

OrbitQuant is benchmarked on various state-of-the-art image (FLUX.1, Z-Image-Turbo) and video (Wan 2.1, CogVideoX) DiTs under multiple bit-widths. The results evidence state-of-the-art PTQ accuracy for weight-and-activation quantization, without any calibration data, even at ultra-low precisions where prior art collapses.

• At W4A4 (4-bit weights, 4-bit activations), OrbitQuant is nearly lossless on GenEval and leads all PTQ baselines.
• Critically, at W2A4 and W3A3, it is the only calibration-free PTQ method producing high-quality generations; all calibration-dependent and static-uniform PTQ methods break down, outputting unusable or noise-like generations.

  Figure 3: Qualitative comparison of OrbitQuant against QuaRot and ViDiT-Q. OrbitQuant maintains output fidelity at low bit-widths where other methods collapse.

Quantitative metrics on VBench for video models confirm that OrbitQuant not only closes the gap to full precision inference but also outperforms calibration-based approaches on critical dimensions, including Overall Consistency, Motion Smoothness, and Scene maintenance.

Efficiency and Latency

• OrbitQuant's runtime overhead for activation quantization is modest: the shared codebook enables a single centroid lookup per coordinate, and the fast RPBH transform limits compute and memory requirements.
• In comparison to existing methods, OrbitQuant achieves the lowest quantization-induced latency and memory overhead on image inference, and favorable results on video.

  Figure 4: Latency and peak memory for image (left, FLUX.1-dev) and video (right, Wan~2.1-1.3B) generation; OrbitQuant offers the most efficient activation quantization pipeline.

Ablations

• Weight-activation quantization in the RPBH basis is robust to random seed and rotation choices, and strictly dominates block-only rotations (without permutation) at low bit.
• Bit-width ablation on AdaLN modulation demonstrates that INT4 is the practical limit for fixed-projection quantization; further reduction degrades performance due to accumulation of quantization error in the modulation pathway.

  Figure 5: AdaLN bit-width ablation: GenEval Overall remains stable until AdaLN weights drop below INT4, where performance collapses.

Implications and Future Directions

Theoretical Implications: OrbitQuant's calibration-free paradigm generalizes the use of distributional codebooks for quantization beyond LLMs, demonstrating that randomization and analytic marginal control can overcome non-stationary activation statistics in generative transformers. This advances robust PTQ theory by decoupling quantizer design from model- and input-dependent calibration.

Practical Implications: Deployment-ready DiTs now benefit from aggressive quantization, enabling massive reductions in inference compute and model size without retraining or calibration datasets. The single quantizer recipe simultaneously supports diverse modalities (image/video) and architectures.

Open Challenges: The current pipeline requires a small, structured runtime rotation per layer; integrating native low-bit matmul kernels optimized for lookup-table-based, non-uniform codebooks could further reduce overhead. Extension to context-dependent or dynamic codebooks, while maintaining analytic guarantees, is an open avenue.

Conclusion

OrbitQuant establishes a new standard for PTQ in generative diffusion transformers by utilizing analytic, calibration-free quantization in a robust, rotated basis. The proposed RPBH rotation with a universal Lloyd–Max codebook yields unequivocal numerical and qualitative gains over calibration-driven and heuristic PTQ for both images and video, pushing non-uniform quantization into regimes where all prior methods fail. This work demonstrates the viability and generality of analytic, distributional PTQ methods for large-scale generative models.

(2607.02461)