Beyond Fixed Formulas: Data-Driven Linear Predictor for Efficient Diffusion Models

Abstract: To address the high sampling cost of Diffusion Transformers (DiTs), feature caching offers a training-free acceleration method. However, existing methods rely on hand-crafted forecasting formulas that fail under aggressive skipping. We propose L2P (Learnable Linear Predictor), a simple data-driven caching framework that replaces fixed coefficients with learnable per-timestep weights. Rapidly trained in ~20 seconds on a single GPU, L2P accurately reconstructs current features from past trajectories. L2P significantly outperforms existing baselines: it achieves a 4.55x FLOPs reduction and 4.15x latency speedup on FLUX.1-dev, and maintains high visual fidelity under up to 7.18x acceleration on Qwen-Image models, where prior methods show noticeable quality degradation. Our results show learning linear predictors is highly effective for efficient DiT inference. Code is available at https://github.com/Aredstone/L2P-Cache.

• The paper presents L²P, a learnable linear predictor that replaces fixed formulas to accelerate Diffusion Transformer computations.
• It achieves significant speedup gains with up to 7.14× latency reduction and maintains high PSNR and visual fidelity under aggressive acceleration.
• Its data-driven approach generalizes across models and domains, requiring minimal training data and negligible computational overhead.

Data-Driven Linear Prediction for Diffusion Transformer Acceleration

Introduction

"Beyond Fixed Formulas: Data-Driven Linear Predictor for Efficient Diffusion Models" (2604.26365) addresses the computational inefficiency in inference for Diffusion Transformers (DiTs) by identifying the limitations of current feature caching and forecasting methods. The work demonstrates that both classical cache-and-reuse and recent temporal forecasting approaches (such as TaylorSeer, FoCa) are fundamentally constrained by their use of hand-crafted, fixed-coefficient linear predictors over past hidden features. The authors provide a detailed empirical and theoretical analysis, establishing strong linear predictability in DiT feature trajectories. Motivated by this, they introduce L²P, a learnable, data-driven linear predictor that replaces static analytical coefficients with optimally trained, per-timestep weights, leading to improved speed-accuracy tradeoffs, especially at aggressive acceleration ratios.

Analysis of Linear Forecasting Paradigm in DiT Inference

The authors unify existing feature prediction approaches as a class of linear predictors, where features at future steps are expressed as weighted sums of a finite number of precomputed features from historical steps. Approximations such as Taylor expansion, Hermite interpolation, and numerical ODE integrators (BDF2, Heun) are shown to have fixed, model-agnostic coefficients determined solely by formulaic priors, not by the actual feature dynamics of the target DiT. This rigidity becomes a primary performance bottleneck under long skip intervals.

To establish the theoretical ceiling for linear methods, the authors empirically project each DiT feature onto the subspace spanned by all previous steps, quantifying the achievable reconstruction accuracy.

Figure 1: Projection fidelity along a 50-step diffusion trajectory. Most interior steps exceed $0.95$, indicating a high upper bound for linear prediction.

The results (Figure 1) reveal that for the vast majority of denoising steps, the feature can be linearly reconstructed from history with projected residuals below $5\%$, supporting the sufficiency of a linear framework for practical inference acceleration.

The L²P Framework: Training Learnable Linear Predictors

The core contribution is L²P, which retains the linear structure but learns the optimal coefficients directly on a small corpus of real DiT runs. During training (Figure 2a), the model collects a dataset of complete diffusion trajectories over as few as 50 diverse prompts. For each timestep, L²P fits a unique set of coefficients that best predict the feature from its historical context using simple L2 regression. This procedure converges within seconds on a single consumer GPU.

During inference (Figure 2b), L²P allows bypassing expensive forward passes for skipped steps by directly forecasting the corresponding features, which are then used in the subsequent computation or as outputs.

Figure 2: Overview of the L²P framework; lightweight per-timestep linear models are trained using a small set of cached diffusion trajectories and then used to efficiently predict skipped features during inference.

This plug-and-play form enables integration into arbitrary DiT architectures with negligible training and computational overhead, as the additional memory footprint is limited to the final-layer features and a small coefficient matrix.

Quantitative and Qualitative Results

L²P is extensively benchmarked on SOTA DiT models for text-to-image (FLUX.1-dev, Qwen-Image, Qwen-Image-Lightning) and text-to-video generation (HunyuanVideo). Comparative results with leading feature caching and forecasting baselines highlight several findings:

• Speedup at Constant Quality: On FLUX.1-dev, L²P at interval $\mathcal{N}=5$ achieves a $4.15\times$ latency speedup and $4.55\times$ FLOPs reduction, maintaining a PSNR of $31.459$. In contrast, fixed-coefficient baselines like TaylorSeer and FoCa degrade to sub-$29.5$ PSNR under the same configuration.
• Robustness Under Aggressive Acceleration: L²P sustains high fidelity at extreme $\mathcal{N}=10$ ($7.14\times$ reduction), with a PSNR of $30.031$, outperforming all baselines by over $5\%$0 dB. This trend is consistent in both image and video settings, with significant headroom at high skip ratios.
• Generalization Across Models and Domains: On Qwen-Image-Lightning, a model already optimized for speed, L²P further accelerates inference by up to $5\%$1 with less than $5\%$2 dB drop relative to the 50-step original, even for the most challenging semantic and generated content.

  Figure 3: MSE loss comparison between L²P, TaylorSeer, and FoCa across prediction intervals (N); L²P consistently demonstrates lower error, especially at large skip sizes.

• Quality of Generated Samples: Visual comparisons demonstrate that L²P maintains semantic content and visual details across scaling factors, avoiding the degradation observed in existing methods under high acceleration.

  Figure 4: Image generation results for different prompt scenarios and methods; L²P retains superior image quality and faithfulness to prompts at high speedup ratios.

  

  Figure 5: Qwen-Image-Lightning with and without L²P, showing visually consistent outputs at increased acceleration factors.

Data Efficiency, Semantic Robustness, and Method Analysis

An extensive ablation study explores the impact of training sample size and the semantic distribution of training prompts.

Figure 6: PSNR performance quickly saturates as training set size increases, with strong gains even at 5 samples and saturation by 50.

Key findings:

• L²P achieves a $5\%$3 dB PSNR improvement over state-of-the-art predictors with as few as 5 training samples. Performance saturates past 50 samples, highlighting remarkable data efficiency.
• Training on semantically irrelevant or even degenerate prompts (counterfactual/gibberish) has negligible effect on test performance, confirming that L²P captures intrinsic feature evolution dynamics, not prompt semantics.

  Figure 7: Example semantic prompts used in ablation study for semantic robustness.

Extension to Video Generation and Analysis of Nonlinearity

On HunyuanVideo, L²P enables $5\%$4 wall-clock acceleration with leading fidelity (PSNR $5\%$5, SSIM $5\%$6) under settings where all existing methods show visible drift and temporal artifacts.

Figure 8: L²P maintains subject consistency and smooth temporal dynamics in early, middle, and late frames from long generated videos, outperforming baseline predictors.

In terms of model design, the authors argue that explicit or learned nonlinearity—using deeper neural networks, spatially-varying weights, or polynomial expansions—provides negligible benefit over homogeneous linear predictors, as the linear subspace encapsulates >95% of the feature evolution's variance. Introducing nonlinearity increases both data and computational costs, negating the acceleration gains without significant quality improvements.

Practical and Theoretical Implications

L²P formalizes and empirically validates the hypothesis that the diffusion feature sequence is highly linear within the learned manifold, and that acceleration via data-driven linear forecasting is both tractable and efficient. By optimizing the linear predictor on real feature trajectories, L²P removes the reliance on ad hoc, hand-crafted formulas, vastly improving robustness to aggressive temporal skipping.

Practically, this approach is directly deployable to a wide spectrum of foundation vision models with no need for full retraining or architecture-specific tuning. The minimal data regime and low memory requirements make L²P attractive for edge and latency-sensitive applications.

Theoretically, the work establishes an equivalence between any linear combination and high-order difference expansions, unifying prior analyses and relating learned predictors to generalized Taylor-like expansions, while highlighting the empirical suboptimality of fixed analytical coefficients in diffusion trajectory forecasting.

Conclusion

This work reforms the design and analysis of DiT inference acceleration by advocating for—and validating—the sufficiency of learnable linear predictors. It demonstrates that hand-designed, fixed-coefficient approaches are fundamentally limited, and fully data-driven linear forecasting bridges the gap to theoretical limits with negligible cost and exceptional generalization. The L²P technique provides a robust, practical foundation for future work on efficient generative model deployment, and suggests that future research should focus on scaling out similar learning-based adaptivity in other portions of the diffusion process, provided the computational/quality tradeoff regime permits.

---

Reference:

"Beyond Fixed Formulas: Data-Driven Linear Predictor for Efficient Diffusion Models" (2604.26365)