Tracking High-order Evolutions via Cascading Low-rank Fitting

Abstract: Diffusion models have become the de facto standard for modern visual generation, including well-established frameworks such as latent diffusion and flow matching. Recently, modeling high-order dynamics has emerged as a promising frontier in generative modeling. Rather than only learning the first-order velocity field that transports random noise to a target data distribution, these approaches simultaneously learn higher-order derivatives, such as acceleration and jerk, yielding a diverse family of higher-order diffusion variants. To represent higher-order derivatives, naive approaches instantiate separate neural networks for each order, which scales the parameter space linearly with the derivative order. To overcome this computational bottleneck, we introduce cascading low-rank fitting, an ordinary differential equation inspired method that approximates successive derivatives by applying a shared base function augmented with sequentially accumulated low-rank components. Theoretically, we analyze the rank dynamics of these successive matrix differences. We prove that if the initial difference is linearly decomposable, the generic ranks of high-order derivatives are guaranteed to be monotonically non-increasing. Conversely, we demonstrate that without this structural assumption, the General Leibniz Rule allows ranks to strictly increase. Furthermore, we establish that under specific conditions, the sequence of derivative ranks can be designed to form any arbitrary permutation. Finally, we present a straightforward algorithm to efficiently compute the proposed cascading low-rank fitting.

• The paper introduces cascading low-rank fitting to capture higher-order derivatives via a shared base function, achieving parameter-efficient high-order modeling.
• It rigorously analyzes the rank evolution of successive derivatives and provides bounds and constructions to guarantee parameter efficiency under linear decomposability.
• Efficient batched computations and specialized data structures, including segment trees and tensor generalizations, enable fast range queries in modern generative frameworks.

Tracking High-order Evolutions via Cascading Low-rank Fitting

Introduction and Problem Setting

This work addresses parameter-efficient modeling of high-order dynamics in modern generative frameworks, particularly diffusion and flow-matching models. Classical approaches in these domains estimate only first-order vector fields to map noise to data, but recent directions have considered learning multiple higher-order derivatives (acceleration, jerk, etc.), yielding a richer class of high-order diffusion variants. Direct instantiation of separate neural networks for each derivative leads to a linear blowup in parameter count. While Low-Rank Adaptation (LoRA) mitigates some of that growth, this paper proposes a sharper method—cascading low-rank fitting—where all derivative orders are captured by a shared base function, augmented with sequential accumulations of low-rank components.

By recasting the multi-derivative estimation problem through the lens of linear ODEs and matrix analysis, the framework identifies how parameter efficiency and dynamic expressivity are governed by the rank structure of successive matrix differences. Cascading low-rank fitting balances parameter cost and approximation power by propagating low-rank structure through the entire family of learned dynamical systems.

Theoretical Analysis of Successive Matrix Derivatives

A central theoretical contribution is a comprehensive analysis of the rank evolution in the sequence of higher-order derivatives, where the $i$-th derivative operator is expressed as

$$L_i(t) = \frac{d^{i-1} W(t)}{dt^{i-1}} - \frac{d^i W(t)}{dt^i}.$$

The main structural insight is the role of linear decomposability: if $L_1(t)$ is $r$-linearly decomposable, i.e., expressed as $L_1(t) = \sum_{j=1}^r f_j(t) u_j v_j^\top,$ then, so long as $u_j, v_j$ are linearly independent, all subsequent derivatives $L_i(t)$ have rank at most $r$ and these ranks are guaranteed to be monotonically non-increasing. This formalizes the conditions under which the parameter allocation to LoRA adapters for higher-order terms can be reduced without loss of expressive power.

The necessity of this structural assumption is established: without linear decomposability, the General Leibniz Rule can cause the rank to strictly increase with successive derivatives. Explicit constructions show matrices where

$r_1 < r_2 < \cdots < r_k,$

with each $L_i(t)$ attaining generic rank $$L_i(t) = \frac{d^{i-1} W(t)}{dt^{i-1}} - \frac{d^i W(t)}{dt^i}.$$0. These results highlight both the scenarios under which rank decay, and thus parameter efficiency, is guaranteed, and those where such decay cannot be expected.

Global upper bounds on the generic rank of higher derivatives are also established: for $$L_i(t) = \frac{d^{i-1} W(t)}{dt^{i-1}} - \frac{d^i W(t)}{dt^i}.$$1 of rank $$L_i(t) = \frac{d^{i-1} W(t)}{dt^{i-1}} - \frac{d^i W(t)}{dt^i}.$$2, the $$L_i(t) = \frac{d^{i-1} W(t)}{dt^{i-1}} - \frac{d^i W(t)}{dt^i}.$$3-th derivative satisfies

$$L_i(t) = \frac{d^{i-1} W(t)}{dt^{i-1}} - \frac{d^i W(t)}{dt^i}.$$4

Stepwise bounds ($$L_i(t) = \frac{d^{i-1} W(t)}{dt^{i-1}} - \frac{d^i W(t)}{dt^i}.$$5) are proved, dictating how sharply the rank can grow in the worst case per differentiation.

Constructive Results and Arbitrary Rank Pattern Realization

The paper further demonstrates, via inductive construction, that any rank sequence $$L_i(t) = \frac{d^{i-1} W(t)}{dt^{i-1}} - \frac{d^i W(t)}{dt^i}.$$6 satisfying the global bound $$L_i(t) = \frac{d^{i-1} W(t)}{dt^{i-1}} - \frac{d^i W(t)}{dt^i}.$$7 (matching the cumulative Leibniz expansion) can be realized. This is achieved by constructing suitable block-diagonal matrix polynomials or sums thereof, ensuring that for each $$L_i(t) = \frac{d^{i-1} W(t)}{dt^{i-1}} - \frac{d^i W(t)}{dt^i}.$$8, $$L_i(t) = \frac{d^{i-1} W(t)}{dt^{i-1}} - \frac{d^i W(t)}{dt^i}.$$9. Furthermore, for any permutation $L_1(t)$0 of $L_1(t)$1 and any chain of relational constraints ($L_1(t)$2, $L_1(t)$3), one can design an $L_1(t)$4 such that the sequence $L_1(t)$5 realizes this ordering. These structural theorems characterize the expressivity limits of parameter-efficient high-order modeling using the proposed methodology.

Algorithmic Formulation: Efficient Batched Computations

The cascading low-rank parameterization admits efficient batched evaluation. For an input batch $L_1(t)$6 and target derivative order $L_1(t)$7, the output is computed as

$L_1(t)$8

where the base weight and cumulatively accumulated low-rank adapters are applied. By carefully organizing the batched matrix multiplies and cumulative summation, total time complexity is reduced to $L_1(t)$9, where $r$0 is the rank of the $r$1-th adapter. This is always asymptotically better than $r$2 for full parameter sharing, and the structure efficiently leverages any monotonic decay in the ranks $r$3.

Data Structures for Low-rank Range Queries

To further facilitate advanced usage (e.g., subrange derivative evaluation), the paper introduces a segment tree data structure supporting range queries over low-rank factorized matrices. For a batch $r$4, and interval $r$5, the sum $r$6 can be returned in $r$7 time. This enables both amortized fast batched evaluation and optimal per-index computation, depending on sparsity and rank patterns.

Tensor Generalization

The segment tree query and factorization methods naturally generalize to higher-order tensors. For example, all data structures and batched algorithms are extended to the setting where each adapter is specified by three low-rank factors $r$8, and the basic operation is

$r$9

with similar $L_1(t) = \sum_{j=1}^r f_j(t) u_j v_j^\top,$0 time bounds for range queries, subject to the contraction structure. This generality makes the method applicable to attention-based and pairwise-interaction settings ubiquitous in transformers.

Practical and Theoretical Implications

The framework developed has both practical and theoretical consequences:

• Parameter Efficiency for High-order Dynamics: The results show exactly when one can expect higher-order derivatives to be parameterized with progressively smaller low-rank adapters, which is crucial for scaling generative models to richer dynamical architectures without uncontrollable parameter growth.
• Expressivity and Limitation Characterization: By precisely characterizing the set of all attainable rank sequences for derivatives, the theorems set hard boundaries for the expressivity of parameter-tied high-order architectures.
• Algorithmic Efficiency: The proposed structures and segment trees enable fast batched and range-based processing, which is relevant for high-throughput training and inference in diffusion and video generation models.

Conclusion

This work sets a rigorous foundation for parameter-efficient high-order modeling in modern generative architectures via cascading low-rank fitting. The detailed analysis of rank evolution, generality of achievable patterns, and provision of efficient batched computational routines, collectively advance the understanding of how high-order dynamics can be efficiently and flexibly represented. Future directions include the explicit integration with softmax-based attention mechanisms and exploration of additional structures for non-linear or non-polynomial derivative patterns, further extending the applicability of this framework to a wider class of neural architectures.

---

For detailed analysis and proofs, see "Tracking High-order Evolutions via Cascading Low-rank Fitting" (2604.10980).