Dynamic Trace Estimation: Adaptive Matrix Tracking

• Dynamic trace estimation is a method to approximate the trace of time-evolving matrices using matrix-vector products and exploiting temporal smoothness.
• The approach builds on static estimators like Hutchinson’s method but advances through recursive, damped techniques that mitigate error accumulation.
• Key applications include online neural network diagnostics, dynamic graph analysis, and spectral density approximation, offering improved query complexities.

to=arxiv_search.search 天天送json {"query":"Dynamic Trace Estimation (Dharangutte et al., 2021) Optimal Query Complexities for Dynamic Trace Estimation (Woodruff et al., 2022) stochastic trace estimation parameter-dependent matrices (Matti et al., 25 Feb 2025)", "max_results": 10} to=arxiv_search.search 诺果json {"query":"Estimation of matrix trace using machine learning (Yoon, 2016) On Symmetric Lanczos Quadrature for Trace Estimation (Li et al., 26 Apr 2025)", "max_results": 10} Dynamic trace estimation is the dynamic version of implicit trace estimation: given matrix-vector product access to a matrix that changes over time, the objective is to maintain accurate approximations of its trace with substantially fewer oracle calls than re-running a static estimator at every snapshot. In the canonical formulation, one observes a sequence $A_1,\dots,A_m\in\mathbb{R}^{n\times n}$, accessible only through matrix-vector multiplications, and seeks estimates $t_1,\dots,t_m$ for $\operatorname{tr}(A_i)$ under explicit $\epsilon$-accuracy and $\delta$-failure guarantees. The central methodological idea is to exploit temporal smoothness—typically a bound on $\|A_{i+1}-A_i\|$—so that trace information from earlier matrices can be reused rather than discarded (Dharangutte et al., 2021, Woodruff et al., 2022).

1. Canonical formulation and oracle model

The basic dynamic problem is posed for a matrix sequence

$A_1,A_2,\dots,A_m\in\mathbb{R}^{n\times n},$

with access restricted to a matrix-vector product oracle. In one representative formulation, the target is to output $t_i$ such that

$$\Pr\big[|t_i-\operatorname{tr}(A_i)|\ge \epsilon\big]\le \delta,$$

under the assumptions

$$\|A_i\|_F\le 1,\qquad \|A_{i+1}-A_i\|_F\le \alpha .$$

Here $t_1,\dots,t_m$0 quantifies temporal drift, and the cost metric is the number of oracle calls $t_1,\dots,t_m$1 (Dharangutte et al., 2021).

A second formulation replaces the Frobenius-norm smoothness condition by a Schatten-norm condition. In particular, for matrices satisfying

$t_1,\dots,t_m$2

or more generally $t_1,\dots,t_m$3 for $t_1,\dots,t_m$4, the objective remains simultaneous additive $t_1,\dots,t_m$5-accurate trace estimation over the whole sequence, again using only matrix-vector queries (Woodruff et al., 2022).

These formulations are motivated by matrix-free settings in which explicit matrix entries are unavailable or unaffordable. The trace is then a hidden scalar functional extracted indirectly from quadratic forms or low-rank sketches. This setting includes Hessian analysis during neural-network optimization, graph quantities such as triangle counts and natural connectivity, and parameterized matrix-function traces arising in spectral density approximation (Dharangutte et al., 2021, Matti et al., 25 Feb 2025).

2. Static estimators as the foundation

Dynamic methods are built on static stochastic trace estimation. The classical baseline is Hutchinson’s estimator,

$t_1,\dots,t_m$6

with Rademacher probes $t_1,\dots,t_m$7. It is unbiased,

$t_1,\dots,t_m$8

and satisfies

$t_1,\dots,t_m$9

Applied independently to every $\operatorname{tr}(A_i)$0, this yields a naive dynamic cost of

$\operatorname{tr}(A_i)$1

matrix-vector products (Dharangutte et al., 2021).

Several static works show how strongly this cost can depend on exploitable structure. A machine-learned trace estimator replaces Hutchinson’s random probes by a small learned set of probing vectors $\operatorname{tr}(A_i)$2, trained over a collection of structurally similar matrices by minimizing squared trace-estimation error. In numerical experiments on random matrices, $\operatorname{tr}(A_i)$3 learned probing vectors achieved precision similar to $\operatorname{tr}(A_i)$4 random noise vectors, and about $\operatorname{tr}(A_i)$5 probing vectors gave accuracy comparable to Hutchinson with $\operatorname{tr}(A_i)$6–$\operatorname{tr}(A_i)$7 random noise vectors (Yoon, 2016).

This suggests a useful conceptual distinction. Static methods already exploit cross-instance regularity—shared sparsity, spectral structure, or learned families of matrices—whereas dynamic trace estimation exploits temporal regularity across a sequence. The underlying principle is the same: redundancy in the target family should be converted into fewer matrix-vector products.

3. Recursive dynamic estimators and variance control

A direct incremental identity,

$\operatorname{tr}(A_i)$8

suggests estimating only the trace of changes. The difficulty is drift: repeatedly adding noisy difference estimates causes error accumulation. The main practical resolution is the damped recursive estimator DeltaShift, defined through

$\operatorname{tr}(A_i)$9

with a damping parameter $\epsilon$0 (Dharangutte et al., 2021).

The role of damping is explicit. Multiplying the previous estimate by $\epsilon$1 shrinks older noise, stabilizes the recursion, and balances carried-forward variance against the variance of the fresh correction term. In the theoretical analysis, setting $\epsilon$2 yields

$\epsilon$3

so Hutchinson applied to $\epsilon$4 has variance on the order of $\epsilon$5 rather than the scale of the full matrix (Dharangutte et al., 2021).

The main complexity theorem states that DeltaShift solves the dynamic trace estimation problem with

$\epsilon$6

total matrix-vector multiplications. In the natural regime $\epsilon$7, the average per-step cost becomes

$\epsilon$8

whereas independent Hutchinson requires

$\epsilon$9

per step. The paper identifies this as a quadratic improvement in $\delta$0 (Dharangutte et al., 2021).

The same work also studies a stronger setting with bounded nuclear norm and a Hutch++-based variant. There the static variance bound

$\delta$1

permits a stronger dynamic algorithm. A plausible implication is that dynamic trace estimation becomes especially favorable when the evolving matrices or updates are near-low-rank (Dharangutte et al., 2021).

4. Optimal query complexity and adaptive path length

The first optimal-complexity treatment organizes the sequence into blocks and estimates a hierarchy of trace differences by a binary-tree summation procedure. For a block $\delta$2, the algorithm estimates $\delta$3, adjacent differences, and then longer-range differences at scales $\delta$4. Because a difference spanning $\delta$5 steps satisfies

$\delta$6

the estimator allocates accuracy level by level and reconstructs each $\delta$7 from only $\delta$8 tree nodes (Woodruff et al., 2022).

Under the Schatten-$\delta$9 condition, this yields query complexity

$\|A_{i+1}-A_i\|$0

and more generally, for $\|A_{i+1}-A_i\|$1,

$\|A_{i+1}-A_i\|$2

The same paper proves matching lower bounds in all relevant parameters, including failure probability, and gives the first unconditional lower bounds for dynamic trace estimation. It also establishes tight static lower bounds for Hutchinson’s estimator in the matrix-vector product model with Frobenius-norm error (Woodruff et al., 2022).

A subsequent general framework replaces worst-case dependence on $\|A_{i+1}-A_i\|$3 by a path-length-style dependence on

$\|A_{i+1}-A_i\|$4

Its adaptive residual-reuse update is

$\|A_{i+1}-A_i\|$5

with $\|A_{i+1}-A_i\|$6 and local budget $\|A_{i+1}-A_i\|$7 chosen as a function of the current step size. The resulting complexity scales like

$\|A_{i+1}-A_i\|$8

rather than $\|A_{i+1}-A_i\|$9, which is sharper for sequences that are mostly stable but contain rare bursts (Gokhale et al., 22 Jun 2026).

The same framework also treats unknown step sizes. In the matrix case, $A_1,A_2,\dots,A_m\in\mathbb{R}^{n\times n},$0 is estimated on the fly by applying Hutchinson to

$A_1,A_2,\dots,A_m\in\mathbb{R}^{n\times n},$1

with only $A_1,A_2,\dots,A_m\in\mathbb{R}^{n\times n},$2 extra queries per step for a constant-factor estimate. The paper emphasizes that, in certain cases, change magnitudes can be estimated with nearly no added cost (Gokhale et al., 22 Jun 2026).

5. Parameter-dependent matrices, matrix functions, and online monitoring

Dynamic trace estimation also appears in parameter-dependent settings. For a continuous family

$A_1,A_2,\dots,A_m\in\mathbb{R}^{n\times n},$3

one often needs $A_1,A_2,\dots,A_m\in\mathbb{R}^{n\times n},$4 for many $A_1,A_2,\dots,A_m\in\mathbb{R}^{n\times n},$5-values. A recent approach modifies the Girard-Hutchinson, Nyström, and Nyström++ estimators by using the same random vectors for every $A_1,A_2,\dots,A_m\in\mathbb{R}^{n\times n},$6. This “constant randomization” makes it possible to reuse matrix-vector products across parameter values, especially when $A_1,A_2,\dots,A_m\in\mathbb{R}^{n\times n},$7 is itself approximated by a Chebyshev expansion in $A_1,A_2,\dots,A_m\in\mathbb{R}^{n\times n},$8, and the analysis shows that the loss of stochastic independence across different $A_1,A_2,\dots,A_m\in\mathbb{R}^{n\times n},$9 does not lead to deterioration (Matti et al., 25 Feb 2025).

For spectral density approximation, the target has the form

$t_i$0

The proposed method combines constant randomization with the Chebyshev recurrence

$t_i$1

so that sketches such as $t_i$2 are computed once and then reused for every $t_i$3. The key theoretical statement is that Nyström++ with constant randomization achieves $t_i$4, independent of low-rank properties of $t_i$5 (Matti et al., 25 Feb 2025).

A distinct online formulation appears in neural-network training, where the target is the trace of each layer’s diagonal Hessian block,

$t_i$6

A stochastic estimator combines Hutchinson probes with a single full-network Hessian-vector product, so that one backward-type pass per probe yields unbiased estimates for all layers simultaneously. The estimator’s conditional variance is

$t_i$7

and the total variance decomposes into probe noise plus mini-batch noise, yielding a critical probe count $t_i$8. The practical recommendation is $t_i$9 for online monitoring (Bolshim et al., 25 May 2026).

These examples broaden the subject without collapsing its distinctions. In the core literature, “dynamic” refers to evolving matrices over time; in parameter-dependent problems it refers to reuse across a continuum of $$\Pr\big[|t_i-\operatorname{tr}(A_i)|\ge \epsilon\big]\le \delta,$$0-values; in training diagnostics it refers to repeated online estimation during optimization.

6. Applications, neighboring methods, and conceptual boundaries

The earliest dynamic-trace applications emphasize three domains. For neural-network optimization, traces of Chebyshev polynomials of the Hessian are used to track moments of the Hessian spectral density. For dynamic graphs, triangle counting relies on

$$\Pr\big[|t_i-\operatorname{tr}(A_i)|\ge \epsilon\big]\le \delta,$$1

while natural connectivity depends on $$\Pr\big[|t_i-\operatorname{tr}(A_i)|\ge \epsilon\big]\le \delta,$$2. Across these applications, damped dynamic estimators achieved lower error or better error-versus-cost tradeoffs than Hutchinson, simple incremental updates, or periodic restarts (Dharangutte et al., 2021).

Recent work on stochastic Lanczos quadrature addresses a complementary issue: when trace estimation of $$\Pr\big[|t_i-\operatorname{tr}(A_i)|\ge \epsilon\big]\le \delta,$$3 is built from quadratic forms approximated by Lanczos-Gauss rules, symmetric quadrature nodes and weights can lower iteration counts. For Jordan-Wielandt matrices, carefully chosen initial vectors guarantee symmetric quadrature and yield unbiased trace estimators for Estrada index computation in bipartite and directed graphs (Li et al., 26 Apr 2025).

Several neighboring methods are dynamic-friendly but not dynamic in the strict sequence-update sense. FlexTrace is a single-pass, exchangeable randomized estimator for $$\Pr\big[|t_i-\operatorname{tr}(A_i)|\ge \epsilon\big]\le \delta,$$4 that uses only matvecs with $$\Pr\big[|t_i-\operatorname{tr}(A_i)|\ge \epsilon\big]\le \delta,$$5. The paper explicitly states that it is about static trace estimation of matrix functions and does not address updating the estimate under matrix changes over time, although its single-pass and function-agnostic properties are close in spirit to streaming constraints (Madhavan et al., 5 Mar 2026).

This terminological boundary matters. Dynamic trace estimation is not simply “Hutchinson on differences,” because error accumulation makes damping or multiscale summation essential. Nor is every adaptive trace estimator dynamic: learned probing vectors, single-pass matrix-function sketches, and variational trace-related estimators can exploit structure without maintaining a stateful estimate across a changing target. The common thread is matrix-free access and aggressive reuse of information; the defining feature of the dynamic problem is that the reuse is indexed by change in the target itself (Dharangutte et al., 2021, Madhavan et al., 5 Mar 2026).

The main limitation is equally consistent across the literature. Dynamic gains require stable structure: small $$\Pr\big[|t_i-\operatorname{tr}(A_i)|\ge \epsilon\big]\le \delta,$$6, representative training families, or smoothly parameterized operators. When updates are large, when the structure shifts abruptly, or when training labels or trace surrogates are too noisy, the advantage over static re-estimation narrows. The lower-bound results show that, under the standard matrix-vector product model, the leading dependence on $$\Pr\big[|t_i-\operatorname{tr}(A_i)|\ge \epsilon\big]\le \delta,$$7, $$\Pr\big[|t_i-\operatorname{tr}(A_i)|\ge \epsilon\big]\le \delta,$$8, $$\Pr\big[|t_i-\operatorname{tr}(A_i)|\ge \epsilon\big]\le \delta,$$9, and $$\|A_i\|_F\le 1,\qquad \|A_{i+1}-A_i\|_F\le \alpha .$$0 is essentially settled up to polylogarithmic factors (Woodruff et al., 2022).