Harvey's Average Polynomial-Time Algorithm

• The paper demonstrates that arithmetic invariants like zeta functions can be computed efficiently using recurrence relations and batch modular reductions via accumulating remainder trees.
• It achieves an average per-prime complexity that is polynomial in the logarithm of the prime, drastically reducing computation times for large-scale arithmetic geometry tasks.
• The method integrates lifting and cohomological techniques to reconstruct full invariants, offering significant practical speedups over traditional point-counting and p-adic methods.

Harvey’s Average Polynomial-Time Algorithm refers to a family of techniques for efficiently computing invariants—primarily zeta functions, L-polynomials, and related arithmetic data—of algebraic curves and motives, by amortizing the computational cost across many primes and leveraging structural and cohomological properties. Central to these methods is the use of accumulating remainder trees, recurrence relations, and fast arithmetic algorithms enabling an average cost per prime that is polynomial (or quasilinear) in the input size, with dramatic practical speedups for large-scale computations in arithmetic geometry and computational number theory.

1. Fundamental Principles of the Algorithm

Harvey's algorithm is designed for settings where the main goal is to compute, for many primes $p$ (typically $p \leq N$), quantities of arithmetic interest: the numerator of the zeta function for a curve (such as $P(T)$ for $Z_p(T) = P(T)/[(1-T)(1-qT)]$), Hasse–Witt matrices, Frobenius traces for motives, or L-functions coefficients. The core approach combines:

• Lifting computations to a space where algebraic structure (e.g., cohomology, recurrences) allows for bulk processing.
• Recurrence relations that propagate the required coefficients, typically setting up linear systems or matrix products.
• Accumulating remainder trees (or forests) that enable simultaneous modular reductions for all primes under consideration.
• Exploitation of redundancies to avoid recomputation: shared subproducts, batched matrix operations, and interval-based techniques.

This paradigm reduces what might be a $O(p)$ or worse per-prime cost to a per-prime cost that is polynomial in $\log p$, yielding total runtime $O(N\log^{3+o(1)}N)$ for $N$ primes, and $O(\log^{4+o(1)} p)$ on average per prime in the hyperelliptic genus $2$ setting (Shi, 14 Aug 2025), and comparable exponents for higher genus or related structures.

2. Essential Algorithmic Components

The workflow of Harvey’s approach commonly comprises the following stages:

• Basis selection: Choose an explicit basis for differential forms suitable for the cohomological setting (e.g., $\{x^i dx / y\}$).
• Lift and reduction: Model the arithmetic object (curve, differential operator, motive) over $\mathbb{Q}$ or $\mathbb{Z}$, and for each $p$, reduce modulo $p$ whilst computing requisite matrices (e.g., Frobenius, Cartier–Manin, Hasse–Witt).
• Recurrence relations: Formulate recurrences for the expansion coefficients; for hyperelliptic curves, propagate vector blocks via rational matrices $M_n$, so that $v_{n+1} = v_n M_n / D_n$ (Harvey et al., 2014).
• Matrix factorials and product trees: For $p$-curvature computations, evaluate products $M(\theta)\cdot M(\theta+1)\cdots M(\theta+p-1)$ using binary splitting and accumulating remainder trees (Pagès, 2021).
• Modular reduction across primes: By representing data generically and implementing product/remainder trees, compute all required values simultaneously with quasi-linear overall cost.
• Final invariants extraction: For each prime, obtain the desired arithmetic invariant (e.g., the zeta function, characteristic polynomial, Frobenius trace) either by direct computation or via lifting routines that use additional group-theoretic or cohomological constraints (Shi, 14 Aug 2025).

The process enables leveraging cohomological dualities, the structure of Frobenius action, and deep arithmetic bounds (such as Weil or Kedlaya–Sutherland bounds for L-polynomials).

3. Recurrence Relation and Tree-Based Computation

A key mathematical structure underlying Harvey’s algorithm is the recurrence propagation of coefficient blocks:

$$v_{n+1} = \frac{v_n M_n}{D_n},\qquad v_{n} = v_0 \frac{M_0 M_1\cdots M_{n-1}}{D_0 D_1\cdots D_{n-1}}$$

The matrices $M_n$ and denominators $D_n$ are generally explicit functions of the curve’s defining polynomial and parameter $n$, often established by combinatorial or cohomological analysis.

The accumulating remainder tree operates on products $P(n) = M_0 M_1 \cdots M_{n-1}$, modular reductions $m_n = p^e$, and organizes computation along a binary tree for batch modular reduction, optimizing both time and space complexity (Harvey et al., 2014).

For point-counting, the method computes for all $p$ up to $N$,

$$J_p = (D_0 \cdots D_{(p-1)/2})^{-1}(M_0 \cdots M_{(p-1)/2}) \pmod{p^\nu}$$

which encodes the arithmetic invariant in the first column or row of $J_p$.

4. Complexity, Trade-Offs, and Lifting

The algorithm achieves:

• Total runtime: $O(N\log^{3+o(1)} N)$ across $N$ primes for genus 2 curves (Shi, 14 Aug 2025).
• Average per-prime complexity: $O(\log^{4+o(1)} p)$, substantially better than previous $O(p^{1/2+\epsilon})$ p-adic methods.
• Single-prime variant: $O(p^{1/2}\log^{1+o(1)} p)$ by adapting local computations.

Lifting steps—critical for reconstructing full integer coefficients from modular reductions—utilize additional arithmetic constraints. For genus 2 L-polynomials, given $a_1 \bmod p$ and $a_2 \bmod p$, the lattice of possibilities is sharply restricted by Weil/Kedlaya–Sutherland bounds. Candidate elimination is performed using 2-rank (from the factorization pattern of the curve polynomial) and Las Vegas randomized group operations on the Jacobian (Shi, 14 Aug 2025).

The lifting routine thus requires only $O(\log^{2+o(1)}p)$ operations per prime to recover the full L-polynomial, yielding end-to-end performance improvements of 10–4100× over previous methods.

5. Applications and Impact

The algorithm is deployed in numerous contexts:

• Large-scale point-counting: Construction of L-functions databases, verification of Sato–Tate distributions, empirical paper of paramodular and Birch–Swinnerton-Dyer conjectures.
• Arithmetic geometry: Fast Jacobian and zeta function computations unlock experimental paper in high genus, modularity, and rational point search.
• Cryptography: Efficient order computations for Jacobians supporting discrete logarithm-based cryptosystems.
• Automated arithmetic verification: Enables comprehensive primality and group structure checks for millions of curve instantiations.
• Extension to general motives: Techniques generalize (with additional analytic machinery) to hypergeometric motives, p-curvature, and general linear differential operators (Pagès, 2021, Costa et al., 2020, Costa et al., 2023).

6. Theoretical Underpinnings and Limitations

• Distributional assumptions: Average polynomial time is ensured by exploiting structure across primes, but relies on nonpathological inputs (i.e., monic squarefree polynomials, sufficiently large primes).
• Scope of method: Generalizes to hyperelliptic and superelliptic curves, and even hypergeometric motives, but practical implementation for full zeta function in higher genus may remain incomplete (Shi, 14 Aug 2025).
• Space/computational constraints: The remainder tree and matrix propagation demand careful memory management; remainder forests and FFT-optimized linear algebra improve practical scaling.
• Limitations: Extreme input heights, small primes, or pathological arithmetic behavior may require fallback to slower, traditional methods for a negligible minority of cases.

7. Key Formulas

• Genus 2 zeta function numerator:

$L_p(T) = p^2 T^4 + p a_1 T^3 + a_2 T^2 + a_1 T + 1$

given $a_1, a_2$ modulo $p$, with bounds for lifting.

• Matrix recurrence for Hasse–Witt matrices:

$$W_p = [w_{ij}]_{1 \leq i,j \leq g},\quad w_{ij} \equiv f^{(p-1)/2}_{pi-j}\pmod{p}$$

• Average polynomial-time complexity (genus 2):

$$O(B \log^{3+o(1)} B),\qquad O(\log^{4+o(1)} p)\ \text{per prime}$$

• Lifting candidate bounds:

$$B_{\text{low}} = \frac{1}{2}(a_1^2 - p\delta^2),\qquad B_{\text{high}} = 2p + \frac{a_1^2}{4}$$

8. Significance in Average-Case Complexity Theory

Harvey’s Average Polynomial-Time Algorithm fundamentally advances the theory and practice of average-case complexity for problems in arithmetic geometry. It demonstrates that for a broad class of algebraic curves and motives, one may efficiently process vast amounts of arithmetic data—not by focusing on the worst-case for individual primes, but by rigorous amortization and exploiting algebraic regularities. The lifting paradigm illustrates the power of modular reductions paired with arithmetic constraints, and the accumulating remainder tree framework provides a blueprint for scalable batched computation.

The methodology is broadly relevant to ongoing developments in algorithmic number theory, the computational Langlands program, and high-throughput arithmetic experimental mathematics.