Building Shor's Algorithm in Lean: An Agentic Formalization of Quantum Attacks on RSA-2048 and P-256

Abstract: LLMs are increasingly assisting with demanding formal theorem-proving tasks, particularly when grounded in machine-checked libraries such as Lean. Agentic systems further amplify this process by searching, reusing, and extending existing formal developments to uncover new discoveries. In quantum computing, Shor's algorithm and its variants present such a demanding case for Lean formalization. In this work, we formalize this algorithm family in Lean through agentic formalization: software agents analyze sources, write Lean code and repair proofs, with human review of the scientific claims and machine checking of the resulting formal proofs. Our formalization develops the mathematical foundations for analyzing quantum attacks in two cryptographic settings: a 2048-bit modulus in the RSA-2048 and the standardized elliptic curve over a 256-bit prime field (P-256). To support these analyses, the formalization ranges from quantum algorithms for order finding to reversible quantum circuits for modular and elliptic-curve arithmetic. Based on [Quantum 5, 433] and [ASIACRYPT 2017, 241--270], we formalize the logical resource estimates for RSA-2048 and P-256, respectively, and provide additional estimates of classical operations. We expect the results pave the way for broader machine-checked quantum cryptanalysis and represent a step toward AI-assisted design and verification of quantum algorithms.

• The paper provides a complete Lean-based formalization of Shor's algorithm that rigorously verifies quantum attacks on RSA-2048 and P-256 with concrete resource estimates.
• It details an agentic workflow that integrates mathematical proofs, quantum circuit design, and classical post-processing to achieve end-to-end verification.
• The work offers practical cryptanalytic insights by establishing explicit bounds on qubits, Toffoli gates, and circuit depth for both RSA-2048 and P-256 attacks.

Formalization of Shor's Algorithm and Quantum Attacks on RSA-2048 and P-256 in Lean

Overview

The paper "Building Shor's Algorithm in Lean: An Agentic Formalization of Quantum Attacks on RSA-2048 and P-256" (2607.14082) provides a rigorous Lean formalization of the mathematical reductions, quantum circuits, and classical post-processing procedures for Shor-type attacks against two cryptographically significant problems: integer factorization for RSA-2048 and discrete logarithms for the standardized P-256 elliptic curve. The methodology leverages agentic formalization, wherein software agents undertake decompositional proof development and source aligning, with verification established via the Lean proof assistant and oversight by domain experts.

Agentic Formalization Workflow

Agentic formalization orchestrates artifact flow from mathematical claims and literature sources through decomposition, Lean code generation, machine-checking, and integration into manuscript claims. This division enforces a trust boundary, ensuring that all formalized components—including quantum and classical operations—are subject to Lean's kernel checking, while resource measures and scientific judgment remain with human curators.

Figure 1: Agentic formalization workflow, delineating the trust boundary and the responsibilities for claim selection, source analysis, Lean code development, machine checking, human review, and integration.

The approach directly supports modular provenance tracing in formal libraries, making theorem assumptions, dependencies, and composability accessible to both software agents and researchers.

Main Results: Resource-Explicit Quantum Cryptanalysis

RSA-2048 Factorization

The formalization covers the quantum-to-classical chain in integer factorization for RSA-2048, encompassing order-finding circuits, reversible modular arithmetic, factor recovery via continued fractions, and resource tallies—logical qubits, Toffoli gates, circuit depth, and classical operations. The principal theorem states that a quantum algorithm constructed via Lean returns a prime factor $d \in \{p,q\}$ with a probability ≥$2/3$, bounded by $6.19 \times 10^3$ logical qubits, $8.1 \times 10^9$ Toffoli gates, $6.42 \times 10^9$ circuit depth, and $3.69 \times 10^4$ classical arithmetic steps. These parameters originate from the Ekerå–Håstad short discrete logarithm construction and gate-level analysis [gidney2019factor].

P-256 Elliptic-Curve Discrete Logarithm

For P-256, the formalization extends to affine prime-field arithmetic, elliptic-curve operations, controlled point additions, and scalar recovery, integrating circuit-level resource estimates. The corresponding theorem certifies scalar recovery with probability ≥$2/3$, using $2.33 \times 10^3$ logical qubits, $1.26 \times 10^{11}$ Toffoli gates, $1.16 \times 10^{11}$ maximal Toffoli-gate depth, with only $2/3$0 classical operations required for post-processing, as obtained from gate-level analysis [roetteler2017ecdlp].

Machine-Checked Proof Chain and Modular Abstraction

The Lean development constructs reusable abstractions bridging number-theoretic reductions, quantum circuit semantics, classical validation (e.g., continued fractions, GCDs), and explicit resource accounting. Proof obligations are decomposed into:

• Modular circuit arithmetic—reversible addition, multiplication, exponentiation, and division;
• Quantum circuit scheduling—controlled modular exponentiation, phase estimation, quantum Fourier transform, and coherent group actions for discrete logs;
• Post-processing—continued fraction validation, classical recovery checks, candidate solution selection;
• Resource certificates—logical qubit footprint, Toffoli gate counts, maximal circuit depth, and classical operation budgets.

All steps are formalized and verified in Lean, with interfaces permitting explicit bridges, certificates, and resource provenance.

Figure 2: Dependency map of Lean formalization for RSA-2048 and P-256, showing implemented paths (solid) and certificate boundaries/interfaces (dashed) culminating in terminal public theorems.

Comparative Formal Verification and Methodology

The paper situates its formalization among prior works in quantum-program verification (Qbricks, Qafny, Coq/CoqQ, Quantum Hoare Logic), noting that Lean achieves end-to-end coverage: constructing gate-level arithmetic, order-finding, success probability bounds, quantum and classical resources, and requiring no abstract arithmetic oracles for its claims.

Strong numerical results are explicitly stated:

• RSA-2048 attack: $2/3$1 Toffoli gates, $2/3$2 circuit depth for three sequential attempts, with success ≥$2/3$3.
• P-256 attack: $2/3$4 Toffoli gates, depth $2/3$5, $2/3$6 qubits per run, success ≥$2/3$7.

These claims contradict the often asymptotic resource descriptions seen elsewhere, providing concrete, machine-checked tuples compatible with cryptanalytic simulation and benchmarking.

Practical and Theoretical Implications

The modular, agentic approach supports future automation of quantum cryptanalysis, with explicit mathematical and circuit-level provenance. The Lean formalization lays a foundation for AI-assisted quantum algorithm design and verification, enabling retrieval, schematic comparison, and reuse of theorem components. The work exposes certificate boundaries for resource claims and output equality, highlighting explicit assumptions and supporting reproducibility and auditability.

Extensions and implications include:

• Integration of resource analysis from approximate arithmetic and fault-tolerant quantum architectures (e.g., LDPC code-based reduction, space-efficient arithmetic circuits).
• Expansion of formal coverage to other cryptographically relevant algorithms (e.g., ECC variants, alternate number-theoretic schemes).
• Systematic derivation of correctness, probability distributions, and resource models from unified formal executions, given only public cryptographic data.

Conclusion

The paper achieves a robust Lean-based formalization of quantum attacks against RSA-2048 and P-256, recording both mathematical reductions and high-fidelity circuit resource estimates. Modular abstractions, agentic source decomposition, and certificate-bound claims collectively advance the reliability and auditability of quantum cryptanalytic modeling. Future directions target integration of circuit noise, fault-tolerance, and space-depth tradeoffs within formal systems, fostering deeper connections between quantum algorithm verification and cryptanalytic benchmarking.