Modified QAOA Factorization Protocol

• The paper introduces a modified QAOA-based factorization method that alters the factoring Hamiltonian, circuit synthesis, and variational parameterization to enhance performance on NISQ hardware.
• Key techniques include Boolean preprocessing, Hamiltonian and circuit reformulations (e.g., GROBNER, SCHALLER), and fixed-point parameterization to reduce interaction locality and improve noise resilience.
• Practical insights highlight reduced CNOT counts, improved residual performance gains, and successful small-instance factorizations, demonstrating feasibility under current quantum constraints.

Modified QAOA-based factorization protocol denotes a class of NISQ-era integer-factorization methods in which the standard Quantum Approximate Optimization Algorithm is altered at the level of the factoring Hamiltonian, circuit synthesis, variational parameterization, or the surrounding hybrid classical–quantum workflow. In the current literature, these modifications include Boolean preprocessing and variational quantum factoring, Hamiltonian and circuit transformations chosen for superconducting-noise resilience, hardware-aware split-cost QAOA on LHZ lattices for dense Ising encodings, fixed-point QAOA with universal parameters inside Schnorr’s smooth-relation workflow, and null-space encodings that replace the squared residual Hamiltonian by a linear one (Anschuetz et al., 2018, Qiu et al., 2020, Lechner, 2018, Zalivako et al., 13 Mar 2025, Pellicer, 13 Nov 2025).

1. Baseline formulation: from binary multiplication to QAOA

The canonical starting point is to encode the factorization problem $m=pq$ in binary. In Variational Quantum Factoring, the factors and target integer are written as

$$m=\sum_{k=0}^{n_m-1} 2^k m_k,\qquad p=\sum_{k=0}^{n_p-1} 2^k p_k,\qquad q=\sum_{k=0}^{n_q-1} 2^k q_k,$$

with binary carry variables $z_{i,j}$ used to express columnwise multiplication constraints. The central clause is

$$0=\sum_{j=0}^{i} q_j p_{i-j}+\sum_{j=0}^i z_{j,i}-m_i-\sum_{j=1}^{n_c} 2^j z_{i,i+j},$$

and factorization is reduced to the zero-energy condition

$0=\sum_{i=0}^{n_c} C_i^2.$

After operator substitution $b_k \mapsto \frac12(1-\sigma^z_{b,k})$, this becomes a 4-local Ising Hamiltonian used as the QAOA cost Hamiltonian. The accompanying preprocessing pass applies deterministic Boolean identities, carry-range truncation, substitution, and propagation. The paper states that preprocessing runtime is $O(n_m^2)$, that the asymptotic qubit count without simplification is $O(n_m\log n_m)$, and that with simplification the empirical scaling is approximately $O(n_m)$ (Anschuetz et al., 2018).

A more explicit small-instance realization appears in the $143=11\times 13$ example of variational quantum factoring on superconducting hardware. There, binary multiplication constraints simplify to

$$m=\sum_{k=0}^{n_m-1} 2^k m_k,\qquad p=\sum_{k=0}^{n_p-1} 2^k p_k,\qquad q=\sum_{k=0}^{n_q-1} 2^k q_k,$$0

giving the factoring cost

$$m=\sum_{k=0}^{n_m-1} 2^k m_k,\qquad p=\sum_{k=0}^{n_p-1} 2^k p_k,\qquad q=\sum_{k=0}^{n_q-1} 2^k q_k,$$1

With $$m=\sum_{k=0}^{n_m-1} 2^k m_k,\qquad p=\sum_{k=0}^{n_p-1} 2^k p_k,\qquad q=\sum_{k=0}^{n_q-1} 2^k q_k,$$2, the problem Hamiltonian becomes

$$m=\sum_{k=0}^{n_m-1} 2^k m_k,\qquad p=\sum_{k=0}^{n_p-1} 2^k p_k,\qquad q=\sum_{k=0}^{n_q-1} 2^k q_k,$$3

and the standard QAOA trial state is

$$m=\sum_{k=0}^{n_m-1} 2^k m_k,\qquad p=\sum_{k=0}^{n_p-1} 2^k p_k,\qquad q=\sum_{k=0}^{n_q-1} 2^k q_k,$$4

with the transverse-field mixer $$m=\sum_{k=0}^{n_m-1} 2^k m_k,\qquad p=\sum_{k=0}^{n_p-1} 2^k p_k,\qquad q=\sum_{k=0}^{n_q-1} 2^k q_k,$$5 (Qiu et al., 2020).

2. Hamiltonian reformulation and locality reduction

A major line of modification changes the Hamiltonian before circuit synthesis. In the superconducting-qubit resiliency study, the conventional DIRECT route algebraically expands the factoring Hamiltonian and leaves a 4-body interaction term. The same factorization objective is then reformulated in three alternative ways: SCHALLER, which uses

$$m=\sum_{k=0}^{n_m-1} 2^k m_k,\qquad p=\sum_{k=0}^{n_p-1} 2^k p_k,\qquad q=\sum_{k=0}^{n_q-1} 2^k q_k,$$6

to reduce the highest locality from 4-body to 3-body; GROBNER, which introduces an auxiliary binary variable $$m=\sum_{k=0}^{n_m-1} 2^k m_k,\qquad p=\sum_{k=0}^{n_p-1} 2^k p_k,\qquad q=\sum_{k=0}^{n_q-1} 2^k q_k,$$7 replacing $$m=\sum_{k=0}^{n_m-1} 2^k m_k,\qquad p=\sum_{k=0}^{n_p-1} 2^k p_k,\qquad q=\sum_{k=0}^{n_q-1} 2^k q_k,$$8 and uses a penalty with coefficients $$m=\sum_{k=0}^{n_m-1} 2^k m_k,\qquad p=\sum_{k=0}^{n_p-1} 2^k p_k,\qquad q=\sum_{k=0}^{n_q-1} 2^k q_k,$$9, $z_{i,j}$0, $z_{i,j}$1 to reduce 4-body to 2-body interactions; and SIM-GROBNER, which replaces the Gröbner penalty by the simpler squared consistency term

$z_{i,j}$2

reducing the maximum interaction order from 4 to 3 while increasing the number of qubits. The modified workflow inserts a transformation-and-selection stage between Hamiltonian construction and QAOA execution (Qiu et al., 2020).

A second and conceptually distinct reformulation is the null-space protocol. The standard digitized-adiabatic/QAOA factorization Hamiltonian is

$z_{i,j}$3

whereas the modified protocol replaces it by

$z_{i,j}$4

Because $z_{i,j}$5 is bilinear in the binary number operators, $z_{i,j}$6 contains only one- and two-body $z_{i,j}$7-type terms, while $z_{i,j}$8 expands as

$z_{i,j}$9

The spectral target also changes: the standard protocol searches for the ground state of a positive semidefinite cost Hamiltonian, whereas the modified protocol searches for a zero-eigenvalue state in $$0=\sum_{j=0}^{i} q_j p_{i-j}+\sum_{j=0}^i z_{j,i}-m_i-\sum_{j=1}^{n_c} 2^j z_{i,i+j},$$0. Accordingly, the initial state changes from $$0=\sum_{j=0}^{i} q_j p_{i-j}+\sum_{j=0}^i z_{j,i}-m_i-\sum_{j=1}^{n_c} 2^j z_{i,i+j},$$1 to the mixer eigenstate $$0=\sum_{j=0}^{i} q_j p_{i-j}+\sum_{j=0}^i z_{j,i}-m_i-\sum_{j=1}^{n_c} 2^j z_{i,i+j},$$2. The same study benchmarks three protocols: standard, with evolution and cost based on $$0=\sum_{j=0}^{i} q_j p_{i-j}+\sum_{j=0}^i z_{j,i}-m_i-\sum_{j=1}^{n_c} 2^j z_{i,i+j},$$3; linear_quadratic, with evolution under $$0=\sum_{j=0}^{i} q_j p_{i-j}+\sum_{j=0}^i z_{j,i}-m_i-\sum_{j=1}^{n_c} 2^j z_{i,i+j},$$4 but cost $$0=\sum_{j=0}^{i} q_j p_{i-j}+\sum_{j=0}^i z_{j,i}-m_i-\sum_{j=1}^{n_c} 2^j z_{i,i+j},$$5; and linear_abs, with evolution under $$0=\sum_{j=0}^{i} q_j p_{i-j}+\sum_{j=0}^i z_{j,i}-m_i-\sum_{j=1}^{n_c} 2^j z_{i,i+j},$$6 and cost $$0=\sum_{j=0}^{i} q_j p_{i-j}+\sum_{j=0}^i z_{j,i}-m_i-\sum_{j=1}^{n_c} 2^j z_{i,i+j},$$7 (Pellicer, 13 Nov 2025).

3. Circuit selection, noise resiliency, and hardware-aware compilation

In the superconducting-qubit setting, the central practical observation is that mathematically equivalent Hamiltonians can produce very different gate-level circuits and therefore very different noise behavior. Two explicit circuit-selection criteria are proposed: use fewer total CNOT gates, and use fewer CNOT gates per qubit. The stated rationale is that gate noise has a much larger impact on VQF than decoherence noise for the instances studied. For factoring $$0=\sum_{j=0}^{i} q_j p_{i-j}+\sum_{j=0}^i z_{j,i}-m_i-\sum_{j=1}^{n_c} 2^j z_{i,i+j},$$8, at noise level $$0=\sum_{j=0}^{i} q_j p_{i-j}+\sum_{j=0}^i z_{j,i}-m_i-\sum_{j=1}^{n_c} 2^j z_{i,i+j},$$9 and QAOA depth $0=\sum_{i=0}^{n_c} C_i^2.$0, the normalized residual performance gain is $0=\sum_{i=0}^{n_c} C_i^2.$1 for GROBNER, $0=\sum_{i=0}^{n_c} C_i^2.$2 for SIM-GROBNER, $0=\sum_{i=0}^{n_c} C_i^2.$3 for SCHALLER, and $0=\sum_{i=0}^{n_c} C_i^2.$4 for DIRECT. For factoring $0=\sum_{i=0}^{n_c} C_i^2.$5, DIRECT has $0=\sum_{i=0}^{n_c} C_i^2.$6 more CNOTs than GROBNER. The same study also reports that, for GROBNER in the $0=\sum_{i=0}^{n_c} C_i^2.$7 instance at fixed noise $0=\sum_{i=0}^{n_c} C_i^2.$8, $0=\sum_{i=0}^{n_c} C_i^2.$9 drops from $b_k \mapsto \frac12(1-\sigma^z_{b,k})$0 to $b_k \mapsto \frac12(1-\sigma^z_{b,k})$1 as $b_k \mapsto \frac12(1-\sigma^z_{b,k})$2 increases from $b_k \mapsto \frac12(1-\sigma^z_{b,k})$3 to $b_k \mapsto \frac12(1-\sigma^z_{b,k})$4, illustrating that deeper QAOA can become counterproductive on NISQ hardware (Qiu et al., 2020).

A different modification addresses hardware topology directly. The LHZ-based QAOA scheme maps an all-to-all Ising problem with logical couplings $b_k \mapsto \frac12(1-\sigma^z_{b,k})$5 into a square-lattice gauge model with

$b_k \mapsto \frac12(1-\sigma^z_{b,k})$6

physical qubits and $b_k \mapsto \frac12(1-\sigma^z_{b,k})$7 plaquette constraints. In this representation,

$b_k \mapsto \frac12(1-\sigma^z_{b,k})$8

The standard cost layer is split into local programmable $b_k \mapsto \frac12(1-\sigma^z_{b,k})$9-field evolution and problem-independent constraint evolution. Three protocols are compared: $O(n_m^2)$0

$O(n_m^2)$1

$O(n_m^2)$2

Each 4-body plaquette unitary is implemented with 6 CNOT gates and one single-qubit $O(n_m^2)$3 rotation, and the full constraint set can be realized in 28 parallel gate operations independent of system size. The paper states that “the separation of local field terms and interaction terms is advantageous,” and that optimization of the constraint strengths $O(n_m^2)$4 “does further improve the fidelity” (Lechner, 2018).

4. Fixed-point parameterization and universal-angle variants

Another modification removes per-instance variational optimization at runtime. In the trapped-ion factoring experiment, QAOA is embedded not in a direct multiplication-table Hamiltonian but in a Schnorr/Babai smooth-relation workflow: the quantum subroutine solves a QUBO corresponding to a closest vector problem refinement step. The modified component is a fixed-point or fixed-angles QAOA strategy in which a single set of universal angles is learned offline from a training set and then reused after Hamiltonian normalization. The paper states that it uses a training set of 100 QUBO subproblems arising during factoring of $O(n_m^2)$5 on $O(n_m^2)$6 qubits, normalizes every QUBO coefficient matrix by its maximal value, maximizes the minimum of $O(n_m^2)$7 over the training set using a random mutations optimization algorithm, and then executes only a single QAOA layer,

$O(n_m^2)$8

with universal parameters

$O(n_m^2)$9

There is no per-instance classical optimization loop over $O(n_m\log n_m)$0 during factoring (Zalivako et al., 13 Mar 2025).

This fixed-point protocol was demonstrated experimentally on a trapped-ion processor for

$O(n_m\log n_m)$1

using 6 qubits, 9 different circuits, and 5 shots per circuit. The run used

$O(n_m\log n_m)$2

The paper reports a total of 45 experimental shots, with 43 shots sufficient to collect 12 sr-pairs and complete the run, and states that in the particular sample run the first 39 shots were already sufficient to factorize $O(n_m\log n_m)$3. It also presents simulation results for

$O(n_m\log n_m)$4

with 10 qubits and

$O(n_m\log n_m)$5

with 15 qubits, while emphasizing that scalability in both the classical and quantum domains requires further study (Zalivako et al., 13 Mar 2025).

A broader fixed-parameter template appears in work on generic QUBO optimization. There, a modified fpQAOA scheme combines three ingredients: optimization for the probability of achieving a target approximation ratio rather than the exact optimum, the choice $O(n_m\log n_m)$6 together with the two-parameter sine–cosine encoding

$O(n_m\log n_m)$7

and Frobenius normalization

$O(n_m\log n_m)$8

The paper does not discuss integer factorization explicitly, but this suggests a transferable fpQAOA template when a factoring method is first reduced to a QUBO/Ising instance. It also reports that omitting even one of these three modifications results in exponential growth of the median shots required on the tested random QUBO ensembles (Chernyavskiy et al., 23 Sep 2025).

5. Reported performance regimes and representative results

The literature does not report a single performance metric. Some papers evaluate squared overlap with the solution manifold, some use success probability of reading out correct factors, some use normalized residual performance gain under noise, and some count unique smooth relation pairs. The resulting picture is therefore heterogeneous rather than uniform.

Work	Modification	Reported outcome
VQF (Anschuetz et al., 2018)	Preprocessing + standard QAOA	Tested $O(n_m\log n_m)$9; easier instances often had no carry bits and $O(n_m)$0 symmetry
Resilient VQF (Qiu et al., 2020)	DIRECT/SCHALLER/GROBNER/SIM-GROBNER + circuit selection	For $O(n_m)$1 at $O(n_m)$2, GROBNER $O(n_m)$3, SIM-GROBNER $O(n_m)$4, SCHALLER $O(n_m)$5, DIRECT $O(n_m)$6
Fixed-point trapped-ion factoring (Zalivako et al., 13 Mar 2025)	Universal-angle $O(n_m)$7 QAOA inside Schnorr pipeline	Factored $O(n_m)$8 with 6 qubits; 43 shots were enough to collect 12 sr-pairs
Null-space linearized protocol (Pellicer, 13 Nov 2025)	$O(n_m)$9	Numerical simulations up to 8 qubits reported comparable or higher fidelities with fewer quantum resources

Within baseline VQF, the structural role of carry bits is especially prominent. The paper states that success probability rises and then plateaus at a large value after approximately $143=11\times 13$0 layers for $143=11\times 13$1, $143=11\times 13$2, and $143=11\times 13$3, but is substantially worse for $143=11\times 13$4, $143=11\times 13$5, and $143=11\times 13$6. It further sharpens this observation by noting that preliminary numerics suggest the mere presence of carry bits negatively affects the algorithm, with relatively weak dependence on the exact number of carry bits at fixed size (Anschuetz et al., 2018).

Within the null-space study, the modified protocol exhibits a distinct spectral signature. The paper reports abrupt fidelity jumps as depth increases and attributes them qualitatively to a more widely spread spectrum for $143=11\times 13$7. The RMS spectral spreads listed for $143=11\times 13$8 to $143=11\times 13$9 are $$m=\sum_{k=0}^{n_m-1} 2^k m_k,\qquad p=\sum_{k=0}^{n_p-1} 2^k p_k,\qquad q=\sum_{k=0}^{n_q-1} 2^k q_k,$$00 for the standard Hamiltonian and $$m=\sum_{k=0}^{n_m-1} 2^k m_k,\qquad p=\sum_{k=0}^{n_p-1} 2^k p_k,\qquad q=\sum_{k=0}^{n_q-1} 2^k q_k,$$01 for the linearized Hamiltonian. It also reports that linear_abs generally outperformed linear_quadratic (Pellicer, 13 Nov 2025).

6. Scope, limitations, and points of dispute

A recurrent misconception is to treat these modifications as a single canonical protocol or as an established replacement for Shor’s algorithm. The literature instead describes several distinct modifications aimed at different bottlenecks: qubit reduction by preprocessing, locality reduction by Hamiltonian reformulation, CNOT reduction by algebraic transformation, connectivity-aware compilation, or elimination of runtime variational search. The claims are correspondingly local. VQF explicitly does not claim asymptotic superiority over classical factoring or Shor’s algorithm; the trapped-ion fixed-point work states that the scalability of the approach in both the classical and quantum domain still requires further studies; and the null-space protocol reports only numerical simulations up to eight qubits and no full hardware demonstration in that work (Anschuetz et al., 2018, Zalivako et al., 13 Mar 2025, Pellicer, 13 Nov 2025).

Another unsettled issue concerns the relation between approximate optimization and exact factor recovery. The generic fpQAOA work studies random QUBO instances rather than factorization and optimizes for the probability of achieving a target approximation ratio, not for exact factors. This suggests a plausible extension to factorization only when a factoring method is first reduced to a QUBO, and even then the mapping from “high approximation ratio” to “recoverable factor pair” is not established by that paper (Chernyavskiy et al., 23 Sep 2025). A further documentary caution concerns "Integer Factorization through Func-QAOA" (Atallah et al., 2023): the provided content is described as containing only a small circuit diagram showing a simplification based on $$m=\sum_{k=0}^{n_m-1} 2^k m_k,\qquad p=\sum_{k=0}^{n_p-1} 2^k p_k,\qquad q=\sum_{k=0}^{n_q-1} 2^k q_k,$$02 and the XOR cancellation law $$m=\sum_{k=0}^{n_m-1} 2^k m_k,\qquad p=\sum_{k=0}^{n_p-1} 2^k p_k,\qquad q=\sum_{k=0}^{n_q-1} 2^k q_k,$$03, with no text supporting claims about factorization, QAOA, Func-QAOA, arithmetic oracles, search-space reduction, or empirical results. Accordingly, it does not presently substantiate a factorization-specific modified protocol (Atallah et al., 2023).

Taken together, the literature supports a precise but narrow conclusion: modified QAOA-based factorization protocols are best understood as ansatz-engineering and compilation strategies for NISQ-constrained factorization workflows. Their demonstrated advances are in Hamiltonian locality, CNOT economy, hardware compatibility, parameter transfer, and small-instance execution, rather than in a settled asymptotic theory of efficient quantum factorization (Lechner, 2018, Qiu et al., 2020, Zalivako et al., 13 Mar 2025).