---
title: Modified QAOA Factorization Protocol
url: https://www.emergentmind.com/topics/modified-qaoa-based-factorization-protocol
type: topic
---

# Modified QAOA Factorization Protocol

Modified QAOA-based factorization protocol denotes a class of NISQ-era integer-factorization methods in which the standard Quantum Approximate Optimization Algorithm is altered at the level of the factoring Hamiltonian, circuit synthesis, variational parameterization, or the surrounding hybrid classical–quantum workflow. In the current literature, these modifications include Boolean preprocessing and variational quantum factoring, Hamiltonian and circuit transformations chosen for superconducting-noise resilience, hardware-aware split-cost QAOA on LHZ lattices for dense Ising encodings, fixed-point QAOA with universal parameters inside Schnorr’s smooth-relation workflow, and null-space encodings that replace the squared residual Hamiltonian by a linear one [1808.08927, 2004.12447, 1802.01157, 2503.10588, 2511.11747].

## 1. Baseline formulation: from binary multiplication to QAOA

The canonical starting point is to encode the factorization problem \(m=pq\) in binary. In Variational Quantum Factoring, the factors and target integer are written as
\[
m=\sum_{k=0}^{n_m-1} 2^k m_k,\qquad
p=\sum_{k=0}^{n_p-1} 2^k p_k,\qquad
q=\sum_{k=0}^{n_q-1} 2^k q_k,
\]
with binary carry variables \(z_{i,j}\) used to express columnwise multiplication constraints. The central clause is
\[
0=\sum_{j=0}^{i} q_j p_{i-j}+\sum_{j=0}^i z_{j,i}-m_i-\sum_{j=1}^{n_c} 2^j z_{i,i+j},
\]
and factorization is reduced to the zero-energy condition
\[
0=\sum_{i=0}^{n_c} C_i^2.
\]
After operator substitution \(b_k \mapsto \frac12(1-\sigma^z_{b,k})\), this becomes a 4-local Ising Hamiltonian used as the QAOA cost Hamiltonian. The accompanying preprocessing pass applies deterministic Boolean identities, carry-range truncation, substitution, and propagation. The paper states that preprocessing runtime is \(O(n_m^2)\), that the asymptotic qubit count without simplification is \(O(n_m\log n_m)\), and that with simplification the empirical scaling is approximately \(O(n_m)\) [1808.08927].

A more explicit small-instance realization appears in the \(143=11\times 13\) example of variational quantum factoring on superconducting hardware. There, binary multiplication constraints simplify to
\[
p_1 + q_1 - 1 = 0,\qquad
p_2 + q_2 - 1 = 0,\qquad
p_2 q_1 + p_1 q_2 - 1 = 0,
\]
giving the factoring cost
\[
C_f = (p_1 + q_1 - 1)^2 + (p_2 + q_2 - 1)^2 + (p_2 q_1 + p_1 q_2 - 1)^2.
\]
With \(\hat z_i = \frac{1-\sigma_i^z}{2}\), the problem Hamiltonian becomes
\[
H_c = (\hat p_1 + \hat q_1 - 1)^2 + (\hat p_2 + \hat q_2 - 1)^2 + (\hat p_2 \hat q_1 + \hat p_1 \hat q_2 - 1)^2,
\]
and the standard QAOA trial state is
\[
\lvert \psi_F(\boldsymbol{\gamma},\boldsymbol{\beta}) \rangle
= \prod_{\ell=1}^{p} e^{-i \beta_\ell H_M} e^{-i \gamma_\ell H_c} \lvert + \rangle^{\otimes n},
\]
with the transverse-field mixer \(H_M=\sum_i \sigma_i^x\) [2004.12447].

## 2. Hamiltonian reformulation and locality reduction

A major line of modification changes the Hamiltonian before circuit synthesis. In the superconducting-qubit resiliency study, the conventional **DIRECT** route algebraically expands the factoring Hamiltonian and leaves a 4-body interaction term. The same factorization objective is then reformulated in three alternative ways: **SCHALLER**, which uses
\[
AB + S = 0 \quad \mapsto \quad 2\left[\frac{1}{2}\left(A+B-\frac12\right)+S\right]^2 - \frac18
\]
to reduce the highest locality from 4-body to 3-body; **GROBNER**, which introduces an auxiliary binary variable \(w_{ij}\) replacing \(p_i q_j\) and uses a penalty with coefficients \(a=-2\), \(b=-2\), \(c=1\) to reduce 4-body to 2-body interactions; and **SIM-GROBNER**, which replaces the Gröbner penalty by the simpler squared consistency term
\[
(p_i q_j - w_{ij})^{+} = (p_i q_j - w_{ij})^2,
\]
reducing the maximum interaction order from 4 to 3 while increasing the number of qubits. The modified workflow inserts a transformation-and-selection stage between Hamiltonian construction and QAOA execution [2004.12447].

A second and conceptually distinct reformulation is the null-space protocol. The standard digitized-adiabatic/QAOA factorization Hamiltonian is
\[
\hat H_{\mathrm{QP}} = \big(N\mathbf 1 - \hat P \hat Q\big)^2,
\]
whereas the modified protocol replaces it by
\[
\hat H_{\mathrm{LP}} = N\mathbf 1 - \hat P \hat Q.
\]
Because \(\hat P\hat Q\) is bilinear in the binary number operators, \(\hat H_{\mathrm{LP}}\) contains only one- and two-body \(Z\)-type terms, while \(\hat H_{\mathrm{QP}}\) expands as
\[
\hat{H}_\mathrm{QP} = n \mathbf 1 + \sum_i a_i \hat{\sigma}_i^z + \sum_{ij} b_{ij} \hat{\sigma}_i^z \hat{\sigma}_j^z + \sum_{ijk} c_{ijk} \hat{\sigma}_i^z \hat{\sigma}_j^z \hat{\sigma}_k^z + \sum_{ijk\ell} d_{ijk\ell} \hat{\sigma}_i^z \hat{\sigma}_j^z \hat{\sigma}_k^z \hat{\sigma}_\ell^z.
\]
The spectral target also changes: the standard protocol searches for the ground state of a positive semidefinite cost Hamiltonian, whereas the modified protocol searches for a zero-eigenvalue state in \(\ker(\hat H_{\mathrm{LP}})\). Accordingly, the initial state changes from \(\ket{+}^{\otimes n}\) to the mixer eigenstate \(\ket{+-+-+\cdots}\). The same study benchmarks three protocols: **standard**, with evolution and cost based on \(\hat H_{\mathrm{QP}}\); **linear\_quadratic**, with evolution under \(\hat H_{\mathrm{LP}}\) but cost \(\langle \hat H_{\mathrm{QP}}\rangle\); and **linear\_abs**, with evolution under \(\hat H_{\mathrm{LP}}\) and cost \(\langle |\hat H_{\mathrm{LP}}| \rangle\) [2511.11747].

## 3. Circuit selection, noise resiliency, and hardware-aware compilation

In the superconducting-qubit setting, the central practical observation is that mathematically equivalent Hamiltonians can produce very different gate-level circuits and therefore very different noise behavior. Two explicit circuit-selection criteria are proposed: use fewer total CNOT gates, and use fewer CNOT gates per qubit. The stated rationale is that gate noise has a much larger impact on VQF than decoherence noise for the instances studied. For factoring \(143\), at noise level \(i=0.4\) and QAOA depth \(p=2\), the normalized residual performance gain is \(47.26\%\) for GROBNER, \(38.79\%\) for SIM-GROBNER, \(17.91\%\) for SCHALLER, and \(11.13\%\) for DIRECT. For factoring \(291311\), DIRECT has \(60\%\) more CNOTs than GROBNER. The same study also reports that, for GROBNER in the \(143\) instance at fixed noise \(i=0.4\), \(G_{i=0.4,p}\) drops from \(63.37\%\) to \(15.21\%\) as \(p\) increases from \(1\) to \(4\), illustrating that deeper QAOA can become counterproductive on NISQ hardware [2004.12447].

A different modification addresses hardware topology directly. The LHZ-based QAOA scheme maps an all-to-all Ising problem with logical couplings \(J_{ij}\) into a square-lattice gauge model with
\[
K=\frac{N(N-1)}{2}
\]
physical qubits and \(K-N+1\) plaquette constraints. In this representation,
\[
U_x(\beta)=\prod_{i=1}^K e^{-i\beta \sigma_x^{(i)}},\qquad
U_z(\gamma)=\prod_{i=1}^K e^{-i\gamma J_i \sigma_z^{(i)}},\qquad
U_c(\Omega)= \prod_{l=1}^{K-N+1} e^{-i\Omega C_l \sigma_z^{(l,n)}\sigma_z^{(l,e)}\sigma_z^{(l,s)}\sigma_z^{(l,w)}}.
\]
The standard cost layer is split into local programmable \(Z\)-field evolution and problem-independent constraint evolution. Three protocols are compared:
\[
U_a = U_p(\gamma_0)U_x(\beta_1)U_p(\gamma_1)\cdots,
\]
\[
U_b = U_z(\gamma_0)U_c(\Omega_0)U_x(\beta_1)U_z(\gamma_1)U_c(\Omega_1)\cdots,
\]
\[
U_c = U_z(\gamma_0)U_c(\Omega_0,C_l)U_x(\beta_1)U_z(\gamma_1)U_c(\Omega_1,C_l)\cdots.
\]
Each 4-body plaquette unitary is implemented with 6 CNOT gates and one single-qubit \(R_Z\) rotation, and the full constraint set can be realized in 28 parallel gate operations independent of system size. The paper states that “the separation of local field terms and interaction terms is advantageous,” and that optimization of the constraint strengths \(C_l\) “does further improve the fidelity” [1802.01157].

## 4. Fixed-point parameterization and universal-angle variants

Another modification removes per-instance variational optimization at runtime. In the trapped-ion factoring experiment, QAOA is embedded not in a direct multiplication-table Hamiltonian but in a Schnorr/Babai smooth-relation workflow: the quantum subroutine solves a QUBO corresponding to a closest vector problem refinement step. The modified component is a fixed-point or fixed-angles QAOA strategy in which a single set of universal angles is learned offline from a training set and then reused after Hamiltonian normalization. The paper states that it uses a training set of 100 QUBO subproblems arising during factoring of \(N=48567227\) on \(n=10\) qubits, normalizes every QUBO coefficient matrix by its maximal value, maximizes the minimum of \(P_q/P_c\) over the training set using a random mutations optimization algorithm, and then executes only a single QAOA layer,
\[
\ket{\beta_*,\gamma_*} = e^{-i\beta_* H_M} e^{-i\gamma_* H_P}\ket{+}^{\otimes n},
\]
with universal parameters
\[
\gamma_* = 2.64,\qquad \beta_* = 0.33.
\]
There is no per-instance classical optimization loop over \((\gamma,\beta)\) during factoring [2503.10588].

This fixed-point protocol was demonstrated experimentally on a trapped-ion processor for
\[
1591 = 37\times 43
\]
using 6 qubits, 9 different circuits, and 5 shots per circuit. The run used
\[
B_1=6,\qquad B_2=11,\qquad c=1.5,\qquad \delta=0.75,\qquad N_{\rm sh}=5.
\]
The paper reports a total of 45 experimental shots, with 43 shots sufficient to collect 12 sr-pairs and complete the run, and states that in the particular sample run the first 39 shots were already sufficient to factorize \(1591\). It also presents simulation results for
\[
74425657=9521\times 7817
\]
with 10 qubits and
\[
35183361263263=4194191\times 8388593
\]
with 15 qubits, while emphasizing that scalability in both the classical and quantum domains requires further study [2503.10588].

A broader fixed-parameter template appears in work on generic QUBO optimization. There, a modified fpQAOA scheme combines three ingredients: optimization for the probability of achieving a target approximation ratio rather than the exact optimum, the choice \(p=n\) together with the two-parameter sine–cosine encoding
\[
\gamma_i = u \sin\!\left[\left(i-\tfrac{1}{2}\right)\frac{\pi}{2n}\right],\qquad
\beta_i = v \cos\!\left[\left(i-\tfrac{1}{2}\right)\frac{\pi}{2n}\right],
\]
and Frobenius normalization
\[
S \mapsto \frac{S}{\|S\|_{\mathrm{Frob}}}.
\]
The paper does not discuss integer factorization explicitly, but this suggests a transferable fpQAOA template when a factoring method is first reduced to a QUBO/Ising instance. It also reports that omitting even one of these three modifications results in exponential growth of the median shots required on the tested random QUBO ensembles [2509.19035].

## 5. Reported performance regimes and representative results

The literature does not report a single performance metric. Some papers evaluate squared overlap with the solution manifold, some use success probability of reading out correct factors, some use normalized residual performance gain under noise, and some count unique smooth relation pairs. The resulting picture is therefore heterogeneous rather than uniform.

| Work | Modification | Reported outcome |
|---|---|---|
| VQF [1808.08927] | Preprocessing + standard QAOA | Tested \(35,77,1207,33667,56153,291311\); easier instances often had no carry bits and \(p\leftrightarrow q\) symmetry |
| Resilient VQF [2004.12447] | DIRECT/SCHALLER/GROBNER/SIM-GROBNER + circuit selection | For \(143\) at \(i=0.4,p=2\), GROBNER \(47.26\%\), SIM-GROBNER \(38.79\%\), SCHALLER \(17.91\%\), DIRECT \(11.13\%\) |
| Fixed-point trapped-ion factoring [2503.10588] | Universal-angle \(p=1\) QAOA inside Schnorr pipeline | Factored \(1591=37\times 43\) with 6 qubits; 43 shots were enough to collect 12 sr-pairs |
| Null-space linearized protocol [2511.11747] | \(\hat H_{\mathrm{LP}}=N\mathbf 1-\hat P\hat Q\) | Numerical simulations up to 8 qubits reported comparable or higher fidelities with fewer quantum resources |

Within baseline VQF, the structural role of carry bits is especially prominent. The paper states that success probability rises and then plateaus at a large value after approximately \(O(n)\) layers for \(35\), \(56153\), and \(291311\), but is substantially worse for \(77\), \(1207\), and \(33667\). It further sharpens this observation by noting that preliminary numerics suggest the mere presence of carry bits negatively affects the algorithm, with relatively weak dependence on the exact number of carry bits at fixed size [1808.08927].

Within the null-space study, the modified protocol exhibits a distinct spectral signature. The paper reports abrupt fidelity jumps as depth increases and attributes them qualitatively to a more widely spread spectrum for \(\hat H_{\mathrm{LP}}\). The RMS spectral spreads listed for \(n=3\) to \(8\) are \(0.59, 0.58, 0.26, 0.24, 0.20, 0.21\) for the standard Hamiltonian and \(0.70, 0.68, 0.41, 0.37, 0.32, 0.32\) for the linearized Hamiltonian. It also reports that **linear\_abs** generally outperformed **linear\_quadratic** [2511.11747].

## 6. Scope, limitations, and points of dispute

A recurrent misconception is to treat these modifications as a single canonical protocol or as an established replacement for Shor’s algorithm. The literature instead describes several distinct modifications aimed at different bottlenecks: qubit reduction by preprocessing, locality reduction by Hamiltonian reformulation, CNOT reduction by algebraic transformation, connectivity-aware compilation, or elimination of runtime variational search. The claims are correspondingly local. VQF explicitly does not claim asymptotic superiority over classical factoring or Shor’s algorithm; the trapped-ion fixed-point work states that the scalability of the approach in both the classical and quantum domain still requires further studies; and the null-space protocol reports only numerical simulations up to eight qubits and no full hardware demonstration in that work [1808.08927, 2503.10588, 2511.11747].

Another unsettled issue concerns the relation between approximate optimization and exact factor recovery. The generic fpQAOA work studies random QUBO instances rather than factorization and optimizes for the probability of achieving a target approximation ratio, not for exact factors. This suggests a plausible extension to factorization only when a factoring method is first reduced to a QUBO, and even then the mapping from “high approximation ratio” to “recoverable factor pair” is not established by that paper [2509.19035]. A further documentary caution concerns "Integer Factorization through Func-QAOA" [2309.15162]: the provided content is described as containing only a small circuit diagram showing a simplification based on \(b=b'\) and the XOR cancellation law \(x\oplus x=0\), with no text supporting claims about factorization, QAOA, Func-QAOA, arithmetic oracles, search-space reduction, or empirical results. Accordingly, it does not presently substantiate a factorization-specific modified protocol [2309.15162].

Taken together, the literature supports a precise but narrow conclusion: modified QAOA-based factorization protocols are best understood as ansatz-engineering and compilation strategies for NISQ-constrained factorization workflows. Their demonstrated advances are in Hamiltonian locality, CNOT economy, hardware compatibility, parameter transfer, and small-instance execution, rather than in a settled asymptotic theory of efficient quantum factorization [1802.01157, 2004.12447, 2503.10588].

Source: https://www.emergentmind.com/topics/modified-qaoa-based-factorization-protocol