Continuous Relaxed Babai's Nearest Plane (CR-BNP)

• CR-BNP is defined as a continuous relaxation of Babai’s nearest-plane algorithm, deferring immediate rounding to preserve the upper-triangular lattice structure.
• It quantifies residual ambiguity to enable adaptive mixed-radix encoding, optimizing the QUBO formulation for hardware-constrained LWE cryptanalysis.
• Quantitative analyses demonstrate that CR-BNP widens the admissible rounding interval, significantly boosting the probability of correct lattice decoding.

Searching arXiv for CR-BNP and closely related Babai nearest-plane work. Continuous Relaxed Babai’s Nearest Plane (CR-BNP) is a continuous relaxation of Babai’s nearest-plane procedure applied to a lattice closest-vector formulation, introduced explicitly as part of the CIM-BDD pipeline for Learning With Errors (LWE) cryptanalysis on Coherent Ising Machine hardware. In that formulation, CR-BNP suspends the discrete nearest-integer rounding step within a designated exploration subspace, solves the resulting upper-triangular system in $\mathbb{R}$, and uses the resulting fractional ambiguities to center and compress a subsequent discrete search. The method is therefore not a replacement for lattice decoding in general, but a geometry-aware intermediate construction: it preserves the Gram–Schmidt / triangular structure of nearest-plane decoding while deferring integrality so that the remaining ambiguity can be encoded more economically (Jiang, 22 Jun 2026).

1. Definition and conceptual scope

CR-BNP is defined on the exploration block of a reduced lattice basis $\mathbf{B}_0$, after Gram–Schmidt decomposition

$\mathbf{B}_0 = \mathbf{B}^* \mathbf{U}_\mu,$

with $$\mathbf{B}^*=[\mathbf{b}_1^*,\dots,\mathbf{b}_{m'}^*]$$ orthogonal and $\mathbf{U}_\mu$ upper unit-triangular. For a target $\mathbf{c}'$, the associated projection coefficients are

$$\alpha_i = \frac{\langle \mathbf{c}', \mathbf{b}_i^*\rangle}{\|\mathbf{b}_i^*\|^2}, \qquad i=1,\dots,m'.$$

Within the trailing exploration subspace, CR-BNP is defined by solving the continuous triangular system

$$\mathbf{U}_{\mu,\text{exp}} \mathbf{y}_c = \boldsymbol{\alpha}_{\text{exp}}$$

instead of performing Babai’s coordinatewise rounding during back-substitution. The relaxed coordinate vector is then rounded only once to obtain the discrete center

$$\mathbf{y}_{\text{center}} = \lfloor \mathbf{y}_c \rceil,$$

and the residual ambiguity is isolated as

$$\boldsymbol{\delta} = \mathbf{y}_c - \mathbf{y}_{\text{center}} \in [-0.5,0.5]^{d_{\text{exp}}}.$$

The paper introducing the term states the definition directly: “Within the exploration subspace, we deliberately defer the discrete rounding operation ($\mathbf{B}_0$0). Instead, we allow the modified algorithm to use continuous projection coefficients to sequentially compute the continuous coordinates ($\mathbf{B}_0$1). We term this approach Continuous Relaxed Babai’s Nearest Plane (CR-BNP)” (Jiang, 22 Jun 2026).

This places CR-BNP between two objects. On one side is classical Babai decoding, whose recursion is integer-forcing at every stage. On the other is a fully continuous least-squares solve in the orthogonalized coordinates. CR-BNP retains the ordered nearest-plane geometry and the same upper-triangular dependency structure, but relaxes the integrality constraint locally and temporarily. A plausible implication is that CR-BNP should be regarded less as a new lattice-decoding endpoint than as a controlled softening of the Babai recursion in those coordinates reserved for downstream exploration.

2. Mathematical construction

The immediate derivation comes from a closest vector problem of the form

$\mathbf{B}_0$2

Expressing $\mathbf{B}_0$3 and projecting $\mathbf{B}_0$4 onto the orthogonal basis produces $\mathbf{B}_0$5. If the integrality restriction is ignored on the exploration block, the projected least-squares minimizer is exactly the continuous solution of

$\mathbf{B}_0$6

Because $\mathbf{B}_0$7 is upper-triangular, the coordinates are obtained by backward substitution: $\mathbf{B}_0$8 The source paper describes this explicitly as an $\mathbf{B}_0$9 Gaussian back-substitution formula (Jiang, 22 Jun 2026).

The conceptual modification relative to Babai’s classical nearest-plane rule is correspondingly narrow. In the standard nearest-plane algorithm, one computes floating-point coefficients and rounds each coefficient immediately to the nearest integer, propagating the discrete choice through the remaining coordinates. In CR-BNP, the same triangular recursion is followed over the exploration coordinates, but the recursion remains real-valued until the end of the block. The output is therefore a triplet: a continuous point $\mathbf{B}_0 = \mathbf{B}^* \mathbf{U}_\mu,$0, a discrete center $\mathbf{B}_0 = \mathbf{B}^* \mathbf{U}_\mu,$1, and a bounded ambiguity vector $\mathbf{B}_0 = \mathbf{B}^* \mathbf{U}_\mu,$2.

This ambiguity vector is not ancillary; it is the object exported to the next stage. The paper interprets $\mathbf{B}_0 = \mathbf{B}^* \mathbf{U}_\mu,$3 as a measure of “spatial ambiguity,” and the subsequent encoding decisions depend directly on $\mathbf{B}_0 = \mathbf{B}^* \mathbf{U}_\mu,$4. In that sense, CR-BNP does not merely approximate a lattice point. It computes a local coordinate chart around a Babai-like center and quantifies how fragile each coordinate’s hard rounding would have been.

3. Relation to classical Babai nearest-plane decoding

Classical Babai nearest-plane decoding is a recursive closest-vector heuristic on a lattice basis. In one formulation, for orthogonalized directions $\mathbf{B}_0 = \mathbf{B}^* \mathbf{U}_\mu,$5 and basis vectors $\mathbf{B}_0 = \mathbf{B}^* \mathbf{U}_\mu,$6, Babai computes

$\mathbf{B}_0 = \mathbf{B}^* \mathbf{U}_\mu,$7

for $\mathbf{B}_0 = \mathbf{B}^* \mathbf{U}_\mu,$8. In upper-triangular coordinates, the same structure appears as sequential scalar rounding of affine residuals (Chen et al., 24 Jul 2025).

CR-BNP changes only the nearest-integer part of this architecture. The projection coefficients, the triangular dependence, and the backward ordering remain intact. What is suspended is the immediate map

$\mathbf{B}_0 = \mathbf{B}^* \mathbf{U}_\mu,$9

The later GPTQ/Babai equivalence paper makes this point especially sharp by observing that, once a triangular basis is chosen, “the only discrete step is the rounding” (Chen et al., 24 Jul 2025). That observation is not itself a definition of CR-BNP, but it identifies exactly why CR-BNP is mathematically natural: a continuous relaxation can preserve the geometry of nearest-plane decoding while relaxing only the scalar integer projection.

Earlier distributed nearest-plane analyses provide the geometric background for this interpretation. They treat Babai decoding as a rectangular approximation to Voronoi decoding, with error localized to the part of a Babai cell occupied by neighboring Voronoi cells. In the two-dimensional setting of the Babai-to-Voronoi partition refinement problem, the residual error after coarse Babai decoding is expressed by overlap areas such as

$$\mathbf{B}^*=[\mathbf{b}_1^*,\dots,\mathbf{b}_{m'}^*]$$0

and successive refinement resolves only those boundary-crossing regions (Vaishampayan et al., 2017). This suggests a useful conceptual reading of CR-BNP: rather than committing to hard rounding everywhere, it maintains a continuous estimate precisely where the nearest-plane decision is most sensitive.

The distributed nearest-plane papers do not define CR-BNP, and they do not introduce differentiable or probabilistic rounding. Their contribution is instead structural. They make explicit that Babai’s approximation error is a boundary mismatch phenomenon, that the mismatch is basis dependent, and that refinement should concentrate on ambiguity regions rather than on the full cell (Bollauf et al., 2017). A plausible implication is that CR-BNP inherits its strongest justification from the same locality principle.

4. Role in adaptive mixed-radix encoding and penalty-free QUBO mapping

In the CIM-BDD construction, CR-BNP is not the final decoder. It is the pre-encoding geometric relaxation that makes a penalty-free closest-vector formulation small enough for current Coherent Ising Machine hardware. The underlying LWE instance is first reduced to a $$\mathbf{B}^*=[\mathbf{b}_1^*,\dots,\mathbf{b}_{m'}^*]$$1-ary lattice CVP with objective

$$\mathbf{B}^*=[\mathbf{b}_1^*,\dots,\mathbf{b}_{m'}^*]$$2

so that the cryptographic noise becomes the quantity to be minimized directly rather than a penalized constraint. The difficulty is then hardware realization: direct encoding of the full integer vector is too large and too precision-demanding. CR-BNP is introduced to localize the remaining search (Jiang, 22 Jun 2026).

The adaptive mixed-radix encoder uses thresholds $$\mathbf{B}^*=[\mathbf{b}_1^*,\dots,\mathbf{b}_{m'}^*]$$3 and $$\mathbf{B}^*=[\mathbf{b}_1^*,\dots,\mathbf{b}_{m'}^*]$$4 on $$\mathbf{B}^*=[\mathbf{b}_1^*,\dots,\mathbf{b}_{m'}^*]$$5. According to the paper’s rule, if $$\mathbf{B}^*=[\mathbf{b}_1^*,\dots,\mathbf{b}_{m'}^*]$$6, the coordinate is treated as unambiguous and assigned $$\mathbf{B}^*=[\mathbf{b}_1^*,\dots,\mathbf{b}_{m'}^*]$$7 bits; if $$\mathbf{B}^*=[\mathbf{b}_1^*,\dots,\mathbf{b}_{m'}^*]$$8, it is given $$\mathbf{B}^*=[\mathbf{b}_1^*,\dots,\mathbf{b}_{m'}^*]$$9 bit with a directional bias; and if $\mathbf{U}_\mu$0, it is assigned $\mathbf{U}_\mu$1 bits corresponding to a ternary local move. The exploration coefficients searched by hardware are then parameterized as

$\mathbf{U}_\mu$2

with $\mathbf{U}_\mu$3 (Jiang, 22 Jun 2026).

The projected exploration-space energy remains quadratic: $\mathbf{U}_\mu$4 where

$\mathbf{U}_\mu$5

Because this objective is already quadratic in $\mathbf{U}_\mu$6, no penalty terms are required. CR-BNP therefore functions as the mechanism that determines the center, the local search directions, and the degree of per-coordinate pruning before the QUBO is formed.

The practical significance stated in the paper is hardware-oriented. In the $\mathbf{U}_\mu$7 experiments with $\mathbf{U}_\mu$8, the encoder compresses the exploration problem to between $\mathbf{U}_\mu$9 and $\mathbf{c}'$0 logical bits depending on the seed, and the paper claims that this “compresses the target combinatorial space, restricting the $\mathbf{c}'$1 instance to fewer than $\mathbf{c}'$2 logical qubits,” while also narrowing the dynamic amplitude range of the QUBO coefficients (Jiang, 22 Jun 2026). In this setting, CR-BNP is best understood as a geometry-aware search-space restriction coupled tightly to an analog hardware budget.

5. Performance characteristics, probabilistic analysis, and limitations

The main formal CR-BNP-specific analysis in the CIM-BDD work concerns coordinate error in the exploration block. If

$\mathbf{c}'$3

then for the last exploration dimension the paper derives

$\mathbf{c}'$4

In its representative $\mathbf{c}'$5 example, $\mathbf{c}'$6, so

$\mathbf{c}'$7

Under standard single-coordinate hard Babai rounding, success requires the error to lie in $\mathbf{c}'$8, which the paper evaluates as

$\mathbf{c}'$9

Allowing a ternary offset $$\alpha_i = \frac{\langle \mathbf{c}', \mathbf{b}_i^*\rangle}{\|\mathbf{b}_i^*\|^2}, \qquad i=1,\dots,m'.$$0 widens the admissible interval to $$\alpha_i = \frac{\langle \mathbf{c}', \mathbf{b}_i^*\rangle}{\|\mathbf{b}_i^*\|^2}, \qquad i=1,\dots,m'.$$1, and the corresponding probability becomes

$$\alpha_i = \frac{\langle \mathbf{c}', \mathbf{b}_i^*\rangle}{\|\mathbf{b}_i^*\|^2}, \qquad i=1,\dots,m'.$$2

This is the clearest quantitative justification offered for CR-BNP: the continuous estimate identifies coordinates for which hard nearest-plane rounding is fragile, and a local ternary search can compensate for that fragility (Jiang, 22 Jun 2026).

The same paper also introduces the statistically bounded early-stopping threshold

$$\alpha_i = \frac{\langle \mathbf{c}', \mathbf{b}_i^*\rangle}{\|\mathbf{b}_i^*\|^2}, \qquad i=1,\dots,m'.$$3

which acts as a one-sided certificate and also as a Decision-LWE distinguisher. In the LWE$$\alpha_i = \frac{\langle \mathbf{c}', \mathbf{b}_i^*\rangle}{\|\mathbf{b}_i^*\|^2}, \qquad i=1,\dots,m'.$$4 experiments, single-pass classical Babai produces residuals around $$\alpha_i = \frac{\langle \mathbf{c}', \mathbf{b}_i^*\rangle}{\|\mathbf{b}_i^*\|^2}, \qquad i=1,\dots,m'.$$5 to $$\alpha_i = \frac{\langle \mathbf{c}', \mathbf{b}_i^*\rangle}{\|\mathbf{b}_i^*\|^2}, \qquad i=1,\dots,m'.$$6, well above $$\alpha_i = \frac{\langle \mathbf{c}', \mathbf{b}_i^*\rangle}{\|\mathbf{b}_i^*\|^2}, \qquad i=1,\dots,m'.$$7. Under baseline BKZ-20 the full pipeline recovers the secret on $$\alpha_i = \frac{\langle \mathbf{c}', \mathbf{b}_i^*\rangle}{\|\mathbf{b}_i^*\|^2}, \qquad i=1,\dots,m'.$$8 seeds; under progressive BKZ-20$$\alpha_i = \frac{\langle \mathbf{c}', \mathbf{b}_i^*\rangle}{\|\mathbf{b}_i^*\|^2}, \qquad i=1,\dots,m'.$$930, on $$\mathbf{U}_{\mu,\text{exp}} \mathbf{y}_c = \boldsymbol{\alpha}_{\text{exp}}$$0 seeds. For successful seeds, the reported energy trajectory is

$$\mathbf{U}_{\mu,\text{exp}} \mathbf{y}_c = \boldsymbol{\alpha}_{\text{exp}}$$1

These figures show the intended use case: plain Babai may fail badly, whereas CR-BNP-guided local exploration may still produce a correct solution after correction (Jiang, 22 Jun 2026).

The limitations are stated with comparable clarity. The pipeline “does not solve the LWE instance exactly, it solves the $$\mathbf{U}_{\mu,\text{exp}} \mathbf{y}_c = \boldsymbol{\alpha}_{\text{exp}}$$2-dimensional exploration QUBO (near-)exactly.” The paper also lists several inexactness sources: omitted true offsets due to the $$\mathbf{U}_{\mu,\text{exp}} \mathbf{y}_c = \boldsymbol{\alpha}_{\text{exp}}$$3-bit window, continuous/integer mismatch, hardware precision limits, and frozen-subspace Babai mis-rounding. CR-BNP is therefore not presented as an exact decoder with a worst-case approximation guarantee. Its analysis is partial, most meaningful when the basis is BKZ-reduced and the instance lies in a bounded-distance-decoding-like noise regime. This distinguishes CR-BNP from exact nearest-lattice decoding and from claims of universal improvement over hard Babai.

6. Broader context and related interpretations

Although the term CR-BNP is introduced in the LWE/CIM work, several earlier and later papers illuminate its mathematical setting. The distributed communication paper on transforming a nearest-plane partition into the Voronoi partition studies how a coarse rectangular Babai partition can be refined progressively until exact Voronoi decoding is achieved, and proves that with unlimited rounds the average cost of zero-error refinement is finite (Vaishampayan et al., 2017). That work is not framed as continuous relaxation, but it offers a precise partition-refinement model of “Babai correction.” This suggests a close conceptual analogy: CR-BNP replaces early hard commitment by a continuous center-and-ambiguity representation, whereas the distributed protocol replaces it by progressively finer communication-driven partitioning.

The earlier communication-cost analyses of Babai decoding in two and higher dimensions make two structural facts explicit. First, the nearest-integer operation is the discrete source of non-smoothness in triangular nearest-plane decoding. Second, the Babai partition is basis dependent, while the Voronoi partition is not (Bollauf et al., 2017, Bollauf et al., 2018). This matters for CR-BNP because relaxing the rounding map alone does not remove basis dependence. A plausible implication is that CR-BNP should be analyzed jointly with basis reduction rather than as an isolated smoothing device.

The 2025 GPTQ geometry paper adds a different perspective by proving that back-to-front GPTQ is mathematically identical to Babai’s nearest-plane algorithm on a Hessian-induced lattice and by emphasizing that the residual propagation step is linear while “the only discrete step is the rounding” (Chen et al., 24 Jul 2025). That paper does not define CR-BNP either, but it makes the natural relaxation point explicit in a high-dimensional, optimization-oriented setting. In that sense, it provides a complementary rationale for CR-BNP: if nearest-plane geometry is already the right coordinate system for certain quantization objectives, then continuous relaxations should preserve the triangular residual flow and modify only the scalar rounding map.

A common misconception is therefore to treat CR-BNP as synonymous with any soft Babai decoder. The current literature does not support that identification. The distributed papers do not use the term; the GPTQ paper treats continuous relaxations as extrapolations rather than established methods; and the CR-BNP definition in the LWE/CIM work is specific to the exploration subspace of a reduced $$\mathbf{U}_{\mu,\text{exp}} \mathbf{y}_c = \boldsymbol{\alpha}_{\text{exp}}$$4-ary lattice CVP instance. Another misconception is to regard CR-BNP as a stand-alone exact lattice algorithm. The available formulation is narrower: a structured continuous nearest-plane relaxation whose principal function is to locate a discrete center, quantify fractional ambiguity, and drive a local mixed-radix search under hardware constraints.

Taken together, the literature supports a precise characterization. CR-BNP preserves Babai’s ordered triangular geometry, suspends its immediate integrality constraint over a selected subspace, and exports the resulting fractional information to a downstream discrete procedure. Its antecedents lie in Babai/Voronoi mismatch analysis, basis-dependent nearest-plane geometry, and the recognition that nearest-plane algorithms are linear except for rounding. Its explicit current realization is a pre-encoding geometric relaxation for penalty-free CVP-to-QUBO compression on Coherent Ising Machine hardware (Jiang, 22 Jun 2026).