Improving ML Attacks on LWE with Data Repetition and Stepwise Regression

Abstract: The Learning with Errors (LWE) problem is a hard math problem in lattice-based cryptography. In the simplest case of binary secrets, it is the subset sum problem, with error. Effective ML attacks on LWE were demonstrated in the case of binary, ternary, and small secrets, succeeding on fairly sparse secrets. The ML attacks recover secrets with up to 3 active bits in the "cruel region" (Nolte et al., 2024) on samples pre-processed with BKZ. We show that using larger training sets and repeated examples enables recovery of denser secrets. Empirically, we observe a power-law relationship between model-based attempts to recover the secrets, dataset size, and repeated examples. We introduce a stepwise regression technique to recover the "cool bits" of the secret.

• The paper demonstrates that high repetition in training data, combined with angular and region-specific embeddings, enables ML models to overcome the three cruel bit barrier.
• It introduces stepwise regression for bit recovery that significantly improves secret reconstruction rates and outperforms previous methods like VERDE.
• Empirical results highlight a power-law scaling in data efficiency and effective use of synthetic data, offering profound security implications for LWE-based post-quantum cryptosystems.

Improving ML Attacks on LWE with Data Repetition and Stepwise Regression

Introduction and Context

The paper "Improving ML Attacks on LWE with Data Repetition and Stepwise Regression" (2604.03903) presents a comprehensive empirical and methodological advance in the machine learning-driven cryptanalysis of the Learning With Errors (LWE) problem. LWE constitutes the hardness assumption for much of lattice-based post-quantum cryptography (PQC). Prior ML attacks such as SALSA, PICANTE, and VERDE demonstrated success on LWE with binary, ternary, and small/sparse secrets, but consistently failed to recover secrets with more than three non-zero ("cruel region") bits after lattice (BKZ) reduction.

This work systematically investigates the underlying reasons for this limitation and provides novel techniques for breaking through the "3 cruel bits" barrier. It uses large-scale experiments (up to 400 million synthetic or BKZ-reduced samples, requiring tens of millions of CPU hours) and introduces a suite of methodological enhancements: large-scale data and repeated examples for model training, and a stepwise regression (including a dual variant) for cool region bit recovery.

Background: LWE Attack Landscape

The foundational LWE problem asks to recover a secret vector $\mathbf{s}$ from noisy modular inner products, where $\mathbf{s}$ can be binary, ternary, or possess small integer entries with variable Hamming weight. Security for most practical parameter choices is under continuous scrutiny.

ML attacks on LWE, beginning with SALSA [salsa], relied on transformers trained on large sets of LWE pairs sharing the same secret. These works achieved limited secret recovery, bounded by the model’s ability to learn modular arithmetic and by the signal amplification and rate amplification produced by lattice reduction (BKZ). Prior empirical evidence and theoretical analyses (e.g., the "Cool and Cruel" decomposition [coolcruel]) established the inability to recover more than 3 cruel bits, where cruel bits are unreduced (high variance, hard) positions post-BKZ.

Methodologies and Experimental Design

Training with Large and Repeated Data

The work demonstrates—via exhaustive experimentation—that increasing not only the total number of samples but, crucially, the level of repetition (i.e., presenting the same reduced LWE instance with noise multiple times during training) materially enhances the transformer’s capacity for cruel bit recovery. Contrary to previous beliefs, simple data scaling alone is insufficient: only with high repetition do secrets with more than 3 cruel bits become accessible.

The experiments examine recovery as a function of dimension ($n=256, 512$), modulus size ($\log_2 q=12, 20, 28, 41$), Hamming weight, and training repetition, isolating the effects across a spectrum of parameter settings.

Angular Embedding and Specialized Model Architecture

Input vectors undergo angular encoding, placing each modular value on the unit circle in $\mathbb{R}^2$ and then projecting to higher-dimensional spaces. Notably, the architecture introduces a specific "cool and cruel" region-positional embedding, providing explicit inductive bias regarding the heterogeneous difficulty across secret coordinates. This improves cruel bit extraction, especially under high repetition regimes.

Stepwise Regression for Cool Bit Recovery

After recovering cruel bits, the traditional approach estimates cool bits by global linear regression, which degrades in the presence of modular wraparound and when noise is amplified by lattice reduction. This work replaces this phase by iterative stepwise regression, which identifies zero (or, dually, one) bits one at a time, removing their contribution and repeating. This one-by-one elimination (and switching between primal and dual modes depending on expected remaining sparsity) exploits the known independence and sparsity structure of the secret. It provides superior accuracy over standard regression, as evidenced by extensive benchmarks in both low and high reduction regimes.

Key Empirical Results

Improvement Over State-of-the-Art

The proposed approach sets new high-water marks for the maximum Hamming weight and number of cruel bits recovered by ML attacks. For example, in settings $(n=512, \log_2 q=41)$, prior state-of-the-art (VERDE) recovers up to $h=63$ binary secrets; this method achieves $h=75$, with high expected recovery rates (91% vs. 15%) for all secrets at that Hamming weight. In harder regimes with less modulus reduction, the improvement is equally significant.

Scaling Laws

A central empirical finding is the existence of a power-law scaling for the number of model-based brute-force attempts $A$ required for secret recovery, in terms of the total number $D$ of (potentially repeated) data points:

$\mathbf{s}$0

where the exponent $\mathbf{s}$1 increases with the repetition factor $\mathbf{s}$2. The presence of repetition not only reduces the need for distinct (costly) examples, but fundamentally alters the data-efficiency scaling of the attack. Models with more parameters do not significantly reduce the number of required attempts compared to data-related factors.

Data Synthesis and Ablation

Synthetic data—constructed to match the statistical properties of BKZ-reduced instances—was shown to be indistinguishable, from the attack’s perspective, from true reduced data. This finding enables cost-effective pretraining and the study of scaling regimes not reachable via lattice reduction alone.

The paper also presents ablation studies showing the importance of the cool/cruel embedding and examines strategies for mitigating model prediction collapse in the angular output space by regularizing the radius.

Practical and Theoretical Implications

Security Implications for PQC: The results demonstrate that the practical security of LWE-based schemes with small, sparse secrets is materially lower than previously estimated by the cryptanalytic community, especially when adversaries are equipped with massive, repeated datasets and can use model-guided search over cruel bits. This is compounded by the demonstration that cool bits can be systematically extracted from a sparse support by modular stepwise regression, undermining the constellation of defenses associated with BKZ reduction.

Data Budget and Attack Feasibility: While the attack requires large reduced datasets (and, consequently, significant computational precomputation for BKZ), synthetic augmentation and aggressive data reuse can bring the cost into the frontier of feasibility for certain cryptanalytic scenarios. The insight that data repetition outperforms mere data size increases (in terms of model search efficacy) will influence future ML-based cryptanalysis and design of cryptanalytic datasets.

Methodological Innovations: Stepwise regression, although considered suboptimal in classical statistics, is precisely matched to the LWE setting post-reduction. The finding that modular arithmetic learning by transformers is deeply data-limited, but not fundamentally model-size limited, restructures approaches to designing ML architectures for cryptanalytic tasks.

Future Outlook

Subsequent research should focus on:

• Extending these techniques to more generalized LWE variants (e.g., module-LWE, ring-LWE) as used in modern PQC standards.
• Examination of non-sparse (dense) secret distributions and their susceptibility to attacks with synthetic or repeated data.
• Exploration of curriculum and synthetic pre-training to minimize costly BKZ reduction, leveraging the equivalence between synthetic and reduced statistics.
• Further analysis of the repetition-induced phase transitions in modular arithmetic learning, possibly generalizable to other hard cryptographic problems.

Conclusion

This work closes a key gap in the empirical cryptanalysis of LWE with small secrets via ML, demonstrating that the prior "cruel bit" barrier is surmountable by leveraging large-scale, repeated data and specialized post-model analysis (stepwise regression), and providing rigorous scaling observations for attack success probability. These results necessitate a re-evaluation of security margins for LWE-based systems with sparse secrets and offer methodological innovations likely to propagate to both cryptanalysis and broader domains requiring modular arithmetic learning.