Memorization Diagnostics for Code LLMs Should be Scale-Aware

Abstract: The extent to which LLMs for code rely on memorization over genuine understanding remains highly debated. While current literature frequently reports widespread memorization, evaluating the underlying probing techniques across dense architectures reveals a severe breakdown in their utility at scale. Traditional encoder-style probes using perturbations such as synonym fuzzing or dead-code insertion struggle to expose memorization in scaled models, even on known-contaminated benchmarks, and decoder-style probes that rely on log probabilities show similar performance degradation. The specific mode of failure for these probes, particularly why such techniques disrupt smaller models but fail to impact larger ones, motivates us to untangle representation load from memorization rather than treating them as a single phenomenon. By applying invertible mathematical transforms to numeric problems, we isolate these two factors and reveal that scaled encoders successfully absorb substantial representation load while still converging on the correct family of solutions. In practical software engineering, this ability to adapt to varying surface forms is what truly matters for usability and generalizability in LLM and agentic applications. Whether a specific solution was seen during training becomes a much less pressing question because although memorization inflates scores on contaminated benchmarks, factoring out representation load makes it debatable how much we should truly care if a functional answer was originally memorized. Future evaluations must therefore be built around separating these phenomena rather than relying on methodologies that quietly entangle them.

• The paper demonstrates that encoder-side perturbation and decoder-side likelihood probes lose discriminative power as models scale, with decoder AUC falling from 100% on Pythia to 75% on Llama-3.1-405B.
• The paper introduces I/O isomorphism, a metamorphic testing protocol using bijective input-output transformations to preserve task correctness while exposing representational-load failures separate from memorization.
• The paper finds that frontier models suffer 14–30 point pass@1 declines under transformed interfaces while retaining similar opcode distributions, indicating decoder-side compliance failures rather than loss of the underlying algorithm.

This paper argues that the standard diagnostics used to detect memorization in code LLMs lose their discriminative power as model scale grows, and that a substantial share of what such diagnostics report is better explained by representational load—the burden of carrying an algorithm through an unfamiliar interface—rather than by recall of training data. The authors, from the University of Luxembourg, test both encoder-side perturbation probes and decoder-side likelihood probes across dense model families spanning 0.4B to 405B parameters, and propose an I/O-isomorphism protocol grounded in metamorphic testing to separate the two phenomena (2608.12771).

Motivation and framing

The prevailing methodology for probing memorization in code generation applies semantics-preserving perturbations to prompts—variable renaming, docstring paraphrase, dead-code insertion—and interprets the resulting pass@$k$ drop as a memorization signal. A complementary line of work, exemplified by CoDeC, measures how in-context examples shift token log-likelihoods to distinguish seen from unseen training data. The paper's central objection is that these probes are validated on small model families and rarely re-examined at frontier scale, where encoder capacity may absorb perturbations entirely and probability mass spreads across a vastly larger space of plausible continuations. On benchmarks with known contamination, the authors find that both probe families fade as model capacity increases—a saturation they treat as the hypothesis to be disproved before proposing an alternative.

The paper introduces representational load as a construct distinct from memorization: the demand imposed on a model when a task is stated in a form distant from training-like surface conventions, whether on the encoder side (parsing auxiliary constraints) or the decoder side (honoring those constraints token by token during generation). The practical motivation is that real software engineering routinely presents models with domain-specific encodings, legacy formats, and project-local conventions that no benchmark captures.

Methodology

The study evaluates 11 dense models for encoder-side and isomorphism experiments (deepseek-coder-6.7b through Llama-3.1-70B, plus GPT-4o, GPT-4o-mini, and two Gemini Flash variants), and a separate 7-model axis for decoder-side scoring (Pythia 410M–12B, Llama-3.1-70B, Nemotron-4-340B, Llama-3.1-405B, davinci-002). The authors deliberately restrict to dense architectures, since the relationship between nominal parameter count and effective capacity is ambiguous for mixture-of-experts models; how decoder diagnostics should be adapted for MoE architectures is left open.

Three benchmarks span a difficulty and contamination axis: MBPP (August 2021, plausibly contaminated for every model), EffiBench (November 2024), and BigOBench (March 2025), the latter two postdating the training cutoffs of all models except Gemini-2.5-Flash. This design creates a natural experiment: if memorization drove performance under perturbation, isomorphic drops should concentrate on MBPP.

The core methodological contribution is the I/O isomorphism protocol. A bijective value transform $T_\theta(t) = at + b$ is applied to every integer in a benchmark's test cases, and an explicit encode/decode contract is appended to the prompt. Because $T_\theta$ is invertible, correctness is preserved exactly:

$$f(x)=y \iff T_\theta\bigl(f\bigl(T_\theta^{-1}(x')\bigr)\bigr)=y'$$

so any accuracy change is attributable to representational handling, not to the task itself. The model must decode every input, run the algorithm, and re-encode every output; memorized retrieval cannot pass this layer. Ablations over base-conversion and cubic bijection families test generality beyond affine transforms.

To distinguish compliance failures from competence failures, the authors compile generated programs to CPython bytecode and compute Shannon entropy over opcode-frequency distributions, then measure per-problem Jensen-Shannon divergence (JSD) between pooled opcode distributions across conditions. Near-zero JSD indicates the same algorithmic families are attempted; a spike indicates the strategy itself is abandoned.

Encoder-side probes saturate at scale

Synonym fuzzing at 20% and 40% substitution rates produces drops under 8 pass@1 points for frontier models across all three benchmarks. Gemini-2.5-Flash loses at most 2.1 points and occasionally gains on unseen benchmarks. Mid-tier models show thresholded, non-linear degradation—Llama-3.1-70B drops 7.9 points on MBPP under Syn-20 but 27.3 under Syn-40—while models below 22B collapse (Codestral-22B loses 36.9 points on MBPP under Syn-40).

A key negative result concerns the contamination signature. If synonym fuzzing disrupted memorized retrieval, effects should concentrate on MBPP, the only benchmark plausibly contaminated for all models. The data do not support this: relative losses on the unseen benchmarks are often comparable or larger, though this is confounded by task difficulty, since small models have low baselines on harder tasks and small absolute drops become large relative declines. The diagnostic therefore conflates encoder fragility with memorization and cannot cleanly separate the two. This directly undercuts perturbation-based memorization claims validated only on smaller models.

Decoder-side probes lose separation at scale

The CoDeC reproduction matches the original behavior on Pythia: seen-dataset scores of 71–75%, unseen scores of 17–24%, and AUC of 100% at all three checkpoints. At scale, both the scores and the seen–unseen gap shrink dramatically. The gap narrows to 12.5 points for davinci-002, 3.5 for Nemotron-4-340B, and 5.5 for Llama-3.1-405B; AUC falls from 100% to 75%. Llama-3.1-70B even inverts the expected ordering (AUC of 25%), though the authors attribute this to the small number of datasets rather than treating it as substantive. Seen-dataset scores fall from 71–75% at the Pythia scale to 7.5% at 405B.

The proposed explanation is twofold: probability mass spreads across a richer representation space, diluting the concentrated retrieval effects CoDeC relies on, and stronger in-context learning lets larger models use added context constructively for both seen and unseen targets, erasing the asymmetry the probe exploits. One caveat the authors concede: the "seen" labels are probable rather than certain, since they rest on corpus-overlap designations, though this asymmetry is shared across models and does not affect the trend.

Representational load degrades performance without erasing the algorithm

The full isomorphism contract drops frontier-model pass@1 by 14–30 absolute points on a provably unchanged task. Decomposing the contract isolates the bottleneck: for Gemini-2.0-Flash on MBPP, transforming inputs alone (Iso Enc) costs 9.7 points, whereas transforming outputs alone (Iso Dec) costs 21.8 points—close to the full contract's 22.7-point penalty. This pattern holds across datasets and scaled models, locating the dominant cost in decoder-side contract compliance during autoregressive generation rather than input parsing.

The opcode analysis then separates compliance from competence. For frontier models, per-problem JSD between Original and Iso generations stays near zero across all benchmarks: on exactly the problems where pass@1 has dropped by 14–30 points, the models select the same loops, comparisons, and control flow, and fail mainly at serializing the encoded interface. A running example on an MBPP Woodall-number task makes the failure mode concrete: across five generations, the algorithmic loop is identical in passing and failing runs, and failures arise only when the model routes boolean returns through the integer contract, producing comparisons that always evaluate to True. The paper terms this a narrow channel—the route from specification to code stays open under load but tightens. Smaller models, by contrast, show JSD spikes on harder benchmarks, abandoning whole solution families—a channel that closes rather than narrows, and a competence failure rather than a compliance failure.

Two ablations strengthen these claims. Alternative bijection families (base conversion, cubic polynomials) produce drops of comparable magnitude, confirming the brittleness reflects general value-space sensitivity. A dead-code insertion ablation matching the Iso contract's token count bounds the prompt-length confound at roughly 7%: dead code costs 1–7% relative while Iso degrades the same models by 24–51%, an order-of-magnitude gap.

Discussion

The paper offers an architectural reading of the decoder-side bottleneck: prompt understanding is parallel and context-rich, while decoding is sequential and must preserve implicit arithmetic state across many steps while maintaining syntax—a form of constrained multi-step generation that next-token training does not explicitly support. The authors are careful to note that their experiments speak to the size of the effect, not its mechanism directly.

On the memorization question, the position is deliberately moderate. The paper does not claim memorization is absent from scaled code LLMs; it claims the surface and likelihood probes inherited from prior work stop discriminating at scale, so a bare pass@$k$ drop or likelihood shift is weak evidence of memorization on its own. The authors also observe that solution multiplicity—multi-modal opcode distributions on unseen benchmarks for scaled models—complicates decoder-side memorization methodologies, since stochastic switching among several correct solutions confounds likelihood-based attribution.

Limitations and open questions

The paper is explicit about scope. The metamorphic guarantee holds only over bijective transforms on numeric I/O, so findings characterize numeric programming tasks specifically; constructing closed bijections on string formats, data layouts, or API conventions remains an open problem. Opcode entropy is a proxy, not a direct measurement—two behaviorally distinct programs can share an opcode histogram, and JSD is invariant to control-flow reorganization. The model panel excludes MoE architectures, covers Python only, and addresses function-level synthesis rather than repository-scale or agentic coding. Greedy decoding at temperature 0 collapses the pass@5/pass@1 gap and may understate headroom under sampling; the exact contract wording is a fixed free variable; and closed-source endpoints offer no determinism or versioning guarantees. Whether the narrow-channel behavior generalizes beyond the three transform families and three benchmarks is untested.

Conclusion

This paper demonstrates that two dominant memorization diagnostics for code LLMs—encoder-side synonym fuzzing and decoder-side likelihood probing—saturate as model scale grows, and that a metamorphic I/O-isomorphism protocol can separate representational load from memorization with a formal correctness guarantee. Its central empirical finding is that scaled models retain their algorithmic core under substantial representational stress, failing at interface serialization rather than at the task itself, while smaller models exhibit genuine competence collapse. The practical recommendation is to treat pass@$k$ drops as a starting point, factor out representational load before attributing degradation to recall, and report solution-space stability alongside correctness. The open questions the paper leaves—metamorphic oracles for non-numeric loads, MoE-specific diagnostics, and the architectural basis of the decoder bottleneck—define a concrete agenda for scale-aware evaluation of code LLMs.