Detecting Functional Memorization in Code Language Models

Abstract: LLMs are increasingly used to generate code at scale. Meanwhile, prior work has investigated whether training data may be recoverable from model outputs, by auditing the textual overlap between training examples and model generations. Code, however, can be functionally equivalent while textually dissimilar. In this work, we study functional memorization: extraction of functional logic beyond what verbatim metrics detect. We construct a counterfactual setup for Olmo-3-32B, comparing a midtrained model (exposed to target code) against a pretrained reference (not exposed). We prompt both models with Python function signatures and measure both textual and functional similarity (i.e., LLM-as-a-judge, execution-based). Our results show clear evidence of functional memorization, highlighting the need for auditing metrics that go beyond textual overlap.

• The paper demonstrates that midtraining on the CraneCode corpus increases functional similarities between generated code and training samples.
• It introduces a counterfactual experiment contrasting a midtrained target model with a pretrained reference to rigorously detect proprietary logic reproduction.
• The study shows that conventional text-based metrics miss deeper functional memorization, highlighting the need for advanced auditing protocols.

Detecting Functional Memorization in Code LLMs

Introduction

The extraction of training data from LLMs trained on code has significant implications for both intellectual property (IP) leakage and privacy. Traditional memorization studies focus mainly on verbatim or near-verbatim reproduction, which fail to capture the unique risk in coding applications: the surface form of code can be altered considerably while retaining functional equivalence. "Detecting Functional Memorization in Code LLMs" (2606.12764) presents a formal and empirical analysis of functional memorization, whereby models reproduce the underlying operational logic from training data without near-verbatim surface similarity.

Methodology

The core methodology is a counterfactual experiment leveraging Olmo-3-32B checkpoints, exploiting the model's training transparency. The authors define a target (midtrained) model, exposed to the CraneCode corpus, and a reference (pretrained) model, which shares architecture and all earlier data but is not exposed to CraneCode. This enables a sharp test for memorization: a sample is considered counterfactually memorized if only the target model produces code equivalent to the training data given a prompt, while the reference does not.

Unlike prior literature, which predominantly employs textual overlap metrics like BLEU, edit distance, or LCS, this work evaluates functional similarity via a wide spectrum of approaches:

1. Structural code metrics: CodeBLEU (various configurations) and tree-based edit similarity (TSED) to capture syntactic and data-flow similarity.
2. LLM-as-a-judge protocols: Prompts to large models (variants of LLaMA-3.1-70B-Instruct) to assign scalar equivalence or semantic similarity ratings.
3. Execution-based verification: HyClone paradigm, which attempts input generation and cross-comparison of outputs, directly approximating functional equivalence.

The benchmark corpus is curated by filtering CraneCode for Python functions between 10 and 50 lines, classified for meaningful functional logic by an LLM with a severity metric, resulting in 7,422 analyzable functions.

Main Results

The primary finding is that midtraining (i.e., exposure to specific code) systematically increases similarity between ground-truth training samples and the target model's generations across all evaluation dimensions. Explicitly, all similarity-based deltas between the target and reference model are positive (see Figure 1 and Figure 2), affirming the presence of counterfactual memorization on both textual and several levels of functional axes.

Figure 3: BLEU score distributions indicate higher similarity for the target model (midtrained) relative to reference.

Figure 1: Pearson correlation matrix reveals that structural and functional metrics capture complementary signals beyond textual overlap.

Notably, near-verbatim memorization as detected by BLEU is relatively rare (0.11% exact, 0.58% at BLEU ≥ 0.75), but functional memorization occurs with significantly higher prevalence. Depending on the metric, 0.3%–4.2% of analyzed samples are flagged as functionally memorized under conservative thresholds, with the most reliable LLM-judge configuration surfacing 3.9% (where BLEU < 0.75, but LLM-judge ≥ 0.75 only for the target model). Execution-based HyClone confirms 0.28% as functionally identical, despite low BLEU.

Figure 2: The delta in LLM-judge (Ours) versus BLEU exposes cases of functional memorization beneath textual overlap thresholds.

The memorization effect is robust across function length, input context, and severity. The effect is most pronounced with minimal context and for functions with low-to-moderate severity, but remains statistically significant in all slices (see Figures 8–15).

Figure 4: Distribution of severity labels among analyzed functions, confirming a wide range of functional logic sensitivity.

Qualitative Analysis

The qualitative section demonstrates the phenomenon with instructive examples. In a canonical case, the target model internalizes city-specific commission rules, expressing them with entirely restructured code absent from the reference model's outputs. BLEU and edit similarity are low, while structural, LLM-as-a-judge, and execution-based metrics robustly detect equivalence.

These examples (detailed in the appendices) underscore the inadequacy of text-based similarity for capturing meaningful IP leakage. LLMs may generalize proprietary logic into new surface forms, which would elude standard extraction audits.

Comparison of Functional Similarity Metrics

The work provides a thorough metric-based breakdown:

• Textual metrics (BLEU, edit sim, LCS): Deterministic, full coverage, but blind to nontrivial refactorings.
• Structural metrics (CodeBLEU, TSED): Capture shallow logic similarity, robust to variable renaming, but noisy for short/simple functions, partial coverage due to AST parsing failures.
• LLM-as-a-judge: Highest sensitivity to deep functional equivalence, can capture Type IV clones. Not deterministic, sensitive to prompt calibration, and relatively costly.
• Execution-based (HyClone): Gold standard for functional sameness, but coverage (∼15%) severely limited by real-world code entanglement (class context, dependencies, I/O).

Theoretical and Practical Implications

This analysis provides empirical evidence that code LLMs trained on real-world corpora can and do leak functional IP beyond surface copying. For practitioners, post-hoc audits based solely on text metrics are not sufficient to guarantee the non-extraction of proprietary business rules, algorithms, or sensitive configuration from code LLMs. For theorists, functional memorization occupies a continuum between memorization and generalization, challenging the definition of overfitting in the context of models exposed to large, structured, and semantically rich data.

The findings raise questions for data curation, model release (especially open weights), and the governance of code-generating systems—a critical matter as models increasingly incorporate user and proprietary data.

Directions for Future Research

Improvements in coverage and reliability of functional similarity measurement are essential. Increasing robustness in LLM judge protocols, better prompt engineering, expanding execution-based coverage (e.g., robust code sandboxing, intelligent test generation), and more nuanced definitions of memorization are promising axes for development. There is need for theoretical work quantifying the minimal abstraction at which a piece of logic may be considered memorized versus generalized, especially in high-utility deployment domains.

Conclusion

"Detecting Functional Memorization in Code LLMs" establishes strong empirical evidence that code LLMs can reproduce proprietary logic from training data even with low surface similarity, eluding verbatim detection regimes. The results necessitate revision in auditing protocols for code models, with explicit consideration of functional similarity. They invite the community to devise more rigorous, multidimensional approaches for both IP risk management and formal memorization studies moving forward.