ClassEval-Obf Benchmark

• The paper introduces ClassEval-Obf, an obfuscation-enhanced benchmark designed to reveal LLMs' overreliance on human-interpretable naming cues.
• It employs deterministic, semantics-preserving obfuscations, such as α-renaming and ambiguous identifiers, to isolate structural comprehension from lexical memorization.
• Empirical results show significant declines in summarization and execution metrics, highlighting the need for more robust, structure-aware code models.

ClassEval-Obf is an obfuscation-enhanced variant of the ClassEval benchmark designed to assess the code understanding and generalization ability of LLMs by systematically suppressing human-interpretable naming cues in code while preserving program structure and semantics. The benchmark aims to disentangle the semantic reasoning capabilities of LLMs from their reliance on lexical or memorized naming patterns, providing a more stringent and reliable testbed for evaluating models on class-level code generation, summarization, and execution tasks, particularly in scenarios where memorization shortcuts are prevalent.

1. Motivation and Objectives

ClassEval-Obf was developed in response to the observed deficiencies in traditional code generation benchmarks, where LLMs can achieve high scores by leveraging superficial naming cues present in code (identifier names, docstrings, and human-intended labels). Many established datasets—including the original ClassEval—retain human-written, semantically informative names, which enable LLMs to exploit memorized associations rather than demonstrating true comprehension of program structure and behavior. ClassEval-Obf systematically removes, distorts, or disguises such cues, thereby forcing models to rely on formal code semantics—syntax, control flow, and execution logic—to perform tasks.

The benchmark’s objectives are to:

• Eliminate the confounding effect of "identifier leakage" on LLM performance.
• Assess genuine semantic reasoning in code generation and summarization by decoupling lexical cues from structural information.
• Provide robust pre-/post-obfuscation metrics (e.g., delta in Pass@1, Pass@3, summarization accuracy) to quantify models’ reliance on names.
• Encourage architectures and training methods that promote structure-aware code understanding.

2. Semantics-Preserving Obfuscation Methodology

ClassEval-Obf employs a suite of deterministic, semantics-preserving obfuscations on class-level Python code. These transformations are designed to remove exploitable natural-language signals while guaranteeing that execution-relevant logic remains untouched. The primary strategies include:

Obfuscation Strategy	Description	Effect on Code
α-renaming	All identifiers renamed to generic, role-specific placeholders (e.g., "class1", "var2")	Semantic structure invariant, lexical cues lost
Ambiguous Identifiers	Renames to visually ambiguous strings (e.g., "lIIIlI")	Confounds recognition/memorable patterns
Cross-domain Substitution	Identifiers replaced with unrelated domain terms (e.g., medical jargon)	Breaks contextual/semantic associations
Misleading Semantics	Names assigned that contradict the actual function or behavior	Natural-language signals corrupted

All obfuscations strictly preserve syntax, control/data flow, and output behaviors. α-renaming, in particular, is widely referenced in the literature as a way to test against model overfitting to names.

These obfuscations act as a precise probe into how code models internalize meaning: a model grounded in syntax and logic should be minimally impacted, whereas a model over-reliant on naming patterns will see substantial drops in task accuracy.

3. Benchmark Construction and Evaluation Protocol

ClassEval-Obf extends the ClassEval suite, which itself is composed of 100 hand-crafted Python class-level tasks with comprehensive test suites for both method-level and class-level correctness (coverage >98%). Obfuscations are applied to both inputs (prompt context, method/class skeletons) and canonical solution/test code.

Evaluation encompasses:

• Generation tasks: LLMs synthesize Python classes given obfuscated specifications.
• Summarization tasks: Models are asked to recover natural-language descriptions of program intent from obfuscated code.
• Execution prediction: Pass@1 and Pass@3 metrics are computed—representing the ratio of generated solutions passing all test cases on the first or within the first three tries—across both naming-retentive ("original") and obfuscated code.

The delta metric (Δ), defined as

$$Δ = \text{Metric}_{\text{original}} - \text{Metric}_{\text{obfuscated}}$$

exposes the impact of naming cues. Smaller Δ indicates robustness to lexical signal removal; larger Δ indicates overreliance on names.

A memorization stress test is employed to detect if models simply regurgitate outputs from training data. When run on obfuscated code, these matches dissipate, highlighting the suppression of memorization shortcuts.

4. Empirical Findings: Performance and Analysis

ClassEval-Obf demonstrates several key effects of naming obfuscation:

• Summarization Tasks: Accuracy of LLMs, including leading models (e.g., GPT-4o), declines sharply on intent-level summaries following obfuscation (e.g., from 87.3 to 58.7 in summary accuracy), indicating that such tasks are highly contingent on identifier signals.
• Execution Tasks: Despite unchanged program logic, execution success (Pass@1, Pass@3) decreases across all models with any obfuscation strategy. This suggests a nontrivial dependency on naming patterns, even in tasks theoretically bound only to code structure.
• Memorization Shortcuts: Under the original benchmark, LLMs sometimes emit outputs directly from training data by recognizing identifier keys. In the obfuscated setting, “old output” matches vanish. This effect demonstrates that naming obfuscation effectively neutralizes one avenue of memorization.
• Delta Metrics: Quantitative tables in the paper confirm negative Δ values for all models and tasks, directly quantifying the robustness—or vulnerability—of different architectures to semantic obfuscation.

Notably, these effects are attenuated on code from domains where identifiers are generally short or generic (e.g., competitive programming), implying that the benchmark’s sensitivity is task- and domain-dependent.

5. Implications for LLM Evaluation and Model Development

ClassEval-Obf reveals that high performance on conventional code benchmarks may frequently overstate true semantic competence, as models exploit human-written, information-rich names. Incorporating semantics-preserving obfuscation into evaluation exposes these overestimations and provides a more demanding, realistic measurement of model capability.

Key implications include:

• The need to report performance using both original and obfuscated conditions, with Δ as a key metric.
• A more reliable basis for gauging actual code understanding and for tracking progress in structural generalization.
• A call to benchmark designers: integrate obfuscation-based testbeds to diminish identifier leakage and increase task difficulty.
• Directions for model development: architectures must be trained and evaluated with less dependence on lexical cues, possibly using techniques that reward structural alignment and penalize overfitting to names.

A plausible implication is that future code LLMs, when robustly evaluated under the ClassEval-Obf paradigm, will be better equipped for real-world programming scenarios where naming patterns are less consistent, more obfuscated, or intentionally adversarial.

6. Mathematical Notation and Reporting Standards

The benchmark references standard mathematical notation in its reporting and method description:

• The use of α-renaming, denoted with the Greek letter α, to refer to identifier-renaming obfuscation.
• Common operators such as $\argmax$ and $\argmin$ appear for expressing optimization routines.
• The Δ (delta) metric is used to report performance change following obfuscation:

$$Δ = \text{Metric}_{\text{original}} - \text{Metric}_{\text{obfuscated}}$$

• Pass@k (e.g., Pass@1, Pass@3) is used as the primary execution metric, representing the fraction of generated solutions that pass the full test suite in k attempts.

Metrics are reported both pre- and post-obfuscation to foreground the direct influence of naming removal.

7. Significance and Future Research Directions

ClassEval-Obf has established a new standard for evaluating the semantic reasoning capabilities of code LLMs, highlighting the importance of disentangling lexical memorization from genuine program understanding. Its methodology provides a direct avenue to stress-test model robustness in scenarios where identifier cues are noisy, misleading, or absent.

The benchmark’s adoption suggests a move toward structural, rather than lexical, generalization as the primary axis of progress in code LLMs. Broader implications include:

• Recommendations to extend obfuscation-based evaluations to other languages, paradigms, and code understanding tasks.
• The potential integration of obfuscation strategies during model pretraining to further encourage semantic abstraction.
• Use as a diagnostic tool to isolate sources of performance gain in future model releases.

Researchers are encouraged to continue developing benchmarks and metrics that mitigate identifier leakage and reward true semantic progress in automated code understanding and synthesis (Le et al., 3 Oct 2025).