Do Machines Struggle Where Humans Do? LLM and Human Comprehension of Obfuscated Code

Abstract: While code obfuscation impairs human code comprehension, it remains unclear if LLMs share these failure modes. Building directly on a recent human study of program comprehension under code obfuscation, we evaluate whether LLMs share the failure modes that obfuscation induces in human programmers. Evaluating several LLMs with five obfuscation tiers using the Block Model, we localize comprehension failures at the atom, block, relational, and macro levels. We find that reasoning-tuned models demonstrate significant alignment with human difficulty patterns across experience levels, whereas instruction and coder-tuned models show near-zero correlation. Chain-of-Thought trace length tracks task difficulty across tasks. Results indicate that performance under control-flow flattening degrades in proportion to state-space complexity, while adversarial identifier renaming disrupts comprehension through the interaction of semantic displacement and identifier-level interference. These findings suggest that reasoning-tuned LLMs approximate human sensitivity to code complexity more effectively than instruction-tuned variants.

• The paper demonstrates that reasoning-tuned LLMs align significantly with human comprehension failures, achieving correlations of 0.30–0.47 across obfuscation levels.
• It employs progressive obfuscation techniques, such as identifier renaming and control-flow flattening, to systematically probe breakdowns in code understanding.
• Chain-of-thought length is used as a marker for reasoning effort, showing longer traces in harder tasks, yet not guaranteeing correct outcomes under adversarial conditions.

Do Machines Struggle Where Humans Do? LLM and Human Comprehension of Obfuscated Code

Introduction and Motivation

The paper "Do Machines Struggle Where Humans Do? LLM and Human Comprehension of Obfuscated Code" (2606.31725) investigates the alignment between human and LLM comprehension processes when faced with progressively obfuscated code. The study is grounded in both software engineering and cognitive science perspectives, leveraging obfuscation as an experimental probe to localize where model and human reasoning breaks down. The objective is to rigorously examine whether LLMs share human failure modes in code comprehension, particularly when deprived of surface-level cues such as meaningful identifiers and canonical control flow.

Theoretical Framework and Research Questions

The analysis is organized using Schulte's Block Model, which decomposes program understanding into atom (token/identifier), block (statement/region), relational (data/control flow), and macro (global algorithm) levels.

Figure 1: The Block Model schema—used to localize comprehension failures at atom, block, relational, and macro levels.

Four central research questions (RQs) structure the empirical work:

1. Task-level Alignment: Do LLMs succeed and fail on the same obfuscated tasks as human programmers?
2. Expertise and Model Stratification: Is the alignment observed stratified by model architecture, size, or training (reasoning vs. coder vs. instruct), analogously to human expertise levels?
3. Process-level Alignment (Effort Scaling): Does CoT (Chain-of-Thought) trace length serve as a monotonic index of reasoning effort and task difficulty for LLMs, as response time does for humans?
4. Failure Mode Alignment (Misleading Identifiers): Do LLMs exhibit confident, high-probability errors under adversarial identifier renaming, paralleling human miscomprehension?

Experimental Design

Datasets and Obfuscation

Two primary datasets are used. The first consists of Python and JavaScript function prediction problems, constructed to match a human participant study (enabling direct human-model comparison), and the second is a model-only set spanning HumanEval-X, CruxEval-X, and LeetCode, with 250 code snippets over five obfuscation tiers (L0: clean; L1: identifier renaming; L1b: adversarial renaming; L2: control-flow flattening; L3: both renaming and flattening). Obfuscations are guaranteed to be semantics-preserving.

Model and Prompt Selection

The study evaluates a broad set of LLMs across coding, instruct, and reasoning-tuned families and multiple parameter scales. Importantly, deliberative reasoning is specifically elicited through prompt variants, distinguishing System 1 (fast/automatic) from System 2 (slow/explicit reasoning) responses.

Figure 2: Prompt suite incorporates controlled manipulations across reasoning depth, cognitive interference, token limits, and scaffolding.

Task-Level Human–Model Alignment

Across increasing obfuscation, both human and LLM performance degrades. The decline is pronounced for novice humans and non-reasoning models, strongly correlated with the structural complexity induced by control-flow flattening and the loss of semantic cues by identifier renaming.

The critical result is that reasoning-tuned LLMs (e.g., DS-R1-Qwen-7B, SmolLM3-3B) display significant alignment with human task-level difficulty patterns: Spearman $\rho = 0.30$–$0.47$ between human and model accuracies across tasks and obfuscation levels. By contrast, coder and instruct-tuned models show negligible correlation to human difficulty, even at comparable or higher model scale.

Figure 3: Accuracy trends across conditions for model types reveal the persistent gap in robustness between reasoning, coder, and instruct-tuned LLMs.

Model architecture is the dominant factor for both raw accuracy and alignment: the best reasoning model is correct on 58–70% of clean Python snippets, outperforming all coder/instruct models; yet its correlation with human failure is not perfect ($\rho < 0.5$), indicating remaining architectural divergence.

Language and Expertise Effects

Python code is more robust to obfuscation in humans than JavaScript, whereas LLMs show weaker, less systematic language effects. Notably, human experts exhibit improved accuracy under adversarial renaming (L1b), peaking in performance as they abstract away from misleading cues—a pattern not replicated by any model family.

Figure 4: Models' mean accuracy (left) exhibits less language-dependent variation than human accuracy (right) across increasing obfuscation.

Reasoning models align with human difficulty gradients stratified by expertise, but no LLM architecture reproduces the accuracy inversion exhibited by human experts on L1b. The bulk of coder and instruct models, regardless of parameter count, are uncorrelated with human performance at any tier or expertise level.

Figure 5: Alignment by human experience level—reasoning models significantly correlate with all tiers, indicating general human-like sensitivity.

Process-Level Analysis: CoT Length as Proxy for Effort

The paper rigorously analyzes CoT trace statistics as process-level indicators of reasoning depth. There is a robust, negative correlation ($\rho = -0.52$) between mean CoT trace length and accuracy per question: harder, more obfuscated tasks elicit longer reasoning traces and are more likely to be answered incorrectly. This effect is monotonic across obfuscation up to L2/L3, mirroring the scaling of cognitive effort in prior human studies, but lacking the non-monotonic switch to “insightful” System 2 reasoning observed in human experts.

Figure 6: There is a significant negative association between accuracy and mean CoT trace length per question, across models and tiers.

Figure 7: Log chain-of-thought lengths increase with obfuscation tier, reflecting increased model effort.

Figure 8: CoT trace length terciles (easy–hard) remain well-separated for all obfuscation tiers, reinforcing length as a difficulty marker.

Failure Mode Analysis: Adversarial Renaming and High-Confidence Errors

A key finding is that adversarial identifier renaming (L1b) can trigger high-confidence, incorrect model answers, especially when high semantic displacement co-occurs with identifier-level generation spikes. However, this failure mode is neither consistent nor global: it requires the interaction of high displacement and localized identifier disruption. By contrast, non-adversarial renaming alone does not reliably induce confident errors. Model response-confidence under L1b is also highly task and architecture dependent, unlike the more uniform human response.

Figure 9: Confidence–accuracy relationship in L1b, stratified by semantic displacement (left: terciles, right: deciles); high displacement destabilizes the calibration of confidence to correctness.

Implications

Software Engineering

• Tool designers should prefer reasoning-tuned LLMs when high alignment with human mental models is essential, e.g., in code review or comprehension-assisted development. Such alignment aids in developing explainable, human-supervised pipelines.
• Interpreting model confidence as a proxy for correctness is hazardous in the presence of strong identifier obfuscation. Additional verification or test scaffolding is warranted when identifier quality is poor.
• Verbose Chain-of-Thought does not inherently signal productive reasoning; excess length correlates with model struggle, not solution reliability.

Software Security

• Despite LLM advances, obfuscation (including control-flow flattening and identifier-based techniques) remains effective for impeding both automated and human analysis. LLM-based analyzers do not obviate obfuscation’s protective value.
• Attackers may exploit identifier-level adversarial strategies to induce “false positive” trust in model-based audits or security scans; thus human–model alignment becomes a new axis in adversarial defense.

Limitations and Threats to Validity

Results are interpreted with caution due to potential training data contamination (particularly in temporally post-dated models), restriction to deterministic/self-contained code snippets, and the inherent limitations of output prediction tasks as proxies for generalized comprehension.

Conclusion

The paper demonstrates that reasoning-tuned LLMs correlate significantly with human code comprehension failure patterns under semantics-preserving obfuscation, while other common model architectures do not. CoT trace length emerges as a robust indicator of reasoning effort and task difficulty, but only up to the point where high confidence is reliably calibrated to correctness. Subtle adversarial tactics in naming can induce confident model errors, marking a concrete gap in robustness.

Future research should focus on architectural and training protocol innovation to further close the gap between model and expert human reasoning—particularly the abstraction over misleading atom-level cues and the development of diagnostic signals for reasoning instability. Stronger human-model alignment in code comprehension opens new avenues for explainable, supervision-compatible AI in both software engineering and security-critical domains.