How LLMs Derive Program Meaning from Source Code

Determine how large language models derive program meaning from source code.

Background

The paper investigates how source code conveys meaning through two complementary channels: structural semantics (syntax, control/data flow, execution behavior) and human-interpretable naming (identifiers and docstrings). The authors argue that a true understanding of program intent should persist under semantics-preserving obfuscations that suppress naming cues.

Empirical results show sharp degradation in intent-level tasks like summarization when identifiers are obfuscated, and surprisingly non-trivial drops in execution-oriented tasks that should depend only on program structure. These findings suggest that existing benchmarks may reward memorization keyed to naming patterns rather than genuine semantic reasoning, motivating clarification of how LLMs actually derive program meaning.

References

LLMs achieve strong results on code tasks, but how they derive program meaning remains unclear.

— When Names Disappear: Revealing What LLMs Actually Understand About Code  (2510.03178 - Le et al., 3 Oct 2025) in Abstract

The competing ``naming from operations alone'' reading (\S\ref{sec:floormethod}) remains open and needs a fourth condition we did not run.

— A Memorization Floor for LLM Refinement of Decompiled Code  (2609.17236 - Asjad, 15 Sep 2026) in Threats to Validity, paragraph “The ablation conflates its target with idiom recognition”

Evaluations revealed that current LLMs struggle significantly across all three of these tasks. Because PLSemanticsBench presented a set of complex evaluations where models needed to articulate intermediate steps and output long execution traces, the models' uniformly low performance leaves an open question regarding exactly where their capability threshold is.

— Predicting Program Exit Code with LLMs and Programming Language Semantics  (2609.00579 - Marinov et al., 1 Sep 2026) in Section 1, Introduction

These methods demonstrate the potential of LLMs for mobile-specific program analysis, but their reliability and interpretability in security-critical settings remain open questions that warrant further investigation.

— Mobile App Analysis in the New Era: Challenges and the Road Ahead  (2610.17872 - Wu et al., 13 Oct 2026) in Section 6.1, subsection “LLM-Assisted Static Analysis”