How LLMs Derive Program Meaning from Source Code

Determine how large language models derive program meaning from source code.

Background

The paper investigates how source code conveys meaning through two complementary channels: structural semantics (syntax, control/data flow, execution behavior) and human-interpretable naming (identifiers and docstrings). The authors argue that a true understanding of program intent should persist under semantics-preserving obfuscations that suppress naming cues.

Empirical results show sharp degradation in intent-level tasks like summarization when identifiers are obfuscated, and surprisingly non-trivial drops in execution-oriented tasks that should depend only on program structure. These findings suggest that existing benchmarks may reward memorization keyed to naming patterns rather than genuine semantic reasoning, motivating clarification of how LLMs actually derive program meaning.