CodeGlance: Multi-Dimensional Benchmark

• CodeGlance is a multi-dimensional benchmark that evaluates dynamic code reasoning in LLMs by focusing on intrinsic logic, API interaction, and unseen function challenges.
• It simulates real-world software tasks such as pull-request review, debugging, and navigating unfamiliar code, using metrics like execution trace length and cyclomatic complexity.
• The benchmark highlights the importance of model scale and tailored augmentation strategies like chain-of-thought and retrieval augmentation in overcoming dynamic reasoning limitations.

CodeGlance is a multi-dimensional benchmark for analyzing code reasoning challenges in LLMs under realistic software-engineering conditions. It is designed around the observation that developers often need to understand code behavior at a glance during pull-request review, debugging, and navigation of unfamiliar codebases, and that this capability depends on reasoning about dynamic program behavior rather than only static code structure. The benchmark studies three scenarios—intrinsic logic reasoning, API interaction reasoning, and unseen function reasoning—and evaluates how model scale, code-complexity features, and augmentation strategies affect performance on these tasks (Wang et al., 15 Feb 2026).

1. Scope and problem formulation

CodeGlance is motivated by a gap in prior code-reasoning evaluation: existing studies are described as focusing primarily on isolated code snippets, thereby overlooking real-world situations involving external API interactions and unfamiliar functions. In response, CodeGlance investigates code reasoning across three realistic scenarios intended to separate challenges arising from pure execution tracing from those arising from externalized semantics or knowledge gaps (Wang et al., 15 Feb 2026).

The three scenarios are defined operationally. Intrinsic Logic Reasoning (ILR) asks a model to predict the concrete output of a self-contained function given specific inputs, purely by mentally simulating its control flow and data transformations. API Interaction Reasoning (AIR) asks for output prediction when the function invokes one or more well-known third-party library APIs such as NumPy, Pandas, or PyTorch. Unseen Function Reasoning (UFR) asks for output prediction when the snippet calls custom or newly introduced functions that the model has never seen in pre-training. These distinctions are central to the benchmark because they isolate different sources of difficulty: trace-based reasoning, API-semantic understanding, and inference under distributional novelty.

The benchmark’s framing emphasizes dynamic behavior. This is visible both in the task format—predicting concrete outputs under specified inputs—and in the later feature analysis, which includes execution-trace and encapsulated-execution measures rather than relying only on static properties such as code length. A plausible implication is that CodeGlance is structured less as a syntactic code-understanding benchmark than as a behavioral reasoning benchmark.

2. Benchmark composition and dataset construction

All snippets in CodeGlance are in Python. The benchmark aggregates material from five sources and organizes them into ILR, AIR, and UFR settings. The reported overall benchmark size is 1,969 samples, with averages of 5.81 code lines, 1.67 cyclomatic complexity, 2.52 external APIs, and 0.47 unseen functions (Wang et al., 15 Feb 2026).

Scenario	Source Dataset	#Samples
ILR	CRUXEval	800
AIR-DS	DS-1000-R	1,019
AIR-TD	TorchdataCode-R	50
UFR	PanNumEval-R	50
UFR	MonkBeatEval-R	50

The benchmark statistics further distinguish the datasets. CRUXEval has average code lines 5.47, average cyclomatic complexity 2.14, and zero external APIs and unseen functions. DS-1000-R has average code lines 5.61, average cyclomatic complexity 1.33, and average external APIs 3.12. TorchdataCode-R has average code lines 9.56, average cyclomatic complexity 1.82, and average external APIs 5.12. PanNumEval-R has average code lines 5.70, average cyclomatic complexity 1.02, average external APIs 4.18, and average unseen functions 0. MonkBeatEval-R has average code lines 5.70, average cyclomatic complexity 1.02, average external APIs 4.18, and average unseen functions 4.18 (Wang et al., 15 Feb 2026).

Dataset construction differs by source. CRUXEval was taken “as-is,” because it already provides code, inputs, and assertions. For DS-1000, TorchdataCode, PanNumEval, and MonkBeatEval, the benchmark uses a three-stage pipeline: (1) an LLM generates simplified assertions such as assert len(result) == ?; (2) the material is standardized into code plus single input/assertion pairs; and (3) humans verify correctness and remove nondeterministic or numerical-heavy cases. This construction protocol matters because it standardizes heterogeneous source tasks into a common output-prediction format.

The evaluation metrics are pass@1 and pass@3. pass@1 is defined as the number of problems solved with greedy decoding divided by the total problems, multiplied by 100%. pass@3 is the “success within three sampled attempts” rate multiplied by 100%. Greedy decoding uses temperature = 0 for pass@1, while sampling uses temperature = 0.8 for pass@3 (Wang et al., 15 Feb 2026).

3. Multi-dimensional feature analysis

A defining contribution of CodeGlance is its extraction of nine features across three categories in order to analyze what makes a code snippet difficult. These are grouped into intrinsic logic features, API interaction features, and a knowledge feature. The feature set is designed to move beyond undifferentiated notions of “hardness” by attaching difficulty to specific structural or semantic dimensions of the code (Wang et al., 15 Feb 2026).

The intrinsic logic features are: Code Lines $L(c)$, the static number of lines in the snippet; Cyclomatic Complexity $M(c)$, the number of independent control-flow paths; and Execution Trace Length $T_{\text{exec}}(c)$, the actual number of statements executed for the given input. The API interaction features are: API Invocation Count $A(c)$, the total number of external API calls; Total API Parameters $P_{\text{total}}(c)$, the sum of all parameters in all API signatures; Specified API Parameters $P_{\text{spec}}(c)$, the number of non-default arguments actually provided; Encapsulated Code Lines $C_{\text{enc}}(c)$, the estimated total lines hidden inside invoked APIs; and Encapsulated Execution Lines $T_{\text{enc}}(c)$, the estimated dynamic steps inside API implementations. The knowledge feature is Unseen Function Count $U(c)$, the number of calls to functions outside the training distribution.

The benchmark provides formal definitions. For code lines,

$L(c)=|lines(c)|.$

For cyclomatic complexity in one-component code with $M(c)$0,

$M(c)$1

where $M(c)$2 is the number of edges and $M(c)$3 the number of nodes in the control-flow graph. For execution trace length,

$M(c)$4

where $M(c)$5 is the number of paths taken under the test input and $M(c)$6 is the number of statements executed on path $M(c)$7.

The remaining definitions are likewise explicit:

$M(c)$8

$M(c)$9

$T_{\text{exec}}(c)$0

$T_{\text{exec}}(c)$1

$T_{\text{exec}}(c)$2

$T_{\text{exec}}(c)$3

This feature design operationalizes two distinct ideas. First, dynamic burden can differ substantially from static size: a short snippet may nevertheless induce a long execution trace or depend on extensive hidden computation inside APIs. Second, knowledge burden can be partially orthogonal to logical burden: a snippet may be easy to trace but still difficult if function semantics are unavailable from pre-training. The benchmark’s later conclusions explicitly state that dynamic execution complexity is more predictive of LLM performance than static code size, which is consistent with this feature taxonomy (Wang et al., 15 Feb 2026).

4. Model evaluations and empirical findings

CodeGlance evaluates seven models: Qwen2.5-Coder (3B, 7B, 14B, 32B), DeepSeek-Coder-6.7B, DeepSeek-Coder-7B, GPT-4o-mini, and GPT-4o (Wang et al., 15 Feb 2026). The reported pass@1 results are given across four test sets: CX for CRUXEval, DS for DS-1000, TD for TorchdataCode, and MB for MonkBeatEval.

Model	CX	DS	TD	MB
Qwen2.5-3B	21.5	37.5	45.0	6.0
Qwen2.5-7B	46.2	49.9	65.0	40.5
Qwen2.5-14B	58.4	62.0	68.0	49.5
Qwen2.5-32B	65.3	66.2	68.5	64.0
DeepSeek-6.7B	40.4	44.8	56.0	36.2
DeepSeek-7B	42.5	47.9	37.4	44.8
GPT-4o-mini	56.6	61.1	66.2	52.2
GPT-4o	64.2	73.2	64.8	56.0

Several empirical findings are highlighted. Model scaling yields large gains: within Qwen2.5, performance on unseen functions rises from 6.0% to 64.0% on MonkBeatEval when scaling from 3B to 32B, a 10.7× increase. Unseen function reasoning is the hardest scenario for smaller models: Qwen2.5-3B drops to 6.0% on MB versus 37.5% on DS, a 31.5-point gap. At larger scale, this gap narrows: for Qwen2.5-32B, DS is 66.2% and MB is 64.0%, only 2.2 points apart (Wang et al., 15 Feb 2026).

The results also indicate a convergence plateau around $T_{\text{exec}}(c)$4–$T_{\text{exec}}(c)$5 pass@1 at large scale, which the benchmark interprets as suggesting persistent core challenges in dynamic reasoning. In addition, model-specific strengths vary: GPT-4o leads on DS with 73.2%, but trails Qwen2.5-32B on MB, where the scores are 56.0% versus 64.0%. This suggests that proficiency in familiar API-heavy settings does not translate uniformly to settings dominated by unseen-function inference.

The overall pattern separates two regimes. Smaller models appear especially vulnerable to distributional novelty in function semantics, whereas larger models reduce—but do not eliminate—the penalty associated with unseen functions. This suggests that scale partially compensates for both reasoning and knowledge deficits, but not to the point of solving dynamic code understanding in general.

5. Augmentation strategies and differential effects

CodeGlance compares three lightweight augmentation strategies: Chain-of-Thought (CoT) Prompting, Retrieval-Augmented Generation (RAG), and Code Search Augmentation (Wang et al., 15 Feb 2026). CoT prompting includes both zero-shot prompting with formulations such as “Let’s think … step by step” and few-shot prompting with 2–3 worked examples. RAG retrieves relevant official API documentation or function source code. Code search augmentation retrieves raw library or source-repository code for unseen functions.

The quantitative effects vary substantially by scenario. On CRUXEval, Qwen2.5 improves markedly with CoT few-shot: 3B moves from 21.5 to 46.1 (+24.6), 7B from 46.2 to 56.2 (+10.0), 14B from 58.4 to 74.8 (+16.4), and 32B from 65.3 to 77.9 (+12.6). On DS-1000-R, the same strategy has much smaller or uneven benefits: 3B +0.4 (37.5→37.9), 7B +8.8, 14B +1.5, and 32B +1.8. For RAG(API doc) on DS-1000-R, the effects are similarly heterogeneous: 3B declines from 37.5 to 18.0 (−19.5), 7B changes from 49.9 to 50.5 (+0.6), 14B +0.4, and 32B +1.5.

On unseen-function reasoning, the benchmark reports separate behavior on MonkBeatEval. CoT few-shot changes Qwen2.5 from 6.0 to 26.0 (+20.0) at 3B, from 40.5 to 43.5 (+3.0) at 7B, from 49.5 to 71.0 (+21.5) at 14B, and from 64.0 to 61.0 (−3.0) at 32B. RAG(API doc) on MonkBeatEval yields +4.0, +1.5, +2.0, and +7.0 for 3B, 7B, 14B, and 32B, respectively. Code Search shows smaller gains than docs overall, though the benchmark notes examples in both directions: 3B +8.0 vs. docs +4.0, but 32B +0.5 vs. docs +7.0 (Wang et al., 15 Feb 2026).

The benchmark offers a functional interpretation of these differences. CoT is said to excel when the primary bottleneck is logical complexity, particularly long traces or multi-branch control flow, because explicit step-by-step decomposition helps even small models. RAG/API-doc is said to help only when genuine knowledge gaps exist, as with unseen functions, and only if the model has enough capacity to filter noisy context; smaller models are described as being overwhelmed. Code search, which supplies raw source, is reported as less effective than natural-language documentation because it imposes a nested comprehension task. This suggests that augmentation is not a generic performance booster but a mechanism that must be aligned with the dominant failure mode.

6. Recommendations, implications, and broader interpretation

The benchmark closes with several practical recommendations. For model selection and routing, it recommends that logic-heavy tasks involving small-to-medium code with complex branches can be handled by medium-sized models (+CoT), whereas heavy API or unseen-function tasks should use large models (30B+) or ensembles that can leverage RAG (Wang et al., 15 Feb 2026). For prompt engineering, it recommends always including a few illustrative CoT examples for intrinsically complex code reasoning and using zero-shot CoT sparingly because it can degrade pass@1 if models lack in-context examples.

For knowledge augmentation, the guidance is selective. In known API tasks, only large models are said to benefit from retrieving official docs, and the recommendation is to skip this for small models. In unseen-function tasks, the benchmark recommends prioritizing concise API documentation over raw code dumps, because natural-language descriptions integrate more easily. For data augmentation, it recommends mixing in analogical code exercises such as monkey/traversal mapping during fine-tuning so that the model learns to generalize on naming conventions, and emphasizing dynamic execution scenarios during training with attention to runtime traces and intermediate state tracking.

The implications for real-world assistants are framed in deployment terms. AI pair programmers should adapt augmentation strategy to context: step-by-step CoT for debugging or logic tracing, live doc retrieval for API unfamiliarity, and model switching when encountering completely new libraries. The benchmark also notes memory and latency trade-offs: CoT increases prompt length but adds minimal latency, whereas RAG incurs retrieval latency and should therefore be used selectively in IDE settings. It additionally proposes continuous learning via a lightweight function-usage tracker that flags when a user invokes an unseen function and triggers documentation or code retrieval.

Three summary claims synthesize the benchmark’s interpretation. First, dynamic execution complexity is more predictive of LLM performance than static code size. Second, model scale and augmentation must be matched to the dominant challenge—logic versus knowledge. Third, no single LLM or strategy suffices across all realistic code-reasoning scenarios (Wang et al., 15 Feb 2026). Taken together, these points position CodeGlance as a benchmark for disentangling failure modes in code reasoning rather than merely ranking models on aggregate accuracy.