---
title: 'CodeGlance: Multi-Dimensional Benchmark'
url: https://www.emergentmind.com/topics/codeglance
type: topic
---

# CodeGlance: Multi-Dimensional Benchmark

CodeGlance is a multi-dimensional benchmark for analyzing code reasoning challenges in large language models (LLMs) under realistic software-engineering conditions. It is designed around the observation that developers often need to understand code behavior at a glance during pull-request review, debugging, and navigation of unfamiliar codebases, and that this capability depends on reasoning about dynamic program behavior rather than only static code structure. The benchmark studies three scenarios—intrinsic logic reasoning, API interaction reasoning, and unseen function reasoning—and evaluates how model scale, code-complexity features, and augmentation strategies affect performance on these tasks [2602.13962].

## 1. Scope and problem formulation

CodeGlance is motivated by a gap in prior code-reasoning evaluation: existing studies are described as focusing primarily on isolated code snippets, thereby overlooking real-world situations involving external API interactions and unfamiliar functions. In response, CodeGlance investigates code reasoning across three realistic scenarios intended to separate challenges arising from pure execution tracing from those arising from externalized semantics or knowledge gaps [2602.13962].

The three scenarios are defined operationally. **Intrinsic Logic Reasoning (ILR)** asks a model to predict the concrete output of a self-contained function given specific inputs, purely by mentally simulating its control flow and data transformations. **API Interaction Reasoning (AIR)** asks for output prediction when the function invokes one or more well-known third-party library APIs such as NumPy, Pandas, or PyTorch. **Unseen Function Reasoning (UFR)** asks for output prediction when the snippet calls custom or newly introduced functions that the model has never seen in pre-training. These distinctions are central to the benchmark because they isolate different sources of difficulty: trace-based reasoning, API-semantic understanding, and inference under distributional novelty.

The benchmark’s framing emphasizes dynamic behavior. This is visible both in the task format—predicting concrete outputs under specified inputs—and in the later feature analysis, which includes execution-trace and encapsulated-execution measures rather than relying only on static properties such as code length. A plausible implication is that CodeGlance is structured less as a syntactic code-understanding benchmark than as a behavioral reasoning benchmark.

## 2. Benchmark composition and dataset construction

All snippets in CodeGlance are in Python. The benchmark aggregates material from five sources and organizes them into ILR, AIR, and UFR settings. The reported overall benchmark size is **1,969** samples, with averages of **5.81** code lines, **1.67** cyclomatic complexity, **2.52** external APIs, and **0.47** unseen functions [2602.13962].

| Scenario | Source Dataset | #Samples |
|---|---|---:|
| ILR | CRUXEval | 800 |
| AIR-DS | DS-1000-R | 1,019 |
| AIR-TD | TorchdataCode-R | 50 |
| UFR | PanNumEval-R | 50 |
| UFR | MonkBeatEval-R | 50 |

The benchmark statistics further distinguish the datasets. **CRUXEval** has average code lines **5.47**, average cyclomatic complexity **2.14**, and zero external APIs and unseen functions. **DS-1000-R** has average code lines **5.61**, average cyclomatic complexity **1.33**, and average external APIs **3.12**. **TorchdataCode-R** has average code lines **9.56**, average cyclomatic complexity **1.82**, and average external APIs **5.12**. **PanNumEval-R** has average code lines **5.70**, average cyclomatic complexity **1.02**, average external APIs **4.18**, and average unseen functions **0**. **MonkBeatEval-R** has average code lines **5.70**, average cyclomatic complexity **1.02**, average external APIs **4.18**, and average unseen functions **4.18** [2602.13962].

Dataset construction differs by source. **CRUXEval** was taken “as-is,” because it already provides code, inputs, and assertions. For **DS-1000, TorchdataCode, PanNumEval, and MonkBeatEval**, the benchmark uses a three-stage pipeline: **(1)** an LLM generates simplified assertions such as `assert len(result) == ?`; **(2)** the material is standardized into code plus single input/assertion pairs; and **(3)** humans verify correctness and remove nondeterministic or numerical-heavy cases. This construction protocol matters because it standardizes heterogeneous source tasks into a common output-prediction format.

The evaluation metrics are **pass@1** and **pass@3**. **pass@1** is defined as the number of problems solved with greedy decoding divided by the total problems, multiplied by 100%. **pass@3** is the “success within three sampled attempts” rate multiplied by 100%. Greedy decoding uses **temperature = 0** for pass@1, while sampling uses **temperature = 0.8** for pass@3 [2602.13962].

## 3. Multi-dimensional feature analysis

A defining contribution of CodeGlance is its extraction of **nine features across three categories** in order to analyze what makes a code snippet difficult. These are grouped into intrinsic logic features, API interaction features, and a knowledge feature. The feature set is designed to move beyond undifferentiated notions of “hardness” by attaching difficulty to specific structural or semantic dimensions of the code [2602.13962].

The **intrinsic logic features** are: **Code Lines** $L(c)$, the static number of lines in the snippet; **Cyclomatic Complexity** $M(c)$, the number of independent control-flow paths; and **Execution Trace Length** $T_{\text{exec}}(c)$, the actual number of statements executed for the given input. The **API interaction features** are: **API Invocation Count** $A(c)$, the total number of external API calls; **Total API Parameters** $P_{\text{total}}(c)$, the sum of all parameters in all API signatures; **Specified API Parameters** $P_{\text{spec}}(c)$, the number of non-default arguments actually provided; **Encapsulated Code Lines** $C_{\text{enc}}(c)$, the estimated total lines hidden inside invoked APIs; and **Encapsulated Execution Lines** $T_{\text{enc}}(c)$, the estimated dynamic steps inside API implementations. The **knowledge feature** is **Unseen Function Count** $U(c)$, the number of calls to functions outside the training distribution.

The benchmark provides formal definitions. For code lines,
$$
L(c)=|lines(c)|.
$$
For cyclomatic complexity in one-component code with $P=1$,
$$
M(c)=E-N+2,
$$
where $E$ is the number of edges and $N$ the number of nodes in the control-flow graph. For execution trace length,
$$
T_{\text{exec}}(c)=\sum_{i=1}^{n} t_i,
$$
where $n$ is the number of paths taken under the test input and $t_i$ is the number of statements executed on path $i$.

The remaining definitions are likewise explicit:
$$
A(c)=\sum_{f\in APIs} count_{calls}(f,c),
$$
$$
P_{\text{total}}(c)=\sum_{call\in APIs}|params(call)|,
$$
$$
P_{\text{spec}}(c)=\sum_{call\in APIs}|nonDefaultParams(call)|,
$$
$$
C_{\text{enc}}(c)=\sum_{f\in APIs} linesImplementation(f),
$$
$$
T_{\text{enc}}(c)=\sum_{f\in APIs} execStepsImplementation(f),
$$
$$
U(c)=\sum_{g\in calls} 1_{g\notin PretrainDist}.
$$

This feature design operationalizes two distinct ideas. First, dynamic burden can differ substantially from static size: a short snippet may nevertheless induce a long execution trace or depend on extensive hidden computation inside APIs. Second, knowledge burden can be partially orthogonal to logical burden: a snippet may be easy to trace but still difficult if function semantics are unavailable from pre-training. The benchmark’s later conclusions explicitly state that dynamic execution complexity is more predictive of LLM performance than static code size, which is consistent with this feature taxonomy [2602.13962].

## 4. Model evaluations and empirical findings

CodeGlance evaluates **seven models**: **Qwen2.5-Coder (3B, 7B, 14B, 32B)**, **DeepSeek-Coder-6.7B**, **DeepSeek-Coder-7B**, **GPT-4o-mini**, and **GPT-4o** [2602.13962]. The reported pass@1 results are given across four test sets: **CX** for CRUXEval, **DS** for DS-1000, **TD** for TorchdataCode, and **MB** for MonkBeatEval.

| Model | CX | DS | TD | MB |
|---|---:|---:|---:|---:|
| Qwen2.5-3B | 21.5 | 37.5 | 45.0 | 6.0 |
| Qwen2.5-7B | 46.2 | 49.9 | 65.0 | 40.5 |
| Qwen2.5-14B | 58.4 | 62.0 | 68.0 | 49.5 |
| Qwen2.5-32B | 65.3 | 66.2 | 68.5 | 64.0 |
| DeepSeek-6.7B | 40.4 | 44.8 | 56.0 | 36.2 |
| DeepSeek-7B | 42.5 | 47.9 | 37.4 | 44.8 |
| GPT-4o-mini | 56.6 | 61.1 | 66.2 | 52.2 |
| GPT-4o | 64.2 | 73.2 | 64.8 | 56.0 |

Several empirical findings are highlighted. **Model scaling yields large gains**: within Qwen2.5, performance on unseen functions rises from **6.0%** to **64.0%** on MonkBeatEval when scaling from **3B** to **32B**, a **10.7×** increase. **Unseen function reasoning is the hardest scenario for smaller models**: **Qwen2.5-3B** drops to **6.0%** on MB versus **37.5%** on DS, a **31.5-point gap**. At larger scale, this gap narrows: for **Qwen2.5-32B**, DS is **66.2%** and MB is **64.0%**, only **2.2 points** apart [2602.13962].

The results also indicate a **convergence plateau around \(\sim 65\)–\(68\%\) pass@1 at large scale**, which the benchmark interprets as suggesting persistent core challenges in dynamic reasoning. In addition, **model-specific strengths vary**: **GPT-4o** leads on **DS** with **73.2%**, but trails **Qwen2.5-32B** on **MB**, where the scores are **56.0%** versus **64.0%**. This suggests that proficiency in familiar API-heavy settings does not translate uniformly to settings dominated by unseen-function inference.

The overall pattern separates two regimes. Smaller models appear especially vulnerable to distributional novelty in function semantics, whereas larger models reduce—but do not eliminate—the penalty associated with unseen functions. This suggests that scale partially compensates for both reasoning and knowledge deficits, but not to the point of solving dynamic code understanding in general.

## 5. Augmentation strategies and differential effects

CodeGlance compares three lightweight augmentation strategies: **Chain-of-Thought (CoT) Prompting**, **Retrieval-Augmented Generation (RAG)**, and **Code Search Augmentation** [2602.13962]. CoT prompting includes both zero-shot prompting with formulations such as “Let’s think … step by step” and few-shot prompting with **2–3 worked examples**. RAG retrieves relevant official API documentation or function source code. Code search augmentation retrieves raw library or source-repository code for unseen functions.

The quantitative effects vary substantially by scenario. On **CRUXEval**, **Qwen2.5** improves markedly with **CoT few-shot**: **3B** moves from **21.5** to **46.1** (**+24.6**), **7B** from **46.2** to **56.2** (**+10.0**), **14B** from **58.4** to **74.8** (**+16.4**), and **32B** from **65.3** to **77.9** (**+12.6**). On **DS-1000-R**, the same strategy has much smaller or uneven benefits: **3B +0.4** (**37.5→37.9**), **7B +8.8**, **14B +1.5**, and **32B +1.8**. For **RAG(API doc)** on DS-1000-R, the effects are similarly heterogeneous: **3B** declines from **37.5** to **18.0** (**−19.5**), **7B** changes from **49.9** to **50.5** (**+0.6**), **14B +0.4**, and **32B +1.5**.

On unseen-function reasoning, the benchmark reports separate behavior on **MonkBeatEval**. **CoT few-shot** changes **Qwen2.5** from **6.0** to **26.0** (**+20.0**) at **3B**, from **40.5** to **43.5** (**+3.0**) at **7B**, from **49.5** to **71.0** (**+21.5**) at **14B**, and from **64.0** to **61.0** (**−3.0**) at **32B**. **RAG(API doc)** on MonkBeatEval yields **+4.0**, **+1.5**, **+2.0**, and **+7.0** for **3B**, **7B**, **14B**, and **32B**, respectively. **Code Search** shows smaller gains than docs overall, though the benchmark notes examples in both directions: **3B +8.0 vs. docs +4.0**, but **32B +0.5 vs. docs +7.0** [2602.13962].

The benchmark offers a functional interpretation of these differences. **CoT** is said to excel when the primary bottleneck is logical complexity, particularly long traces or multi-branch control flow, because explicit step-by-step decomposition helps even small models. **RAG/API-doc** is said to help only when genuine knowledge gaps exist, as with unseen functions, and only if the model has enough capacity to filter noisy context; smaller models are described as being overwhelmed. **Code search**, which supplies raw source, is reported as less effective than natural-language documentation because it imposes a nested comprehension task. This suggests that augmentation is not a generic performance booster but a mechanism that must be aligned with the dominant failure mode.

## 6. Recommendations, implications, and broader interpretation

The benchmark closes with several practical recommendations. For **model selection and routing**, it recommends that **logic-heavy tasks** involving small-to-medium code with complex branches can be handled by **medium-sized models (+CoT)**, whereas **heavy API or unseen-function tasks** should use **large models (30B+) or ensembles that can leverage RAG** [2602.13962]. For **prompt engineering**, it recommends always including a few illustrative CoT examples for intrinsically complex code reasoning and using zero-shot CoT sparingly because it can degrade pass@1 if models lack in-context examples.

For **knowledge augmentation**, the guidance is selective. In **known API tasks**, only large models are said to benefit from retrieving official docs, and the recommendation is to skip this for small models. In **unseen-function tasks**, the benchmark recommends prioritizing **concise API documentation** over **raw code dumps**, because natural-language descriptions integrate more easily. For **data augmentation**, it recommends mixing in **analogical code exercises** such as monkey/traversal mapping during fine-tuning so that the model learns to generalize on naming conventions, and emphasizing **dynamic execution scenarios** during training with attention to runtime traces and intermediate state tracking.

The implications for real-world assistants are framed in deployment terms. **AI pair programmers** should adapt augmentation strategy to context: **step-by-step CoT** for debugging or logic tracing, **live doc retrieval** for API unfamiliarity, and **model switching** when encountering completely new libraries. The benchmark also notes **memory and latency trade-offs**: CoT increases prompt length but adds minimal latency, whereas RAG incurs retrieval latency and should therefore be used selectively in IDE settings. It additionally proposes **continuous learning** via a lightweight function-usage tracker that flags when a user invokes an unseen function and triggers documentation or code retrieval.

Three summary claims synthesize the benchmark’s interpretation. First, **dynamic execution complexity is more predictive of LLM performance than static code size**. Second, **model scale and augmentation must be matched to the dominant challenge—logic versus knowledge**. Third, **no single LLM or strategy suffices across all realistic code-reasoning scenarios** [2602.13962]. Taken together, these points position CodeGlance as a benchmark for disentangling failure modes in code reasoning rather than merely ranking models on aggregate accuracy.

Source: https://www.emergentmind.com/topics/codeglance