RepoReason Framework

• RepoReason is a diagnostic framework that evaluates repository-level reasoning by tracking state mutations and interdependent code across multi-file repositories.
• It uses execution-driven mutation, semantic oracles, and dynamic program slicing to ensure verifiable and deterministic assertion regeneration.
• The framework introduces metrics such as ESV, MCL, and DFI to measure context fragmentation, simulation depth, and integration challenges in LLM code reasoning.

RepoReason is a white-box diagnostic benchmark framework developed to rigorously evaluate the repository-level reasoning capabilities of LLM agents. Repository-level reasoning is defined as an agent’s ability to maintain logical consistency and track state mutations across extensive, interdependent source file ecosystems, a task which surpasses the confined context of isolated function or snippet analysis. RepoReason aims to address methodological, cognitive, and anti-contamination challenges in assessing agentic code reasoning by combining semantically faithful mutation, abductive assertion verification, dynamic diagnostic slicing, and orthogonally structured cognitive metrics (Li et al., 7 Jan 2026).

1. Motivation and Conceptual Scope

The repository-level reasoning challenge encapsulates several core difficulties unique to real-world, multi-file codebases:

• Context fragmentation demands that agents discover, correlate, and reason over artifacts distributed across many files and modules, including traversing long-range imports.
• Long-chain state mutations require agents to follow variable or object state transitions that may originate in initialization routines, evolve through multiple transformation layers, and culminate in test assertions.
• Information aggregation tasks compel the synthesis of multi-source knowledge, frequently needing the agent to integrate divergent data points or logic from disparate call sites.
• Shortcut learning risk arises due to LLM pretraining on public code and test corpora, which might encourage superficial pattern matching rather than genuine reasoning.

RepoReason specifically addresses these issues by shifting from black-box, outcome-based benchmarks toward diagnostics that expose the “why” behind an agent’s failure, focusing on abductive assertion verification, anti-memorization mutational protocols, and orthogonally decomposed cognitive stress testing.

2. Execution-Driven Mutation and Semantic Oracle

Central to RepoReason is the notion of the Semantic Oracle: leveraging real Python execution as the ultimate ground-truth judge after applying disciplined mutations to both code and tests. This protocol entails:

• Test selection and mutation: Curated real-world test functions are systematically mutated by a teacher LLM, introducing both syntax-level (e.g., variable renaming, reformatting) and logic-level (e.g., altered constants/inputs) changes, while keeping call sequences invariant.
• Probe injection and dynamic execution: All assertions are programmatically rewritten as print("DEBUG_RESULT:", <expression>) statements. The system executes the mutated code using sys.settrace() to capture runtime values.
• Assertion regeneration: The teacher model re-infers assertion targets based on the observed runtime states, restoring stylistic alignment with the original assertions.

Strict validation ensures that mutated tasks only enter the benchmark if they: (a) execute without failure, (b) have passing regenerated assertions, and (c) preserve the original API call sequence. Deterministic value protocols filter out instability from, for example, timestamps or UUIDs.

Pseudocode summary for mutation and validation:

def execution_driven_mutation(repo, test_path, teacher_model): mutated_test = teacher_model.mutate(test_path) probed_test = inject_debug_prints(mutated_test) write_to_disk(probed_test) debug_values = run_and_capture(repo, probed_test, marker="DEBUG_RESULT:") new_assertions = teacher_model.style_consistent_assertions(probed_test, debug_values) final_test = replace_probes_with_asserts(probed_test, new_assertions) if not (executes_without_error(final_test) and all_assertions_pass(final_test) and api_call_sequence_preserved(test_path, final_test)): discard_variant() else: return final_test

This process directly undermines memorization-based strategies, compelling agents to reason over the mutated, execution-derived ground truth.

3. Dynamic Program Slicing and Diagnostic Infrastructure

RepoReason systematically leverages dynamic backward program slicing to construct a Minimal Causal Computational Subgraph for each abductive verification task:

1. The slicing root is the (masked) assertion event.
2. For each element $E$ in the candidate slice, all inputs are recursively resolved to their last-defining events.
3. This continues until all variables in the backward slice are resolved, producing the minimal set of statements influencing the assertion outcome.

This dynamic subgraph is strictly runtime-trace-based, isolating only those computational steps empirically required for the assertion. The resulting slice forms the analytic foundation from which all further diagnostics and metrics are derived.

Dynamic slicing pseudocode:

def dynamic_slice(trace, assertion_event): slice_set = {assertion_event} worklist = [assertion_event] while worklist: e = worklist.pop() for arg in e.inputs: producer = trace.find_definition(arg, before=e.timestamp) if producer not in slice_set: slice_set.add(producer) worklist.append(producer) return slice_set

4. Cognitive Metrics: ESV, MCL, and DFI

RepoReason decomposes reasoning demands into three orthogonally defined metrics, each quantifying a distinct axis of computational load in the dynamic slice:

• Effective Sliced Volume (ESV) measures reading load—the sum of code volume (in tokens or lines) of all functions or methods contributing to the assertion:

$\mathrm{ESV} = \sum_{i=1}^{n}|S_i|$

A high ESV signals context overload, with the agent facing a diffuse set of source artifacts.

• Mutation Chain Length (MCL) quantifies simulation depth—the total number of dynamic state-update events or the maximal dependency chain length:

$$\mathrm{MCL} = \sum_{s\in S} \mathrm{freq}(s) \qquad \text{or}\qquad \mathrm{MCL} = \max_{p\in P} |\mathrm{path}(p)|$$

High MCL corresponds to deep or repeated state-tracking requirements.

• Dependency Fan-in (DFI) captures integration width—the aggregate count of unique upstream sources (variables, outputs) implicated in forming the assertion:

$\mathrm{DFI} = \left| \bigcup_{t=1}^T F_t \right|$

High DFI directly measures the aggregation challenge, identifying tasks where an agent must synthesize many semantically independent inputs.

Empirical results reveal that DFI is the primary bottleneck for state-of-the-art LLM agents, sharply curtailing accuracy on tasks requiring wide fan-in dependency synthesis.

5. Benchmark Task Construction Methodology

RepoReason tasks are structured around abductive assertion verification:

• Assertions such as assert len(cache) == 5 are masked to assert len(cache) == <mask>, requiring the agent to infer the appropriate value by tracing the mutated execution.
• Execution-driven mutation ensures that every task variant is novel and eliminates contamination from pretraining corpora.
• Only stable, deterministic expressions are permitted as valid assertion fill-ins, enforced by a whitelist of literal or constructor types.
• Inclusion criteria stipulate successful execution, unique and deterministic result, and assertion validity on the mutated repository.

This methodology systematically prevents shortcut learning and maximizes the logical depth of challenge for agentic reasoning.

6. Empirical Results and Diagnostic Insights

RepoReason’s evaluation utilizes the OpenHands ReadOnlyAgent framework with LLMs including Claude-4.5-Sonnet, DeepSeek-v3.1-Terminus, GPT-5.2, Kimi-K2, and Qwen3-Coder-480B, under a standardized inference configuration (temperature 0, max tokens 8192).

Overall Pass@1 Results:

Model	Pass@1
Claude-4.5-Sonnet	66.98%
DeepSeek-v3.1-Terminus	60.96%
GPT-5.2	56.86%
Kimi-K2	54.74%
Qwen3-Coder-480B	50.56%

• Performance declines across the easy-medium-hard spectrum, with Claude maintaining the lead overall.
• Most notably, accuracy degrades steeply as DFI increases; specifically, when $\mathrm{DFI}>20$, most models fall below 40% accuracy.
• Pearson correlation analysis confirms that DFI manifests the strongest negative correlation with agent accuracy (e.g., GPT-5.2: $-0.234$ with DFI).
• This aggregation deficit demonstrates that wide-fan-in integration remains the principal unsolved challenge for current agentic code reasoning models.

7. Implications and Prospective Developments

Key implications for future research in agentic LLM reasoning include:

• Prioritizing wide-context integration mechanisms capable of handling large dependency fan-in, which may entail specialized network modules or persistent memory features for tracking distributed state.
• Incorporating dynamic tracing modules or enhanced memory buffers to facilitate partial state update retention across arbitrarily deep call hierarchies.
• Developing aggregation attention mechanisms to better extract, represent, and merge independent upstream dependencies.

Potential extensions of RepoReason include broadening language coverage (Java, C++, Rust) via analogous dynamic slicing infrastructure, merging static and dynamic program slicing to prune infeasible code paths, adaptive task generation to incrementally increase ESV/MCL/DFI as agent proficiency improves, and integrating code repair capabilities for agents to propose remediations alongside diagnostic reasoning.

By uniting rigorous mutation, dynamic slicing, and fine-grained metrics, RepoReason establishes the first white-box, repository-level code reasoning benchmark. Its results argue that, while state-tracking and context fragmentation pose ongoing obstacles, the frontier bottleneck now lies in aggregating many independent logical sources—codified by DFI—as the most urgent target for next-generation agentic reasoning systems (Li et al., 7 Jan 2026).