ReMIND Dataset for Deductive Code Reasoning

• The ReMIND dataset is a benchmark layer built on HumanEval and LiveCodeBench, filtering tasks to retain only functionally correct Python implementations.
• It uses a multi-agent pipeline—Mutator, Executor, and Inspector—to generate code mutations, execution traces, and control-flow validations for deductive reasoning analysis.
• The dataset evaluates LLM performance in self- and cross-execution settings, revealing biases and robustness in semantic execution prediction across diverse implementations.

Searching arXiv for the specified ReMind paper and closely related benchmarks to ground the article in the cited literature. The ReMIND dataset, in the sense established by "ReMind: Understanding Deductive Code Reasoning in LLMs," is not a standalone named benchmark but a ReMind-specific data artifact for studying deductive code reasoning in LLMs. It is constructed by filtering two public code-generation benchmarks—HumanEval and LiveCodeBench—and then attaching multiple functionally correct LLM-generated implementations, multiple test cases, and ReMind-generated artifacts such as mutations, execution traces, and control-flow-guided annotations. Its purpose is not to evaluate code synthesis, but to isolate whether an LLM can act as an interpreter of already-correct programs by predicting execution outputs purely through reasoning (Gao et al., 1 Nov 2025).

1. Definition and scope

Within the ReMind framework, the term “ReMIND dataset” denotes a benchmark layer built on top of existing public tasks rather than a newly named corpus. The underlying sources are HumanEval and LiveCodeBench, but the effective dataset is the filtered subset produced by ReMind’s validation protocol and multi-agent pipeline. In practice, this artifact consists of benchmark problems, multiple correct LLM-written solutions per problem, associated test cases, and runtime-generated reasoning artifacts.

A central defining property is that all retained code is functionally correct by construction. The paper prompts five LLMs to generate Python solutions, executes those solutions against official benchmark tests, and retains only those problems for which all five models pass all test cases. This removes ordinary program bugs as a confound. Any failure during evaluation is therefore attributed to deductive reasoning failure rather than implementation error. A common misconception is that ReMind is a code-generation benchmark; in fact, its operative task is semantic execution reasoning over correct programs, not synthesis, repair, or bug localization.

Another important boundary is that ReMind does not provide manually authored reasoning chains. Its annotation layers are algorithmic: test outcomes, code mutations, stepwise execution traces, and control-flow validation derived from Python abstract syntax trees. This makes the dataset especially relevant for studying reasoning dynamics, source-style bias, and execution-faithfulness rather than imitation of human chain-of-thought.

2. Construction protocol and corpus composition

The dataset is built from two public benchmarks. HumanEval contributes Python programming tasks oriented toward algorithmic reasoning, while LiveCodeBench contributes more realistic competitive-programming problems collected from sources such as LeetCode, AtCoder, and Codeforces. ReMind uses HumanEval’s official test cases and LiveCodeBench release_v5, then applies a filtering protocol that retains only problems for which all five generator models produce functionally correct solutions (Gao et al., 1 Nov 2025).

The five code generators are GPT-4o-mini, DeepSeek-V3, DeepSeek-R1, OpenAI o1 (low reasoning effort), and OpenAI o1 (high reasoning effort). For each retained problem, the dataset contains one problem specification, multiple test cases, and five distinct correct implementations—one from each model. Test cases per problem are capped at 15 for computational reasons.

Benchmark	Questions	Avg. Tests
HumanEval	152	6.9
LiveCodeBench	328	14.6

These counts arise after filtering. HumanEval starts from 164 Python tasks and retains 152. LiveCodeBench retains 328 problems. Because each retained problem has five correct implementations, the dataset contains five distinct program snippets per problem. This yields a multi-implementation benchmark in which functional equivalence is held fixed while code style and control-flow realization vary across sources. That structure is essential for measuring self-reasoning bias and cross-source degradation.

The LiveCodeBench subset is additionally selected from Oct 2023–Jan 2025, later than the knowledge cutoff of o1, to mitigate contamination. HumanEval is acknowledged as an older public benchmark and is used primarily as a simpler deductive-reasoning setting.

3. Task formalization and metric

The task formalized over the dataset is deductive code reasoning. Given a program $\mathcal{P}$ and an input $\mathcal{I}$, an LLM executor $E$ must predict the true output $\mathcal{O}$ without executing the code:

$$E(\mathcal{P}, \mathcal{I}) \longrightarrow \tilde{\mathcal{O}}, \qquad \text{with target } \tilde{\mathcal{O}} = \mathcal{O}.$$

This formulation makes the dataset a benchmark for execution tracing, variable tracking, branch resolution, and loop reasoning. It is not a bug-finding task, because the programs are known to satisfy the benchmark test suites. It is also not simply question answering over code, because correctness depends on faithful simulation of program behavior on concrete inputs.

Evaluation uses a strict per-problem accuracy analogous to Pass@1 over multiple test cases:

$$\mathrm{Accuracy} = \frac{1}{N} \sum_{i=1}^{N} \prod_{j=1}^{m_i} \bigl[ O_{ij} = \tilde{O}_{ij} \bigr].$$

Here, $N$ is the number of benchmark instances, $m_i$ is the number of test cases for instance $i$, $O_{ij}$ is the true output, and $\mathcal{I}$0 is the predicted output. The product term means that a problem counts as correct only if all of its test cases are reasoned correctly. This metric penalizes brittle reasoning that succeeds on isolated inputs but fails to maintain semantic consistency across the input set for a single implementation.

4. Runtime artifacts and annotation layers

Although the base benchmark is defined by problems, correct implementations, and test cases, ReMind enriches it at test time with additional artifacts that function as annotation layers. These arise from the three-agent architecture: Mutator, Executor, and Inspector (Gao et al., 1 Nov 2025).

The Mutator produces code variants from an original snippet. The paper distinguishes semantic mutations, such as variable renaming and reordering independent statements, from structural mutations, such as rewriting loop forms or conditions. Functional equivalence is desirable but not required. This is deliberate: when a mutation changes behavior, the divergence becomes a signal about reasoning-sensitive regions of the program. From a dataset perspective, this converts each original implementation into a local cluster of related variants.

The Executor then acts as an LLM interpreter. For a program-input pair, it generates a step-by-step reasoning trajectory that includes control-flow decisions, variable updates, intermediate states, and the final predicted output. The paper distinguishes “healthy execution,” which aligns with actual execution, from “problematic execution,” which includes both explicit errors and implicit errors. Explicit errors yield visibly incorrect intermediate or final states. Implicit errors are mistaken intermediate steps whose effect may be masked later, potentially leaving the final output correct or misleading.

The Inspector adds a structural validation layer. It constructs control-flow graphs for original and mutated snippets using Python’s ast module, compares predicted trajectories with those graphs, identifies infeasible branches, loop misinterpretations, and branch-condition mis-evaluations, and then emits control-flow refinement suggestions. These CFG-based checks are algorithmic rather than human-authored. A plausible implication is that the ReMind artifact is useful not only for benchmarking final-answer accuracy but also for studying the first point at which a reasoning trace diverges from executable semantics.

The paper does not specify a formal schema for released files, but it indicates that the released data likely include the filtered problem sets, LLM-generated correct code, and possibly mutated code, reasoning logs, CFG extraction outputs, and error reports. What is explicit is that there is no manually annotated chain-of-thought corpus.

5. Evaluation regimes and empirical profile

The dataset supports three evaluation regimes. Self-Execution (RQ1) asks an executor model to reason about code generated by itself. Cross-Execution (RQ2) asks it to reason about code generated by other LLMs. Zero-shot Reasoning (RQ3) evaluates on the filtered LiveCodeBench subset to test generalization on later, more complex problems (Gao et al., 1 Nov 2025).

These regimes are designed to expose three phenomena identified in the paper. First, there is an intrinsic gap between code generation and code reasoning: even when a model writes correct code, it may not reliably reason about that code’s execution. On HumanEval self-execution, for example, DeepSeek-V3 reaches 85.5% reasoning accuracy despite 100% generation correctness by filtering, while GPT-4o-mini reaches 55.3%. Second, there is a consistent bias toward code sources: models reason best about their own implementations and degrade on functionally equivalent code written by other models. DeepSeek-V3 is reported to lose up to 43% relative performance on some cross-execution combinations. Third, zero-shot generalization is weak on the more complex LiveCodeBench subset. The paper reports that o1-High drops to 64% self-execution on LiveCodeBench, and DeepSeek-R1 to about 54.1%.

The same dataset is also used to compare prompting and orchestration strategies. Baselines include vanilla chain-of-thought (CoT), Mutation, Chain-of-Code (CoC), and RHDA, with the underlying problem/code/test units held fixed. ReMind improves substantially under this protocol. On HumanEval with DeepSeek-V3 as executor, ReMind reaches 91.6%, compared with 59.2% for CoT, 64.5% for CoC, and 68.7% for RHDA. On LiveCodeBench with o1-High, ReMind reaches 78.4%, compared with 59.1% for CoT, 57.6% for CoC, and 61.1% for RHDA. The paper also reports lower standard deviation across code sources, indicating more stable behavior under cross-source variation. This suggests that the dataset is not merely difficult; it is specifically structured to reveal whether a method can reduce source-style bias while preserving semantic reasoning.

6. Access, limitations, and research use

The paper states that “the code and data are available at https://anonymous.4open.science/r/remind-71F0/” (Gao et al., 1 Nov 2025). No explicit license is stated in the paper itself. Reuse therefore depends both on the repository contents and on the licenses of the upstream benchmarks, HumanEval and LiveCodeBench.

Methodologically, the dataset operates in a zero-shot, test-time setting rather than in a supervised train/validation/test regime. The paper does not define conventional splits because the experiments do not fine-tune models on ReMind-annotated data. Adaptation occurs through test-time multi-agent computation: mutation, execution tracing, and inspection. This makes the artifact especially suitable for prompt engineering, inference-time orchestration, and reasoning-diagnostics research.

Several limitations are explicit. The current benchmark is restricted to Python and to sequential, deterministic programs. The authors note likely difficulty for concurrency, asynchronous code, and side-effect-heavy or I/O-intensive programs. Scale is also selective rather than massive: the benchmark emphasizes high-quality filtered instances rather than millions of programs. Computational overhead is nontrivial, with about 5.0 API calls per HumanEval instance and 5.2 per LiveCodeBench instance. A plausible implication is that the dataset is best viewed as an analysis-oriented benchmark for semantic reasoning quality, not as a broad web-scale corpus.

In practical use, the ReMIND dataset provides a controlled platform for evaluating deductive execution reasoning, probing self-reasoning bias, testing multi-agent inference methods, and collecting structured traces for error analysis. Its distinctive contribution is the combination of multiple correct implementations per problem with test cases and control-flow-grounded reasoning artifacts. That combination makes it possible to ask a sharper question than standard code benchmarks usually permit: not whether an LLM can write correct code, but whether it can reliably understand the execution of correct code across stylistic and structural variation.