---
title: 'LogiNumSynth: Synthetic Logical-Numerical Benchmark'
url: https://www.emergentmind.com/topics/loginumsynth
type: topic
---

# LogiNumSynth: Synthetic Logical-Numerical Benchmark

Searching arXiv for the named topic and closely related work to ground the article in recent literature.
LogiNumSynth is a synthetic data generation framework for natural-language problems that require the joint application of logical reasoning and numerical computation. It was introduced to address a gap in reasoning benchmarks: many existing datasets isolate deduction or arithmetic, whereas LogiNumSynth synthesizes tasks in which a model must combine rule chaining with numerical evaluation inside a formally constructed reasoning world. Its defining characteristics are explicit control over world richness, logical reasoning depth, and numerical computation complexity; generation of gold reasoning traces for process-level analysis; and use as both an evaluation benchmark and a source of targeted supervision for language models [2510.11031].

## 1. Conceptual definition and motivation

LogiNumSynth synthesizes reasoning instances of the form $\langle Facts, Rules, Query\rangle$, then converts them into fluent natural language [2510.11031]. Facts provide base information, rules encode conditional inference, and the query asks for the value of a target entity-attribute pair. A generated sample is solvable only by combining logical reasoning, such as chaining rules and matching conditions, with numerical reasoning, such as computing expressions in rule conclusions [2510.11031].

The framework was proposed because prevalent reasoning benchmarks tend to be single-modality in their demands. The cited landscape includes logical datasets such as RuleTaker, ProofWriter, FOLIO, FLD, and LogicNLI, alongside numerical datasets such as GSM8K, MATH, MAWPS, FinQA, SVAMP, and AIME [2510.11031]. This division leaves limited support for evaluating whether a model can integrate symbolic inference with arithmetic computation in a single problem. The paper identifies RuleArena as the only prior integrated benchmark mentioned, but characterizes it as having fixed scenarios and limited control over complexity [2510.11031]. LogiNumSynth is intended to fill that gap by supporting benchmark generation, controllable difficulty, process evaluation, and training-data synthesis within one system [2510.11031].

A plausible implication is that LogiNumSynth reframes integrated reasoning as a parameterized synthesis problem rather than as a static benchmark design problem. That interpretation is consistent with its emphasis on controllable reasoning worlds and difficulty knobs, rather than on a single immutable dataset.

## 2. Formal representation of the reasoning world

Each synthesized sample is built from entities, attributes, and relationships. The paper denotes these as $\mathbf{E} = \{e_1,e_2,\dots\}$, $\mathbf{A} = \{a_1,a_2,\dots\}$, and $\mathbf{R} = \{r_1,r_2,\dots\}$ [2510.11031]. Two fact types are used: relationship facts of the form
$$
r_k(e_i,e_j)
$$
and attribute facts of the form
$$
\mathrm{is}(e_i,a_j,\mathrm{num}),
$$
where $\mathrm{num}$ is a numerical attribute value [2510.11031].

Rules are implications
$$
Condition \rightarrow Conclusion,
$$
with the condition given as a conjunction of one or more atoms [2510.11031]. In the current implementation, these atoms are either attribute facts or relationship facts. Variables in the conclusion must appear in the condition [2510.11031]. The conclusion is either a relationship fact or an attribute fact whose value is computed by an expression:
$$
\mathrm{is}\left(e_\gamma,a_j,\mathrm{expr}(\dots)\right).
$$

The numerical expression language supports four expression types [2510.11031]:

1. **Constant expression**: a fixed value.
2. **Retrieval expression**:
   $$
   e_\alpha[a_i]
   $$
   meaning the value of attribute $a_i$ on entity $e_\alpha$.
3. **Calculation expression**:
   $$
   k \times e_\alpha[a_i] + b
   $$
   where $k,b$ are sampled parameters.
4. **Aggregation expression** over two subexpressions using max, min, addition, or subtraction; an example given is
   $$
   \max(k\times e_\alpha[a_i] + b,\ e_\beta[a_j]).
   $$

This formalization makes the reasoning world self-contained: the target answer depends only on internally generated entities, attributes, relations, rules, and values, with no dependence on external domain knowledge [2510.11031]. The paper also notes that the system is extensible to richer arithmetic or logic, such as exponentiation, trigonometric functions, disjunction, and negation [2510.11031]. This suggests that the present formulation is intentionally minimal but structurally open-ended.

## 3. Synthesis pipeline and reasoning DAG construction

The generation pipeline comprises four main modules: an initializer, a reasoning DAG constructor, a distraction synthesizer, and a natural language generator [2510.11031].

The initializer randomly selects entities, attributes, and relationships from predefined pools. The paper specifies the actual pools used: **7,944 entities** from `nltk.corpus.names`, **1,366 attributes** from `nltk.corpus.treebank`, and **976 relationships** from `nltk.corpus.wordnet` [2510.11031]. Synonyms are excluded within a sample to avoid ambiguity, and one entity plus one attribute are selected as the query target [2510.11031].

The core synthesis step is the construction of a reasoning directed acyclic graph. This DAG is built backward from the query and represents the gold reasoning structure: the root is the query, internal nodes are intermediate conclusions, and leaves are supporting facts [2510.11031]. The constructor recursively decides whether each node is supported by a direct fact or by a rule plus prerequisite subnodes, continuing until the configured reasoning depth is reached [2510.11031].

The paper gives pseudocode with functions `ConstructDAG`, `BuildNode`, `SynthesizeFact`, `SynthesizeRule`, and `UpdateConclusions` [2510.11031]. Its high-level algorithm is described as follows: start from an empty state $(\mathcal{G}, Facts, Rules, \mathcal{C})$, build the query node, synthesize a fact if maximum depth is reached, otherwise choose rule versus fact, recursively build condition nodes for synthesized rules, check consistency and acyclicity after each addition, and use forward reasoning to derive the gold answer and gold process [2510.11031].

After the gold DAG is created, the system adds irrelevant facts and irrelevant rules as distractors, subject to the constraint that they do not conflict with the gold knowledge [2510.11031]. The natural language generator then verbalizes the formal sample in two stages: template rendering followed by LLM refinement. The template stage uses hundreds of templates, and the refinement stage uses **Qwen3-8B** with thinking disabled to improve fluency [2510.11031]. The paper reports **222 attribute-fact templates**, **20 relationship-fact templates**, and **26 implication templates**, along with smaller template sets for expressions [2510.11031].

A plausible implication is that the backward DAG construction gives LogiNumSynth unusually strong control over solvability and traceability relative to forward-only random generation. The paper’s emphasis on gold reasoning paths and fixed-point verification supports that interpretation [2510.11031].

## 4. Difficulty control and dataset configurations

One of the framework’s central contributions is explicit control over difficulty along three axes: world richness, logical reasoning depth, and numerical computation complexity [2510.11031].

World richness is defined through scale and density. Scale refers to the number of entities, attributes, and relationships, while density refers to the number of facts and rules and thus to how densely connected the world is [2510.11031]. Logical reasoning depth is controlled directly by the DAG depth parameter; deeper recursive chains require more inference steps [2510.11031]. Numerical complexity is controlled by the expression types allowed, the probability assigned to each expression type, the operand ranges, and the ranges for parameters $k,b$ in calculation expressions [2510.11031].

The paper synthesizes **7 datasets**, divided into evaluation datasets and training datasets [2510.11031]. The evaluation sets are:

- **EL-EN**: Easy Logical, Easy Numerical
- **EL-HN**: Easy Logical, Hard Numerical
- **HL-EN**: Hard Logical, Easy Numerical
- **HL-HN**: Hard Logical, Hard Numerical
- **exHL-HN**: Extremely Hard Logical, Hard Numerical

The training sets are:

- **EL-Train**
- **EN-Train** [2510.11031]

The following table condenses configuration details explicitly reported in the paper.

| Dataset | Key configuration highlights | Size |
|---|---|---:|
| EL-EN | 10 entities, 15 attributes, 10 relationships, 15 facts, 15 rules, depth 1–3, one condition, operand range [1,10] | 500 |
| EL-HN | Same structure as EL-EN, but hard numerical range [-100,100] | 500 |
| HL-EN | Depth 4–6, 2–3 conditions, easy numbers | 500 |
| HL-HN | Depth 4–6, 2–3 conditions, hard numbers | 500 |
| exHL-HN | 30 entities, 40 attributes, 40 relationships, 15×depth facts, 5×depth rules, depth 7–10, 3–6 conditions | 400 |
| EL-Train | Training dataset | 5000 |
| EN-Train | Training dataset | 5000 |

A special detail is that exHL-HN is built as four depth-specific subsets, each containing 100 samples [2510.11031]. This staged construction indicates that the benchmark is intended not merely to distinguish easy from hard settings, but to expose degradation profiles as logical depth increases.

This control-oriented design contrasts with fixed-scenario benchmarks. The paper frames that distinction explicitly by noting that earlier mixed reasoning resources had limited complexity control and repetitive rule sets [2510.11031].

## 5. Correctness guarantees and evaluation methodology

The appendix formalizes `UpdateConclusions` as repeated application of rules until no new conclusions arise, framed as closure under a monotone consequence operator
$$
T(\mathcal{C}) = \mathcal{C} \cup \mathrm{Infer}(\mathcal{C}, Rules).
$$
The universe is defined as
$$
\mathcal{U}=\{\mathrm{is}(e,a,v)\mid e\in\mathbf{E}, a\in\mathbf{A}, v\in\mathbb{V}\}\cup\{r(e_i,e_j)\mid r\in\mathbf{R}, e_i,e_j\in\mathbf{E}\}
$$
and is finite, so the ascending chain of conclusions must stabilize at a fixed point [2510.11031]. Conflicts and cycles are rejected during synthesis, which guarantees uniqueness and consistency of answers [2510.11031].

This fixed-point account is significant because it grounds the synthetic benchmark in a well-defined inference semantics rather than in ad hoc trace generation. It also underpins the benchmark’s process-level supervision: a gold reasoning DAG can be validated against the closure of the rule system.

The evaluation uses **29 LLMs**, including Llama3.2-1B/3B/3.1-8B, the Qwen2.5 and Qwen3 families, Phi4-mini, Phi4, Phi4-reasoning, Phi4-reasoning-plus, the GLM4 series, DeepSeek-V3, DeepSeek-R1, GPT-4o, and GPT5-mini [2510.11031]. Models are tested with Chain-of-Thought prompting in 0-shot and few-shot settings, with additional 1-shot to 5-shot ablations reported in the appendix [2510.11031].

Two evaluation metrics are central [2510.11031]:

- **Answer accuracy**: whether the final answer is correct.
- **Process accuracy**: whether the reasoning steps are correct relative to the gold reasoning DAG.

To compute process accuracy, the model is prompted to summarize its reasoning explicitly; a text-based parser and an LLM-based structured extractor recover the reasoning chain; and each step is checked against the gold DAG for rule-condition satisfaction and correctness of the derived conclusion [2510.11031]. Process accuracy can award partial credit, can remain below 1 even when the final answer is correct, and can accept a different valid path if the synthesizer admits multiple valid reasoning paths [2510.11031]. The paper argues that this makes process accuracy more informative than answer accuracy for diagnosing reasoning quality, particularly in settings where transparency matters [2510.11031].

A recurrent misconception in benchmark interpretation is that final-answer correctness is a sufficient proxy for reasoning competence. LogiNumSynth explicitly rejects that assumption by construction: the benchmark is designed so that hidden reasoning faults can be measured independently of final answer status [2510.11031].

## 6. Experimental findings, failure modes, and training utility

The main experimental result is that many language models perform poorly on joint logical-numerical reasoning, especially under hard logical or hard numerical settings [2510.11031]. The paper gives illustrative results on HL-HN:

- **Llama3.1-8B-instruct**: process accuracy **0.83**, answer accuracy **5.00**
- **GPT-4o**: process accuracy **5.12**, answer accuracy **38.80** [2510.11031]

The strongest models reported are **DeepSeek-R1** and **GPT5-mini** [2510.11031]. On the four main datasets, their results are:

| Model | EL-EN | EL-HN | HL-EN | HL-HN |
|---|---|---|---|---|
| DeepSeek-R1 | 97.87 / 99.00 | 91.93 / 92.60 | 86.77 / 98.20 | 76.35 / 92.60 |
| GPT5-mini | 98.40 / 99.60 | 92.93 / 92.00 | 92.50 / 99.40 | 79.72 / 94.40 |

Each entry is reported as process accuracy / answer accuracy [2510.11031]. Even for these stronger systems, the move from EL-HN to HL-HN produces a larger drop in process accuracy than in answer accuracy [2510.11031]. The paper interprets this as evidence that models may still reach correct answers while failing to maintain a clean, logically valid reasoning trace under greater logical depth [2510.11031].

The exHL-HN stress test produces a marked collapse:

- **DeepSeek-R1**: **12.27** process / **38.50** answer
- **GPT5-mini**: **23.23** process / **54.75** answer [2510.11031]

Few-shot prompting does not consistently help; the paper reports that it sometimes improves performance and sometimes hurts it [2510.11031]. This is attributed to the intrinsic complexity of the tasks, where examples can guide but also distract or overload the model [2510.11031].

The failure modes identified by the paper include incorrect rule application, incorrect numerical computations, and incorrect intermediate results such as lexical mistakes in entity or attribute names, reversed relationship direction, or wrong intermediate conclusions that nonetheless yield the correct final answer [2510.11031]. The case studies sharpen these distinctions. In one case, an arithmetic error is masked by a later `min` operation, producing answer accuracy $=1$ but process accuracy $=0.6$; in another, process accuracy is $5/6$ while answer accuracy is $0$; in another, a shorter valid reasoning path than the gold path is accepted as correct [2510.11031].

The framework is also used for targeted training. The paper fine-tunes **Llama3.2-1B-instruct** and **Qwen3-1.7B** using supervised fine-tuning with LoRA and, in some cases, Recall Adam optimizer [2510.11031]. It uses **EL-Train** for improving numerical reasoning and **EN-Train** for improving logical reasoning [2510.11031]. Reported gains include:

- On numerical benchmarks such as GSM8K, MATH, MATHQA, SVAMP, MAWPS, and AIME:
  - Llama3.2-1B-instruct average: **37.27 → 38.44**
  - Qwen3-1.7B average: **72.54 → 73.20** [2510.11031]

- On formal logical benchmarks such as RuleTaker, ProofWriter, FOLIO, and FLD:
  - Qwen3-1.7B on FOLIO: **52.36 → 62.56** (+10.20)
  - Qwen3-1.7B average: **64.53 → 69.47** (+4.94) [2510.11031]

- On broader reasoning benchmarks such as LogiQA, ReClor, AbductionR, and RuleArena:
  - Llama3.2-1B-instruct on LogiQA: **12.85 → 23.35**
  - Qwen3-1.7B on RuleArena: **8.18 → 10.30** [2510.11031]

These results support the paper’s claim that LogiNumSynth is not only a diagnostic benchmark but also a useful supervision source whose benefits transfer to external reasoning tasks [2510.11031].

## 7. Position within the benchmark landscape and broader significance

Within the benchmark ecosystem, LogiNumSynth occupies a specific niche: controllable synthesis of natural-language tasks that integrate rule-based inference with arithmetic computation [2510.11031]. This distinguishes it from fixed-rule logical benchmarks and from purely mathematical problem sets. The paper’s framing implies that its main novelty lies not in inventing a new isolated task family, but in combining four capabilities rarely unified in one resource: formal controllability, process-level annotation, distractor-rich natural-language realization, and use for targeted training [2510.11031].

Its emphasis on process accuracy is particularly consequential. The benchmark treats reasoning traces as first-class evaluation objects rather than incidental explanatory text. This orientation aligns with broader concerns in theorem proving, diagnosis, and auditability, where correct outputs without reliable intermediate inference are often inadequate [2510.11031]. A plausible implication is that LogiNumSynth can be used to study not only whether a model solves a problem, but whether its generated rationale remains semantically faithful under increasing reasoning depth.

The framework’s extensibility is another salient feature. The paper notes that richer arithmetic and logic could be incorporated, including exponentiation, trigonometric functions, disjunction, and negation [2510.11031]. This suggests that the current system should be understood as a base formalism rather than a closed ontology of reasoning types.

In summary, LogiNumSynth is a controllable synthetic benchmark generator for joint logical-numerical reasoning, built around backward construction of gold reasoning DAGs, fixed-point verified inference, distractor injection, and natural-language verbalization [2510.11031]. Its experiments show that even strong frontier models degrade substantially on deeper integrated reasoning, especially in process fidelity, and its training results indicate that synthesized data of this type can improve performance beyond the benchmark itself [2510.11031].

Source: https://www.emergentmind.com/topics/loginumsynth