LogiNumSynth: Synthetic Logical-Numerical Benchmark

• LogiNumSynth is a framework that generates synthetic reasoning tasks by integrating logical deduction with numerical computation within a controlled formal world.
• It constructs a backward-directed acyclic graph to produce gold reasoning traces and injects distractors to validate both answer and process accuracy.
• The system explicitly controls world richness, reasoning depth, and numerical complexity, serving as both an evaluation benchmark and a supervision source.

Searching arXiv for the named topic and closely related work to ground the article in recent literature. LogiNumSynth is a synthetic data generation framework for natural-language problems that require the joint application of logical reasoning and numerical computation. It was introduced to address a gap in reasoning benchmarks: many existing datasets isolate deduction or arithmetic, whereas LogiNumSynth synthesizes tasks in which a model must combine rule chaining with numerical evaluation inside a formally constructed reasoning world. Its defining characteristics are explicit control over world richness, logical reasoning depth, and numerical computation complexity; generation of gold reasoning traces for process-level analysis; and use as both an evaluation benchmark and a source of targeted supervision for LLMs (Liu et al., 13 Oct 2025).

1. Conceptual definition and motivation

LogiNumSynth synthesizes reasoning instances of the form $\langle Facts, Rules, Query\rangle$, then converts them into fluent natural language (Liu et al., 13 Oct 2025). Facts provide base information, rules encode conditional inference, and the query asks for the value of a target entity-attribute pair. A generated sample is solvable only by combining logical reasoning, such as chaining rules and matching conditions, with numerical reasoning, such as computing expressions in rule conclusions (Liu et al., 13 Oct 2025).

The framework was proposed because prevalent reasoning benchmarks tend to be single-modality in their demands. The cited landscape includes logical datasets such as RuleTaker, ProofWriter, FOLIO, FLD, and LogicNLI, alongside numerical datasets such as GSM8K, MATH, MAWPS, FinQA, SVAMP, and AIME (Liu et al., 13 Oct 2025). This division leaves limited support for evaluating whether a model can integrate symbolic inference with arithmetic computation in a single problem. The paper identifies RuleArena as the only prior integrated benchmark mentioned, but characterizes it as having fixed scenarios and limited control over complexity (Liu et al., 13 Oct 2025). LogiNumSynth is intended to fill that gap by supporting benchmark generation, controllable difficulty, process evaluation, and training-data synthesis within one system (Liu et al., 13 Oct 2025).

A plausible implication is that LogiNumSynth reframes integrated reasoning as a parameterized synthesis problem rather than as a static benchmark design problem. That interpretation is consistent with its emphasis on controllable reasoning worlds and difficulty knobs, rather than on a single immutable dataset.

2. Formal representation of the reasoning world

Each synthesized sample is built from entities, attributes, and relationships. The paper denotes these as $\mathbf{E} = \{e_1,e_2,\dots\}$, $\mathbf{A} = \{a_1,a_2,\dots\}$, and $\mathbf{R} = \{r_1,r_2,\dots\}$ (Liu et al., 13 Oct 2025). Two fact types are used: relationship facts of the form

$r_k(e_i,e_j)$

and attribute facts of the form

$\mathrm{is}(e_i,a_j,\mathrm{num}),$

where $\mathrm{num}$ is a numerical attribute value (Liu et al., 13 Oct 2025).

Rules are implications

$Condition \rightarrow Conclusion,$

with the condition given as a conjunction of one or more atoms (Liu et al., 13 Oct 2025). In the current implementation, these atoms are either attribute facts or relationship facts. Variables in the conclusion must appear in the condition (Liu et al., 13 Oct 2025). The conclusion is either a relationship fact or an attribute fact whose value is computed by an expression:

$$\mathrm{is}\left(e_\gamma,a_j,\mathrm{expr}(\dots)\right).$$

The numerical expression language supports four expression types (Liu et al., 13 Oct 2025):

1. Constant expression: a fixed value.
2. Retrieval expression:

$e_\alpha[a_i]$

meaning the value of attribute $\mathbf{E} = \{e_1,e_2,\dots\}$0 on entity $\mathbf{E} = \{e_1,e_2,\dots\}$1.

1. Calculation expression:

$\mathbf{E} = \{e_1,e_2,\dots\}$2

where $\mathbf{E} = \{e_1,e_2,\dots\}$3 are sampled parameters.

2. Aggregation expression over two subexpressions using max, min, addition, or subtraction; an example given is

$\mathbf{E} = \{e_1,e_2,\dots\}$4

This formalization makes the reasoning world self-contained: the target answer depends only on internally generated entities, attributes, relations, rules, and values, with no dependence on external domain knowledge (Liu et al., 13 Oct 2025). The paper also notes that the system is extensible to richer arithmetic or logic, such as exponentiation, trigonometric functions, disjunction, and negation (Liu et al., 13 Oct 2025). This suggests that the present formulation is intentionally minimal but structurally open-ended.

3. Synthesis pipeline and reasoning DAG construction

The generation pipeline comprises four main modules: an initializer, a reasoning DAG constructor, a distraction synthesizer, and a natural language generator (Liu et al., 13 Oct 2025).

The initializer randomly selects entities, attributes, and relationships from predefined pools. The paper specifies the actual pools used: 7,944 entities from nltk.corpus.names, 1,366 attributes from nltk.corpus.treebank, and 976 relationships from nltk.corpus.wordnet (Liu et al., 13 Oct 2025). Synonyms are excluded within a sample to avoid ambiguity, and one entity plus one attribute are selected as the query target (Liu et al., 13 Oct 2025).

The core synthesis step is the construction of a reasoning directed acyclic graph. This DAG is built backward from the query and represents the gold reasoning structure: the root is the query, internal nodes are intermediate conclusions, and leaves are supporting facts (Liu et al., 13 Oct 2025). The constructor recursively decides whether each node is supported by a direct fact or by a rule plus prerequisite subnodes, continuing until the configured reasoning depth is reached (Liu et al., 13 Oct 2025).

The paper gives pseudocode with functions ConstructDAG, BuildNode, SynthesizeFact, SynthesizeRule, and UpdateConclusions (Liu et al., 13 Oct 2025). Its high-level algorithm is described as follows: start from an empty state $\mathbf{E} = \{e_1,e_2,\dots\}$5, build the query node, synthesize a fact if maximum depth is reached, otherwise choose rule versus fact, recursively build condition nodes for synthesized rules, check consistency and acyclicity after each addition, and use forward reasoning to derive the gold answer and gold process (Liu et al., 13 Oct 2025).

After the gold DAG is created, the system adds irrelevant facts and irrelevant rules as distractors, subject to the constraint that they do not conflict with the gold knowledge (Liu et al., 13 Oct 2025). The natural language generator then verbalizes the formal sample in two stages: template rendering followed by LLM refinement. The template stage uses hundreds of templates, and the refinement stage uses Qwen3-8B with thinking disabled to improve fluency (Liu et al., 13 Oct 2025). The paper reports 222 attribute-fact templates, 20 relationship-fact templates, and 26 implication templates, along with smaller template sets for expressions (Liu et al., 13 Oct 2025).

A plausible implication is that the backward DAG construction gives LogiNumSynth unusually strong control over solvability and traceability relative to forward-only random generation. The paper’s emphasis on gold reasoning paths and fixed-point verification supports that interpretation (Liu et al., 13 Oct 2025).

4. Difficulty control and dataset configurations

One of the framework’s central contributions is explicit control over difficulty along three axes: world richness, logical reasoning depth, and numerical computation complexity (Liu et al., 13 Oct 2025).

World richness is defined through scale and density. Scale refers to the number of entities, attributes, and relationships, while density refers to the number of facts and rules and thus to how densely connected the world is (Liu et al., 13 Oct 2025). Logical reasoning depth is controlled directly by the DAG depth parameter; deeper recursive chains require more inference steps (Liu et al., 13 Oct 2025). Numerical complexity is controlled by the expression types allowed, the probability assigned to each expression type, the operand ranges, and the ranges for parameters $\mathbf{E} = \{e_1,e_2,\dots\}$6 in calculation expressions (Liu et al., 13 Oct 2025).

The paper synthesizes 7 datasets, divided into evaluation datasets and training datasets (Liu et al., 13 Oct 2025). The evaluation sets are:

• EL-EN: Easy Logical, Easy Numerical
• EL-HN: Easy Logical, Hard Numerical
• HL-EN: Hard Logical, Easy Numerical
• HL-HN: Hard Logical, Hard Numerical
• exHL-HN: Extremely Hard Logical, Hard Numerical

The training sets are:

• EL-Train
• EN-Train (Liu et al., 13 Oct 2025)

The following table condenses configuration details explicitly reported in the paper.

Dataset	Key configuration highlights	Size
EL-EN	10 entities, 15 attributes, 10 relationships, 15 facts, 15 rules, depth 1–3, one condition, operand range [1,10]	500
EL-HN	Same structure as EL-EN, but hard numerical range [-100,100]	500
HL-EN	Depth 4–6, 2–3 conditions, easy numbers	500
HL-HN	Depth 4–6, 2–3 conditions, hard numbers	500
exHL-HN	30 entities, 40 attributes, 40 relationships, 15×depth facts, 5×depth rules, depth 7–10, 3–6 conditions	400
EL-Train	Training dataset	5000
EN-Train	Training dataset	5000

A special detail is that exHL-HN is built as four depth-specific subsets, each containing 100 samples (Liu et al., 13 Oct 2025). This staged construction indicates that the benchmark is intended not merely to distinguish easy from hard settings, but to expose degradation profiles as logical depth increases.

This control-oriented design contrasts with fixed-scenario benchmarks. The paper frames that distinction explicitly by noting that earlier mixed reasoning resources had limited complexity control and repetitive rule sets (Liu et al., 13 Oct 2025).

5. Correctness guarantees and evaluation methodology

The appendix formalizes UpdateConclusions as repeated application of rules until no new conclusions arise, framed as closure under a monotone consequence operator

$\mathbf{E} = \{e_1,e_2,\dots\}$7

The universe is defined as

$\mathbf{E} = \{e_1,e_2,\dots\}$8

and is finite, so the ascending chain of conclusions must stabilize at a fixed point (Liu et al., 13 Oct 2025). Conflicts and cycles are rejected during synthesis, which guarantees uniqueness and consistency of answers (Liu et al., 13 Oct 2025).

This fixed-point account is significant because it grounds the synthetic benchmark in a well-defined inference semantics rather than in ad hoc trace generation. It also underpins the benchmark’s process-level supervision: a gold reasoning DAG can be validated against the closure of the rule system.

The evaluation uses 29 LLMs, including Llama3.2-1B/3B/3.1-8B, the Qwen2.5 and Qwen3 families, Phi4-mini, Phi4, Phi4-reasoning, Phi4-reasoning-plus, the GLM4 series, DeepSeek-V3, DeepSeek-R1, GPT-4o, and GPT5-mini (Liu et al., 13 Oct 2025). Models are tested with Chain-of-Thought prompting in 0-shot and few-shot settings, with additional 1-shot to 5-shot ablations reported in the appendix (Liu et al., 13 Oct 2025).

Two evaluation metrics are central (Liu et al., 13 Oct 2025):

• Answer accuracy: whether the final answer is correct.
• Process accuracy: whether the reasoning steps are correct relative to the gold reasoning DAG.

To compute process accuracy, the model is prompted to summarize its reasoning explicitly; a text-based parser and an LLM-based structured extractor recover the reasoning chain; and each step is checked against the gold DAG for rule-condition satisfaction and correctness of the derived conclusion (Liu et al., 13 Oct 2025). Process accuracy can award partial credit, can remain below 1 even when the final answer is correct, and can accept a different valid path if the synthesizer admits multiple valid reasoning paths (Liu et al., 13 Oct 2025). The paper argues that this makes process accuracy more informative than answer accuracy for diagnosing reasoning quality, particularly in settings where transparency matters (Liu et al., 13 Oct 2025).

A recurrent misconception in benchmark interpretation is that final-answer correctness is a sufficient proxy for reasoning competence. LogiNumSynth explicitly rejects that assumption by construction: the benchmark is designed so that hidden reasoning faults can be measured independently of final answer status (Liu et al., 13 Oct 2025).

6. Experimental findings, failure modes, and training utility

The main experimental result is that many LLMs perform poorly on joint logical-numerical reasoning, especially under hard logical or hard numerical settings (Liu et al., 13 Oct 2025). The paper gives illustrative results on HL-HN:

• Llama3.1-8B-instruct: process accuracy 0.83, answer accuracy 5.00
• GPT-4o: process accuracy 5.12, answer accuracy 38.80 (Liu et al., 13 Oct 2025)

The strongest models reported are DeepSeek-R1 and GPT5-mini (Liu et al., 13 Oct 2025). On the four main datasets, their results are:

Model	EL-EN	EL-HN	HL-EN	HL-HN
DeepSeek-R1	97.87 / 99.00	91.93 / 92.60	86.77 / 98.20	76.35 / 92.60
GPT5-mini	98.40 / 99.60	92.93 / 92.00	92.50 / 99.40	79.72 / 94.40

Each entry is reported as process accuracy / answer accuracy (Liu et al., 13 Oct 2025). Even for these stronger systems, the move from EL-HN to HL-HN produces a larger drop in process accuracy than in answer accuracy (Liu et al., 13 Oct 2025). The paper interprets this as evidence that models may still reach correct answers while failing to maintain a clean, logically valid reasoning trace under greater logical depth (Liu et al., 13 Oct 2025).

The exHL-HN stress test produces a marked collapse:

• DeepSeek-R1: 12.27 process / 38.50 answer
• GPT5-mini: 23.23 process / 54.75 answer (Liu et al., 13 Oct 2025)

Few-shot prompting does not consistently help; the paper reports that it sometimes improves performance and sometimes hurts it (Liu et al., 13 Oct 2025). This is attributed to the intrinsic complexity of the tasks, where examples can guide but also distract or overload the model (Liu et al., 13 Oct 2025).

The failure modes identified by the paper include incorrect rule application, incorrect numerical computations, and incorrect intermediate results such as lexical mistakes in entity or attribute names, reversed relationship direction, or wrong intermediate conclusions that nonetheless yield the correct final answer (Liu et al., 13 Oct 2025). The case studies sharpen these distinctions. In one case, an arithmetic error is masked by a later min operation, producing answer accuracy $\mathbf{E} = \{e_1,e_2,\dots\}$9 but process accuracy $\mathbf{A} = \{a_1,a_2,\dots\}$0; in another, process accuracy is $\mathbf{A} = \{a_1,a_2,\dots\}$1 while answer accuracy is $\mathbf{A} = \{a_1,a_2,\dots\}$2; in another, a shorter valid reasoning path than the gold path is accepted as correct (Liu et al., 13 Oct 2025).

The framework is also used for targeted training. The paper fine-tunes Llama3.2-1B-instruct and Qwen3-1.7B using supervised fine-tuning with LoRA and, in some cases, Recall Adam optimizer (Liu et al., 13 Oct 2025). It uses EL-Train for improving numerical reasoning and EN-Train for improving logical reasoning (Liu et al., 13 Oct 2025). Reported gains include:

• On numerical benchmarks such as GSM8K, MATH, MATHQA, SVAMP, MAWPS, and AIME:
  • Llama3.2-1B-instruct average: 37.27 → 38.44
  • Qwen3-1.7B average: 72.54 → 73.20 (Liu et al., 13 Oct 2025)
• On formal logical benchmarks such as RuleTaker, ProofWriter, FOLIO, and FLD:
  • Qwen3-1.7B on FOLIO: 52.36 → 62.56 (+10.20)
  • Qwen3-1.7B average: 64.53 → 69.47 (+4.94) (Liu et al., 13 Oct 2025)
• On broader reasoning benchmarks such as LogiQA, ReClor, AbductionR, and RuleArena:
  • Llama3.2-1B-instruct on LogiQA: 12.85 → 23.35
  • Qwen3-1.7B on RuleArena: 8.18 → 10.30 (Liu et al., 13 Oct 2025)

These results support the paper’s claim that LogiNumSynth is not only a diagnostic benchmark but also a useful supervision source whose benefits transfer to external reasoning tasks (Liu et al., 13 Oct 2025).

7. Position within the benchmark landscape and broader significance

Within the benchmark ecosystem, LogiNumSynth occupies a specific niche: controllable synthesis of natural-language tasks that integrate rule-based inference with arithmetic computation (Liu et al., 13 Oct 2025). This distinguishes it from fixed-rule logical benchmarks and from purely mathematical problem sets. The paper’s framing implies that its main novelty lies not in inventing a new isolated task family, but in combining four capabilities rarely unified in one resource: formal controllability, process-level annotation, distractor-rich natural-language realization, and use for targeted training (Liu et al., 13 Oct 2025).

Its emphasis on process accuracy is particularly consequential. The benchmark treats reasoning traces as first-class evaluation objects rather than incidental explanatory text. This orientation aligns with broader concerns in theorem proving, diagnosis, and auditability, where correct outputs without reliable intermediate inference are often inadequate (Liu et al., 13 Oct 2025). A plausible implication is that LogiNumSynth can be used to study not only whether a model solves a problem, but whether its generated rationale remains semantically faithful under increasing reasoning depth.

The framework’s extensibility is another salient feature. The paper notes that richer arithmetic and logic could be incorporated, including exponentiation, trigonometric functions, disjunction, and negation (Liu et al., 13 Oct 2025). This suggests that the current system should be understood as a base formalism rather than a closed ontology of reasoning types.

In summary, LogiNumSynth is a controllable synthetic benchmark generator for joint logical-numerical reasoning, built around backward construction of gold reasoning DAGs, fixed-point verified inference, distractor injection, and natural-language verbalization (Liu et al., 13 Oct 2025). Its experiments show that even strong frontier models degrade substantially on deeper integrated reasoning, especially in process fidelity, and its training results indicate that synthesized data of this type can improve performance beyond the benchmark itself (Liu et al., 13 Oct 2025).