RepublicQA: Logical-Semantic Benchmark

• RepublicQA is a benchmark for logical reasoning that intertwines logical and semantic complexities using abstract propositions from Plato’s Republic.
• It employs a three-valued format (True, False, Uncertain) alongside multi-hop deductions and a semiotic structure inspired by Greimas’ Semiotic Square.
• The dataset is manually curated with double annotation and rigorous FOL mapping, enabling nuanced evaluations of philosophical arguments.

to=arxiv_search.search 彩神争霸能json {"query":"(Zhang et al., 29 Sep 2025)", "max_results": 5} to=arxiv_search.search 时时彩后രം 天天中彩票怎么json {"query":"(Chen et al., 2024)", "max_results": 5} RepublicQA is a benchmark for logical reasoning that is explicitly designed to couple logical complexity with semantic complexity rather than treating them as separable dimensions. Introduced together with LogicAgent, it is built from Plato’s Republic and targets reasoning over philosophically abstract, context-sensitive propositions involving ambiguity, conflicting stances, and both contrary and contradictory relations. Its task format is three-valued—True, False, or Uncertain—and its reported reading difficulty reaches FKGL = 11.94, placing it at college level. In the source paper, RepublicQA is positioned as a complement to benchmarks such as ProntoQA, ProofWriter, FOLIO, and ProverQA, which are described as formally well structured but comparatively limited in semantic richness and indeterminacy (Zhang et al., 29 Sep 2025).

1. Benchmark purpose and problem setting

RepublicQA was introduced to fill a gap in logical reasoning evaluation. The benchmark is motivated by the observation that contemporary datasets often emphasize concrete everyday scenarios, largely unambiguous propositions, template-based generation, and shallow or purely combinatorial logical structure. Under that diagnosis, they test whether a model can follow explicit logical rules, but they do not adequately probe settings in which propositions are philosophically abstract, language is lexically and syntactically complex, implicit assumptions matter, and reasoning must distinguish contraries from contradictions.

The benchmark’s primary role is twofold. First, it serves as an evaluation under abstraction and ambiguity: models must track philosophical arguments, opposing stances, and context-dependent norms rather than merely manipulate symbolic rules on semantically simple content. Second, it functions as a stress test for semiotic-grounded reasoning, because it was paired with LogicAgent, a framework that explicitly leverages Greimas’ Semiotic Square and a three-valued decision scheme.

This design yields a more human-like reasoning regime than the source paper attributes to prior logical benchmarks. The target phenomena include abstract concepts such as justice, virtue, knowledge, art, the soul, and the state; nuanced stances expressed through dialogue; and cases in which a conclusion is neither provable nor refutable from the premises and must therefore be labeled Uncertain.

2. Philosophical source and semiotic organization

RepublicQA is derived from Plato’s Republic, described as a foundational text rich in discussions of justice, morality, governance, knowledge, education, the soul, art, and imitation. Its dialogical structure is central to the benchmark: Socratic exchanges generate claims, counterclaims, and reformulations that naturally instantiate opposed positions.

The benchmark’s conceptual organization is tightly associated with Greimas’ Semiotic Square, which the source paper restates in four positions:

• $S_1$: primary proposition
• $S_2$: its contrary
• $\lnot S_1$: contradictory of $S_1$
• $\lnot S_2$: contradictory of $S_2$

The paper gives the following formal properties for the square:

$$S_1 \ \bot\ S_2 \quad\text{(cannot both be true, but may both be false)}$$

$S_1 \leftrightarrow \lnot\lnot S_1$

$$S_1 \Rightarrow \lnot S_2 \quad\text{and}\quad S_2 \Rightarrow \lnot S_1$$

These relations are important because RepublicQA is meant to contain propositions for which such oppositions are semantically meaningful. The dataset is not mechanically defined as explicit quadruples of $(S_1, S_2, \lnot S_1, \lnot S_2)$ for each item, but its philosophical origin and abstract propositions are chosen so that those structures are frequent and useful. A plausible implication is that RepublicQA evaluates not only truth-functional competence, but also whether a model can discriminate between propositions that are strict negations and propositions that are merely opposed.

3. Construction, formalization, and annotation

RepublicQA is described as manual and curated, rather than template-driven. Its construction pipeline comprises source extraction from philosophically salient passages of The Republic; extraction of contextual premises and a target philosophical statement; translation of premises into first-order logic (FOL); formulation of a natural-language question of the form “Is the statement ‘X’ correct?”; assignment of one of the three labels; and quality control through double annotation and formal checking.

The benchmark uses a benchmark-agnostic FOL mapping in which entities become unary predicates such as $S_2$0, actions and relations become $S_2$1-ary predicates, roles become unary predicates such as $S_2$2, and evaluative properties become predicates over actions or states such as $S_2$3 or $S_2$4. A CFG-based parser using NLTK is used to enforce syntactic correctness of FOL formulas.

The annotation protocol is explicitly human-centered. Contexts and propositions are extracted manually, with double annotation by two graduate students and cross-validation for logical and semantic consistency. The paper also reports full FOL checking through CFG validation and internal consistency checks on premise–conclusion relations. Additional semantic checking is used in experiments, but the benchmark’s core ground truth is human-defined.

A representative case concerns the question “Is the statement ‘The just man turns out to be a thief’ correct?” The example illustrates several design principles at once: the proposition is morally loaded, the context distinguishes ability from actual moral behavior, and the judgment requires interpretation of Plato’s argumentative stance rather than mere lexical matching. In the LogicAgent pipeline, this proposition is then embedded in a semiotic structure, but the raw benchmark itself provides only context + question + gold label.

4. Task format, logical setting, and dataset profile

Each RepublicQA instance consists of a long natural-language context, a natural-language question centered on a target philosophical proposition, and a label drawn from {True, False, Uncertain}. The task definition is: given premises $S_2$5 and a proposition $S_2$6, choose one of the three labels. The benchmark itself provides context, question, and gold label; FOL representations and semiotic-square structures are generated internally by LogicAgent at inference time rather than distributed as raw benchmark fields.

The logical setting is FOL with standard quantifiers and connectives, including $S_2$7, $S_2$8, $S_2$9, $\lnot S_1$0, $\lnot S_1$1, $\lnot S_1$2, $\lnot S_1$3, and $\lnot S_1$4. The contexts frequently require multi-hop reasoning. The paper reports an average explanation length of 6.10 steps, which suggests that even short questions often demand several chained inferences.

Property	Value
Total samples	200
Unique topics	58
Average context length	637.42 tokens
Average proposition length	166.42 tokens
Average explanation steps	6.10
Explanation completeness	100%
FKGL	11.94
TTR	0.685
MTLD	74.81
UBR	0.929
Vocabulary size	2,083 unique terms

Several of these quantities are used to operationalize semantic richness. The paper reports that RepublicQA has a relatively high proportion of “Uncertain” answers (37.5%), interpreting this as a reflection of the nuanced nature of philosophical reasoning. It also reports high lexical diversity and low repetition, with UBR = 0.929, which is presented as evidence that the dataset is resistant to fixed-pattern or template-fitting strategies.

The reading-difficulty measure is the Flesch–Kincaid Grade Level, reported with the formula

$\lnot S_1$5

In context, the reported FKGL = 11.94 is used to characterize the benchmark as college level.

5. Reasoning demands and empirical behavior

RepublicQA is explicitly constructed to combine logical complexity and semantic complexity. On the logical side, the paper emphasizes multi-step deductions, quantification over individuals, actions, and properties, conditionals, and chained causal or normative reasoning. On the semantic side, it emphasizes abstract concepts such as justice, virtue, soul, knowledge, and art; competing moral theories; implicit assumptions; and ambiguous or context-sensitive propositions that cannot be settled without careful reading.

The benchmark therefore requires several distinct modes of reasoning. One is multi-hop deduction, reflected in the mean reasoning length of 6.10 steps. Another is reasoning under ambiguity, where the correct output is often Uncertain because the premises neither entail nor contradict the target proposition. A third is handling opposing stances, since many questions restate one side of a Platonic dialogue and the model must determine whether that stance is actually supported by the provided premises. A fourth is discriminating contraries from contradictions. A fifth is managing existential import in a three-valued setting: vacuous statements are not simply accepted as trivially true when the relevant antecedent is unsupported.

The paper also describes six canonical logical patterns used in LogicAgent’s construction of contraries and contradictories around RepublicQA propositions. These patterns cover universal claims, conjunctions, biconditionals, existential claims, implications, and disjunctions, with some patterns marked as Strict and others as Conditional under constraints such as $\lnot S_1$6, $\lnot S_1$7, or $\lnot S_1$8. Although these rules are used by LogicAgent rather than by human annotators during dataset creation, they indicate the logical shapes that RepublicQA is intended to exercise.

On the benchmark itself, evaluation is performed by straightforward accuracy over the three labels:

$\lnot S_1$9

The reported RepublicQA results are as follows (Zhang et al., 29 Sep 2025):

Method	Qwen2.5-32B	GPT-4o
Naive	68.50	74.00
CoT	72.00	75.00
Logic-LM	70.00	73.50
SymCoT	76.00	80.50
Aristotle	74.50	82.50
LogicAgent	82.50	87.00

In the accompanying discussion, LogicAgent is reported to achieve 84.75 average accuracy on RepublicQA, with a 6.25% average gain over strong baselines. The paper further notes that Logic-LM performs comparably to the naive baseline, which is taken as evidence that tool-augmented symbolic reasoning alone brings little advantage when symbolic and semantic ambiguity are both high. This suggests that RepublicQA is specifically difficult for methods that rely on formal deduction without equally strong handling of abstract language and stance-sensitive interpretation.

6. Comparative position, limitations, and prospective extensions

RepublicQA is compared against four commonly used logical reasoning benchmarks: ProntoQA, ProofWriter, FOLIO, and ProverQA. In that comparison, its distinctive features are semantic richness and abstractness, a large Uncertain fraction, a high prevalence of contrary cases, and resistance to template fitting. The paper reports that approximately 73% of RepublicQA instances involve meaningful contraries, compared with lower proportions in FOLIO and ProverQA and almost none in ProntoQA and ProofWriter. It also reports topic modeling results spanning at least five major philosophical clusters: political philosophy, individual psychology, metaphysics/epistemology, art/reality, and ethics/justice.

The benchmark’s limitations are equally explicit. RepublicQA is based entirely on Plato’s Republic, which introduces a clear domain focus and philosophical bias. Its labels align with the argumentative structure of that text rather than with contemporary philosophical consensus or alternative traditions. Its scale is also limited: 200 samples is small relative to large synthetic benchmarks, even if each item is complex. In logical terms, the benchmark is evaluated in FOL with additional existential import checks and a three-valued label scheme, rather than in modal, deontic, or paraconsistent frameworks that may also be relevant to philosophical reasoning. Finally, annotation retains an irreducible element of subjectivity, because judgments about Platonic propositions are interpretive even under double annotation and formal validation.

The source paper also implies several future directions: extending the construction approach to other philosophical corpora and traditions; enriching the opposition structure beyond contraries and contradictions; scaling the dataset while maintaining quality; studying dataset-specific reasoning complexity thresholds; and publicly releasing both the benchmark and the code for dataset construction. A plausible implication is that RepublicQA is best understood not only as a finished benchmark, but also as a methodological proposal for evaluating symbolic–semantic alignment under ambiguity in domains where language, interpretation, and logic are inseparable.