LingBench++: Linguistic Benchmark for LLMs

• The paper's main contribution is introducing a benchmark that integrates expert-verified reasoning traces and a dual-objective scoring system to assess LLM performance on complex linguistic puzzles.
• The framework offers a novel multi-agent architecture that combines solver, grammar, and aggregator agents for iterative hypothesis testing and enhanced accuracy.
• LingBench++ incorporates diverse linguistic tasks, rich typological metadata, and stepwise metrics to diagnose internal reasoning and pinpoint failure modes.

Searching arXiv for the specified paper and closely related context. LingBench++ is a linguistically-informed benchmark and reasoning framework for evaluating LLMs on complex linguistic tasks inspired by the International Linguistics Olympiad (IOL). It combines a fully text-encoded corpus of IOL problems, a two-tiered scoring scheme for final solutions and rule explanations, expert-verified gold reasoning references, and a multi-agent architecture that integrates grammatical knowledge retrieval, tool-augmented reasoning, and deliberate hypothesis testing. Its stated aim is to advance linguistically grounded, culturally informed, and cognitively plausible reasoning in LLMs, while moving evaluation beyond final-answer accuracy alone (Lian et al., 22 Jul 2025).

1. Corpus design and problem format

The core dataset consists of 96 full IOL problems and 225 sub-problems drawn from IOL archives (2003–2024). Each problem is transcribed into four parts—Introduction, Corpus, Tasks, and Notes—exactly as in the IOL, but fully text-encoded. This preserves the original structure of the source material while making it suitable for LLM evaluation.

The benchmark spans the full range of IOL problem styles: Rosetta, Match-up, Monolingual, Pattern, Computational, and Text. These formats correspond to familiar IOL rubrics such as fill-in tables, translation, correspondence, and rule description. Multimodal elements, including non-Latin scripts, IPA, and diagrams, are encoded in text or simplified via OCR. The resulting benchmark is therefore not limited to plain translation-like tasks; it is organized around the broader class of self-contained micro-data puzzles characteristic of olympiad linguistics (Lian et al., 22 Jul 2025).

A central design property is that the problems are described as self-contained and drawn from unfamiliar languages, with no outside knowledge assumed. This distinguishes the benchmark from settings in which broad world knowledge is necessary for task completion. A plausible implication is that LingBench++ is intended to isolate reasoning over structured linguistic evidence rather than retrieval of general background facts.

2. Task taxonomy and linguistic scope

LingBench++ organizes its problems by a set of linguistic task categories labeled “Subjects”: Compounding, Morphology, Numbers, Phonology & Phonetics, Semantics, Syntax, and Writing System. The paper notes that these reflect classic divisions of IOL puzzles; for example, Rosetta problems often focus on morphology + syntax, whereas Pattern tasks are described as heavy on phonology.

The benchmark also uses a typological metadata schema referred to as UKLO classification. Under this scheme, each problem is tagged along three axes. Subject ranges over the seven linguistic categories above. Type ranges over Rosetta, Match-up, Monolingual, Pattern, Computational, and Text. Theme ranges over Classical, Comparative, Encrypted, Kinship, Maps, Mystery, MFL, Senses & Feelings, Stories, Poetry, and No Theme. For each language, the metadata further maps ISO 639-3 code to Glottocode, Language Family, and estimated number of speakers (Lian et al., 22 Jul 2025).

The paper gives an explicit example annotation: Sub-problem 2 | Subject=Numbers | Type=Pattern | Language=Egyptian Arabic | Speakers=68 M | glottocode=egyp1253 | Family=Semitic. This exemplifies the benchmark’s attempt to connect individual reasoning instances to typological and sociolinguistic descriptors. Such metadata also underpins the reported cross-family performance analyses.

3. Scoring model and stepwise evaluation

LingBench++ defines the final solution score for each sub-problem as a weighted combination of answer correctness and rule explanation quality:

$$S_{\rm final} = w_{\rm ans} S_{\rm ans} + w_{\rm rule} S_{\rm rule},$$

with default weights $w_{\rm ans} = w_{\rm rule} = 0.5$. The answer credit $S_{\rm ans}$ uses exact-match or, when multiple translations are valid, a small set of normalized variants. The rule explanation credit $S_{\rm rule}$ is based on how many gold-standard rules in a hand-crafted checklist are correctly mentioned. This makes the benchmark explicitly dual-objective: correct outputs matter, but so does articulated rule induction (Lian et al., 22 Jul 2025).

Beyond the final score, the benchmark introduces a stepwise evaluation protocol called “Check-of-Thought.” For each problem, the authors first prompted Gemini-2.5-Pro with a linguistics-expert template to generate a chain of thought. These chains were then manually verified/refined by seven linguists + three IOL contestants into gold reasoning references (GRR). The GRR functions as an expert-aligned reference trajectory for evaluating intermediate reasoning rather than just terminal predictions.

The protocol comprises five complementary evaluation dimensions. Under Information Extraction & Structuring, it uses SLVS (Stepwise Logical Validity Score) and ISC (Information Structuring Completeness). Under Hypothesis Generation & Rule Induction, it uses HGA (Hypothesis Generation Adequacy), RIC (Rule Induction Coverage), and IJC (Inference Justification Coverage). Under Completeness & Coverage, it uses CCS (Chain-of-Thought Continuity Score) and SCR (Subtask Coverage Rate). Under Logical Deduction & Internal Consistency, it uses SLVS$_{\rm internal}$ and ACR (Application Consistency Rate). Under Contradiction Handling & Self-Revision, it uses CDA (Contradiction Detection Ability) and ETS (Error Traceability Score).

Each metric is rated on a 1–5 scale, where 1=poor, 3=acceptable, 5=excellent. A judge LLM is prompted with an in-paper system prompt to compare the target chain against the GRR and issue scores. The paper provides an example result excerpt in which SLVS = 2, ISC = 2, HGA = 1, RIC = 1, … ETS = 1. This scoring regime is designed to expose distinct failure modes that are invisible under pure exact-match evaluation.

4. Cross-cultural and typological coverage

The benchmark’s coverage extends to 90 + low-resource and endangered languages, including examples from Austronesian, Atlantic-Congo, Australian, Papuan, Niger-Congo, and other language groupings. It also includes rare scripts, specifically Tifinagh, IPA, and pictographs, together with associated cultural contexts (Lian et al., 22 Jul 2025).

The paper reports that Figures show that Syntax & Morphology dominate IOL, and that Rosetta type accounts for ~44 problems. These distributional statistics are important because they delimit the empirical profile of the benchmark: LingBench++ is broad in typological reach, but its problem inventory still reflects the historical preferences of olympiad linguistics. This suggests that benchmark performance should be interpreted in relation to the composition of the underlying IOL archives rather than as a uniform sample over linguistic phenomena.

The typological tags are not merely descriptive. The paper states that cross-family performance analyses exploit these tags to reveal where LLMs succeed/fail. In this sense, typological metadata serves as an analysis scaffold for evaluating systematic generalization across families, scripts, and puzzle types, rather than only as auxiliary cataloging information.

5. Multi-agent reasoning framework

LingBench++ pairs its benchmark with a multi-agent architecture composed of three interacting agent types. The Solver Agent generates one or more candidate solution chains or hypotheses from the raw problem. The Grammar Agent retrieves relevant reference-grammar excerpts via RAG over 1 100+ grammars, each chunked at 256 tokens. The Aggregator Agent implements a Mixture-of-Agents strategy in which, at each round, multiple solver chains are pooled and combined, with the final answer extracted from the aggregator’s best chain (Lian et al., 22 Jul 2025).

The grammar retrieval pipeline is specified in concrete terms. Texts are OCR’d (if needed), chunked, embedded (Qwen-Embedding-4B), and indexed in LanceDB for hybrid full-text + vector search by Glottocode, language family, etc. This makes the retrieval component sensitive not only to lexical similarity but also to typological metadata. The paper further states that, at each hop, agents can call the Grammar Agent to confirm a morphosyntactic rule “X→Y” by searching the reference grammar.

The architecture is summarized by a pseudocode sketch:

$w_{\rm ans} = w_{\rm rule} = 0.5$5

The paper also gives a higher-level round description: for $r \in 1 \ldots R$, each of $N$ Solver Agents proposes a chain $C_{r,i}$, after which the Aggregator pools $\{C_{r,i}\}$ and may optionally produce its own chain $C_{r,\mathrm{agg}}$. A plausible implication is that LingBench++ treats linguistic problem solving as an iterative hypothesis-testing process rather than a single-pass generation task.

6. Empirical performance and interpretability

The reported experiments compare single-pass baselines, grammar-augmented configurations, and Mixture-of-Agents variants. In the baseline setting, OpenAI-o4-mini (single-pass) achieves avg score = 0.193, with 61/89 problems scored <0.25. Gemini-2.5-pro (single-pass) achieves avg score = 0.381, with 38/95 <0.25. A “Guided” prompt that adds an IOL training guide yields avg score = 0.307, which is reported as worse than raw Gemini (Lian et al., 22 Jul 2025).

Adding the Grammar Agent to Gemini-2.5-pro yields avg score = 0.387 on 77 problems, described as a slight lift over raw Gemini. The larger gains come from the Mixture-of-Agents setup. With 2 rounds and $w_{\rm ans} = w_{\rm rule} = 0.5$0 Solver Agents, using Gemini-2.5-pro as final aggregator, the system reaches avg = 0.419, with 39/94 problems $w_{\rm ans} = w_{\rm rule} = 0.5$1. With 3 rounds, it reaches avg = 0.425, identified as the best overall. OpenAI-o4-mini under MoA also improves from 0.312 to 0.350, but remains below Gemini.

The paper quantifies the main accuracy gain as a relative gain from single-pass to MoA-3 for Gemini of $w_{\rm ans} = w_{\rm rule} = 0.5$2. By comparison, the Grammar Agent alone yields a +1.6 points lift (0.387–0.381). These results support the paper’s stated claim that models equipped with external knowledge sources and iterative reasoning outperform single-pass approaches on these tasks.

Interpretability is presented as a second major outcome. Because LingBench++ provides structured reasoning traces aligned against GRR, it allows evaluators to pinpoint failure modes, such as low RIC or CDA. The Check-of-Thought metrics therefore function as a fine-grained diagnostic profile beyond “got it right or wrong.” This makes the framework relevant not only for benchmarking aggregate accuracy but also for analyzing the internal structure and revision behavior of model reasoning.

7. Position within LLM evaluation

LingBench++ is explicitly framed as differing from prior benchmarks that focus solely on final answer accuracy. Its combination of expert-verified chains of thought, stepwise evaluation, typological metadata, and agentic reasoning with grammar retrieval makes it both a benchmark and a reasoning framework (Lian et al., 22 Jul 2025).

Two clarifications follow from the paper’s design choices. First, although the problems are linguistically and culturally diverse, they are still IOL-inspired self-contained micro-data puzzles; the benchmark is therefore centered on structured inference from bounded evidence rather than unconstrained language understanding. Second, the framework does not equate higher answer accuracy with better reasoning in a simplistic sense: the separate scoring of $w_{\rm ans} = w_{\rm rule} = 0.5$3 and $w_{\rm ans} = w_{\rm rule} = 0.5$4, together with metrics such as HGA, RIC, CDA, and ETS, reflects an explicit distinction between obtaining a correct answer and exhibiting a coherent, justified, and self-corrective solution process.

In summary, LingBench++ integrates a richly annotated IOL problem set, a final-answer plus rule-explanation scoring model, a five-dimension Check-of-Thought protocol, and a proof-of-concept multi-agent reasoning stack. The benchmark’s main contribution is to operationalize evaluation of LLMs on linguistically grounded, cross-cultural, multi-step inference while preserving diagnostic access to intermediate reasoning structure.