IN22 Benchmark Evaluation

• IN22 benchmark comprises two evaluations: one for multilingual machine translation across English and 22 Indian languages, and another for inductive theorem proving using OEIS-based conjectures.
• It employs robust methodologies, including BLEU/chrF metrics for MT and SMT-LIB encoding with paired, synthesized programs for theorem proving.
• The benchmark drives advances in low-resource language processing and automated reasoning, highlighting practical challenges and opportunities for future expansions.

The term IN22 refers to two distinct and influential benchmarks in contemporary computational research: (1) the IN22 benchmark for multilingual machine translation (MT) involving English and the 22 scheduled Indian languages, and (2) the “IN22” benchmark for inductive theorem proving, which encompasses 29,687 conjectures over OEIS integer sequences. Both benchmarks have become essential for evaluating LLMs and automated reasoning systems in their respective domains. This article describes both benchmarks in detail, emphasizing design, methodology, empirical findings, and their ongoing impact.

1. IN22 Benchmark for Multilingual Machine Translation

Overview and Motivation

IN22 evaluates the translation capabilities of MT systems or LLMs for English ↔ 22 scheduled Indian languages, addressing both general-domain (encyclopedic/news) and conversational-domain translation. The benchmark was introduced to quantify progress in Multilingual MT for low- and mid-resource Indian languages, which remain underrepresented in mainstream NLP model pre-training (Mujadia et al., 2023, Brahma et al., 2024).

Dataset Structure and Languages

IN22 comprises only test splits; no official train/dev sets are released. All test sentences are professionally translated and sentence-aligned. Two key subsets are provided:

• IN22_conversational_test (“IN22_conv”): 1,502 parallel sentences (per direction) of conversational, day-to-day usage.
• IN22_general_test (“IN22_gen”): 1,023 parallel sentences (per direction) of general-domain content.

The benchmark covers the following languages:

Family	Languages (ISO codes)
Indo-Aryan	Assamese, Bangla, Bodo, Dogri, Konkani, Gujarati, Hindi, Kashmiri, Maithili,
	Marathi, Nepali, Odia, Punjabi, Sanskrit, Santali, Sindhi, Urdu
Dravidian	Kannada, Malayalam, Tamil, Telugu
Sino-Tibetan	Meitei (Manipuri)

The primary training resource for MT models is the BPCC-Human subset (≈2.2M English-Indic pairs), but this is distinct from the IN22 test data.

Evaluation Protocol and Metrics

Task directions include both English→Indic and Indic→English translation. Evaluated regimes span:

• Zero-shot: Raw LLM prompting with generic translation instructions.
• In-Context Learning (ICL): Single example demonstration per pair.
• Fine-tuned: LoRA parameter-efficient adaptation, full-parameter fine-tuning (FSDP), or a two-stage protocol (full fine-tune followed by LoRA).

Two principal metrics are used:

• BLEU: Precision-oriented n-gram overlap with brevity penalty (per [Papineni et al., 2002]):

$$\text{BLEU} = \text{BP} \times \exp\left(\sum_{n=1}^{N} w_n \log p_n\right)$$

• chrF / chrF++: Character n-gram F-score (with $\beta=2$ and word n-gram backoff for ++ variant).

Main Results and Model Configurations

The LLaMA-2–13B LLM, tuned in a two-stage regime (full fine-tune then LoRA), achieves state-of-the-art performance on IN22:

Direction	Test Set	BLEU	chrF
En→Indic	IN22_conv	13.42	43.98
	IN22_gen	15.93	46.99
Indic→En	IN22_conv	14.03	36.71
	IN22_gen	16.65	40.44

Comparable systems, such as NLIP_Lab–IITH’s “IndicRASP-Seed,” evaluated on IN22 and comparable test sets, further validate the robustness and competitiveness of IN22 as a multilingual MT benchmark (Brahma et al., 2024).

Analysis and Recommendations

• Fine-tuning Regimes: Raw LLMs and one-shot ICL regimes fall well below conventional MT systems (BLEU < 5). Adapter-based fine-tuning (LoRA) provides significant gains ($\text{BLEU}\sim6$–$10$), but two-stage tuning approaches or matches the best commercial MT models for many languages.
• Underrepresented Languages: Syntax-rich, low-resource languages with unique scripts not covered during LLM pre-training benefit most from IN22-specific adaptation, yet their absolute BLEU values remain lowest.
• Bilingual vs. Multilingual Training: Multilingual fine-tuning using a single model across 22 directions significantly outperforms separate bilingual models.

Suggested directions for advancing IN22-based MT include expanding script/vocabulary coverage in pre-training, releasing official train/dev splits, augmentation with new domains (health, law), and interface with retrieval-augmented or monolingual techniques (Mujadia et al., 2023).

2. IN22 Benchmark for Inductive Theorem Proving

Definition and Rationale

The second IN22 benchmark (from (Gauthier et al., 2023)) is an extensive SMT-LIB repository designed to evaluate the performance of automated inductive theorem provers. Each problem asserts the equivalence of two distinct, synthesized programs that generate the same OEIS sequence.

Formally, given an integer sequence $(s_0, s_1, ..., s_n)$ and two programs $P_{\text{Small}}$, $P_{\text{Fast}}$, the benchmark asserts:

$$\forall x \in \mathbb{N}. \ f_{P_{Small}}(x,0) = f_{P_{Fast}}(x,0)$$

after verifying equivalence on $x = 0...100$.

Programming Language Specification

Programs are constructed in a minimalist, recursion-oriented DSL:

• Terminals: $0, 1, 2, X, Y$
• Arithmetic Ops: $\beta=2$0
• Conditional: $\beta=2$1
• Recursion/Loops: $\beta=2$2, $\beta=2$3 (mutual), $\beta=2$4

Operators are first-order except for loop bodies, which may be higher-order.

Benchmark Construction and Stratification

• Synthesis Loop: A tree-neural network and learning-guided search generate both “smallest” and “fastest” programs over 29687 OEIS sequences.
• Conjecture Generation: Paired programs with divergent syntax but matching output define each conjecture; coverage is validated empirically.
• SMT-LIB encoding: All problems are encoded as unsatisfiable queries negating the target equality.

Problems are stratified using both syntactic and semantic filters to estimate inductive difficulty:

Filter	Problems	Criterion
Syntactic	23163	Loops with bounds depending on X; at least one body depends on X/Y
Semantic	16197	Output with no bounded-cycle tail over $\beta=2$5

Difficulty is empirically indexed by the problem subset solved by Z3 (no explicit induction), Vampire, and CVC5 (both with induction).

Representative Problems

Examples include equivalence of arithmetic and recursive formulations for triangular numbers, sum of cubes, Fibonacci numbers, binary exponentiation, and double factorials. For instance, the triangular number conjecture:

• Small: $\beta=2$6
• Fast: $\beta=2$7
• Conjecture: $\beta=2$8

Evaluation Protocol and Baseline Results

Solver	No Filter (29,687)	Syntactic (23,163)	Semantic (16,197)
Z3	4,757 (16.0%)	487 (2.1%)	7 (0.04%)
Vampire	2,195 (7.4%)	278 (1.2%)	83 (0.5%)
CVC5	2,428 (8.2%)	893 (3.9%)	504 (3.1%)

Modifying queries to include "successor" constraints (C1: enforce equality for $\beta=2$9 and $\text{BLEU}\sim6$0) markedly increases coverage for inductive provers (e.g., CVC5 coverage on semantic subset rises to 11.1%).

CVC5 (with strengthened conjectures) and Vampire (with tailored induction) show complementary strengths in solving deeply inductive cases. Z3 remains best only for non-inductive or bounded-loop problems (Gauthier et al., 2023).

3. Comparative Significance of the Two IN22 Benchmarks

The two IN22 benchmarks function in different domains: MT and inductive automated reasoning. The former provides the principal yardstick for LLM-based translation with a focus on South Asian linguistic diversity and cross-domain generalization. The latter enforces strong structural and semantic diversity for algorithmic and inductive reasoning, foregrounding both proof search strategies and advances in automated synthesis.

Both benchmarks set high standards for algorithmic generalization, diversity, and coverage, promoting rigorous evaluation under multilingual or mathematical abstraction constraints.

4. Challenges, Limitations, and Recommendations

Multilingual MT (IN22)

• Resource Imbalance: Coverage is weakest for low-resource and script-divergent languages. Poor sub-word vocabularies impede adaptation, especially for Indic→English translation.
• Benchmark Completeness: Lack of official training/development sets prevents end-to-end reproducibility.
• Extension Needs: Domain coverage and Indian→Indian translation directions are underexplored.

Inductive Proving (IN22)

• Inductive Bottleneck: Most semantically filtered cases are beyond current non-inductive solvers.
• Conjecture Strengthening: Adding successor/pair constraints is an effective mitigation for difficult instances.
• Benchmark Utility: The programmatic DSL and paired benchmarks reveal both the promise and gap in synthesizing and verifying structural properties across a broad mathematical space.

5. Prospects and Future Directions

IN22 (MT) is poised for expansion through:

• Inclusion of additional domains (e.g., law, health), extended Indian→Indian translation directions, and larger pre-training vocabularies.
• Release of new train/dev/test splits and richer metadata.

For the inductive theorem proving IN22:

• Progress in learning-guided synthesis and automated induction scheduling is required for substantial further solver coverage.
• Incorporation of generalized conjecture-strengthening protocols and new neural-symbolic methods could boost success rates.

A plausible implication is that benchmarks structured with paired syntactic/semantic abstractions, as in both IN22 testbeds, provide powerful levers for both qualitative and quantitative progress in NLP and automated reasoning.