Multilingual-IRT: A Psychometric Evaluation

• Multilingual-IRT is a psychometric framework that extends IRT to evaluate parallel multilingual benchmarks by decomposing overall ability and language-specific residuals.
• The model introduces per-language difficulty deviations and split discriminability to separate content effects from translation and cultural perturbations.
• Empirical results show 11–16% lower error rates and improved diagnostics for translation errors and culture-specific item recovery in multilingual evaluations.

Searching arXiv for papers on Multilingual-IRT and closely related usages to ground the article. Multilingual-IRT is a psychometric framework for evaluating LLMs on parallel multilingual benchmarks, where the same item appears in many languages. It extends Item Response Theory with per-language difficulty deviations, split discriminability separating content from language effects, and per-language ability residuals, so that multilingual performance can be decomposed into general capability, language-specific strengths and weaknesses, item-intrinsic difficulty, and language-specific perturbations introduced by translation or cultural localization (Lior et al., 14 Jun 2026).

1. Scope, problem setting, and terminological boundaries

In the strict psychometric sense, Multilingual-IRT addresses a specific setting: a benchmark with $I$ items, $L$ languages, and $J$ models, producing a response tensor $Y_{ijl}$ of size $I \times J \times L$. The motivating claim is that multilingual evaluation has structure that standard benchmark analyses ignore, and that explicitly modeling this structure yields both more efficient evaluation and more meaningful diagnosis of benchmark artifacts (Lior et al., 14 Jun 2026).

The framework is motivated by three problems. First, exhaustive evaluation is expensive because evaluation cost scales roughly linearly with the number of languages. Second, automatic translation introduces errors that can make an item easier or harder, alter the logic of answer choices, or even change the correct answer. Third, some benchmark items conflate general and culture-specific knowledge, so multilingual differences may be overinterpreted as language ability when they partly reflect culturally localized content (Lior et al., 14 Jun 2026).

A recurrent misconception is that any multilingual IRT-adjacent system qualifies as Multilingual-IRT. The package inrep is instead described as a multilingual CAT delivery framework, with a localized Shiny interface, support for multilingual test presentation, UTF-8 and RTL rendering, and deployment features; it does not provide substantive multilingual psychometric methodology for cross-language comparability such as multilingual calibration, linking or equating, DIF testing, invariance analysis, or translation validation (Selva, 20 Jul 2025). Conversely, some papers use nearby acronyms in non-psychometric senses: mFollowIR is a multilingual benchmark for instruction-following in retrieval rather than item response theory (Weller et al., 31 Jan 2025), MLAIRE is a multilingual language-aware information retrieval evaluation protocol (Jang et al., 8 May 2026), IMSAE is relevant only if “IRT” is read as iterative representation transformation/transfer (Shao et al., 12 Jun 2025), and MITT is relevant only if “IRT” is read as initial reasoning transfer (Bajpai et al., 21 May 2025). In that sense, “Multilingual-IRT” names a psychometric model in one literature and an acronymically similar but conceptually different family of multilingual retrieval or representation methods in others.

2. Statistical formulation

The starting point is ordinary 2PL IRT, written as

$p_{ij}=\frac{1}{1+e^{-a_i(\theta_j-b_i)}}$

or equivalently

$\operatorname{logit}(p_{ij}) = a_i(\theta_j-b_i).$

The argument against using this model directly is that translated versions of the same item are not independent. Standard IRT would either treat each language-specific item as unrelated, wasting the parallel structure, or require a multidimensional setup whose axes are not identifiable in a linguistically meaningful way. The paper identifies rotational indeterminacy in standard MIRT as the core obstacle to interpreting one dimension as “general ability” and another as “language-specific residual ability” unless extra structure is imposed in advance (Lior et al., 14 Jun 2026).

The full Multilingual-IRT model is

$Y_{ijl} \sim \operatorname{Bernoulli}(p_{ijl}),$

$$\operatorname{logit}(p_{ijl}) = a_i^{\text{base}} \theta_j + a_i^{\text{lang}} (\varepsilon_j)_l - (b_i + d_{il}).$$

Two sum-to-zero constraints define the decomposition: $$\sum_{l=1}^{L} d_{il} = 0, \qquad \sum_{l=1}^{L} (\varepsilon_j)_l = 0.$$ The per-language residual vector has a cross-lingual prior

$L$0

where $L$1 is a learned correlation matrix (Lior et al., 14 Jun 2026).

This construction separates four terms inside the logit: $L$2 The paper also defines a family of variants called M$L$3. Indep-IRT fits a separate IRT model independently in each language; Parallel-IRT shares item parameters with language-specific difficulty deviations; Coupled-IRT reparameterizes ability as shared plus residual; and Multilingual-IRT adds the split discriminability $L$4 that separates content effects from language effects (Lior et al., 14 Jun 2026).

3. Parameter interpretation and identifiability

The central interpretive move is that each parameter has a multilingual benchmark meaning. The baseline item difficulty $L$5 is the language-averaged difficulty of item $L$6. The language-specific deviation $L$7 measures how much harder or easier the translation or localization in language $L$8 makes that item, so $L$9 is the item difficulty in language $J$0. The scalar $J$1 is the model’s overall ability, while $J$2 is the model’s relative strength or weakness in language $J$3 compared to its own average across languages. The split discriminabilities $J$4 and $J$5 indicate whether an item differentiates models mainly by their overall ability or by their language-specific residuals (Lior et al., 14 Jun 2026).

This parameterization is used to distinguish different kinds of multilingual phenomena. A universally difficult item has large $J$6 without necessarily large $J$7. A language-specific translation artifact appears as unusually large positive $J$8, meaning one language version is much harder than the cross-lingual baseline for that item. Source intrusion often appears as negative $J$9, because retained English terms can make a supposedly translated item easier. A culture-specific item is expected to have relatively large $Y_{ijl}$0 compared to $Y_{ijl}$1, because changing the language modulates how much the item separates models (Lior et al., 14 Jun 2026).

The paper’s identifiability claim is that the asymmetric scalar-plus-residual design avoids the rotational ambiguity of ordinary MIRT. That argument is operationalized by the sum-to-zero constraints on $Y_{ijl}$2 and $Y_{ijl}$3, by the separate loadings $Y_{ijl}$4 and $Y_{ijl}$5, and by the learned cross-lingual covariance structure $Y_{ijl}$6 (Lior et al., 14 Jun 2026). Empirically, the learned residual correlation matrix recovers meaningful linguistic structure, including a Romance block $Y_{ijl}$7, a Slavic block $Y_{ijl}$8, a South Asian block $Y_{ijl}$9, and an African block $I \times J \times L$0, without typological supervision (Lior et al., 14 Jun 2026).

The priors are explicitly structured. The appendix specifies

$I \times J \times L$1

$I \times J \times L$2

$I \times J \times L$3

with centering

$I \times J \times L$4

and

$I \times J \times L$5

Inference is performed with stochastic variational inference in NumPyro, using Adam with learning rate $I \times J \times L$6, 30,000 SVI steps, a rank-10 multivariate normal guide, and 1,000 posterior samples from the fitted guide for posterior means and standard deviations (Lior et al., 14 Jun 2026).

4. Empirical applications and reported evidence

The reported experiments use MMLU-Pro-X, a parallel multilingual extension of MMLU-Pro with 11,829 items per language, 10 answer options per question, and 29 languages, evaluated on 25 LLMs (Lior et al., 14 Jun 2026). The framework is presented as supporting three practical applications: predicting unobserved instances, surfacing candidate translation errors, and recovering culture-specific items.

Application	Signal	Reported evidence
Efficient evaluation	$I \times J \times L$7 from fitted model	11–16% lower binary cross-entropy than the strongest accuracy-based baseline
Translation auditing	$I \times J \times L$8	Detections spread across all 28 non-English languages; no single language > 11%
Culture-specificity recovery	$I \times J \times L$9	26.1% culture-specific items in top 2K versus 24.2% for best accuracy baseline

For missing-entry prediction, the model is fit on a random observed fraction $p_{ij}=\frac{1}{1+e^{-a_i(\theta_j-b_i)}}$0 of the response tensor and used to predict the remaining entries. The headline result is 11–16% lower BCE than the strongest accuracy-based baseline across observation fractions, with smaller but consistent ROC-AUC gains. Most of the predictive improvement comes from sharing item parameters across languages, that is, from the jump from Indep-IRT to Parallel-IRT; the richer components, including $p_{ij}=\frac{1}{1+e^{-a_i(\theta_j-b_i)}}$1 and split discriminability, do not substantially improve missing-entry prediction. BCE improves by only 4.2% when increasing the observed fraction from $p_{ij}=\frac{1}{1+e^{-a_i(\theta_j-b_i)}}$2 to $p_{ij}=\frac{1}{1+e^{-a_i(\theta_j-b_i)}}$3, with most gains already achieved by $p_{ij}=\frac{1}{1+e^{-a_i(\theta_j-b_i)}}$4, leading to the practical conclusion that 60% of instances can be left unseen with negligible loss in prediction quality (Lior et al., 14 Jun 2026).

For translation-error detection, the ranking score is

$p_{ij}=\frac{1}{1+e^{-a_i(\theta_j-b_i)}}$5

Large positive values indicate that an item is much harder than expected in one language with high certainty. The annotation pipeline uses Gemini-2.5-Flash as an LLM judge with severity labels Critical / Minor / None and categories Semantic Shift / Logic Alteration / Source Intrusion / Formatting Failure / None. Human validation of 77 judged cases gave 67.5% strict precision and 79.2% lenient precision, while a three-judge comparison on a 60K-item balanced subset showed Gemini-2.5-Flash agreed with the majority in 86.3% of cases. Accuracy baselines concentrate $p_{ij}=\frac{1}{1+e^{-a_i(\theta_j-b_i)}}$6 of detected critical errors in Wolof, Yoruba, and Zulu, whereas Multilingual-IRT distributes detections across all 28 non-English languages, with no single language exceeding 11% (Lior et al., 14 Jun 2026).

The negative tail,

$p_{ij}=\frac{1}{1+e^{-a_i(\theta_j-b_i)}}$7

is used to find unexpectedly easy item-language pairs. This surfaces Source Intrusion errors—English words left in supposedly translated non-English questions. Among the top 1000 items ranked by $p_{ij}=\frac{1}{1+e^{-a_i(\theta_j-b_i)}}$8, Multilingual-IRT finds 16.8% Source Intrusion errors, compared with 5% for the best accuracy baseline and 4% for a random baseline (Lior et al., 14 Jun 2026).

For culture-specific item recovery, the scoring signal is the ratio

$p_{ij}=\frac{1}{1+e^{-a_i(\theta_j-b_i)}}$9

Items ranked by this ratio are judged by Gemini-2.5-Flash as culturally specific or universal, with six types: Region/Country, Religion/Philosophy, Language-internal, Named-entity, Social-convention, and Universal. The top 2,000 items from the ratio contain 26.1% culture-specific items, compared with 24.2% for the best accuracy-based method and 19.8% for a random baseline. Pairwise Jaccard overlap with accuracy baselines is $\operatorname{logit}(p_{ij}) = a_i(\theta_j-b_i).$0, indicating that the ratio surfaces largely different candidates. The paper also reports that culture-specific items are harder overall, that U.S./U.K.-tied items are easier in English than non-cultural items are $\operatorname{logit}(p_{ij}) = a_i(\theta_j-b_i).$1 vs $\operatorname{logit}(p_{ij}) = a_i(\theta_j-b_i).$2, $\operatorname{logit}(p_{ij}) = a_i(\theta_j-b_i).$3, and that non-English region-tied items are easier in their own region’s language than under random reassignment $\operatorname{logit}(p_{ij}) = a_i(\theta_j-b_i).$4 vs $\operatorname{logit}(p_{ij}) = a_i(\theta_j-b_i).$5, $\operatorname{logit}(p_{ij}) = a_i(\theta_j-b_i).$6 (Lior et al., 14 Jun 2026).

5. Relation to adjacent IRT-based evaluation frameworks

Multilingual-IRT belongs to a broader family of recent attempts to adapt IRT to contemporary model evaluation, but its novelty lies in explicitly modeling parallel multilingual structure. M3IRT extends classical IRT to multimodal LLM evaluation by decomposing both model ability and item difficulty into image-only, text-only, and cross-modal components, and uses the cross-modal parameters to identify shortcut questions and select compact, high-quality benchmark subsets (Uebayashi et al., 3 Mar 2026). The transferable lesson is the decomposition strategy: benchmark validity improves when ability and difficulty are decomposed along the dimensions that actually govern answerability.

IrtNet is a neural multidimensional IRT-style framework for learning compact latent representations of LLM abilities from binary correctness data. Its response model

$\operatorname{logit}(p_{ij}) = a_i(\theta_j-b_i).$7

uses a model ability vector, a query discrimination vector, and a query difficulty scalar, with query parameters generated from semantic embeddings through a dense Mixture-of-Experts network. The experiments are not multilingual, but the framework is relevant as a neural multidimensional 2PL-style backbone that could be extended to multilingual item populations (Chen et al., 1 Oct 2025).

By contrast, inrep should be classified as multilingual infrastructure rather than multilingual psychometric methodology. It provides a comprehensive framework for computerized adaptive testing in R, supports 1PL, 2PL, 3PL, GRM, real-time ability estimation, multiple item selection algorithms, sophisticated stopping criteria, and multilingual UI features such as 40+ languages, right-to-left script rendering, and UTF-8 throughout. However, it does not describe multilingual calibration, linking or equating, DIF testing, invariance analysis, or translation validation, so it supports multilingual administration but not multilingual score comparability in the strict psychometric sense (Selva, 20 Jul 2025).

A plausible implication is that Multilingual-IRT occupies a specific niche within the emerging IRT-for-model-evaluation literature: it is neither a generic MIRT model nor a multilingual deployment layer, but a structured latent-variable model for parallel multilingual benchmarks.

6. Limitations, misconceptions, and neighboring usages

Several limitations are explicit. The framework requires aligned parallel items and does not directly apply to independently sourced monolingual datasets. It uses binary correctness labels $\operatorname{logit}(p_{ij}) = a_i(\theta_j-b_i).$8, so it does not directly incorporate confidence, partial credit, free-form generations, or richer response structure. With only 25 LLMs, some parameters—especially $\operatorname{logit}(p_{ij}) = a_i(\theta_j-b_i).$9 and off-diagonal entries of $Y_{ijl} \sim \operatorname{Bernoulli}(p_{ijl}),$0—are less precisely recovered, and the simulation study shows weaker recovery for those components than for the global ability and difficulty parameters (Lior et al., 14 Jun 2026).

Another misconception is that the model’s improvements are uniform across all use cases. The paper explicitly reports that most of the improvement in missing-entry prediction comes from sharing item parameters across languages, whereas the richer parts of the model matter more for auditing and interpretation than for pure imputation (Lior et al., 14 Jun 2026). This is consistent with the parameter-analysis correlations

$Y_{ijl} \sim \operatorname{Bernoulli}(p_{ijl}),$1

$Y_{ijl} \sim \operatorname{Bernoulli}(p_{ijl}),$2

which show that difficulty and ability mostly reproduce smoothed versions of intuitive accuracy summaries, while discriminability captures signal that accuracy misses (Lior et al., 14 Jun 2026).

A further source of confusion is the acronym itself. In multilingual retrieval, mFollowIR studies instruction-following retrieval across Russian, Chinese, and Persian, evaluates with nDCG@20 and p-MRR, and shows that instruction-following retrieval is materially easier when the instruction is in English than when the model must process non-English instructions directly (Weller et al., 31 Jan 2025). MLAIRE instead measures semantic retrieval accuracy and query-language preference with metrics such as Language Preference Rate and Lang-nDCG, showing that semantically strong retrievers may return correct content in a non-query language while retrievers with stronger query-language preference may retrieve less semantically relevant passages (Jang et al., 8 May 2026). Those frameworks are part of multilingual information retrieval evaluation, not item response theory. Likewise, IMSAE and MITT use “IRT”-adjacent language to denote iterative representation transformation/transfer and initial reasoning transfer, respectively, rather than psychometric latent-trait modeling (Shao et al., 12 Jun 2025, Bajpai et al., 21 May 2025).

In the strict encyclopedia sense, then, Multilingual-IRT refers most precisely to the 2026 extension of 2PL/MIRT for parallel multilingual LLM evaluation, centered on the decomposition

$Y_{ijl} \sim \operatorname{Bernoulli}(p_{ijl}),$3

and designed to support efficient evaluation, translation auditing, and culture-specificity auditing in multilingual benchmark analysis (Lior et al., 14 Jun 2026).