DEPART: DEcomposing PARiTy across Multilingual LLMs

Abstract: Multilingual LLMs (mLLMs) leaderboards report per-language accuracy but rarely explain why disparities emerge, leaving systemic biases unattributed and offering practitioners no actionable levers. We first establish that these gaps are systematic rather than artifacts of sampling noise via distribution-free Friedman and Kruskal--Wallis tests, then introduce a two-step Bayesian hierarchical framework that decomposes multilingual performance variance into interpretable components. First, isolating the variance attributable to language identity, we show that observable language features (script, family, typological distance) explain $R^2_{\text{ling}} = 79\%$ of this variance on understanding tasks and $92\%$ on reasoning, with a model's internal representational similarity to English emerging as the dominant predictor across both task buckets. Second, decomposing the full (model$\times$benchmark$\times$language) cube, we find that NLU and reasoning have fundamentally divergent variance profiles: model identity dominates understanding ($66.7\%$ of variance), whereas the benchmark$\times$model interaction dominates reasoning ($46.3\%$). Together these results recast multilingual evaluation from passive performance mapping into an explainable, diagnostic framework with concrete levers for targeting the root drivers of language disparity.

• The paper introduces a two-step Bayesian hierarchical mixed-effects model to decompose cross-lingual variance and identify structural drivers of performance gaps.
• It demonstrates that internal representational similarity to English significantly boosts accuracy, explaining 79% to 92% of variance in multilingual tasks.
• The study highlights actionable interventions targeting language features and tokenizer fertility to enhance equitable performance across low- and high-resource languages.

Decomposing Parity Across Multilingual LLMs: An Expert Appraisal

Motivation and Context

The performance of multilingual LLMs (mLLMs) consistently varies across languages, with low-resource languages suffering disproportionately from reduced accuracy and reasoning capacities. Conventional leaderboards often report per-language performance, Gini coefficients, or English-vs-X deltas, but rarely provide actionable explanations for these disparities or identify their systemic roots. This paper introduces a diagnostic, explainable framework for variance decomposition, yielding both predictive insight and concrete levers for intervention targeting structural drivers of cross-lingual inequality.

Methodological Overview

The authors evaluate seven open-weight mLLMs, spanning 3.5B–122B parameters and covering diverse tokenizer architectures, across 15 multilingual benchmarks and 63 languages. The evaluation suite is partitioned into two primary task buckets: multilingual understanding (NLU) and multilingual reasoning, encompassing both parallel and non-parallel datasets. The principal analytical technique is a two-step Bayesian hierarchical mixed-effects model, calibrated with non-parametric validation. This model is constructed to quantify the interplay of language identity, model architecture, benchmark/task specificity, and their random interactions, allowing the attribution of variance to observable features and residuals.

The first stage isolates variance attributed to language identity, explained via structural features such as script, language family, typological distance to English, resource class, tokenizer fertility, and internal representation alignment. The second stage expands this to the full (model × benchmark × language) cube, decomposing total variance into constituent components for each task bucket.

Validation of Systematic Disparity

Distribution-free Friedman and Kruskal-Wallis tests establish that cross-lingual performance gaps are systematic, rejecting the null hypothesis of equal language ranks across all benchmarks with high inter-model agreement (median Kendall’s W ≈ 0.75), and confirming that disparity tracks resource tier monotonically. Mean accuracy nearly doubles from the lowest to the highest resource classes, and all pairwise tier contrasts are statistically significant, validating the structured nature of observed gaps.

Hierarchical Variance Decomposition

Language-Level Predictive Features

On NLU tasks, language-level structural features explain $R_{\text{ling}} = 79\%$ of cross-language variance; on reasoning tasks, $R_{\text{ling}} = 92\%$ is explained. The most dominant predictor is internal representational similarity (CKA alignment) to English: a one-SD increase yields $+6.5$ accuracy points (NLU) and $+9.4$ points (Reasoning). Resource class is also independently significant for Reasoning. Typological and tokenizer features are collinear with categorical variables or random intercepts, contributing to joint but not individual predictive signal.

Full-Design Variance Structure

Variance decomposition reveals distinct structural characteristics for the two buckets:

• NLU (Understanding): Model identity accounts for $66.7\%$ of variance, with modest benchmark-model interaction ($13.6\%$), indicating consistency across knowledge-centric tasks.
• Reasoning: Benchmark-model interaction dominates ($46.3\%$), exceeding the model main effect ($12.7\%$), confirming that each task probes disparate capabilities not pooled across benchmarks.

Model-language specific residuals persist at $7$-$9\%$, indicating unobserved effects primarily attributable to pretraining data mixture and quality.

Implications for Multilingual Model Development

The decomposition reframes multilingual evaluation from passive metric reporting to an actionable diagnostic regime. Large portions of the cross-lingual gap are predictable and structural; interventions targeting language-level features, tokenizer fertility, and cross-lingual geometric alignment can be directly evaluated and optimized prior to final checkpoint selection. Critical practical implication: reporting variance components ($R_{\text{ling}} = 92\%$0, $R_{\text{ling}} = 92\%$1) alongside per-language scores identifies whether improvements stem from genuine capability enhancement or merely structural alignment (e.g., increased similarity to English).

Representation alignment to English is robust across task buckets and NLG generation, offering an inexpensive proxy for ranking multilingual checkpoints during pretraining and alignment, independent of actual benchmark inference. Tokenizer fertility affects reasoning performance heterogeneously, with substantial variance across tokenizer families; per-model random slopes on fertility are mandatory for accurate attribution.

The model-language residual sets the ceiling for feature-driven mitigation, highlighting the role of pretraining-data curation as the next frontier. Model releases that disclose mixture ratios and data quality could enable direct targeting of these residuals, further closing the gap.

Theoretical Considerations

The split in variance structure between understanding and reasoning tasks indicates fundamental differences in how multilingual cognitive processes are encoded—model identity is more salient for knowledge mapping, while task-specific skills dominate reasoning. Aggregating all reasoning benchmarks obscures substantive differences, necessitating per-benchmark reporting. The correlation between cross-lingual headroom and English-aligned geometric similarity is consistent with findings that decoder LLMs route multilingual computation through English-centric representations (2605.28163).

These results converge with independent signals on tokenizer-induced fairness [Ahuja et al., 2023], representation alignment [Li et al., 2025], and systemic disparity [Blasi et al., 2022], but for the first time jointly quantify their relative contributions and residual ceilings in a calibrated, hierarchical model.

Prospects for Future Research

Future developments should focus on:

1. Controlled experiments altering pretraining data mix and alignment objectives to validate causal claims underlying representation alignment and residuals.
2. Expansion of benchmarks covering broader scripts and language families to resolve collinear feature effects, particularly for NLG tasks.
3. Routine reporting of model-language random slopes and full variance components in leaderboard settings.
4. Exploration of alignment methods that raise cross-lingual representational similarity without increasing English dependence, potentially improving equitable performance for low-resource languages.

Conclusion

This work establishes a formal, explainable foundation for diagnosing and mitigating cross-lingual disparities in mLLMs. Structural language features and internal representation alignment can explain the majority of observed performance gaps, with systematic evaluation revealing concrete intervention points for pretraining, alignment, and tokenization. Residuals set practical ceilings, focusing future efforts on transparent data curation. The diagnostic approach advocated here is essential for socially equitable, scientifically robust multilingual LLM development.