Behavioral and Representational Evidence of Binomial Ordering Preferences in Large Language Models

Abstract: LLMs can readily reproduce conventional expressions, yet their ability to model gradient frequency distributions remains underexplored. We investigate this using linguistic binomials, such as men and women, where both word permutations are grammatically valid but exhibit distinct, cross-linguistic variations in conventionality. We formalize binomial ordering as a distributional alignment problem, and construct a multilingual dataset of 600 binomial pairs across 8 languages. With categorical and distributional metrics, we measure and compare the corpus-derived preferences with model-induced ordering probabilities of 6 open-weight LLMs. While models often behaviorally recover the dominant corpus-preferred order, particularly for strongly conventionalized pairs, they align less well with the exact corpus preference distributions. This suggests that apparent directional order overstates how faithfully LLMs capture the statistical nuances of language use. Sparse probing verifies that the concept of preference strength is partially encoded among middle-to-late layers, and steering along probe-derived directions alters model-induced ordering distributions, demonstrating that the statistical behavioral preference of LLMs can be mechanistically measured and manipulated via internal representations.

• The paper demonstrates that multilingual LLMs reliably recover corpus-based binomial ordering directionality, achieving over 78% accuracy in some cases.
• The paper employs a framework integrating dataset curation, behavioral evaluation, sparse probing, and causal steering to map preference distributions to hidden states.
• The paper reveals that targeted interventions in model subspaces can modulate gradient ordering strengths, highlighting practical implications for style transfer and bias control.

Binomial Ordering Preferences in Multilingual LLMs: Behavioral and Representational Evidence

Introduction: Problem Formalization and Methodological Framework

This study investigates the extent to which LLMs encode and express the gradient conventions governing binomial orderings—expressions such as "men and women" versus "women and men"—across multiple languages. Instead of focusing solely on binary acceptability judgments, the work formalizes binomial ordering as a distributional alignment problem between corpus-derived and model-induced preference distributions. The framework builds on recent advances in probing and activation steering to link behavioral metrics to representational structure, enabling both the localization and causal intervention of preference strength within LM hidden states.

Figure 1: The analysis pipeline integrates dataset curation, behavioral evaluation, probing, and causal internal intervention to connect corpus, model outputs, and internal representations.

Dataset and Evaluation Protocol

The authors construct a typologically and script-diverse dataset comprising 600 binomial pairs mapped to eight languages (EN, DE, RU, ID, AR, TR, JA, ZH), with corpus-derived statistics from Sketch Engine’s TenTen corpora. Each pair is evaluated for both possible orderings, extracting conditional frequency counts to define reference probability distributions under Bayesian smoothing. LLM preference distributions are elicited via pooled log-likelihoods across four prompt templates to reduce context specificity.

Model--corpus alignment is quantified at both the categorical (direction) and distributional (strength) levels, using Spearman rank correlation, pairwise preferred-order accuracy, and probability-level distances (MAE, Jensen–Shannon divergence).

Behavioral Results: Directional Robustness Versus Graded Divergence

The principal empirical finding is that LLMs display systematic behavioral alignment with corpus-preferred binomial directions, most robustly for highly conventionalized pairs. For Llama-3.1-8B, for example, average pairwise accuracy on preferred ordering exceeds 78% across languages with Spearman $\rho$ peaking above 0.90 for JA and ZH, indicating robust directionality recovery. However, probability-level alignment is uniformly weaker: mean absolute error frequently exceeds 0.19, and Jensen–Shannon divergences indicate significant systematic probability displacement for less conventionalized or less corpus-polarized items.

Figure 2: Item-level scatter plots for Llama-3.1-8B visualize systematic deviations from corpus probability diagonal, with color marking corpus evidence tiers.

Behavioral misalignment is strongly structured by language. LLMs trained on multilingual corpora, for instance, display stronger alignment for ZH/JA than for ID/RU, paralleling the distributional sharpness of corpus preferences. Off-diagonal correlations in model–corpus heatmaps (Figure 3) suggest that induced orderings combine language-specific evidence with broader latent cross-linguistic patterns—an effect attributed to transfer and smoothing both at dataset and model architectural levels.

Figure 3: Spearman correlation heatmap, model–corpus by language, displaying strong diagonality but also substantial cross-language structure.

Representational Probing: Localization of Preference Strength Encoding

The work exploits sparse probing (Lasso-regularized linear regression) to decode the absolute deviation of corpus-based preference probability from 0.5 from Qwen3-4B hidden states. Probing results identify a sharply peaked decoding window in the model’s mid-to-late layers (layer 14 for last-token, layer 23 for mean-pooling), with maximum cross-validated $\rho=0.576$ (layer 14, minimal regularization). The correlation holds with highly compact probes, retaining substantial linear signal with fewer than 0.5% of dimensions active—a hallmark of abstract, non-diffuse feature encoding. This layer profile matches canonical findings in interpretability literature for compositional and semantic constructs.

Causal Steering: Manipulating Gradient Preferences via Representational Subspace

Direct intervention along the decoded probe direction highlights the causal role of these subspaces in model behavior. Activation steering at relevant layers (e.g., last-token at layer 14, Qwen3-4B) modulates model-induced binomial probabilities, systematically flattening or sharpening the sigmoid-shaped ordering curves as the steering strength $\lambda$ is varied. The intervention primarily attenuates/extenuates the strength of gradient preferences, rather than inverting directionality, and the effect is language-dependent—stronger for languages or items with more moderate baseline preferences.

Figure 4: Steering curves for Qwen3-4B at varying $\lambda$ values, showing controlled shifts in binomial ordering probability distributions by language.

Item-level case studies (Figure 5) further illustrate that steering modifies preference strength but respects high-confidence directionality, with ceiling effects for highly conventionalized orders. In languages with milder baseline preferences, the shift is more prominent and symmetric.

Figure 5: Controlled intervention (steering) for the "men and women" binomial, showing probability reallocation across direction and languages as a function of intervention strength.

Theoretical and Practical Implications

This study demonstrates that LLMs are not strictly categorical in their linguistic conventions but encode a notion of gradient conventionality, expressed in both external behavior and internal representation. Preferred directionality is robust, but precise probability-level alignment—essential for modeling empirically observed language use and idiomaticity—remains incomplete and varies systematically by language and evidence tier.

The presence of recoverable, causal subspaces for preference strength supports the view that LMs internalize abstract, compact features accessible to linear control, extending recent findings on monosemantic circuits and direct steering of factual or stylistic features. This grants a mechanism for controlled modification and analysis of conventions, raising both opportunities for targeted style transfer and risks for the amplification or attenuation of corpus-born cultural or societal biases, especially in sensitive or power-asymmetric binomial pairs.

These findings provoke further inquiry into the limits of inductive generalization in multilingual LMs, the differential transfer of language-specific conventions versus abstract features, and the potential for more advanced architectural or finetuning regimes to better capture the nuanced distributional texture of conventionality beyond categorical acceptability.

Conclusion

Behavioral and representational analysis of binomial ordering in LLMs reveals a nuanced competence: while models robustly infer and generalize dominant binomial orders, they less faithfully recover the nuanced distributional gradient of conventionality, a gap that is both measurable and mechanistically interpretable. Internal representations localize causally active subspaces encoding preference strength, enabling direct steering of output distributions. This delineates clear theoretical boundaries for the current state of LLM linguistic knowledge and poses concrete challenges for model and dataset design in multilingual and convention-rich settings.

Reference: "Behavioral and Representational Evidence of Binomial Ordering Preferences in LLMs" (2606.21645)