Language-Native Ranking Overview

• Language-Native Ranking is a design orientation where language models and intrinsic linguistic features form the core of ranking decisions.
• It leverages PLMs to score query-document pairs, enabling pairwise and listwise comparisons that capture nuances like scalar intensity and humor.
• Real-world applications, such as Baidu Search and TreeRanker for code completion, demonstrate enhanced efficiency and adaptability in complex retrieval tasks.

Language-native ranking denotes a family of ranking formulations in which language is not merely an auxiliary feature source but the primary object, signal, or computational substrate of the ranking decision. The surveyed work suggests that the term has several recurrent meanings: a LLM can serve as the central ranking function over textual candidates; ranking can be defined over intrinsically linguistic properties such as scalar intensity, humor, metaphor novelty, or borrowing nativeness; and ranking can be made native to comparison itself, replacing unstable absolute scores with pairwise or listwise preference judgments. In that sense, language-native ranking is better understood as a design orientation than as a single algorithmic template (Zou et al., 2021, Wang et al., 22 Apr 2026, Soler et al., 2021, Zheng et al., 18 Mar 2026).

1. Conceptual scope

A useful distinction within the literature is between language-model-native, language-sensitive, and comparison-native ranking. In industrial search, the ranking function becomes language-model-native when relevance is computed primarily by a PLM that directly reads query, title, and selected document content, rather than by manually engineered lexical and behavioral features. In multilingual settings, the same phrase can instead refer to whether a ranker privileges the query’s native language or English, sometimes incorrectly. In linguistic semantics, it refers to ranking structures that are native to a language’s own lexical inventory, morphology, or usage patterns. In evaluation, it refers to systems trained to compare items directly, rather than to reproduce unstable absolute scales (Zou et al., 2021, Wang et al., 22 Apr 2026, Soler et al., 2021, Zheng et al., 18 Mar 2026).

The boundary of the concept is also clarified by contrast cases. “Language Ranker” ranks languages by similarity of target-language and English last-token hidden states at layers $\{5,10,15,20,25\}$, using English as the baseline; the paper itself is best interpreted as an English-referenced multilingual ranking metric rather than a fully language-native one. By contrast, NN-Rank ranks source languages for zero-shot transfer from nearest-neighbor structure in multilingual hidden states built from unlabeled target text, so the ranking is target-aware and model-aware rather than anchored to English alone (Li et al., 2024, Ebrahimi et al., 3 Oct 2025).

2. LLMs as first-class ranking functions

In large-scale search systems, language-native ranking becomes most concrete when the ranking problem is written as learning a scoring function $$f(\cdot,\cdot): \mathbb{Q}\times\mathbb{D}\rightarrow\mathbb{R}$$ and the score is produced by a PLM that directly contextualizes query and document text. The Baidu Search deployment study is exemplary: it identifies four obstacles to this shift—latency from quadratic self-attention, long-document handling for web pages whose content is often longer than 4000 tokens, mismatch between generic PLM pre-training and retrieval relevance, and score compatibility inside a committee system containing freshness, quality, and authority modules. The practical implication is that a language-model-native ranker in production must be accurate, low-latency, and blendable with non-neural ranking components (Zou et al., 2021).

The same paper operationalizes this with QUITE, a linear-time query-dependent summarizer with complexity $\mathcal{O}(N_c)$; Pyramid-ERNIE, which splits a 12-layer model into 9 lower representation layers and 3 upper interaction layers; relevance-oriented pre-training on $538{,}314{,}000$ queries and $2{,}986{,}664{,}000$ query-document pairs; and human-anchored fine-tuning on about 10 million 0–4 relevance labels. Pyramid-ERNIE reported about 30% inference-time reduction relative to original ERNIE, while the full CAP+REP+HINT stack improved online $\Delta DCG_2$ by 2.85% and $\Delta DCG_4$ by 1.58% despite a slight offline PNR reduction, showing that score stability and committee compatibility can dominate pure pairwise accuracy in production. A related deployment logic appears in code completion: TreeRanker keeps static candidate generation external, but makes ranking language-model-native by organizing candidates in a trie and collecting token-level scores in a single greedy decoding pass, nearly matching exhaustive Beam@All ranking while giving up to 30x lower inference time (Zou et al., 2021, Cipollone et al., 4 Aug 2025).

3. Multilinguality, native-language bias, and language-agnostic correction

In multilingual RAG, language-native ranking becomes problematic when rankers conflate language match with answer utility. The LAURA study shows that current rerankers systematically favor English and the query’s native language: for the widely used BGE reranker, more than 70% of the top-5 retrieved documents, averaged across 13 languages, come from English and the query language alone. Yet the estimated oracle over the same retrieved pool is much higher—e.g., 63.6 character 3-gram recall for Llama-8B-Instruct versus 48.9 for BGE reranking—and score–answer correlations remain below 0.2. LAURA addresses this by defining positives through downstream generative utility, constructing language-debiased subsets, filtering documents with $\theta=0.8$, and fine-tuning with a listwise objective. The result is lower divergence from the oracle language distribution, higher PEER fairness, and consistent downstream gains, showing that optimal multilingual evidence is often distributed across multiple languages rather than concentrated in English or the query language (Wang et al., 22 Apr 2026).

A different multilingual ranking problem arises in zero-shot cross-lingual transfer: given a target language, which source language should be selected for training? NN-Rank answers this by encoding unlabeled target text and all source datasets with mBERT or XLM-R, retrieving top-$k$ nearest source subword representations for each target subword, and tallying how often each source dataset is retrieved. On POS and NER, it improves over feature-based baselines by up to 35.56 NDCG and 18.14 NDCG, remains competitive even when only Bible text is available, and reaches 92.8% of full-data NDCG with as few as 25 target examples. The ranking here is native to the multilingual model’s hidden-space geometry rather than to hand-crafted language descriptors (Ebrahimi et al., 3 Oct 2025).

The multilingual literature also contains two cautionary contrasts. First, “Language Ranker” correlates with pretraining proportions but explicitly mixes English affinity, pretraining exposure, and some multilingual generalization; it does not isolate native-language competence on each language’s own terms. Second, earlier multilingual entity ranking already showed that global importance and language-specific salience diverge: a system built from Wikipedia and Freebase ranked 27 million entities, yet examples such as Morocco ranking much higher in Arabic than in English make clear that cross-language importance is not a single universal scalar (Li et al., 2024, Bhattacharyya et al., 2017).

4. Ranking intrinsic linguistic structure and nativeness

One of the clearest forms of language-native ranking is ranking lexical items by an intrinsic semantic dimension. “Scalar Adjective Identification and Multilingual Ranking” builds MULTI-SCALE by translating English scalar adjective datasets into French, Spanish, and Greek, then ranks adjectives by cosine similarity to an intensity direction $dVec$. The ranking task is thus language-sensitive at both the dataset and model levels: translations are context-specific, ties can change by language, and monolingual contextual encoders outperform multilingual BERT in almost all cases. The same paper shows that scalar ordering and scalar identification are distinct tasks: on the English SCAL-REL benchmark, a logistic regression probe over BERT adjective representations reaches 0.946 accuracy, whereas DIFFVEC-style scalarity scoring is only about 0.549, indicating that a one-dimensional intensity direction is effective for ordering known scalar adjectives but not sufficient for deciding scalarity itself (Soler et al., 2021).

A closely related problem is ranking creative language characteristics. For humorousness and metaphor novelty, the small-data study compares GPPL with DirectRanker and finds that standard neural baselines collapse when training data shrinks, whereas DirectRanker remains effective and stacking DirectRanker with GPPL improves Spearman’s $$f(\cdot,\cdot): \mathbb{Q}\times\mathbb{D}\rightarrow\mathbb{R}$$0 by 14% and 16% on average over prior state of the art. The ranked objects are texts themselves, ordered by intrinsic creative properties rather than by external relevance. This makes the task a strong example of language-native ranking in the literal sense: the target variable is a graded linguistic characteristic of the utterance (Siekiera et al., 2020).

Nativeness can also be defined sociolinguistically rather than semantically. In Hindi-English social media, borrowing detection is posed as ranking English-origin words by how native-like they have become in Hindi discourse. The proposed UUR, UTR, and UPR statistics quantify use in Hindi or Hindi-majority contexts, and their rankings correlate with human nativeness judgments at about 0.62–0.63 Spearman, compared with 0.26 for the newspaper-frequency baseline. For highly ranked words in Hindi-heavy contexts, annotators preferred changing the language tag from English to Hindi in about 88% of cases. Here, nativeness is distributional integration into local usage, not etymological origin (Patro et al., 2017).

A different but related ranking view appears in native language identification and in large-scale language statistics. ProDAPT casts NLI as ranking candidate L1s by language-model loss, replacing one full GPT-2 model per language with a shared backbone plus adapters; this reduces storage from 15.95 GB to 2.4 GB, lowers GPU memory from 16.6 GB to 4.1 GB, and yields 13x faster inference while still reaching 84.2% test accuracy on TOEFL11. At a much broader scale, rank-dynamics analysis of Google Books $$f(\cdot,\cdot): \mathbb{Q}\times\mathbb{D}\rightarrow\mathbb{R}$$1-grams shows that several languages share a stable low-rank core and a volatile higher-rank periphery, captured by rank diversity $$f(\cdot,\cdot): \mathbb{Q}\times\mathbb{D}\rightarrow\mathbb{R}$$2, change probability $$f(\cdot,\cdot): \mathbb{Q}\times\mathbb{D}\rightarrow\mathbb{R}$$3, rank entropy $$f(\cdot,\cdot): \mathbb{Q}\times\mathbb{D}\rightarrow\mathbb{R}$$4, and rank complexity $$f(\cdot,\cdot): \mathbb{Q}\times\mathbb{D}\rightarrow\mathbb{R}$$5. This suggests that language-native ranking is not only a modeling choice but also a statistical property of language use itself (Uluslu et al., 2022, Morales et al., 2018).

5. Comparison-native and representation-native ranking

Several works shift the ranking primitive from absolute scoring to direct comparison or latent ranking structure. “Language Modelling via Learning to Rank” reframes next-token prediction as ranking a branching set of plausible continuations rather than optimizing one-hot cross-entropy alone. Using a Plackett–Luce objective over teacher-generated top-$$f(\cdot,\cdot): \mathbb{Q}\times\mathbb{D}\rightarrow\mathbb{R}$$6 words, it improves Wiki02 Transformer-XL perplexity from 65.27 to 55.94, outperforming KL-based distillation at 56.70, and even a non-probabilistic $$f(\cdot,\cdot): \mathbb{Q}\times\mathbb{D}\rightarrow\mathbb{R}$$7-gram teacher is competitive with BERT and Born-Again teachers. The implication is that language modeling itself can be made ranking-native when multiple plausible continuations are treated as an ordered set rather than as uniformly wrong negatives (Frydenlund et al., 2021).

Other work asks whether ranking knowledge is already encoded inside LM representations. Contrast-Consistent Ranking extends CCS from binary truthfulness to ranking by learning a probe from hidden states under consistency constraints, with MarginCCR, TripletCCR, and OrdRegCCR variants; across models and datasets, CCR probing often outperforms or matches pairwise, pointwise, and listwise prompting, with TripletCCR strongest on average. “Ranking LLMs without Ground Truth” pushes this further by ranking LLMs from triplets of models that judge one another; under sufficient separation and low overlap in incorrect answers, Greedy Triplet Ranking recovers the correct ordering except for the top-two ambiguity, and empirically the method matches reference-based rankings closely on summarization, multiple choice, and dialog (Stoehr et al., 2023, Dhurandhar et al., 2024).

Once rankings are built from comparisons, aggregation becomes a central methodological issue. In head-to-head LLM evaluation, Bradley–Terry preserved transitivity best, with 77.29% on Chatbot Arena and 56.67% on SLAM, while Glicko was recommended for large uneven datasets because it models uncertainty explicitly; Elo was not recommended because of instability, order sensitivity, and hyperparameter sensitivity. CNPE applies the same comparison-native logic to scientific paper evaluation: graph-based pair sampling, comparison-oriented SFT, RL with comparison-based rewards, and Bradley–Terry aggregation. On ICLR-2025, this yielded an average relative improvement of 21.8% over DeepReview-14B and generalized to five unseen datasets, supporting the broader claim that relative judgment can be more robust than absolute score regression when evaluation scales drift across venues or time periods (Daynauth et al., 2024, Zheng et al., 18 Mar 2026).

6. Evaluation protocols, misconceptions, and open problems

A recurring misconception is that language-native ranking simply means preferring the user’s native language, the most frequent language, or the strongest-resource language. The multilingual RAG literature shows the opposite: same-language and English favoritism can suppress answer-critical evidence. Another misconception is that ranking can be recovered safely from absolute scores after the fact; comparison-native paper evaluation argues that score scales vary across conferences and years, so isolated scoring encourages models to learn narrow conventions rather than stable judgment. A third misconception is that English-referenced multilingual metrics are automatically language-neutral; the Language Ranker analysis explicitly rejects that interpretation (Wang et al., 22 Apr 2026, Zheng et al., 18 Mar 2026, Li et al., 2024).

The protocol used to elicit judgments is itself part of the ranking system. RankME combines continuous scales with relative assessment and shows that experimental design changes reliability substantially: in Setup 2, ICC rose to 0.42 for naturalness and 0.68 for quality, outperforming Likert and plain magnitude estimation, and RankME plus TrueSkill produced the same system rankings as direct RankME aggregation. This suggests that language-native ranking is not only a property of models and objectives; it is also a property of how humans are asked to compare language (Novikova et al., 2018).

Open problems recur across the literature. Multilingual RAG still localizes the main pathology at the reranking stage and leaves retriever-generator redesign open. Scalar ranking remains anchored to translated English resources and calls for language-specific scalarity benchmarks. Triplet-based model ranking provides theory for GTR but not for FTR. Source-language ranking remains sensitive to domain mismatch and has not yet addressed multi-source transfer. Live leaderboard systems still need uncertainty-aware aggregation that is robust to sparse, imbalanced comparison graphs. Taken together, these directions suggest that the field is moving from isolated scoring toward ranking systems that are simultaneously language-aware, model-aware, and comparison-aware, but the balance between efficiency, fairness, calibration, and cross-domain generalization remains unresolved (Wang et al., 22 Apr 2026, Soler et al., 2021, Dhurandhar et al., 2024, Ebrahimi et al., 3 Oct 2025, Daynauth et al., 2024).