Position Bias Score (PBS) Overview

• Position Bias Score (PBS) is a family of metrics that quantify the extent to which observed outcomes depend on positional effects versus content relevance.
• PBS applies in diverse areas such as ranking, Vision Transformer classification, and LLM evaluation, each with tailored definitions and methods.
• Estimating PBS involves contextual and causal techniques to adjust for position-dependent biases, thereby improving model performance and fairness.

Searching arXiv for the cited papers and recent context. Position Bias Score (PBS) denotes a family of quantitative measures that capture how strongly an observed prediction, interaction, or evaluation outcome depends on position rather than content alone. In the cited literature, the term does not have a single canonical definition. In ranking and recommendation, PBS usually denotes an examination propensity at rank $k$, either as $\theta_k$ or as a top-normalized quantity such as $\theta_k/\theta_1$. In Vision Transformer classification, PBS is the average Position-SHAP score over correctly classified test samples for a model–dataset pair. In rubric-based LLM-as-a-judge, PBS coincides with a Bias Cost measuring deviation from position-neutral rubric selection, while in multiple-choice reasoning audits it is the Euclidean distance between the model’s absolute-position answer distribution and the uniform distribution. Closely related notions also appear in graph learning, extractive question answering, and long-context retrieval (Bruintjes et al., 19 May 2025, Wood et al., 2022, Xu et al., 2 Feb 2026, Wang, 21 Apr 2026).

1. Nomenclature and formal variants

The same acronym is used for several mathematically distinct estimands. What unifies them is not a shared formula but a shared role: each PBS isolates a positional effect that would otherwise be conflated with relevance, appearance, semantics, or true task difficulty.

Setting	PBS definition	Interpretation
PBM ranking	$PBS_k \equiv \theta_k$ or $PBS_k \equiv \theta_k/\theta_1$	examination propensity at rank $k$
Contextual ranking	$PBS(q,k)=\widehat{\Pr}(E=1\mid q,k)$	context-specific examination probability
ViT classification	average of $\phi_P/(\phi_P+\phi_I)$	reliance on positional embeddings
Rubric-based LLM judging	$\mathrm{Cost}(\pi)=\sum_p |P(p\mid \pi_p)-1/k|$	deviation from position-neutral score choice
MCQ reasoning audit	$\mathrm{PBS}(q)=\|\bar p_q-u\|_2$	absolute-position answer skew
GNN label bias	$\theta_k$0	proximity of node $\theta_k$1 to labeled nodes

In the ranking literature, PBS is typically embedded in a Position-Based Model (PBM) or Contextual Position-Based Model (CPBM), where the probability of click or engagement decomposes into a relevance term and an examination term. In that usage, PBS is a latent or estimated attention probability, and inverse-propensity correction is the primary downstream use (Demsyn-Jones, 2022, Mayor et al., 2021).

In Vision Transformers, the role of PBS is different. There it quantifies how much a classifier’s output depends on positional embeddings rather than appearance features. The relevant sample-level statistic is Position-SHAP, and PBS is the average of that statistic over correctly classified test examples (Bruintjes et al., 19 May 2025).

In LLM evaluation, the rubric-based and MCQ-reasoning papers define PBS as a direct behavioral deviation from positional neutrality. The first uses an absolute-deviation cost over rubric positions; the second uses an $\theta_k$2 distance from uniform over answer-option positions (Xu et al., 2 Feb 2026, Wang, 21 Apr 2026).

Some adjacent work studies position bias without introducing a single scalar called PBS. Extractive QA work measures position bias through out-of-position generalization failure and layer-wise representational dependence on early tokens, while long-context sorting work models a per-prompt bias curve $\theta_k$3 rather than a single number (Ko et al., 2020, Tang et al., 26 Jun 2026). This suggests that PBS is best understood as a family of position-sensitivity estimands rather than a universal metric.

2. PBS as examination propensity in ranking and recommendation

The classical ranking interpretation of PBS is examination probability. Under the PBM, user engagement is factorized as

$\theta_k$4

with $\theta_k$5. In some papers the Position Bias Score at rank $\theta_k$6 is exactly $\theta_k$7; in others it is normalized as $\theta_k$8, so that $\theta_k$9 and deeper ranks are interpreted relative to the top slot (Demsyn-Jones, 2022, Li, 2020).

The Thumbtack study operationalizes this definition through randomized adjacent swaps on 50% of search traffic. For eligible searches, one of 11 swap pairs—$\theta_k/\theta_1$0—is chosen uniformly at random, and the resulting contact counts are used to estimate pairwise ratios $\theta_k/\theta_1$1. Chaining adjacent ratios yields

$\theta_k/\theta_1$2

The resulting PBS curve is then used both to debias historical rate features via COEC and to define an IPS learning objective. In visitor-randomized A/B tests, replacing old COEC features with PBS-corrected $\theta_k/\theta_1$3 improved the rate of queries with at least one contact by $\theta_k/\theta_1$4 $\theta_k/\theta_1$5, while an IPS-based ranker improved the same metric by $\theta_k/\theta_1$6 $\theta_k/\theta_1$7 (Demsyn-Jones, 2022).

The Tripadvisor Hotels study uses a related but operationally distinct PBS methodology. There, $\theta_k/\theta_1$8, but the key observation is conditional on the user’s last click. All positions at or above the last click are treated as certainly examined,

$\theta_k/\theta_1$9

whereas positions below the last click are sampled with probability

$PBS_k \equiv \theta_k$0

This sampling-based approximation plays the role that inverse-propensity weighting would play in a fully explicit IPS formulation. In a two-week online A/B test, constant 80% below-last-click sampling produced $PBS_k \equiv \theta_k$1 clicks per session, whereas propensity sampling produced $PBS_k \equiv \theta_k$2 clicks $PBS_k \equiv \theta_k$3 relative to control (Li, 2020).

A third variant appears in deterministic recommender logging. Because the usual importance ratio $PBS_k \equiv \theta_k$4 degenerates when $PBS_k \equiv \theta_k$5 is deterministic, PBS is reinterpreted as the probability that a user examines the recommended item at its displayed position. If an item moves from logging position $PBS_k \equiv \theta_k$6 to counterfactual position $PBS_k \equiv \theta_k$7, the importance weight becomes $PBS_k \equiv \theta_k$8. In two large-scale Just Eat Takeaway.com A/B test replays, the resulting PBS-weighted offline estimator tracked online CTR with day-by-day correlations of approximately $PBS_k \equiv \theta_k$9 and $PBS_k \equiv \theta_k/\theta_1$0, albeit with a small constant underestimation bias (Wood et al., 2022).

3. Contextual, causal, and sparse-data estimation

A major limitation of non-contextual PBS curves is that they assume a single examination profile for all queries, users, and interfaces. CPBM-style work relaxes this by letting examination depend on both rank and observable context. In the intervention-harvesting formulation, the relevant quantity is $PBS_k \equiv \theta_k/\theta_1$1, and a normalized context-specific PBS can be written as $PBS_k \equiv \theta_k/\theta_1$2. The estimator exploits natural variation from multiple historical ranking functions rather than explicit randomized swaps. Interventional sets

$PBS_k \equiv \theta_k/\theta_1$3

and the AllPairs objective jointly estimate a propensity model $PBS_k \equiv \theta_k/\theta_1$4 and an average-relevance model $PBS_k \equiv \theta_k/\theta_1$5. Real-world ArXiv search experiments showed that simple and complex queries have different examination curves, and semi-synthetic Yahoo LTR experiments reported substantially lower estimation error—up to $PBS_k \equiv \theta_k/\theta_1$6–$PBS_k \equiv \theta_k/\theta_1$7 reduction—relative to PBM propensities (Fang et al., 2018).

Ranker-agnostic contextual EM regression provides a second contextual route. There, the Position Bias Score is defined directly as

$PBS_k \equiv \theta_k/\theta_1$8

with $PBS_k \equiv \theta_k/\theta_1$9 modeling examination and $k$0 modeling relevance. EM alternates between posterior estimates

$k$1

and M-step regression updates for $k$2 and $k$3. The reported empirical result is that this contextual estimator outperforms existing position-bias estimators in relative error when examination varies across queries, and that the learned propensities improve downstream ranking even when no context dependency is present (Mayor et al., 2021).

Sparse $k$4 data create a different estimation problem. The REM-with-Embedding approach retains the PBM definition $k$5 but mitigates sparsity by replacing item identity with a low-dimensional embedding assignment $k$6. Each logged click is split into pseudo-observations over latent embedding coordinates, and REM is run on the transformed data. On sparse Open Bandit, vanilla REM had RMSE $k$7, VAE+REM reduced this to $k$8, and LSI+REM further reduced it to $k$9. On Rakuten Ichiba, the corresponding RMSEs were $PBS(q,k)=\widehat{\Pr}(E=1\mid q,k)$0, $PBS(q,k)=\widehat{\Pr}(E=1\mid q,k)$1, and $PBS(q,k)=\widehat{\Pr}(E=1\mid q,k)$2. The same estimated PBS curves improved downstream ranking metrics, with Rakuten MRR rising from $PBS(q,k)=\widehat{\Pr}(E=1\mid q,k)$3 under REM to $PBS(q,k)=\widehat{\Pr}(E=1\mid q,k)$4 under LSI+REM (Ishikawa et al., 2023).

A distinct causal line of work does not estimate examination propensity directly, but instead treats position bias as a causal position effect. Using historical A/B tests as instruments, the LinkedIn study defines item-level and system-level effects

$PBS(q,k)=\widehat{\Pr}(E=1\mid q,k)$5

Two-stage least squares then estimates the average causal effect of a position change. In feed ads, a representative IV estimate gives a coefficient on feed position of $PBS(q,k)=\widehat{\Pr}(E=1\mid q,k)$6 (SE $PBS(q,k)=\widehat{\Pr}(E=1\mid q,k)$7), corresponding to a $PBS(q,k)=\widehat{\Pr}(E=1\mid q,k)$8 increase in click probability for a one-slot rise; in PYMK, a one-slot bump increases invite probability by $PBS(q,k)=\widehat{\Pr}(E=1\mid q,k)$9–$\phi_P/(\phi_P+\phi_I)$0 at the edge level (Friedberg et al., 2022). A plausible implication is that the ranking literature contains two related but distinct estimands: PBS as exposure propensity, and position effect as causal response to movement across ranks.

4. PBS in Vision Transformer classifiers

In Vision Transformer image classification, PBS measures positional reliance in the classifier rather than user exposure. The motivating claim is that positional information can be either spurious or useful, depending on the data: camera placement may be arbitrary, favoring translation invariance, but datasets may also contain center bias, capture bias, or stable scene layout such as “the sky is up” (Bruintjes et al., 19 May 2025).

The formal device used to define PBS is Position-SHAP, an extension of Kernel SHAP to handle position embeddings. In a ViT input, there are two groups of features at the first layer: patch embeddings $\phi_P/(\phi_P+\phi_I)$1 and position embeddings $\phi_P/(\phi_P+\phi_I)$2. Because $\phi_P/(\phi_P+\phi_I)$3 is constant with respect to image content, naive SHAP assigns it no importance. Position-SHAP instead treats the collection of PE tokens $\phi_P/(\phi_P+\phi_I)$4 as a background distribution and masks by randomly reassigning them among positions. The local linear approximation is written as

$\phi_P/(\phi_P+\phi_I)$5

and the sample-level score is

$\phi_P/(\phi_P+\phi_I)$6

Averaging this quantity over all correctly classified test samples yields the Position Bias Score for a model–dataset pair (Bruintjes et al., 19 May 2025).

The empirical picture is a spectrum from near-zero to very high PBS. In the toy CIFAR-10-Position dataset, position-dependent samples produce significantly higher Position-SHAP than randomly cornered samples, with Mann–Whitney $\phi_P/(\phi_P+\phi_I)$7. On real datasets, EuroSAT has low PBS, approximately $\phi_P/(\phi_P+\phi_I)$8, and removing position embeddings slightly improves accuracy. SVHN has very high PBS, approximately $\phi_P/(\phi_P+\phi_I)$9, and removing position embeddings drops performance from approximately $\mathrm{Cost}(\pi)=\sum_p |P(p\mid \pi_p)-1/k|$0 to approximately $\mathrm{Cost}(\pi)=\sum_p |P(p\mid \pi_p)-1/k|$1. CIFAR-10, Flowers-102, and Imagenette exhibit intermediate PBS, approximately $\mathrm{Cost}(\pi)=\sum_p |P(p\mid \pi_p)-1/k|$2–$\mathrm{Cost}(\pi)=\sum_p |P(p\mid \pi_p)-1/k|$3, with moderate accuracy loss when positional encoding is removed. The paper reports a tight correlation between a dataset’s PBS under a baseline APE model and the amount of accuracy lost when dispensing with positional embeddings entirely (Bruintjes et al., 19 May 2025).

This PBS framework is used to motivate Auto-PE, a one-parameter gating mechanism

$\mathrm{Cost}(\pi)=\sum_p |P(p\mid \pi_p)-1/k|$4

initialized at $\mathrm{Cost}(\pi)=\sum_p |P(p\mid \pi_p)-1/k|$5 and optimized end-to-end with learning rate $\mathrm{Cost}(\pi)=\sum_p |P(p\mid \pi_p)-1/k|$6. When $\mathrm{Cost}(\pi)=\sum_p |P(p\mid \pi_p)-1/k|$7, position is effectively turned off; when $\mathrm{Cost}(\pi)=\sum_p |P(p\mid \pi_p)-1/k|$8, position is amplified. For RoPE, the rotary angle $\mathrm{Cost}(\pi)=\sum_p |P(p\mid \pi_p)-1/k|$9 is scaled by $\mathrm{PBS}(q)=\|\bar p_q-u\|_2$0, so $\mathrm{PBS}(q)=\|\bar p_q-u\|_2$1 becomes $\mathrm{PBS}(q)=\|\bar p_q-u\|_2$2. On EuroSAT, Auto-APE drives $\mathrm{PBS}(q)=\|\bar p_q-u\|_2$3, approaching the no-PE optimum; on SVHN, Auto-APE learns $\mathrm{PBS}(q)=\|\bar p_q-u\|_2$4, matching or slightly amplifying the APE optimum. Position-affecting augmentations reduce PBS, and using a $\mathrm{PBS}(q)=\|\bar p_q-u\|_2$5 token instead of global-average pooling tends to increase PBS (Bruintjes et al., 19 May 2025).

5. Graph learning and extractive QA

In semi-supervised graph learning, the relevant positional bias is proximity to the labeled subset rather than rank in a list. The corresponding score is the Label Proximity Score, which the cited summary also treats as a Position Bias Score. Let

$\mathrm{PBS}(q)=\|\bar p_q-u\|_2$6

with $\mathrm{PBS}(q)=\|\bar p_q-u\|_2$7 for labeled nodes and $\mathrm{PBS}(q)=\|\bar p_q-u\|_2$8 otherwise. The score vector is

$\mathrm{PBS}(q)=\|\bar p_q-u\|_2$9

This quantity measures the total random-walk proximity from node $\theta_k$00 to the labeled set. Empirically, grouping nodes by LPS yields nearly monotonic accuracy curves, whereas grouping by degree or shortest-path distance is less consistent. The proposed LPSL optimization learns a new propagation matrix $\theta_k$01 such that all nodes share equal proximity to the labeled set while remaining smooth with respect to the original graph. On Cora, backbone GCN improves from $\theta_k$02 to $\theta_k$03, and APPNP from $\theta_k$04 to $\theta_k$05; across eight datasets, LPSL variants improve or match the best baseline in $\theta_k$06 splits (Han et al., 2023).

Extractive QA studies an adjacent form of position bias but does not define a single scalar PBS. Instead, it operationalizes position bias through performance collapse when test answer positions differ from training answer positions, and through layer-wise measures of representational dependence on early tokens. In a biased SQuAD setting where training answers lie only in the first sentence, BERT achieves approximately $\theta_k$07 F1 on first-sentence dev examples but only approximately $\theta_k$08 on dev examples whose answers lie later in the passage. Sentence-level prior ensembling with a learned mixin recovers performance on out-of-position answers to $\theta_k$09 while preserving high in-position accuracy, and full-train SQuAD performance remains approximately $\theta_k$10 F1 with sentence-level prior product or mixin (Ko et al., 2020).

The conceptual relation between these two lines is that both define position not as rank alone but as access to supervision. In the GNN case, supervision arrives through message passing from labeled nodes; in extractive QA, supervision arrives through a skewed answer-position prior. This suggests that PBS-like quantities can be defined whenever position changes the amount of usable signal reaching the learner, even outside standard ranking interfaces.

6. PBS in LLM-as-a-judge and reasoning audits

Rubric-based LLM evaluation treats the rubric list as an ordered multiple-choice interface. The relevant PBS is the Bias Cost

$\theta_k$11

with $\theta_k$12 in the reported experiments. Under no position bias, each score option would be selected with equal probability $\theta_k$13 at every rubric position. The paper estimates these position-conditional probabilities by running ten balanced permutations: five forward cyclic rotations and five reverse rotations, so that each score appears exactly twice in each position. This design supports both explicit estimation of $\theta_k$14 and permutation averaging, which cancels first-order position effects (Xu et al., 2 Feb 2026).

The observed bias is substantial and model-dependent. Aggregated slot-selection probabilities exceed the $\theta_k$15 ideal at slot 1 for all models, and many models also exceed $\theta_k$16 at slot 5. Qwen3-8B selects position 1 about $\theta_k$17–$\theta_k$18 of the time, while GPT-4.1 selects it about $\theta_k$19–$\theta_k$20. Table 9 reports default Cost values for the canonical order $\theta_k$21, including approximately $\theta_k$22 for GPT-4.1, $\theta_k$23 for Qwen3-8B, and $\theta_k$24 for GPT-OSS-120B. Choosing the minimum-bias cyclic ordering can materially lower the score; GPT-4.1 falls to approximately $\theta_k$25 under reversal. Averaging across balanced permutations improves human correlation on rubric benchmarks: on HANNA, GPT-4.1 Spearman rises from $\theta_k$26 to $\theta_k$27, the average $\theta_k$28 across all models is $\theta_k$29, and on SummEval the average $\theta_k$30 is $\theta_k$31 (Xu et al., 2 Feb 2026).

A different PBS definition appears in reasoning-model auditing for four-option multiple-choice question answering. For each question $\theta_k$32, the answer options are cyclically permuted so that the correct answer appears in each absolute position. Let $\theta_k$33 be the empirical answer-probability vector for shift $\theta_k$34, let $\theta_k$35, and let $\theta_k$36. The per-question PBS is

$\theta_k$37

High PBS means that the model persistently favors some absolute option positions after the content has been permuted away from them (Wang, 21 Apr 2026).

This formulation is used to study length-driven bias in chain-of-thought reasoning. Across thirteen reasoning-mode configurations, twelve show positive partial correlation between reasoning length and PBS after controlling for accuracy, with values in $\theta_k$38 and all $\theta_k$39. All twelve open-weight reasoning-mode configurations exhibit monotonically increasing PBS across length quartiles. For R1-Qwen-7B on MMLU, mean PBS rises from $\theta_k$40 in the shortest quartile to $\theta_k$41, $\theta_k$42, and $\theta_k$43 in the longest quartile. DeepSeek-R1 at $\theta_k$44B has aggregate PBS $\theta_k$45, but the longest quartile still shows PBS $\theta_k$46, indicating that high overall accuracy can suppress visible bias without eliminating the underlying mechanism (Wang, 21 Apr 2026).

The truncation intervention provides causal evidence for this length effect. Around the Commitment Change Point, trajectories are truncated at relative offsets and decoding is resumed. For R1-Qwen-7B on MMLU, shifts toward the model’s preferred position increase from $\theta_k$47 at early truncation points to $\theta_k$48 at late ones. The same study distinguishes direct-answer baseline bias from chain-of-thought length-accumulated bias: Llama-Instruct-direct has strong direct-mode PBS, Qwen-Instruct-direct has weaker direct-mode PBS, and direct-answer bias is uncorrelated with trajectory length, whereas CoT reasoning introduces a length-linked bias signature (Wang, 21 Apr 2026).

7. Interpretation, comparability, and limitations

PBS values are not comparable across domains without first specifying the estimand and normalization. A ranking PBS of $\theta_k$49 may mean a relative examination probability, a ViT PBS of $\theta_k$50 may mean modest reliance on positional embeddings, and a reasoning PBS of $\theta_k$51 may mean substantial absolute-position skew in answer selection. This is a definitional issue rather than a statistical one.

A common misconception is that PBS is always a causal quantity. In PBM-based ranking, PBS is generally an examination propensity inferred under assumptions such as separability between relevance and examination, homogeneity of $\theta_k$52 within an interface, and absence of trust bias. The Thumbtack study states these assumptions explicitly and notes that if separability fails, $\theta_k$53 absorbs some trust bias rather than isolating pure examination (Demsyn-Jones, 2022). In deterministic-list offline evaluation, PBS-weighted IPS is unbiased only up to model misspecification, and its validity depends on the examination model being a reasonably accurate approximation of user behavior (Wood et al., 2022).

A second misconception is that correcting for PBS is necessarily sufficient. Long-context attention-sorting results provide a counterexample. There, a per-prompt bias curve $\theta_k$54 is estimated from the low-attention majority of documents, and a position-specific baseline $\theta_k$55 is subtracted or divided out of raw attention scores before one-pass sorting. On LLaMA-2-7B-32K-Instruct, this debiasing produces the same containment accuracy as uncalibrated one-pass sorting, $\theta_k$56, and on YaRN-Llama-2-7B-64K it improves one-pass sorting from $\theta_k$57 to $\theta_k$58 but remains $\theta_k$59 percentage points behind iterative sorting, closing only $\theta_k$60 of the gap (Tang et al., 26 Jun 2026). This suggests that position-bias correction may be necessary yet insufficient when iterative interaction changes the effective evidence available to the model.

A third misconception is that position bias is always undesirable. The ViT results show that the utility of position information is data-dependent: EuroSAT benefits from effectively turning position off, whereas SVHN depends on it strongly (Bruintjes et al., 19 May 2025). By contrast, ranking systems usually treat position bias as a nuisance because it contaminates labels rather than improving the signal of interest (Demsyn-Jones, 2022).

Taken together, these literatures establish PBS as a general methodological pattern: identify the portion of observed behavior attributable to position, quantify it with a task-appropriate estimand, and then either debias downstream learning or adapt the model so that its use of positional information matches the structure of the data.