Context-Performance Paradox

Updated 5 July 2026

Context-Performance Paradox is a phenomenon where a system’s measured competence reverses with context changes, highlighting inconsistencies between specialized and general tasks.
It is formalized with metrics such as the Domain Specificity Index (DSI) and Performance Inversion Metric (PIM), which reveal stark performance inversions in large language models.
Empirical studies across AI, HCI, and decision theory demonstrate that conventional averaging obscures key context-dependent dynamics, urging refined evaluation methods.

Searching arXiv for papers directly relevant to context-dependent performance inversions and related paradox formulations. search_arxiv({"query":"\"The Rosetta Paradox\" Domain-Specific Performance Inversions in LLMs", "max_results": 5}) The Context-Performance Paradox denotes a family of context-dependent reversals in which measured competence, success, fidelity, or utility is not stable across settings and can invert as the relevant context changes. In current LLM research, the closest formalization is the “Rosetta Paradox,” defined as the case in which LLMs “excel in highly specialized or domain-specific applications” yet “demonstrate poor performance on the so-called general, seemingly simpler tasks”; related formulations appear in long-context reasoning, personalization, risky choice, organizational collaboration, and social learning (Jha et al., 2024, Du et al., 6 Oct 2025, Ontañón et al., 2021, Honda et al., 2024, Hu et al., 2022, Collins et al., 16 Jan 2025).

1. Cross-domain structure of the paradox

Across the cited literatures, the paradox has a common architecture: performance is favorable when measured in one context, but the same system degrades, inverts, or becomes mischaracterized when the context is altered. In these works, “context” can denote a knowledge domain, prompt length, retrieved corpus, adaptive interface, organizational layer, social-learning environment, or lottery choice set. The common methodological move is to separate quantities that are often conflated—specialization versus generalization, retrieval versus reasoning, user modeling versus intervention, performance versus success, or logical structure versus coarse-grained observables (Jha et al., 2024, Yucesoy et al., 2015, Ontañón et al., 2021, Liu et al., 4 Jan 2026).

This family resemblance is especially clear in cases where average or aggregate scores conceal asymmetry. The Rosetta Paradox argues that “average benchmark score” can hide domain-specific inversions (Jha et al., 2024). “Untangling Performance from Success” separates an athlete’s objective competitive record from socially mediated popularity (Yucesoy et al., 2015). The personalization literature shows that a system can become better at adaptation while becoming a worse observer of the user it is modeling (Ontañón et al., 2021). A plausible implication is that the paradox is less a single theorem than a recurrent pattern in which context-blind evaluation merges distinct mechanisms into one headline number.

2. Domain-specific inversion in LLMs

The most explicit LLM formalization appears in “The Rosetta Paradox: Domain-Specific Performance Inversions in LLMs” (Jha et al., 2024). There the paradox is defined as a counterintuitive inversion in which specialized-context performance exceeds general-context performance even when the general tasks are intuitively simpler. The paper formalizes this with the Domain Specificity Index (DSI), intended to measure how specialized a dataset or task is by the ratio of domain-specific terms to total terms, and the Performance Inversion Metric (PIM), which compares specialized-task accuracy with general-task accuracy and normalizes it. Higher DSI denotes a more specialized domain; a positive PIM denotes better performance on specialized tasks than on general tasks.

The reported examples are sharply asymmetric:

Model	Specialized / General accuracy	PIM
BioBERT	94% / 70%	+0.48
LEGAL-BERT	92% / 68%	+0.41
GPT-3 (175B)	89% / 86%	+0.03
BERT (Base)	82% / 83%	-0.01

The same pattern appears in cross-domain transition tests. BioBERT drops from 92% specialized knowledge accuracy to 66% transition-task accuracy, and LEGAL-BERT from 90% to 64%, whereas GPT-3 changes from 89% to 85% and BERT base from 81% to 80% (Jha et al., 2024). The experimental design spans GPT-3 (175B), BERT (base), BERT (large), BioBERT, and LEGAL-BERT; specialized datasets include MedQA and an arXiv Physics corpus, while general datasets include CommonCrawl and OpenBookQA.

The paper argues that this is “likely not just an artifact of biased training data,” but an “intrinsic architectural and emergent property” of deep neural networks (Jha et al., 2024). Training-data bias and catastrophic forgetting are treated as contributors, not exhaustive explanations. The broader significance is evaluative: standard single-score benchmarks can obscure whether a model is a strong specialist, a strong generalist, or both, and those regimes are not equivalent.

3. Prompt context, long context, and fidelity in LLM systems

A second major formulation concerns prompt length and retrieval. “Context Length Alone Hurts LLM Performance Despite Perfect Retrieval” shows that long-context degradation is not reducible to retrieval failure: across 5 open- and closed-source LLMs on math, question answering, and coding tasks, performance degrades by 13.9%–85% as input length increases despite perfect retrieval (Du et al., 6 Oct 2025). The benchmark structure separates evidence, distraction tokens, and question; retrieval is measured by strict exact match. A concrete example is Llama-3.1-8B-Instruct, which perfectly retrieves evidence on 970 of 1000 MMLU problems extended to 30K tokens, yet its accuracy drops by 24.2% relative to the short-context case. The effect persists when distractions are replaced with whitespace, when evidence is placed immediately before the question, and even when irrelevant tokens are masked so the model attends only to relevant tokens. The mitigation is to retrieve and recite the evidence first, then solve; on RULER, GPT-4o improves by up to 4%.

“Predicting Task Performance with Context-aware Scaling Laws” treats this as a scaling problem in which downstream performance depends jointly on training compute, prompt length, and model context limit (Montgomery et al., 16 Oct 2025). The proposed law is

$\mathcal{P}(C, n_\text{pmt}, n_\text{ctx}) = \Bigl[1 - \exp\Bigl(-A \Bigl(\frac{C}{C^c}\Bigr)^\alpha\Bigr)\Bigr] \times \Bigl[1 - \exp\Bigl(-B \Bigl(\frac{n_\text{pmt}}{n_\text{pmt}^c}\Bigr)^\beta\Bigr)\Bigr] \times \sigma(n_\text{pmt} - n_\text{ctx}),$

with saturating gains from compute and usable context and a sharp penalty when prompts exceed the model’s reliable window. The framework is fit on 65,500 instances across arithmetic reasoning, common sense reasoning, and machine translation using YaRN-extended Llama-2-7B and Llama-2-13B checkpoints from 4k to 128k context. The key result is not that more context is useless, but that it helps only up to a task-dependent saturation regime and can reverse once the effective context limit is crossed.

A third variant concerns lossy context compression. “When Less is More: The LLM Scaling Paradox in Context Compression” identifies a “Size-Fidelity Paradox” in compressor–decoder setups: larger compressors, spanning 0.6B to 90B, reach lower loss and may preserve BLEU while becoming less faithful to source context under 4×, 16×, and 64× compression (Guo et al., 10 Feb 2026). The two named mechanisms are knowledge overwriting—e.g., “the white strawberry” $\to$ “the red strawberry”—and semantic drift—e.g., “Alice hit Bob” $\to$ “Bob hit Alice.” Effective rank of the latent memory embeddings increases monotonically with model scale in Qwen3 from 0.6B to 32B and is strongly negatively correlated with knowledge-faithfulness accuracy, with Pearson $r = -0.931$ and Spearman $\rho = -0.996$ . Conditional entropy also becomes negatively associated with QA faithfulness, with Pearson $r = -0.823$ and Spearman $\rho = -0.876$ .

A fourth long-context variant is safety-conditioned retrieval. “The Injection Paradox: Brand-Level Suppression in Safety-Trained LLM Recommendations via RAG Context Injection” studies a static-corpus RAG recommendation task for wireless earbuds with 40 documents across 9 brands (Paeng, 8 Jun 2026). Only 1 of 40 documents is manipulated, corresponding to 2.5% of the corpus, yet in Claude Opus 4.6 the target brand drops from a 54% baseline to zero top-2 recommendations across all 50 trials in the combined condition. The same paper reports that GPT-4o-mini moves in the opposite direction, from 17% baseline to 40% under injection and 67% in the combined condition. The effect is interpreted as brand-level suppression propagation: one injected document appears to trigger a trustworthiness penalty that extends to the target brand’s unmodified documents.

4. Adaptive systems, user modeling, and the personalization paradox

“The Personalization Paradox: the Conflict between Accurate User Models and Personalized Adaptive Systems” defines a closely related phenomenon in adaptive HCI (Ontañón et al., 2021). The paradox arises because user modeling seeks to estimate stable user properties, while personalization intervenes in the environment and thereby changes the behavior being modeled. The paper identifies two manifestations. In feedback loops or self-reinforcing loops, the system pushes the user toward the behavior it already predicts; the Netflix and predictive-text examples illustrate how constrained exposure can make the model appear correct by limiting alternatives. In moving targets, the user actually changes because the context has changed, so the model becomes stale even if it was initially accurate.

The core mechanism is the adaptation loop: user model $\rightarrow$ adaptation $\rightarrow$ altered context/behavior $\rightarrow$ new observations $\to$ 0 updated user model (Ontañón et al., 2021). In the exergames case study, a web-based physical-activity platform uses Fitbit data and multi-armed bandits to infer social comparison preferences and adapt which comparison targets users see. The system intentionally omits competition and goals in order to remain as neutral as possible, allows users to choose among different profiles, and attempts to separate modeling from intervention by modeling expected physical activity in different contexts rather than only comparison choice. The broader methodological recommendation is to decide explicitly whether model accuracy or behavioral change takes precedence, and, where possible, to separate a neutral data-collection phase from a later intervention phase.

This formulation generalizes the paradox from benchmarking to closed-loop systems. A model can become better at personalizing while becoming less reliable as a model of the user. That is a different failure mode from the Rosetta Paradox, but it preserves the same structural inversion: context-sensitive optimization degrades the stability of the performance claim being made.

“Untangling Performance from Success” distinguishes performance from success in a setting where both can be independently measured (Yucesoy et al., 2015). Performance is operationalized by ATP ranking $\to$ 1 and tournament-related variables such as prestige $\to$ 2, matches played $\to$ 3, opponent quality $\to$ 4, and career length $\to$ 5; success is measured as Wikipedia page views. The main multiplicative model PROMO is

$\to$ 6

Most individual linear fits have $\to$ 7, except $\to$ 8 with $\to$ 9, whereas the final PROMO model reaches $\to$ 0. For retired players, the prediction reduces to

$\to$ 1

The paper’s resolution of the apparent contradiction between merit and fame is analytic separation: once performance and success are measured independently, visibility becomes largely predictable from structured performance inputs.

At the organizational level, “A ‘Distance Matters’ Paradox: Facilitating Intra-Team Collaboration Can Harm Inter-Team Collaboration” extends Olson & Olson’s Distance Matters framework from intra-team collaboration to an inter-team layer (Hu et al., 2022). The same concepts—common ground, collaboration readiness, collaboration technology readiness, coupling of work, and organizational managerial aspects—persist at the inter-team scale but are actualized differently. The central finding from the ten-month ethnography is that within-team collaboration can be excellent while between-team collaboration is poor. At PANL, the tool ecosystem contained 9 communication tools, 21 collaboration/project-management tools, and 26 data storage locations. Specialized tools and local workflows improve intra-team productivity but create inter-team interoperability and support problems. The Supply Chain case partially avoided this through embedded boundary-spanners, customer-success scorecards, and consolidation onto PeopleSoft.

At population scale, “Revisiting Rogers’ Paradox in the Context of Human-AI Interaction” extends the social-learning equilibrium to networks in which humans learn from AI systems that are themselves socially learning from humans (Collins et al., 16 Jan 2025). In the baseline model, the world changes with probability $\to$ 2, population size is $\to$ 3, individual-learning cost is $\to$ 4, individual-learning success rate is $\to$ 5, survival if adapted is $\to$ 6, and survival if not adapted is $\to$ 7. Individual learning succeeds with

$\to$ 8

The paper concludes that cheap AI-mediated social learning does not, on its own, improve long-run collective world understanding; critical social learning does improve equilibrium quality, while negative feedback can arise when repeated AI use reduces later individual-learning ability through

$\to$ 9

Taken together, these works show that context can mediate not only individual model behavior but also whether local performance converts into fame, inter-team coordination, or population-level adaptation.

6. Formal decision-theoretic and logical contextuality formulations

In decision theory, “Non-Allais Paradox and Context-Dependent Risk Attitudes” axiomatizes a representation in which the same lottery is evaluated differently depending on the probability of disappointing outcomes (Honda et al., 2024). The context variable is

$r = -0.931$ 0

where $r = -0.931$ 1 is a disappointment threshold and prizes in $r = -0.931$ 2 are “disappointing.” For each $r = -0.931$ 3, the model assigns a utility function $r = -0.931$ 4, yielding the expected contextual utility representation

$r = -0.931$ 5

The crucial claim is that independence fails not because of probability weighting alone, but because the utility function itself changes with context. Empirically, 78 of 150 subjects switched choices in Stage 1, 92 of 150 switched in Stage 2, and a logit regression in Stage 3 reports $r = -0.931$ 6. The paper states that the observed pattern is incompatible with rank-dependent utility, cumulative prospect theory, and models that preserve a single utility function while only distorting probabilities.

In quantum foundations, “The Equivalence between Hardy-type paradox and Logical Contextuality” resolves a different but formally related context-dependence problem (Liu et al., 4 Jan 2026). The paper introduces a logical Hardy-type paradox defined by events $r = -0.931$ 7 satisfying

$r = -0.931$ 8

with a probability assignment such that one event has positive probability and the others are certain. Its central theorem is

$r = -0.931$ 9

It also proves

$\rho = -0.996$ 0

The paper revisits Mansfield’s $\rho = -0.996$ 1 scenario and argues that the earlier non-equivalence claim resulted from a coarse-grained, observable-based notion of Hardy paradox rather than the event-based logical formulation. In the KCBS scenario, only one kind of Hardy-type paradox is admitted, with success probability $\rho = -0.996$ 2 for the specific realization studied.

These decision-theoretic and logical formulations are more formal than the LLM and HCI cases, but they preserve the same core lesson. Context dependence need not be dismissed as noise or inconsistency. In these models it is built into the representation itself: evaluation rules, observable contradictions, and preference orderings are conditional on the context under which they are defined.