OBLIQ-Bench: Exposing Overlooked Bottlenecks in Modern Retrievers with Latent and Implicit Queries

Abstract: Retrieval benchmarks are increasingly saturating, but we argue that efficient search is far from a solved problem. We identify a class of queries we call oblique, which seek documents that instantiate a latent pattern, like finding all tweets that express an implicit stance, chat logs that demonstrate a particular failure mode, or transcripts that match an abstract scenario. We study three mechanisms through which obliqueness may arise and introduce OBLIQ-Bench, a suite of five oblique search problems over real long-tail corpora. OBLIQ-Bench exposes an overlooked asymmetry between retrieval and verification, where reasoning LLMs reliably recognize latent relevance whenever relevant documents are surfaced, but even sophisticated retrieval pipelines fail to surface most relevant documents in the first place. We hope that OBLIQ-Bench will drive research into retrieval architectures that efficiently capture latent patterns and implicit signals in large corpora.

• The paper demonstrates the retrieval–verification gap where LLM verifiers vastly outperform first-stage retrievers on oblique, latent queries.
• It introduces a novel benchmark with descriptive, analogue, and tip-of-the-tongue tasks to assess the challenges in implicit attribute retrieval.
• Quantitative results reveal major limitations of current IR architectures, motivating the development of models that internalize latent signal extraction.

OBLIQ-Bench: Exposing Retrieval–Verification Asymmetry in Latent and Implicit Query Regimes

Problem Formulation and Motivation

OBLIQ-Bench identifies and formalizes a critical deficiency in current scalable IR systems: the inability to surface relevant documents for queries defined via latent or implicit attributes, even as reasoning LLMs reliably verify such relevance given direct exposure to candidate documents. Existing benchmarks, including MS MARCO, BEIR, and BRIGHT, are largely saturated and fail to expose evaluation difficulty genuinely rooted in surfacing latent structure. The authors introduce the concept of oblique queries—search requests whose relevance is predicated on implicit, abstract, or analogical relationships, rather than direct lexical or topical cues. This work amplifies the retrieval–verification asymmetry: verification-models (i.e., LLM rerankers with full query-document context) can separate relevant instances once surfaced, but first-stage retrieval systems cannot efficiently produce candidate pools containing those instances in the first place.

Figure 1: OBLIQ-Bench tasks are characterized by a large gap between retriever and verifier performance—unlike prior benchmarks where this gap is small or absent.

Task Suite and Modes of Obliqueness

OBLIQ-Bench encompasses five search tasks spanning long-tail, real-world corpora with three distinct categories of oblique information needs:

• Descriptive Queries: Targeting implicit stances or behavioral features not explicitly stated (e.g., tweets indicating a stance only via irony or conversations manifesting LLM failure modes unnoted in their transcripts).
• Analogue Queries: Seeking analogues sharing abstract structure or reasoning techniques (e.g., math problems solvable by the same proof idea, or text snippets sharing an author's style across topics).
• Tip-of-the-Tongue: Retrieving a specific passage from a corpus based on a partial, indirect recollection, deliberately omitting direct surface clues (typically for obscure congressional hearing transcripts).

  Figure 2: Examples of the three query types deployed in OBLIQ-Bench, each verifiable by reasoning models but systematically unreachable to current retrieval stack.

The construction pipeline involves human definition of a latent label, a full corpus-wide annotation pass via an LLM under that lens, clustering in attribute space, and LLM-driven generation of abstract queries. This enables large-scale, high-recall, and robust relevance labeling for latent tasks.

Figure 3: Schematic of the corpus annotation, clustering, query generation, and pooling workflow used to enable oblique benchmark creation.

Retrieval–Verification Gap Quantification

The core empirical contribution is the operationalization of the retrieval–verification gap: $V_t - R_t$, with $V_t$ the score of a prohibitively expensive re-ranker (e.g., listwise GPT-5.2, fed large pools), and $R_t$ the best attainable by scalable retrievers or agentic multi-hop pipelines.

Scaling the pool of candidate documents reveals that, in OBLIQ-Bench, even as the LLM verifier's head scores approach their attainable upper bound quickly when golds are forcibly injected, single-stage and agentic retrievers' recall remains poor. Notably, multi-hop agents close little of this gap in tasks where the relevance signal is highly abstract or orthogonal to query surface.

Figure 4: As the pool size grows, the LLM verifier rapidly separates positives, while strong dense retrievers (Gemini-2-Embedding) plateau at substantially lower recall—revealing the core bottleneck.

Quantitative Results

Across all five OBLIQ-Bench tasks, the best dense (Gemini-2-Embedding), late interaction (LateOn), and LLM-augmented agentic retrievers achieve NDCG@10 scores well below 0.22. By contrast, the Oracle GPT-5.2 Tournament reranker yields 0.33–0.51 (NDCG@10, pooled) in descriptive/analogue cases and up to 0.91 (NDCG@10, Recall@10=0.96) on Tip-of-the-Tongue, given candidate pools containing the true golds. Notably, Writing-Style (authorship analogues) is an outlier where BM25 outperforms some larger dense models, since stylistic cues can leave lexical residue even across topics, but even here no scalable retriever is competitive with the verifier.

In tasks such as Congress Hearings Tip-of-the-Tongue, all classic and modern retrievers (BM25, Gemini, Qwen, LateOn, query rewrites, and multi-hop agents) yield near-zero recall at 10 and 50 returned passages. Only the LLM verifier, with golds forcibly injected, approaches perfect recovery. This empirical reality signals a regime for which both representation learning and iterative agentic query rewriting are inadequate; no surface decomposition or parametric prior can bridge the gap when the target is specified by latent or multimodal constraints.

Architectural and Modeling Implications

Several core insights emerge:

• Current indexing architectures (dual-encoder, late-interaction, or hybrid retrieval) are structurally incapable of exposing dense signals for latent or implicit attributes absent explicit supervision or multi-stage processing integrating full-query–document context.
• Agentic retrieval (multi-hop query generation and feedback) provides modest gains when alternative phrasings can heuristically approximate the relevance set, but is generally ineffective for distributed or highly abstract meta-labels (e.g., behavioral error types in conversations or authorial style).
• Training data regimes and domain adaptation matter: Gemini-2-Embedding shows clear improvements over open-source baselines in several tasks, suggesting proprietary pretraining and curation is impactful for certain forms of non-oblique reasoning.
• Late-interaction models outperform in cases where weak surface structure connects query and golds (Congress), but falter otherwise.

The results collectively argue for new retrieval architectures that internalize latent signal extraction and align corpus indexing with implicit attribute access—possibly via retrieval-augmented segmentation, modular reasoning, or scalable multi-hop search with long-range cross-document attribute propagation.

Theoretical and Practical Implications

The retrieval–verification asymmetry posed by OBLIQ-Bench undermines the validity of current IR benchmarks as a proxy for information-seeking or reasoning in real-world downstream tasks, especially those involving policy audit, attribution, or “unknown unknown” discovery. Future research directions include:

• Development of retrofittable or train-time procedures for infusing latent or implicit attribute supervision into retriever representations.
• Hybrid cascades that elevate more compute-intensive, reasoning-aware filtering to earlier stages, at scalable cost.
• Exploration of dynamic, corpus-wide annotation and clustering, using LLMs or hybrid neuro-symbolic systems, followed by query-time alignment to derived latent taxonomies.

Conclusion

OBLIQ-Bench exposes the intrinsic limitations of current scalable retrieval systems along dimensions relevant to implicit, abstract, or analogical information needs. By operationalizing the retrieval–verification gap, it provides both a diagnostic and a directive for future research. As reasoning LLMs become trivial verifiers over surfaced candidates, retrieval architectures must be designed to surface candidates with respect to attributes that are not easily indexed or represented by shallow correlation with query surface. This work establishes that latent attribute retrieval is a rigorous and unsolved challenge, motivating architectural innovations that may realign IR progress with human-centered search and discovery.