Prism-Reranker: Beyond Relevance Scoring -- Jointly Producing Contributions and Evidence for Agentic Retrieval

Abstract: Modern retrieval pipelines increasingly serve downstream consumers like retrieval-augmented generation (RAG) and autonomous agents that need more than a scalar relevance score. A reranker that only tells the caller "how relevant" forces the agent to dump entire documents into the language-model context, wasting tokens on tangential passages and boilerplate. We introduce Prism-Reranker, a family of reranker models built on Qwen3.5 at four sizes (0.8B, 2B, 4B, 9B) that goes beyond scalar scoring. In addition to the standard yes/no relevance judgement, whenever the verdict is yes the model emits (i) a contribution statement summarizing how the document helps the query, and (ii) an evidence passage: a self-contained rewrite that preserves every query-relevant signal while discarding noise. Prism-Reranker is trained with a hybrid objective combining point-wise distillation from a strong commercial reranker API with supervised fine-tuning on contribution and evidence targets. We curate training data from KaLM-Embedding's open-source aggregation, augmented with real web documents retrieved via commercial search APIs for open-domain queries and LLM-synthesized variants, and rewrite a portion of queries into keyword-style reformulations to adapt the model to agent-issued traffic. To reconcile inconsistent labels across open corpora and obtain crisp binary supervision, we relabel data with an LLM-as-Judge ensemble aggregating votes from five frontier LLMs. On a QA subset of BEIR and on an LLM-judged evaluation of contribution and evidence quality, Prism-Reranker attains solid results across all four sizes. We further show that the same recipe extends existing LLM-based rerankers, augmenting Qwen3-Reranker-4B with contribution and evidence capabilities while improving its average BEIR-QA NDCG@10 by +1.54 over the base model. Model weights, training recipe, and evaluation suite are released.

• The paper introduces Prism-Reranker, which jointly produces relevance scores, contributions, and evidence passages to enhance agentic retrieval.
• It leverages the QWEN3.5 backbone with dual-objective training—including point-wise distillation and supervised fine-tuning—to generate structured, high-fidelity outputs.
• Experimental results show competitive ranking performance while reducing extraneous content and token usage in retrieval-augmented pipelines.

Prism-Reranker: Joint Contribution and Evidence Generation for Agentic Retrieval

Motivation and Problem Formulation

The utility of contemporary retrieval pipelines in RAG and agentic settings is fundamentally limited by reranker architectures that emit only scalar relevance scores. As downstream consumers shift from human search to autonomous agents and generative models, rerankers must not only inform which documents are relevant but also explicitly produce structured signals—namely, summaries of a document's utility to the query (contribution) and compact contextually faithful evidence passages. Prism-Reranker directly addresses three bottlenecks: wasteful over-inclusion of tangential content, inability to filter web/noisy documents, and lack of actionable planning signals for agent control flow.

Methodology

Model Architecture

Prism-Reranker leverages the QWEN3.5 backbone (available in 0.8B, 2B, 4B, and 9B parameter configurations), retaining the standard causal Transformer architecture. The model produces, in a single forward pass per (query, document) pair:

• A calibrated yes/no relevance decision, recoverable as a probability score.
• A contribution sentence describing how the document addresses the query.
• An evidence passage: a compressed, rewrite of document text strictly containing query-relevant information.

Importantly, the generation of contribution and evidence is gated by the relevance prediction, ensuring that only positively relevant documents incur generation cost.

Training Regimen

The model is trained under a dual-objective scheme:

• Point-wise Distillation: MSE regression aligns the student’s relevance score with that output by a strong commercial reranker, operating as a teacher.
• Supervised Fine-Tuning (SFT): When the binary verdict is positive, SFT supervises the joint generation of contribution and evidence. Negative cases receive just a 'no' token.

Data for SFT is generated using advanced LLMs (DEEPSEEK-V4-PRO) to ensure high-quality structured outputs.

Data Pipeline and Annotation

Training data merges open-source retrieval corpora (KaLM-Embedding aggregation with English, Chinese, and multilingual coverage), real web-crawled documents retrieved via Tavily and Exa, and keyword-style query reformulations to mimic agent-driven traffic.

To address cross-corpus labeling inconsistencies, a robust LLM-as-Judge ensemble system is formulated, involving five strong yet pairwise decorrelated LLMs. Binary labels are assigned via 3-of-5 majority decision, standardizing supervision.

Extension Recipe

The framework allows for SFT-driven augmentation of existing generative rerankers (e.g., QWEN3-RERANKER-4B), where self-distillation anchors relevance quality while SFT adds structured outputs, without dependence on a commercial teacher.

Experimental Results

Relevance Ranking

On BEIR QA-style datasets, Prism-Reranker models (0.8B–9B) attain competitive NDCG@10 scores, with scaling trends consistent with model capacity. Compared to the commercial teacher, a 2.8–5.3 point drop is observed, consistent with the increased supervised output complexity and the multi-task burden; the extension approach over a BEIR-strong reranker (Prism-Reranker-4B-exp) yields a +1.54 NDCG@10 gain over its base model, demonstrating that the joint objective can either match or surpass the anchor when based on a robust teacher.

Structured Output Quality

Assessment involves both rule-based and LLM-judged evaluation measures:

• Label accuracy (correct yes/no) climbs with model size, peaking at 0.845 for the 9B model and 0.851 for the augmented 4B-exp.
• Entity fidelity consistently exceeds 0.97, indicating minimal hallucination.
• LLM-judged metrics (faithfulness, coverage, self-containedness) range 3.1–3.8 on a 1–5 scale, validating output quality.
• Compression ratio: Evidence passages exhibit median lengths roughly 50–56% that of source documents, with maximal reduction for long, noisy web content.

Relative to a strong LLM baseline (DEEPSEEK-V4-FLASH), Prism-Reranker compactly delivers structured outputs with competitive language consistency and entity fidelity, despite operating with markedly fewer parameters.

Key Findings

• Structured signals add utility: The evidence field provides context-optimized input for downstream LLMs, reducing token consumption and filtering extraneous content, while the contribution field supplies explicit agentic planning cues absent from prior reranker architectures.
• Tradeoff between ranking and multi-task output: Direct student-teacher distillation on relevance yields teacher-comparable ranking; SFT on structured output introduces a modest ranking penalty, which can be mitigated or reversed in extension mode over strong base models.
• LLM-ensemble labeling ensures consistency: Multi-LLM judgment reconciles label conventions across training sources, improving supervision for structured output generation.
• Compression is intrinsic: Context-size reductions accrue most notably on the longest documents, directly addressing efficiency constraints in RAG and agentic architectures.

Implications and Future Directions

Prism-Reranker makes a compelling case for rerankers as multi-output models, architected for autonomous agent and RAG pipelines, not just human search. The ability to generate evidence in addition to ranking substantially streamlines context management, improves answer faithfulness, and allows agents or LLMs to operate on precisely the relevant kernel of information.

Potential future directions include:

• Reinforcement learning: Direct optimization for entity-level faithfulness and conciseness, possibly using rule-based or LLM-based reward models, holds promise for further suppressing hallucinations and optimizing evidence utility.
• Ablation studies and curriculum learning: Thorough ablation of ensemble labeling, data balancing, and mixed loss weightings can expose optimal configurations for transfer and scaling.
• Language coverage and generalization: Extending robust evaluation to low-resource languages or domain-specific corpora is feasible via expanded training with KaLM-Embedding-style aggregation.
• End-to-end agentic evaluation: Measuring downstream task success—such as QA accuracy or planning efficacy using only evidence outputs—remains a critical practical metric for deployment relevance.

Conclusion

Prism-Reranker establishes a new baseline for rerankers in agentic and retrieval-augmented pipelines, demonstrating that compact, efficient models can reliably emit both ranking signals and high-fidelity, query-specific context in one forward pass. This approach materially bridges the gap between retrieval and reasoning, lowering the cost of deploying capable, context-aware agentic systems without relying on frontier-scale LLMs or duplicative post-processing modules.

Model checkpoints, full training, and evaluation recipes are released to the community to advance further work on practical, structured-output ranking in agentic information retrieval (2604.23734).