PRInTS: Reward Modeling for Long-Horizon Information Seeking (2511.19314v1)

Abstract: Information-seeking is a core capability for AI agents, requiring them to gather and reason over tool-generated information across long trajectories. However, such multi-step information-seeking tasks remain challenging for agents backed by LLMs. While process reward models (PRMs) can guide agents by ranking candidate steps at test-time, existing PRMs, designed for short reasoning with binary judgment, cannot capture richer dimensions of information-seeking steps, such as tool interactions and reasoning over tool outputs, nor handle the rapidly growing context in long-horizon tasks. To address these limitations, we introduce PRInTS, a generative PRM trained with dual capabilities: (1) dense scoring based on the PRM's reasoning across multiple step quality dimensions (e.g., interpretation of tool outputs, tool call informativeness) and (2) trajectory summarization that compresses the growing context while preserving essential information for step evaluation. Extensive evaluations across FRAMES, GAIA (levels 1-3), and WebWalkerQA (easy-hard) benchmarks on multiple models, along with ablations, reveal that best-of-n sampling with PRInTS enhances information-seeking abilities of open-source models as well as specialized agents, matching or surpassing the performance of frontier models with a much smaller backbone agent and outperforming other strong reward modeling baselines.

• The paper introduces PRInTS which leverages dense information gain estimation and recursive summarization to improve decision-making over long trajectories.
• It achieves significant performance gains, with up to a +9.3% improvement on benchmarks and superior sample efficiency using only 1–2k training samples.
• The approach efficiently compresses extended contexts, providing robust reward signals for agentic LLMs without the pitfalls of context explosion.

PRInTS: Reward Modeling for Long-Horizon Information Seeking

Motivation and Limitations of Existing Process Reward Models

Long-horizon information-seeking represents a critical challenge for agentic LLM architectures. Tasks in domains such as mathematical problem solving, software engineering, and research demand agents that can operate over thousands of tokens, reason over complex tool outputs, and make planning decisions in highly multimodal, evolving contexts. While prior process reward models (PRMs) have yielded significant headway in guiding agentic reasoning via step-level evaluations, they remain fundamentally limited in two respects. First, they evaluate only short, isolated reasoning snippets or tool calls, typically with binary judgments that fail to capture the fine-grained, multi-dimensional nature of decision quality over full trajectory steps that include tool interactions and subsequent reasoning updates. Second, as the context window grows with each step, accumulated trajectories introduce significant context explosion, which existing PRMs are ill-equipped to compress, leading to noisy, inefficient evaluation and diminished reward signal.

These deficits are highlighted in a comparative schematic:

Figure 1: Comparison of information-seeking challenges for existing PRMs and the PRInTS framework.

The PRInTS Framework: Joint Scoring and Summarization

PRInTS (Progress Reward via Information gain scoring and Trajectory Summarization) directly addresses these issues by providing a unified, generative PRM with two tightly integrated capabilities:

1. Dense Step-Level Scoring via Information Gain Estimation: PRInTS scores each candidate trajectory step—comprising reasoning and tool invocation—along multiple quality dimensions, including informativeness, uncertainty resolution, and the interpretability of tool outputs. This scoring is implemented as an information gain estimate, concretely defined as the marginal change in answer accuracy induced by each step, computed with Monte Carlo rollouts over an agentic environment.
2. Trajectory Summarization for Context Compression: Rather than supplying the entire unbounded context trajectory, PRInTS recursively compresses the growing history into a bounded, dense summary that retains all essential information required for downstream step evaluation while filtering noise and redundancy.

The operational flow of PRInTS is depicted below:

Figure 2: PRInTS serving both as a step scorer (left, evaluating candidates and selecting the highest-value step) and a recursive summarizer (right, maintaining a compact trajectory representation).

Data Annotation, Training Pipeline, and Reward Composition

PRInTS is trained on an automatically constructed corpus of annotated candidate step pairs and recursively generated trajectory summaries. For each possible agent step within a trajectory, PRInTS estimates information gain via Monte Carlo rollouts (measuring pre/post step accuracy differentials), then constructs preference pairs between higher and lower gain steps. The reward model is trained with composite rewards: a scalar score reward for direct regression of information gain, and a pairwise comparison loss for relative ranking. Notably, an adaptive weighting schema accentuates signal from step pairs with large, reliable gain differences while mitigating the influence of label noise for ambiguous pairs.

The data annotation and dual-task training regime are shown in:

Figure 3: End-to-end PRInTS data annotation and training pipeline integrating both score and summary supervision.

Distributional properties of information gain labels further ensure balanced, high-fidelity reward signal for robust model optimization.

Figure 4: Distribution of PRInTS-annotated information gain across trajectory steps, ensuring a robust margin between winning and losing preference pairs.

Experimental Results

Extensive empirical validation is performed with three distinct LLM-agent architectures (Qwen3-32B, Tongyi DeepResearch-30B-A3B, Gemini-2.5-Flash) and across the FRAMES, GAIA (Levels 1–3), and WebWalkerQA benchmarks. Key findings:

• Strong Numerical Improvements: PRInTS consistently surpasses prior step-level evaluation methods. On Qwen3-32B, PRInTS confers an average +9.3% absolute gain (from 29.5% to 38.8%) over baseline. DeepResearch-30B-A3B benefits by +3.9% absolute. Frontier agent Gemini-2.5-Flash also sees a +4.0% boost.
• Outperforms Advanced Baselines: Unlike standard PRMs (StepWiser, GenPRM, Web-Shepherd), PRInTS maintains strong, positive gains as agents become “stronger,” not suffering from reward saturation.
• Sample Efficiency: Comparable performance improvements are attained using only 1–2k training samples, contrasting with the 10k–100k data regime of full agent fine-tuning.

Additional ablations demonstrate the criticality of context compression (summarization) for accurate evaluation and establish that combined regression/ranking reward (with adaptive weighting) yields superior generalizability compared to pure regression or pure ranking objectives.

PRInTS exhibits strong best-of-n test-time scaling, reliably leveraging additional candidate computation to select superior reasoning steps.

Figure 5: PRInTS best-of-n scaling curves on GAIA, demonstrating further performance increase with additional candidate evaluation.

Moreover, the method scales favorably with training data:

Figure 6: PRInTS achieves substantial performance even with reduced annotation set sizes, exhibiting strong sample efficiency.

Trajectory Summarization and Dual Ability Synergy

PRInTS's joint training of step-wise scoring and recursive summarization enables it to maintain evaluation quality as context grows, outperforming PRMs trained only for scoring (without summarizing) or hybrid approaches where summarization and scoring are separated. The unified approach yields better transfer and regularization between trajectory compression and quality estimation.

Summarizer prompt structures and evaluation behavior are provided for transparency:

Figure 7: PRInTS scorer prompting for step evaluation during test-time.

Figure 8: PRInTS summarizer prompting for context update.

Qualitative examples further illustrate the ability to differentiate high-quality steps (that acknowledge epistemic uncertainty and elicit informative tool interaction) from low-quality steps (overconfident, unsupported guesses):

Figure 9: PRInTS distinguishes between effective uncertainty-resolving actions and unsupported, spurious outputs.

Implications and Future Directions

• Practical: PRInTS can be integrated as a plug-in reward model for any agent architecture without fine-tuning the backbone model. This enables fine-grained, data-efficient test-time selection and improvement for information-seeking agents in environments with unbounded context and multimodal tool interaction.
• Theoretical: By framing step value in terms of information gain, PRInTS ties reward modeling to a principled utility assignment compatible with information-theoretic and probabilistic approaches to reasoning. The summarization component can be developed into a more general context management layer for scalable agentic reasoning.
• Future Directions: Potential advancements include extending PRInTS to other agent architectures, incorporating richer tool chains (including multimodal or embodied interactions), combining trajectory compression with abstraction-based planning, and enhancing preference pair sampling objectives. Integration with policy optimization pipelines and autoregressive planning for agents is also promising.

Conclusion

PRInTS introduces a robust, scalable reward modeling paradigm for long-horizon information-seeking, combining dense information gain estimation with recursive trajectory summarization within a single generative PRM framework. It demonstrates consistent, state-of-the-art improvements across open-source and frontier agents, offering strong sample efficiency and extensibility. PRInTS’s architecture and methodology represent a substantial advance in the design of test-time reward models for agentic LLMs operating under complex, evolving contexts (2511.19314).