RICE-PO: Retrieval Interaction Policy Optimizer

• RICE-PO is a critic-free policy optimization framework for interactive retrieval, deriving training signals from retrieval interaction structure.
• It employs uncertainty-guided anchor selection, local counterfactual branching, and gated credit propagation to precisely assign rewards.
• The approach outperforms standard RL baselines by improving credit assignment in query-side retrieval using a fixed retriever setup.

RICE-PO, short for Retrieval-Interaction Credit Estimation for Policy Optimization, is a critic-free policy optimization framework for interactive retrieval agents in which training signal is derived from the structure of the retrieval interaction itself rather than from a learned value model, a process reward model, or an external judge. It is designed for query-side interactive retrieval settings where a language agent alternates between latent reasoning, executable summaries or queries, and retrieved evidence, and where the main training difficulty is credit assignment: executable actions can be scored directly by retrieval metrics, whereas latent reasoning spans influence retrieval only indirectly through subsequent summaries. RICE-PO addresses this asymmetry by selecting uncertain executable actions as anchors, evaluating local counterfactual branches, and propagating credit to latent reasoning only when the local signal is both influential and stable under future interaction (Li et al., 25 May 2026).

1. Problem formulation and motivation

RICE-PO is formulated for query-side interactive retrieval. Given an initial input

$x=(q_0,D_0),$

the agent operates over multiple steps. At step $t$, it observes the original query $q_0$, the previous retrieved set $D_{t-1}$, and the interaction history $h_{t-1}$, then generates a latent reasoning segment $z_t \in \mathcal{V}^{n_z}$ and an executable summary segment $s_t \in \mathcal{V}^{n_s}$. The summary is issued to the retriever as the effective query:

$D_t=\mathcal{R}(s_t).$

A trajectory therefore has the form

$\tau=\{(z_1,s_1,D_1),\ldots,(z_T,s_T,D_T)\},$

under a policy

$\pi_\theta(z_t,s_t \mid q_0,D_{t-1},h_{t-1}),$

with a final scalar reward $t$0 such as NDCG@10 (Li et al., 25 May 2026).

The motivation is the mismatch between what is directly observable and what is causally important. A summary action can be evaluated immediately by the retriever, but a reasoning span is latent and affects retrieval only through the neighboring summary and later interaction. Uniformly broadcasting the final trajectory reward to all reasoning tokens is therefore unreliable. The same final reward may assign credit to reasoning steps that did not actually shape retrieval success, especially when later turns recover from or overwrite earlier mistakes. RICE-PO is built around the premise that this asymmetry requires selective backward credit propagation rather than indiscriminate outcome-level reward assignment.

2. Interaction-derived credit estimation

The algorithm has three stated stages: uncertainty-guided anchor selection, local counterfactual branching and evaluation, and gated credit propagation. The corresponding diagnostics are policy uncertainty, reasoning-to-summary influence, and future residual stability (Li et al., 25 May 2026).

Anchor selection begins by pooling summary actions from trajectories sampled for the same query and ranking them by average token entropy. For a summary $t$1 in trajectory $t$2, entropy is defined as

$t$3

and the anchor set is the top-$t$4 summaries by this score:

$t$5

The stated rationale is that high-uncertainty executable decisions are both more informative to inspect and cheaper to analyze than every step in every trajectory.

For each selected summary anchor, RICE-PO constructs local counterfactual branches from the same prefix. At trigger point $t$6, it fixes the history before the reasoning action and generates $t$7 local branches in addition to the original branch. Each branch produces a reasoning span $t$8, its neighboring summary $t$9, and the corresponding retriever response. The summary-level local reward is

$q_0$0

computed by using the generated summary together with the input query to retrieve documents with a fixed retriever and then scoring the result with NDCG@10. The Monte Carlo estimates are

$q_0$1

The variance $q_0$2 is interpreted as a proxy for local sensitivity: if alternative reasoning spans under the same history produce strongly different summary rewards, then the reasoning step materially influences the executable action.

RICE-PO then estimates future residual effects by rolling each branch out to the final depth and comparing final reward with local summary reward:

$q_0$3

with residual variance

$q_0$4

A small $q_0$5 indicates that later interaction has similar residual effect across branches, so future turns are not strongly reordering or canceling the local difference. Credit is copied backward from summary to reasoning only if

$q_0$6

The paper explicitly states that these Monte Carlo statistics are not independent rewards; they determine only how strongly summary-level credit should be propagated to the paired reasoning span.

3. Advantage construction and PPO-style optimization

RICE-PO begins from a group-normalized trajectory advantage $q_0$7 computed from the final-step retrieval reward, specifically the NDCG@10 of the last summary in trajectory $q_0$8. This trajectory-level advantage is broadcast to all tokens except those belonging to selected summary or reasoning spans. Selected summary tokens receive a summary-level advantage $q_0$9, while reasoning tokens receive either a copied-back reasoning advantage $D_{t-1}$0 or, if the local signal is weak or unstable, the original trajectory-level advantage (Li et al., 25 May 2026).

The defining mechanism is therefore not reward shaping in the usual sense, but token-level reassignment of advantage according to local interaction evidence. If the estimated reasoning-to-summary influence is high and the residual future effect is stable, then reasoning tokens inherit the summary-level signal. Otherwise, they remain attached to the coarser trajectory-level signal. This prevents latent reasoning from automatically inheriting local credit when the local branch analysis suggests that later interaction may have overridden the effect.

Optimization uses a PPO-style clipped objective with token-level importance ratios

$D_{t-1}$1

and a KL penalty weighted by $D_{t-1}$2. The formulation preserves the standard clipped-policy structure while replacing uniformly assigned token advantages with localized advantages constructed from trajectory-level, summary-level, and gated reasoning-level signals.

The theoretical justification is stated as Summary-to-Reasoning Credit Approximation. If

$D_{t-1}$3

where $D_{t-1}$4 is the summary retrieval reward and $D_{t-1}$5 is the remaining future effect, then

$D_{t-1}$6

If the residual term is bounded, summary-level credit approximates reasoning-level credit. This is the formal basis for copying summary credit backward only when future residual effects are sufficiently controlled. A plausible implication is that RICE-PO treats executable summaries as empirically observable “credit anchors” and latent reasoning as a secondary object whose supervision must be justified rather than assumed.

4. Experimental setting

The empirical evaluation uses two benchmark suites. BRIGHT comprises 12 reasoning-intensive retrieval tasks, spanning StackExchange, LeetCode, TheoremQA, and domains including biology, economics, programming, and mathematics. BEIR is instantiated with five datasets: DBPedia-Entity, FiQA-2018, SciFact, Touché-2020, and TREC-COVID. The main metric throughout is NDCG@10 (Li et al., 25 May 2026).

A central design choice is the fixed retriever setting. For all RL-based methods, the retriever is held constant as Diver-4B + BM25, following Diver. The claim under evaluation is therefore about improving the interaction policy under a common retrieval backend rather than about changing the retriever itself. This makes the comparison specifically about credit assignment and policy learning.

The baseline set spans both prompted systems and RL methods. On BRIGHT, comparisons include RaDeR, DeepRetrieval, TongSearch-QR, Diver, and the RL baselines GRPO, GiGPO, HGPO, and Tree-GRPO. On BEIR, comparisons include BM25S, MonoT5-3B, RankLLaMA-7B/13B, Rank1-7B/14B/32B, together with the same RL baselines. RL evaluations use DeepSeek-R1-Distill-Qwen-1.5B, Qwen3-4B-Thinking-2507, and, on BEIR, Qwen2.5-3B-Instruct.

The appendix-level training configuration reported for RICE-PO is specific: actor KL coefficient $D_{t-1}$7, top $D_{t-1}$8 summary anchors by entropy, 5 Monte Carlo branches per selected anchor, influence threshold $D_{t-1}$9, residual-effect threshold $h_{t-1}$0, batch size $h_{t-1}$1, actor learning rate $h_{t-1}$2, and maximum rollout depth of 5 retrieval steps. BRIGHT-style training data are constructed from ReasonIR with sampled splits of 4,011 train / 1,384 test for BRIGHT and 4,008 train / 1,447 test for BEIR.

Setting	BRIGHT	BEIR
Scope	12 reasoning-intensive retrieval tasks	5 datasets
Main metric	NDCG@10	NDCG@10
Fixed retriever	Diver-4B + BM25	Diver-4B + BM25

5. Quantitative performance and ablation results

On BRIGHT, the strongest prompt-based baseline is Diver, which averages 37.20 with the 14B summarizer/retrieval pipeline. RICE-PO reports 27.49 average with DeepSeek-R1-Distill-Qwen-1.5B and 37.76 average with Qwen3-4B-Thinking-2507. With the 4B backbone, the method exceeds Diver’s average of 37.20 and improves over GRPO, GiGPO, HGPO, and Tree-GRPO by about 1.1–1.5 points depending on the comparison. With the 1.5B backbone, it outperforms the RL baselines and substantially improves over plain prompting; the paper presents this as evidence that interaction-derived credit assignment can compensate for smaller model size (Li et al., 25 May 2026).

On BEIR, RICE-PO achieves the best reported average on both evaluated backbones: 65.66 with DeepSeek-R1-Distill-Qwen-1.5B and 65.68 with Qwen2.5-3B-Instruct. The reported gains over the RL baselines are smaller than on BRIGHT. The stated interpretation is that BEIR tasks are less reasoning-intensive and depend more on surface matching, so the advantage of localized reasoning credit is present but attenuated.

The ablation results isolate the contribution of the two main design choices. For entropy-based triggering versus random triggering, the average BRIGHT score is 27.49 versus 25.82. The paper further reports that 9 out of 12 BRIGHT tasks benefit from entropy-based triggering, with a median gain of about $h_{t-1}$3. Tasks such as TheoremQA-Questions and Psychology are described as gaining substantially. This supports the specific role of uncertainty as a branching-budget allocator rather than merely as a descriptive statistic.

A second ablation evaluates the gating criteria. On BRIGHT, Effect Only gives 25.89, Influence Only gives 26.56, and Both (Ours) gives 27.49. The reported interpretation is that the two gates are complementary: influence-only helps when intermediate summaries matter, effect-only helps when local changes survive later turns, and the conjunction is best because it avoids propagating noisy or overwritten local credit. A further case-selection ablation compares several ways of assigning reasoning credit—current-step summary reward, final-summary reward, random selection between them, and RICE-PO’s gated propagation—and finds the gated method best. This result is used to argue that neither always-local nor always-final reward assignment is adequate for latent reasoning supervision.

6. Relation to adjacent RL approaches, limitations, and scope

RICE-PO is positioned explicitly against three families of methods. Relative to critic-based RL, it is critic-free: it does not train a value model, a process reward model, or an external judge. Relative to group-based RL baselines such as GRPO, GiGPO, HGPO, and Tree-GRPO, it does not primarily redistribute final trajectory reward; instead, it changes the source of credit by using executable summaries as intermediate anchors, constructing local counterfactual branches from the same history, and gating backward propagation with influence and residual-stability tests. Relative to prompt-based agents such as Diver and ThinkQE, it is trained from retrieval interaction rather than relying on fixed prompting strategies at inference time (Li et al., 25 May 2026).

The paper’s stated takeaways are correspondingly narrow and technical. First, retrieval interaction contains usable intermediate supervision because executable summaries are directly evaluable. Second, latent reasoning should not automatically inherit final reward; copied-back credit is warranted only when the reasoning-to-summary relationship is strong and future residual effects are stable. Third, uncertainty is a good trigger for local analysis, since entropy-based anchor selection improves the usefulness of counterfactual branching. Fourth, critic-free localized learning can outperform standard RL baselines under a fixed retriever.

The limitations are also explicit. The framework uses a fixed retriever, depends on a bounded local branching budget, and leaves open the extension to adaptive retrievers, larger backbones, and more efficient estimators of influence and residual stability. A plausible implication is that the method is best understood not as a general solution to long-horizon reasoning credit assignment, but as a retrieval-specific framework that exploits the executable–latent asymmetry of interactive search. Within the provided literature summaries, the label RICE-PO is also applied to a motion-coupled canopy-mapping algorithm for hybrid rice fields (Feng et al., 22 Feb 2025). This suggests that the acronym is not unique across domains; in retrieval research, however, the established expansion is Retrieval-Interaction Credit Estimation for Policy Optimization.