Long-Term Optimization for Large-Scale Generative Retrieval with Off-Policy REINFORCE

Abstract: Generative retrieval has become a popular paradigm for large-scale recommendation. However, it is typically trained with supervised next-item prediction objectives that do not directly optimize long-term user satisfaction. In this work, we formulate recommendation as a session-level sequential decision-making problem and introduce an autoregressive approach for training generative retrievers with off-policy REINFORCE on pre-collected data. Unlike the one-step off-policy correction used in prior work, we propose a multi-step approximation of importance weights enabled by the autoregressive formulation. To support offline evaluation, we train a user feedback model that simulates user responses to generated recommendations. This lets us adapt doubly robust off-policy evaluation for sequential decision-making to recommendation, a setting that has received limited attention. We further introduce a feedback-model-based test-time scaling procedure that simulates future responses and selects recommendations with the highest predicted long-term returns. Experiments on the public large-scale Yambda-5B dataset show that our RL agent improves offline estimates of cumulative session reward over next-item and next-positive prediction baselines, while largely preserving retrieval quality. Moreover, allocating more inference-time compute to simulating future responses improves model-based long-term return estimates without updating the policy.

• The paper introduces a novel autoregressive off-policy REINFORCE pipeline to optimize long-term user satisfaction in generative retrieval.
• It leverages multi-step importance sampling and a dedicated user feedback model to simulate and maximize cumulative rewards over recommendation sessions.
• Empirical results on the Yambda-5B dataset confirm robust improvements in offline evaluation metrics while maintaining competitive recall.

Long-Term Optimization for Large-Scale Generative Retrieval with Off-Policy REINFORCE

Introduction and Motivation

Large-scale recommendation systems are predominantly trained using supervised, next-item prediction objectives that offer only myopic optimization based on immediate feedback. Such approaches are agnostic to the longer-term user satisfaction, a limitation that has motivated increasing interest in RL-based frameworks for recommender systems. The paper "Long-Term Optimization for Large-Scale Generative Retrieval with Off-Policy REINFORCE" (2607.02818) addresses this gap by adopting a session-level, sequential decision-making protocol, casting the next-item recommendation process as a Markov Decision Process (MDP). Rather than optimizing solely for instantaneous proxies (clicks, likes), the approach aims to maximize the cumulative discounted reward over recommendation sessions.

The key innovation lies in developing an autoregressive off-policy REINFORCE pipeline for generative retrieval, leveraging multi-step importance sampling for enhanced off-policy correction, and integrating an explicit user feedback model for both test-time scaling and rigorous offline evaluation.

Methodology

Session-Level RL Formulation

Recommendation is framed as a trajectory-based RL problem where the state encapsulates the user's interaction history and the action corresponds to the next track recommendation. The reward signal comprises both binary like indicators and listened ratios, aggregated as $r_t = \mathbbm{1}\{like_t\} + \frac{1}{10} ratio_t$, supporting dense reward propagation across long sessions.

Off-Policy Autoregressive REINFORCE

Contrary to previous approaches that leverage one-step off-policy correction [chen2018topk], this work performs autoregressive off-policy REINFORCE (Figure 1). Both the behavior and target policies are parameterized with identical causal transformer decoders, with the target network initialized from the trained behavior policy. Trajectories collected under the behavior policy are processed in a single forward pass rather than state-by-state, enabling sequence-level importance weighting.

Figure 1: Autoregressive off-policy REINFORCE training, utilizing a frozen transformer to estimate behavior policy probabilities and applying REINFORCE loss to the target policy transformer on session trajectories.

Multi-step importance weighting is achieved by retaining up to $K$ policy-ratio factors per step, with gradient variance controlled by capping the weights. The optimal $K$ is determined via ablation, with $K=10$ yielding the best tradeoff between performance and variance.

User Feedback Model for Model-Based Evaluation

A central component is the user feedback model—a causal transformer trained to predict both the probability of a user liking a recommendation and the listened ratio (Figure 2). This model is critical for two tasks: (1) direct model-based estimation of discounted returns for new policies and (2) simulating user dynamics for multi-step roll-outs in test-time scaling.

Figure 2: User feedback model architecture, with separate scalar heads for like probability and listened ratio for each candidate, enabling reward prediction for model-based evaluation.

Multi-Step Test-Time Scaling

To exploit the learned user feedback model, the inference pipeline is augmented with multi-step test-time scaling (Figure 3). For a given user state, the system samples multiple $T$-step roll-outs from the policy, simulates feedback/reward trajectories via the user feedback model, and ranks candidate first actions by predicted cumulative discounted return. The item with the highest modeled long-term return is selected as the recommendation.

Figure 3: Multi-step test-time scaling procedure with roll-out generation and scoring using a learned user feedback model, returning the first action from the highest-scoring trajectory.

Off-Policy Evaluation and Baselines

Model evaluation leverages three OPE estimators tailored for sequential, multi-step settings:

• Stepwise Inverse Propensity Scoring (Step-IS)
• Direct Method (DM) using the feedback model for simulated roll-outs
• Doubly Robust (DR) estimator combining historical rewards and model predictions for minimum bias/variance

Baselines include supervised next-item prediction (NIP-CE) and a positive-only candidate generator (Positive-CE), both matched to the RL agent in terms of model architecture and compute.

Empirical Evaluation

Data and Protocol

Experiments are conducted on Yambda-5B [ploshkin2025yambda], a public dataset with 5B user interactions, partitioned to prevent leakage. All models share the same training budget and are evaluated with retrieval (recall@100) and OPE metrics.

Results: Long-Term Optimization

The RL agent optimized with multi-step off-policy REINFORCE achieves superior discounted session return estimates (OPE) relative to both supervised baselines, without significantly degrading recall@100 (Table 1 in the paper). The improvement persists across all OPE estimators, with the DR estimator providing the most robust offline value estimate due to its theoretical bias correction properties.

Ablation Studies

Ablation on importance weighting demonstrates:

• Off-policy correction is indispensable: Removing importance weighting (i.e., training as if on-policy) significantly degrades all OPE metrics even if recall@100 on high-return states may appear superficially higher.
• Multi-step correction: Expanding from single- to multi-step factors in importance weighting yields monotonic improvements up to $K=10$.
• Clipping: Confirms a bias-variance tradeoff, with moderate importance weight clipping leading to optimal RL policy performance.

Test-Time Scaling

The efficacy of multi-step test-time scaling is pronounced: with more roll-outs at inference, the predicted model-based (DM) return increases consistently for all models, including supervised baselines (Figure 4). This validates the practical value of test-time compute for post-hoc reward maximization and suggests that explicit RL optimization and model-based roll-out reranking are complementary strategies.

Figure 4: DM estimate as a function of the number of inference-time roll-outs, demonstrating improved model-based long-term return for all models with increasing rollout budget.

Practical and Theoretical Implications

This work strengthens the feasibility of long-term reward optimization for large-scale generative recommenders under realistic, production-scale data constraints. The autoregressive off-policy REINFORCE framework enables efficient learning with trajectory-level multi-step importance correction, mitigating the distributional shift and optimizing for true long-horizon user utility rather than short-term proxies.

The deployment of a user feedback model unlocks not only improved OPE and candidate reranking but also provides a pathway towards unified architectures with a single transformer backbone—amalgamating candidate generation, value estimation, and test-time reranking.

The main limitation remains the reliance on offline evaluation; future work must empirically validate the correlation between OPE improvements and online A/B test gains, especially given the known deficiencies of offline metrics for evaluating long-horizon interventions in recommender systems.

Future Directions

Anticipated developments include:

• Online validation of OPE improvements via A/B testing in live recommender systems, specifically for session-based short-video recommendation.
• Scaling model capacity and analyzing the effect of increased test-time compute in capturing long-term reward via roll-out expansion.
• Exploring unified, multi-head architectures integrating behavior, target, and reward models for end-to-end inference and learning efficiency.

Conclusion

The paper introduces a technically rigorous, empirically validated methodology for long-term session-level optimization in generative retrieval recommenders. The combination of autoregressive off-policy REINFORCE, multi-step importance correction, a powerful user feedback model, and computation-efficient test-time scaling constitutes a practical toolkit for both academic research and industrial deployment in sequential recommender optimization (2607.02818).