RecZero: RL-Driven Autonomous Recommender

• The paper introduces a novel RL-based paradigm that eliminates the teacher–student pipeline by training a single LLM to autonomously develop multi-step reasoning.
• It employs a 'Think-before-Recommendation' prompt that decomposes rating prediction into user analysis, item feature extraction, compatibility matching, and final rating decision.
• RecZero, and its hybrid variant RecOne, demonstrably outperform traditional baselines on rating benchmarks, showcasing faster convergence and improved accuracy.

RecZero is a reinforcement learning-based paradigm for recommender systems, specifically designed to transcend the limitations of distillation-driven reasoning transfer by enabling LLMs to autonomously develop reasoning skills for rating prediction. The hallmark of RecZero is the elimination of the teacher–student pipeline, instead training a single LLM end-to-end via RL to generate, optimize, and apply multi-step reasoning about user preferences and item features in recommendation tasks. Beneath its surface-level architecture, RecZero systematically unifies structured reasoning, reward-driven optimization, and integration of user–item compatibility signals in a cohesive framework that demonstrably outperforms traditional baselines on rating prediction benchmarks.

1. Rationale and Framework Design

RecZero was conceived to address intrinsic drawbacks in LLM-based recommender systems that rely on teacher model reasoning traces. These systems typically suffer from:

• Teacher models lacking domain specialization, resulting in subpar reasoning quality.
• Resource costs and rigid supervision associated with generating static reasoning traces.
• Superficial transfer of reasoning ability, failing to cultivate genuine reasoning autonomy.

RecZero abandons the multi-stage distillation approach. Instead, it utilizes a pure RL setup where the entire reasoning process—from user analysis to final rating decision—is optimized in one model. This enables the LLM to learn reasoning heuristics natively in the context of the recommendation task, robustly adapting to rating objectives and user-item compatibility nuances.

2. Structured Reasoning via Prompt Engineering

The core of the RecZero architecture is a "Think-before-Recommendation" prompt construction. This schema decomposes the rating prediction procedure into explicit chain-of-thought substeps, enforced through token-demarcated template segments:

• {analyze user}...{/analyze user}: Extraction of positive ([pos]/[like]) and negative ([neg]/[dislike]) user attributes from historical interactions.
• {analyze item}...{/analyze item}: Identification and annotation of item features.
• {match}...{/match}: Assessment of user–item compatibility based on prior step outputs.
• {rate}...{/rate}: Final scalar rating output.

Each section requires explicit information tagging, ensuring that the model performs genuine, interpretable reasoning rather than shortcutting directly to the answer. The prompt template is illustrated in the paper’s appendix with a system prompt and example reasoning trace, providing reproducible benchmarks for structured reasoning outputs.

3. Reward Modeling and RL Optimization

RecZero employs a rule-based reward function, calculated to simultaneously encourage format adherence and predictive accuracy:

• Format Reward (R_format): Binary signal rewarding outputs that strictly follow the structured template (e.g., R_format = 0.5 for correct format, –0.5 for incorrect).
• Answer Reward (R_answer): Proportional to rating accuracy, computed as

$$R_{\text{answer}} = 1 - \frac{|y - \hat{y}|}{\max_{\text{error}}}$$

where $\hat{y}$ is the predicted rating, $y$ is the ground-truth, and $\max_{\text{error}}$ is the maximum possible discrepancy (typically 4 for ratings in [1,5]).

The total reward is additive:

$R = R_{\text{format}} + R_{\text{answer}}$

Policy optimization uses Group Relative Policy Optimization (GRPO): for each input, multiple trajectories $(R_1, \dots, R_G)$ are sampled, and the relative advantage for trajectory $i$ is computed as

$$\hat{A}_i = \frac{R_i - \text{mean}(R_1, \dots, R_G)}{\text{std}(R_1, \dots, R_G)}$$

The policy update applies a clipped objective, thereby stabilizing learning and directly linking reasoning quality to reward gradient.

4. Hybrid Approach: RecOne

The paper further introduces RecOne, a hybrid paradigm that initializes the LLM via supervised fine-tuning (SFT) on cold-start, high-quality reasoning samples before proceeding with RL optimization. The process involves:

• Generation of cold-start samples divided into D_align (naturally correct traces) and D_misalign (requiring correction via rationalized teacher traces).
• Autoregressive SFT loss pretraining:

$$\max_{\theta} \sum_{(x, y^{\text{trace}}) \in D_{\text{trace}}} \sum_t \log P_\theta ( y^{\text{trace}}_t | x, y^{\text{trace}}_{<t} )$$

• Subsequent RL fine-tuning as in RecZero.

RecOne leverages the sample efficiency and rapid domain adaptation of SFT while still benefiting from the policy-driven optimization advantages of RL.

5. Experimental Performance and Benchmarks

Experiments were conducted on standard datasets in domains including Books, Music, and Yelp. Metrics include:

• Mean Absolute Error (MAE)
• Root Mean Square Error (RMSE)

Results show that both RecZero and RecOne substantially outperform CF-based approaches, review-based models, and prior LLM-based recommendation systems (such as Reason4Rec, Rec-SAVER, EXP3RT). Particularly, RecOne achieves the lowest MAE/RMSE across all datasets, with RecZero also outperforming the strongest distillation baselines.

Supporting figures (e.g., training curve plots) indicate faster convergence rates and lower final error for RecOne, attributed to its hybrid SFT+RL training strategy.

6. Implications, Advantages, and Future Directions

RecZero presents several notable implications:

• Enables fully autonomous reasoning in recommender systems, enhancing interpretability and adaptability without reliance on teacher models or static reasoning traces.
• Streamlines deployment pipelines with a single, end-to-end trainable model.
• Demonstrates cost-efficiency and adaptability in cold-start and distribution-drift settings by exploiting RL’s sample-efficient reward mechanism.
• Lays groundwork for iterative self-optimization in future recommender frameworks.

Potential research directions include scaling RecZero to larger model architectures, exploring iterative or multi-round RL schemes, and replacing teacher models in self-distillation paradigms.

Table: Reward Structure Components (as implemented in RecZero)

Reward Component	Mathematical Formulation	Function
Format Reward	$R_\text{format} = 0.5$ or $-0.5$	Enforces strict prompt adherence
Answer Reward	$$R_\text{answer} = 1 - \frac{|y - \hat{y}|}{\max_\text{error}}$$	Promotes rating accuracy
Total Reward	$R = R_\text{format} + R_\text{answer}$	Guides reasoning and prediction
GRPO Advantage	$$\hat{A}_i = \frac{R_i - \text{mean}(R_1,\dots,R_G)}{\text{std}(R_1,\dots,R_G)}$$	Relative performance scaling

References to Related Work

• The RecZero approach sets itself apart from distillation-based frameworks and prior LLM-powered recommenders cited within the paper (e.g., Reason4Rec, Rec-SAVER, EXP3RT), exhibiting superior empirical results (Kong et al., 27 Oct 2025).

Conclusion

RecZero demonstrates that a pure RL paradigm, focused on autonomous reasoning and direct reward-driven optimization within a single LLM, yields significant advances in recommender system performance and interpretability. The "Think-before-Recommendation" prompt and GRPO-driven reward optimization induce robust step-wise analytical capabilities, establishing a foundation for future research in reasoning-empowered recommendation and related areas.