FlexRec: Need-Conditioned LLM Ranking

• FlexRec is a reinforcement-learning framework that uses explicit need instructions to adapt LLM ranking policies dynamically across recommendation scenarios.
• It introduces a causally grounded item-level reward via local counterfactual swaps, addressing coarse sequence-level credit assignment and sparse feedback.
• The method incorporates a critic-guided, uncertainty-aware RL optimization that significantly improves metrics like NDCG and Recall over traditional approaches.

FlexRec is a reinforcement-learning post-training framework for LLM-based recommendation in which a single autoregressive ranker is conditioned not only on user context and a fixed candidate pool, but also on an explicit need instruction that specifies the current recommendation objective. It is designed for need-specific recommendation, where ranking behavior must change on demand across scenarios such as maximizing expected interest, promoting exploration, surfacing trends, or handling product search. The framework addresses two obstacles that the paper identifies in applying RL to this setting—coarse sequence-level credit assignment and sparse, noisy feedback—through a causally grounded item-level reward and critic-guided, uncertainty-aware scaling during policy optimization (Pan et al., 12 Mar 2026).

1. Conceptual scope and recommendation setting

FlexRec is formulated for recommendation environments in which objectives are not fixed. The motivating premise is that modern recommenders must respond to dynamic user intents and changing business objectives, whereas traditional recommenders are usually trained for a single static target such as click-through rate or purchase probability. In FlexRec, these varying objectives are represented explicitly as need instructions supplied at inference time, so the model is expected to alter ranking behavior without retraining for each scenario (Pan et al., 12 Mar 2026).

The framework studies this problem in closed-set autoregressive ranking. Given a user context, a candidate set, and a target need, the model generates an ordered output by selecting one candidate at a time from the remaining pool. This is not an open-ended generation problem; it is a permutation or top-$K$ ordering problem over a fixed candidate set. The paper treats this formulation as especially suitable for LLM post-training because ranking is sequential, listwise objectives such as NDCG are non-differentiable, and need instructions can be expressed naturally in language (Pan et al., 12 Mar 2026).

The needs instantiated in the experiments are Maximizing Interest, Explore New Topics / Niche Discovery, Trend Promotion, and Product Search. These correspond, respectively, to ranking items the user is most likely to enjoy or engage with, promoting novel but relevant items, balancing personalization with recent popularity, and ranking products by relevance to a textual query. This suggests a broader view of recommendation alignment in which the objective is not reduced to a single latent preference score, but is instruction-conditioned and scenario-specific (Pan et al., 12 Mar 2026).

2. Formal task formulation and autoregressive policy

The paper defines the recommendation context as

$x \triangleq (U, C, M),$

where $U$ denotes user information, $C$ the candidate set, and $M$ metadata or task instruction. Let

$n \in \mathcal{N}$

denote the target need. The model generates a ranking

$$y = (a_1,\dots,a_K), \qquad a_k \in C \setminus \{a_1,\dots,a_{k-1}\},$$

so each decision is made from the remaining candidates after the prefix has been fixed (Pan et al., 12 Mar 2026).

This induces the autoregressive ranking policy

$\pi_\theta(a_k \mid a_{<k}, x, n),$

which conditions on the context $x$, the need $n$, and the already selected prefix $x \triangleq (U, C, M),$0. The paper assumes a need-specific ranking objective

$x \triangleq (U, C, M),$1

and instantiates the listwise objective as NDCG@K in experiments. The sequential formulation is central to the method: the value of placing an item at rank $x \triangleq (U, C, M),$2 depends on the current prefix and on the remaining candidate pool, so rank-level decisions cannot be treated as context-free local classifications (Pan et al., 12 Mar 2026).

FlexRec therefore differs from conventional reranking pipelines that optimize a static score function over items independently. Its policy is an instruction-following LLM—Qwen2.5-3B-Instruct in the main experiments—that receives natural-language descriptions of user history and candidate items, plus the active need instruction, and then generates the ranking autoregressively. The paper also allows the model to generate optional reasoning tokens, which matters because FlexRec assigns different training signals to item tokens and non-item tokens during RL (Pan et al., 12 Mar 2026).

3. CausalSwap: item-level reward and credit assignment

The first technical contribution is a causally grounded item-level reward based on local counterfactual swaps. The paper argues that sequence-level rewards give the same scalar signal to all actions in a rollout, which is too coarse for ranking, while naive per-rank rewards are not directly comparable across rollouts because the meaning of a rank-$x \triangleq (U, C, M),$3 decision depends on the prefix and the remaining candidate set. FlexRec addresses this by holding the earlier prefix fixed and evaluating the chosen item only against alternatives that were still available at that step (Pan et al., 12 Mar 2026).

For a ranking

$x \triangleq (U, C, M),$4

and positions $x \triangleq (U, C, M),$5, the paper defines the swap operator $x \triangleq (U, C, M),$6 and the swapped ranking

$x \triangleq (U, C, M),$7

The change in objective under that swap is

$x \triangleq (U, C, M),$8

From this, FlexRec defines the expected local swap effect

$x \triangleq (U, C, M),$9

and estimates it with the CausalSwap reward

$U$0

By construction, $U$1 is described as an unbiased estimator of $U$2 (Pan et al., 12 Mar 2026).

The causal restriction is the crucial design choice. The contribution of $U$3 is defined relative to the remaining pool

$U$4

not relative to earlier positions that had already been fixed. The paper presents this as a direct response to the autoregressive structure of ranking. Computationally, the reward construction incurs $U$5 overhead, which the paper considers acceptable in reranking because candidate sets are limited (Pan et al., 12 Mar 2026).

FlexRec then normalizes these item rewards within each prompt group. If $U$6 is the reward for item $U$7 in rollout $U$8 of group $U$9, the pooled rewards define

$C$0

with group mean and standard deviation

$C$1

The resulting item-level advantage is

$C$2

Item tokens receive these item-level advantages, while non-item tokens receive conventional sequence-level GRPO-style advantages. This produces a hybrid token-credit assignment scheme rather than a uniform sequence reward over the whole output (Pan et al., 12 Mar 2026).

4. Critic-guided uncertainty-aware reinforcement learning

The second technical contribution is a critic that completes sparse rewards and explicitly models uncertainty. The paper emphasizes that recommendation data are dominated by unobserved user-item interactions, so RL reward often must be imputed rather than read off directly from logged outcomes. Using such estimates naively can destabilize optimization because predicted rewards vary substantially in confidence across users, items, and contexts (Pan et al., 12 Mar 2026).

FlexRec therefore trains a two-tower MLP-based critic that predicts both an expected reward

$C$3

and a predictive variance

$C$4

The critic uses an MLP user-history encoder, fixed item embeddings, hidden dimension $C$5, interaction features $C$6, $C$7, $C$8, and $C$9, along with mean and log-variance heads, LayerNorm, and dropout $M$0. Its heteroscedastic regression objective is

$M$1

with $M$2 in the paper’s implementation (Pan et al., 12 Mar 2026).

Sequence-level uncertainty is obtained by summing per-item variances: $M$3 Ignoring cross-item covariance, this yields a variance estimate $M$4 for each rollout. FlexRec then computes inverse-variance confidence

$M$5

and rescales the advantage as

$M$6

Low-confidence rewards are therefore down-weighted rather than treated on par with high-confidence ones (Pan et al., 12 Mar 2026).

At the optimization level, FlexRec is described as item-level GRPO with uncertainty-aware GRPO. The policy update uses item-level advantages for item tokens, sequence-level advantages for non-item tokens, KL regularization against a reference policy with coefficient $M$7, and entropy regularization with coefficient $M$8. The paper frames this combination as a stabilization mechanism for RL on sparse recommendation feedback, rather than as a change to the base LLM architecture itself (Pan et al., 12 Mar 2026).

5. Need instructions, data representation, and evaluation protocol

The framework operationalizes flexible needs through natural-language instructions and task-specific reward construction. User histories and candidate items are converted into textual descriptions, and the active need is specified in the prompt. For sequential recommendation datasets, the user context uses the most recent

$M$9

historical items; the closed candidate set has size

$n \in \mathcal{N}$0

Candidate generation is performed by a fixed semantic retriever following STAR: items are embedded with Gemini text-embedding-004, the user query embedding is a temporally discounted average of historical item embeddings, top-$n \in \mathcal{N}$1 similar items are retrieved, and previously interacted items are excluded (Pan et al., 12 Mar 2026).

The paper evaluates FlexRec on KuaiRec, MovieLens-1M / ML-1M++ setup, and ESCI, covering short-video recommendation, movie sequential recommendation, and product search. Need-specific targets are derived from existing signals because public benchmarks do not directly annotate these needs. For example, KuaiRec uses watch ratio for Maximizing Interest; MovieLens uses rating-based relevance transformed via $n \in \mathcal{N}$2; niche discovery introduces a novelty bonus; trend promotion combines semantic similarity and recent popularity; and ESCI maps $n \in \mathcal{N}$3 to $n \in \mathcal{N}$4 for graded relevance (Pan et al., 12 Mar 2026).

Need	Instantiation in the paper	Instructional emphasis
Maximizing Interest	Watch ratio or rating-based relevance	Strong alignment with known interests
Explore New Topics / Niche Discovery	Novelty bonus on unseen topics/genres	Niche but plausibly relevant items
Trend Promotion	$n \in \mathcal{N}$5, with $n \in \mathcal{N}$6	Popular in the last 24 hours if relevant
Product Search	ESCI relevance labels mapped to graded scores	Query-item relevance over title, description, and attributes

Evaluation uses NDCG@$n \in \mathcal{N}$7, Recall@5, MRR@5, and, in some tables, Precision@5. The generalization protocol includes training on Maximizing Interest and testing zero-shot on Explore New Topics and Trend Promotion. Main implementation details are also explicit: the backbone is Qwen2.5-3B-Instruct, training is full-parameter RL post-training, the learning rate is $n \in \mathcal{N}$8, compute is 4 × NVIDIA A100 GPUs, the training framework is Verl, and rollout inference uses vLLM (Pan et al., 12 Mar 2026).

6. Empirical results, ablations, and limitations

Across need-specific ranking scenarios, the paper reports that FlexRec improves NDCG@5 by up to 59\% and Recall@5 by up to 109.4\%, and under generalization settings it achieves up to 24.1\% Recall@5 improvement. These gains are reported against strong traditional recommenders and LLM-based baselines and are presented as evidence that instruction-conditioned RL post-training can produce a more adaptable recommender than static rerankers or prompt-only LLM use (Pan et al., 12 Mar 2026).

Dataset / setting	FlexRec result	Relative improvement over Qwen2.5-3B-Instruct
KuaiRec, Maximizing Interest	N@5 0.597, R@5 0.335, MRR@5 0.840	N@5 +59.2%, R@5 +109.4%, MRR@5 +63.1%
MovieLens-1M, Maximizing Interest	N@5 0.615, N@10 0.665, MRR@5 0.368	N@5 +23.7%, N@10 +20.3%, MRR@5 +50.2%
ESCI, Product Search	N@5 0.528, P@5 0.277, R@5 0.678	N@5 +17.6%, P@5 +17.9%, R@5 +15.9%

The generalization experiments are more specific. When trained on Maximizing Interest and tested on Explore New Topics, FlexRec reaches R@5 = 0.165 on both KuaiRec and MovieLens-1M, with +17.9\% and +24.1\% improvements over Qwen2.5-3B-Instruct, respectively. Under Trend Promotion, transfer is generally strong but not uniformly dominant on every metric-dataset pair; the paper explicitly notes settings in which R@5 and MRR@5 are lower than Qwen on KuaiRec even though NDCG remains improved. This suggests that the learned policy generalizes across needs, but not identically across all reward definitions and domains (Pan et al., 12 Mar 2026).

The ablation studies attribute the main gains to the two core components. On KuaiRec dense under Maximizing Interest, CausalSwap (ours) reaches N@5 = 0.607 under GRPO and 0.621 under PPO, clearly above Independent contribution and Non-causal swap variants. In reward-completion ablations, Raw critic reaches N@5 = 0.566, whereas the FlexRec uncertainty-aware critic reaches 0.595 and improves R@5 from 0.280 to 0.319. The training-dynamics analysis further reports that sequence-level reward plateaus early, swap-based item-level reward sustains validation NDCG improvements, and removing uncertainty-aware updates yields early gains followed by later degradation (Pan et al., 12 Mar 2026).

The paper’s limitations are explicit. FlexRec is restricted to closed-set reranking only, with a predefined candidate pool. Need-specific labels are derived from existing signals such as watch ratio, novelty bonuses, and trend scores rather than collected as direct need annotations. The system also does not model retrieval or open-world recommendation dynamics. Future work is therefore directed toward retrieval-augmented personalization, larger and evolving item spaces, and more general open-world recommendation settings. Within its stated scope, however, FlexRec is presented as a step toward a universal, instruction-conditioned recommender whose ranking behavior can be reconfigured through explicit need instructions rather than retraining a separate model for each objective (Pan et al., 12 Mar 2026).