Batch-Constrained Distributional RL

• The paper integrates a batch constraint via behavior cloning and IQN-based full return distribution estimation to robustly learn policies from fixed, offline datasets.
• It formalizes session-based recommendation as an MDP with sequential user interactions, mitigating extrapolation error through hard support constraints.
• Experimental evaluations show that BCD-RL achieves competitive performance in CTR/BR metrics while improving diversity and reducing popularity bias compared to baselines.

Batch-constrained distributional reinforcement learning (BCD-RL) is a class of algorithms designed for learning policies from fixed, offline datasets where no online user interaction is permitted, and where the intrinsically stochastic nature of returns is explicitly modeled. The BCD4Rec framework exemplifies this paradigm in the context of session-based recommendation, combining batch-constrained action selection with full return distribution estimation to address the challenges of large discrete action spaces, biased logs, and user preference heterogeneity (Garg et al., 2020).

1. Formalization in Session-Based Recommendation

The session-based recommendation problem is modeled as a Markov Decision Process (MDP) with the following components:

• State space $\mathcal{S}$: At each time $t$, state $s_t$ is the sequence of the user’s most recent $L$ positive interactions (e.g., clicked/bought items), each embedded as a $d$-dimensional vector. The complete sequence is encoded by a Bi-GRU followed by a linear layer to yield a real-valued state embedding $\mathbf{s} \in \mathbb{R}^d$.
• Action space $\mathcal{A}$: The agent selects an item $$a_t \in \mathcal{I} \setminus \{\text{already clicked}\}$$ from the catalog. Each candidate is represented by its embedding $\mathbf{a} \in \mathbb{R}^d$.
• Transition dynamics $P(s'|s,a)$: After an action, the user's response yields a positive (clicked/bought) or negative (skip) outcome. Positive feedback updates $t$0 by appending $t$1 and removing the oldest entry; otherwise, the state is unchanged or absorbing.
• Reward $t$2: Scalar immediate feedback, e.g., skip $t$3, click $t$4, buy $t$5 in the Diginetica environment.
• Discount factor $t$6.

The objective is to learn a policy $t$7 maximizing the expected discounted return: $t$8

2. Batch-Constrained Policy Learning

In the batch (offline) RL setting, only a fixed dataset $t$9 collected by an unknown policy $s_t$0 is available. Standard off-policy Q-learning is susceptible to overestimating for actions rarely or never taken by $s_t$1 ("out-of-distribution" actions), resulting in extrapolation error.

2.1 Action Support Constraint

A state-conditioned generative model $s_t$2 is trained via behavior cloning: $s_t$3 Typically,

$s_t$4

During learning, Q-value maximization is conducted only over actions $s_t$5 with $s_t$6, for some threshold $s_t$7. This hard support constraint prevents the policy from excessively bootstrapping on unsupported actions.

2.2 Absence of Penalty Terms

BCD4Rec applies a hard cutoff via $s_t$8 for batch constraints, without introducing KL-divergence or $s_t$9 penalties in Bellman backups.

3. Distributional RL via Implicit Quantile Networks

Instead of only predicting $L$0, BCD4Rec estimates the full distribution of returns $L$1 with Implicit Quantile Networks (IQN):

• Distributional Bellman operator:

$L$2

This operator is applied in the distributional (not mean) sense.

• IQN parameterization:
  • Sample $L$3 quantile fractions $L$4.
  • For each $L$5, compute a quantile-conditional state embedding $L$6, where $L$7 is a learned transformation (e.g., cosine embedding + ReLU).
  • Quantile estimate: $L$8.
• Learning objective: The quantile Huber loss is computed as

$L$9

with $d$0 the standard Huber loss. The total loss is

$d$1

where $d$2, and $d$3 is the maximizing action under the batch constraint.

4. Implementation Protocols

Algorithmic workflow consists of the following:

1. Pre-train item embeddings $d$4 (e.g., using SRGNN next-item prediction) or random initialization.
2. Initialize the IQN network $d$5, a corresponding target network $d$6 (with $d$7), and the behavior model $d$8.
3. Iterate over training epochs by:
   • Sampling minibatches from $d$9.
   • Fitting $\mathbf{s} \in \mathbb{R}^d$0 by supervised learning.
   • For each sample, drawing $\mathbf{s} \in \mathbb{R}^d$1 quantiles and computing $\mathbf{s} \in \mathbb{R}^d$2.
   • For each quantile in $\mathbf{s} \in \mathbb{R}^d$3, searching over $\mathbf{s} \in \mathbb{R}^d$4 with $\mathbf{s} \in \mathbb{R}^d$5 to maximize mean $\mathbf{s} \in \mathbb{R}^d$6.
   • Computing quantile-Huber losses and updating $\mathbf{s} \in \mathbb{R}^d$7.
   • Periodically synchronizing target network parameters.

Main architectural elements include a learned item table, Bi-GRU session encoder, MLP/cosine quantile embedding, and dot-product final quantile scoring.

5. Theoretical Properties

• Convergence and optimization: IQN with expressive function classes converges to fixed points of the distributional Bellman operator in the 1-Wasserstein metric, subject to standard contraction arguments.
• Stochasticity and robustness: Modeling $\mathbf{s} \in \mathbb{R}^d$8 (the full return distribution) allows improved handling of reward stochasticity from latent user interests, mitigating overestimation for actions with high-variance but low means.
• Error reduction: The batch support constraint (hard cutoff) combined with quantile regression reduces extrapolation error endemic to learning from purely offline datasets.

6. Experimental Evaluation

BCD4Rec was evaluated in both real and simulated environments:

• Datasets:
  • Diginetica (DN): CIKM-2016 click/buy logs (~6,700 items). Training utilizes user interaction logs and a GRU-based simulator.
  • RecSim: Google’s RecSim, simulating interest evolution with 200 items and click/skip responses; behavior data is drawn from random and various checkpointed policies.
• Baselines: Heuristic (MostPop, SKNN, STAN), supervised (GRU, SRGNN), off-policy RL (DQN, BCQ), and distributional RL (QR-DQN, QR-BCQ, IQN without batch constraint).
• Metrics:
  • Online: Buy Rate (BR) and Click-Through Rate (CTR).
  • Diversity: Coverage@3, the fraction of unique items recommended in the top-3 across sessions.
  • Offline proxies: Recall@3 and average Q-value $\mathbf{s} \in \mathbb{R}^d$9 on held-out $\mathcal{A}$0.

6.1 Performance Summary

Task	Behavior	DQN	BCQ	IQN	BCD4Rec	Online-oracle
DN (BR)	4.1%	4.8%	13.3%	23.5%	24.1%	26.6%
RecSim-1 (CTR)	63.1%	50.7%	65.9%	73.5%	76.4%	85.9%
RecSim-2 (CTR)	68.3%	52.1%	72.3%	75.9%	79.3%	85.9%
RecSim-3 (CTR)	79.9%	54.0%	76.3%	81.5%	83.2%	85.9%

BCD4Rec achieves the highest performance among offline methods in both environments, closely matching the online-oracle.

6.2 Analysis of Diversity and Bias

• Latent category coverage: BCD4Rec stays within users’ latent preferred categories, mitigating overestimation on out-of-category items.
• Popularity bias: The frequency of item recommendations under BCD4Rec approaches the oracle distribution, contrasting with the skew towards popular items typical of DQN, BCQ, and QR-DQN.

6.3 Ablation Findings

• Pre-training item embeddings (via SRGNN) yields a 3–5 point improvement in BR/CTR, especially in larger catalogs.
• For hyperparameter selection, Recall@3 poorly correlates with online reward; the average $\mathcal{A}$1-value on a validation set shows strong correlation with CTR/BR and should be preferred as an offline proxy.

7. Synthesis and Implications

BCD4Rec integrates a strict batch constraint via a learned behavior model, full distributional value estimation using IQN, and scalable session/action encoding. It exhibits substantial improvement over both conventional offline RL and non-RL baselines in realistic recommendation environments. Its design effectively controls extrapolation from batch data, captures user response stochasticity, increases category diversity, and reduces popularity bias in recommendation outputs (Garg et al., 2020).