DeepChain: Multi-Agent RL Framework

• DeepChain is a multi-agent reinforcement learning framework for whole-chain recommendations that formulates user sessions as a single MDP with coordinated scenario-specific policies.
• It employs specialized actor networks with attention mechanisms and a global critic to learn recommendation strategies across heterogeneous interaction phases.
• The framework integrates a model-based component to mitigate data scarcity and imbalanced rewards, achieving superior performance on benchmark datasets.

DeepChain is a multi-agent reinforcement learning (RL) framework for whole-chain recommendations, designed to jointly optimize multiple recommendation strategies across consecutive user scenarios within a session. Departing from previous approaches that either apply a single policy to all scenarios or treat scenario-specific strategies independently, DeepChain models the entirety of a user’s session—spanning heterogeneous interaction phases such as entrance and item-detail pages—as a single Markov decision process (MDP) where specialized recommender agents (RAs) coordinate to maximize cumulative reward. It incorporates a model-based component to address sample complexity and imbalanced reward distributions common in large-scale recommender systems (Zhao et al., 2019).

1. Whole-Chain Recommendation as Multi-Agent RL

DeepChain formalizes the user session as an MDP $(\mathcal{S}, \mathcal{A}, P, R, \gamma)$, where:

• $\mathcal{S}$: At step $t$, the state $s_t$ encodes the chronologically ordered item interaction history $s_t = (i_1, i_2, ..., i_{N(t)})$ across all scenarios.
• $\mathcal{A}$: An action $a_t$ is a single item recommendation ($a_t \in \mathcal{A}$).
• $R$: Upon $a_t$, the user response yields reward $\mathcal{S}$0, with typical values: skip = $\mathcal{S}$1, click or purchase = $\mathcal{S}$2, leave = $\mathcal{S}$3.
• $\mathcal{S}$4: The transition kernel $\mathcal{S}$5 captures both user feedback and scenario transitions (e.g., entrance page to item-detail page).
• $\mathcal{S}$6: Discount factor in $\mathcal{S}$7.

There are $\mathcal{S}$8 scenario-specific RAs, $\mathcal{S}$9, each uniquely responsible for a scenario. For a session trajectory $t$0 and joint policy $t$1, the objective is to maximize expected return:

$t$2

At each $t$3, only the agent for the active scenario selects $t$4, with all agents sharing user-history memory.

2. State, Action, and Reward Specification

For illustrative purposes, consider two scenarios: entrance page ($t$5) and item-detail page ($t$6).

• State $t$7: A sequence of $t$8 item IDs, each with a pre-trained embedding $t$9 (e.g., via word2vec trained on click sequences), yielding $s_t$0.
• Action $s_t$1: Item recommendation, represented by its embedding $s_t$2.
• Immediate reward $s_t$3 depends on user action:
  • Skip $s_t$4
  • Click or purchase $s_t$5
  • Leave $s_t$6
• Transition probabilities for each scenario $s_t$7: Click/go-to-detail $s_t$8, skip/stay $s_t$9, leave $s_t = (i_1, i_2, ..., i_{N(t)})$0.

3. Joint Multi-Agent Policy Learning Objective

The goal is to learn scenario-specific policies $s_t = (i_1, i_2, ..., i_{N(t)})$1 so as to maximize the session's cumulative discounted return,

$s_t = (i_1, i_2, ..., i_{N(t)})$2

A global action-value function $s_t = (i_1, i_2, ..., i_{N(t)})$3 (the Critic) is used, and the designated actor at step $s_t = (i_1, i_2, ..., i_{N(t)})$4 applies $s_t = (i_1, i_2, ..., i_{N(t)})$5, with $s_t = (i_1, i_2, ..., i_{N(t)})$6.

4. DeepChain Architecture

Agents, Critic, and Model-Based Component

The architecture comprises the following:

• Shared Memory: All agents operate on a common user-history state, $s_t = (i_1, i_2, ..., i_{N(t)})$7.
• Actor Networks for each Scenario ($s_t = (i_1, i_2, ..., i_{N(t)})$8):
  • Preference Encoder: A GRU processes the sequence $s_t = (i_1, i_2, ..., i_{N(t)})$9, outputting hidden vectors $\mathcal{A}$0. An attention mechanism computes per-item weights:

  $\mathcal{A}$1

  yielding the user preference embedding $\mathcal{A}$2. - Recommendation Decoder: For candidate item $\mathcal{A}$3 with embedding $\mathcal{A}$4, score is

  $\mathcal{A}$5

  and the top-scoring item is recommended.

• Global Critic $\mathcal{A}$6: GRU+attention encoder (two heads, one per scenario) produces context representations concatenated with the action embedding, followed by MLP layers to estimate $\mathcal{A}$7.
• Model-Based Probability Network ($\mathcal{A}$8): Learns to predict next-scenario transitions ($\mathcal{A}$9, $a_t$0, $a_t$1 for each $a_t$2) using two softmax heads given $a_t$3, facilitating model-based target computation and reducing reliance on rare real user signals.

5. Training Methodology

DeepChain follows an off-policy actor–critic paradigm similar to DDPG, with four trainable neural nets: actors $a_t$4, critic $a_t$5, probability network $a_t$6, and their soft-updated targets $a_t$7. A replay buffer $a_t$8 stores experience tuples.

The model-based multi-scenario Bellman backup is given by (for mini-batch update on scenario $a_t$9):

• If $a_t \in \mathcal{A}$0:

$a_t \in \mathcal{A}$1

• If $a_t \in \mathcal{A}$2:

$a_t \in \mathcal{A}$3

Loss functions:

• Critic: Mean squared error loss,

$a_t \in \mathcal{A}$4

• Actor: Deterministic policy gradient,

$a_t \in \mathcal{A}$5

• Probability Net: Cross-entropy loss on next-scenario labels.

Target networks are updated via soft updates with $a_t \in \mathcal{A}$6 (e.g., $a_t \in \mathcal{A}$7).

6. Addressing Data Scarcity and Imbalanced Rewards

To overcome the data demands and highly skewed reward distribution (dominance of skip/zero-reward events), DeepChain leverages:

• Model-Based Rollouts: The probability network $a_t \in \mathcal{A}$8 enables generation of imagined transitions, mitigating the scarcity of positive signals.
• Multi-Scenario Bellman Backup: Model-based targets aggregate over possible user behaviors and scenario transitions, smoothing sparse rewards.
• Replay Buffer and Target Networks: These mechanisms stabilize learning and further reduce the reliance on infrequent reward events during policy updates.

7. Empirical Evaluation and Comparative Analysis

Experiments were conducted on a JD.com dataset with 500K user sessions (19.7M interactions, train/validation/test split at 80/20). Key configuration details:

• User-history length $a_t \in \mathcal{A}$9
• Embedding dimension $R$0, GRU hidden units $R$1, discount $R$2, target soft update $R$3
• Reward assignments: skip = 0; click = +1; leave = –2

Offline evaluation utilized MAP and NDCG@40 as metrics, with DeepChain compared against W$R$4D (Wide & Deep), DeepFM, GRU4Rec, DDPG (single-agent), and MA (model-free multi-agent) baselines. DeepChain achieved 4–19% higher scores than the strongest baselines (e.g., entrance page MAP: DeepChain $R$5 vs. MA $R$6 vs. DDPG $R$7).

Ablation studies confirmed that using separate actors per scenario and the model-based component accelerated learning and improved convergence relative to single-agent or model-free variants. Performance sensitivity analysis showed session length/history length ($R$8 up to 50) notably improved entrance page recommendations, reflecting the impact of historical diversity on scenario-specific policy efficacy.

For further architectural and algorithmic details, see (Zhao et al., 2019).