---
title: 'DeepChain: Multi-Agent RL Framework'
url: https://www.emergentmind.com/topics/deepchain
type: topic
---

# DeepChain: Multi-Agent RL Framework

DeepChain is a multi-agent reinforcement learning (RL) framework for whole-chain recommendations, designed to jointly optimize multiple recommendation strategies across consecutive user scenarios within a session. Departing from previous approaches that either apply a single policy to all scenarios or treat scenario-specific strategies independently, DeepChain models the entirety of a user’s session—spanning heterogeneous interaction phases such as entrance and item-detail pages—as a single Markov decision process (MDP) where specialized recommender agents (RAs) coordinate to maximize cumulative reward. It incorporates a model-based component to address sample complexity and imbalanced reward distributions common in large-scale recommender systems [1902.03987].

## 1. Whole-Chain Recommendation as Multi-Agent RL

DeepChain formalizes the user session as an MDP $(\mathcal{S}, \mathcal{A}, P, R, \gamma)$, where:

- $\mathcal{S}$: At step $t$, the state $s_t$ encodes the chronologically ordered item interaction history $s_t = (i_1, i_2, ..., i_{N(t)})$ across all scenarios.
- $\mathcal{A}$: An action $a_t$ is a single item recommendation ($a_t \in \mathcal{A}$).
- $R$: Upon $a_t$, the user response yields reward $r_t = R(s_t, a_t)$, with typical values: skip = $0$, click or purchase = $+1$, leave = $-2$.
- $P$: The transition kernel $P(s_{t+1} \mid s_t, a_t)$ captures both user feedback and scenario transitions (e.g., entrance page to item-detail page).
- $\gamma$: Discount factor in $[0,1]$.

There are $M$ scenario-specific RAs, $\{\pi_1, ..., \pi_M\}$, each uniquely responsible for a scenario. For a session trajectory $\tau = (s_1, a_1, ..., s_T, a_T)$ and joint policy $(\pi_1, ..., \pi_M)$, the objective is to maximize expected return:
$$
J(\pi_1, ..., \pi_M) = \mathbb{E}_{\tau \sim P, \pi} \Big[ \sum_{t=1}^T \gamma^{t-1} r_t \Big].
$$
At each $t$, only the agent for the active scenario selects $a_t = \pi_m(s_t)$, with all agents sharing user-history memory.

## 2. State, Action, and Reward Specification

For illustrative purposes, consider two scenarios: entrance page ($m$) and item-detail page ($d$).

- State $s_t$: A sequence of $N$ item IDs, each with a pre-trained embedding $e_j \in \mathbb{R}^E$ (e.g., via word2vec trained on click sequences), yielding $(e_1, ..., e_N)$.
- Action $a_t$: Item recommendation, represented by its embedding $a_t \in \mathbb{R}^E$.
- Immediate reward $r_t$ depends on user action:
  - Skip $\rightarrow r_t = 0$
  - Click or purchase $\rightarrow r_t = +1$
  - Leave $\rightarrow r_t = -2$
- Transition probabilities for each scenario $\mu \in \{m, d\}$: Click/go-to-detail $p_\mu^c(s, a)$, skip/stay $p_\mu^s(s, a)$, leave $p_\mu^l(s, a)$.

## 3. Joint Multi-Agent Policy Learning Objective

The goal is to learn scenario-specific policies $(\pi_m, \pi_d)$ so as to maximize the session's cumulative discounted return,
$$
J(\pi_m, \pi_d) = \mathbb{E}\left[\sum_{t=1}^T \gamma^{t-1} r_t\right].
$$
A global action-value function $Q(s, a)$ (the Critic) is used, and the designated actor at step $t$ applies $a_t = \pi_\mu(s_t)$, with $\mu \in \{m, d\}$.

## 4. DeepChain Architecture

### Agents, Critic, and Model-Based Component

The architecture comprises the following:

- **Shared Memory:** All agents operate on a common user-history state, $s_t = (e_1, ..., e_N)$.
- **Actor Networks for each Scenario ($\pi_\mu$):**
  - **Preference Encoder:** A GRU processes the sequence $(e_1, ..., e_N)$, outputting hidden vectors $h_j$. An attention mechanism computes per-item weights:
    $$
    \alpha_j^\mu = \mathrm{softmax}_j\left(w_\mu^\top \tanh(W_h h_j + b_h)\right),
    $$
    yielding the user preference embedding $a_t^\mu = \sum_{j=1}^N \alpha_j^\mu h_j \in \mathbb{R}^H$.
  - **Recommendation Decoder:** For candidate item $k$ with embedding $e_k$, score is
    $$
    \mathrm{Score}_k = {a_t^\mu} \cdot {e_k}^\top
    $$
    and the top-scoring item is recommended.
- **Global Critic $Q(s, a)$:** GRU+attention encoder (two heads, one per scenario) produces context representations concatenated with the action embedding, followed by MLP layers to estimate $Q$.
- **Model-Based Probability Network ($P_\phi$):** Learns to predict next-scenario transitions ($p_\mu^c$, $p_\mu^s$, $p_\mu^l$ for each $\mu$) using two softmax heads given $(s, a)$, facilitating model-based target computation and reducing reliance on rare real user signals.

## 5. Training Methodology

DeepChain follows an off-policy actor–critic paradigm similar to DDPG, with four trainable neural nets: actors $(\pi_m, \pi_d)$, critic $Q_\mu(s, a)$, probability network $P_\phi(s'|s, a)$, and their soft-updated targets $(\pi'_m, \pi'_d, Q_\mu')$. A replay buffer $\mathcal{D}$ stores experience tuples.

The model-based multi-scenario Bellman backup is given by (for mini-batch update on scenario $\mu$):
- If $\mu = m$:
  $$
  y = p_m^s(s, a)\, \gamma Q'(s', \pi'_m(s')) + p_m^c(s, a)\, [r + \gamma Q'(s', \pi'_d(s'))] + p_m^l(s, a)\, r
  $$
- If $\mu = d$:
  $$
  y = p_d^s(s, a)\, \gamma Q'(s', \pi'_m(s')) + p_d^c(s, a)\, [r + \gamma Q'(s', \pi'_d(s'))] + p_d^l(s, a)\, r
  $$

Loss functions:
- **Critic:** Mean squared error loss,
  $$
  L(\mu) = \mathbb{E}_{\text{batch}}[(Q_\mu(s, a) - y)^2]
  $$
- **Actor:** Deterministic policy gradient,
  $$
  \nabla_{\theta_\mu} J \approx \mathbb{E}_s\left[\nabla_a Q_\mu(s, a)\Big|_{a = \pi_\mu(s)} \cdot \nabla_{\theta_\mu} \pi_\mu(s)\right]
  $$
- **Probability Net:** Cross-entropy loss on next-scenario labels.

Target networks are updated via soft updates with $\tau$ (e.g., $0.01$).

## 6. Addressing Data Scarcity and Imbalanced Rewards

To overcome the data demands and highly skewed reward distribution (dominance of skip/zero-reward events), DeepChain leverages:
- **Model-Based Rollouts:** The probability network $P_\phi(s'|s,a)$ enables generation of imagined transitions, mitigating the scarcity of positive signals.
- **Multi-Scenario Bellman Backup:** Model-based targets aggregate over possible user behaviors and scenario transitions, smoothing sparse rewards.
- **Replay Buffer and Target Networks:** These mechanisms stabilize learning and further reduce the reliance on infrequent reward events during policy updates.

## 7. Empirical Evaluation and Comparative Analysis

Experiments were conducted on a JD.com dataset with 500K user sessions (19.7M interactions, train/validation/test split at 80/20). Key configuration details:
- User-history length $N=50$
- Embedding dimension $E=20$, GRU hidden units $H=64$, discount $\gamma=0.95$, target soft update $\tau=0.01$
- Reward assignments: skip = 0; click = +1; leave = –2

Offline evaluation utilized MAP and NDCG@40 as metrics, with DeepChain compared against W$^2$D (Wide & Deep), DeepFM, GRU4Rec, DDPG (single-agent), and MA (model-free multi-agent) baselines. DeepChain achieved 4–19% higher scores than the strongest baselines (e.g., entrance page MAP: DeepChain $0.126$ vs. MA $0.121$ vs. DDPG $0.117$).

Ablation studies confirmed that using separate actors per scenario and the model-based component accelerated learning and improved convergence relative to single-agent or model-free variants. Performance sensitivity analysis showed session length/history length ($N$ up to 50) notably improved entrance page recommendations, reflecting the impact of historical diversity on scenario-specific policy efficacy.

For further architectural and algorithmic details, see [1902.03987].

Source: https://www.emergentmind.com/topics/deepchain