Hierarchical Semantic RL

• Hierarchical Semantic Reinforcement Learning is a framework that uses hierarchical policy architectures and semantic abstraction to tackle large, dynamic action spaces in recommendation systems.
• It employs residual quantization via k-means across multiple levels to generate fixed-length Semantic IDs, ensuring consistent mapping despite catalog changes.
• Empirical results demonstrate significant improvements in reward, user depth, and conversion rates, validating HSRL’s scalability and efficiency in production environments.

Hierarchical Semantic Reinforcement Learning (HSRL) refers to a class of reinforcement learning frameworks that leverage hierarchical policy architectures, semantic abstraction, and interpretable action spaces to address challenges in domains with complex, large, or dynamic action spaces. HSRL departs from conventional, flat RL policy optimization by introducing a multi-level decision process wherein higher-level policies generate semantically meaningful representations (such as Semantic IDs), which are subsequently refined by lower-level components, enabling stable and efficient learning even in environments such as large-scale recommender systems with volatile catalog dynamics (Wang et al., 10 Oct 2025).

1. Motivation and Problem Definition

Traditional RL-based recommender systems suffer from the high cardinality and dynamic nature of item-level action spaces. Directly learning a policy over items is computationally prohibitive, results in significant instability as catalogs change, and often creates a substantial mismatch between feature representations and the operational action space, which degrades policy effectiveness. To address these issues, HSRL introduces the concept of operating in a fixed Semantic Action Space (SAS), within which actions encode semantic intent rather than directly mapping to item identities. Each action is expressed as a Semantic ID (SID), a finite, structured sequence of tokens, abstracting away from the underlying, changeable set of items.

2. Semantic Action Space and Semantic IDs

The Semantic Action Space (SAS) is constructed offline by encoding each item in the catalog into a fixed-length token sequence, termed the Semantic ID (SID). This process employs residual quantization with k-means (“RQ-k-means”) iteratively across $L$ levels: at each level $\ell$, the item embedding vector $x_i$’s current residual $r^{(\ell)}_i$ is assigned to the nearest centroid in a fixed vocabulary $\mathcal{V}_\ell$, i.e.,

$$z_\ell(i) = \arg\min_k \|\mathbf{r}^{(\ell)}_i - \mathbf{c}^{(\ell)}_k\|_2$$

with the residual updated as

$$\mathbf{r}^{(\ell+1)}_i = \mathbf{r}^{(\ell)}_i - \mathbf{c}^{(\ell)}_{z_\ell(i)}$$

After $L$ quantization steps, the complete SID is $z(i) = [z_1(i), z_2(i), \ldots, z_L(i)]$.

A static, invertible lookup codebook is maintained to map any SID back to its corresponding item (or set of items) at serving time. This mechanism guarantees that action space semantics and dimensionality do not fluctuate with catalog growth or churn, addressing both scaling and stability challenges in RL-based recommendation.

3. Hierarchical Policy Network (HPN)

The policy network in HSRL is designed to operate hierarchically, generating SIDs in a coarse-to-fine, autoregressive manner. Given user state $s$ and contextualized embedding $c_0$ (e.g., from a sequence encoder), the policy produces a probability distribution over the level-1 semantic vocabulary: $p(z_1 | c_0) = \mathrm{softmax}(W_1 \cdot c_0)$ After sampling or weighting via $p(z_1 | c_0)$, the expected embedding $e_1$ is computed using a dedicated embedding matrix $E_1$: $e_1 = (p(z_1 | c_0))^T E_1$ The residual context for level 2 is then computed as

$c_1 = \mathrm{LayerNorm}(c_0 - e_1)$

This process is recursively repeated for all $L$ levels. Each sub-policy thus generates a token conditioned on the context after subtracting committed semantics, producing the joint policy over SIDs as

$$\pi_\theta(z | s) = \prod_{\ell=1}^L p(z_\ell | c_{\ell-1})$$

This residual architecture ensures each level’s decision focuses on semantics unresolved at prior coarser levels, regularizes the representation-decision alignment, and helps avert overfitting to spurious correlation in the raw item space.

4. Multi-Level Critic (MLC) and Credit Assignment

Complex action generation as token sequences makes it non-trivial to assign credit to individual decision steps: rewards are typically assigned only after the final semantic action is executed and mapped back to an item. HSRL addresses this through a Multi-Level Critic (MLC), which provides per-level value estimates for each intermediate context. Specifically, for each level $\ell$ context $c_\ell$ the critic predicts $V_\phi(s, \ell) = f_\phi(c_\ell)$; these are then aggregated: $$w_\ell = \frac{\exp(w_\ell)}{\sum_j \exp(w_j)}, \quad \hat{V}_\phi(s) = \sum_{\ell=0}^L w_\ell V_\phi(s, \ell)$$ This design supports fine-grained temporal-difference (TD) learning, allowing stable and data-efficient credit assignment across the hierarchical policy’s decisions.

5. Empirical Performance and Online Deployment

HSRL was evaluated on standard public recommendation benchmarks such as RL4RS and MovieLens-1M, as well as a production-scale short-video ad platform. On RL4RS, HSRL achieved a Total Reward of 12.308 and average user Depth of 13.084, exceeding the best prior baseline (HAC) by over 22% in Total Reward and 17.9% in Depth. On MovieLens-1M, the advantage was over 7% in both metrics. In online A/B testing within a production system, HSRL delivered an 18.421% increase in conversion rate (CVR) with a cost increase of only 1.251%.

Ablation confirms that hierarchical structure, entropy regularization, behavioral cloning, and the multi-level critic all contribute materially to performance and learning efficiency. HSRL’s design allows deployment without changes to upstream recommendation infrastructure, as decision-making over SIDs can be seamlessly mapped to items via the offline codebook.

6. Theoretical and Practical Implications

By decoupling the policy from the item-centric action space, HSRL resolves the representation-decision mismatch inherent in RL-based recommenders with dynamic item pools. Hierarchical residual state modeling ensures that each policy stage operates over an appropriately abstracted context, preventing information redundancy and mode collapse at lower semantic levels. The use of SAS and SIDs ensures stability against catalog churn, supporting continual training and deployment.

The multi-level critic’s token-level TD estimates improve policy optimization in autoregressive RL settings, where delayed or sparse rewards complicate standard policy gradient methods. This architecture materially enhances scalability, stability, and credit assignment in large-scale recommendation environments.

7. Relevant Mathematical Formulation

Component	Formula	Description
MDP definition	$$\mathcal{M} = (\mathcal{S}, \mathcal{A}_{\text{SAS}}, \mathcal{P}, \mathcal{R}, \gamma)$$	Markov Decision Process on SAS
SID generation	$z(i) = [z_1(i), ..., z_L(i)]$	Residual quantization tokens per item
Per-level action	$$p(z_\ell \mid c_{\ell-1}) = \mathrm{softmax}(W_\ell \cdot c_{\ell-1})$$	Token distribution per semantic level
Context update	$c_\ell = \mathrm{LayerNorm}(c_{\ell-1} - e_\ell)$	Residual semantic context
Multi-level value	$\hat{V}(s) = \sum_{\ell=0}^L w_\ell V(s,\ell)$, $\;\; w_\ell = \frac{\exp(w_\ell)}{\sum_j\exp(w_j)}$	Weighted token-level value aggregation
Policy likelihood	$$\pi_\theta(z \mid s) = \prod_{\ell=1}^L p(z_\ell \mid c_{\ell-1})$$	Joint policy over SID sequence

The above architecture and empirically validated outcomes establish HSRL as a scalable, interpretable, and production-ready solution for RL-based recommendation under dynamic action space constraints (Wang et al., 10 Oct 2025).