SCORER: Stackelberg RL Framework

• SCORER is a game-theoretically motivated framework that models RL as a Stackelberg leader–follower game to couple perception and control effectively.
• It employs a two-timescale bi-level optimization where the perception network anticipates the control network’s rapid adaptations for enhanced learning efficiency.
• Empirical findings indicate that SCORER achieves faster convergence and higher final returns compared to conventional DQN variants across diverse benchmarks.

Stackelberg Coupled Representation and Reinforcement Learning (SCORER) is a game-theoretically motivated framework for integrating distinct representation learning and policy optimization modules in reinforcement learning (RL) via a leader–follower structure. It formalizes the interplay between a perception (representation) network and a control (policy) network as a Stackelberg game, seeking to improve sample efficiency and final policy performance through anticipatory, bi-level optimization dynamics.

1. Stackelberg Game-Theoretic Formulation

SCORER frames RL agent design as a two-player Stackelberg game: the perception network $f_{\phi}$ (“leader”) strategically adapts features to benefit the control network $Q_{\theta}$ (“follower”), which seeks to minimize its BeLLMan error given the current latent representation. Formally, the equilibrium structure is characterized by a bi-level program: $$\min_{\phi} \;\; \mathcal{L}_{\mathrm{leader}}(f_{\phi}, Q_{\theta^*(\phi)}) \quad\text{where}\quad \theta^*(\phi) = \arg\min_{\theta} \;\; \mathcal{L}_{\mathrm{follower}}(\theta, \phi)$$ Here, $\mathcal{L}_{\mathrm{follower}}(\theta, \phi)$ is typically the Mean Squared BeLLMan Error (MSBE), and $\mathcal{L}_{\mathrm{leader}}$ measures the quality of representations by the effect on the follower’s task (e.g., variance or magnitude of MSBE). This explicit leader–follower hierarchy operationalizes the coupling by requiring the leader to anticipate the follower’s update path.

The approach contrasts with end-to-end joint optimization, which interleaves representation and policy updates with no explicit modeling of their interaction, and with methods employing auxiliary or self-supervised objectives for representation shaping.

2. Two-Timescale Bi-Level Optimization

SCORER employs a two-timescale stochastic optimization algorithm to approximate the Stackelberg equilibrium:

• Follower (control head) update: Parameter $\theta$ is updated on a fast timescale using standard RL techniques (e.g., DQN MSBE) with the representation fixed:

$$\theta_{k+1} \leftarrow \theta_k - \alpha_\theta \nabla_{\theta} \mathcal{L}_{\mathrm{follower}}(\theta_k, f_{\phi}(s))$$

The representation branch $f_{\phi}$ is “detached” when computing gradients, preventing gradient leakage through $\phi$ along this path.

• Leader (perception head) update: Parameter $\phi$ is updated more slowly, optimizing a leader loss with respect to pre-update follower weights (i.e., “anticipating” the result of the follower’s rapid adaptation):

$$\phi_{k+1} \leftarrow \phi_k - \alpha_\phi \nabla_{\phi} \mathcal{L}_{\mathrm{leader}}(f_{\phi}, Q_{\theta_{\mathrm{bu}}})$$

Here, $Q_{\theta_{\mathrm{bu}}}$ are follower parameters from before the most recent batch of updates, which provides a “look-ahead” effect. The cross-derivative term $\nabla_{\phi} \theta^*(\phi)$ is ignored in practice due to timescale separation.

This scheme aims to capture essential Stackelberg dynamics—anticipatory adjustment of representations to the trajectory of control updates—while remaining computationally feasible, in contrast to full implicit bi-level gradient methods.

3. Leader and Follower Objectives

Key loss functions include:

Network	Update Frequency	Loss	Notes
Perception ($\phi$)	Slow	$$\mathcal{L}_{\mathrm{leader}}(f_{\phi}, Q_{\theta_{\mathrm{bu}}})$$	MSBE, variance, or Var+representation norm; uses pre-update Q-network
Control ($\theta$)	Fast	$$\mathcal{L}_{\mathrm{follower}}(\theta, \phi)\!=\!\mathbb{E}[(Y-Q_\theta(f_\phi(s), a))^2]$$	Follows standard DQN/Double DQN/variant loss on detached feature encoding

Variants explored include $\mathcal{L}_{\mathrm{leader}} = \text{MSBE}$, $$\mathcal{L}_{\mathrm{leader}} = \mathrm{Var}(\mathrm{MSBE})$$ to encourage stability, and combinations with $\ell_2$ normalization of feature vectors.

4. Empirical Performance and Findings

Empirical results demonstrate that SCORER provides substantial performance gains over standard DQN variants (including Double DQN, Dueling DQN) and baseline decoupled or naive end-to-end approaches on a range of benchmarks (Breakout-MinAtar, Asterix, Space Invaders):

• Sample efficiency: SCORER MSBE nearly doubles final performance in Breakout-MinAtar and reaches fixed reward thresholds in approximately half the number of environment steps relative to DQN baselines.
• Final performance: On multiple tasks, SCORER improves interquartile mean (IQM) returns and achieves statistically significant gains (via Welch’s t-test) over baseline methods.
• Design ablations: Comparing against “team coupling” (non-anticipatory), the anticipatory leader update in SCORER results in both faster convergence and more stable, higher final returns.

SCORER’s structured anticipation is shown to be essential: leader loss conditioned on the follower’s pre-update (“look-ahead”) state enables the perception network to “steer” the representation space toward control-favorable regions.

5. Structural Advantages and Theoretical Principles

The SCORER framework demonstrates that effective coupling of representation and policy in RL does not require:

• Architectural complexity (e.g., multiple feature heads, heavy multi-task decoders)
• External auxiliary/self-supervised tasks
• Differentiation through full follower update “unrolling” (as in full bi-level optimization)

Instead, the simple two-timescale Stackelberg game abstraction ensures that the perceptual module directly supports the RL task, efficiently reducing the BeLLMan error along the control branch. This approach regularizes representation learning by using the RL objective itself, thereby improving generalization and robustness with minimal algorithmic overhead.

6. Broader Impact and Applicability

SCORER generalizes across diverse RL algorithms and tasks. It can be applied wherever the agent architecture admits a separation between perception and control, including value-based, actor–critic, and model-based RL. Its bi-level design opens avenues for future extensions such as:

• Multi-level Stackelberg games (e.g., hierarchical or multi-stage networks)
• Structured anticipation in meta-learning and continual learning contexts
• Real-world applications where sample efficiency and robust feature shaping under sparse rewards are required

Its principled formulation as a Stackelberg game positions SCORER as a foundational approach for further research on the algorithmic design of tightly coupled RL representation–policy systems.