Score-Based Action Learning

• Score-based action learning is a framework that uses gradients of log-density and scalar feedback to guide policy learning in settings like offline RL, imitation, and action quality assessment.
• It employs diffusion models and score-matching techniques to generate multimodal action distributions, enhancing data efficiency and robustness.
• The approach integrates counterfactual signals and regret-based scoring to accelerate credit assignment and optimize policies in dynamic, strategic environments.

Score-based action learning refers to the class of methods in which the concept of a "score"—interpreted as the gradient of log-density in generative models, as quantitative human-provided scalar or trajectory evaluations, or as counterfactual signals such as regret—is central to policy learning, action selection, or action evaluation. This paradigm encompasses a diverse range of settings, including score-based diffusion policies in offline and imitation learning, reinforcement learning using scalar trajectory scores or regret-based stepwise feedback, and fine-grained action assessment in domains such as sports or gaming. Contemporary methods leverage both the mathematical underpinnings of score-matching and the practical utility of human or learned scoring signals to achieve data efficiency, robustness, multimodal action generation, and interpretability in a range of learning settings.

1. Mathematical Foundations of Score-Based Action Learning

Two foundational interpretations of "score" are prevalent: (i) the score function as the gradient of the log-probability density with respect to action or state variables (i.e., $\nabla_a \log p(a|s)$ or $\nabla_x \log p(x)$), primarily in score-based generative modeling; and (ii) externally provided or learned scalar assessments of action or trajectory quality, either as trajectory returns, direct human ratings, or regret-based signals.

In score-based generative models such as diffusion policies, the policy $\pi_\theta(a|s,g)$ is defined implicitly by modeling the reverse-time diffusion of data distributions and learning the score function $$s_\theta(a_t, t; s, g) \approx \nabla_{a_t} \log p_t(a_t|s,g)$$. These models are optimized via denoising score matching, e.g. minimizing

$$\mathcal{L}(\theta) = \mathbb{E}_{(s,g,a), \sigma, \varepsilon} \left[ \alpha(\sigma) \left\| D_\theta(a + \varepsilon, s, g, \sigma) - a \right\|^2 \right]$$

and used to reconstruct action samples from noise through iterative denoising steps (Reuss et al., 2023).

In scalar score learning, human teachers or automatic critics provide scores $s_i \in [0,10]$ to agent trajectories $\tau_i$, which are then used to train learned reward functions $r_\theta(s,a)$ or preference models in interactive RL frameworks (Liu et al., 2023).

Regret-based frameworks define an instantaneous score as the difference between the value of the current and the optimal action (according to a reference "teacher" Q-network), i.e.,

$$\mathrm{regret}_t = Q_\text{opp}(s_t, a^*_\text{opp}(s_t)) - Q_\text{opp}(s_t, a_t)$$

which is incorporated as a dense per-step feedback signal in the RL update (Xu, 3 Feb 2026).

2. Score-Based Generative Policies and Diffusion Methods

Score-based generative diffusion models have emerged as high-expressivity action policies in both goal-conditioned imitation learning (GCIL) and offline reinforcement learning.

In BESO (BEhavior generation with ScOre-based diffusion policies) (Reuss et al., 2023), a transformer-based denoising network $D_\theta$ approximates the score function, enabling both goal-conditioned and unconditional policy learning via classifier-free guidance (CFG). Sampling employs fast ODE solvers (e.g. 3-step DDIM), dramatically reducing latency relative to prior diffusion-based policies while expressing multi-modal action distributions. Training is decoupled from environment interaction and relies solely on play data.

Conservative Denoising Score-based Algorithms (CDSA) employ denoising score-matching to model $\nabla_a \log p_d(s,a)$ and $\nabla_s \log p_d(s,a)$ from offline datasets. Learned score fields are used to adjust actions generated by a pre-trained policy towards higher data density, reinforcing conservatism and improving risk metrics (Liu et al., 2024).

Contractive Diffusion Policies (CDPs) introduce contraction in the sampling dynamics by penalizing the largest symmetric Jacobian eigenvalue of the score with respect to actions, thereby conferring robustness to discretization, score-matching errors, and seed sensitivity in offline control (Abyaneh et al., 2 Jan 2026).

Recent advances in fine-tuning diffusion models align score-based policies with human preference using continuous-time RL. The "Scores as Actions" formalism realizes policy optimization where the diffusion score function is treated as the controller and fine-tuned with terminal reward regularization and pathwise KL penalties, using a stochastic control and continuous-time policy gradient framework (Zhao et al., 2024, Zhao et al., 3 Feb 2025).

Model/Framework	Key Idea	Domain/Application
BESO (Reuss et al., 2023)	Score-based diffusion policy, GCIL	Goal-conditioned imitation, play dataset offline RL
CDSA (Liu et al., 2024)	Conservative action refinement	Offline RL, D4RL
CDP (Abyaneh et al., 2 Jan 2026)	Contractive dynamics via Jacobian loss	Robust offline IL/RL, real-world robot manipulation
Scores as Actions (Zhao et al., 2024, Zhao et al., 3 Feb 2025)	Score as continuous-time action	Fine-tuning diffusion T2I with RLHF

3. Score Signals in Interactive RL and Action Quality Assessment

Score-based action learning also encompasses settings where scalar feedback or trajectory-level scores provide the primary learning signal.

Interactive RL with scalar human feedback, as in "Boosting Feedback Efficiency by Adaptive Learning from Scores" (Liu et al., 2023), leverages dense trajectory scores instead of pairwise preferences, utilizing adaptive label smoothing and prioritized sampling to mitigate scorer inconsistency and maximize feedback efficiency. The reward function is trained via a preference model and integrated with off-policy actor-critic learning (SAC), yielding up to $3\times$ reduction in human feedback for near-optimal policy acquisition on standard tasks.

In action-quality assessment, such as sporting events or multiplayer online games, dedicated architectures map action sequences to scalar quality scores. For Olympic event assessment, 3D CNNs extract spatiotemporal features and regress to scalar action scores using SVR or LSTM-based frameworks (Parmar et al., 2016), with partial-score temporal structure aiding interpretable feedback. In multiplayer games, action sequences are scored by their contribution to team outcome using GRU-based deep models, where model training is fully supervised only at the match level (Jang et al., 2022).

4. Regret and Counterfactual Scoring in Reinforcement Learning

Regret-based score signals provide a dense, informative shaping signal in domains where rewards are sparse or delayed. The StepScorer/PRM mechanism computes regret at every step as the difference between the Q-value of the agent's action and the optimal action (per a fixed teacher network), integrating this measure into the reward signal used for policy gradient updates (Xu, 3 Feb 2026). This approach, grounded in behavioral economics, improves credit assignment, accelerates convergence, and stabilizes learning on continuous control tasks, exemplified by large speedups in LunarLander-v3 benchmarks.

5. Score-Based Approaches in Game-Theoretic and Discrete Action Domains

Score-based action learning carries nontrivial implications for agent preference and optimality in strategic settings. As established in "Score vs. Winrate in Score-Based Games" (Pasqualini et al., 2022), optimizing for expected score rather than win/lose can induce risk-seeking (higher score variance) when losing and risk-aversion when ahead, resulting in suboptimal winrates under function approximation or search error. This highlights the necessity of reward design aligned with the game’s true objective.

Hybrid estimators combining score-based (likelihood ratio) and pathwise (reparameterization) gradients, as in the Relaxed Policy Gradient (RPG) method (Levy et al., 2017), extend sample-efficient policy optimization to discrete action spaces by relaxing deterministic dynamics and unifying continuous and discrete-action updates in a low-variance fashion. The RPG framework yields large sample-complexity reductions over pure REINFORCE and evolution strategies on classic benchmarks.

6. Practical Considerations and Limitations

Key empirical takeaways across score-based action learning paradigms include:

• Decoupled score learning and sampling enables architectural flexibility and accelerated inference (Reuss et al., 2023).
• Score-matching regularization naturally enforces multimodal expressivity and risk aversion via data density gradients (Liu et al., 2024, Abyaneh et al., 2 Jan 2026).
• Contractive regularization improves dynamical robustness with minimal computational overhead (Abyaneh et al., 2 Jan 2026).
• Label smoothing and adaptive feedback selection mitigate noise in human-based scoring (Liu et al., 2023).
• Score signals as stepwise regret accelerate learning and credit assignment in sparse reward domains (Xu, 3 Feb 2026).
• Score-based training can negatively impact outcome-optimality in game-theoretic settings under misspecified reward proxy (Pasqualini et al., 2022).

Limitations include computational cost for high-dimensional SDE simulation, sensitivity to hyperparameters (e.g. contraction weight, regularization penalties), challenge of value estimation in continuous-time RL fine-tuning, and potential overfitting to the scoring function rather than the ultimate task metric. Theoretical guarantees are typically local and assumptions for global safety, especially in OOD regimes, remain open problems.

7. Significance and Current Frontiers

Score-based action learning unites advances in generative modeling (diffusion/SDE approaches), interactive feedback (trajectory-level scoring), counterfactual reasoning (regret), and low-variance policy gradients across continuous and discrete actions. Its influence is expanding, underpinning state-of-the-art results in offline RL, imitation learning, generative model alignment (RLHF with diffusion policies), robust feedback-efficient RL, and interpretable action attribution in strategic domains. Ongoing work addresses solver-agnostic continuous-time RL for diffusion fine-tuning, hybridization with value-based and conservative offline RL, and the extension of contractive, score-based regularization to diverse learning architectures (Zhao et al., 2024, Zhao et al., 3 Feb 2025, Abyaneh et al., 2 Jan 2026).