Logical Reward Decomposition in RL

• Logical Reward Decomposition is a framework that decomposes reward signals into logical, interpretable components for compositional task solving and enhanced generalization.
• It employs formal languages such as Linear Temporal Logic and reward machines to encode tasks and transform non-Markovian rewards into Markovian signals.
• Practical implementations demonstrate improved interpretability, multi-agent coordination, and performance in complex environments through modular and hierarchical reward structures.

Logical Reward Decomposition refers to the formal structuring, learning, and application of decomposed reward signals in machine learning and reinforcement learning (RL), particularly by encoding rewards as logical or semantically structured components. The field spans temporal logic–based reward shaping, finite-state automata encodings (“reward machines”), axiomatic and functional decompositions, and data-driven or judge-annotated criterion-based reward signals. Logical reward decomposition is deployed to enable interpretability, compositional task solving, generalization, and efficient multi-agent or multi-task RL.

1. Formalisms: Logical Task Decomposition and Specification

Logical reward decomposition is operationalized using formal languages, most notably Linear Temporal Logic (LTL), Boolean specification formulas, and reward machines.

• LTL-based Decomposition: A task is specified as a co-safe LTL formula $\varphi$ over a finite set of atomic propositions $\mathcal{AP}$, built from the grammar:

$$\varphi ::= p\ |\ \neg\varphi\ |\ \varphi_1 \wedge \varphi_2\ |\ \varphi_1 \vee \varphi_2\ |\ \bigcirc\varphi\ |\ \Box\varphi\ |\ \Diamond\varphi\ |\ \varphi_1\ U\ \varphi_2\ |\ \varphi_1\ R\ \varphi_2$$

with semantics defined on truth-assignment traces $\sigma = \langle\sigma_0, \ldots, \sigma_t\rangle$ (Liu et al., 2024).

• Reward Machines (RM, HRM): Tasks are encoded as finite automata whose transitions are labeled with logical formulas over high-level events $P$ (Furelos-Blanco et al., 2022). Hierarchical Reward Machines (HRM) further allow RMs to call sub-machines, supporting modular subgoal structure and hierarchical execution.
• Boolean Non-Markovian Specifications: A "specification" $\varphi$ is a subset of trace space $\varphi \subset (S \times A)^\tau$, indicating which trajectories are "rewarded" (accepting). Logical composition (conjunction, disjunction, implication) admits safe, interpretable compositionality (Vazquez-Chanlatte et al., 2017).
• Judge-based Decomposition: For language or reasoning tasks, rubrics with $M$ weighted criteria $R = \{c_1, ..., c_M\}$ define reward as $r(x, y) = \sum_j \alpha_j z_j(x,g,y)$, where $\mathcal{AP}$0 are normalized per-criterion scores assigned by a judge LLM or program (Bhattarai et al., 8 May 2026).

2. Algorithms for Logical Reward Extraction and Shaping

Logical reward decomposition supports both hand-crafted and learned extraction of task structure and shapes learning via progressive, interpretable signals.

a. LTL Progression and Markovization

• Reward Shaping via LTL Progression: The non-Markovian terminal reward

$\mathcal{AP}$1

is Markovized by progressing $\mathcal{AP}$2 after each step: $\mathcal{AP}$3, and defining

$\mathcal{AP}$4

(Liu et al., 2024).

b. Learning Logical Specifications from Demonstrations

• MAP Inference of Temporal Logic: The most likely specification $\mathcal{AP}$5 is inferred from demonstrations via a closed-form max-entropy MAP objective

$\mathcal{AP}$6

where $\mathcal{AP}$7 is the number of demos satisfying $\mathcal{AP}$8, and $\mathcal{AP}$9 is its satisfaction rate under uniform random play (Vazquez-Chanlatte et al., 2017).

c. Hierarchical Policy Learning

• Meta-Controller and Subgoal Policies: In hierarchical MAHRL, the agent maintains a meta-controller to select unresolved logical subgoals, while sub-policies solve subtasks, each optimized with shaped rewards tied to LTL progression or RM subgoal acceptance (Liu et al., 2024, Furelos-Blanco et al., 2022).

3. Multi-Component and Structured Reward Decomposition

Modern reward decomposition often uses explicit multi-component structures, where each component targets an interpretable behavioral, logical, or functional axis.

Framework	Decomposition Type	Reward Definition
LTL / RM / HRM	Logical/temporal (structural)	Progression/acceptance–based
Independently-Obtainable Rewards	Functional (policy-disentangling)	Learn sub-rewards $$\varphi ::= p\ |\ \neg\varphi\ |\ \varphi_1 \wedge \varphi_2\ |\ \varphi_1 \vee \varphi_2\ |\ \bigcirc\varphi\ |\ \Box\varphi\ |\ \Diamond\varphi\ |\ \varphi_1\ U\ \varphi_2\ |\ \varphi_1\ R\ \varphi_2$$0; sum to $$\varphi ::= p\ |\ \neg\varphi\ |\ \varphi_1 \wedge \varphi_2\ |\ \varphi_1 \vee \varphi_2\ |\ \bigcirc\varphi\ |\ \Box\varphi\ |\ \Diamond\varphi\ |\ \varphi_1\ U\ \varphi_2\ |\ \varphi_1\ R\ \varphi_2$$1
Rubric-Grounded RL	Verifiable multi-criterion (rubric-based)	$$\varphi ::= p\ |\ \neg\varphi\ |\ \varphi_1 \wedge \varphi_2\ |\ \varphi_1 \vee \varphi_2\ |\ \bigcirc\varphi\ |\ \Box\varphi\ |\ \Diamond\varphi\ |\ \varphi_1\ U\ \varphi_2\ |\ \varphi_1\ R\ \varphi_2$$2
ToolRLA	Orthogonal correctness/format/compliance	$$\varphi ::= p\ |\ \neg\varphi\ |\ \varphi_1 \wedge \varphi_2\ |\ \varphi_1 \vee \varphi_2\ |\ \bigcirc\varphi\ |\ \Box\varphi\ |\ \Diamond\varphi\ |\ \varphi_1\ U\ \varphi_2\ |\ \varphi_1\ R\ \varphi_2$$3 (with multiplication in $$\varphi ::= p\ |\ \neg\varphi\ |\ \varphi_1 \wedge \varphi_2\ |\ \varphi_1 \vee \varphi_2\ |\ \bigcirc\varphi\ |\ \Box\varphi\ |\ \Diamond\varphi\ |\ \varphi_1\ U\ \varphi_2\ |\ \varphi_1\ R\ \varphi_2$$4) (Liu, 2 Mar 2026)
Sycophancy Disentanglement (GRPO)	Behavioral axis–targeted (5 terms)	Linear sum: pressure, fidelity, etc.
Vision-Language Confidence Decomposition	Skill-clustered (perception/reasoning)	Intra-cluster normalized advantages

Notably, (Liu et al., 2024, Furelos-Blanco et al., 2022, Liu, 2 Mar 2026, Bhattarai et al., 8 May 2026, Mohsin et al., 7 Apr 2026) all implement multi-component decomposed reward with explicit logic or axiomatic separation.

4. Theoretical Properties and Performance Guarantees

Reward decomposition frameworks offer uniquely formal guarantees, transfer, and learning speedups relative to monolithic scalar rewards.

• Transformation Equivalence: Any non-Markovian logical reward game with LTL rewards can be transformed via progression into a Markov game that preserves optimal policies (Liu et al., 2024).
• Saturation and Disjointness: When maximizing the disentanglement objective for independently-obtainable rewards, optimal solutions assign the environment reward to exactly one sub-reward per state, inducing near-disjoint policies and state partitions (Grimm et al., 2019).
• Compositional Safety: Boolean specifications and HRMs guarantee safe recombination: logical subgoal satisfaction is preserved under conjunction, preventing the reward hacking that can occur with scalar reward summation (Vazquez-Chanlatte et al., 2017, Furelos-Blanco et al., 2022).
• Reward Decomposition Theorems: In multimodal or multi-component RL (e.g., ToolRLA, Visual-ARFT), the sub-optimality gap between independently optimizing reward components and joint optimization is upper-bounded as $$\varphi ::= p\ |\ \neg\varphi\ |\ \varphi_1 \wedge \varphi_2\ |\ \varphi_1 \vee \varphi_2\ |\ \bigcirc\varphi\ |\ \Box\varphi\ |\ \Diamond\varphi\ |\ \varphi_1\ U\ \varphi_2\ |\ \varphi_1\ R\ \varphi_2$$5 times the average pairwise covariance plus normalization error, where $$\varphi ::= p\ |\ \neg\varphi\ |\ \varphi_1 \wedge \varphi_2\ |\ \varphi_1 \vee \varphi_2\ |\ \bigcirc\varphi\ |\ \Box\varphi\ |\ \Diamond\varphi\ |\ \varphi_1\ U\ \varphi_2\ |\ \varphi_1\ R\ \varphi_2$$6 is component count and $$\varphi ::= p\ |\ \neg\varphi\ |\ \varphi_1 \wedge \varphi_2\ |\ \varphi_1 \vee \varphi_2\ |\ \bigcirc\varphi\ |\ \Box\varphi\ |\ \Diamond\varphi\ |\ \varphi_1\ U\ \varphi_2\ |\ \varphi_1\ R\ \varphi_2$$7 is group size (Adams et al., 21 Apr 2026). When reward axes are weakly correlated, decomposition is near-optimal.

5. Practical Implementations and Empirical Benefits

Application domains for logical reward decomposition span multi-agent systems, tool-augmented and vision-LLMs, alignment tasks, and dialogue agents.

• Multi-Agent Hierarchical RL: Logical subgoal-extraction via LTL enables multi-task learning, interpretable execution traces, and agent coordination; experiments confirm improved completion rates and agent cooperation in Minecraft-like environments (Liu et al., 2024).
• Dialogue and Alignment: LLM-based reward decomposition translates session-level feedback into turn-level labels, supporting robust RL fine-tuning and generalization in settings with weak supervision (Lee et al., 21 May 2025).
• Tool-Integrated Agents: Fine-grained, logically-structured reward (e.g., ToolRLA's gating, multiplicative correctness, and vetoed compliance penalty) yields substantial error rate reductions and improved compliance versus additive or monolithic reward (Liu, 2 Mar 2026).
• De-sycophancy in LLMs: Decomposed reward over discrete behavioral axes (pressure resistance, factuality, etc.) isolates and corrects failure modes otherwise conflated by scalar judge models (Mohsin et al., 7 Apr 2026).

Paper	Domain	Empirical Effect
(Liu et al., 2024)	Multi-agent MAHRL	Improved task completion, enhanced interpretability
(Furelos-Blanco et al., 2022)	Hierarchical RL	2–10x speedup, scalable to long-horizon composition
(Liu, 2 Mar 2026)	Tool-integrated RL	47% higher completion, 93% lower violation
(Bhattarai et al., 8 May 2026)	Reasoning with rubrics	+5.13pp transfer accuracy, reduction in null rewards
(Lee et al., 21 May 2025)	Dialogue agent alignment	$$\varphi ::= p\ |\ \neg\varphi\ |\ \varphi_1 \wedge \varphi_2\ |\ \varphi_1 \vee \varphi_2\ |\ \bigcirc\varphi\ |\ \Box\varphi\ |\ \Diamond\varphi\ |\ \varphi_1\ U\ \varphi_2\ |\ \varphi_1\ R\ \varphi_2$$8 reduction in global loss over baselines
(Mohsin et al., 7 Apr 2026)	Sycophancy control	Up to 17pp reduction in sycophancy metrics

6. Advanced Topics: Heterogeneous and Programmatic Decomposition

• Skill/Cluster-Aligned Decomposition: In heterogeneous settings (e.g., vision-language), step-level reward is decomposed intra-cluster according to unsupervised skill partition (visual vs. textual reasoning), using programmatic metrics such as Visual Dependence Score and per-cluster normalization. This prevents majority skill dominance and restores meaningful learning signals in minority-step clusters (Yoon et al., 13 May 2026).
• Multiplicative vs. Additive Composition: For correctness dimension as in ToolRLA, multiplicative composition of subrewards (e.g., tool name validity $$\varphi ::= p\ |\ \neg\varphi\ |\ \varphi_1 \wedge \varphi_2\ |\ \varphi_1 \vee \varphi_2\ |\ \bigcirc\varphi\ |\ \Box\varphi\ |\ \Diamond\varphi\ |\ \varphi_1\ U\ \varphi_2\ |\ \varphi_1\ R\ \varphi_2$$9 completeness $\sigma = \langle\sigma_0, \ldots, \sigma_t\rangle$0 parameter accuracy) enforces prerequisite chain logic otherwise violated by additive rewards, sharply reducing pathological solutions (Liu, 2 Mar 2026).
• Normalization and Variance Attenuation: Partial-credit and multi-axis normalization (rubric or GRPO-based methods) stabilize policy updates and improve credit assignment, particularly with noisy or discrete reward models (Bhattarai et al., 8 May 2026, Mohsin et al., 7 Apr 2026).

7. Open Challenges and Future Directions

Despite robust formal properties, logical reward decomposition systems face domain-specific calibration issues, prompt and rubric sensitivity, and risk of semantical drift when deploying learned or LLM-inferred sub-reward oracles (Lee et al., 21 May 2025, Bhattarai et al., 8 May 2026). Future work focuses on robustifying the extraction of logical decompositions from weak feedback, integrating continuous state and action abstractions with formal logic, and automating the design of compositional reward structures in novel domains.

---

References (by arXiv ID):

• (Liu et al., 2024, Furelos-Blanco et al., 2022, Vazquez-Chanlatte et al., 2017, Grimm et al., 2019, Lee et al., 21 May 2025, Liu, 2 Mar 2026, Adams et al., 21 Apr 2026, Yoon et al., 13 May 2026, Bhattarai et al., 8 May 2026, Mohsin et al., 7 Apr 2026)