Universal Decision Learners

Abstract: Many theories of decision making -- planning, reinforcement learning, causal intervention, online learning, and game-theoretic equilibrium -- turn local information into globally coherent behavior. This paper proposes a common categorical formulation: a Universal Decision Learner (UDL) extends a partially specified decision functor from observed contexts to new contexts by a pair of universal constructions. Left Kan extensions express rollout, aggregation, and candidate generation; right Kan extensions express consistency, constraint satisfaction, and fixed-point semantics. The central claim is not that every decision problem has the same algorithm, but that many decision formalisms instantiate the same universal problem: extend local behavioral data canonically, then characterize the globally coherent extensions. We give the abstract UDL construction, prove its universal comparison property, define Kan-invariant behavioral equivalence and minimal abstractions, and show how Bellman equations, planning recursions, causal interventions, online regret, and equilibria arise as special cases. The supplementary material develops the reinforcement-learning specialization in more detail.

• The paper introduces a unifying framework that uses Kan extensions to extend local decision data into globally consistent models.
• It formalizes decision-making across planning, reinforcement learning, game theory, online learning, and causal inference using categorical constructs.
• The approach offers principled criteria for model abstraction and fixed-point semantics, with significant implications for deep RL and algorithm design.

Universal Decision Learners: A Categorical Synthesis of Decision-Making Paradigms

Abstract Framework and Contribution

The concept of Universal Decision Learners (UDLs) introduces a categorical framework to unify diverse decision-making paradigms such as reinforcement learning (RL), planning, causal inference, online learning, and game-theoretic equilibrium (2605.30694). Rather than focusing on algorithmic commonalities, UDLs formalize the semantic task shared by these settings: canonically extending partial, local decision data to globally coherent decision-making structures. The core apparatus is the Kan extension, with left Kan extensions capturing rollout and aggregation (e.g., planning trajectories or interpolation of local knowledge), and right Kan extensions capturing consistency, constraint satisfaction, and fixed-point semantics (as in Bellman equations or Nash equilibria).

The UDL formalism is not an algorithmic proposal but a universal semantic framework. It operationalizes decision learning as a two-stage categorical process: first, the left Kan extension propagates and aggregates local decision models (forming candidate solutions); then, the right Kan extension enforces global consistency, yielding the set of decision models agreeing with all downstream constraints and observable consequences.

Kan Extensions and Decision Learning

Let $D$ be a subcategory of observed or locally accessible decision contexts, $C$ the target (global) context category, and $F: D \to E$ a functor representing local decision data into a value category $E$. The inclusion $J: D \to C$ expresses how local contexts relate to global ones.

• Left Kan Extension ($Lan_J F$): Universally "rolls out" local data to new contexts, aggregating over all ways of reaching a target context. This captures forward semantics: planning (path rollouts), aggregating memory fragments, or constructing candidate behaviors.
• Right Kan Extension ($Ran_J F$): Universally applies constraints and consistency, solving for global decision semantics compatible with all local observations. This realizes backward semantics: Bellman-style value consistency, equilibrium fixed points, or feasibility constraints.

A Universal Decision Learner is defined via the composite:

$UDL_J(F) = Ran_J(Lan_J F)$

This composition first propagates local knowledge, then filters for global consistency.

Theoretical Properties and Universalities

Two main universal properties establish UDLs as canonical within the class of global decision models:

1. Canonical Rollout: $Lan_J F$ is initial among all global decision models extending the local functor $F$. Any comparison with another global model factors through $C$0, guaranteeing minimality and universality of rollout.
2. Canonical Consistency: $C$1 is terminal among all global decision models whose localizations agree with $C$2. Any global model compatible with $C$3 factors uniquely through $C$4.

This duality underpins the UDL comparison principle: a global model is semantically equivalent to the UDL precisely when it is isomorphic under the canonical comparison induced by Kan extension.

Fixed Point Semantics and Coinduction

A significant class of problems falls within the right Kan extension's fixed-point semantics. For example, RL's Bellman equation corresponds to the right Kan fixed point. The UDL view formalizes value functions as globally consistent assemblies of local continuation constraints, and the fixed points are characterized via (potentially metric) coinductive reasoning. The coalgebraic structure embraces not only standard MDPs but also PSRs, partially observed systems, and distributed or asynchronous environments.

Abstraction, Bisimulation, and Representation

Abstraction is encoded via Kan invariance and Kan bisimulation:

• Kan-Invariant Equivalence: Two contexts are identified if their Kan-extended behaviors are naturally isomorphic.
• Minimal Kan-Invariant Quotient: There is always a coarsest abstraction preserving decision semantics, constructed via quotienting the context category according to Kan-equivalence classes.

This semantics-driven abstraction generalizes classical bisimulation in MDPs and underlies the correctness of representational learning and model compression.

A further generalization incorporates homotopy Kan invariance, recognizing models as equivalent if their universal semantics are equivalent up to continuous deformation. This is salient in causal inference (identifiability modulo latent structure) and approximate RL (semantic invariance under function approximation).

Specializations to Classical Problems

UDLs instantiate established formalisms as follows:

• Planning: Left Kan extension computes optimal rollouts over aggregated plan fragments.
• Reinforcement Learning: Right Kan extension encodes Bellman consistency; classical algorithms approximate this fixed point within restricted hypothesis classes (cf. projected Kan extension in function approximation).
• Game Theory: Equilibrium conditions correspond to global sections surviving all best-response (consistency) constraints, i.e., a right Kan fixed point.
• Online Learning: Left Kan extension aggregates losses; right Kan extension defines consistency with comparator classes; sublinear regret corresponds to approximate right Kan consistency.
• Causal Inference: The extension of interventional or observational semantics arises via Kan extension; identifiability is Kan-invariance under different causal representations.

Function Approximation and Deep Reinforcement Learning

Approximate computation of Kan extensions is essential in contemporary RL/artificial intelligence, especially with large or structured state spaces. Deep RL variants (e.g., DQN, graph neural architectures) can be recognized as learning projected right Kan extensions, where parameterized hypothesis spaces (e.g., neural nets, basis expansions) only approximate the universal semantic object. The framework suggests decomposing objectives into Kan (Bellman) consistency, morphism (structural), and regularization (diagrammatic coherence) losses, clarifying the roles of learned representations as approximate coalgebra morphisms preserving Kan-extended semantics.

Implications and Future Directions

The UDL framework provides a rigorous semantic backbone for decision making across domains. Practically, it supplies principled criteria for comparing and designing algorithms through the lens of universal semantic objects, independently of the specifics of computation, optimization, or architecture. The minimal, semantics-driven approach to abstraction informs optimal state aggregation, representation learning, and model reduction.

Theoretically, UDL elevates the study of abstraction from syntactic equivalence to semantic indistinguishability. It supports the unification of fixed-point methods, dynamic programming, equilibrium computation, and causal identifiability within a single, compositional categorical language. The recognition of higher categorical structure (homotopy, diagrammatic identities) points toward new modes of reasoning about approximate equivalence, invariance, and robustness in AI systems.

Future developments may explore algorithmic approximations of Kan extensions in nontrivial categories, further integration with homotopical and diagrammatic machine learning, or the systematic use of UDL principle for model selection, regularization, and transfer across multiple decision-making paradigms.

Conclusion

Universal Decision Learners synthesize a diversity of decision-making settings under the framework of Kan extension, providing both a unifying theoretical account and a set of canonical semantic constructions for understanding rollout, constraint satisfaction, consistency, and abstraction. By shifting focus from algorithmic procedure to semantic extension, UDL offers a robust foundation for both practical applications in learning and planning, and for the development of new theory across AI, control, causal reasoning, and strategic interaction.