Action-Controllable Factorization (ACF)

• ACF is a framework that factorizes complex state-action spaces into independently controllable latent factors for tractable learning.
• It employs contrastive losses, mutual information clustering, and factorized policy architectures to isolate sparse action effects.
• ACF enhances sample efficiency and interpretability in reinforcement learning, model-based planning, and multi-agent systems.

Action-Controllable Factorization (ACF) refers to a family of methodologies aimed at discovering, representing, and exploiting decompositions of state and action spaces into independently controllable factors within complex dynamical systems. The fundamental objective is to enable tractable, efficient learning and planning by leveraging sparsity in the system’s underlying decoupling structure—whether known a priori or uncovered from high-dimensional observations—so that each action predominantly affects a single or a small subset of latent state variables, while all other components evolve independently or under default dynamics.

1. Theoretical Foundations and Motivation

Conventional Markov decision processes (MDPs) become intractable as the joint state-action space grows, especially in environments with many interacting entities or high-dimensional observations. Factored MDPs address this by positing that the state $s = (s_1, ..., s_K)$ and action $a = (a_1, ..., a_M)$ can be partitioned such that the transition kernel and reward function decompose:

$$P(s' | s, a) = \prod_{k} P_k(s'_k | s_k, a_k), \quad R(s, a) = f\left(R_1(s_1, a_1), ..., R_K(s_K, a_K)\right)$$

The optimal policy then factorizes as $π^*(a|s) = \prod_k π^*_k(a_k|s_k)$, yielding exponential gains in sample and computational efficiency, provided the factorization captures the environment's true dependency structure (Losapio et al., 2024, Rodriguez-Sanchez et al., 2 Oct 2025).

When state observations are high-dimensional (e.g., images), mapping observations $x$ to latent factors $z = f_\phi(x) = (z_1, ..., z_K)$ that satisfy the above factorization—without explicit knowledge of the mapping or the factors themselves—becomes central to extending this sample efficiency to deep RL and model-based planning in real domains (Rodriguez-Sanchez et al., 2 Oct 2025, Thomas et al., 2017, Wang et al., 18 Feb 2026).

2. Formal Criteria: Action-Controllable and Independently Controllable Factors

ACF seeks to recover latent variables that are both Markovian (i.e., their evolution is conditionally independent given the factors themselves and actions) and independently controllable: each action dimension or skill should primarily affect a single latent factor, with minimal interference to the others.

Mathematically, suppose $f_\phi: X \rightarrow (z_1, ..., z_K)$ is an encoder. Each $z_i$ corresponds to an independently controllable aspect of the environment if for each $i$, there exists a policy $π_i$ such that executing $a = (a_1, ..., a_M)$0 predictably changes $a = (a_1, ..., a_M)$1 while leaving $a = (a_1, ..., a_M)$2 as unchanged as possible (Thomas et al., 2017, Rodriguez-Sanchez et al., 2 Oct 2025). Quantitatively, this is formalized using selectivity or controllability losses:

$a = (a_1, ..., a_M)$3

Maximizing this selectivity encourages factor disentanglement and minimal cross-feature interference (Thomas et al., 2017).

In model-based variants, contrastive objectives compare the probability density of next states under target actions versus a reference (typically, a "no-op" action) to identify which factors undergo nontrivial transition changes per action, thus revealing sparse action-factor dependencies (Rodriguez-Sanchez et al., 2 Oct 2025).

3. Algorithms and Mechanisms for Decomposing State-Action Spaces

Several algorithmic paradigms instantiate ACF:

Mutual Information–Driven State-Action Clustering

In domain-agnostic approaches, mutual information (MI) between state/action components and next-state variables is estimated empirically:

$a = (a_1, ..., a_M)$4

A thresholded MI adjacency matrix is permuted into block-diagonal form, grouping highly correlated dimensions and defining independent sub-MDPs $a = (a_1, ..., a_M)$5. This results in data-driven segmentation of the overall system into tractable subproblems, each associated with a subset of the action space and its directly affected state components (Losapio et al., 2024).

Contrastive Latent Factorization

In pixel-based domains, ACF employs a contrastive loss leveraging action sparsity: most actions affect only a small subset of state variables. The encoder and energy-based models are trained so that for each latent factor $a = (a_1, ..., a_M)$6, the model predicts sharp transitions only under the actions controlling $a = (a_1, ..., a_M)$7, with all other factors evolving as under the no-op action (Rodriguez-Sanchez et al., 2 Oct 2025). Sampling positive and negative pairs by contrasting transitions under different actions, the model aligns each latent to its respective controllable factor.

Policy and Value Function Factorization

In action-rich discrete domains, Factored Action space Representations (FAR) decompose both policies and Q-functions into factor-specific outputs, combined (e.g., summed and softmaxed) to yield the overall policy or value estimate for joint actions:

$a = (a_1, ..., a_M)$8

Updates backpropagate through all factor-heads simultaneously from each sample, facilitating parallelized, factor-specific learning (Sharma et al., 2017).

4. Architectural Blueprints

Implementations span a range of architectures, adapted to the spatial-temporal and combinatorial nature of the underlying domain:

• Residual CNN encoders and energy-based networks for pixel observations to produce latent factors (for RL and world modeling) (Rodriguez-Sanchez et al., 2 Oct 2025).
• Slot attention and transformer blocks in factored latent world models to maintain temporal consistency of entity-specific slots and their associated latent actions (Wang et al., 18 Feb 2026).
• Parallel output heads for each action factor in deep RL architectures (FARA3C/FARAQL), exploiting the compositional structure of action spaces (Sharma et al., 2017).
• Nonparametric MI estimators and graph-theoretic block clustering for MDP factorization, enabling distributed policy optimization (Losapio et al., 2024).

5. Sample Efficiency and Empirical Performance

Empirical results consistently demonstrate that ACF approaches yield significant improvements in disentanglement, sample efficiency, and interpretability:

Approach/Domain	Metric/Effect	Results/Findings
ACF (pixels→factors)	Diag. $a = (a_1, ..., a_M)$9 (DoorKey)	0.56 vs GCL 0.32, DMS 0.40, Markov 0.48 (Rodriguez-Sanchez et al., 2 Oct 2025)
FAR on Atari 2600	Final game scores	FARA3C beats A3C on 9/14 games, FARAQL outperforms on 9/13 (Sharma et al., 2017)
FLAM (multi-entity video)	PSNR/SSIM/FVD	FLAM outperforms AdaWorld, Genie across 5 datasets (Wang et al., 18 Feb 2026)
MI-block MDPs (powergrid)	Fact. recovery error	Perfect recovery in synthetic toy, Frobenius error ≃0.02 (Losapio et al., 2024)

In all cases, the factorization reduces the complexity of learning and planning, enabling low-dimensional model-building or control in domains where monolithic RL or world modeling would be computationally prohibitive.

6. Limitations, Open Issues, and Variants

Several assumptions and open challenges apply:

• True Prior Factorization: Success depends on the environment admitting a (possibly approximate) sparse transition/reward decomposition. Weak coupling or hidden interactions between factors can degrade performance (Losapio et al., 2024, Rodriguez-Sanchez et al., 2 Oct 2025).
• Threshold and Hyperparameter Selection: MI thresholding and the number of factors $$P(s' | s, a) = \prod_{k} P_k(s'_k | s_k, a_k), \quad R(s, a) = f\left(R_1(s_1, a_1), ..., R_K(s_K, a_K)\right)$$0 often require domain-sensitive tuning; automated criteria remain an open problem (Losapio et al., 2024, Wang et al., 18 Feb 2026).
• Controllability Signals: Selectivity or contrastive objectives require that the environment’s transition dynamics are sufficiently deterministic and that each action uniquely controls a latent direction (Thomas et al., 2017, Rodriguez-Sanchez et al., 2 Oct 2025).
• Scalability: Attention-based and transformer modules may encounter computational limits with large numbers of factors or very high-dimensional observations (Wang et al., 18 Feb 2026).
• Binding and Discrete Object Representations: Current approaches struggle with variable numbers of entities or object “binding,” as there is no explicit slot-addressing in most encoders (Thomas et al., 2017, Wang et al., 18 Feb 2026).

Potential extensions include multi-step controllability, slot-based or attentive binding architectures, and richer generative priors for visual fidelity or exploration.

7. Applications and Integration in Reinforcement Learning

ACF methodologies have demonstrated utility in diverse contexts:

• Distributed and modular RL: Each discovered factor can be assigned a dedicated agent or sub-policy, facilitating distributed or federated RL, particularly effective in large-scale domains such as power grids (Losapio et al., 2024).
• Efficient model-based planning: Compact, factored world models permit use of classically efficient planners (e.g., Factored Value Iteration, DBNs), as well as interpretable goal-conditioned skills targeting specific factors (Rodriguez-Sanchez et al., 2 Oct 2025, Wang et al., 18 Feb 2026).
• Action-free sequential modeling: In video prediction and control synthesis, factored latent actions enable scene manipulation and policy learning with minimal supervision, especially in settings with multiple independently acting entities (Wang et al., 18 Feb 2026).
• Interpretable skill decomposition: Policies aligned with controllable factors can generate options or skills that target specific aspects of the environment, enhancing interpretability and transferability (Thomas et al., 2017, Rodriguez-Sanchez et al., 2 Oct 2025).

Action-Controllable Factorization provides a theoretically grounded and empirically validated toolkit for uncovering and exploiting latent dynamical structure in high-dimensional, multi-entity domains, leading to substantial gains in learning efficiency, control, and interpretation across classical and deep RL paradigms.