Tensor-Based Agents in Scalable RL

• Tensor-based agents are frameworks using tensor algebra to decompose high-dimensional decision spaces into manageable components.
• They employ low-rank factorization methods such as CP, Tucker, and tensor trains to efficiently model multi-agent reinforcement learning, robotic control, and predictive state representations.
• These methods improve scalability, parameter efficiency, and generalization, enabling successful application in traffic management, robotic manipulation, and high-DoF motion tasks.

Tensor-based agents are learning architectures and control frameworks that leverage high-order tensor representations, tensor factorization, and tensor networks to model, plan, and control decision-making in high-dimensional, structured domains. By explicitly structuring policies, value functions, dynamical models, or agent populations as tensors—and then exploiting algebraic decompositions or tensor-invariant pipelines—these agents achieve scalability, parameter efficiency, and improved generalization in problems such as multi-agent reinforcement learning, robust robotic control, and large-scale predictive state modeling.

1. Foundations of Tensor-Based Agent Methodologies

Tensor-based agents make high-order structure a first-class modeling primitive. In multi-agent reinforcement learning (MARL), the joint action–value function $Q(s, a^1, ..., a^N)$, with $N$ agents, is naturally an $(N+1)$-way tensor (state plus $N$ agent action modes). However, the parameter count in such representations scales exponentially with the number of agents or system components, rendering naive approaches intractable for $N \gg 5$.

To address this, tensor-based agents use algebraic factorization, especially low-rank representations (e.g., CANDECOMP/PARAFAC (CP), Tucker, tensor train/MPS), to approximate high-dimensional tensors by compositional objects with O($N$) or O(poly($N$)) parameters. Examples include:

• Factorized Q-Learning (FQL), in which the multi-agent Q-tensor is decomposed into independent and pairwise interaction terms, dropping all higher-order terms and achieving a manageable parameterization (Zhou et al., 2018).
• Tesseract (Tensorised Actors), which employs CP-decomposition for the Q-tensor, enabling both efficient Bellman updates and actor-critic training in deep MARL (Mahajan et al., 2021).
• Model-based RL with tensor decompositions, representing transition and reward as low-CP-rank tensors, and fitting these models via alternating least squares or power iteration (Vaart et al., 2021).

The approach generalizes to tensor-network modeling for generative world models, e.g., matrix product state (MPS) active-inference agents (Wauthier et al., 2022), or tensor-based predictive state representation (PSR) in multi-agent PSRs (Chen et al., 2020).

2. Architectures and Decomposition Schemes

Tensor-based agent designs are structured around modular neural, algorithmic, or physical modeling blocks that directly realize tensor factorization or leverage tensor invariance:

Example: FQL Composite Network

For $N$ homogeneous agents, FQL factorizes the Q-function as

$$Q^i(s, a^i, a^{-i}) \approx Q(s^i, a^i; \theta) + \lambda V(s^i, a^i; \beta_1)^\top \overline U(s^{-i}, a^{-i}; \beta_2)$$

with

$$\overline U(s^{-i}, a^{-i}; \beta_2) = \frac{1}{N-1} \sum_{j \ne i} U(s^j, a^j; \beta_2)$$

implemented with three sub-networks: Q-net, V-net, and U-net, each mapping from an agent’s perspective $N$0 to a scalar or embedding; parameters are shared within groups, yielding an O($N$1) parameter regime for $N$2 groups (Zhou et al., 2018).

Example: Tesseract Q-Tensor CP Decomposition

For a joint Q-tensor $N$3 (order $N$4), Tesseract approximates:

$N$5

allowing for $N$6 parameterization rather than $N$7 (Mahajan et al., 2021). The architecture supports both value-based and actor-critic RL (TD learning and direct policy gradients) and is PAC-sample efficient for low-rank domains.

Example: Model-based Decomposition

Transition (order $N$8) and reward (order $N$9) tensors are represented as:

$(N+1)$0

learned via alternating least squares from transitions; planning is then via standard policy/value iteration on the completed tensor model (Vaart et al., 2021).

Predictive State, Control, and MPS

• Multi-agent PSR: system-dynamics tensors, $(N+1)$1, admitting CP/Tucker decompositions for learning prediction vectors and compressed system states (Chen et al., 2020).
• Tensor-invariant robotic control: coordinate-free, non-recursive DAEs parameterized by tensor mass–inertia, Jacobians, and constraint forces (Eslami et al., 15 Sep 2025).
• MPS world models: generative factorization across observation–action sequences, supporting active inference (Wauthier et al., 2022).

3. Training and Inference Algorithms

Tensor-based agents employ update rules and optimization schemes matched to the tensor structure, often reducing the computational complexity of high-dimensional search and training.

Approach	Update/Inference Algorithm	Parameter Sharing
FQL	Agent-wise coordinate maximization, SGD over shared subnets	All agents in group
Tesseract	Bellman update + CP projection (model-based); TD targets + CP-factor critic (model-free)	Global Q-tensor
Model-based CP	Alternating least squares, power iteration, CP tensor completion	Global tensor
MPS Agents	DMRG-style sweeps for NLL, truncation via SVD	Bond dim adaptive
Tensor-Predictive	Tensor regression, mode-specific factor extraction	Shared transition
Tensor-based control	Closed-form port-LHS, data-driven port-RHS equivariant NN	Control/loss split

For FQL, instead of global joint maximization (O($(N+1)$2)), tractable TD-targets are computed by sequential argmax for each agent, freezing others’ actions at recent values:

$(N+1)$3

followed by standard experience replay and SGD (Zhou et al., 2018).

For Tesseract/CP RL, one projects the updated Q-tensor after the Bellman step or critic regression back onto the low-rank manifold using truncated CP-decomposition, and factor gradients/parameterization are exploited in the critic/actor updates (Mahajan et al., 2021, Mahajan et al., 2021).

MPS agents use left-right DMRG-inspired sweeps to minimize negative log-likelihood of the generative model, with sweeping SVD-based rank truncation maintaining efficient state space representation (Wauthier et al., 2022, Howard, 2024).

4. Scalability, Tractability, and Invariance Gains

Tensor-based agents address the curse of dimensionality both in parameter count and computational cost.

• Model count: FQL replaces O($(N+1)$4) separate networks with O($(N+1)$5) shared sub-networks (often $(N+1)$6) (Zhou et al., 2018).
• Joint-action maximization: O($(N+1)$7) exhaustive search is reduced to O($(N+1)$8) decomposition or tractable greedy maximization (Zhou et al., 2018, Mahajan et al., 2021).
• Sample efficiency: The sample complexity of a CP-decomposed Q-tensor is O($(N+1)$9 poly$N$0) rather than O($N$1), where $N$2 is the number of agents (Mahajan et al., 2021, Mahajan et al., 2021).
• In robotic control, full coordinate invariance is achieved by retaining all dynamics and control constraints in tensor (rather than coordinate) form; controllers and learned corrections generalize across reference frames and scenarios (Eslami et al., 15 Sep 2025).

Empirical scaling is substantial: FQL solves $N$3 agent pure-cooperation games with a single GPU, outperforming both independent and mean-field Q-learning (Zhou et al., 2018).

5. Applications and Empirical Results

Tensor-based agent frameworks have been validated in diverse scenarios:

• Large-scale MARL: Pure cooperative (traffic) and competitive (battle) tasks, with FQL outperforming IQL, MF-Q, and MAAC baselines, especially at $N$4 (Zhou et al., 2018). In SMAC and tensor game benchmarks, Tesseract is the only approach to solve hard coordination tasks as $N$5 increases (Mahajan et al., 2021, Mahajan et al., 2021).
• Model-based RL: Synthetic MDP experiments show that CP-tensor model-based RL converges rapidly if the underlying dynamics are truly low-rank, generalizing effectively to never-visited state-action pairs (Vaart et al., 2021).
• Multi-agent PSR: CP/Tucker-tensor-based predictive state models achieve lower one-step error than TPSR/CPSR baselines as agent count scales, with competitive runtime and memory footprint (Chen et al., 2020).
• Robotic control: Tensor-invariant DAC agents yield coordinate-free, interpretable controllers for multibody systems, supporting robust stabilization and fast adaptation to new physical conditions (Eslami et al., 15 Sep 2025).
• High-DoF motion: Transition Motion Tensors (TMT) enable seamless and robust switching between template policies in locomotion and manipulation, outperforming gating and value-based switching in terms of transition success and computational cost (Soeseno et al., 2021).
• Batched RL: TensorFlow Agents leverages batch tensors to vectorize environments and agent computations, yielding near-linear scaling up to core count in physical simulation throughput (Hafner et al., 2017).
• Tensor-network generative models: MPS-based active inference agents support latent state learning and planning in discrete environments, integrating naturally with variational free-energy principles (Wauthier et al., 2022).

6. Limitations, Open Challenges, and Theoretical Guarantees

Despite their advantages, tensor-based agent designs carry inherent challenges:

• Rank selection: Optimal CP/Tucker rank selection is NP-hard; practitioners rely on cross-validation or domain intuition (Mahajan et al., 2021, Mahajan et al., 2021).
• Expressivity vs tractability: Under-parameterized (low-rank) models may underfit, while excessive rank causes increased computation and potential overfitting (Vaart et al., 2021).
• Learning dynamics: ALS and power iteration may get stuck in suboptimal local minima; tensor completion can be numerically unstable when data is sparse (Vaart et al., 2021).
• Theoretical guarantees: For systems where the transition and reward tensors are low-rank, approximation and evaluation errors can be made arbitrarily small, with sample complexity showing an exponential ($N$6) gain relative to full-table methods (Mahajan et al., 2021).
• Transfer and invariance: Tensor-invariant control architectures are maximally data-efficient only if the coordination and physical properties are truly geometric; coordinate-dependent methods are strictly less general (Eslami et al., 15 Sep 2025).
• Scalability in nonlinear or nonstationary regimes may require hybrid architectures, such as integrating neural networks with invariant tensor kernels (Eslami et al., 15 Sep 2025, Xu et al., 2020).

7. Summary Table of Representative Tensor-Based Agent Methods

Method	Factorization	Target Domain	Unique Features	Reference
FQL	Independent + Pairwise	Large-scale MARL	O($N$7) net sharing	(Zhou et al., 2018)
Tesseract	CP-rank	MARL, factored RL	Bellman CP-projection	(Mahajan et al., 2021)
Model-based CP	CP, ALS	Model-based MARL	Generalization to unvisited (s,a)	(Vaart et al., 2021)
Tensor PSR	CP, Tucker	Multi-agent PSR	Prediction + transition regression	(Chen et al., 2020)
TensorNet RL	TT (MPS)	MARL, MDP	DMRG policy updates	(Howard, 2024)
Tensor-Inv Ctrl	Unreduced tensor mechanics	Robotic control	Port splitting, equivariant learning	(Eslami et al., 15 Sep 2025)
TMT	4-mode empirical tensor	Motion planning	Transition quality as tensor lookup	(Soeseno et al., 2021)

These frameworks, unified by the tensor formalism, represent a generalizable and scalable paradigm for agent-based modeling, learning, and control in high-dimensional, structured, and physically rich environments.