---
title: Tensor-Based Agents in Scalable RL
url: https://www.emergentmind.com/topics/tensor-based-agents
type: topic
---

# Tensor-Based Agents in Scalable RL

Tensor-based agents are learning architectures and control frameworks that leverage high-order tensor representations, tensor factorization, and tensor networks to model, plan, and control decision-making in high-dimensional, structured domains. By explicitly structuring policies, value functions, dynamical models, or agent populations as tensors—and then exploiting algebraic decompositions or tensor-invariant pipelines—these agents achieve scalability, parameter efficiency, and improved generalization in problems such as multi-agent reinforcement learning, robust robotic control, and large-scale predictive state modeling.

## 1. Foundations of Tensor-Based Agent Methodologies

Tensor-based agents make high-order structure a first-class modeling primitive. In multi-agent reinforcement learning (MARL), the joint action–value function $Q(s, a^1, ..., a^N)$, with $N$ agents, is naturally an $(N+1)$-way tensor (state plus $N$ agent action modes). However, the parameter count in such representations scales exponentially with the number of agents or system components, rendering naive approaches intractable for $N \gg 5$.

To address this, tensor-based agents use algebraic factorization, especially low-rank representations (e.g., CANDECOMP/PARAFAC (CP), Tucker, tensor train/MPS), to approximate high-dimensional tensors by compositional objects with O($N$) or O(poly($N$)) parameters. Examples include:

- Factorized Q-Learning (FQL), in which the multi-agent Q-tensor is decomposed into independent and pairwise interaction terms, dropping all higher-order terms and achieving a manageable parameterization [1809.03738].
- Tesseract (Tensorised Actors), which employs CP-decomposition for the Q-tensor, enabling both efficient Bellman updates and actor-critic training in deep MARL [2106.00136].
- Model-based RL with tensor decompositions, representing transition and reward as low-CP-rank tensors, and fitting these models via alternating least squares or power iteration [2110.14524].

The approach generalizes to tensor-network modeling for generative world models, e.g., matrix product state (MPS) active-inference agents [2208.08713], or tensor-based predictive state representation (PSR) in multi-agent PSRs [2005.13706].

## 2. Architectures and Decomposition Schemes

Tensor-based agent designs are structured around modular neural, algorithmic, or physical modeling blocks that directly realize tensor factorization or leverage tensor invariance:

### Example: FQL Composite Network

For $N$ homogeneous agents, FQL factorizes the Q-function as
$$
Q^i(s, a^i, a^{-i}) \approx Q(s^i, a^i; \theta) + \lambda V(s^i, a^i; \beta_1)^\top \overline U(s^{-i}, a^{-i}; \beta_2)
$$
with
$$
\overline U(s^{-i}, a^{-i}; \beta_2) = \frac{1}{N-1} \sum_{j \ne i} U(s^j, a^j; \beta_2)
$$
implemented with three sub-networks: Q-net, V-net, and U-net, each mapping from an agent’s perspective $(s^i, a^i)$ to a scalar or embedding; parameters are shared within groups, yielding an O($G$) parameter regime for $G$ groups [1809.03738].

### Example: Tesseract Q-Tensor CP Decomposition

For a joint Q-tensor $\hat Q^\pi(s)$ (order $n$), Tesseract approximates:
$$
Q^\pi(s, a^1, ..., a^n) \approx \sum_{r=1}^R u_r^{(1)}(a^1) \cdots u_r^{(n)}(a^n) u_r^{(n+1)}(s)
$$
allowing for $O(R n |A|)$ parameterization rather than $|A|^n$ [2106.00136]. The architecture supports both value-based and actor-critic RL (TD learning and direct policy gradients) and is PAC-sample efficient for low-rank domains.

### Example: Model-based Decomposition

Transition (order $n+2$) and reward (order $n+1$) tensors are represented as:
$$
T_{s,a_1,...,a_n,s'} \approx \sum_{l=1}^{r_T} w_l x_l(s) y_l^{(1)}(a_1) \cdots y_l^{(n)}(a_n) z_l(s')
$$
learned via alternating least squares from transitions; planning is then via standard policy/value iteration on the completed tensor model [2110.14524].

### Predictive State, Control, and MPS

- Multi-agent PSR: system-dynamics tensors, $\mathcal D_{i_1\cdots i_N,k}$, admitting CP/Tucker decompositions for learning prediction vectors and compressed system states [2005.13706].
- Tensor-invariant robotic control: coordinate-free, non-recursive DAEs parameterized by tensor mass–inertia, Jacobians, and constraint forces [2509.11688].
- MPS world models: generative factorization across observation–action sequences, supporting active inference [2208.08713].

## 3. Training and Inference Algorithms

Tensor-based agents employ update rules and optimization schemes matched to the tensor structure, often reducing the computational complexity of high-dimensional search and training.

| Approach             | Update/Inference Algorithm          | Parameter Sharing    |
|----------------------|-------------------------------------|---------------------|
| FQL                  | Agent-wise coordinate maximization, SGD over shared subnets | All agents in group |
| Tesseract            | Bellman update + CP projection (model-based); TD targets + CP-factor critic (model-free) | Global Q-tensor     |
| Model-based CP       | Alternating least squares, power iteration, CP tensor completion | Global tensor       |
| MPS Agents           | DMRG-style sweeps for NLL, truncation via SVD | Bond dim adaptive   |
| Tensor-Predictive    | Tensor regression, mode-specific factor extraction | Shared transition   |
| Tensor-based control | Closed-form port-LHS, data-driven port-RHS equivariant NN | Control/loss split  |

For FQL, instead of global joint maximization (O($\prod_j |A_j|$)), tractable TD-targets are computed by sequential argmax for each agent, freezing others’ actions at recent values:
$$
a_{t+1}^i = \arg\max_{a^i} Q(s_{t+1}^i, a^i, a_t^{-i}; \Theta)
$$
followed by standard experience replay and SGD [1809.03738].

For Tesseract/CP RL, one projects the updated Q-tensor after the Bellman step or critic regression back onto the low-rank manifold using truncated CP-decomposition, and factor gradients/parameterization are exploited in the critic/actor updates [2106.00136], [2110.14538].

MPS agents use left-right DMRG-inspired sweeps to minimize negative log-likelihood of the generative model, with sweeping SVD-based rank truncation maintaining efficient state space representation [2208.08713], [2401.03896].

## 4. Scalability, Tractability, and Invariance Gains

Tensor-based agents address the curse of dimensionality both in parameter count and computational cost.

- Model count: FQL replaces O($N$) separate networks with O($G$) shared sub-networks (often $G=1$) [1809.03738].
- Joint-action maximization: O($\prod_j |A_j|$) exhaustive search is reduced to O($\sum_j |A_j|$) decomposition or tractable greedy maximization [1809.03738], [2106.00136].
- Sample efficiency: The sample complexity of a CP-decomposed Q-tensor is O($|U|^{n/2}$ poly$(k, ...)$) rather than O($|U|^n$), where $n$ is the number of agents [2106.00136], [2110.14538].
- In robotic control, full coordinate invariance is achieved by retaining all dynamics and control constraints in tensor (rather than coordinate) form; controllers and learned corrections generalize across reference frames and scenarios [2509.11688].

Empirical scaling is substantial: FQL solves $N=500$ agent pure-cooperation games with a single GPU, outperforming both independent and mean-field Q-learning [1809.03738].

## 5. Applications and Empirical Results

Tensor-based agent frameworks have been validated in diverse scenarios:

- Large-scale MARL: Pure cooperative (traffic) and competitive (battle) tasks, with FQL outperforming IQL, MF-Q, and MAAC baselines, especially at $N\gg100$ [1809.03738]. In SMAC and tensor game benchmarks, Tesseract is the only approach to solve hard coordination tasks as $N$ increases [2106.00136], [2110.14538].
- Model-based RL: Synthetic MDP experiments show that CP-tensor model-based RL converges rapidly if the underlying dynamics are truly low-rank, generalizing effectively to never-visited state-action pairs [2110.14524].
- Multi-agent PSR: CP/Tucker-tensor-based predictive state models achieve lower one-step error than TPSR/CPSR baselines as agent count scales, with competitive runtime and memory footprint [2005.13706].
- Robotic control: Tensor-invariant DAC agents yield coordinate-free, interpretable controllers for multibody systems, supporting robust stabilization and fast adaptation to new physical conditions [2509.11688].
- High-DoF motion: Transition Motion Tensors (TMT) enable seamless and robust switching between template policies in locomotion and manipulation, outperforming gating and value-based switching in terms of transition success and computational cost [2111.15072].
- Batched RL: TensorFlow Agents leverages batch tensors to vectorize environments and agent computations, yielding near-linear scaling up to core count in physical simulation throughput [1709.02878].
- Tensor-network generative models: MPS-based active inference agents support latent state learning and planning in discrete environments, integrating naturally with variational free-energy principles [2208.08713].

## 6. Limitations, Open Challenges, and Theoretical Guarantees

Despite their advantages, tensor-based agent designs carry inherent challenges:

- Rank selection: Optimal CP/Tucker rank selection is NP-hard; practitioners rely on cross-validation or domain intuition [2106.00136], [2110.14538].
- Expressivity vs tractability: Under-parameterized (low-rank) models may underfit, while excessive rank causes increased computation and potential overfitting [2110.14524].
- Learning dynamics: ALS and power iteration may get stuck in suboptimal local minima; tensor completion can be numerically unstable when data is sparse [2110.14524].
- Theoretical guarantees: For systems where the transition and reward tensors are low-rank, approximation and evaluation errors can be made arbitrarily small, with sample complexity showing an exponential ($\sqrt{|U|^n}$) gain relative to full-table methods [2106.00136].
- Transfer and invariance: Tensor-invariant control architectures are maximally data-efficient only if the coordination and physical properties are truly geometric; coordinate-dependent methods are strictly less general [2509.11688].
- Scalability in nonlinear or nonstationary regimes may require hybrid architectures, such as integrating neural networks with invariant tensor kernels [2509.11688], [2010.13209].

## 7. Summary Table of Representative Tensor-Based Agent Methods

| Method         | Factorization      | Target Domain            | Unique Features        | Reference      |
|----------------|-------------------|-------------------------|-----------------------|---------------|
| FQL            | Independent + Pairwise | Large-scale MARL      | O($G$) net sharing    | [1809.03738]  |
| Tesseract      | CP-rank           | MARL, factored RL       | Bellman CP-projection | [2106.00136]  |
| Model-based CP | CP, ALS           | Model-based MARL        | Generalization to unvisited (s,a) | [2110.14524]|
| Tensor PSR     | CP, Tucker        | Multi-agent PSR         | Prediction + transition regression | [2005.13706] |
| TensorNet RL   | TT (MPS)          | MARL, MDP               | DMRG policy updates   | [2401.03896]  |
| Tensor-Inv Ctrl| Unreduced tensor mechanics | Robotic control | Port splitting, equivariant learning | [2509.11688] |
| TMT            | 4-mode empirical tensor | Motion planning | Transition quality as tensor lookup | [2111.15072] |

These frameworks, unified by the tensor formalism, represent a generalizable and scalable paradigm for agent-based modeling, learning, and control in high-dimensional, structured, and physically rich environments.

Source: https://www.emergentmind.com/topics/tensor-based-agents