---
title: 'ReLATE: Adaptive Tensor Encoding in RL'
url: https://www.emergentmind.com/topics/reinforcement-learned-adaptive-tensor-encoding-relate
type: topic
---

# ReLATE: Adaptive Tensor Encoding in RL

Searching arXiv for the cited papers to ground the article.
Reinforcement-Learned Adaptive Tensor Encoding (ReLATE) denotes reinforcement-driven procedures that learn tensor-structured representations from interaction signals rather than prescribing them a priori. In the recent arXiv literature, the term is used in two closely related but non-identical senses. In reinforcement learning proper, ReLATE refers to policy-gradient or value-function methods in which policy parameters or value functions are encoded as low-rank PARAFAC/CP tensors and updated from returns, advantages, or temporal-difference errors [2501.04879; 2201.09736]. In sparse tensor decomposition, ReLATE is the explicit name of a framework that uses deep reinforcement learning to discover an efficient sparse tensor encoding for tensor-decomposition kernels such as MTTKRP, with reward defined by runtime speedup relative to ALTO [2509.00280]. Across these settings, the common idea is that multilinear structure is not fixed merely for compression; it is adapted by reinforcement signals to exploit locality, redundancy, and workload structure.

## 1. Terminological scope and conceptual core

The literature associates ReLATE with adaptive tensor structure learned from a reward-bearing process rather than from supervised labels. In the policy-gradient formulation, the encoded object is a probabilistic policy $\pi(a \mid s;\theta)$ whose parameters are collected into a tensor $X$ and constrained to have low PARAFAC/CP rank [2501.04879]. In the value-based formulation, the encoded object is a matrix or tensor approximation of $Q^\pi(s,a)$ or $Q^*(s,a)$, updated online and model-free by TD-style errors [2201.09736]. In the tensor-decomposition systems formulation, the encoded object is a linearized sparse representation of an $N$-way tensor, learned by an autonomous agent interacting with a tensor-decomposition environment [2509.00280].

| Setting | Encoded object | Reinforcement signal |
|---|---|---|
| Policy-based RL | Policy parameter tensor $X$ | Returns or advantages |
| Value-based RL | Value-function matrix/tensor | TD errors |
| Sparse tensor decomposition | Linearized sparse encoding $p$ | Runtime speedup vs ALTO |

A frequent misconception is to treat ReLATE as a single standardized algorithm. The available papers do not support that interpretation. "Tensor and Matrix Low-Rank Value-Function Approximation in Reinforcement Learning" explicitly states that the paper does not mention ReLATE by name, although its tensor TD updates align closely with the notion of adaptive tensor encoding learned from reinforcement signals [2201.09736]. By contrast, "ReLATE: Learning Efficient Sparse Encoding for High-Performance Tensor Decomposition" uses ReLATE as the formal name of a learning-augmented systems framework [2509.00280]. This suggests that ReLATE is best understood as a family resemblance centered on reinforcement-driven adaptation of tensor structure.

## 2. Tensor parameterization in reinforcement learning

In the policy-gradient instantiation, the state space is factorized as $S = S_1 \times \cdots \times S_D$, and policy parameters are placed in a tensor whose modes align with state factors and, for discrete actions, an action mode. For discrete action spaces $A=\{1,\ldots,C\}$, the logits are entries of a $(D+1)$-mode tensor $X \in \mathbb{R}^{N_1 \times \cdots \times N_D \times C}$. For continuous actions, the state-dependent mean $\mu_s$ is stored in a $D$-mode tensor, with one additional mode per action dimension for multi-dimensional actions [2501.04879].

The central multilinear ansatz is the CP/PARAFAC model
$$
X \approx \sum_{r=1}^{R} u_r^{(1)} \otimes u_r^{(2)} \otimes \cdots \otimes u_r^{(m)},
$$
with factor matrices $U^{(k)} = [u_1^{(k)} \ \cdots \ u_R^{(k)}] \in \mathbb{R}^{n_k \times R}$. Entrywise,
$$
X[i] = \sum_{r=1}^{R} \prod_{k=1}^{m} U^{(k)}[i_k,r].
$$
The low-rank constraint reduces degrees of freedom from $\prod_k n_k$ to $R\sum_k n_k$, which is the formal source of the parameter-efficiency claim in the policy paper [2501.04879].

For discrete control, the softmax logits at state $s$ with mode indices $i^s=(i_1^s,\ldots,i_D^s)$ are
$$
z_a(s) = \sum_{r=1}^{R} U^{(1)}[i_1^s,r] \cdots U^{(D)}[i_D^s,r] \cdot U^{(D+1)}[a,r],
$$
and
$$
\pi(a \mid s;X) = \frac{\exp(\beta z_a(s))}{\sum_{b=1}^{C} \exp(\beta z_b(s))}, \qquad \beta \ge 0.
$$
For scalar continuous actions,
$$
\mu_s = \sum_{r=1}^{R} U^{(1)}[i_1^s,r] \cdots U^{(D)}[i_D^s,r], \qquad \pi(a \mid s;X)=\mathcal{N}(a;\mu_s,\sigma^2).
$$
The same paper also gives an equivalent feature-based formulation,
$$
y(s)=\langle X, s^{(1)} \otimes \cdots \otimes s^{(m)} \rangle = X \times_1 s^{(1)} \times_2 \cdots \times_m s^{(m)},
$$
and a matricized/Khatri-Rao expression for efficient computation [2501.04879].

The value-function line uses the same multilinear principle on $Q$. With $D=D_S+D_A$ modes and cardinalities $C_d$, the Q-tensor has shape $C_1 \times \cdots \times C_D$, and
$$
\hat Q(i_1,\ldots,i_D) = \sum_{r=1}^{K} \prod_{d=1}^{D} A^{(d)}_{i_d,r}.
$$
The matrix special case is $\hat Q \approx UV^T$ with $\hat Q_{i,j} \approx \sum_{r=1}^{R} U_{i,r}V_{j,r}$ [2201.09736]. In both formulations, the factor rows act as learned low-dimensional embeddings for per-mode indices.

## 3. Learning mechanisms, gradients, and adaptive rank

The policy-gradient objective is
$$
J(\theta)=\mathbb{E}_{\tau^T \sim g(\theta)}[G_T(\tau^T)],
$$
with gradient
$$
\nabla_\theta J(\theta)=\mathbb{E}\!\left[\sum_{t=1}^{T} \nabla_\theta \log \pi(a_t \mid s_t;\theta)\cdot A_t \right].
$$
For CP-parameterized policies, the chain rule yields sparse factor updates. If $\Theta=\{U^{(k)}\}$ denotes the CP factors, then
$$
\frac{\partial X[i^s]}{\partial U^{(d)}[i,k]} = \mathbf{1}_{i=i_d^s}\cdot \prod_{j \ne d} U^{(j)}[i_j^s,k],
$$
so only $R \cdot D$ gradient entries are nonzero per sample $(s,a)$ [2501.04879].

The distribution-dependent factor depends on the policy class. For Gaussian policies with fixed $\sigma$,
$$
\frac{\partial \log \pi(a \mid s;\Theta)}{\partial X[i^s]} = \frac{a-\mu_s}{\sigma^2},
$$
hence
$$
\frac{\partial \log \pi(a \mid s;\Theta)}{\partial U^{(d)}[i,k]}
=
\frac{a-\mu_s}{\sigma^2}
\left(\prod_{j \ne d} U^{(j)}[i_j^s,k]\right)\mathbf{1}_{i=i_d^s}.
$$
For softmax policies with temperature $\beta$,
$$
\frac{\partial \log \pi(a \mid s;\Theta)}{\partial z_b(s)}=\mathbf{1}_{a=b}-\operatorname{softmax}_b(z(s)),
$$
leading to the corresponding sum over action fibers [2501.04879].

The paper instantiates these scores in REINFORCE, actor-critic, TRPO, and PPO variants. The stochastic REINFORCE update is
$$
\Theta \leftarrow \Theta + \eta \cdot \frac{1}{U}\sum_{u=1}^{U}\sum_{t=1}^{T} G_t^u \nabla_\Theta \log \pi(A_t^u \mid S_t^u;\Theta),
$$
while the actor-critic version replaces $G_t^u$ with $A_\omega(S_t^u)$ and learns a critic by minimizing
$$
L(\omega)=\frac{1}{2}\sum (G_t - V_\omega(S_t))^2.
$$
The trust-region formulations solve the usual score-based subproblem in TRPO and the clipped surrogate in PPO, but with tensor-factor parameters [2501.04879].

ReLATE’s adaptive-rank extension modifies the fixed-rank base method by rank growth and pruning. Growth is triggered when training stalls, for example when the moving average of return or advantage improvement is below $\epsilon_g$ over a window $W$; the method then adds a CP component using small random factors or CP-ALS fitted to current gradient residuals. Pruning uses a saliency score
$$
s_r = \sum_k \lVert u_r^{(k)} \rVert_2
$$
or gradient-aligned importance, removing components with $s_r < \tau_p$. A held-out batch governs validation-based acceptance, and group-sparsity
$$
\lambda \sum_r \left(\sum_k \lVert u_r^{(k)} \rVert_2\right)
$$
provides implicit rank selection [2501.04879].

In the value-function formulation, the updates are TD-like rather than policy-gradient-based. For the matrix case,
$$
\delta_t = r_t + \gamma \max_a (U_{s_{t+1},:}\cdot V_{a,:}) - (U_{s_t,:}\cdot V_{a_t,:}),
$$
followed by row updates to $U_{s_t,:}$ and $V_{a_t,:}$. For the tensor case,
$$
\delta_t = r_t + \gamma \max_{a'} \hat Q([s_{t+1};a']) - \hat Q([s_t;a_t]),
$$
and each active factor row is updated by the product of the other active factor rows, optionally with $\ell_2$ regularization [2201.09736]. The paper emphasizes that these procedures are model-free, online, and do not require a transition model.

## 4. Theoretical properties and computational characteristics

For projected tensor low-rank policy gradient, the theory is stated under bounded rewards, Lipschitz conditions on policy scores and trajectory distributions, and bounded score variance. With
$$
L_G := D(T^2 L_0 R + L_1)
$$
and any $\bar L > L_G$, the projected update
$$
\Theta^{h+1}=P_C(\Theta^h + \eta^h \hat g(\Theta^h))
$$
satisfies
$$
\frac{1}{H}\sum_{h=1}^{H}\mathbb{E}\big[\lVert \nabla_\Theta J^{1/\bar L}(\Theta^h)\rVert^2\big]
\le
\frac{\bar L}{\bar L-L_G}
\cdot
\frac{
J^{1/\bar L}(\Theta^1)-J^{1/\bar L,*} + \frac{\bar L G^2}{2}\sum_{h=1}^{H}(\eta^h)^2
}{
\sum_{h=1}^{H}\eta^h
},
$$
where $J^\lambda(\Theta)$ is the Moreau envelope of $-J$. Choosing $\eta^h=1/\sqrt{H}$ yields an $O(1/\sqrt{H})$ stationarity rate [2501.04879]. The paper states that dependence on rank enters only through constants, whereas the computational and sample advantages arise from the reduced parameterization.

The value-function paper provides a different theoretical lens. For synchronous low-rank Bellman iteration, if the truncation error of the rank-$K$ operator is bounded by $B$ in the matrix case, then
$$
\limsup_{t\to\infty}\lVert \hat q_t^\pi - q^\pi \rVert_\infty \le \frac{B}{1-\gamma},
$$
and the same form holds for truncated PARAFAC in the tensor case with bound $B'/(1-\gamma)$ [2201.09736]. The interpretation given there is that the low-rank truncation acts as a contractive perturbation of Bellman’s operator.

The computational savings are expressed explicitly in parameter counts. For policy tensors with fixed rank $R$, degrees of freedom scale as $R\sum_k n_k$ instead of $\prod_k n_k$ [2501.04879]. For value approximation, matrix low rank uses $(C_S + C_A)K$ parameters rather than $C_S \times C_A$, while CP tensor low rank uses $(\sum_{d=1}^{D} C_d)K$ rather than $\prod_{d=1}^{D} C_d$ [2201.09736]. The policy paper further states that computing $X[i^s]$ costs $O(DR)$, and whole fibers across actions cost $O(R(D+1))$ per action when using Khatri-Rao and matricization structure [2501.04879].

These guarantees do not remove the usual non-convexity caveats. Both RL-oriented papers note ill-conditioning or local minima as practical concerns, and both recommend regularization, projection, or normalized/scaled gradients for stability [2501.04879; 2201.09736].

## 5. ReLATE for sparse tensor decomposition and systems optimization

In "ReLATE: Learning Efficient Sparse Encoding for High-Performance Tensor Decomposition," the problem is no longer RL control but the construction of a sparse tensor representation that accelerates tensor-decomposition kernels, especially MTTKRP [2509.00280]. The background operator is
$$
M^{(n)} = X_{(n)}\left(\bigodot_{k\ne n} A^{(k)}\right),
$$
and the paper emphasizes that sparse encodings directly affect traversal order, fiber reuse, load balance, synchronization costs, and memory-access behavior. The performance model is given through the Roofline relation
$$
P = \min\big(P_{\text{peak}},\; BW_{\text{peak}}\cdot OI\big).
$$

ReLATE formulates encoding construction as an MDP $(S,A,T,R,\gamma)$. The state $s_t$ is an $N \times \ell(p)$ encoding matrix with one-hot columns, where
$$
\ell(p)=\sum_{n=1}^{N}\ell^{(n)}, \qquad \ell^{(n)}=\lceil \log_2 I_n \rceil.
$$
At each step, the action chooses which mode supplies the next lower-order bit to the linearized position. The agent must respect per-mode bit budgets, so the valid-action mask is
$$
M_n(t)=
\begin{cases}
1, & \text{if } c_n(t) < \ell^{(n)},\\
0, & \text{otherwise}.
\end{cases}
$$
This guarantees that every terminal encoding is functionally correct [2509.00280].

The terminal reward is runtime speedup relative to ALTO:
$$
r_{\text{term}}=\text{speedup} = \frac{\text{time(ALTO)}}{\text{time(ReLATE encoding)}},
$$
with shaping
$$
r_t = \frac{1}{\ell(p)} \log(r_{\text{term}}), \qquad t=0,\ldots,\ell(p)-1.
$$
The objective is
$$
\max_\pi \; \mathbb{E}\!\left[\sum_{t=0}^{\ell(p)-1} \gamma^t r_t\right].
$$
The number of valid interleavings is reduced to the multinomial
$$
\binom{\ell(p)}{\ell^{(1)},\ldots,\ell^{(N)}}=\frac{\ell(p)!}{\prod_{n=1}^{N}\ell^{(n)}!},
$$
because internal bit-order preservation is enforced within each mode [2509.00280].

The learning algorithm is a hybrid model-free/model-based scheme. Its model-free core is Double DQN with prioritized replay, using target
$$
y = r + \gamma Q\!\left(s',\arg\max_{a'}Q(s',a';\theta);\theta^-\right)
$$
and loss
$$
\mathcal{L}(\theta)=\big(y-Q(s,a;\theta)\big)^2.
$$
The network architecture consists of two convolutional layers with $3\times 3$ filters and 16 and 32 feature maps, followed by two fully connected layers that emit $N$ action values. A lightweight reward model predicts terminal rewards and enables imagined actions once its accuracy exceeds $90\%$ [2509.00280].

A defining systems contribution is the combination of rule-driven action masking, dynamics-informed action filtering, and a reward cache. These mechanisms ensure correctness, avoid evaluating invalid encodings, and bound execution time during training. The framework operates in an offline, decoupled client-server setup: the client learns the policy, and the server evaluates kernel runtime [2509.00280].

## 6. Empirical behavior, limitations, and broader significance

The RL policy paper reports experiments on MountainCar and Pendulum, including both continuous and discrete actions, as well as a multi-dimensional wireless communications task described as opportunistic multiple access. It states that tensor low-rank policies consistently converge faster and require fewer parameters than neural-network baselines in both continuous and discrete tasks, while achieving similar returns; in a wireless setup with 6D state and 2D continuous action, PTLRPO reached steady state significantly faster than NN-PPO, with fewer parameters [2501.04879]. It also reports that logit and mean tensors discretized from neural-network policies exhibit rapidly decaying normalized Frobenius error as CP rank increases, which indicates empirical low-rankness.

The value-function paper presents broad evidence of low-rank structure in Q-functions. Singular-value analyses on FrozenLake, Racetrack, Jack’s Car Rental, Taxi-v3, Pendulum, CartPole, MountainCar, and Acrobot show that only a few singular values dominate, and CP approximations of Q-tensors display rapidly decreasing normalized Frobenius error with rank [2201.09736]. Reported policy-quality results include low normalized cumulative reward error for truncated-SVD-induced policies, including 0.00% on FrozenLake, 0.84% on Jack’s Car Rental, and 0.42% on Pendulum. The same paper reports that tensor low-rank value approximation often attains comparable returns with far fewer parameters than tabular Q-learning, matrix low rank, or DQN, and that TLR variants converge faster and require fewer samples than DQN on a high-dimensional highway task [2201.09736].

The sparse tensor-decomposition paper evaluates ReLATE on large real-world tensors from FROSTT, including DARPA, FB-M, FLICKR-3D/4D, DELI-3D/4D, NELL-1, AMAZON, PATENTS, and REDDIT, on dual-socket Intel Xeon Platinum 8592+ hardware. It reports speedups versus the best expert-designed format of up to $2\times$, with geometric means of $1.4\times$ on original tensors and $1.46\times$ on randomly shuffled tensors; versus SPLATT, speedups reach up to $16.9\times$, with geometric means of $3.28$-$3.69\times$ [2509.00280]. Storage is stated to be equal to ALTO, while SPLATT can require up to $4.2\times$ more memory due to per-mode copies. Example memory-system effects include a 43% reduction in main-memory volume on DELI-3D and a 41% reduction on REDDIT, alongside improved cache behavior [2509.00280].

Several limitations recur across the literature. In RL, fixed CP rank can limit expressivity relative to deep neural networks; extremely nonlinear mappings may require higher rank or richer tensor formats such as Tensor Train or Hierarchical Tucker [2501.04879]. CP models can be ill-conditioned, and discretization choices can strongly affect value-based tensor methods [2201.09736]. In sparse tensor decomposition, extremely small tensors show limited benefit, rapidly changing data distributions can undermine an offline-trained encoding, and hardware portability may require retraining or re-optimization for different memory hierarchies [2509.00280].

Taken together, these works situate ReLATE as a technically specific strategy for exploiting multilinear structure under reinforcement signals. In RL, the adaptive object is the policy or value tensor; in tensor-decomposition systems, it is the sparse encoding itself. The common principle is not merely low rank or sparsity, but the use of reward-guided adaptation to discover tensor representations that better match the intrinsic structure of states, actions, or nonzero distributions [2501.04879; 2201.09736; 2509.00280].

Source: https://www.emergentmind.com/topics/reinforcement-learned-adaptive-tensor-encoding-relate