Vector Symbolic Policy Gradient

Abstract: We answer this question with Vector-Symbolic Policy Gradient (VSPG), a discrete-action actor that represents each action by a unit-norm hypervector and scores it by similarity to the encoded state. Under the standard softmax policy-gradient surrogate, we prove that its update is exactly advantage-weighted hypervector bundling followed by normalization, and therefore supports standard advantage estimators. We further show that each trained action hypervector is a fixed-size compressed kernel memory, storing an advantage-weighted kernel expansion over visited states and transferring evidence according to the encoder-induced similarity. This provides a concrete mechanism that can support sample-efficient learning without increasing inference-time memory. Finally, for bipolar action memories, we prove that greedy action selection is stable under random bit flips, with failure probability decaying exponentially in the hypervector dimension. VSPG thus connects VSA action memories, log-linear policy gradients, and kernel policy search while providing a quantitative robustness guarantee.

• The paper introduces VSPG, an exact softmax policy-gradient method that uses advantage-weighted hypervector bundling and row normalization without backpropagation or optimizer state.
• VSPG interprets each action memory as a fixed-size, advantage-weighted kernel expansion over visited states, with performance generally improving as hypervector dimension increases and saturating beyond D=1,000 on CartPole.
• Experiments show competitive or faster learning across control and grid-world tasks, including a Warm-Humid SustainGym reward of −7.11 ± 0.81, while bipolar memories receive exponential robustness guarantees against random bit flips.

VSPG (Vector-Symbolic Policy Gradient) is a discrete-action actor for reinforcement learning that parameterizes a categorical softmax policy entirely with Vector Symbolic Architecture (VSA) primitives: each action is represented by a unit-norm hypervector, actions are scored by inner-product similarity to a fixed encoded state, and the policy-gradient update reduces exactly to advantage-weighted bundling followed by row-wise normalization (2608.18404). The paper's central contribution is to show that this update is not merely analogous to policy gradient — it is the softmax policy-gradient step, computed in closed form with no backpropagation and no optimizer state — and that the resulting action memories are fixed-size compressed kernel expansions over visited states with provable robustness to random bit flips.

Method

The actor maintains a matrix $\mathbf{C}\in\mathbb{R}^{|\mathcal{A}|\times D}$ of unit-norm action hypervectors over a fixed, never-trained encoder $\boldsymbol{\phi}$. The policy is $$\pi(a\mid x)=\operatorname{softmax}_a(\tau\,\mathbf{C}\boldsymbol{\phi}(x))$$, so each logit is a bounded scaled cosine in $[-\tau,\tau]$, where the temperature $\tau$ jointly controls policy sharpness and effective step size. Given advantages $A_t$, the update is

$$\mathbf{C}\leftarrow \operatorname{row\text{-}norm}\bigl(\mathbf{C}+\eta\,\boldsymbol{\Lambda}^{\top}\mathbf{S}\bigr), \qquad \Lambda_{t,a}=A_t\,\tau\bigl(\mathbf{1}[a=a_t]-\pi(a\mid x_t)\bigr),$$

which is one matrix multiplication plus normalization. Three fixed random encoders are considered: random Fourier features (RFF), FHRR (complex phase encoding), and a Basis map (identity or sign-thresholded random projection). After normalization, encoder inner products approximate a cosine kernel $\kappa(x,y)$, so dimensionality $D$ governs kernel-approximation fidelity while the encoder family determines the induced similarity structure.

Theoretical results

Three formal results anchor the method. Proposition 1 establishes that $$\nabla_{\mathbf{C}}\hat J=\boldsymbol{\Lambda}^{\top}\mathbf{S}$$ exactly, so the bundling rule is a projected gradient-ascent step on the sampled surrogate; a corollary shows the normalized step is first-order equivalent to Riemannian gradient ascent on the product of unit spheres whenever $\boldsymbol{\phi}$0. This places VSPG squarely within log-linear softmax policy-gradient theory (2608.18404).

Proposition 2 gives an exact expansion: after any number of updates, each trained action hypervector equals a positively rescaled initialization plus a superposition of advantage-weighted encodings of all visited inputs, so the deployed logit decomposes as a sum of kernel evaluations $\boldsymbol{\phi}$1 against experience. The deployed policy is therefore approximately a softmax over advantage-weighted kernel scores against stored transitions — a compressed RKHS-style expansion held in $\boldsymbol{\phi}$2 fixed coordinates and never enumerated at inference. Notably, the sign of each coefficient preserves the sign of the advantage-weighted softmax score at that visit.

Proposition 3 proves bit-flip stability for bipolar action memories: under independent coordinate flips with probability $\boldsymbol{\phi}$3, greedy action selection fails with probability at most $\boldsymbol{\phi}$4, where $\boldsymbol{\phi}$5 is the clean similarity margin. The proof exploits the fact that flips primarily scale logits by $\boldsymbol{\phi}$6 while residual perturbations concentrate via Hoeffding's inequality. The authors are careful to note this bound applies only to genuinely bipolar memories; multi-bit quantized real-valued memories are evaluated empirically rather than claimed under the same guarantee.

Experiments

Evaluations span classic control (CartPole-v1, LunarLander-v2, Acrobot-v1), MiniGrid (Empty-5x5, DoorKey-5x5/8x8), and multi-agent SustainGym building control, against DNN actors, raw-linear actors, and QHD as the closest prior VSA baseline. VSPG learns substantially faster than DNN and linear actors on CartPole and Acrobot, remains competitive across MiniGrid including DoorKey-8x8, and matches baselines on LunarLander. QHD degrades sharply with task complexity, collapsing on DoorKey-5x5 and learning almost nothing on DoorKey-8x8. On SustainGym after 500 episodes, FHRR-VSPG achieves $\boldsymbol{\phi}$7 average reward per step on Warm-Humid versus $\boldsymbol{\phi}$8 for the DNN and $\boldsymbol{\phi}$9 for Raw-Linear, with RFF-VSPG strongest on Hot-Dry ($$\pi(a\mid x)=\operatorname{softmax}_a(\tau\,\mathbf{C}\boldsymbol{\phi}(x))$$0).

Under post-training quantization (1–8-bit signed affine) followed by independent bit flips, VSPG action memories degrade more gracefully than DNN and Raw-Linear actors, supporting the paper's positioning of VSPG as an actor for unreliable edge systems. Ablations confirm the two representation-level predictions of Proposition 2: performance improves with $$\pi(a\mid x)=\operatorname{softmax}_a(\tau\,\mathbf{C}\boldsymbol{\phi}(x))$$1 (saturating beyond $$\pi(a\mid x)=\operatorname{softmax}_a(\tau\,\mathbf{C}\boldsymbol{\phi}(x))$$2 on CartPole), and qualitative neighbor retrieval shows that successful encoder configurations (Basis, FHRR on DoorKey-8x8) produce coherent kernel neighborhoods while the failing RFF configuration does not.

Limitations and open questions

The paper concedes several points plainly. No encoder is uniformly effective: RFF-VSPG fails on DoorKey-8x8 while succeeding on SustainGym, and Basis-VSPG performs poorly with high variance ($$\pi(a\mid x)=\operatorname{softmax}_a(\tau\,\mathbf{C}\boldsymbol{\phi}(x))$$3 on Warm-Humid), indicating that VSPG inherits its inductive bias from the encoder-induced similarity without a principled selection procedure. Supplementary analysis further reveals a practical sensitivity: with zero initialization, immediate row normalization can amplify weak early evidence to full-scale action directions, causing normalized variants to stall where unnormalized ones continue learning — so neither nonzero initialization nor normalization is required by the theory, but their interaction matters in practice. The bit-flip bound covers only bipolar memories under direct sign flips, not the quantized real-valued memories used in the main robustness experiments. Finally, the kernel-expansion interpretation assumes a fixed encoder whose similarity transfers advantage evidence usefully; how to learn or adapt this similarity remains open, as does extension beyond the small-to-medium discrete-action benchmarks evaluated here.

Conclusion

VSPG demonstrates that a categorical actor built purely from VSA primitives admits an exact policy-gradient identity, a fixed-size kernel-memory interpretation, and exponential-in-$$\pi(a\mid x)=\operatorname{softmax}_a(\tau\,\mathbf{C}\boldsymbol{\phi}(x))$$4 robustness guarantees for bipolar storage. Empirically it offers competitive returns, favorable sample efficiency relative to matched neural and linear actors, and markedly graceful degradation under quantization and memory corruption, at the cost of encoder-dependent generalization behavior that the theory identifies but does not resolve.