---
title: Value Vectors Overview
url: https://www.emergentmind.com/topics/value-vectors
type: topic
---

# Value Vectors Overview

“Value vectors” is a polysemous technical term whose meaning depends strongly on the mathematical and architectural context. In the cited literature, it denotes: the fundamental vector-space-valued data units processed by vector-valued neural networks; the payload vectors propagated by transformer attention; the fixed columns of MLP down-projection matrices in transformer feedforward blocks; the $\alpha$-vectors that represent piecewise-linear-convex value functions in POMDPs; and vector-valued objects in signal processing, optimal transport, dual-quaternion analysis, and multivariate risk theory [2309.07716] [2606.02780] [2511.08825] [2509.17932] [1404.2492] [1611.09946] [2111.04491] [1708.01324]. Across these usages, the common thread is that a single mathematical object aggregates multiple correlated scalar quantities and is manipulated as one entity rather than as independent channels or coordinates.

## 1. Terminological scope and principal meanings

The term has no single universal definition. In vector-valued neural networks, a value vector is an element $x \in \mathbb{V}$ of a finite-dimensional real vector space equipped with a multiplication, written
$$
x = \sum_{i=1}^n x_i e_i,
$$
with coordinates in $\mathbb{R}^n$ under an isomorphism $\varphi : \mathbb{V} \to \mathbb{R}^n$ [2309.07716]. In this setting, a group of correlated channels is treated as one algebraic object rather than as separate scalars.

In transformer attention, value vectors are the vectors collected and averaged by the attention weights. For a standard block with residual stream $\mathbf{x}_i$, queries, keys, and values are
$$
\mathbf{Q} = \mathbf{x}_i \mathbf{W}_Q, \quad
\mathbf{K} = \mathbf{x}_i \mathbf{W}_K, \quad
\mathbf{V} = \mathbf{x}_i \mathbf{W}_V,
$$
and each head outputs
$$
\mathbf{O}^{(h)} = \mathrm{softmax}\!\left( \frac{\mathbf{Q}^{(h)} {\mathbf{K}^{(h)}}^{\!\top}}{\sqrt{d_h}} \right) \mathbf{V}^{(h)}.
$$
Here the values are the payload written back into the residual stream [2606.02780]. In approximate-attention work, these same value vectors are the objects whose aggregation should be approximated directly, rather than approximating only the attention weights [2103.09857].

In transformer MLPs, the term denotes something different. For GLU-style feedforward blocks,
$$
m = W_{\text{down}}\big(f(W_{\text{gate}} h) \odot (W_{\text{up}} h)\big),
$$
which can be rewritten as
$$
m = \sum_{i=1}^{d'} k_i v_i,
$$
with
$$
k_i = f(w_{\text{gate}, i}^\top h)\,(w_{\text{up}, i}^\top h), \quad
v_i = w_{\text{down}, i}.
$$
Thus each column of $W_{\text{down}}$ is a fixed value vector in an MLP key–value memory interpretation [2509.17932].

In POMDP planning, a value vector is an $\alpha$-vector, namely an element of $\mathbb{R}^{|S|}$ whose inner product with a belief $b$ gives the value of a conditional plan,
$$
V(b) = \max_{\alpha \in \Gamma} \alpha \cdot b.
$$
These vectors furnish the piecewise-linear-convex representation of finite-horizon POMDP value functions [2511.08825].

A concise comparison is useful.

| Area | “Value vector” denotes | Core role |
|---|---|---|
| Vector-valued NNs | Element of $\mathbb{V}$ | Jointly encodes correlated channels |
| Transformer attention | Row/value in $V$ | Payload aggregated by attention |
| Transformer MLPs | Column of $W_{\text{down}}$ | Fixed memory content weighted by key activations |
| POMDPs | $\alpha$-vector in $\mathbb{R}^{|S|}$ | Hyperplane defining value over beliefs |
| Signal/OT/risk settings | Vector-valued sample, density, or risk point | Multi-component state treated as one object |

A common misconception is that these usages are interchangeable. They are not. The attention value vector, the MLP value vector, and the POMDP $\alpha$-vector occupy different spaces, are produced by different mechanisms, and serve different analytical purposes [2606.02780] [2509.17932] [2511.08825].

## 2. Value vectors as algebraic feature units in vector-valued neural networks

In the framework of vector-valued neural networks, “value vectors” are the fundamental data units processed by V-nets. Instead of treating channels independently, the network treats tuples of channels as elements of a vector space $\mathbb{V}$ endowed with multiplication [2309.07716]. A vector-valued signal is therefore an array whose entries are vectors in $\mathbb{V}$, and a value vector encodes “a small group of correlated scalar channels” processed as a single algebraic object [2309.07716].

The algebra structure is specified by a multiplication table
$$
e_i e_j = \sum_{k=1}^n p_{ijk} e_k,
$$
so that for
$$
x = \sum_i x_i e_i,\quad y = \sum_j y_j e_j,
$$
their product becomes
$$
xy = \sum_{k=1}^n \mathcal{B}_k(x,y)\,e_k,
$$
with bilinear forms determined by the coefficients $p_{ijk}$ [2309.07716]. This means that inter-channel coupling is built into the multiplication itself rather than left entirely to unconstrained learned weights.

A vector-valued dense layer takes $\boldsymbol{x}=(x_1,\dots,x_N)\in\mathbb{V}^N$ and computes
$$
\boldsymbol{y} = \boldsymbol{\psi}(\boldsymbol{s} + \boldsymbol{b}),\quad
s_i = \sum_{j=1}^N w_{ij}x_j,
$$
with vector-valued weights and biases [2309.07716]. Convolution is defined analogously:
$$
(\mathbf{W} * x)(p,k) = \sum_{c=1}^C \sum_{q\in D} \mathbf{W}(q,c,k)\,x(p+S(q),c).
$$
The multiplication $\mathbf{W}(q,c,k)\,x(\cdot,c)$ mixes the channels inside each value vector in a structured way [2309.07716].

A central claim of the framework is parameter efficiency. A dense V-net layer with input dimension $N$ value vectors and output dimension $M$ value vectors uses $nMN$ weight parameters plus $nM$ biases, whereas the unconstrained real layer on the same effective dimensions uses $n^2MN$ weights plus $nM$ biases, yielding a factor-of-$n$ reduction in the weights when $n>1$ [2309.07716]. The paper further states that V-nets “usually have fewer parameters and often undergo more robust training than traditional neural networks,” while a universal approximation theorem is given for V-MLPs under a non-degeneracy assumption and split activations [2309.07716].

The same framework subsumes hypercomplex-valued neural networks. A hypercomplex-valued neural network is treated as a V-net in which $\mathbb{V}$ is a hypercomplex algebra with identity, so quaternions, complex numbers, Clifford algebras, and parametrized “hypercomplex” constructions become special cases [2309.07716]. The paper also shows that V-nets can be implemented in TensorFlow or PyTorch as real-valued networks by constraining real weight matrices to Kronecker-structured forms such as
$$
\mathcal{M}_L(\boldsymbol{W}) = \sum_{k=1}^n \boldsymbol{W}_k \otimes \boldsymbol{P}_{k:}^T.
$$
This makes the value-vector interpretation architectural rather than software-dependent [2309.07716].

## 3. Value vectors in transformers: attention payloads, approximation targets, and context-free banks

In attention mechanisms, value vectors are the vectors that are actually aggregated once the query–key similarities have been computed. For a single query, the output is
$$
o = \sum_{i=1}^L \alpha_i v_i, \quad
\alpha_i = \frac{\kappa(q,k_i)}{\sum_j \kappa(q,k_j)},
$$
so the attention weights determine how to mix the value vectors, but the output itself lives in value space [2103.09857].

This distinction is the basis of the paper on value-aware approximate attention. That work argues that many efficient-attention methods are “value-oblivious” because they optimize approximation of the attention distribution or kernel matrix while ignoring the contribution of $V$ to the final output [2103.09857]. It defines a value-aware objective over the set
$$
C_r = \left\{ \sum_{i=1}^{L} \beta_i v_i : \beta_i \ge 0,\ \sum_i \beta_i = 1,\ |\{i : \beta_i>0\}|\le r \right\},
$$
and seeks
$$
\tilde{o}^\star = \operatorname*{argmin}_{\tilde{o}\in C_r} \|o - \tilde{o}\|^2.
$$
A key theoretical point is that by Carathéodory’s theorem, if $r \ge d+1$ for $d$-dimensional values, then zero approximation error is possible because $o$ lies in the convex hull of the value vectors [2103.09857]. The paper further shows that ignoring values can be arbitrarily bad in the $r=1$ regime, and that the effect is especially pronounced for less skewed kernels such as low-degree polynomial or ELU kernels [2103.09857].

A separate line of work asks whether deep-layer value vectors need to depend on context at all. Standard attention computes $\mathbf{V}=\mathbf{x}_i\mathbf{W}_V$ from the current residual stream, but systematic ablations on 135M and 780M models show that in the last third of layers, replacing these context-dependent values with context-free token-specific vectors improves validation loss over standard attention and, at 780M, improves the average score across 21 benchmarks [2606.02780]. The resulting Bank of Values (BoV) stores a learned lookup table
$$
\mathbf{E}_v \in \mathbb{R}^{|\mathcal{V}| \times d},
$$
and for token id $i_p$ uses
$$
\mathbf{v}_p = \gamma_v\, \mathbf{E}_v[i_p].
$$
Keys and queries remain standard; only the value path is replaced in the last third of layers [2606.02780].

The reported quantitative findings are specific. On the 12-layer 135M model, baseline validation loss is 0.854 BPB, while the best context-free variants in the last four layers reach as low as 0.845 BPB [2606.02780]. On the 24-layer 780M model, BoV achieves validation loss $0.714 \pm 0.008$ versus 0.722 for baseline, and CORE benchmark score $0.272 \pm 0.012$ versus 0.260 [2606.02780]. The method also changes the compute–memory tradeoff: value computation becomes a gather rather than a dense matrix multiplication, and value caches for the BoV layers can be eliminated because values can be looked up from token ids [2606.02780].

A plausible implication is that the value path in deep transformer layers can function more as a token-identity-preserving channel than as a context-dependent feature synthesizer. The paper itself states the empirical result more cautiously: once deep layers have access to a good context-free value vector, “adding back the context-dependent component provides little additional benefit for aggregate benchmark performance” [2606.02780].

## 4. Value vectors in transformer MLPs and training-free truthfulness detection

Transformer MLPs introduce another, more parameter-centric notion of value vectors. Under the GLU decomposition
$$
m = \sum_{i=1}^{d'} k_i v_i,
$$
the vectors $v_i = w_{\text{down}, i}$ are fixed columns of the down-projection matrix, while the coefficients $k_i$ are input-dependent scalar key activations [2509.17932]. This realizes the MLP as a key–value memory in which the hidden-state update is a weighted sum of stored value vectors.

The truth-detection method TruthV exploits this structure. For a multiple-choice question with candidate answers $a_j$, the model is run on each sequence $[q; a_j]$, and for each MLP unit $c$ the final-token key activations $k_{i,j}^c$ are recorded [2509.17932]. A single value vector then becomes a weak predictor under either an argmax rule,
$$
a_i^c = \arg\max_j k_{i,j}^c,
$$
or an argmin rule,
$$
a_i^c = \arg\min_j k_{i,j}^c.
$$
Its accuracy on a labeled selection set is
$$
\text{Acc}_c = \frac{1}{|\mathcal{D}_{\text{sel}}|}
\sum_{i=1}^{|\mathcal{D}_{\text{sel}}|} [a_i^c = l_i],
$$
and the top-ranked value vectors are ensembled by majority vote [2509.17932].

The key empirical finding is that a small subset of MLP value vectors exhibits truthfulness-related statistical patterns. Most units are near random, but some are substantially predictive under argmax and others under argmin; these patterns are complementary, and combining them slightly improves performance [2509.17932]. The method is training-free in the sense that it does not fit a probe: it only ranks existing units by simple accuracy statistics computed from a small labeled set [2509.17932].

On the NoVo benchmark, TruthV substantially exceeds both the log-likelihood baseline and the earlier attention-based NoVo method. For Gemma-2-2B-it, average accuracy is 49.25% for log-likelihood, 61.67% for NoVo, 70.33% for TruthV(argmax), and 70.25% for TruthV(argmin). For Qwen3-4B, the corresponding averages are 40.98%, 63.26%, 70.28%, and 69.78% [2509.17932]. The top truthfulness-related value vectors are concentrated in middle and late layers, while early layers contribute virtually none [2509.17932].

The paper also emphasizes an important limitation of simplistic interpretations. The distributions of key activations for true and false candidates overlap substantially, so there is no global threshold separating truthful from untruthful answers. What matters is a question-conditioned relative pattern across candidates [2509.17932]. This suggests that these value vectors do not encode “truth” as an isolated scalar marker. Rather, they participate in comparative internal computations that are useful for selecting the correct answer.

## 5. Value vectors in planning, control, and risk: $\alpha$-vectors, dual-quaternion vectors, and vector-valued CVaR

In POMDP theory, the canonical value vectors are the $\alpha$-vectors used to represent a finite-horizon value function over beliefs:
$$
V_t^*(b) = \max_{\alpha \in \Gamma_t} \alpha \cdot b.
$$
Each $\alpha \in \mathbb{R}^{|S|}$ defines a hyperplane over belief space and corresponds to a conditional plan or a node in a finite-state controller [2511.08825]. Classical point-based value iteration performs Bellman backups directly on these value vectors, but their $|S|$-dimensional representation becomes intractable in large state spaces. Neural Value Iteration reinterprets an $\alpha$-vector as a function $S \to \mathbb{R}$ and replaces each tabular vector with a neural network $\alpha_{\text{NN}}$, maintaining the max-over-elements structure
$$
V(b) = \max_{\alpha_{\text{NN}} \in \mathcal{A}} \bigg(\sum_s b(s)\,\alpha_{\text{NN}}(s)\bigg)
$$
inside a finite network controller [2511.08825]. The paper reports that on RockSample$(20,20)$ with more than $4.19 \times 10^8$ states, NVI achieves $11.63 \pm 1.09$ return, whereas MCVI gets $2.36 \pm 2.23$ and SARSOP is inapplicable [2511.08825].

In dual-quaternion analysis, the relevant objects are dual quaternion vectors
$$
x = (x_1,\dots,x_n)^T,\quad x_i \in \mathbb{DQ},
$$
whose norms take values in the dual numbers $\mathbb{D}$ rather than in $\mathbb{R}$ [2111.04491]. The paper defines 1-, $\infty$-, and 2-norms on $\mathbb{DQ}^n$:
$$
\|x\|_1 = \sum_{i=1}^n |x_i|,\qquad
\|x\|_\infty = \max_i |x_i|,
$$
and a case-dependent $\|x\|_2$ that is well-defined both when some components are appreciable and when all are infinitesimal [2111.04491]. In this literature, the phrase “value vectors” refers to vectors whose norm is a dual-number-valued quantity capturing both standard and infinitesimal magnitude [2111.04491]. That usage is not architectural but geometric.

In multivariate risk theory, “value vectors” appear as elements of the set-valued multivariate Conditional Value-at-Risk. For a discrete random loss vector $X \in \mathbb{R}^d$, one first defines the multivariate VaR set $MVaR_p(X)$ via $p$-level efficient points, then constructs
$$
MCVaR_p(X,\eta) = \eta + \frac{1}{1-p}E[(X-\eta)_+],
$$
and finally retains only the non-dominated vectors:
$$
VMCVaR_p(X) = Min\{\,MCVaR_p(X,\eta):\ \eta\in MVaR_p(X)\}.
$$
Each element of $VMCVaR_p(X)$ is a risk vector whose coordinates are dimensionwise tail-risk quantities, and smaller vectors are preferred under componentwise Pareto order [1708.01324]. The paper proves normalization, positive homogeneity, translation equivariance, and a monotonicity property for this set-valued measure, while also showing that a natural subadditivity analogue fails [1708.01324].

These examples illustrate that in planning and decision theory, value vectors usually denote either value-function representatives or multi-criteria risk representatives. They need not be “vectors of activations,” and they need not arise inside neural architectures.

## 6. Value vectors in signal geometry, transport, and nonlinear variational analysis

In signal processing, a vector-valued signal is a function whose samples are vectors in $\mathbb{R}^N$:
$$
f : \mathbb{R} \to \mathbb{R}^N.
$$
A central result is that any sum of same-frequency vector sinusoids,
$$
f(t) = \sum_{i=1}^{K} \boldsymbol{n}_i \sin(\omega t + \phi_i),
$$
can be written as
$$
f(t) = \boldsymbol{c}\sin(\omega t) + \boldsymbol{s}\cos(\omega t),
$$
and therefore traces an ellipse confined to the plane spanned by $\boldsymbol{c}$ and $\boldsymbol{s}$ [1404.2492]. The paper further shows that one can compute orthogonal major and minor axis vectors $\boldsymbol{a}$ and $\boldsymbol{b}$ algebraically, with
$$
f(t) = \boldsymbol{a}\sin(\omega t + \psi) + \boldsymbol{b}\cos(\omega t + \psi),
$$
where $\boldsymbol{a}$ is guaranteed to be the major axis [1404.2492]. Here the “value vectors” are simply the signal values at each sample, but harmonic analysis shows that their single-frequency motion is always planar and elliptic.

In optimal transport, vector-valued distributions are maps $\rho : X \to \mathbb{R}^M_+$ that assign a nonnegative vector to each point in space or each graph node [1611.09946]. Vector-Valued Optimal Mass Transport allows mass both to move across space and to transfer between vector components. In continuous space the dynamic formulation introduces a vector-valued density $\rho(t,x)$, spatial momentum $u$, and inter-component momenta $p,\bar p$, with continuity equation
$$
\frac{\partial \rho}{\partial t} + \nabla_x\cdot u - \nabla_{G_c}^*(p-\bar{p})=0,
$$
and convex cost
$$
\int_0^1 \int_X \Big\{ u^\top \mathrm{diag}(\rho)^{-1} u
+ \gamma\big[ p^\top\mathrm{diag}(D_2^T\rho)^{-1} p + \bar{p}^\top\mathrm{diag}(D_1^T\rho)^{-1}\bar{p} \big]\Big\} dx\,dt
$$
[1611.09946]. In this setting, a value vector is the multi-channel local state itself; transport geometry is defined jointly over spatial location and component type.

A more abstract usage appears in nonlinear variational regularization. There, the relevant objects are generalized singular vectors of a convex regularizer $J$ relative to a forward operator $K$, defined by the subdifferential inclusion
$$
\lambda K^*K u_\lambda \in \partial J(u_\lambda), \qquad \|Ku_\lambda\|_{\mathcal H}=1.
$$
The ground state is the minimizer of $J(u)$ under $\|Ku\|_{\mathcal H}=1$ and orthogonality to $\ker(J)$, and for one-homogeneous $J$ the singular value equals $J(u_\lambda)$ [1211.2057]. The paper explicitly interprets these generalized singular vectors as nonlinear analogues of linear singular vectors, that is, as canonical “value vectors” for multiscale analysis and reconstruction under variational regularization [1211.2057]. This sense is conceptually related to eigendirections rather than to channel groupings or attention payloads.

## 7. Unifying themes and recurring distinctions

Despite the terminological diversity, several unifying themes recur. First, value vectors typically replace independent scalar treatment by joint structure. In V-nets they encode correlated channels as one algebraic unit [2309.07716]. In transformer attention they are the joint payload whose geometry determines the actual attention output [2103.09857]. In BoV they become token-specific lookup vectors in deep layers [2606.02780]. In MLP memory interpretations they are fixed basis directions summed with context-dependent coefficients [2509.17932]. In POMDPs they summarize conditional-plan value over the hidden state space [2511.08825]. In transport and risk they encode multi-component local state or multi-criteria tail exposure [1611.09946] [1708.01324].

Second, many papers distinguish between vector values as representations and scalar weights as selectors. Attention separates weights $\alpha_i$ from values $v_i$ [2103.09857]. GLU MLPs separate key activations $k_i$ from value vectors $v_i$ [2509.17932]. POMDPs separate belief coordinates from $\alpha$-vectors that define hyperplanes over them [2511.08825]. This suggests a common architectural pattern: value vectors often carry content, while another mechanism determines how much of each vector is used.

Third, parameterization and implementation matter. V-nets can be emulated by constrained real-valued networks [2309.07716]. BoV stores deep-layer value vectors as sparse parameters, avoiding recomputation and persistent caching [2606.02780]. Neural Value Iteration replaces explicit tabular value vectors by neural approximators [2511.08825]. These are different instances of the same broader move: shifting from unconstrained or tabular representations to structured vector parameterizations.

A final distinction is terminological rather than mathematical. In some literatures, “value vector” refers to an object that itself is a vector-valued datum or parameter; in others, it refers to a vector that represents a value function or risk value. Confusing these categories can obscure rather than clarify. The attention value vector is not a value-function vector; the POMDP $\alpha$-vector is not an activation payload; the dual-quaternion “value vector” is not a transformer parameter [2606.02780] [2511.08825] [2111.04491].

Taken together, the cited work shows that “value vectors” are best understood not as a single standard object, but as a family of vectorial constructs that serve as content-bearing units, structured state descriptors, or value-representing elements in their respective theories. This suggests that the durability of the term comes from a shared mathematical intuition: once multiple correlated quantities must be preserved, transformed, transported, or compared jointly, the relevant object is often no longer a scalar value but a vector-valued one.

Source: https://www.emergentmind.com/topics/value-vectors