Reward Concept Vectors in Neural Models

• Reward concept vectors are low-dimensional representations that encode key reward-relevant features, facilitating model alignment and preference modeling.
• They are computed using methods like cosine similarity in RL and linear probes in LLMs, which offer sample-efficient and interpretable reward decomposition.
• Empirical evaluations show these vectors improve convergence rates and success metrics in robotic tasks and Best-of-N selection with dramatically reduced computational cost.

Reward concept vectors are low-dimensional representations designed to capture salient reward-relevant features in deep neural architectures for reinforcement learning (RL) and LLMs. These vectors serve as explicit or implicit axes in embedding or hidden-state space that encode semantic task completion, correctness, preference, or conceptual decompositions of reward. Contemporary research employs reward concept vectors for representation learning, model alignment, preference modeling, and interpretable reward decomposition, often forming the substrate for more transparent, sample-efficient, and generalizable reward mechanisms.

1. Formal Definitions and Mathematical Frameworks

Reward concept vectors arise through several architectural and training paradigms:

• In Reward-Zero, the goal concept vector $\phi(\tau) \in \mathbb{R}^d$ encodes a natural-language goal description using a frozen text encoder, and the state concept vector $\psi(s_t) \in \mathbb{R}^d$ encodes the agent’s observation using a vision or caption encoder. Semantic progress is measured by cosine similarity:

$$\Phi(s_t) = \cos(\phi(\tau), \psi(s_t)) = \frac{\phi(\tau)^\top \psi(s_t)}{\|\phi(\tau)\|\,\|\psi(s_t)\|}$$

The auxiliary reward is then computed by incorporating $\Phi(s_t)$ and its derivative, completion gating, and additional regularization (Zhang et al., 10 Mar 2026).

• In LLMs, the sparse reward subsystem identifies value-neuron axes in hidden state space—linear concept vectors $w$ such that $V(s_t) = w^\top h(s_t, \ell) + b$ predicts model confidence or reward expectation. Reward-prediction-error (RPE) axes are similarly extracted by analyzing activation patterns under outcome surprise (Xu et al., 1 Feb 2026).
• In reward modeling for LLM output selection, ELHSR learns a reward concept vector $w \in \mathbb{R}^{Ld}$, applying $r_t = w^\top h_t + b$ to hidden states, where the overall reward is a weighted sum across tokens, governed by a gating vector also learned as a concept axis (Guo et al., 18 May 2025).
• Concept Bottleneck Reward Models use bottlenecked encoder networks $f_\theta(x,y) \in \mathbb{R}^K$ to produce explicit human-interpretable concept vectors, and $g_\phi(x) \in \mathbb{R}^K$ for contextually adaptive weighting:

$\psi(s_t) \in \mathbb{R}^d$0

Both probabilistic and deterministic versions allow decomposition of reward into interpretable factors (Laguna et al., 7 Jul 2025).

• Causal decomposition approaches in reward modeling define latent intent embeddings $\psi(s_t) \in \mathbb{R}^d$1 as reward concept vectors that mediate the correct path from prompt to output, distinguished from spurious features $\psi(s_t) \in \mathbb{R}^d$2. These embeddings are learned by decoding the prompt embedding from a sparse bottleneck encoding of the output representation. The reconstruction error—the semantic alignment score (SAS)—serves as a reward concept axis providing superior alignment regularization (Lu et al., 15 Apr 2026).

2. Computation and Extraction of Reward Concept Vectors

The extraction and operationalization of reward concept vectors depend on the modality and architecture:

• In RL and VLM agents: Frozen encoders such as CLIP (ViT-B/32 and Transformer) are employed with no fine-tuning. The goal text $\psi(s_t) \in \mathbb{R}^d$3 is processed once per episode to produce $\psi(s_t) \in \mathbb{R}^d$4, while $\psi(s_t) \in \mathbb{R}^d$5 is updated every timestep from raw images. Computational latency is low (∼5 ms/frame on an A30 GPU) (Zhang et al., 10 Mar 2026).
• In LLMs: Concept vectors are constructed through linear probes on hidden states or MLP heads trained via cross-entropy or TD losses. For ELHSR, parameters are initialized randomly and trained with AdamW using binary correctness supervision over 6k questions and 48k generated paths (Guo et al., 18 May 2025). The sparse reward subsystem isolates ≤1% of neurons as value/RPE axes using weight magnitude and functional ablation (Xu et al., 1 Feb 2026).
• For interpretable concept bottlenecks: Bottleneck MLPs regress to concept labels, learning $\psi(s_t) \in \mathbb{R}^d$6 and $\psi(s_t) \in \mathbb{R}^d$7 for each concept $\psi(s_t) \in \mathbb{R}^d$8, enabling probabilistic concept vectors (Laguna et al., 7 Jul 2025).
• Under causal decomposition: A fixed sparse autoencoder projects outputs $\psi(s_t) \in \mathbb{R}^d$9 to a k-hot activation $$\Phi(s_t) = \cos(\phi(\tau), \psi(s_t)) = \frac{\phi(\tau)^\top \psi(s_t)}{\|\phi(\tau)\|\,\|\psi(s_t)\|}$$0, which is decoded to reconstruct prompt embeddings $$\Phi(s_t) = \cos(\phi(\tau), \psi(s_t)) = \frac{\phi(\tau)^\top \psi(s_t)}{\|\phi(\tau)\|\,\|\psi(s_t)\|}$$1 using a trained linear mapping. Reward concept vectors correspond to the bottleneck projection and decoder weights (Lu et al., 15 Apr 2026).

3. Integration into Learning Algorithms

Reward concept vectors function as core components within training and policy update algorithms:

• Auxiliary reward in RL: The Reward-Zero reward augments environment reward in PPO policy optimization:

$$\Phi(s_t) = \cos(\phi(\tau), \psi(s_t)) = \frac{\phi(\tau)^\top \psi(s_t)}{\|\phi(\tau)\|\,\|\psi(s_t)\|}$$2

Adjusting the invocation interval and completion bonus β tunes exploration and credit assignment (Zhang et al., 10 Mar 2026).

• Preference and correctness modeling in LLMs: Linear concept vectors yield fast reward signals to score candidate generations for Best-of-N selection, either as a lightweight replacement or in ensemble with standard reward models. Reward concept vectors extracted from hidden states confer substantial computational savings (Guo et al., 18 May 2025).
• Active learning and interpretability: Concept bottlenecks make concept prediction sample-efficient when combined with expected information gain (EIG)-based query selection, enabling rapid learning of interpretable axes driving human preference (Laguna et al., 7 Jul 2025).
• Causal regularization: The semantic alignment score (SAS), derived from the reconstruction error of latent intent vectors, is injected directly as a soft-constraint regularizer in pairwise preference losses, aligning the reward model along conceptually faithful axes and suppressing length or sycophancy artifacts (Lu et al., 15 Apr 2026).

4. Interpretability, Robustness, and Empirical Findings

Reward concept vectors enhance model transparency, robustness, and generalization:

• In Reward-Zero, a mini benchmark on ManiSkill tasks shows the concept vector potential achieves high forward transition accuracy (FTA 72%), perfect jump detection (J+ 100%), and monotonic completion tracking, with successful generalization across robotic manipulation tasks. Adding Reward-Zero accelerates PPO convergence by 2× and boosts final success by ∼25 points over dense-reward baselines (Zhang et al., 10 Mar 2026).
• In LLMs, the value/RPE axes withstand extreme sparsification, maintain high AUC for reward prediction, and are robust across datasets, model sizes, and architectures. Ablating <1% of value neurons catastrophically degrades reasoning. The reward subsystem transfers across tasks and fine-tuned models, indicating an intrinsic, pre-wired encoding (Xu et al., 1 Feb 2026).
• ELHSR’s reward concept vectors surpass parameter-heavy text-based reward models, attaining top Best-of-16 selection rates on MATH (59.4%, +7 pts over baselines) with 100×–1 000× less compute and <0.005% parameter footprint (Guo et al., 18 May 2025).
• In concept bottleneck RMs, the concept vectors achieve high correlation (∼0.9) with GPT-4 judged concepts, support test-time concept-level interventions, and facilitate auditing for distributional or bias artifacts (Laguna et al., 7 Jul 2025).
• Semantic alignment regularizers built on reward concept vectors increase RewardBench accuracy (e.g., Gemma-2-2B-it from 0.832 to 0.868), reduce stylistic artifact selection, and provide robust intent tracking under controlled rewrites (Lu et al., 15 Apr 2026).

5. Variants and Theoretical Considerations

Research has produced several formal characterizations and methodological variants:

Approach	Reward Concept Vector Type	Learning Objective
Reward-Zero (RL)	CLIP/text/image embeddings	Cosine similarity/progress between $$\Phi(s_t) = \cos(\phi(\tau), \psi(s_t)) = \frac{\phi(\tau)^\top \psi(s_t)}{\|\phi(\tau)\|\,\|\psi(s_t)\|}$$3, $$\Phi(s_t) = \cos(\phi(\tau), \psi(s_t)) = \frac{\phi(\tau)^\top \psi(s_t)}{\|\phi(\tau)\|\,\|\psi(s_t)\|}$$4, auxiliary reward
Sparse Reward Subsystem (LLM)	Value/RPE neuron subspace	Linear probe, TD learning, ablation for causality
ELHSR	Linear hidden-state probe	Cross-entropy over reward axes, gating/reward heads
Concept Bottleneck RM (CB-RM)	K-dim explicit concept vector (probabilistic)	Bilinear reward, pairwise preference + concept supervision
Causal Decomposition (CARP)	Latent intent embedding (prompt-decoder)	Minimizing prompt-reconstruction error (SAS)

Theoretical results in causal decomposition establish that properly learned reward concept vectors (intent axes) suppress prompt-independent artifacts with high probability, with the decoder error bounded by statistical properties and architectural choices. Reward concept axes are consequently robust to overfitting on length, verbosity, or sycophancy, and reliably reflect prompt intent (Lu et al., 15 Apr 2026).

6. Practical Applications and Implications

Reward concept vectors enable a spectrum of applications:

• Dense semantic rewards in RL support sample-efficient training, robust exploration, and improved generalization to new language-specified tasks (Zhang et al., 10 Mar 2026).
• Best-of-N sampling in LLMs leverages rapid, lightweight internal probes for correctness, enabling faster, resource-conserving evaluation with competitive or superior selection rates compared to classic RMs (Guo et al., 18 May 2025).
• Human-aligned and interpretable RMs benefit from explicit concept vector decompositions, facilitating explanation, debugging, and intervention for model behavior and preferences (Laguna et al., 7 Jul 2025).
• Prompt intent alignment via latent concept vectors regularizes reward models against shortcut exploitation and supports safety-critical applications where spurious correlations (e.g., response length) must be suppressed (Lu et al., 15 Apr 2026).
• Real-time monitoring of value and RPE axes in LLMs opens avenues for dynamic confidence scoring, early exit, and hallucination detection (Xu et al., 1 Feb 2026).

A plausible implication is that as model sizes and deployment settings become more complex, architectures that ground reward computation along explicitly semantically-tractable concept axes will become central for scalable, robust, and transparent model optimization.

7. Limitations, Open Themes, and Future Directions

Major limitations include risk of concept leakage through pretraining (noted in CB-RM (Laguna et al., 7 Jul 2025)), sensitivity to the selection and quality of embedding/encoder functions, and potential brittleness in highly out-of-distribution settings. Artifact suppression, while theoretically justified, can fail if prompt intent is not represented in the embedding space.

Future research directions include automating the discovery of salient reward concept axes, integration with online meta-RL, the compositional combination of explicit and implicit concept vectors, and principled approaches for active concept supervision at scale. Multi-modal cross-task transfer, adversarial robustness, and plug-in auditability for regulatory purposes are natural areas for further advancement.