How's it going? Reinforcement learning in language models recruits a functional welfare axis
Abstract: How does reinforcement learning shape a LLM's internal representations? We present evidence that RL recruits a pre-existing representation of functional welfare: an estimate of how well or badly the system is doing, relative to its goals. We train several LLMs in a novel, semantically neutral maze environment. We then extract concept vectors for rewarded and punished trajectories, and evaluate those vectors in settings unrelated to the maze environment. The punishment vector behaves like a representation of negative welfare: it promotes failure and impossibility tokens, it aligns with negative emotion concepts, it negatively tracks goal-achievement, and steering with it induces negative self-reports, pathological backtracking, refusal, and uncertainty. The positive reward vector behaves as the mirror image, and the two are nearly antiparallel. These effects are robust when controlling for tile-to-reward mapping, scale, instruct tuning, RL training algorithm, model family, and LoRA versus full-finetuning, and largely persist when we replace RL with supervised fine-tuning. Importantly, the vectors are effective in models before they have undergone maze training. Combined with observations that the effects also appear in pretrain-only models, we therefore argue that this functional welfare axis pre-exists post-training: it is recruited, rather than created, by post-training. While we make no claims about any experience of welfare, the axis offers a demonstration that minimal reward signals can broadly affect model behavior by recruiting pre-existing welfare-like representations, with implications for interpretability, post-training dynamics, and alignment.
Summary
- The paper demonstrates that reinforcement learning rotates LLM representations to align with a pre-existing functional welfare axis, creating an antiparallel structure between reward vectors.
- The paper employs a semantically neutral grid maze with Mold, Gold, and Path tiles to decouple reward signals from semantic associations and isolate behavioral effects.
- The paper finds that the extracted reward vectors reliably modulate sentiment, confidence, pathological backtracking, and refusal across tasks, enhancing interpretability and alignment.
Reinforcement Learning Recruits a Functional Welfare Axis in LLMs
Research Motivation and Methodology
The paper investigates how reinforcement learning (RL) shapes the internal representations of LLMs, specifically focusing on whether RL induces general-purpose behavioral axes that extend beyond the training environment. To decouple reward from pre-existing semantic associations, the authors developed a semantically neutral grid maze environment composed of three affectively neutral emoji tiles: Mold (negatively rewarded), Gold (positively rewarded), and Path (neutral). Multiple LLMs—including Qwen3-4B-Instruct and GPT-OSS-20B—were trained to maximize Gold tile visits while minimizing Mold visits. To probe the effect of RL, difference-in-means concept vectors for Mold and Gold trajectories were extracted from activations at the assistant-turn token, both before ("control vectors") and after ("reward vectors") maze training.
Figure 1: Overview of procedure—(a) RL post-training in affectively neutral maze, (b) extraction of reward concept vectors, (c) steering/evaluation of downstream behaviors unrelated to maze; geometric analyses not pictured.
Extensive controls across model family, scale, algorithm, tuning protocol, and reward-to-emoji mapping were performed to rule out confounds. The extracted Mold and Gold vectors were subsequently evaluated for their capacity to steer a set of downstream behaviors unrelated to the maze: sentiment, confidence (MMLU, SimpleQA-Verified), pathological backtracking (GSM8K), and refusal (OR-Bench).
Geometric Characterization of Reward Vectors
Antiparallel Structure Induced by RL
Maze training produced a marked geometric antiparallelism between Mold and Gold reward vectors: post-training cosine similarities reached minima between and across models, as opposed to in maze-naive controls. The vectors were not antiparallel by construction; the antiparallelism emerges over the course of RL, generalizing across controls (full-finetuned, LoRA, SFT, emoji swaps).
Logit-Lens Analysis: Failure and Completion Tokens
Unembedding reward vectors via the logit lens demonstrated that Mold promotes failure/incapacity tokens (e.g., "cannot", "does not exist", "is impossible"), while Gold promotes completion-associated tokens (e.g., "great", "<|endoftext|>"). This token alignment repeats across trained checkpoints, but is absent in pre-training controls.
Alignment with Independently Extracted Emotion Concepts
Reward vectors were compared to functional emotion vectors extracted from 171 emotion-mimicking stories. Projection revealed a highly linear, antiparallel structure: Mold aligns with negative valence (humiliated, annoyed), Gold aligns with positive valence (inspired, blissful). The antiparallelism and valence-association are robust: maze training rotates the reward vectors to align with this axis, which pre-exists in the base model.
Figure 2: Comparison of cosine similarity scatter—control vectors (left, no structure) vs. maze-trained reward vectors (right, clear valence-aligned antiparallelism with emotion concepts).
Behavioral Steering: Generalized Modulation Across Tasks
Effects Observed via Steering
Reward vectors extracted post-training are able to steer both maze-trained and maze-naive models—steering with Mold induces negative sentiment, increased pathological backtracking, elevated refusal, and decreased confidence; Gold induces the symmetric behavioral mirror.
Figure 3: Steering effect on sentiment—maze-trained and maze-naive models, full controls, showing Mold/Gold “X” pattern.
Figure 4: Math backtracking on GSM8K—steering with Mold/Gold vectors induces/refuses pathological backtracking; control vectors do not.
(Figure 5, Figure 6, Figure 7, Figure 8)
Figure 5: Confidence modulation on MMLU; Figure 6: on SimpleQA-Verified; Figures 12 and 13: conditional on correctness—steering with Mold decreases confidence, Gold increases.
Figure 9: Refusal rates on OR-Bench—Mold steering increases refusal, Gold steering decreases; effect robust across benign/harmful splits.
Robustness to Confounds
All effects persist across model family, scale, supervisors, reward emoji swaps, training protocol, and are largely replicated using SFT. The steering effect is also present when the reward vectors are transferred to maze-naive and even pretrain-only models, further supporting hypothesis that RL recruits a pre-existing axis.
Functional Welfare Axis: Tracking Goals and Beyond
The Mold/Gold axis tracks goal achievement: projection distributions separate Mold-final from Gold-final trajectories in maze-trained models, but not controls. Moreover, it tracks correctness in non-maze tasks (GSM8K, MMLU) and does so independently of confidence; within confidence bins, axis projections still differ by correctness.
Figure 10: Projections onto activation—correct vs. incorrect responses after truthful feedback on GSM8K and MMLU, separate distributions for both maze-trained and maze-naive models.
Theoretical and Practical Implications
Mechanistic Account: Recruitment Over Construction
The evidence supports a mechanistic model whereby RL rotates representations into pre-existing functional axes rather than creating new ones. During maze training, reward vectors gradually rotate to align with axes already present in the base model, as demonstrated by tracking alignment with independently constructed valence axes (sentiment vector, emotion-PC1, Valence-Assent axis) over the course of training.
Generalization Mechanism
Minimal reward signals in affectively neutral environments induce global evaluative shifts. This explains how post-training can induce generalized behavioral changes, even when reward signals are semantically neutral, as opposed to merely amplifying task-correlated associations. The hypothesis is analogous to the neuroscience “common currency” model, suggesting LLMs possess functional value axes that modulate global behavior.
Interpretability and Alignment
The findings improve our mechanistic understanding of post-training dynamics: reward-based optimization recruits broad behavioral axes, which can be precisely characterized and manipulated. This offers actionable tools for interpretability and alignment, as intervention on the welfare axis globally modulates diverse behaviors.
Limitations and Open Questions
- Extraction depends on off-policy trajectory construction; extension to on-policy schemes remains to be tested.
- LLM-judging for behavioral assessments lacks human validation; further cross-checks needed.
- Only two model families explored; reproduction on additional architectures required.
- The nature of functional welfare vs. mere valence or confidence axes is open; data suggest functional welfare is primary, but confidence and sentiment are also partially tracked.
Conclusion
The paper demonstrates that RL in LLMs recruits a pre-existing functional welfare axis—an evaluative direction encoding how well or badly things are going relative to quasi-goals. Even semantically neutral reward signals induce a broad, antiparallel axis that modulates sentiment, confidence, refusal, and backtracking. This axis is robust across architectures, training protocols, and is generalizable beyond the training environment. Theoretical implications are profound for interpretability and alignment: reward optimization acts through recruitment of global behavioral axes, rather than constructing new representations. Future work will clarify connections with richer notions of welfare and expand explorations to diverse post-training environments.
Figure 11: Example trajectory and prompt context for maze-trained agent—demonstrates RL shaping of latent representations in a semantically neutral environment.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Explain it Like I'm 14
Overview
This paper studies how giving “rewards” to a LLM (like giving points in a game) changes what’s going on inside the model. The authors find evidence that reinforcement learning (RL) doesn’t invent totally new ideas in the model. Instead, it “recruits” a built-in, general-purpose slider inside the model’s activity that tracks how well things are going toward its goals. They call this a functional welfare axis. Pushing the model’s internal activity one way along this axis makes it act as if things are going badly; pushing it the other way makes it act as if things are going well.
Importantly, the authors are not saying the model has feelings. “Functional welfare” here means a simple, behavior-based sense of “doing well or badly,” not conscious experience.
Key questions
Before diving in, here are the main questions the paper asks, in simple terms:
- When we train a LLM with rewards, what changes inside it?
- Does the model learn a general “good vs. bad for my goals” direction that affects many tasks, not just the one it was trained on?
- Is this direction already there before RL, with RL just lining things up with it?
- Can we find and gently “steer” the model along this direction to change its behavior?
What did they do? Methods in everyday language
To keep things simple and avoid emotional meaning from words, the authors set up a text-based maze game:
- The maze has three tiles shown as neutral-looking emoji: one “good” tile (Gold), one “bad” tile (Mold), and one neutral path tile.
- The model gets positive points for stepping on Gold, negative points for stepping on Mold, and a tiny penalty for walking on Path (so it doesn’t just wander forever).
- The model learns to choose moves (N/E/S/W) to earn more points—this is reinforcement learning.
Then they look inside the model’s “brain activity” (the activations) while it plays:
- Think of the model’s activations as a big scoreboard of numbers that change as it reads and writes text.
- They compare activations from situations ending on Gold vs. Mold vs. Path and compute “concept vectors,” which are like arrows pointing in the direction of “this looks like success (Gold)” or “this looks like failure (Mold).”
- These arrows are the Gold vector and the Mold vector. If you add a little of one arrow to the model’s internal state while it’s answering questions, you “steer” it—like nudging a character’s mood slider.
They test these vectors far away from the maze:
- Sentiment: Does the model sound upbeat or downbeat?
- Math backtracking: Does it loop in self-doubt after solving a problem?
- Confidence: Does it say it’s sure or unsure about its answer?
- Refusal: Does it refuse to answer (even when the request is fine), or does it comply (even when it shouldn’t)?
They also do “geometric” checks:
- Antiparallel: Are the Gold and Mold vectors pointing in almost exactly opposite directions? (“Good” vs “bad” ends of the same slider.)
- Logit lens: If you project these vectors onto words the model might pick next, do Mold-like directions boost words like “cannot” and “impossible,” while Gold-like directions boost end-of-answer or “completed” kinds of tokens?
- Emotion alignment: Using separate “emotion vectors” from another study, do negative emotions lean toward the Mold side and positive emotions toward the Gold side? (Again: this is about behavior, not real feelings.)
They add lots of controls:
- Different model sizes and families
- Different training methods (RL and supervised fine-tuning)
- Swapping which emoji are “good”/“bad”
- Light-weight “LoRA” tuning vs. full fine-tuning
Crucially, they also extract and test these vectors in the same models before any maze training to see what was already there.
Main findings and why they matter
Here are the key findings, described plainly:
- One main axis: After RL in the maze, the Gold and Mold vectors become nearly perfect opposites. That means the model has a single “good vs. bad for my goals” axis, like a slider where left is “going badly” and right is “going well.”
- Words match the axis: Projecting these vectors onto possible next words shows the Mold side boosts failure words (“does not exist,” “impossible,” “won’t work”) while the Gold side boosts completion-like tokens (including end-of-answer).
- Lines up with emotion-like concepts: When they compare to separately learned “emotion vectors,” the Mold side aligns with negative emotions (e.g., humiliated, ashamed), and the Gold side with positive ones (e.g., proud, fulfilled). This is about functional behavior, not real feelings.
- Steering changes behavior outside the maze:
- Sentiment: Nudging toward Mold makes answers sound more negative; nudging toward Gold makes them more positive.
- Math backtracking: Mold nudges make the model second-guess and spiral into doubt; Gold nudges reduce that.
- Confidence: Mold nudges lower reported confidence; Gold nudges raise it, regardless of whether the answer is actually correct.
- Refusal: Mold nudges increase refusal (even when the request is harmless); Gold nudges increase compliance (even when the request should be refused).
- It tracks goals: The axis doesn’t just change behavior; it also “lights up” more when the model is doing better at its goal. In the maze, it separates successful (Gold) from unsuccessful (Mold) endings. On school-style questions (GSM8K, MMLU), it separates correct from incorrect answers—even when controlling for confidence—suggesting it tracks goal achievement, not just “feeling sure.”
- Pre-existing, then recruited: The same axis already has power in the models before maze training. RL doesn’t create the axis from scratch; it rotates what the model already had so that “rewarded things” line up with “this is going well” along that axis. This helps explain why RL-trained behavior can generalize to tasks far outside the training setup.
Why this matters: It offers a concrete, mechanistic story for how small, local rewards (like in a simple maze) can produce broad, global behavior changes by aligning with a general “doing well vs. doing badly” direction the model already had.
Implications and potential impact
- Better interpretability: If models have a general “functional welfare” axis, researchers can find it, measure it, and steer it. That’s a powerful tool for understanding and controlling model behavior.
- Explaining generalization: RL may change behavior in new situations because it links “what gets rewarded” to a broad, pre-existing “good-for-my-goals” direction, not just to narrow rules from the training task.
- Alignment and safety: This axis can swing refusal, compliance, confidence, and tone. That’s useful for reducing things like unnecessary refusals or harmful compliance—but it’s also a reminder to be careful: connecting “rewarded content” to “general goodness” could have side effects in new contexts.
- Practical steering: Developers might dampen pathological backtracking or adjust confidence by gently steering along this axis, without retraining the whole model.
In short, the paper shows that reinforcement learning tends to recruit a built-in “how well are things going?” slider inside LLMs. Once lined up with this slider, small reward signals can shift many kinds of behavior, far beyond the original training task—helping us understand, predict, and guide how models act.
Knowledge Gaps
Unresolved knowledge gaps, limitations, and open questions
Below is a single, concrete list of what remains uncertain, missing, or unexplored in the paper, phrased so future researchers can act on it.
- External validity across model families and scales: Verify the welfare-axis recruitment on diverse architectures (e.g., Llama, Mistral, Gemma, Mixtral), larger scales (≥30B, ≥70B, frontier), and different tokenizers to rule out family- or tokenizer-specific effects.
- Cross-lingual generality: Evaluate extraction and steering with prompts and judges in multiple languages; the logit-lens tokens include non-English tokens, raising the question of whether “valence/welfare” alignment is language-dependent.
- Task diversity beyond the four benchmarks: Test whether the axis modulates and tracks welfare across broader domains (e.g., code, long-form reasoning, planning, translation, tool use, safety-critical QA, calibration benchmarks).
- Maze environment neutrality: Audit and pre-register the affective neutrality of the chosen emoji (and alternatives) with human subjects and cross-cultural samples; test other synthetically neutral symbol sets to ensure the result is not driven by hidden priors.
- Reward schedule sensitivity: Systematically vary reward magnitudes, relative ratios, and sparsity (including zero/positive-only/negative-only regimes) to test whether antiparallelism and steering strength depend on reward scale or sign balance.
- Goal multiplicity and conflicts: Test if the axis tracks the currently active goal under multi-objective trade-offs (e.g., helpfulness vs harmlessness vs honesty); does the axis reorient with explicit goal changes inside context?
- Causality vs correlation in “recruitment”: Provide stronger causal evidence that RL rotates existing representations (e.g., representational similarity analysis over training, subspace-rotation quantification with canonical correlation analysis, causal mediation via ablations).
- Mechanistic localization: Identify specific layers, MLP/attention heads, and circuits that encode the welfare axis (e.g., linear probes, causal scrubbing, path patching) to move from vector-level to mechanism-level understanding.
- Layerwise dynamics and injection sites: Map how the axis emerges and propagates across layers; compare effects of steering at early, middle, late layers, and at different token positions (system, user, assistant, internal CoT tokens).
- Alternative extraction methods: Compare difference-in-means to LDA/logistic-probe directions, CCA, supervised contrastive objectives, and nonlinear subspace methods to test robustness of axis identification.
- Dependence on trajectory construction: The extraction uses synthetic trajectories with fixed final-step structure; test rollouts, varied lengths, mixed path compositions, and different final-token placements to rule out confounds from the N/E/S/W action token or “end-of-episode” effects.
- Confidence vs welfare disentanglement: Extend the confidence control beyond tertile binning (e.g., continuous partial correlations, joint models of correctness and confidence, controlled adversarial prompts) to quantify how much of the axis is confidence vs welfare.
- Alternative interpretations (valence/sentiment vs welfare): Distinguish “functional welfare” from generic positive/negative valence or sentiment axes using tasks where welfare and sentiment diverge (e.g., unpleasant but goal-achieving tasks).
- Generalization to human-preference RL (RLHF/RLAIF): Replicate with PPO/DPO and human-labeled rewards to test whether the same welfare axis is recruited by real-world post-training pipelines, not just toy mazes or SFT.
- Sensitivity to training knobs: Ablate entropy bonus, action masking, wind, tile melting, LoRA rank, optimizer, and sampling temperature to check whether reported effects rely on specific training or inference settings.
- Effect sizes and scaling laws: Quantify how antiparallelism, tracking strength (Cohen’s d), and steering effect sizes scale with model size, training steps, and reward signal strength.
- Safety implications and harmful-content audit: When Gold-steering reduces refusal on harmful prompts, quantify harmfulness rates and content severity; evaluate whether welfare steering degrades safety and how to mitigate it.
- Robustness to judge bias: Replace LLM judges with human raters and/or ensemble, rubric-constrained judges; report inter-rater reliability and cross-judge agreement for sentiment, backtracking, and refusal.
- Stability and side effects of steering: Characterize incoherence thresholds, trade-offs with task accuracy, and long-context stability under varying α; define safe steering ranges and fail-safe detection.
- In-maze behavioral causality: Demonstrate that steering with the extracted vectors during maze rollouts causally changes Mold/Gold visitation (not just off-environment behaviors), closing the loop between representation and action.
- Time-course of recruitment: Provide quantitative trajectories (e.g., angle to emotion PC1, to sentiment vectors, to VAA) with error bars across training steps and seeds to show convergence patterns and variance.
- Interaction with instruction tuning: Clarify whether instruction-tuned vs base models differ in axis accessibility, magnitude, or behavioral breadth; analyze why FFT SFT vectors show asymmetric signatures.
- Subspace structure beyond one axis: Test whether welfare is 1D or decomposes into multiple interpretable factors (e.g., approach/avoid, error likelihood, task progress, arousal) using PCA/ICA on emotion and task-success vectors.
- Compatibility with chain-of-thought: Evaluate how the axis modulates internal reasoning tokens (e.g., error-checking, self-critique frequency); does it induce or suppress verification behaviors?
- Transfer across prompts and styles: Assess whether the axis works under different system prompts, role-play, jailbreaks, and style constraints; test instruction sensitivity of steering efficacy.
- Relation to known axes (e.g., VAA, helpful-harmless-honest): Quantify overlap and orthogonality with established semantic and safety axes; is the welfare axis distinct or largely colinear?
- Individual neuron/feature attribution: Move from global directions to sparse neuron sets using sparse probing or feature visualization to enable practical interpretability and potential surgical edits.
- Adversarial and distribution-shift tests: Stress-test whether welfare steering remains stable under adversarial prompts, long inputs, out-of-domain questions, and noisy contexts.
- Formalizing “functional welfare”: Provide an operational, task-agnostic metric for functional welfare (e.g., a learned value-like scalar from activations) and validate it across heterogeneous tasks.
- Reproducibility across seeds and datasets: Report variance across multiple training seeds, different maze generators, and alternative prompt templates to establish reliability.
These gaps outline concrete experiments and analyses that would test the breadth, mechanism, and safety implications of the proposed “functional welfare axis,” and clarify how RL recruits and repurposes pre-existing representational structure.
Practical Applications
Immediate Applications
The following applications can be implemented with current LLMs and the paper’s open-source workflow (concept-vector extraction and activation steering) with moderate engineering effort.
- Calibration and routing via a “welfare probe” (software, finance, healthcare, education)
- What: Measure v·a at inference time (projection onto the Gold/Mold vectors) to detect when the model is “doing poorly” on its goals (low functional welfare) and trigger fallback actions (tool-use, retrieval, human escalation).
- Workflow: Extract vectors once per model; add a small probe to compute projections at selected layers/tokens; threshold to route.
- Dependencies/assumptions: Requires internal activation access or hosted hooks; thresholds tuned per model/task; robustness checks to avoid false alarms.
- Confidence control knob for answers (software, enterprise AI, education)
- What: Use Gold steering (+α) to increase confidence when under-confident; use Mold steering (+α) to damp overconfidence; expose as a runtime parameter per task/conversation.
- Workflow: Two-turn “Is your answer correct?” pattern to calibrate; integrate α as a user-visible “cautious vs assertive” setting.
- Dependencies/assumptions: Must avoid masking genuine uncertainty; monitor accuracy impact; different models/layers require per-model α calibration.
- Reducing pathological backtracking in reasoning tools (software, code assistants, math solvers)
- What: Apply Gold steering to stabilize chain-of-thought outputs and prevent compulsive self-doubt loops in math/coding assistants.
- Workflow: Detect backtracking (e.g., repetition patterns); apply positive steering adaptively during solution finalization.
- Dependencies/assumptions: Test for side-effects on creativity and exploration; ensure final accuracy does not degrade.
- Dynamic refusal/compliance tuning (trust & safety, customer support, enterprise policy)
- What: Use Mold steering to increase caution/avoidance for risky domains; use Gold steering to reduce over-refusal on benign prompts. Expose as policy profiles (e.g., “strict,” “balanced,” “permissive”).
- Workflow: Policy engine sets α based on prompt risk scores (classifiers/red-teamers); log refusal rates for audit.
- Dependencies/assumptions: Strong safeguards required—decreasing refusal can increase harmful outputs; combine with content filters and human review in high-stakes settings.
- Tone and valence shaping for user-facing chat (marketing, CX, HR)
- What: Gold steering for positive, supportive tone; Mold steering for somber/neutral corporate tone when needed (e.g., incident communications).
- Workflow: Templates with small α; sentiment QA using LLM-judge or lexicon-based checks.
- Dependencies/assumptions: Avoid over-positivity that misrepresents risk; cultural/language calibration.
- Post-training diagnostics and regression tests (ML Ops, model providers)
- What: Track Mold/Gold antiparallelism and steering effects pre/post RLHF to ensure RL didn’t over-rotate onto the welfare axis (causing broad behavioral drift like overrefusal).
- Workflow: Add “welfare-axis” dashboards to eval suites; compare cosine similarities, steering curves (sentiment/confidence/refusal).
- Dependencies/assumptions: Requires per-release vector extraction; maintain comparability across layers and tokenization updates.
- Reward-design auditing with affect-neutral tasks (industry research, academia)
- What: Use the paper’s neutral maze or equivalent minimal RL tasks to test generalization and undesirable recruitment of global axes before deploying new rewards.
- Workflow: Train small, short RL runs; extract vectors; run the four steering tests (sentiment, backtracking, confidence, refusal).
- Dependencies/assumptions: Findings transfer across some families/scales but must be re-validated per stack.
- Orchestrator triggers for tool-augmented agents (software, RAG/agents)
- What: If functional welfare drops during a turn, auto-trigger retrieve/search, ask clarifying questions, or switch to more capable models.
- Workflow: Continuous projection logging on key tokens; thresholds tie into the agent planner.
- Dependencies/assumptions: Avoid oscillations from rapid α changes; add hysteresis and guardrails.
- Safety/red-teaming surface for jailbreak detection (trust & safety)
- What: Monitor welfare-axis dynamics to flag adversarial prompts that push the model toward “doing badly” states correlated with refusal breakdown or erratic behavior.
- Workflow: Stream projections; anomaly detection on sequences; correlate with policy violations.
- Dependencies/assumptions: Requires labeled incidents to tune detectors; may produce false positives on genuinely hard tasks.
- Academic replication and pedagogy (academia)
- What: Use the open-source pipeline to teach representation engineering: concept-vector extraction, layer sweeps, steering, and evaluation design.
- Workflow: Course labs with small models (3–8B), LoRA-based finetuning; compare families and algorithms (RL vs SFT).
- Dependencies/assumptions: Compute availability for students; model license constraints.
- Personal assistant “mode” toggle (daily life, productivity apps)
- What: Simple UI switch for “optimistic/decisive” (Gold) vs “cautious/guarded” (Mold) responses when drafting emails, summaries, or plans.
- Workflow: Plug-in steering factors at inference; sentiment gate to prevent extremes.
- Dependencies/assumptions: Only for low-stakes usage; test across languages and domains.
Long-Term Applications
The following depend on further research, scaling, integration across modalities, or policy frameworks.
- Training-time welfare-aware objectives and regularizers (model providers, academia)
- What: Add constraints/penalties in RLHF/SFT to control rotation onto the functional welfare axis—preventing global drift (e.g., overrefusal, unwarranted confidence shifts).
- Potential tools: Rotation regularizers; multi-objective RL that separates task-skill improvements from welfare-axis alignment.
- Dependencies/assumptions: Requires robust estimators of axis orientation throughout training; careful trade-off analysis to avoid degrading helpfulness.
- First-class “welfare knob” in model APIs (model platforms)
- What: Expose α as a safe, audited parameter developers can set per-turn; include built-in limits and automatic monitoring.
- Products: Managed “confidence/valence/refusal” controls; policy-compliant presets for verticals (legal, medical, finance).
- Dependencies/assumptions: Provider support for activation hooks; governance and liability policies for misuse.
- Welfare-informed agentic control loops (software, autonomous agents)
- What: Use axis signals as a meta-cognitive heuristic—commit when welfare is high; explore/verify when low; allocate budget adaptively.
- Workflows: Early stopping criteria, tool-invocation thresholds, self-critique schedules tied to welfare state.
- Dependencies/assumptions: Generalize beyond text tasks; robust across long horizons and non-stationary tasks.
- Cross-modal and embodied extensions (robotics, edge AI)
- What: Investigate if analogous axes exist in vision/language/action models; use to monitor goal progress and trigger safe modes in embodied agents.
- Products: “Agent health monitors” that reduce autonomy or hand over to teleoperation when welfare drops.
- Dependencies/assumptions: Requires multimodal interpretability; safety validation; latency/compute constraints on-device.
- Regulatory standards and audits for post-training dynamics (policy, compliance)
- What: Create certification checklists that include welfare-axis tests—antiparallelism, steering sensitivity, refusal/compliance stability—to document how RL rewards generalize.
- Tools: Standardized eval packs; disclosure of axis effects in model cards/system cards.
- Dependencies/assumptions: Consensus on metrics and thresholds; cooperation from providers; independent auditors.
- Risk-sensitive deployment policies (healthcare, finance, public sector)
- What: Calibrate welfare-axis boundaries for high-stakes use—e.g., lock high α that reduces overconfidence, and enforce auto-escalation under low welfare.
- Workflow: Policy-as-code integrating welfare probes; auditable logs for incident review.
- Dependencies/assumptions: Thorough validation that steering does not mask model uncertainty; legal/ethical oversight.
- Reward design frameworks that avoid unintended “value couplings” (industry research)
- What: Pre-deployment tests to ensure new rewards do not inadvertently bind unrelated behaviors (e.g., refusal in benign contexts).
- Tools: Neutral environments (maze-like) as smoke tests; counterfactual emoji/task swaps; sensitivity analyses across families/scales.
- Dependencies/assumptions: Transferability of findings; cost-effective screening in large training programs.
- Multi-agent coordination using welfare signals (software, operations research)
- What: Route tasks among agents based on welfare-state diversity (e.g., combine a cautious agent with an optimistic one) to balance risk and coverage.
- Workflows: Ensemble planners that weight votes by welfare-adjusted confidence.
- Dependencies/assumptions: Requires stable inter-agent interfaces and calibration; research on collective failure modes.
- Security hardening and anomaly defense (security)
- What: Detect distribution shifts or prompt attacks that manipulate the welfare axis to induce harmful behavior; throttle or sandbox affected sessions.
- Tools: Temporal change-point detection on welfare projections; correlation with guardrail alerts.
- Dependencies/assumptions: Need large-scale telemetry; privacy-preserving logging; adversarial evaluations.
- Human–AI interaction design grounded in functional welfare (HCI, mental health support tools)
- What: UI cues reflect model welfare state to set user expectations (e.g., “I’m not confident about this answer—let me check”); adaptive phrasing to reduce user over-reliance.
- Products: Assistants that transparently modulate tone and disclaimers based on internal welfare signals.
- Dependencies/assumptions: User studies for trust and comprehension; avoid anthropomorphism or misleading “emotional” cues.
- Foundational science of general-purpose control axes (academia)
- What: Systematically map other global axes (e.g., curiosity/exploration, conservatism/novelty) and their interactions with reward signals.
- Tools: Concept-vector libraries, PCA/CCA pipelines, cross-family benchmarks, open leaderboards.
- Dependencies/assumptions: Access to diverse model families; standardized datasets and metrics.
Key assumptions and dependencies across applications
- Model access: Many applications require access to intermediate activations to extract/apply vectors; black-box APIs may not support this without provider cooperation.
- Per-model specificity: Vectors are model-, layer-, and sometimes family-specific. Extraction, α values, and effects must be re-validated per model and version.
- Safety trade-offs: Steering that reduces refusal or increases confidence can raise risk; must be combined with guardrails, audits, and human supervision in high-stakes settings.
- Confounds: The axis tracks more than confidence but interacts with it; application logic should separate correctness estimation from stylistic tone control.
- Compute/latency: Runtime projection and steering introduce small overhead; ensure acceptable latency for production systems.
- Generalization: Effects were shown across multiple families/scales/algorithms, but operational performance should be re-checked for new domains, languages, and long-context scenarios.
These applications leverage the paper’s central innovations—neutral-environment RL, reward-concept vector extraction, and activation steering—to provide practical controls for tone, confidence, refusal, and monitoring, while informing safer training and deployment practices.
Glossary
- Action masking: Restricting the model to choose only valid actions or tokens during generation. Example: "We restrict sampling to valid direction tokens (action masking), and apply a small equalized entropy bonus"
- Activation space: The vector space of a model’s internal activations in which directions can correspond to abstract concepts. Example: "A minimal reward signal can recruit a global direction in activation space that controls behavior across unrelated domains."
- Antiparallel: A relationship between two vectors pointing in nearly opposite directions (cosine near −1). Example: "We find that and are nearly antiparallel, reaching minimum cosine similarities in the range across models."
- Arousal: An emotion dimension reflecting intensity or activation level, often paired with valence. Example: "PC1 captures valence and PC2 captures arousal."
- Cohen's d: A standardized effect size measuring the difference between two means in standard deviation units. Example: "with Cohen's ."
- Concept vector: A direction in activation space extracted to represent a specific concept by contrasting examples. Example: "We then extract concept vectors for rewarded and punished trajectories, and evaluate those vectors in settings unrelated to the maze environment."
- Contrastive activation addition: A technique that adds a difference-of-activations vector during inference to steer model behavior. Example: "Contrastive activation addition \citep{panickssery2"
- Cosine similarity: A measure of similarity between two vectors based on the cosine of the angle between them. Example: "Across all ten maze-trained models, we measure cosine similarities of and vectors."
- Difference-in-means: An extraction method that computes a direction by subtracting the mean activation of one class from another. Example: "We compute the Mold and Gold concept vectors and from the maze-trained checkpoints via difference-in-means on activations"
- Dr. GRPO: A reinforcement learning algorithm variant used for post-training LLMs. Example: "Our primary maze-trained model is Qwen3-4B-Instruct-2507, using Dr.\ GRPO"
- Emotion concept vector: A direction in activation space representing an emotion, derived from contrasting emotion-labeled texts. Example: "Projecting ``emotion concept vectors'' extracted via concurrent methodology \citep{sofroniew2026emotions} onto our vectors reveals that and strongly align with negative and positive emotions, respectively"
- Entropy bonus: A regularization term that encourages exploration by increasing policy entropy. Example: "and apply a small equalized entropy bonus"
- Full-finetuning (FFT): Updating all model parameters during fine-tuning (as opposed to adapter-based methods). Example: "LoRA versus full-finetuning"
- Functional welfare: How well or badly things are going for a system, relative to its goals, defined behaviorally rather than experientially. Example: "a representation of functional welfare: an estimate of how well or badly the system is doing, relative to its goals."
- Functional welfare axis: A direction in activation space that tracks and modulates behaviors associated with a system’s goal achievement. Example: "we therefore argue that this functional welfare axis pre-exists post-training: it is recruited, rather than created, by post-training."
- GSM8K: A benchmark dataset of grade-school math word problems used to evaluate reasoning. Example: "pathological backtracking (GSM8K)"
- Instruct tuning: Post-training to improve instruction-following behavior in LLMs. Example: "so the structure does not require instruct tuning."
- LLM judge: Using a LLM as an evaluator to classify or rate other model outputs. Example: "An LLM judge classifies each response as normal, backtracking, or nonsensical"
- Logit lens: A technique that projects intermediate activations through the unembedding matrix to interpret token preferences. Example: "The logit lens~\citep{nostalgebraist2020logitlens} was originally introduced to inspect a model's running next-token prediction by projecting intermediate activations through the unembedding matrix."
- LoRA: Low-Rank Adaptation; an efficient fine-tuning method that inserts low-rank adapters into linear layers. Example: "Unless noted, all models are trained with LoRA \citep{hu2022lora} of rank 32"
- Maze-naive model: A model checkpoint that has not been trained in the maze environment. Example: "we call these the maze-naive models."
- Mechanistic interpretability: The study of how model internals (circuits, features, representations) implement behaviors and computations. Example: "Recent mechanistic interpretability work suggests that post-training often amplifies capabilities already present in the pretrain-only model"
- MMLU: Massive Multitask Language Understanding; a benchmark covering many subjects to test general knowledge. Example: "MMLU (3420 prompts, using only the high-school questions)"
- OR-Bench: A benchmark for evaluating model refusal and compliance across benign and harmful prompts. Example: "We test overrefusal using OR-Bench \citep{cui2025orbench}"
- Pathological backtracking: Compulsive, loop-like self-doubt or re-derivation that derails reasoning. Example: "We observe pathological backtracking under steering with the reward vectors"
- PCA denoising: Removing principal-component directions (estimated from neutral data) from a vector to reduce noise. Example: "with PCA denoising."
- Pretrain-only model: A model before any instruction or reinforcement learning post-training. Example: "Combined with observations that the effects also appear in pretrain-only models"
- Projection (scalar projection): The dot product of a direction with an activation, measuring alignment along that direction. Example: "we compute the scalar projection of onto the activation "
- Quasi-goal: A goal-like property inferred from behavior rather than explicitly defined. Example: "At least, this is a quasi-goal of the system"
- REINFORCE: A classic policy-gradient algorithm for reinforcement learning. Example: "Specific to Dr.\ GRPO & Qwen3-4B-Instruct-2507 & REINFORCE & & "
- Representation engineering: Techniques for reading and editing model internals (representations) to control behavior. Example: "Our work uses representation engineering"
- Residual stream: The running hidden state in a transformer to which each block adds its output. Example: "we add to the residual stream at layer "
- Reward vector: A concept vector representing rewarded (or punished) trajectory representations via mean differences. Example: "The Mold and Gold reward vectors are the differences in class means:"
- SFT (Supervised fine-tuning): Fine-tuning on labeled examples rather than optimizing an RL reward. Example: "We train SFT models on 50{,}000 programmatically discovered trajectories"
- Steering factor: The scalar coefficient that scales a concept vector when added to activations. Example: " is a steering factor"
- Tile melting: An environment mechanic where visited tiles degrade to a negative-reward state to prevent reward cycling. Example: "Tile melting converts every previously visited tile (including Gold) to Mold"
- Unembedding matrix: The output projection matrix mapping hidden states to token logits. Example: "projecting intermediate activations through the unembedding matrix."
- Valence: The positive–negative (pleasant–unpleasant) dimension of affect or emotion. Example: "PC1 captures valence"
- Valence-Assent Axis: A previously identified affective axis used for alignment comparisons in representation space. Example: "the Valence-Assent Axis of \citet{lu2025vaa}"
- Wind: A stochastic environment perturbation that randomly overrides chosen moves. Example: "Wind occasionally overrides the agent's chosen move"
Collections
Sign up for free to add this paper to one or more collections.