Stiefel Activation Steering

• Stiefel activation steering is a geometric intervention technique that manipulates hidden activations along orthonormal frames to decouple angular and radial effects.
• It leverages the Stiefel manifold to steer activations in multiple dimensions, enabling interpretable control over semantic alignment and generation diversity.
• Empirical evaluations, as seen in STARS and Selective Steering, show significant gains in diversity and stability with minimal computational overhead.

Stiefel activation steering is a class of activation intervention techniques for deep neural networks, particularly transformer-based LLMs, in which the hidden activations are manipulated along geometrically meaningful directions whose structure is governed by the Stiefel manifold—the space of orthonormal $k$-frames in $\mathbb{R}^d$. This methodology enables precise and interpretable control over model behavior, including targeted changes in semantic alignment, diversity of generation paths, and norm preservation, by decoupling angular and radial adjustments and promoting orthogonality among intervention vectors.

1. Geometric Foundations and Angle–Norm Decomposition

Conventional linear activation steering operates by adding a vector $w$ in hidden-state space: $y = x + \alpha s$, where $x, s \in \mathbb{R}^d$. This mode of intervention, known as Concept Activation Addition (CAA), conflates two geometric effects: the direction (angle) of the hidden state with respect to the conceptual vector $s$, and the magnitude (norm) of the hidden state. Any steering vector can be decomposed as $w = r u$, with $r = \|w\|$ (radial component) and $u = w / \|w\|$ (unit direction), highlighting the entanglement of angular and radial effects (Aparin et al., 4 Jun 2026).

Spherical (unit-sphere) steering seeks to address this entanglement. For a given activation $x$ of norm $\mathbb{R}^d$0, one can set the steered state to have a prescribed alignment (cosine $\mathbb{R}^d$1) with a concept direction $\mathbb{R}^d$2:

$\mathbb{R}^d$3

where $\mathbb{R}^d$4 is a unit vector orthogonal to $\mathbb{R}^d$5, and norm is preserved. Spherical steering thereby separates angular alignment (conceptual control) from radial scaling (stability and model confidence).

2. Stiefel Manifold Generalization

The rank-1 frameworks above generalize to higher-rank interventions using the Stiefel manifold $\mathbb{R}^d$6, the set of $\mathbb{R}^d$7-tuples of orthonormal vectors in $\mathbb{R}^d$8. A steering frame $\mathbb{R}^d$9 comprises an orthonormal basis $w$0 ($w$1) and a diagonal or positive-definite scaling $w$2, supporting interventions that operate along multiple independent conceptual directions.

A steering update in this context can be cast as moving along a geodesic on $w$3, potentially via a matrix exponential in the tangent space, and combining this with a controlled change in the multi-radius $w$4. This geometric construction preserves interpretability while enabling multidimensional manipulation of activations (Aparin et al., 4 Jun 2026).

3. Inference-Time Stiefel Steering for Generation Diversity

Inference-time Stiefel activation steering operationalizes these geometric ideas for practical generative control. In the STARS algorithm, $w$5 parallel generation paths are steered by selecting $w$6 orthogonal perturbation vectors organized as columns of $w$7, with $w$8 (scaled Stiefel manifold). To promote diversity, the method maximizes the geometric volume spanned by the steered activations, formalized as maximizing $w$9, where $y = x + \alpha s$0 stacks the original (unsteered) activations (Zhu et al., 29 Jan 2026).

The optimization problem:

$y = x + \alpha s$1

can be solved by Riemannian gradient descent on $y = x + \alpha s$2. For efficiency, a closed-form one-step update is introduced, capturing $y = x + \alpha s$3 of the optimality gap while incurring only $y = x + \alpha s$4 the runtime of full Riemannian optimization. This enables real-time use at each generation step, with $y = x + \alpha s$5 typically in the range $y = x + \alpha s$6–$y = x + \alpha s$7 and only $y = x + \alpha s$8–$y = x + \alpha s$9 increase in inference latency.

Empirical results show that STARS achieves $x, s \in \mathbb{R}^d$0–$x, s \in \mathbb{R}^d$1 improvements in generative diversity and coverage metrics (e.g., code line coverage up to $x, s \in \mathbb{R}^d$2 vs $x, s \in \mathbb{R}^d$3 for nucleus sampling), without significant loss in core correctness, across a range of LLMs and benchmarks (Zhu et al., 29 Jan 2026).

4. Norm Preservation and Selective Steering

Norm-preserving variants of Stiefel activation steering, exemplified by Selective Steering, ensure intervention operators are unitary transformations—orthogonal matrices acting on the whole activation space—thus maintaining the statistical properties of activations across all layers (Dang et al., 27 Jan 2026). The method constructs a $x, s \in \mathbb{R}^d$4-D steering plane $x, s \in \mathbb{R}^d$5 spanned by orthonormal vectors $x, s \in \mathbb{R}^d$6 and applies a rotation of angle $x, s \in \mathbb{R}^d$7 via the matrix

$x, s \in \mathbb{R}^d$8

where $x, s \in \mathbb{R}^d$9 projects to the orthogonal complement of $s$0, $s$1 is the basis of $s$2, and $s$3 is a $s$4 rotation. This approach provably preserves $s$5 for any activation $s$6, with $s$7 itself constituting an element of the Stiefel manifold. In contrast, prior angular steering methods fail to preserve norm in general and can cause downstream distribution shifts or generation collapse, especially in smaller models (Dang et al., 27 Jan 2026).

5. Empirical and Algorithmic Trade-offs

Empirical analysis demonstrates that, for a wide range of LLMs and downstream tasks:

• Directional probes (on $s$8 or $s$9) achieve nearly identical classification accuracy, while probes on scalar norm $w = r u$0 remain at chance, confirming that concepts are predominantly encoded in angular structure, not norm.
• Allowing explicit norm scaling in spherical steering (via $w = r u$1) can reduce perplexity by up to $w = r u$2 at high $w = r u$3 with minimal downstream metric impact. Pure norm preservation, however, can degrade downstream accuracy and increase perplexity at high angular targets compared to approaches with relaxed norm constraints.
• In selective norm-preserving steering, applying interventions only at discriminative layers—where class means are sign-opposed—enables high attack success rates (up to $w = r u$4 prior methods) without introducing coherence loss or capability degradation (Dang et al., 27 Jan 2026).

A summary of intervention design options and observed trade-offs is given below:

Method	Angle Control	Norm Control	Trade-offs (Empirical)
CAA (raw)	Yes	Entangled	Norm inflation, less interpretable
Spherical (S)	Yes	Preserved	High PPL/cap drop at large angle
CAA-m	Yes	Free	Lower PPL/cap drop, norm varies
Selective	Yes	Preserved	Stable, layer-efficient, high controllability

6. Practical Applications and Computational Considerations

Stiefel activation steering is applied in both behavior alignment and generation diversity contexts. In STARS, the technique is used to encourage divergent generation paths through orthogonal steering, achieving significantly higher coverage in program synthesis and scientific idea generation without loss in syntactic or semantic quality. The low computational overhead of one-step Stiefel steering makes it suitable for real-time deployment in multi-path or ensemble inference settings (Zhu et al., 29 Jan 2026).

Norm-preserving Stiefel steering, as in Selective Steering, underpins robust adversarial controllability for alignment and refusal tasks by maintaining hidden-state distribution stability. Discriminative layer selection further optimizes both computational load (steering at $w = r u$5–$w = r u$6 layers out of $w = r u$7–$w = r u$8) and downstream performance (Dang et al., 27 Jan 2026).

7. Open Directions and Interpretability

While current research provides a detailed account for rank-1 (vector) interventions and proposes frameworks for higher-rank (multi-frame) Stiefel steering, the extension of angle–norm decomposition and interpretability guarantees to full $w = r u$9 interventions remains an open question. Notions such as per-frame angular scores and radius scaling compatible with multi-dimensional Stiefel geometry are currently areas for further development (Aparin et al., 4 Jun 2026). A plausible implication is that future systems may parameterize interventions by disentangled angular and radial controls over orthogonal subspaces, generalizing linear and spherical steering to higher-dimensional concept subspaces.

---

References:

• "A Geometric Account of Activation Steering through Angle-Norm Decomposition" (Aparin et al., 4 Jun 2026)
• "Exploring Diverse Generation Paths via Inference-time Stiefel Activation Steering" (Zhu et al., 29 Jan 2026)
• "Selective Steering: Norm-Preserving Control Through Discriminative Layer Selection" (Dang et al., 27 Jan 2026)