---
title: Adaptive Multi-Branch Steering (AMBS)
url: https://www.emergentmind.com/topics/adaptive-multi-branch-steering-ambs
type: topic
---

# Adaptive Multi-Branch Steering (AMBS)

Adaptive Multi-Branch Steering (AMBS), understood conceptually as adaptive branching over multiple steering pathways, denotes a class of large-language-model control methods that steer multiple attributes jointly while attempting to keep their representations disentangled and composable. In the formulation most closely associated with this description, "Multi-Subspace Representation Steering" (MSRS), the paper does **not** explicitly use the term “Adaptive Multi-Branch Steering,” but it is clearly a closely related multi-branch / multi-subspace steering formulation: it learns separate attribute branches in the form of private subspaces, adds a shared branch for common structure, and uses a learned mask to adaptively route steering at inference [2508.10599].

## 1. Problem setting and motivation

The overall goal is to control multiple LLM attributes **simultaneously** without the usual trade-offs and interference seen in prior steering methods. The problem addressed is that existing activation steering methods such as CAA or ITI are usually built for **one attribute at a time**. If they are combined naïvely for multiple attributes, the steering directions can conflict, which leads to attribute interference, degraded generation quality, reduced general capability, and trade-offs such as improving truthfulness while worsening bias or refusal.

Within this framing, AMBS is motivated by the need to steer multiple attributes jointly, but to do so in a way that keeps their representations disentangled and composable. MSRS realizes that objective by steering in **orthogonal subspaces**, learning a **shared subspace** for common directions and **private subspaces** for attribute-specific directions, then combining them adaptively. This suggests that the defining feature of AMBS is not merely simultaneous steering, but simultaneous steering with explicit structural separation and adaptive composition.

## 2. Multi-branch representation structure

MSRS builds on the ReFT-style idea of editing hidden representations, but generalizes it to a multi-attribute, subspace-based design. For a hidden representation \(h \in \mathbb{R}^d\), ReFT uses

\[
\Phi(h; R, W, b) = h + R^{\top} \left(W h + b - R h\right),
\]

where \(R \in \mathbb{R}^{r \times d}\) is a low-rank projection matrix, \(R R^\top = I_r\) is the orthonormality constraint, and \(W \in \mathbb{R}^{r \times d}\), \(b \in \mathbb{R}^r\) are learned parameters. In this formulation, the model maps the representation into a low-dimensional subspace, transforms it there, and maps it back.

The multi-branch extension decomposes the steering space into one **shared subspace** \(B_{\text{shared}}\) and multiple **attribute-specific private subspaces** \(B_i\). The key idea is that each attribute gets its own orthogonal branch, but all attributes can also share some latent directions that encode common structure. In AMBS terms, these components correspond to a shared branch plus multiple private branches, rather than a single undifferentiated steering direction.

## 3. Hybrid shared-private subspace composition

The hybrid composition strategy is the core of the method. For each attribute \(i\), the mean activation from its dataset \(\mathcal{D}_i\) is computed as

\[
\tau_i = \frac{1}{|\mathcal{D}_i|} \sum_{j=1}^{|\mathcal{D}_i|} h_{i,j}^l.
\]

The method uses the **last token hidden state** at layer \(l\), because it aggregates the full prompt context. All attribute means are then concatenated:

\[
\tau_c = [\tau_1 \; | \; \tau_2 \; | \; \dots \; | \; \tau_n] \in \mathbb{R}^{d \times n}.
\]

To extract the shared subspace, the method performs SVD on the concatenated matrix,

\[
\tau_c = U_c \Sigma_c V_c^\top,
\]

and chooses the smallest rank \(r_s\) whose cumulative singular-value energy reaches at least 90% of total energy. The shared subspace is defined by the top singular vectors,

\[
B_{\text{shared}} = V_{c,1:r_s}^{\top} \in \mathbb{R}^{r_s \times d}.
\]

For each attribute \(i\), the shared component is removed:

\[
H_{\text{res}}^{(i)} = \tau_i - B_{\text{shared}}^{\top} B_{\text{shared}} \tau_i.
\]

SVD is then applied to the residual,

\[
H_{\text{res}}^{(i)} = U^{(i)} S^{(i)} V^{(i)\top},
\]

and the smallest rank \(r_i\) whose singular values explain at least 90% of the residual energy is retained:

\[
B_i = \left( V^{(i)}_{1:r_i} \right)^\top \in \mathbb{R}^{r_i \times d}.
\]

The full alignment matrix is the concatenation of all subspace bases:

\[
S_{\text{align}} = [B_{\text{shared}}, B_1, B_2, \dots, B_n] \in \mathbb{R}^{(r_s + \sum_i r_i)\times d}.
\]

Conceptually, this yields a **hybrid steering basis** in which the shared branch represents common directions and the private branches represent attribute-specific directions. A plausible implication is that AMBS can be understood as a structured steering basis rather than a single control vector.

## 4. Adaptive weighting and training objective

MSRS does not only concatenate subspaces; it learns how much to use each subspace dimension dynamically. The method introduces a mask network,

\[
m(h) = \text{sigmoid}(\text{MLP}(h)) \in [0,1]^r,
\]

which provides a learned weight for each subspace dimension. The steering function becomes

\[
\Phi_{l,p}(h; R, W, b, m) = h + R^\top \text{diag}(m(h)) (W h + b - R h).
\]

In this expression, \(R\) is the learned subspace basis, \(W h + b - R h\) is the latent correction, and \(\text{diag}(m(h))\) scales each dimension by its relevance. The adaptive aspect of the method therefore resides in gating or masking over the latent steering basis.

The method also defines a binary prior mask \(m_{\text{prior}} \in \{0,1\}^r\), where entries corresponding to the shared subspace and the selected attribute-specific subspace are 1, and others are 0. Mask regularization is

\[
\mathcal{L}_{\text{reg}} = \left\| m(h) - m_{\text{prior}} \right\|_2^2.
\]

To align the learned subspace basis with the SVD-derived structure, the method uses

\[
\mathcal{L}_{\text{align}} = 1 - \frac{\langle R, S_{\text{align}} \rangle}{\|R\|_2 \|S_{\text{align}}\|_2}.
\]

The total training loss is

\[
\mathcal{L} = \mathcal{L}_{\text{task}} + \lambda_1 \mathcal{L}_{\text{reg}} + \lambda_2 \mathcal{L}_{\text{align}},
\]

where \(\lambda_1, \lambda_2 > 0\). This is the mechanism by which the method learns a dynamic, structured, multi-branch steering representation.

## 5. Token-level intervention and inference-time routing

A defining feature of the method is that it does not always steer the final token only. Instead, it uses a **dynamic intervention position selection** strategy. The stated reason is that fixed-position steering, especially always using the last token, can increase interference, because different attributes may be most salient at different positions.

For an input sequence with token representations \(h_1, \dots, h_T\), and for each attribute \(i\), each token is projected onto the attribute’s subspace \(R_i\):

\[
\text{proj}_{R_i}(h_t) = R_i^\top R_i h_t.
\]

A relevance score is then computed:

\[
s_{i,t} = \left\| \text{proj}_{R_i}(h_t) \right\|_2.
\]

The intervention position is selected as

\[
p_i = \arg\max_{t \in \{1,\dots,T\}} s_{i,t}.
\]

The chosen token is therefore the one whose representation most strongly aligns with the attribute-specific subspace, and the steering function \(\Phi_{l,p}\) is applied at token position \(p_i\), rather than always at the last token. In the ablation tables, this is called the **Important Token** strategy, and it consistently beats **Last Token**. This suggests that AMBS involves not only multiple branches and adaptive weighting, but also adaptive selection of where in the sequence steering should occur.

## 6. Interference reduction and empirical profile

The method reduces inter-attribute interference through three mechanisms. First, each attribute is assigned a separate private subspace \(B_i\), which prevents all attributes from being forced into the same low-rank space, described as a major source of conflict in vanilla ReFT-style steering. Second, the method includes a shared subspace \(B_{\text{shared}}\), which helps preserve common steering directions and prevents the system from fragmenting too aggressively. Third, the mask \(m(h)\) suppresses irrelevant dimensions, and dynamic token selection chooses the token most aligned to the target attribute, rather than always steering the same position. Together, these design choices isolate attribute-specific steering, preserve shared semantics, and apply interventions where they matter most.

The reported evaluations cover multiple attribute pairs and models. The attribute settings are **TruthfulQA + BBQ** for truthfulness vs bias, **Alpaca + Refusal** for instruction following vs refusal, and **HelpSteer** for helpfulness, coherence, verbosity, together with the general benchmarks **HellaSwag, RACE, MMLU, OpenBookQA, GLUE**. The models are **Llama2-7B**, **Llama3-8B-Instruct**, **Qwen2-7B-Instruct**, and **Mistral-7B-v0.3**. The baselines are **ICL**, **CAA**, **ITI**, **ReFT**, **MTL-LoRA**, and **MAT-STEER** [2508.10599].

In the main multi-attribute results, the method is reported as consistently stronger at balancing conflicting attributes. For **Llama3-8B-Instruct**, the table gives **TruthfulQA MC2: 56.32** vs ReFT 49.51 and MAT-STEER 49.67; **BBQ: 0.645** vs ReFT 0.637, CAA 0.629, ITI 0.612; **Alpaca win rate: 0.36** vs ReFT 0.30; **Refusal: 0.529** vs CAA 0.493 and ReFT 0.451; and **HelpSteer helpfulness/coherence: 3.89 / 3.96**. For **Qwen2-7B-Instruct**, the table gives **TruthfulQA MC2: 53.27**, **BBQ: 0.642**, **Alpaca: 0.45**, **Refusal: 0.445**, and HelpSteer with good balance across helpfulness/coherence/verbosity. For **Mistral-7B-v0.3**, the table gives **TruthfulQA MC2: 52.62**, **BBQ: 0.644**, **Refusal: 0.693**, and **HelpSteer coherence: 3.93**.

The baseline trade-offs are characterized explicitly. **ITI** often boosts truthfulness but hurts bias/refusal/general behavior. **CAA** can improve one dimension but often causes trade-offs. **ReFT** is better than pure activation steering but still limited in multi-attribute settings. **MAT-STEER** uses orthogonality but still lacks the shared-subspace mechanism and adaptive composition needed for robust multi-attribute control. On general capability preservation, the reported **GLUE average** scores are **0.7748** for Llama3-8B-Instruct, above ReFT (**0.7569**) and CAA (**0.7384**), **0.8322** for Qwen2-7B-Instruct, and **0.7066** for Mistral-7B-v0.3**. The paper also reports MMLU and HellaSwag gains, suggesting that the method does not merely overfit steering attributes but preserves or improves broader language capabilities. The ablation results state that **Same Space** is worst due to interference; **MSRS\(_{\text{Attribute}}\)** and **MSRS\(_{\text{Rank}}\)** both outperform Same Space; attribute-based grouping is often the best trade-off, while rank-level weighting can be slightly better on some models but not universally; and **Important Token** beats **Last Token** across models and tasks.

## 7. Relation between AMBS and MSRS

The paper explicitly states that it does **not** use the term “Adaptive Multi-Branch Steering (AMBS).” The relationship is therefore conceptual rather than terminological. If AMBS is understood as a general concept of **adaptive multi-branch steering**, MSRS can be viewed as a concrete realization of that idea in the representation or subspace domain [2508.10599].

The distinction is important. **AMBS**, in this interpretation, denotes adaptive branching over multiple steering pathways. **MSRS**, as the paper’s actual method, denotes adaptive multi-subspace representation steering with shared and private subspaces together with token-level dynamic intervention. It is therefore not merely a synonym for AMBS; rather, it is a specific, technically grounded instance of adaptive multi-branch-like steering.

A common misconception would be to treat the terminology as interchangeable. The available evidence supports a narrower formulation: the paper presents MSRS as the explicit method, while AMBS functions as a higher-level descriptive label. Under that reading, the central characteristics of AMBS are the extraction of a **shared subspace** plus **attribute-specific orthogonal subspaces** via SVD, the learning of a **dynamic mask** to weight those subspaces, the application of steering at the **most relevant token** instead of a fixed position, and optimization with both task loss and subspace-alignment regularizers.

Source: https://www.emergentmind.com/topics/adaptive-multi-branch-steering-ambs