MAT-Steer: Fine Token-Level Steering

• The paper introduces MAT-Steer, a unified framework for per-token control over multiple generative attributes.
• It leverages explicit token insertions, activation-space interventions, and selective masking to manage conflicting attribute cues.
• Empirical evaluations demonstrate significant gains in compositionality, alignment, and disentanglement across varied generative settings.

Selective Token-level Multi-Attribute Steering (MAT-Steer) is a unified paradigm for fine-grained, multi-objective behavioral control in deep generative models, including diffusion transformers for multimodal tasks and LLMs. The MAT-Steer framework enables per-token or per-block steering toward multiple, potentially conflicting, attributes by means of explicit token insertions, activation-space interventions, attention masking, or learned subspaces. This mechanism provides scalable, architecture-agnostic, and empirical gains in compositionality, alignment, and attribute disentanglement across diverse generative settings (Zhang et al., 7 Feb 2026, Nguyen et al., 18 Feb 2025, Radevski et al., 8 Jan 2026, Herbster et al., 9 Apr 2026, Jiang et al., 14 Aug 2025).

1. Foundations and Problem Definition

MAT-Steer addresses the need for controlled generation aligned with multiple, often heterogeneous, user-defined or task-centric attributes. Attributes may include style, content, subject, identity, correctness, helpfulness, coercion, toxicity, and more. The challenge is to enable models to interpret and synthesize results from multiple reference sources or requirements without interference or uncontrolled blending of attributes.

Core to selective token-level MAT-Steer is the introduction of explicit per-attribute "steering"—which may take the form of learnable tokens (Zhang et al., 7 Feb 2026, Radevski et al., 8 Jan 2026), attribute-conditioned activation offsets (Nguyen et al., 18 Feb 2025, Herbster et al., 9 Apr 2026, Jiang et al., 14 Aug 2025), specialized attention masks (Zhang et al., 7 Feb 2026), or reinforcement learning over sparse feature spaces (Cho et al., 11 Feb 2026). The framework is designed both for inference-time intervention (runtime control without model retraining) and as a post-training adaptation layer atop frozen or unified generative backbones.

2. Selective Multi-Attribute Tokens and Steering Mechanisms

Attribute Tokens in Diffusion/Multimodal Models

In diffusion transformers, such as SIGMA (Zhang et al., 7 Feb 2026), a small vocabulary of attribute tokens $$T = \{\langle \mathrm{Style} \rangle, \langle \mathrm{Content} \rangle, \langle \mathrm{Subject} \rangle, \langle \mathrm{Identity} \rangle, \langle \mathrm{Layout} \rangle, \dots \}$$ labels each reference, e.g., marking brushstroke style, facial identity, object content, or selection target. The attribute token $\tau_i$ is prepended to each image block, and the following image embedding is combined with a learnable per-attribute vector: $t_i = v_i + W_\tau(\tau_i)$ where $v_i$ is the patch embedding of image $i$ and $W_\tau$ is the embedding table indexed by $\tau_i$. This enforces that only $\tau_i$-marked subspaces are used when conditioning the diffusion process, selectively injecting attribute-specific information.

Compositional Steering Tokens in LLMs

For LLMs, MAT-Steer can be instantiated using dedicated behavior tokens $t_i$ for each attribute, with a learned composition token $t_{(\mathrm{comp})}$ ("<and>") to signal composition (Radevski et al., 8 Jan 2026). The input embedding sequence is constructed by interleaving these tokens with the prompt: $\tau_i$0 Steering tokens are trained via self-distillation from natural language instructions, and the composition token is regularized for orthogonality to ensure clean attribute disentanglement.

Activation and Subspace-based Interventions

MAT-Steer can also be realized through selective, layerwise activation-space steering (Nguyen et al., 18 Feb 2025, Jiang et al., 14 Aug 2025, Herbster et al., 9 Apr 2026). Here, for each token representation $\tau_i$1 and attribute $\tau_i$2, a gate $\tau_i$3 determines applicability: $\tau_i$4 where steering vectors $\tau_i$5 are learned per attribute with constraints for sparsity and orthogonality. Gates $\tau_i$6 are parameterized via sigmoid functions of the activations, allowing only relevant tokens to be steered. Normalized activations ensure that the magnitude of interventions does not distort model behavior.

In MSRS (Jiang et al., 14 Aug 2025), multi-attribute steering proceeds via orthogonal subspaces for each attribute—constructed by SVD on activation means—alongside a shared subspace. A dynamic weighting mask selects which subspace directions are engaged at each token.

3. Mathematical Formulation and Regularization

The fundamental mathematical structure of MAT-Steer involves:

• Attribute-wise steering vectors: Each attribute $\tau_i$7 is assigned $\tau_i$8 (or subspace $\tau_i$9) that, when added to or projected onto a token's hidden state, pushes its representation toward the "positive" (desirable) region of the attribute.
• Selective gating functions: $t_i = v_i + W_\tau(\tau_i)$0, where $t_i = v_i + W_\tau(\tau_i)$1 are learnable parameters, provide soft control over which tokens are modified.
• Multi-objective loss formulation: Learning is guided by a combination of loss functions:
  • Maximum Mean Discrepancy (MMD) to align steered "negative" activations with the positive distribution,
  • Preservation ($t_i = v_i + W_\tau(\tau_i)$2) and sparsity ($t_i = v_i + W_\tau(\tau_i)$3) regularization to minimize unnecessary or harmful interventions,
  • Orthogonality losses $t_i = v_i + W_\tau(\tau_i)$4 to decouple attribute effects.
• Normalization of activations: To prevent overamplification, the output representation is renormalized.

For compositional steering tokens (Radevski et al., 8 Jan 2026), training employs a temperature-scaled KL divergence between teacher and student outputs, with an additional orthogonality penalty on the composition token.

4. Architectural Modifications and Attention Masking

In unified diffusion transformers, SIGMA (Zhang et al., 7 Feb 2026) integrates attribute token embeddings and applies a group-scoped binary attention mask $t_i = v_i + W_\tau(\tau_i)$5 at every self- and cross-attention layer: $t_i = v_i + W_\tau(\tau_i)$6 where $t_i = v_i + W_\tau(\tau_i)$7 is a causal mask, $t_i = v_i + W_\tau(\tau_i)$8 is the intra-image mask, and $t_i = v_i + W_\tau(\tau_i)$9 restricts group-wise attention. This ensures that reference images only steer the attributes they are tagged for, blocking off-target flow of information between attribute groups.

For LLMs, MAT-Steer is typically agnostic to transformer architecture, requiring only minimal additions (steering tokens to the embedding layer, or activation adjustment at the desired layer) (Radevski et al., 8 Jan 2026, Nguyen et al., 18 Feb 2025).

MSRS further introduces a two-layer dynamic mask network $v_i$0 to gate the use of orthogonal steering subspaces per attribute, allowing precise and context-dependent selective control (Jiang et al., 14 Aug 2025).

5. Training Procedures and Experimental Protocols

Diffusion and Multimodal

Post-training is performed on large corpora of interleaved multi-attribute sequences comprising up to 700,000 examples spanning compositional generation, selective extraction, stylization, relation transfer, editing, and layout (Zhang et al., 7 Feb 2026). The optimization objective remains the denoising loss, but with multi-attribute token-conditioned input sequences.

LLMs

For compositional steering tokens (Radevski et al., 8 Jan 2026), attribute tokens and the composition token are distilled on automatically verifiable behaviors and structures. Experimental setups systematically evaluate seen, unseen, and compositional generalization.

For activation steering (Nguyen et al., 18 Feb 2025, Jiang et al., 14 Aug 2025), labeled datasets are split into positive and negative samples per attribute. Steering vectors, gates, and masks are learned by minimizing MMD-based objectives with auxiliary regularization. Careful ablations confirm the necessity of orthogonality, sparsity, and normalization.

6. Empirical Evaluations and Analysis

Extensive experimental analysis demonstrates that MAT-Steer leads to:

• Superior compositionality: In SIGMA, compositional generation, selective attribute transfer, and layout guidance show substantial gains over prior unified baselines (Bagel), including improvements of +7.64 to +26.42 points on CLIP and DINO metrics (Zhang et al., 7 Feb 2026). Visual outputs are more faithful to each attribute-labeled reference, with reduced off-target feature leakage (CLIP-ES).
• Attribute fidelity and reduced interference: In LLM settings, MAT-Steer outperforms parameter-efficient finetuning and alternative ITI approaches, achieving an average QA accuracy increase of 3.3% over the best existing baselines (Nguyen et al., 18 Feb 2025). Orthogonality regularization and token-level sparsity lead to reduced destructive interference—ablations show 4–5% drops in QA accuracy when these are removed.
• Generalization and compositional robustness: MAT-Steer tokens enable zero-/few-shot generalization to unseen attribute combinations, including 3-way behaviors, with accuracy and robustness not achievable by baseline activation or adapter methods (Radevski et al., 8 Jan 2026).
• Cross-condition consistency: Attention masking and token-specific interventions maintain high subject identity and attribute alignment even in the presence of 4–6 interleaved references (Zhang et al., 7 Feb 2026).
• Scalability and efficiency: MAT-Steer requires only lightweight, post-hoc updates to embedding tables or gating functions, achieving state-of-the-art controllability with minimal computational and data cost.

7. Limitations and Extensions

Limitations of MAT-Steer span several axes:

• Attribute scaling: Extremely dense sets of reference attributes can challenge current formulations, degrading spatial coherence or leading to residual attribute leakage (Zhang et al., 7 Feb 2026).
• Token selection granularity: While selective gating and masking are effective, thresholding and context-dependent token relevance require further optimization (Jiang et al., 14 Aug 2025).
• Scope of attributes: While MAT-Steer is validated on a range of objective and subjective attributes (style, bias, toxicity, helpfulness, formatting), generalization to highly subjective or adversarial traits may require more nuanced steering mechanisms and data (Nguyen et al., 18 Feb 2025, Herbster et al., 9 Apr 2026).
• Model accessibility: Activation-space steering methods require white-box access to model activations.
• Potential for dual-use: Malicious or misaligned steering vectors can degrade performance or be exploited for adversarial purposes if not secured (Herbster et al., 9 Apr 2026).

Future directions include parameter-efficient LoRA-style tuning for dynamically extending the attribute vocabulary, video-rate and spatio-temporal masking, dynamic token-type discovery, and hybridization with new concept subspace extraction modalities (Zhang et al., 7 Feb 2026, Jiang et al., 14 Aug 2025).

Summary Table: Key Instantiations of MAT-Steer

Paper (arXiv ID)	Model/Modality	Steering Method
(Zhang et al., 7 Feb 2026)	Diffusion transformer	Attribute tokens + attention masking
(Radevski et al., 8 Jan 2026)	LLM	Steering tokens + composition token
(Nguyen et al., 18 Feb 2025)	LLM	Gated, per-token activation steering
(Herbster et al., 9 Apr 2026)	LLM	Projection-aware activation steering
(Jiang et al., 14 Aug 2025)	LLM	Multi-subspace orthogonal activation steering

Selective Token-level Multi-Attribute Steering constitutes an effective, rigorously validated methodology for modular, compositional, and fine-grained control in both vision and language generation, establishing new state of the art across multiple benchmarks and domains.