Spectral Editing Key Amplification (SEKA)

• SEKA is a training-free method enabling precise attention steering in transformers by editing key embeddings via spectral decomposition for focused prompt highlighting.
• It leverages low-rank projectors derived from SVD to amplify relevant and suppress irrelevant token contributions, maintaining compatibility with memory-efficient kernels like FlashAttention.
• Adaptive SEKA (AdaSEKA) extends this approach with query-adaptive multi-expert routing to dynamically tailor attention steering across various semantic tasks.

Spectral Editing Key Amplification (SEKA) is a training-free method for attention steering in transformer architectures. SEKA intervenes directly in the key embedding space using spectral decomposition techniques, allowing for low-rank, geometrically interpretable control of attention focus without modifying model weights or requiring full-matrix attention computation. This approach is particularly compatible with memory-efficient attention kernels such as FlashAttention. An adaptive extension, AdaSEKA, further introduces dynamic, query-adaptive multi-expert routing for context-specific highlighting and robust control across multiple semantic tasks (Li et al., 1 Mar 2026).

1. Motivation and Core Principles

SEKA is designed to enable efficient, precise, and geometrically grounded steering of transformer attention in scenarios such as prompt highlighting, where model focus must be directed to user-specified tokens. The principal motivations for SEKA are:

• Training-Free Steering: SEKA alters only key embeddings at inference time, avoiding fine-tuning or parameter addition.
• Pre-Attention Intervention: It modifies the key vectors before any softmax or scaling steps, ensuring no need to instantiate the full attention matrix, which is essential for memory-efficient kernels (e.g., FlashAttention).
• Low-Rank Control: Empirical analysis reveals that changes in token relevance under different prompts concentrate in a low-dimensional subspace of key embeddings. SEKA exploits spectral decomposition to target those directions.
• Model Compatibility: The method operates solely on query (Q) and key (K) embeddings, leaving all downstream (value, MLP) computations unchanged and invariant to the original model’s behavior except for the specific effect of attention steering.

2. Mathematical Foundations

The SEKA method is grounded in spectral analysis of key embedding dynamics across “neutral,” “relevant,” and “irrelevant” prompt conditions. The main steps are as follows:

• Cross-Covariance and SVD: For each layer $l$ and attention head $h$, three sets of key embeddings $(h^n, h^+, h^-)$ are collected under neutral, relevant, and irrelevant prompts, respectively. Cross-covariance matrices $\Sigma^+ = (h^n)^\top h^+$ and $\Sigma^- = (h^n)^\top h^-$, both in $\mathbb{R}^{d \times d}$, are computed and decomposed via singular value decomposition (SVD).
• Rank Selection Criteria: Using a variance retention threshold $\gamma \in (0,1)$, the smallest $k_+$ (largest singular values of $S^+$) and $k_-$ (smallest singular values of $S^-$) are selected such that respective cumulative singular values exceed $\gamma$ times the total.
• Projector Construction: Two projectors are defined: $P^+ = U^+_{:,1:k_+}(U^+_{:,1:k_+})^\top$ (relevant directions) and $P^- = U^-_{:,d-k_-+1:d}(U^-_{:,d-k_-+1:d})^\top$ (irrelevant directions).
• Key Editing Operation: At inference, the key vector $k$ is edited as $k' = k + g^+ P^+ k + g^- P^- k$, where $g^+, g^-$ are scalar gains (typically $g^- \leq 0$ to suppress irrelevant directions). The altered key vector $k'$ leads to amplification or suppression of attention logits, geometrically interpreted as boosting the relevant component while controlling the irrelevant one.

3. Algorithmic Implementation and FlashAttention Compatibility

Offline Preparation

• Assemble a small set (approximately 50–200) of contrastive synthetic prompts, generating $(h^n, h^+, h^-)$ triplets for each layer-head pair.
• Compute $\Sigma^+$, $\Sigma^-$ and their SVDs, determine $k_+$ and $k_-$ according to $\gamma$, and construct $P^+_{l,h}$, $P^-_{l,h}$.
• Evaluate each head’s average $\| h^+ - h^- \|_2$; retain only those exceeding a threshold $\delta_{\min}$, restricting steering to empirically relevance-sensitive heads.

Inference Workflow

• At each attention call in layer $l$ and head $h$, intercept the key tensor $K \in \mathbb{R}^{B\times T\times d}$ and a binary mask $H \in \{0,1\}^T$ that marks highlighted tokens.
• For each position $t$ with $H[t]=1$, apply the editing: $$K[b, t, :] \leftarrow K[b, t, :] + g^+ P^+_{l,h} K[b, t, :] + g^- P^-_{l,h} K[b, t, :]$$.
• Pass the edited $K$ to FlashAttention. As only selected key vectors are edited and no full attention matrix is constructed, SEKA maintains full compatibility and efficiency with memory-optimized attention implementations.

4. Adaptive SEKA: Query-Adaptive Multi-Expert Extension

AdaSEKA generalizes SEKA for scenarios demanding semantic adaptivity, introducing a modular “expert bank.” Each expert learns a distinct relevance subspace from a specific task (e.g., factual recall, instruction following).

Procedure Overview

• Offline: For each expert $m = 1\ldots M$, independently replicate the SEKA SVD routine to obtain $U^+_{m,l,h}$, $S^+_{m,l,h}$.
• Inference:

1. Extract the current query vector $q_{l,h}$ at each head.
2. Calculate alignment scores $a_m^{(l,h)}$ as the normalized sum of $q_{l,h}$ projections onto the expert’s top $K$ singular vectors, weighted by singular values.
3. Construct the dynamic projector: $$P_\text{dyn}^{(l,h)} = \sum_{m=1}^M a_m^{(l,h)}\, U^+_{m,l,h,:,1:K} (U^+_{m,l,h,:,1:K})^\top$$.
4. Modify each highlighted key as $k' = k + g P_\text{dyn}^{(l,h)} k$.

SEKA is a special case of AdaSEKA with $M=1$ and $a_1 = 1$. The method flexibly routes among expert subspaces, enabling contextually aware attention steering without supervision or manual task assignment.

5. Empirical Benchmarks and Performance

Comprehensive benchmarking on Qwen3-4B and Qwen3-8B, using FlashAttention, demonstrates the efficiency and effectiveness of SEKA and AdaSEKA relative to strong baselines such as PASTA. Evaluation metrics and results include:

Benchmark	Metric	SEKA	AdaSEKA (if different)	PASTA
CounterFact	ES	99.02%	—	97.16%
CounterFact	PS	98.61%	—	96.03%
Bias in Bios	Top-1 Accuracy	91.02%	—	89.58%
Pronouns Changing	P.Score	95.18%	Up to 99.52%	95.82%
Lost-in-the-Middle	Mid-context EM	≈0.60	—	≈0.20

• SEKA achieves near-perfect Efficacy and Paraphrase Scores on CounterFact and strong results on Bias in Bios.
• AdaSEKA’s multi-expert mechanism yields pronounced gains, notably in the Pronouns Changing benchmark (up to 99.52%).
• Overhead remains minimal: SEKA incurs +0.03 s/sample and +0.03 GB memory. PASTA imposes +1.03 s/sample and +23 GB memory, precluding FlashAttention compatibility. AdaSEKA adds ≈+0.27 s/sample.

Ablation studies confirm the necessity of learned spectral projectors and head selection. Random projections decrease performance by 5–10 points, and the elimination of SVD and filtering steps leads to catastrophic performance collapse (e.g., Pronoun A.P.Score drops from 90.52 to 36.95).

6. Practical Guidance for Deployment

For effective integration of SEKA and AdaSEKA, several practitioner recommendations arise:

• Hyperparameter Selection:
  • Variance threshold $\gamma$: 0.85–0.99
  • Head-selection $\delta_{\min}$: 0.10–0.25
  • Gain $g^+$: 0.2–2.5 (typically $g^-$ near zero)
  • AdaSEKA requires only $\delta_{\min}$ and one gain $g$
• Data Requirements: 50–100 synthetic contrastive samples are generally sufficient to compute stable projections; additional data primarily reduces variance, not peak accuracy.
• Integration Steps:

1. Precompute and store $P^+$ and $P^-$ for each selected $(l,h)$.
2. Register a hook at key vector construction or as input to FlashAttention.
3. Mask and edit only the key vectors for highlighted tokens during generation.
4. For long-context workloads, steer only in mid-to-late layers and for heads meeting $\delta_{\min}$ to prevent excessive amplification.
5. Multi-task settings benefit from AdaSEKA’s adaptive expert routing without manual subspace assignment.

SEKA thus provides an immediately deployable, interpretable, and computationally efficient paradigm for attention steering in contemporary LLMs, especially where hardware and latency constraints preclude the use of full-rank or retraining-based post-hoc steering methods (Li et al., 1 Mar 2026).