Training-Free Value Vector Methods

• Training-free value vector methods are techniques that manipulate transformer value vectors using pre-trained activations, bypassing gradient-based retraining.
• They enable practical applications in image editing, unlearning, multimodal retrieval, and truthfulness detection by leveraging forward-pass activation statistics.
• Their efficiency and interpretability allow scalable, zero-shot control over model behavior across diverse modalities and tasks.

Training-free value vector methods are a class of approaches that intervene directly in the internal value representations of deep learned models, leveraging pre-trained weights and architecture-induced semantics without any gradient-based retraining. Unlike parameter-efficient tuning or classical feature probing, these methods modify or extract model behavior by manipulating, aggregating, or contextualizing value vectors within transformer architectures via either architectural or prompt-level controls. The term “value vector” here refers to the output memory slots of either attention or multilayer perceptron (MLP/FFN) modules, which encode critical information for aggregation or downstream prediction. Training-free value vector approaches have achieved competitive results in domains including image editing, unlearning, multimodal retrieval, and truthfulness detection.

1. Conceptual Foundations of Value Vector Manipulation

In transformer-based models, value vectors are key constituents of both attention and MLP submodules. In multi-modal attention, values $V$ determine the content to be aggregated, while in MLP “key-value memories,” output states are constructed as linear combinations of learned vectors. The pronounced bias–delta structure—where token representations cluster around a layer bias with informative deviations—enables interpretable and systematic manipulation (Li et al., 20 Feb 2026).

Key distinctions that underpin the training-free paradigm include:

• Non-gradient-based intervention: All operations rely on pre-trained activations and forward passes, with no parameter updates through backpropagation (Kim et al., 29 Jan 2026).
• Activation and similarity statistics: Selection and control are based on the statistical patterns of value activations or their contributions to output, often leveraging calibration or prompt engineering (Liu et al., 22 Sep 2025, Kim et al., 18 Apr 2026).
• Dual-channel abstraction: Some frameworks explicitly disentangle routing from feature aggregation, e.g., modulating the “where” (Key) and the “what” (Value) axes in transformer attention (Li et al., 20 Feb 2026).

2. Principal Frameworks

Dual-Channel Attention Guidance (DCAG)

DCAG generalizes attention-based editing in Diffusion Transformers (DiT) by treating both Key and Value channels as independently scalable axes of control. Every multi-modal attention layer for image tokens in DiT is decomposed as:

$$K^i = \bar{K} + \Delta K^i,\quad V^i = \bar{V} + \Delta V^i$$

with $\hat{K}^i = \bar{K} + \delta_k \Delta K^i$, and $\hat{V}^i = \bar{V} + \delta_v \Delta V^i$, where $\delta_k, \delta_v$ are user-chosen “delta-scales.” Scaling $\delta_k$ nonlinearly influences attention weights, while $\delta_v$ produces linear changes in feature aggregation.

Coordinated tuning of $(\delta_k, \delta_v)$ exposes a two-dimensional fidelity–editability trade-off space, outperforming prior Key-only methods. For example, on the PIE-Bench benchmark, $(\delta_k, \delta_v) = (1.10, 1.15)$ achieves a 1.8% reduction in LPIPS over Key-only, with up to 4.3% gain for localized object deletions (Li et al., 20 Feb 2026).

mEOL: Instruction-Guided Multimodal Embedding

The mEOL method extracts a value vector as a compact semantic embedding from a large multimodal LLM (MLLM) by prompt-crafting: any input (text, image, SVG code) is summarized into a single token hidden state via a prompt such as “This X [instruction] in one word:”. The resulting embedding, taken from the penultimate model layer of the generated token, aligns structured (SVG), visual, and textual modalities without training additional heads. Enhanced by a semantic SVG rewriting module that assigns perceptual IDs to vector primitives, mEOL achieves zero-shot retrieval performance surpassing learned multimodal encoders on VGBench (Kim et al., 18 Apr 2026).

Knowledge Vector Weakening (KVW) for Unlearning

KVW directly intervenes on the value vectors (“knowledge vectors”) in the FFN submodules of vision–language transformers to forcibly remove specific knowledge. By accumulating usage statistics for vectors during forward passes over “forget” and “retain” data, KVW exponentially attenuates those vectors that are disproportionately active for forbidden knowledge:

$$\mathcal{A}_i = \max(0, \log(\mathcal{C}_f / \mathcal{C}_r))\qquad \tilde{\mathbf{v}}_i = e^{-\gamma \mathcal{A}_i} \mathbf{v}_i$$

Here, $$K^i = \bar{K} + \Delta K^i,\quad V^i = \bar{V} + \Delta V^i$$0 is a global weakening strength, and $$K^i = \bar{K} + \Delta K^i,\quad V^i = \bar{V} + \Delta V^i$$1 are the average coefficients for forget and retain sets. This allows efficient, rank-agnostic, training-free “unlearning” of sensitive content, yielding a Pareto-optimal forget–retain trade-off at a fraction of the computational cost of gradient-based or LoRA-based approaches (Kim et al., 29 Jan 2026).

TruthV: Truthfulness Detection Using Value Vectors

TruthV leverages the per-candidate dynamics of MLP value-vector activations in LLMs for truthfulness detection in multiple-choice QA. For each candidate, activations $$K^i = \bar{K} + \Delta K^i,\quad V^i = \bar{V} + \Delta V^i$$2 are computed for all MLP value vectors. Top vectors are selected based on their calibration-set accuracy in predicting the ground-truth candidate via argmax or argmin patterns. At inference, an ensemble of these value vectors votes for the most likely truthful answer by ranking $$K^i = \bar{K} + \Delta K^i,\quad V^i = \bar{V} + \Delta V^i$$3 activations, consistently outperforming attention-only (NoVo) and log-likelihood baselines by 8–10% accuracy on the NoVo benchmark (Liu et al., 22 Sep 2025).

3. Algorithmic Implementations and Statistical Paradigms

All training-free value vector approaches share a reliance on forward-pass statistics and structural dissection of model internals.

• Pseudocode for DCAG Value Intervention (Li et al., 20 Feb 2026): $\hat{K}^i = \bar{K} + \delta_k \Delta K^i$7
• mEOL Embedding Extraction (Kim et al., 18 Apr 2026): $\hat{K}^i = \bar{K} + \delta_k \Delta K^i$8
• KVW Knowledge Attenuation (Kim et al., 29 Jan 2026):
  1. Compute average knowledge coefficients $$K^i = \bar{K} + \Delta K^i,\quad V^i = \bar{V} + \Delta V^i$$4
  2. For each value vector, compute $$K^i = \bar{K} + \Delta K^i,\quad V^i = \bar{V} + \Delta V^i$$5 and apply $$K^i = \bar{K} + \Delta K^i,\quad V^i = \bar{V} + \Delta V^i$$6
  3. Scale $$K^i = \bar{K} + \Delta K^i,\quad V^i = \bar{V} + \Delta V^i$$7 in FFN layers

• TruthV Voting Ensemble (Liu et al., 22 Sep 2025):

  1. For each MCQ candidate, compute value activations $$K^i = \bar{K} + \Delta K^i,\quad V^i = \bar{V} + \Delta V^i$$8.
  2. Preselect top $$K^i = \bar{K} + \Delta K^i,\quad V^i = \bar{V} + \Delta V^i$$9 of vectors by argmax/argmin accuracy on a calibration set.
  3. For inference, each selected vector votes for the candidate with the highest/lowest $\hat{K}^i = \bar{K} + \delta_k \Delta K^i$0; answers with the majority vote are selected.

4. Domains of Application and Empirical Performance

Method	Primary Domain	Distinctive Mechanism	Empirical Gain (Reported)
DCAG	Image editing (DiT)	$\hat{K}^i = \bar{K} + \delta_k \Delta K^i$1 2D control	up to 4.3% LPIPS reduction (Li et al., 20 Feb 2026)
mEOL	Multimodal retrieval	Prompt-Guided 1-token vector	R@1: 0.35 vs. 0.15 for CLIP (Kim et al., 18 Apr 2026)
KVW	Unlearning (LVLMs)	Exponential vector gating	$\hat{K}^i = \bar{K} + \delta_k \Delta K^i$2 speedup v. LoRA (Kim et al., 29 Jan 2026)
TruthV	Truthful QA in LLMs	Calibrated ensemble vote	$\hat{K}^i = \bar{K} + \delta_k \Delta K^i$3–$\hat{K}^i = \bar{K} + \delta_k \Delta K^i$4 acc. over NoVo (Liu et al., 22 Sep 2025)

Performance improvements are typically measured on domain-relevant metrics, such as LPIPS for image fidelity, Recall@K for retrieval, and accuracy/ROUGE for unlearning and truthfulness detection. Notably, these methods exhibit high efficiency and interpretability; for example, DCAG exposes explicit controls, KVW and TruthV select or attenuate vectors based on interpretable statistics, and mEOL produces semantically meaningful embeddings without auxiliary projection heads.

5. Comparative Advantages and Practical Considerations

• Efficiency: All frameworks eschew backpropagation, reducing overall FLOPs by factors of $\hat{K}^i = \bar{K} + \delta_k \Delta K^i$5–$\hat{K}^i = \bar{K} + \delta_k \Delta K^i$6 and halving or better the GPU memory footprint compared to fine-tuned adaptation (Kim et al., 29 Jan 2026).

• Interpretability: Each intervention or selected vector corresponds to directly measurable model activations, with fine-grained control exposed (as in DCAG’s 2D parameter plane) or ensemble-weights assignable (as in TruthV’s calibration).
• Generalizability: These methods are adaptable to diverse modalities; for instance, mEOL extends prompt-level guidance to text, image, SVG, and their combinations via instruction templating (Kim et al., 18 Apr 2026).
• Zero-shot/Prompt-level Control: By leveraging existing model generalization, these approaches avoid domain-specific retraining or data, demonstrating strong zero-shot capabilities particularly in retrieval and editing tasks (Kim et al., 18 Apr 2026, Li et al., 20 Feb 2026).

6. Limitations and Prospects for Extension

Limitations include restriction to settings where value vector activations are either sufficiently interpretable or statistically informative (e.g., TruthV currently limited to MCQ, with open-ended text left unresolved (Liu et al., 22 Sep 2025)). The reliance on forward statistics introduces sensitivity to activation drift in evolving model architectures. Cross-task transferability of calibrated ensembles (as in TruthV and KVW) sometimes exhibits only moderate generalization, constraining universal applicability.

Proposed extensions involve combining value-based and attention-based signals, structured attribute-aware embeddings in mEOL, and adapting these paradigms to new modalities such as video, 3D, or graphs (Kim et al., 18 Apr 2026). The effectiveness of such methods suggests broader opportunities for interpretability, editing, and safety in generative models.

7. Connections and Broader Impact

Training-free value vector methods represent a shift from parameter-centric adaptation to activation- and structure-aware exploitation of model internals. By exposing control knobs and selection mechanisms rooted in the value space, these approaches provide both practical advantages—efficient, domain-agnostic tailoring—and conceptual insights into the loci of semantic representation in large neural systems. Their success in domains ranging from image editing to knowledge unlearning and factuality detection points toward a growing repertoire of non-mutative model governance strategies. This suggests growing interest in value vector-centric paradigms for scalable, interpretable, and efficient model manipulation across architectures, modalities, and downstream objectives (Li et al., 20 Feb 2026, Kim et al., 18 Apr 2026, Kim et al., 29 Jan 2026, Liu et al., 22 Sep 2025).