Decoupled-Value Attention (DVA)

• DVA is a distinct attention mechanism that decouples input-driven affinity from output-specific value propagation, enhancing interpretability and adherence to domain biases.
• It is applied in PFNs for surrogate modeling and in text-to-image diffusion models, where value-only adaptation improves error rates and concept disentanglement.
• Empirical results show DVA outperforms conventional attention by mimicking Gaussian process properties, achieving lower errors and efficient high-dimensional performance.

Decoupled-Value Attention (DVA) denotes a distinct class of attention mechanisms characterized by the strict separation (decoupling) of the sources for the attention affinity and the values to be propagated. In DVA, attention weights (computed from queries and keys) are derived solely from input or context signals, whereas value embeddings exclusively encode output, label, or concept information. This paradigm has been independently introduced and rigorously analyzed in two lines of work: in surrogate modeling of physical systems via prior-data fitted networks (PFNs) (Sharma et al., 25 Sep 2025), and in multi-concept personalization of text-to-image diffusion models (Lim et al., 6 Oct 2025). DVA provides both theoretical and empirical advantages over conventional, fully entangled attention, improving locality, interpretability, and faithfulness to domain-specific inductive biases such as those found in Gaussian-process regression or multi-entity compositional reasoning.

1. Formalism and Core Principles

The defining feature of DVA is the complete decoupling of the attention affinity computation from the channel that conveys output or label information. In canonical transformer attention, the inputs to the query, key, and value projections may come from the same embedding. By contrast, DVA enforces the following schematic separation:

• Affinity Channel: $\mathbf{Q}$ (query) and $\mathbf{K}$ (key) are functions solely of input variables or context features (e.g., $\phi_x(x)$, where $x$ is an input vector).
• Value Channel: $\mathbf{V}$ (value) is exclusively a function of output variables, reference labels, or concept embeddings (e.g., $\phi_y(y)$).

Attention weights $\alpha_{ij}$ are thus computed as

$$\alpha_{ij} = \frac{\exp(\langle Q^*_i, K_j\rangle/\sqrt{d_k})}{\sum_{\ell=1}^n \exp(\langle Q^*_i, K_\ell\rangle/\sqrt{d_k})}$$

where $Q^*_i$ is the query for the $i$th test point and $\mathbf{K}$0 the $\mathbf{K}$1th context key (see [Eq 7], (Sharma et al., 25 Sep 2025)). The output aggregation uses only the scalar or vector representations in $\mathbf{K}$2, which encode label or concept-specific information:

$\mathbf{K}$3

The process mirrors a kernel-weighted summation over reference values, analogous to the predictive mean in Gaussian processes (GPs).

2. Applications and Implementation Variants

A. Prior-Data Fitted Networks (PFNs) and Surrogate Modeling

In PFNs (Sharma et al., 25 Sep 2025), DVA is motivated by the GP property that predictive means are determined by input-input similarity (via a kernel) and outputs are aggregated as a weighted sum. Formally, for a context set $\mathbf{K}$4 and targets $\mathbf{K}$5:

• $\mathbf{K}$6, $\mathbf{K}$7 for input encodings,
• $\mathbf{K}$8 for output encodings.

For query points $\mathbf{K}$9, similarities are computed entirely based on $\phi_x(x)$0 and $\phi_x(x)$1. Output predictions are obtained as

$\phi_x(x)$2

where $\phi_x(x)$3 is a final non-linear head.

B. Diffusion Model Personalization via ConceptSplit

In text-to-image diffusion, DVA is realized through Token-wise Value Adaptation (ToVA) (Lim et al., 6 Oct 2025), targeting disentangled multi-concept personalization. All query and key projections ($\phi_x(x)$4, $\phi_x(x)$5) remain frozen, and only the value projection ($\phi_x(x)$6) is adapted for each personalized concept through low-rank adapters:

$\phi_x(x)$7

where $\phi_x(x)$8, $\phi_x(x)$9 are trainable and $x$0 is the token embedding of concept $x$1. For a sequence $x$2,

$x$3

ensuring each token receives a separate value contribution. The cross-attention maps remain fixed throughout, preserving spatial and semantic alignment for each concept token.

3. Theoretical Connections and Motivations

DVA in PFNs is directly inspired by the update equations in GP regression. The GP predictive mean is:

$x$4

with coefficients $x$5 from input-only kernels. DVA recovers this structure with learned, positive, and normalized attention weights replacing GP kernel weights, and output lines analogous to $x$6 propagated solely through $x$7. Unlike kernel attention (RBF), DVA employs trainable dot-product similarity, making it kernel-free and learnable across diverse function classes.

In ConceptSplit, decoupling value adaptation avoids the destabilizing effect of key modification on attention map sharpness and spatial localization. Empirically, adaptation of $x$8 or joint $x$9 increases attention entropy and induces concept mixing, while ToVA (value-only) maintains per-token spatial focus and leads to superior disentanglement (Lim et al., 6 Oct 2025).

4. Training and Inference Procedures

PFNs with DVA

The PFN pipeline embeds context pairs $\mathbf{V}$0 using $\mathbf{V}$1 and $\mathbf{V}$2, computes Q, K, V via learned projections, and applies dot-product attention from query input points to context. Attention-aggregated values $\mathbf{V}$3 are mapped to predictions by head $\mathbf{V}$4. Training minimizes negative log-likelihood (NLL) or MSE over batched context/target sets. Architecture search varies width, depth, number of heads, and other hyperparameters, with DVA used as a plug-in to Transformer or CNN backbones.

ConceptSplit: ToVA and LODA

Personalization for concept $\mathbf{V}$5 optimizes only the LoRA-style adapter $\mathbf{V}$6 for the corresponding token; the loss is the standard diffusion denoising objective over concept images. During inference, Latent Optimization for Disentangled Attention (LODA) further separates concept attention maps in two stages:

1. Latent Optimization: For timesteps $\mathbf{V}$7, gradients of a KL disentanglement loss across token attention maps are backpropagated to the latents, forcing attention peaks apart.
2. Attention Fixing Guidance (AFG): For later timesteps, hard masks are constructed per-token to maintain separation and prevent re-entanglement, and manipulated attention logits guide denoising.

Algorithmic details and pseudocode appear in (Lim et al., 6 Oct 2025).

5. Empirical Findings and Comparative Performance

Surrogate Modeling and Physical Equations

PFNs with DVA substantially reduce validation loss and mean squared error (MSE) compared to vanilla attention (VA), especially in high-dimensional tasks (5D, 10D, 64D):

Dim	Backbone	Attention	MSE	Final Val Loss
5D	Tx	VA	2.43e-4	–2.04
5D	Tx	DVA	2.84e-5	–4.05
10D	CNN	VA	3.55e-3	–0.81
10D	CNN	DVA	5.49e-4	–1.51

In the IEEE 33-bus test problem (64D), DVA-PFNs achieve errors on the order of $\mathbf{V}$8, closely matching GP accuracy but with over $\mathbf{V}$9 speedup (Sharma et al., 25 Sep 2025).

Multi-Concept Diffusion Models

ConceptSplit with DVA achieves higher disentanglement and per-concept accuracy. In two-object scenarios without background, it reports TA=0.238, C-IA=0.761, D-IA=0.809, GE=0.902, outperforming prior adapter-based approaches (e.g., EDLoRA D-IA=0.566, GE=0.342). Ablations confirm that purely value-based adaptation yields best performance; modifying $\phi_y(y)$0 irreparably degrades attention separation (Lim et al., 6 Oct 2025). LODA’s latent optimization and AFG further bolster multi-concept compositionality.

6. Limitations and Open Research Problems

Softmax-normalized DVA cannot represent negative attention weights, while GP coefficients may be negative; practical architectures compensate via a downstream nonlinearity. In PFNs, omitting label information from attention affinity may under-utilize output-informative patterns. In scalability, the memory footprint of DVA may be prohibitive for very high-dimensional regimes or large context sets, motivating local/hierarchical or memory-efficient variants. Hybrid attentions that admit selective output signals in affinities while preserving input locality represent an open research area (Sharma et al., 25 Sep 2025). For multi-concept diffusion, fine-tuning $\phi_y(y)$1 alone may not suffice for edge cases involving semantic ambiguity or highly entangled prompts, and compositional generalization outside the training distribution remains challenging.

7. Significance and Theoretical Implications

Decoupled-Value Attention operationalizes the principle of structure-preserving attention, faithfully mirroring inductive biases found in probabilistic inference (GPs) and multi-entity compositionality. By enforcing input-driven affinity and output-only value propagation, DVA recovers essential locality and disentanglement, providing plug-in compatibility for both transformer and convolutional backbones. DVA’s success in direct surrogate learning, high-dimensional physical modeling, and concept disentanglement in text-to-image diffusion underscores its role as a foundational primitive for structure-aware neural attention mechanisms (Sharma et al., 25 Sep 2025, Lim et al., 6 Oct 2025).