AttnMVP: Mixed Value Pooling in Recommenders

• The paper introduces AttnMVP, an early-fusion generative recommender system that replaces interleaved item–action streams with a causality-aware Transformer using mixed value pooling.
• It computes queries and keys solely from item embeddings while integrating corresponding action signals into the value stream, explicitly encoding the causal i→a dependency.
• Empirical results demonstrate improved evaluation loss, reduced training time, and enhanced efficiency compared to conventional interleaved attention methods on large-scale recommendation data.

Searching arXiv for the AttnMVP paper and closely related pooling/attention work to ground the article. Attention-based Mixed Value Pooling (AttnMVP) is an early-fusion architecture for generative recommender systems that replaces interleaved item–action token streams with a causality-aware Transformer in which queries and keys are item representations and values are mixtures of item and action embeddings. Introduced in "Beyond Interleaving: Causal Attention Reformulations for Generative Recommender Systems" (Cheng, 11 Mar 2026), it is designed to explicitly encode the dependency $i_n \rightarrow a_n$, reduce the structural inefficiencies of interleaving, and preserve the expressive power of Transformer-based sequence modeling while shortening the effective sequence by half.

1. Problem formulation and motivation

AttnMVP arises in the setting where user behavior is modeled as a sequence of item–action pairs,

$(i_0,a_0), (i_1,a_1), \dots, (i_T,a_T),$

with $i_n \in \mathcal{I}$ denoting an item token and $a_n \in \mathcal{A}$ denoting an action token. Standard generative recommender systems interleave these into a single causal sequence,

$$\mathbf{x} = [i_0, a_0, i_1, a_1, \dots, i_T, a_T],$$

and apply a causal Transformer over the resulting token stream (Cheng, 11 Mar 2026).

The reformulation leading to AttnMVP starts from a structural critique of interleaving. Interleaving doubles the sequence length, so if the original behavior length is $N$, the attention cost shifts from $O(N^2)$ to

$O((2N)^2) = 4O(N^2).$

The same formulation also mixes semantically heterogeneous item and action tokens in a shared self-attention space. With positional mechanisms such as RoPE or relative bias, adjacent tokens interact strongly, so the fixed alternation pattern induces spurious dependencies: $i_n$ acquires a strong bias toward $a_{n-1}$, and $(i_0,a_0), (i_1,a_1), \dots, (i_T,a_T),$0 attends broadly to past items and actions. The paper characterizes this as structural ambiguity and attention noise rather than a faithful representation of the underlying causal process (Cheng, 11 Mar 2026).

A central modeling claim is that the action distribution is better expressed as

$(i_0,a_0), (i_1,a_1), \dots, (i_T,a_T),$1

where history $(i_0,a_0), (i_1,a_1), \dots, (i_T,a_T),$2 modulates the parameters relevant to the current item rather than entering through a flat interleaved token stream. Within this view, interleaved self-attention is reinterpreted as an inefficient proxy for similarity-weighted action pooling, which motivates architectures that preserve item–action causality explicitly and remove action tokens from the query/key space.

2. Mixed-value self-attention architecture

AttnMVP is the paper’s main early-fusion alternative to interleaved generative recommenders. It keeps one position per item–action pair, so the sequence length is $(i_0,a_0), (i_1,a_1), \dots, (i_T,a_T),$3 rather than $(i_0,a_0), (i_1,a_1), \dots, (i_T,a_T),$4. At each position $(i_0,a_0), (i_1,a_1), \dots, (i_T,a_T),$5, the model maintains an item embedding $(i_0,a_0), (i_1,a_1), \dots, (i_T,a_T),$6 and an action embedding $(i_0,a_0), (i_1,a_1), \dots, (i_T,a_T),$7, and initializes the Transformer state with

$(i_0,a_0), (i_1,a_1), \dots, (i_T,a_T),$8

Across $(i_0,a_0), (i_1,a_1), \dots, (i_T,a_T),$9 Transformer layers, with $i_n \in \mathcal{I}$0 in the reported experiments, queries and keys are computed from item representations only, while values are formed by mixing item and action information (Cheng, 11 Mar 2026).

At layer $i_n \in \mathcal{I}$1, the architecture uses

$i_n \in \mathcal{I}$2

and defines the mixed value at position $i_n \in \mathcal{I}$3 as

$i_n \in \mathcal{I}$4

Equivalently, after value projection,

$i_n \in \mathcal{I}$5

This is the mechanism denoted “mixed value pooling”: the attention map is computed in a homogeneous item space, while the value stream carries both item semantics and the action observed for that item. In the experiments, $i_n \in \mathcal{I}$6, and performance is reported as stable for $i_n \in \mathcal{I}$7 (Cheng, 11 Mar 2026).

Attention itself is standard causal self-attention. For query position $i_n \in \mathcal{I}$8,

$i_n \in \mathcal{I}$9

and the pre-residual update is

$a_n \in \mathcal{A}$0

After the Transformer stack, AttnMVP optionally applies an AttnLFA-like pooling stage in which final item representations serve as queries and keys, action embeddings serve as values, and pooling is performed under strict causality. The implementation uses standard causal attention and RoPE, so it remains compatible with standard kernels such as FlashAttention (Cheng, 11 Mar 2026).

3. Causal semantics and representation dynamics

The defining conceptual property of AttnMVP is that it encodes the direct causal edge $a_n \in \mathcal{A}$1 architecturally rather than asking self-attention over interleaved tokens to rediscover it. At position $a_n \in \mathcal{A}$2, the only action that directly enters the local value term is $a_n \in \mathcal{A}$3. Historical actions $a_n \in \mathcal{A}$4 for $a_n \in \mathcal{A}$5 influence $a_n \in \mathcal{A}$6 only if position $a_n \in \mathcal{A}$7 attends to position $a_n \in \mathcal{A}$8, and that attention is determined entirely by similarity in the item-derived query/key space (Cheng, 11 Mar 2026).

This yields a specific causal path. The local edge is

$a_n \in \mathcal{A}$9

Historical action effects are mediated through

$$\mathbf{x} = [i_0, a_0, i_1, a_1, \dots, i_T, a_T],$$0

with the route gated by item similarity between $$\mathbf{x} = [i_0, a_0, i_1, a_1, \dots, i_T, a_T],$$1 and $$\mathbf{x} = [i_0, a_0, i_1, a_1, \dots, i_T, a_T],$$2. Relative to interleaving, action tokens are therefore removed from the key/query geometry. The result is a homogeneous attention space for matching history items and a heterogeneous value space for importing the associated feedback.

The paper describes this as a reduction of attention noise. Self-attention remains a content-based pooling mechanism, but the pooling is restricted so that queries and keys reflect item semantics, whereas values encode “item plus observed response.” Over multiple layers, $$\mathbf{x} = [i_0, a_0, i_1, a_1, \dots, i_T, a_T],$$3 is expected to evolve from pure content semantics toward user-conditioned semantics such as “dog the user likes” or “cat the user tends to skip.” A direct empirical probe of this claim is the AttnMVP–LFA ablation, which retains the early mixed-value Transformer layers but removes the final late-fusion attention. That ablation nearly matches full AttnMVP, indicating that the primary gains come from early, causally constrained integration of actions via mixed values rather than from the final pooling stage alone (Cheng, 11 Mar 2026).

4. Empirical behavior

The reported evaluation uses large-scale product recommendation data from a major professional social network, with histories of up to 1024 events over the past 12 months. The model is trained as a multi-task classifier over actions using Binary Cross Entropy, and evaluation focuses on candidate items in a temporally held-out segment. Primary task metrics include Evaluation Loss and Normalized Entropy (NE) for Long Dwell, Contribution, and Like (Cheng, 11 Mar 2026).

Model	Eval Loss	Training time
AttnLFA	$$\mathbf{x} = [i_0, a_0, i_1, a_1, \dots, i_T, a_T],$$4	$$\mathbf{x} = [i_0, a_0, i_1, a_1, \dots, i_T, a_T],$$5
AttnMVP	$$\mathbf{x} = [i_0, a_0, i_1, a_1, \dots, i_T, a_T],$$6	$$\mathbf{x} = [i_0, a_0, i_1, a_1, \dots, i_T, a_T],$$7
AttnMVP–LFA	$$\mathbf{x} = [i_0, a_0, i_1, a_1, \dots, i_T, a_T],$$8	$$\mathbf{x} = [i_0, a_0, i_1, a_1, \dots, i_T, a_T],$$9

For AttnMVP specifically, the paper reports Long Dwell NE of $N$0, Contribution NE of $N$1, and Like NE of $N$2 relative to the interleaved baseline (Cheng, 11 Mar 2026). The abstract summarizes the same comparison in rounded form as evaluation loss improvements of $N$3 and $N$4 for AttnLFA and AttnMVP, accompanied by training time reductions of $N$5 and $N$6, respectively.

The ablation pattern is as informative as the headline numbers. AttnMVP–LFA achieves nearly the same Evaluation Loss and NE improvements as full AttnMVP while slightly improving training time, which isolates the contribution of mixed-value integration inside the Transformer layers. AttnLFA is more efficient than AttnMVP but yields materially smaller gains, which places AttnMVP at the higher-accuracy end of the causal reformulation spectrum.

5. Position within the pooling literature

AttnMVP belongs to a broader family of pooling mechanisms in which learned weights determine how token-, channel-, field-, or view-level values are aggregated. In text classification with HAL embeddings, attention-based pooling replaces mean pooling with a temperature-scaled additive attention so that the final sequence representation becomes a weighted mixture of token vectors (Sakour et al., 20 Mar 2026). In recurrent text encoders, max-attention pooling first derives a sentence-specific query by max-pooling hidden states and then performs attention-weighted aggregation, explicitly combining max-like saliency detection with soft attention over predictive tokens (Maini et al., 2020).

In recommender systems and tabular prediction, related mixed-pooling patterns appear in different forms. MMBAttn for CTR prediction combines max pooling, mean pooling, and a bit-wise attention branch, so field-level and dimension-level reweighting are both learned from pooled summaries of embeddings (Saribas et al., 2023). In healthcare representation learning, MPVAA aggregates encoder states using a stochastic convex combination of temporal max-pooling and mean-pooling,

$N$7

before multi-view attention fuses demographic, laboratory, and notes views (Chowdhury et al., 2019).

In convolutional attention modules, the same conceptual move appears at the channel level. SPEM replaces global average pooling with a self-adaptive mixture of global max-pooling and global min-pooling, using

$N$8

and reports that this mixed descriptor can match or exceed GAP-based attention baselines (Zhong et al., 2022). Universal Pooling pushes the idea further by treating pooling as channel-wise local spatial attention: scores are produced by a learnable module, normalized by block-wise softmax, and used to form weighted sums within each pooling window (Hyun et al., 2019).

In LLM embedding models, the design space extends to trainable token-level and layer-level pooling. Last-Layer Trainable Pooling applies cross-attention over last-layer token states, while Multi-Layers Trainable Pooling attends over per-layer representations derived from all hidden layers and is reported as statistically superior in text similarity and retrieval tasks under matched training conditions (Tang et al., 2024). Relative to these formulations, AttnMVP is distinguished by its asymmetric role assignment: the primary sequence type remains in the query/key space, while the auxiliary signal is attached to the value stream at each aligned position (Cheng, 11 Mar 2026).

6. Limitations and scope

The reported limitations are structurally tied to the design. First, the action stream is not independently modeled: actions never become queries or keys, so AttnMVP assumes that the primary sequence geometry should be item-defined rather than action-defined (Cheng, 11 Mar 2026). Second, the formulation fits regimes with a small action vocabulary and semantically simple actions such as click, long dwell, like, comment, or share. The details explicitly note that if actions become as rich and structured as items, a more symmetric architecture may be preferable. Third, the fusion inside the Transformer is linear,

$N$9

so more expressive gates or MLP-based value composition remain outside the reported design.

At the same time, the architecture is operationally conservative. It requires no custom kernels, keeps standard causal attention and RoPE, and can be implemented on top of existing sequential recommenders that already support causal Transformers (Cheng, 11 Mar 2026). A plausible implication is that the same design principle may extend to other sequential settings with a dominant token type and a small aligned response vocabulary, provided the intended causal structure is likewise “primary token $O(N^2)$0 local auxiliary signal.”