State-Grounded Cross-Attention (SGCA)

• SGCA is a cross-modal fusion mechanism that grounds attention in evolving state recurrences, enabling dynamic integration of various modalities.
• It generalizes standard dot-product attention with state-driven kernels and asymmetric mappings to reduce computational complexity and enhance expressivity.
• Frameworks like CrossMamba, CrossWKV, and CLaD demonstrate SGCA's practical benefits in robotics, audio-visual tasks, and text-to-image synthesis.

State-Grounded Cross-Attention (SGCA) refers to a class of cross-modal attention mechanisms that fuse multiple input streams—such as kinematic, visual, or audio-visual modalities—by grounding the attention computation in state-based, dynamically evolving recurrences. SGCA generalizes and extends the concept of standard cross-attention by integrating either the hidden-state dynamics of recurrent or state-space models (SSMs) or explicit cross-modal asymmetries, thereby achieving efficient, expressive fusion and robust planning or perception capabilities across tasks such as robotic manipulation, target sound extraction, and text-to-image synthesis. Key instantiations include the CrossMamba, CrossWKV, and CLaD frameworks, each tailored to their application domains but united by the principle of grounding cross-attention computation in state evolution, not solely in learned dot-product kernels.

1. Core Principles and Mathematical Formulation

Central to SGCA is the replacement or augmentation of dot-product/softmax-based attention weights with kernels or update rules derived from the evolving hidden state of an SSM or RNN, or by introducing explicit asymmetries in the cross-modal query-key-value mappings. In contrast to standard Transformer cross-attention, which computes $\mathrm{softmax}(QK^\top/\sqrt{d})V$ and incurs $O(T^2)$ complexity, SGCA leverages stateful recurrences or asymmetric message-passing to offer linear compute and memory scaling.

• In CrossMamba (Wu et al., 2024), the attention kernel is

$\alpha_{i,j} = Q_i K_j H_{i,j}$

where $Q_i$ is derived from the query sequence (e.g., a condition or clue), $K_j$ and $V_j$ from the value sequence (e.g., audio features), and $H_{i,j}$ is a state-transition kernel computed from the chain product of per-step transition matrices, i.e., $H_{i,j} = \prod_{k=j+1}^i \exp(\Delta_k A)$. The output is

$y_i = \sum_{j=1}^i Q_i K_j H_{i,j} V_j$

The SGCA score thus grounds attention in both the input stream and the SSM recurrrences.

• In CrossWKV for RWKV-7 (Xiao et al., 19 Apr 2025), the state update is formulated via a generalized delta rule:

$$S_t = S_{t-1} \left[\mathrm{diag}(\mathbf{w}_t) - \mathbf{k}_t^\top (\mathbf{a}_t \otimes \mathbf{k}_t) \right] + \mathbf{v}_t^\top \mathbf{k}_t$$

with vector-valued gates $O(T^2)$0, keys $O(T^2)$1, values $O(T^2)$2, and text-driven receptance $O(T^2)$3, providing cross-modal fusion in a recurrent, constant-memory fashion.

• In CLaD (Jeong et al., 31 Mar 2026), SGCA is realized as an asymmetric multi-head cross-attention where proprioceptive transitions query semantic transitions:

$O(T^2)$4

with $O(T^2)$5, $O(T^2)$6, $O(T^2)$7 taken from learned transition embeddings of the proprioceptive and semantic histories.

Distinctive to all variants is that the cross-attention operation is either directly parameterized by, or tightly coupled with, the evolving state; the modalities' information or state trajectories modulate the kernels/gates governing attention.

2. Architectures and Cross-Modal Alignment

SGCA has been instantiated in several architectural paradigms:

• Mamba-based Models (CrossMamba): SSMs parameterize attention with O($O(T^2)$8) complexity. Input-dependent gating and kernel updates yield feature fusion without explicit O($O(T^2)$9) softmax operations, making this design particularly effective in target sound extraction pipelines, such as AV-SepMamba and WaveMamba (Wu et al., 2024).
• RWKV-7 and CrossWKV: Combines linear-complexity RWKV with text-conditioned, non-diagonal, input-dependent state transitions to yield global cross-attention without key-value caching. This supports high expressivity and fast, memory-efficient text-to-image synthesis (Xiao et al., 19 Apr 2025).
• CLaD and Robotic Planning: Embeds kinematic (proprioceptive) and semantic token sequences, then employs SGCA for asymmetric querying (proprioceptive-over-semantic), yielding a shared dynamics vector for foresight prediction in diffusion-action policy generation (Jeong et al., 31 Mar 2026).

The unifying trait is the explicit modeling of state or transition dynamics grounding the cross-modal attention. This allows for complex, context-sensitive fusion—e.g., in CLaD, the kinematic state queries extract action-relevant knowledge from high-dimensional semantic tokens, crucial for long-horizon plan generation.

3. Computational Efficiency and Expressivity

SGCA mechanisms provide significant gains in both efficiency and theoretical capacity.

Efficiency: By leveraging hidden-state recurrences or sequence-wise update rules rather than full attention matrices,

• CrossMamba achieves $\alpha_{i,j} = Q_i K_j H_{i,j}$0 time and $\alpha_{i,j} = Q_i K_j H_{i,j}$1 memory, compared to Transformer's $\alpha_{i,j} = Q_i K_j H_{i,j}$2 time and $\alpha_{i,j} = Q_i K_j H_{i,j}$3 memory (Wu et al., 2024).
• RWKV-7’s CrossWKV maintains constant memory, supporting long-context or high-resolution tasks (e.g., 256×256 image generation in 0.52 seconds at 4.5 GB memory) (Xiao et al., 19 Apr 2025).

Expressivity: State-driven cross-attention with non-diagonal, input-conditioned recurrences enables:

• TC$\alpha_{i,j} = Q_i K_j H_{i,j}$4 hardness: CrossWKV can simulate DFA transitions, thus recognize all regular languages and perform nontrivial permutation or state-tracking tasks (e.g., S$\alpha_{i,j} = Q_i K_j H_{i,j}$5 group modeling) within bounded depth, which is unreachable for diagonal-only or fixed-weight RNNs (Xiao et al., 19 Apr 2025).
• Dynamic and asymmetric dependency modeling, e.g., CLaD’s ability to align cross-modal kinematic/semantic state evolution for complex robotic behaviors (Jeong et al., 31 Mar 2026).

4. Self-Supervised Grounding and Loss Formulations

SGCA’s capacity to model grounded foresight or cross-modal dependencies is anchored by auxiliary self-supervised objectives:

• In CLaD, predicted latent foresights for both proprioceptive and semantic modalities are calibrated against EMA (momentum) targets. The loss is:

$\alpha_{i,j} = Q_i K_j H_{i,j}$6

with an additional L$\alpha_{i,j} = Q_i K_j H_{i,j}$7 reconstruction loss:

$\alpha_{i,j} = Q_i K_j H_{i,j}$8

and total objective $\alpha_{i,j} = Q_i K_j H_{i,j}$9, $Q_i$0 (Jeong et al., 31 Mar 2026).

This dual-objective (stable self-supervision plus observation decoding) ensures that latent predictions are both discriminative and readily mapped back to observable state/action spaces, preventing representational collapse and securing robust policy conditioning.

5. Applications and Empirical Results

State-Grounded Cross-Attention mechanisms have demonstrated strong empirical performance across diverse application domains:

Model/Framework	Application Domain	Core SGCA Mechanism	Empirical Results / Gains
CLaD (Jeong et al., 31 Mar 2026)	Robotic planning	Asymmetric kinematic→semantic attention	94.7% SR on LIBERO-LONG (0.66B params, >7× smaller than OpenVLA with 93.8%)
CrossMamba (Wu et al., 2024)	Target sound extraction	SSM-based Mamba hidden-attention kernel	60-73% MAC reduction, SI-SNR equal/better vs Transformer baselines
CrossWKV (Xiao et al., 19 Apr 2025)	Text-to-image generation	RWKV-7, vector-gated, non-diagonal recurrence	FID = 2.88, CLIP score=0.33 (ImageNet 256x256, competitive with DiT-XL/2)

CLaD’s ablation studies show that cross-modal grounding in SGCA is essential; eliminating L$Q_i$1 degrades SR by 8.6 points (94.7%→86.1%). Asymmetric attention—letting kinematic tokens query semantic ones—yields higher success rate (94.7%) than the reverse (93.8%) or symmetric self-attention (86.7%) (Jeong et al., 31 Mar 2026).

In audio-visual separation, CrossMamba outperforms or matches Transformer-based baselines with parameter reduction (up to 32%) and dramatic MACs savings. For image synthesis, CrossWKV’s linear-state mechanism attains global fusion and high fidelity at modest compute.

6. Theoretical Significance and Broader Implications

SGCA’s introduction marks a unification of state-space model paradigms (e.g., RNN/SSM) and attention, providing the global dependency modeling of attention with the scalability and stateful capacity of recurrence. The resulting mechanisms can:

• Scale cross-modal attention to long sequences or high-resolution outputs without explicit attention caches.
• Implement theoretical constructs (DFA, S$Q_i$2 permutation modeling, regular language recognition) directly within the state evolution (Xiao et al., 19 Apr 2025).
• Enable robust fusion in any domain where learnable state evolution and dynamic cross-modality are essential (e.g., video, streaming, neural symbolic reasoning).

Broader implications include pathways to unified SSM-attention architectures and new forms of efficient multimodal reasoning and planning beyond Transformer expressivity limits. Applications range from long-horizon robotics to real-time audio-visual tasks and high-fidelity generative modeling.

7. Comparison with Transformer Cross-Attention and Limitations

In contrast to standard Transformers, which store and operate over the full key-value history for quadratic costing, SGCA:

• Maintains a fixed-size, dynamically updated state or kernel that accumulates all necessary cross-modal information.
• Linearizes compute and memory scaling, making it suitable for resource-constrained or long-context deployments.
• Embeds global cross-modal dependencies directly in recurrent updates or asymmetric attention flows.

Limitations reflect the design-specific expressivity or optimization requirements—e.g., effective engineering of the recurrence, specialized gating or normalization layers, and the necessity for carefully designed auxiliary losses to ensure stable grounding and avoid collapse.

SGCA continues to drive research into more expressive, efficient cross-modal fusion, establishing itself as a foundational component in next-generation state-space and attention hybrid models (Jeong et al., 31 Mar 2026, Wu et al., 2024, Xiao et al., 19 Apr 2025).