Self-Evolving Gated Attention (SEGA)

• Self-Evolving Gated Attention (SEGA) is a temporal module that compresses long observation streams into a fixed-size latent state, enabling efficient and robust imitation learning in robotics.
• The mechanism uses cross-attention-derived gating to update the latent state, filtering out irrelevant disturbances and maintaining temporal sparsity in complex tasks.
• Empirical evaluations on simulated and real-world benchmarks demonstrate that SEGA significantly improves success rates over standard diffusion policies, especially in long-horizon and noisy environments.

Self-Evolving Gated Attention (SEGA) is a temporal modeling module introduced as the core of SeedPolicy, a diffusion-policy-based imitation learning method for long-horizon robot manipulation. It is designed to address the temporal bottleneck of standard Diffusion Policy (DP), whose performance degrades as observation horizons increase because temporal context is treated as a stack of image frames without explicit temporal reasoning. SEGA maintains a compact, time-evolving latent state, updates that state through cross-attention, and regulates the update with a data-dependent gate derived directly from attention maps. In SeedPolicy, this mechanism compresses long observation streams into a fixed-size representation, filters temporally irrelevant disturbances, and enables horizon scaling with moderate overhead (Gui et al., 5 Mar 2026).

1. Problem setting and motivation

SEGA was proposed in the context of imitation learning for robotic manipulation, where Diffusion Policy is effective at modeling multi-modal expert behaviors but becomes unreliable as the observation horizon grows. The motivating diagnosis is specific: standard DP aggregates temporal context as stacked frames, fails to capture complex dependencies across time, and can suffer paradoxical performance collapse at large horizons because it indiscriminately integrates redundant or noisy frames such as background shifts and occlusions. In this formulation, the long-horizon failure is not merely a matter of insufficient capacity; it is a temporal modeling deficiency in how historical information is represented and filtered (Gui et al., 5 Mar 2026).

SEGA is the proposed remedy. It augments DP with a dedicated temporal module that preserves a fixed-size latent sequence encoding historical context and updates that sequence recurrently. The update is regulated by a self-evolving gate computed from intrinsic cross-attention scores rather than from a separate parametric recurrent controller. The intended effect is twofold: compression of arbitrarily long histories into a stable latent representation, and suppression of semantically irrelevant frames before they pollute the evolving state. This design is presented as reversing DP’s degradation with longer horizons while keeping horizon scaling computationally tractable.

A common misconception is that SEGA is simply another temporal attention layer added on top of DP. The paper’s description is narrower and more structured: SEGA is a dual-stream Transformer module with an explicit recurrent latent state, an update path, and a retrieval path. Its core novelty lies not only in attending across time, but in using cross-attention maps themselves as update gates for latent-state evolution.

2. Architectural formulation

At time step $t$, SEGA operates on encoded observations $O_t \in \mathbb{R}^{N_o \times D}$, a latent state $S_{t-1} \in \mathbb{R}^{N_s \times D}$, and produces both an updated state $S_t \in \mathbb{R}^{N_s \times D}$ and enhanced observation features $EObs_t \in \mathbb{R}^{N_o \times D}$. The observation features are produced from the current RGB image $I_t$ and joint pose $P_t$. The module is organized as two coordinated processes: an upper-stream State Update that evolves the latent state, and a lower-stream State Retrieval that reads relevant historical context from the latent state to enrich the current observation features (Gui et al., 5 Mar 2026).

The pipeline is specified as

$$\begin{aligned} O_t &= \text{Encoder}(I_t, P_t) \ S_t &= \text{Update}(S_{t-1}, O_t) \ EObs_t &= \text{Retrieve}(O_t, S_{t-1}) \ A_t &= \text{Diffusion}(EObs_t). \end{aligned}$$

Before cross-attention, both the latent state and current observation features undergo residual self-attention preprocessing:

$$S'_{t-1} = S_{t-1} + \text{MSA}(S_{t-1}), \qquad O'_t = O_t + \text{MSA}(O_t).$$

The State Update path then extracts candidate state content and raw cross-attention maps:

$$\text{Inter} \cdot S_t, \mathcal{A} = \text{CA}(S'_{t-1}, O'_t, O'_t),$$

where $O_t \in \mathbb{R}^{N_o \times D}$0 collects the raw attention scores across layers and heads. The self-evolving gate is computed directly from these cross-attention maps:

$O_t \in \mathbb{R}^{N_o \times D}$1

with

$O_t \in \mathbb{R}^{N_o \times D}$2

The latent state is then updated by gated fusion:

$O_t \in \mathbb{R}^{N_o \times D}$3

Finally, the State Retrieval path enriches the current observation features using the previous latent state:

$O_t \in \mathbb{R}^{N_o \times D}$4

Several details are structurally important. First, the gate $O_t \in \mathbb{R}^{N_o \times D}$5 is a per-state-token update gate. Second, the gate is derived by aggregating cross-attention scores over layers $O_t \in \mathbb{R}^{N_o \times D}$6, heads $O_t \in \mathbb{R}^{N_o \times D}$7, and observation tokens $O_t \in \mathbb{R}^{N_o \times D}$8, rather than by learning a separate recurrent gating network. Third, the paper does not expand MSA and CA into explicit $O_t \in \mathbb{R}^{N_o \times D}$9 or softmax equations; the emphasis is instead on the use of cross-attention maps as intrinsic relevance signals. The stated consequence is temporal sparsity: irrelevant or noisy frames are suppressed, reducing “state pollution” over long horizons.

3. Role within SeedPolicy and computational properties

Within SeedPolicy, SEGA is placed between the visual encoder and the diffusion “Action Expert.” The encoder produces $S_{t-1} \in \mathbb{R}^{N_s \times D}$0, SEGA updates the latent state and produces $S_{t-1} \in \mathbb{R}^{N_s \times D}$1, and the diffusion model conditions on $S_{t-1} \in \mathbb{R}^{N_s \times D}$2 to predict a sequence of future 14-DoF actions. The diffusion backend follows a DDPM-style formulation with $S_{t-1} \in \mathbb{R}^{N_s \times D}$3-prediction, 100 training timesteps, 100 inference timesteps, and a Squared Cosine Cap v2 beta schedule with $S_{t-1} \in \mathbb{R}^{N_s \times D}$4 and $S_{t-1} \in \mathbb{R}^{N_s \times D}$5. The paper explicitly states that the closed-form diffusion loss is not restated and that the method follows standard conditional denoising diffusion policy training, using $S_{t-1} \in \mathbb{R}^{N_s \times D}$6 as the conditioning input (Gui et al., 5 Mar 2026).

The module’s computational rationale is tied to horizon scaling. Full temporal attention over $S_{t-1} \in \mathbb{R}^{N_s \times D}$7 frames is expensive and standard Transformer temporal encoders incur quadratic cost in $S_{t-1} \in \mathbb{R}^{N_s \times D}$8. SEGA avoids this by maintaining a fixed-size latent state $S_{t-1} \in \mathbb{R}^{N_s \times D}$9 of length $S_t \in \mathbb{R}^{N_s \times D}$0, so the per-step overhead is constant in the observation horizon. The cost instead grows primarily with $S_t \in \mathbb{R}^{N_s \times D}$1, $S_t \in \mathbb{R}^{N_s \times D}$2, and the number of SEGA attention blocks $S_t \in \mathbb{R}^{N_s \times D}$3 and heads $S_t \in \mathbb{R}^{N_s \times D}$4. The paper emphasizes “efficient recurrent updates” qualitatively and does not provide explicit asymptotic complexity formulas.

The comparison to other temporal mechanisms is also specific. Relative to LSTM and GRU, SEGA’s temporal integration intensity is described as content-adaptive and non-parametric with respect to the gate, because the gate is driven by cross-attention weights rather than fixed parametric recurrent gates. Relative to vanilla Transformer temporal encoders, SEGA replaces full-window temporal attention with a compact recurrent state and attention-derived gating along the temporal dimension. This suggests a hybrid of recurrent statefulness and attention-based relevance estimation rather than a conventional RNN or a full temporal Transformer.

At inference time, the latent state is updated recurrently as new observations arrive. Because historical information is compressed into $S_t \in \mathbb{R}^{N_s \times D}$5, SeedPolicy can increase the effective observation horizon without the collapse reported for vanilla DP. The paper attributes the improved long-horizon behavior to this recurrent compression-and-filtering mechanism.

4. Empirical results in simulation and on real robots

SEGA was evaluated through SeedPolicy on the RoboTwin 2.0 benchmark, which contains 50 simulated manipulation tasks with an ALOHA-AgileX robot. Each policy was trained on 50 expert demonstrations for 600 epochs and evaluated with 100 rollouts per task. Mean success rates were reported over three independent trials in two configurations: Easy ($S_t \in \mathbb{R}^{N_s \times D}$6) and Hard ($S_t \in \mathbb{R}^{N_s \times D}$7) (Gui et al., 5 Mar 2026).

Across both CNN and Transformer backbones, SeedPolicy achieved an average relative improvement of 36.8% over DP in Easy settings and 169% in Hard settings. The reported aggregate results are:

Model	Easy	Hard
DP-Transformer	33.10%	1.44%
SeedPolicy-Transformer	40.08%	4.28%
DP-CNN	28.04%	0.64%
SeedPolicy-CNN	42.76%	1.54%
RDT-1B	34.50%	13.72%

The parameter counts reported alongside these results are 20.61M for DP-Transformer, 33.36M for SeedPolicy-Transformer, 96.80M for DP-CNN, 147.26M for SeedPolicy-CNN, and 1.2B for RDT. The comparison to large vision-language-action models is framed in terms of efficiency: SeedPolicy-CNN surpasses RDT on Easy settings with two orders of magnitude fewer parameters, whereas RDT remains stronger in Hard settings, which the paper attributes to large-scale pretraining.

The ablation study isolates SEGA’s components on three tasks: Turn Switch, Place Empty Cup, and Stack Bowls Two. For DP the success rates are 51%, 24%, and 33%; adding Temporal Attention yields 51%, 26%, and 48%; adding a recurrent State without gate yields 51%, 28%, and 65%; adding Cross-Attention-based Gating, i.e. SeedPolicy, yields 54%, 32%, and 73%; and replacing that gate with FFN Gating yields 53%, 21%, and 70%. The paper interprets these results as showing that temporal attention alleviates the fixed-window bottleneck, that the recurrent state mechanism contributes both efficiency and performance, and that the cross-attention-based gate produces the largest gains, especially on long-horizon tasks.

The real-robot experiments were conducted on Dexmal Dos W1 with an Intel RealSense D435, using 50 demonstrations per task and 20 trials per task on Looping_Place-Retrieval, Sequential_Picking, and Bottle_Handover. The qualitative summary reports that SeedPolicy significantly boosts success rates and mitigates execution stagnation and spatial positioning errors; Bottle Handover is given as an example, improving from 15% to 56%.

Robustness claims are most pronounced in randomized settings. In the Hard configuration, the gating mechanism is described as suppressing disturbances better than standard stacking. Reported examples include Grab Roller at 51% for SeedPolicy-Transformer versus near zero for the baseline, and Shake Bottle at 23% for SeedPolicy-CNN versus 8% for the baseline. The paper also states that performance margins widen as task length increases across Short, Medium, and Long categories.

5. Relation to gated-attention theory

A separate theoretical treatment of gated attention, “A Statistical Theory of Gated Attention through the Lens of Hierarchical Mixture of Experts,” provides analytical context for SEGA-like mechanisms by recasting each entry in a gated attention matrix or a multi-head self-attention matrix as a hierarchical mixture of experts (Nguyen et al., 1 Feb 2026). In that formulation, a multi-head self-attention entry can be written as

$S_t \in \mathbb{R}^{N_s \times D}$8

where $S_t \in \mathbb{R}^{N_s \times D}$9 are softmax routing weights and $EObs_t \in \mathbb{R}^{N_o \times D}$0 are linear experts. The theory argues that gating introduces nonlinear experts, breaks the PDE interaction

$EObs_t \in \mathbb{R}^{N_o \times D}$1

and thereby improves expert identifiability and sample efficiency.

The paper’s central quantitative distinction is between vanilla MHA and gated attention with nonlinearities placed either after the value projection or after the scaled dot-product attention output. For MHA, expert estimation is described as slower than any polynomial order, with sample complexity $EObs_t \in \mathbb{R}^{N_o \times D}$2 to achieve expert approximation error $EObs_t \in \mathbb{R}^{N_o \times D}$3. For the two gated settings, the reported expert estimation rates lie between $EObs_t \in \mathbb{R}^{N_o \times D}$4 and $EObs_t \in \mathbb{R}^{N_o \times D}$5, with sample complexity $EObs_t \in \mathbb{R}^{N_o \times D}$6. The same theory provides a justification for placing gates at the SDPA output or values rather than at queries, keys, or the final dense output.

This theory is not a formal derivation of SeedPolicy’s specific recurrent update rule. SEGA’s gate is computed from cross-attention maps and applied to latent-state evolution, rather than being introduced exactly as a value-map or SDPA-output nonlinearity. Nevertheless, the theory supplies a relevant lens for several claims made around SEGA: that gating can mitigate attention sink, that nonlinear or content-adaptive routing can improve expressiveness beyond low-rank mappings, and that attention-derived relevance signals can support more selective temporal integration. A plausible implication is that SEGA’s use of attention maps as update gates occupies a broader family of attention-routed mechanisms for which mixture-of-experts reasoning is informative even when the exact theorem statements do not directly instantiate the SeedPolicy architecture.

6. Failure modes, limitations, and recommended configurations

The failure modes that motivate SEGA are concrete. Vanilla DP is reported to exhibit execution stagnation and state aliasing in multi-stage tasks such as Looping_Place-Retrieval and Put_Bottles_Dustbin, along with spatial positioning errors such as air grabs and collisions due to lack of explicit depth. SEGA’s proposed mitigation is a continuous latent state that tracks task progression and breaks deadlocks, while temporal integration of motion cues compensates for missing depth by inferring spatial geometry from history. The paper presents these effects as qualitative explanations for improved long-horizon performance and robustness (Gui et al., 5 Mar 2026).

The reported limitations are equally specific. In highly randomized Hard settings, SeedPolicy’s generalization remains below that of billion-parameter VLA models such as RDT. The paper identifies integration of SEGA with VLA architectures as a direction for stronger open-world performance. This limitation qualifies the efficiency comparison: SEGA is markedly more parameter-efficient, but not uniformly superior under strong distributional variation.

The implementation details provided for SEGA within SeedPolicy include an observation history length $EObs_t \in \mathbb{R}^{N_o \times D}$7, stacked RGB frames at $EObs_t \in \mathbb{R}^{N_o \times D}$8, and a 14-DoF joint pose. The latent state is configured with $EObs_t \in \mathbb{R}^{N_o \times D}$9 and $I_t$0. Training uses AdamW, batch size 128, learning rate $I_t$1, cosine decay, 500 warmup steps, 600 epochs, EMA decay 0.75, gradient accumulation 1, and a single NVIDIA RTX 4090D GPU. The default backbone for real experiments is the Transformer, chosen for parameter efficiency, although the CNN backbone is also evaluated.

The hyperparameter sensitivities reported in the appendix are directly relevant to practical deployment. For latent state length, $I_t$2 underfits, $I_t$3 is best, and $I_t$4 plateaus or slightly degrades due to distraction; Stack Bowls Two at 73% is given as a representative improvement at $I_t$5. For SEGA depth $I_t$6, increasing the number of attention blocks from 2 to 6 improves performance, exemplified by Put Bottles Dustbin from 36% to 48%, whereas depth 8 degrades because of overfitting, with Move Can Pot reported as falling from 71% to 52%.

The paper’s practical takeaways therefore recommend SEGA for imitation learning and diffusion policies in long-horizon robotic manipulation when temporal sparsity and visual disturbances are prevalent, and when increasing the observation horizon causes standard DP to degrade. The recommended configuration is a latent state length $I_t$7, feature dimension $I_t$8, SEGA depth $I_t$9, $P_t$0-prediction DDPM with about 100 timesteps and a Squared Cosine Cap v2 schedule, and a Transformer backbone for parameter efficiency. In that usage regime, SEGA is presented as providing robust horizon scaling, a compact fixed-size temporal representation, efficient recurrent updates, and improved resilience against execution stagnation and state aliasing.