MixANT: Action Anticipation & Mix-Audio Augmentation

• MixANT is a state-space model extension for stochastic dense action anticipation that replaces static forget gates with an observation-dependent mixture-of-experts routing mechanism.
• The architecture preserves linear-time scanning while generating multiple plausible per-frame future action predictions, showing improved results on datasets like Breakfast, Assembly101, and 50Salads.
• Additionally, MixANT denotes a mix-audio augmentation strategy in music source separation that sums same-stem audio segments without label interpolation to enhance target clarity.

MixANT denotes two distinct constructs in the cited literature. In stochastic video understanding, it is an architecture for long-term dense action anticipation that introduces observation-dependent memory propagation into Mamba-like state space models by replacing the static forget gate $A$ with a routed mixture of experts (Wasim et al., 14 Sep 2025). In music source separation, the same acronym has also been used for “mix-audio augmentation,” a waveform-level training strategy that sums same-stem audio segments within CatNet (Song et al., 2021). The dominant technical usage is the former: a model for generating multiple plausible per-frame future action sequences over long horizons while preserving the linear-time scan of selective SSMs.

1. Task definition and motivation

MixANT was introduced for stochastic long-term dense action anticipation, defined as: given an observed prefix of a video, predict per-frame action labels for a long future horizon under uncertainty by generating multiple plausible future label sequences (Wasim et al., 14 Sep 2025). “Dense” denotes per-frame future labeling rather than a coarse transcript, so the model outputs an action label for every future frame and captures durations implicitly. “Stochastic” denotes multi-sample inference, intended to reflect uncertainty and multimodality when several continuations are equally plausible.

The evaluation protocol is parameterized by the observed fraction $\alpha$ and future fraction $\beta$. For a video $v$ with $n_v$ frames, the model observes the first $P=\alpha n_v$ frames and anticipates the next $F=\beta n_v$ frames, with $\alpha \in \{0.2,0.3\}$ and $\beta \in \{0.1,0.2,0.3,0.5\}$. Inference produces $S=25$ sampled futures. Performance is reported with Mean MoC and Top-1 MoC: $\alpha$0

The architectural motivation is a limitation in recent selective SSMs such as Mamba. These models already make $\alpha$1, $\alpha$2, and $\alpha$3 input-dependent, but the forget gate—the $\alpha$4 matrix that governs state memory decay and propagation—remains static in vanilla Mamba. In dense anticipation, this is restrictive for three reasons stated explicitly in the paper: temporal memory should be context-aware; future frames in the conditioning are zero-padded and should sometimes be selectively ignored; and making $\alpha$5 directly input-dependent with attention-like query-key mechanisms or large MLPs adds prohibitive overhead or breaks linear-time scanning. A common misconception is therefore that Mamba is already fully input-selective; in this setting, the critical memory-propagation term is not.

2. Observation-dependent memory propagation

MixANT retains the discrete-time selective-scan SSM formulation

$\alpha$6

with step-size discretization

$\alpha$7

but replaces the static $\alpha$8 with an input-dependent mixture-of-experts forget gate (Wasim et al., 14 Sep 2025).

Each MixMamba block maintains $\alpha$9 expert matrices $\beta$0. A lightweight router computes a gating vector from observed features only: $\beta$1 Hard routing then selects a single expert,

$\beta$2

so the block acquires input dependence in $\beta$3 while preserving an $\beta$4 scan. MixANT keeps Mamba’s input-dependent $\beta$5, $\beta$6, and $\beta$7; only $\beta$8 changes from static to routed mixture.

The resulting MixMamba layer is bi-directional. For each layer, the input is processed by a small feed-forward subnetwork and a 1D convolution $\beta$9 in forward and temporally reversed order, followed by an $v$0 selective scan using the routed $v$1. The forward and backward outputs are then modulated by a residual gate $v$2, the backward stream is flipped back in time, and the two directions are merged by a final feed-forward layer. The router is unified across forward and backward scans rather than independent; the reported analysis states that this yields better accuracy and improves bi-directional consistency.

To avoid expert collapse, the model applies load-balancing regularization. For expert usage in layer $v$3,

$v$4

and the balancing objective encourages a uniform allocation: $v$5 This design is the core of MixANT’s claim to observation-dependent memory propagation: each expert $v$6 can specialize to a distinct temporal decay or propagation regime, and routing selects among them with negligible per-step cost.

3. End-to-end model pipeline and optimization

The full anticipation model is a conditional diffusion system whose generator $v$7 is a stack of $v$8 blocks, comprising $v$9 initial standard bi-directional Mamba blocks followed by $n_v$0 MixMamba blocks (Wasim et al., 14 Sep 2025). The input conditioning uses pre-extracted per-frame visual features $n_v$1: I3D for Breakfast and 50Salads, and TSM for Assembly101. Future frames are represented by zero padding,

$n_v$2

During diffusion sampling, the latent label tensor $n_v$3 is concatenated with $n_v$4 along channels to form the block input $n_v$5. A final MLP maps the last block output to per-frame logits $n_v$6.

Training follows a conditional diffusion objective: $n_v$7 The reconstruction term compares the denoised sample with one-hot ground truth labels: $n_v$8 The total training loss combines reconstruction and load balancing,

$n_v$9

The reported ablation indicates that $P=\alpha n_v$0 is effective.

The training recipe is fully specified. MixANT uses $P=\alpha n_v$1 experts per MixMamba block, a unified router, and routing conditioned on observed frames only. Optimization uses AdamW with $P=\alpha n_v$2 and $P=\alpha n_v$3; the learning rate is $P=\alpha n_v$4 for Breakfast and Assembly101 and $P=\alpha n_v$5 for 50Salads; training lasts 90 epochs on all three datasets; diffusion training uses $P=\alpha n_v$6 steps; inference uses DDIM with 50 steps on Breakfast and Assembly101 and 10 steps on 50Salads; and training is performed on a single NVIDIA A100 80GB GPU. Source and additional materials are provided through the project page linked in the paper.

4. Empirical performance and computational profile

MixANT is evaluated on Breakfast, Assembly101, and 50Salads, and the paper reports consistent improvements over prior methods across all settings (Wasim et al., 14 Sep 2025). Breakfast uses 4-fold cross-validation, 48 classes, I3D features, and anticipation up to 5.4 minutes. Assembly101 uses the validation split, 202 coarse actions, TSM features, and anticipation beyond 12 minutes. 50Salads uses 5-fold cross-validation, 17 actions, I3D features, and anticipation up to 5.1 minutes.

On Breakfast, Mean MoC at $P=\alpha n_v$7 is $P=\alpha n_v$8 for $P=\alpha n_v$9, compared with MANTA’s $F=\beta n_v$0. At $F=\beta n_v$1, Mean MoC is $F=\beta n_v$2, compared with $F=\beta n_v$3. Top-1 MoC at $F=\beta n_v$4 is $F=\beta n_v$5, compared with $F=\beta n_v$6; at $F=\beta n_v$7, it is $F=\beta n_v$8, compared with $F=\beta n_v$9.

On Assembly101, Mean MoC at $\alpha \in \{0.2,0.3\}$0 is $\alpha \in \{0.2,0.3\}$1, compared with MANTA’s $\alpha \in \{0.2,0.3\}$2. At $\alpha \in \{0.2,0.3\}$3, Mean MoC is $\alpha \in \{0.2,0.3\}$4, compared with $\alpha \in \{0.2,0.3\}$5. Top-1 MoC at $\alpha \in \{0.2,0.3\}$6 is $\alpha \in \{0.2,0.3\}$7, compared with $\alpha \in \{0.2,0.3\}$8; at $\alpha \in \{0.2,0.3\}$9, it is $\beta \in \{0.1,0.2,0.3,0.5\}$0, compared with $\beta \in \{0.1,0.2,0.3,0.5\}$1.

On 50Salads, Mean MoC at $\beta \in \{0.1,0.2,0.3,0.5\}$2 is $\beta \in \{0.1,0.2,0.3,0.5\}$3, compared with MANTA’s $\beta \in \{0.1,0.2,0.3,0.5\}$4. At $\beta \in \{0.1,0.2,0.3,0.5\}$5, Mean MoC is $\beta \in \{0.1,0.2,0.3,0.5\}$6, compared with $\beta \in \{0.1,0.2,0.3,0.5\}$7. Top-1 MoC at $\beta \in \{0.1,0.2,0.3,0.5\}$8 is $\beta \in \{0.1,0.2,0.3,0.5\}$9, compared with $S=25$0; at $S=25$1, it is $S=25$2, compared with $S=25$3.

The paper also reports additional evidence on EK-100 short-term anticipation: Testra+MixMamba achieves overall verb/noun/action recall@5 of $S=25$4, compared with $S=25$5 for an attention baseline and $S=25$6 for a Mamba baseline.

The efficiency comparison is reported for Breakfast with $S=25$7, $S=25$8, and 25 samples, on a per-video basis:

Model	Params / Memory / Time	Top-1 MoC
GTDA	3.9M / 19.2GB / 71.8s	48.9
Transformer (15 blocks)	1.2M / 11.3GB / 15.6s	48.8
Transformer (18 blocks)	1.4M / 13.2GB / 18.2s	50.3
MANTA	1.4M / 10.2GB / 1.1s	52.7
Large MLP for $S=25$9	8.0M / 38.7GB / 137.4s	47.4
Query-Key for $\alpha$00	2.3M / 22.7GB / 24.4s	52.3
MixANT ($\alpha$01)	1.6M / 10.9GB / 1.7s	54.1

These results support the paper’s central empirical claim: hard-routing the forget gate preserves the linear-time scan and adds only small overhead relative to MANTA, while naive realizations of input-dependent $\alpha$02 are substantially more expensive. The reported expert-scaling analysis further notes a small increase in speed cost from 1 to 2 experts and minimal additional overhead beyond that, which is presented as evidence for the practicality of $\alpha$03.

5. Ablations, qualitative analyses, and theoretical observations

The ablation studies isolate several architectural choices that materially affect performance (Wasim et al., 14 Sep 2025). The number of experts $\alpha$04 improves performance from $\alpha$05 (equivalent to Mamba) to a peak around $\alpha$06, after which accuracy declines because of sparse gradient flow and reduced usage. The number of initial static blocks $\alpha$07 is also sensitive: Top-1 values across $\alpha$08 are $\alpha$09, and $\alpha$10 is used in the main configuration. The paper interprets this as a trade-off: too few static blocks harm early feature extraction, whereas too many reduce adaptive capacity.

Router design is another decisive factor. A unified router shared by forward and backward scans outperforms independent routers. Conditioning the router on observed frames only also outperforms conditioning on observed-plus-future inputs, because zero-padded future frames dilute the routing signal. Load balancing with $\alpha$11 yields the best accuracy and avoids second-expert dominance.

The paper’s analytical discussion centers on expressivity and controllability. Making $\alpha$12 input-dependent through MoE increases the capacity of temporal memory control because each $\alpha$13 can specialize to a distinct decay or propagation regime. At the same time, MixANT does not impose explicit stability constraints such as spectral radius bounds on $\alpha$14. Empirical training is reported as stable without such constraints. The authors note that, if necessary, stability could be further encouraged through spectral normalization or diagonal-plus-low-rank parameterizations. This suggests a separation between the method’s empirical effectiveness and formal stability control: the former is demonstrated directly, while the latter remains a possible extension rather than a built-in guarantee.

The qualitative analyses report that, on Breakfast videos involving pancake, sandwich, and coffee preparation, MixANT predictions align better with ground truth temporally than MANTA and produce diverse yet plausible stochastic samples. The paper also reports that t-SNE visualizations of expert selections cluster by high-level activities such as “making salad” and “making tea,” even though training is performed on atomic actions. A plausible implication is that expert routing is capturing semantically structured temporal regimes rather than only low-level duration statistics.

6. Relation to adjacent model families, limitations, and future directions

MixANT is positioned against several neighboring model classes (Wasim et al., 14 Sep 2025). Relative to Transformers, the comparison emphasized in the paper is complexity: attention is quadratic in sequence length, which limits long-horizon dense anticipation, whereas MixANT retains near-linear SSM scans while improving accuracy and inference time. Relative to S4/S6 and vanilla Mamba, the key distinction is selectivity: S4/S6 offer efficient long-sequence modeling, Mamba adds input-dependent $\alpha$15, $\alpha$16, and $\alpha$17, and MixANT extends this progression by making the forget gate $\alpha$18 input-dependent through MoE routing. The paper explicitly characterizes this as the first such modification inside Mamba-like blocks.

The relation to other MoE variants around Mamba is narrower. Prior MoE formulations are described as targeting MLP components outside SSM blocks or retaining a shared $\alpha$19. MixANT’s novelty is therefore not the presence of experts per se, but the routing of the forget gate itself. Within action anticipation, the paper contrasts deterministic dense methods, which output a single future, with diffusion-based stochastic approaches such as GTDA and DiffAnt. MANTA is the immediate antecedent because it introduced Mamba into diffusion for efficient stochastic dense anticipation; MixANT is presented as an improvement over MANTA through observation-dependent memory propagation.

The reported limitations are correspondingly specific. Router collapse can occur if load balancing is insufficient, leading to underutilized experts and reduced generalization. Performance is sensitive to the choice of $\alpha$20 and $\alpha$21. Domain transfer is also identified as a concern: expert specializations learned on cooking or assembly data may not transfer directly to different domains without re-routing or retraining.

The future directions named in the paper follow directly from those limitations. They include stability-aware parameterizations of $\alpha$22, soft routing or temperature annealing, modality-aware routing for multi-modal inputs, hierarchical experts, per-time-step routing when computation permits, and uncertainty-aware decoders beyond the reported $\alpha$23 reconstruction objective, such as class-conditional diffusion losses and calibration-oriented designs.

7. Homonymous usage in music source separation

An earlier and conceptually separate use of the term “MixANT” appears in CatNet for music source separation, where it abbreviates mix-audio augmentation rather than a state-space architecture (Song et al., 2021). In that formulation, the augmentation randomly mixes multiple audio segments drawn from the same source stem. If $\alpha$24 are $\alpha$25 segments from one source class, the augmented stem is

$\alpha$26

For multi-stem separation, the mixture presented to the network is

$\alpha$27

where $\alpha$28 indexes stems such as vocals, drums, bass, and other. Because summation is intra-source, the target remains within the same stem identity; no label interpolation is used.

This augmentation is applied to 3-second waveform segments sampled uniformly at random from the same stem’s training pool. The paper describes direct summation, does not specify mixing coefficients or amplitude normalization, and treats MixANT as effectively always-on in the reported “+ aug” runs. It is explicitly distinguished from mixup and SpecAugment: mixup uses convex combinations with interpolated labels, whereas MixANT uses intra-source waveform summation; SpecAugment applies time-frequency masking, whereas MixANT operates directly on raw waveforms.

Within CatNet, MixANT is integrated into a two-branch end-to-end system consisting of a spectrogram UNet branch and a time-domain WavUNet branch. The spectrogram pathway computes STFTs inside the network with fixed differentiable 1D convolution kernels using a Hann window of size $\alpha$29 and hop $\alpha$30, predicts a spectrogram mask, reconstructs with mixture phase, and returns to waveform through differentiable iSTFT. The time-domain branch processes the waveform directly with 1D convolutions. The two outputs are fused by time-domain summation, and training uses an $\alpha$31 waveform loss.

On MUSDB18, CatNet + MixANT reports a median vocals SDR of $\alpha$32 dB, compared with $\alpha$33 dB for MMDenseNet. In the ablation table, MixANT improves vocals SDR for both WavUNet and CatNet, although bass shows mixed effects. The two meanings of MixANT are therefore technically unrelated beyond the acronym: one is a routed forget-gate mechanism for stochastic dense action anticipation, and the other is an intra-source audio-mixing augmentation for music source separation.