Mamba-OTR: Online Take & Release Detection

• Mamba-OTR is a Mamba-based method that detects the last frame of take and release actions in untrimmed egocentric videos with high temporal precision.
• It employs a recurrent backbone combined with focal loss and Fixed Window regularization to address extreme class imbalance and enforce single-point predictions.
• Empirical results on EPIC-KITCHENS-100 show superior performance with 45.48 mp-mAP in sliding-window mode and robust streaming inference compared to transformer baselines.

Searching arXiv for the cited Mamba-OTR paper and closely related Mamba-based vision work for grounding. Mamba-OTR is a Mamba-based method for online detection of the end point of “take” and “release” actions in long, untrimmed egocentric video. It is formulated in the ODAE-style setting in which, at each current time $t'$, the system may use only frames observed up to $t'$ and must decide whether the current frame is the last frame of a take or release action; if so, it emits a single prediction with class label and confidence, otherwise it outputs background (Catinello et al., 22 Jul 2025). The method is designed around three constraints that define the task: temporally sparse positive labels, strict temporal precision, and causal online deployment. Its main technical combination is a Mamba temporal backbone, focal loss for class imbalance, and a Fixed Window regularization scheme intended to align training behavior with point-level evaluation. Reported results on the EPIC-KITCHENS-100 take/release subset include 45.48 mp-mAP in sliding-window mode and 43.35 mp-mAP in streaming mode, compared with 20.32 for a vanilla transformer and 25.16 for a vanilla Mamba baseline under the cited comparison setting (Catinello et al., 22 Jul 2025).

1. Task definition and problem setting

Mamba-OTR addresses Online Take and Release (OTR) detection in untrimmed egocentric video. The task is not generic action recognition over clips, but precise point detection: each ground-truth action is represented as a single endpoint annotation rather than a temporal segment. The paper defines each action as

$a = (c, t),$

where $c$ is the action class and $t$ is the end timestamp, and each prediction as

$\hat{a} = (\hat{c}, \hat{t}, s),$

where $\hat{c}$ is the predicted class, $\hat{t}$ the predicted timestamp, and $s$ the confidence score (Catinello et al., 22 Jul 2025).

The difficulty of the task follows directly from the labeling protocol. Only the ending frame of a take or release action is positive; all other frames are negative or background. In long first-person videos this yields extreme class imbalance, because one positive endpoint is surrounded by many negative frames. The task also requires high temporal precision: predictions that are early, late, or duplicated around the true endpoint are penalized. In the paper’s use of “online,” predictions at time $t'$ may depend only on observations up to $t'$0, and the intended deployment is real-time operation on full untrimmed videos without expensive sliding-window recomputation or long memory buffers (Catinello et al., 22 Jul 2025).

Experiments are conducted on a curated subset of EPIC-KITCHENS-100 containing only two action classes: take and release. Features are sampled at 4 fps, so the training clip lengths considered correspond to concrete durations: 4 frames = 1 second, 8 frames = 2 seconds, 12 frames = 3 seconds, 20 frames = 5 seconds, and 40 frames = 10 seconds (Catinello et al., 22 Jul 2025). This framing makes Mamba-OTR a method for causal, frame-level decision making in egocentric video rather than a generic temporal classifier.

2. Model architecture and temporal processing

All models in the paper consume pre-extracted visual features and output framewise predictions over three classes: take, release, and background. The final Mamba-OTR architecture is an optimized version of a Mamba temporal model. It ingests linearly projected frame features, passes them through stacked Mamba layers, and applies a classification head to produce per-frame class probabilities (Catinello et al., 22 Jul 2025).

The best reported Mamba-OTR configuration uses 3 Mamba layers, 20 input frames during training, and Fixed Window regularization with window size 4 frames. The input features are inherited from prior work and extracted “from [Shou et al. 2018],” with feature extraction time excluded from temporal-model timing comparisons. The model is trained on short fixed-length snippets but evaluated both in sliding-window mode and in true streaming mode (Catinello et al., 22 Jul 2025).

The architectural comparison in the paper includes three temporal baselines. TeSTra is described as a transformer with a dual-memory mechanism; Transformer is a simplified TeSTra using self-attention over a short temporal window; and Mamba is a temporal model that linearly projects frame features, feeds them through stacked Mamba layers, then uses a classification head. Mamba-OTR differs from a vanilla Mamba baseline not by introducing a new state-space recurrence, but by combining the temporal backbone with focal loss and a metric-aligned regularization scheme (Catinello et al., 22 Jul 2025).

A central design claim is that Mamba is particularly suitable because it permits parallelizable training on short clips while supporting recurrent inference over long sequences. This allows Mamba-OTR to be trained on short snippets for computational convenience yet process full-length videos in streaming mode by carrying forward recurrent state without resetting at every chunk (Catinello et al., 22 Jul 2025). This is the core distinction from window-bound transformer baselines.

3. Training objective, label imbalance, and regularization

The overall training objective is

$t'$1

where $t'$2 is a dataset example, $t'$3 is the model, $t'$4 is focal loss, $t'$5 is a regularization term, and $t'$6 (Catinello et al., 22 Jul 2025). The paper explicitly attributes major importance to focal loss because the endpoint labels are extremely sparse. In its reported ablation, adding focal loss changes performance as follows: TeSTra $t'$7, Transformer $t'$8, and Mamba $t'$9 (Catinello et al., 22 Jul 2025).

The paper evaluates three regularizers intended to better match the point-level mAP metric, which greedily assigns at most one prediction to each ground-truth endpoint and penalizes nearby duplicates. The first is entropy minimization,

$a = (c, t),$0

which encourages confident predictions but is temporally agnostic. The second is Sliding Window regularization,

$a = (c, t),$1

which suppresses multiple activations within local neighborhoods but applies everywhere, including background regions. The third, and preferred, method is Fixed Window regularization,

$a = (c, t),$2

where $a = (c, t),$3 is the set of ground-truth action-ending frames (Catinello et al., 22 Jul 2025).

Fixed Window regularization is the distinctive training idea in Mamba-OTR. Because the evaluation metric rewards one correct point and treats additional nearby detections as false positives, the regularizer penalizes distributed probability mass around each ground-truth endpoint and encourages a single temporally concentrated peak. Empirically, with 3 layers and 20-frame clips, performance changes from 42.98 with no regularization to 43.42 with Sliding Window and 45.48 with Fixed Window (Catinello et al., 22 Jul 2025).

Window size also matters. For Fixed Window regularization, the reported ablation is: 4 frames: 45.48, 12 frames: 44.05, 20 frames: 44.40. The selected configuration is therefore a 4-frame window, corresponding to 1 second at 4 fps (Catinello et al., 22 Jul 2025).

4. Inference modes and online deployment behavior

Mamba-OTR is evaluated in two inference modes. In sliding-window inference, the model is applied over overlapping short windows, as is typical for short-context temporal models. In streaming inference, the model processes the full video one frame at a time, carrying forward hidden state without chunking or repeated recomputation (Catinello et al., 22 Jul 2025).

This distinction is decisive in the paper’s argument. A transformer trained on fixed windows is described as learning positional interactions tied to that training configuration, whereas Mamba’s recurrent state can continue beyond the training clip length more naturally. The streaming update behavior is described as: initialize hidden state, feed each incoming frame feature, update the Mamba recurrent state, emit the prediction for the current frame, and continue without resetting state (Catinello et al., 22 Jul 2025).

The final comparison reported for sliding versus streaming is stark. Mamba-OTR declines from 45.48 in sliding-window mode to 43.35 in streaming mode, whereas the strongest transformer variant falls from 38.96 to 0.04 (Catinello et al., 22 Jul 2025). The paper presents this as direct evidence that Mamba’s recurrence enables generalization from short clip training to long untrimmed streaming inference.

The timing numbers reported in the detailed table are:

• Mamba-OTR: 0.14 s per video, 8 ns per frame
• Transformer: 0.14 s per video, 8 ns per frame

The paper notes that both add minimal overhead over the backbone, but emphasizes that the practical advantage of Mamba is not only raw temporal-head runtime under this timing table; it is the preservation of high accuracy in streaming mode (Catinello et al., 22 Jul 2025). The abstract additionally reports a comparison against a “standard Transformer module” in which Mamba-OTR achieves mp-mAP 51.76 with mean inference time 0.14 s, versus 20.32 and 0.28 s for the Transformer under that reported setting (Catinello et al., 22 Jul 2025). The paper explicitly presents multiple settings, so these values coexist with the 45.48/43.35 sliding/streaming results.

5. Evaluation protocol and empirical results

The evaluation metric is point-level mean Average Precision (p-mAP), averaged over tolerance thresholds $a = (c, t),$4 seconds to obtain mp-mAP. A prediction matches a ground truth if the class matches and

$a = (c, t),$5

satisfies $a = (c, t),$6. Matching is greedy in descending confidence order, and each prediction and each ground-truth point can be matched at most once (Catinello et al., 22 Jul 2025).

The main reported ablations can be summarized as follows.

Setting	Result
Vanilla Transformer	20.32
Vanilla Mamba	25.16
Mamba + focal loss	41.01
Mamba + focal loss + no regularization	42.98
Mamba + focal loss + Sliding Window	43.42
Mamba-OTR (Fixed Window)	45.48

These figures refer to the cited ablation setting on the EPIC-KITCHENS-100 take/release subset (Catinello et al., 22 Jul 2025).

Depth and clip-length studies further specify the selected design. With focal loss and 4 training frames, the number-of-layer ablation yields 41.01 for 2 layers, 41.63 for 3 layers, and 40.52 for 4 layers; hence 3 layers is chosen. With 3 layers, the training-clip-length ablation yields 41.63 for 4 frames, 41.84 for 8 frames, 42.47 for 12 frames, 42.98 for 20 frames, and 41.77 for 40 frames; thus 20 frames / 5 s is selected (Catinello et al., 22 Jul 2025).

The final sliding and streaming comparison is:

Model	Sliding	Streaming
Mamba-OTR	45.48	43.35
Transformer	38.96	0.04

This comparison is one of the paper’s strongest empirical claims because it directly tests train/test mismatch under true streaming inference (Catinello et al., 22 Jul 2025).

Qualitative analysis in the paper reports three progressive behaviors: Mamba without focal loss produces noisy, poorly correlated predictions; focal loss improves localization substantially; and Fixed Window regularization further concentrates predictions into a single well-localized spike per action (Catinello et al., 22 Jul 2025). This aligns with the formal goal of point detection rather than dense clip classification.

6. Interpretation, significance, and limitations

Mamba-OTR’s significance lies in the conjunction of three task-aligned elements: a recurrent Mamba temporal backbone, imbalance-aware focal loss, and a metric-aware Fixed Window regularizer. The backbone addresses the mismatch between short training windows and full-video streaming inference; focal loss addresses the extreme rarity of positive endpoints; and Fixed Window regularization addresses the fact that p-mAP penalizes duplicate detections around a single event (Catinello et al., 22 Jul 2025).

A plausible implication is that Mamba-OTR should be understood less as a fundamentally new state-space model than as a carefully matched solution to a point-detection task with causal deployment constraints. The paper is explicit that the contribution is not a new selective-scan recurrence rule; rather, it is the adaptation of stacked Mamba layers to OTR together with loss and regularization design (Catinello et al., 22 Jul 2025).

The paper also leaves several limitations visible. It does not provide extensive optimizer hyperparameters such as learning rate schedules beyond a few core settings, nor detailed data augmentations. Internal Mamba details are not specialized mathematically for OTR, and improvements are attributed largely to recurrence plus training design rather than to a novel sequence operator. Results are confined to a two-class take/release subset of EPIC-KITCHENS-100, so broader generalization to larger action vocabularies is not established (Catinello et al., 22 Jul 2025).

In a broader Mamba context, Mamba-OTR exemplifies a pattern also seen in contemporaneous Mamba-based vision systems: Mamba is not merely substituted for attention as a generic backbone, but used where linear-time recurrent state propagation is operationally advantageous. This suggests a wider role for Mamba in online visual decision problems, particularly when inference must extend beyond the train-time clip length without repeated recomputation.