---
title: 'Mamba-OTR: Online Take & Release Detection'
url: https://www.emergentmind.com/topics/mamba-otr
type: topic
---

# Mamba-OTR: Online Take & Release Detection

Searching arXiv for the cited Mamba-OTR paper and closely related Mamba-based vision work for grounding.
Mamba-OTR is a Mamba-based method for **online detection of the end point of “take” and “release” actions** in long, untrimmed egocentric video. It is formulated in the ODAE-style setting in which, at each current time \(t'\), the system may use only frames observed up to \(t'\) and must decide whether the current frame is the **last frame** of a take or release action; if so, it emits a single prediction with class label and confidence, otherwise it outputs background [2507.16342]. The method is designed around three constraints that define the task: temporally sparse positive labels, strict temporal precision, and causal online deployment. Its main technical combination is a Mamba temporal backbone, **focal loss** for class imbalance, and a **Fixed Window regularization** scheme intended to align training behavior with point-level evaluation. Reported results on the EPIC-KITCHENS-100 take/release subset include **45.48 mp-mAP** in sliding-window mode and **43.35 mp-mAP** in streaming mode, compared with **20.32** for a vanilla transformer and **25.16** for a vanilla Mamba baseline under the cited comparison setting [2507.16342].

## 1. Task definition and problem setting

Mamba-OTR addresses **Online Take and Release (OTR)** detection in untrimmed egocentric video. The task is not generic action recognition over clips, but **precise point detection**: each ground-truth action is represented as a single endpoint annotation rather than a temporal segment. The paper defines each action as
\[
a = (c, t),
\]
where \(c\) is the action class and \(t\) is the end timestamp, and each prediction as
\[
\hat{a} = (\hat{c}, \hat{t}, s),
\]
where \(\hat{c}\) is the predicted class, \(\hat{t}\) the predicted timestamp, and \(s\) the confidence score [2507.16342].

The difficulty of the task follows directly from the labeling protocol. Only the **ending frame** of a take or release action is positive; all other frames are negative or background. In long first-person videos this yields **extreme class imbalance**, because one positive endpoint is surrounded by many negative frames. The task also requires high temporal precision: predictions that are early, late, or duplicated around the true endpoint are penalized. In the paper’s use of “online,” predictions at time \(t'\) may depend only on observations up to \(t'\), and the intended deployment is real-time operation on full untrimmed videos without expensive sliding-window recomputation or long memory buffers [2507.16342].

Experiments are conducted on a curated subset of **EPIC-KITCHENS-100** containing only two action classes: **take** and **release**. Features are sampled at **4 fps**, so the training clip lengths considered correspond to concrete durations: 4 frames = 1 second, 8 frames = 2 seconds, 12 frames = 3 seconds, 20 frames = 5 seconds, and 40 frames = 10 seconds [2507.16342]. This framing makes Mamba-OTR a method for causal, frame-level decision making in egocentric video rather than a generic temporal classifier.

## 2. Model architecture and temporal processing

All models in the paper consume pre-extracted visual features and output framewise predictions over three classes: **take**, **release**, and **background**. The final Mamba-OTR architecture is an optimized version of a Mamba temporal model. It ingests linearly projected frame features, passes them through stacked Mamba layers, and applies a classification head to produce per-frame class probabilities [2507.16342].

The best reported Mamba-OTR configuration uses **3 Mamba layers**, **20 input frames** during training, and **Fixed Window regularization** with window size **4 frames**. The input features are inherited from prior work and extracted “from [Shou et al. 2018],” with feature extraction time excluded from temporal-model timing comparisons. The model is trained on short fixed-length snippets but evaluated both in sliding-window mode and in true streaming mode [2507.16342].

The architectural comparison in the paper includes three temporal baselines. **TeSTra** is described as a transformer with a dual-memory mechanism; **Transformer** is a simplified TeSTra using self-attention over a short temporal window; and **Mamba** is a temporal model that linearly projects frame features, feeds them through stacked Mamba layers, then uses a classification head. Mamba-OTR differs from a vanilla Mamba baseline not by introducing a new state-space recurrence, but by combining the temporal backbone with focal loss and a metric-aligned regularization scheme [2507.16342].

A central design claim is that Mamba is particularly suitable because it permits **parallelizable training on short clips** while supporting **recurrent inference over long sequences**. This allows Mamba-OTR to be trained on short snippets for computational convenience yet process full-length videos in streaming mode by carrying forward recurrent state without resetting at every chunk [2507.16342]. This is the core distinction from window-bound transformer baselines.

## 3. Training objective, label imbalance, and regularization

The overall training objective is
\[
\mathcal{L} = \sum_i^m FL(h(x_i)) + \lambda R,
\]
where \(x_i\) is a dataset example, \(h\) is the model, \(FL\) is focal loss, \(R\) is a regularization term, and \(\lambda = 0.01\) [2507.16342]. The paper explicitly attributes major importance to focal loss because the endpoint labels are extremely sparse. In its reported ablation, adding focal loss changes performance as follows: TeSTra \(20.10 \rightarrow 36.59\), Transformer \(20.32 \rightarrow 38.48\), and Mamba \(25.16 \rightarrow 41.01\) [2507.16342].

The paper evaluates three regularizers intended to better match the **point-level mAP** metric, which greedily assigns at most one prediction to each ground-truth endpoint and penalizes nearby duplicates. The first is entropy minimization,
\[
\mathcal{R}_{entropy} = -\sum_{i} p_i \log(p_i),
\]
which encourages confident predictions but is temporally agnostic. The second is Sliding Window regularization,
\[
\mathcal{R}_{SW} = \sum_{f} \left( \sum_{i=f - \lfloor w/2 \rfloor}^{f + \lfloor w/2 \rfloor} p_i \right),
\]
which suppresses multiple activations within local neighborhoods but applies everywhere, including background regions. The third, and preferred, method is Fixed Window regularization,
\[
\mathcal{R}_{FW} = \sum_{g \in G} \left( \sum_{i=g - \lfloor w/2 \rfloor}^{g + \lfloor w/2 \rfloor} p_i \right),
\]
where \(G\) is the set of ground-truth action-ending frames [2507.16342].

Fixed Window regularization is the distinctive training idea in Mamba-OTR. Because the evaluation metric rewards **one** correct point and treats additional nearby detections as false positives, the regularizer penalizes distributed probability mass around each ground-truth endpoint and encourages a single temporally concentrated peak. Empirically, with 3 layers and 20-frame clips, performance changes from **42.98** with no regularization to **43.42** with Sliding Window and **45.48** with Fixed Window [2507.16342].

Window size also matters. For Fixed Window regularization, the reported ablation is: **4 frames: 45.48**, **12 frames: 44.05**, **20 frames: 44.40**. The selected configuration is therefore a **4-frame** window, corresponding to **1 second** at 4 fps [2507.16342].

## 4. Inference modes and online deployment behavior

Mamba-OTR is evaluated in two inference modes. In **sliding-window inference**, the model is applied over overlapping short windows, as is typical for short-context temporal models. In **streaming inference**, the model processes the full video one frame at a time, carrying forward hidden state without chunking or repeated recomputation [2507.16342].

This distinction is decisive in the paper’s argument. A transformer trained on fixed windows is described as learning positional interactions tied to that training configuration, whereas Mamba’s recurrent state can continue beyond the training clip length more naturally. The streaming update behavior is described as: initialize hidden state, feed each incoming frame feature, update the Mamba recurrent state, emit the prediction for the current frame, and continue without resetting state [2507.16342].

The final comparison reported for sliding versus streaming is stark. Mamba-OTR declines from **45.48** in sliding-window mode to **43.35** in streaming mode, whereas the strongest transformer variant falls from **38.96** to **0.04** [2507.16342]. The paper presents this as direct evidence that Mamba’s recurrence enables generalization from short clip training to long untrimmed streaming inference.

The timing numbers reported in the detailed table are:
- **Mamba-OTR**: 0.14 s per video, 8 ns per frame
- **Transformer**: 0.14 s per video, 8 ns per frame

The paper notes that both add minimal overhead over the backbone, but emphasizes that the practical advantage of Mamba is not only raw temporal-head runtime under this timing table; it is the preservation of high accuracy in streaming mode [2507.16342]. The abstract additionally reports a comparison against a “standard Transformer module” in which Mamba-OTR achieves mp-mAP **51.76** with mean inference time **0.14 s**, versus **20.32** and **0.28 s** for the Transformer under that reported setting [2507.16342]. The paper explicitly presents multiple settings, so these values coexist with the 45.48/43.35 sliding/streaming results.

## 5. Evaluation protocol and empirical results

The evaluation metric is **point-level mean Average Precision (p-mAP)**, averaged over tolerance thresholds \(\phi \in \{1,2,\dots,10\}\) seconds to obtain **mp-mAP**. A prediction matches a ground truth if the class matches and
\[
\delta = |\hat{t} - t|
\]
satisfies \(\delta \le \phi\). Matching is greedy in descending confidence order, and each prediction and each ground-truth point can be matched at most once [2507.16342].

The main reported ablations can be summarized as follows.

| Setting | Result |
|---|---:|
| Vanilla Transformer | 20.32 |
| Vanilla Mamba | 25.16 |
| Mamba + focal loss | 41.01 |
| Mamba + focal loss + no regularization | 42.98 |
| Mamba + focal loss + Sliding Window | 43.42 |
| Mamba-OTR (Fixed Window) | 45.48 |

These figures refer to the cited ablation setting on the EPIC-KITCHENS-100 take/release subset [2507.16342].

Depth and clip-length studies further specify the selected design. With focal loss and 4 training frames, the number-of-layer ablation yields **41.01** for 2 layers, **41.63** for 3 layers, and **40.52** for 4 layers; hence **3 layers** is chosen. With 3 layers, the training-clip-length ablation yields **41.63** for 4 frames, **41.84** for 8 frames, **42.47** for 12 frames, **42.98** for 20 frames, and **41.77** for 40 frames; thus **20 frames / 5 s** is selected [2507.16342].

The final sliding and streaming comparison is:

| Model | Sliding | Streaming |
|---|---:|---:|
| Mamba-OTR | **45.48** | **43.35** |
| Transformer | 38.96 | **0.04** |

This comparison is one of the paper’s strongest empirical claims because it directly tests train/test mismatch under true streaming inference [2507.16342].

Qualitative analysis in the paper reports three progressive behaviors: Mamba without focal loss produces noisy, poorly correlated predictions; focal loss improves localization substantially; and Fixed Window regularization further concentrates predictions into a single well-localized spike per action [2507.16342]. This aligns with the formal goal of point detection rather than dense clip classification.

## 6. Interpretation, significance, and limitations

Mamba-OTR’s significance lies in the conjunction of three task-aligned elements: a recurrent Mamba temporal backbone, imbalance-aware focal loss, and a metric-aware Fixed Window regularizer. The backbone addresses the mismatch between short training windows and full-video streaming inference; focal loss addresses the extreme rarity of positive endpoints; and Fixed Window regularization addresses the fact that p-mAP penalizes duplicate detections around a single event [2507.16342].

A plausible implication is that Mamba-OTR should be understood less as a fundamentally new state-space model than as a carefully matched solution to a point-detection task with causal deployment constraints. The paper is explicit that the contribution is not a new selective-scan recurrence rule; rather, it is the adaptation of stacked Mamba layers to OTR together with loss and regularization design [2507.16342].

The paper also leaves several limitations visible. It does not provide extensive optimizer hyperparameters such as learning rate schedules beyond a few core settings, nor detailed data augmentations. Internal Mamba details are not specialized mathematically for OTR, and improvements are attributed largely to recurrence plus training design rather than to a novel sequence operator. Results are confined to a **two-class take/release subset of EPIC-KITCHENS-100**, so broader generalization to larger action vocabularies is not established [2507.16342].

In a broader Mamba context, Mamba-OTR exemplifies a pattern also seen in contemporaneous Mamba-based vision systems: Mamba is not merely substituted for attention as a generic backbone, but used where linear-time recurrent state propagation is operationally advantageous. This suggests a wider role for Mamba in online visual decision problems, particularly when inference must extend beyond the train-time clip length without repeated recomputation.

Source: https://www.emergentmind.com/topics/mamba-otr