ActivityNet-RTL: Real-Time Temporal Localization

• ActivityNet-RTL is a research direction emphasizing real-time temporal localization through causal inference and low-latency event detection in streaming video.
• It integrates techniques from temporal action proposal and online dense captioning to improve event boundary accuracy and ensure irreversibility of predictions.
• The approach leverages efficient multi-scale temporal convolutions and novel streaming-aware metrics to address latency and partial observation challenges.

Searching arXiv for the cited ActivityNet-RTL-related papers to ground the article in current records. ActivityNet-RTL denotes ActivityNet’s real-time/temporal localization focus: finding where actions happen in time and making those detections usable at low latency. In the available literature, the term is used as a real-time or online setting on ActivityNet that emphasizes low-latency event processing and causal inference rather than a single formally standardized benchmark (Lin et al., 2017, Blanco-Fernández et al., 2024). Within this scope, two strands are especially salient. One is temporal action proposal and localization, where the principal concern is proposal quality for identifying action intervals in untrimmed video. The other is online dense captioning, where event detection and language generation must operate causally on streams without future frames. Together, these works frame ActivityNet-RTL as a family of temporal inference problems defined by causality, partial observations, temporal overlap accuracy, and practical latency constraints (Lin et al., 2017, Blanco-Fernández et al., 2024).

1. Terminological scope and historical position

ActivityNet-RTL is not introduced in the cited works as a standalone official benchmark with a single immutable protocol. The 2017 submission to the ActivityNet Challenge addresses the temporal action proposal task and temporal action localization task, and explicitly treats proposal quality as the main bottleneck because “the accuracy in action classification task is already very high (nearly 90% in ActivityNet dataset)” (Lin et al., 2017). The 2024 work on Live Video Captioning states that it does not define a separate “RTL” benchmark, but notes that its formulation aligns with and extends the spirit of ActivityNet-RTL by enforcing strict causality, operating with partial observations and temporal anticipation, introducing streaming-aware evaluation, reporting throughput and latency for an online demo, and validating on the ActivityNet Captions dataset (Blanco-Fernández et al., 2024).

This usage establishes ActivityNet-RTL as an umbrella designation for ActivityNet-based research that prioritizes temporal localization under real-time or online constraints. A plausible implication is that the term functions less as a fixed dataset name than as a research orientation linking temporal localization, streaming inference, and low-latency evaluation.

2. Core problem formulations

In the 2017 challenge setting, the focus is temporal action proposal and temporal action localization. The proposal task is evaluated by Area under the AR–AN curve (AUC), where Average Recall is averaged over temporal Intersection-over-Union thresholds $\{0.50, 0.55, \ldots, 0.95\}$ and AR@AN fixes the average number of proposals per video (Lin et al., 2017). The localization task is evaluated by mean Average Precision averaged across the same tIoU threshold set (Lin et al., 2017). Temporal overlap is defined for segments $[s_1,e_1]$ and $[s_2,e_2]$ as

$$\text{tIoU} = \frac{\max(0, \min(e_1, e_2) - \max(s_1, s_2))}{\max(e_1, e_2) - \min(s_1, s_2)}.$$

In the 2024 online captioning setting, Live Video Captioning requires partial observations, causality, low latency, streaming constraints, and temporal anticipation (Blanco-Fernández et al., 2024). Predictions cannot be revised once emitted, a property termed “Irreversibility” in the source description (Blanco-Fernández et al., 2024). The model processes a video stream segmented into fixed-length chunks and emits event segments and captions online, again under temporal IoU-based localization accounting, but with additional streaming-aware caption metrics (Blanco-Fernández et al., 2024).

The conceptual continuity between the two formulations is that both are concerned with locating events in time, but the latter adds the requirement that the system act causally on a stream and produce language as well as boundaries. This suggests that ActivityNet-RTL has evolved from proposal-centric temporal localization toward broader online event understanding.

3. Proposal-centric temporal localization in ActivityNet 2017

The 2017 submission, "Temporal Convolution Based Action Proposal: Submission to ActivityNet 2017" (Lin et al., 2017), introduces a temporal convolutional anchor-based proposal network, Prop-SSAD, together with an actionness-based boundary refinement mechanism based on TAG. The backbone is a 1D temporal convolutional stack that produces multi-scale temporal feature maps. As in single-shot detectors for images such as YOLO and SSD, Prop-SSAD associates anchors to these feature maps and predicts proposal scores for them (Lin et al., 2017).

Multi-scale temporal anchor layers are attached at seven feature map scales with anchor lengths $\{1,2,4,8,16,32,64\}$ measured in snippets (Lin et al., 2017). Each anchor is a candidate temporal segment defined by center time and duration. In SSAD, the detector predicts category, overlap score, and boundary offsets for each anchor; in Prop-SSAD, only the overlap score is trained and used to rank proposals, while boundaries are later refined by TAG (Lin et al., 2017). The primary architectural change from SSAD to Prop-SSAD is the expansion of anchor layers from three scales $(4,8,16)$ to seven scales $(1\text{–}64)$ (Lin et al., 2017).

The feature pipeline uses two-stream features only, with no C3D, I3D, or external data (Lin et al., 2017). Videos are segmented into non-overlapping 16-frame snippets. For appearance, the central frame of a 16-frame non-overlapping snippet is processed by a spatial network, and the “Flatten-673” layer of a ResNet provides the appearance vector. For motion, optical flow from 6 frames around the snippet center is processed by a temporal BN-Inception network, and the “global-pool” layer provides the motion vector. These are concatenated into a 3072-D snippet feature (Lin et al., 2017). The sequence of snippet-level features is linearly interpolated to a fixed length of 256 to standardize TCN input (Lin et al., 2017).

Proposal generation proceeds in two stages. Prop-SSAD performs anchor-based scoring via the TCN. For each anchor $a$ at scale $L$, the network outputs an overlap score indicating how well it matches an action segment with high tIoU (Lin et al., 2017). TAG is re-implemented as a 1-hidden-layer MLP that predicts actionness per snippet; grouping with multiple thresholds produces proposals (Lin et al., 2017). For boundary refinement, for each TAG proposal $p_t$, the method computes its IoU with all Prop-SSAD proposals $[s_1,e_1]$0. If $[s_1,e_1]$1, the Prop-SSAD proposal’s boundaries are replaced with the TAG proposal’s boundaries, yielding the refined proposal set $[s_1,e_1]$2 (Lin et al., 2017). The paper does not detail NMS; refinement is identified as the key postprocess (Lin et al., 2017).

The training setup is deliberately narrow. Prop-SSAD uses only an overlap loss to train proposal scoring; no category classification or boundary regression loss is used in this submission, and the exact overlap loss form is not specified (Lin et al., 2017). TAG is trained as a 1-hidden-layer MLP for snippet actionness using supervision from ground-truth segments, implemented similarly to the original TAG approach (Lin et al., 2017). The absence of boundary regression inside Prop-SSAD is presented as a limitation, and the paper notes future work to train the whole pipeline end-to-end (Lin et al., 2017).

4. Evaluation protocols and empirical behavior

For ActivityNet temporal action proposal on validation, Prop-SSAD reports AR@10 $[s_1,e_1]$3, AR@100 $[s_1,e_1]$4, and AUC $[s_1,e_1]$5 (Lin et al., 2017). Refined Prop-SSAD with TAG boundary replacement reports AR@10 $[s_1,e_1]$6, AR@100 $[s_1,e_1]$7, and AUC $[s_1,e_1]$8 (Lin et al., 2017). The source notes that the refinement step notably boosts recall at higher tIoU, improving overall AUC by approximately $[s_1,e_1]$9 points (Lin et al., 2017).

For ActivityNet temporal action localization on validation, the 2017 paper reports the following comparisons (Lin et al., 2017):

Method	mAP@{0.5, 0.75, 0.95, avg}
Shou et al. (CDC)	{43.83, 25.88, 0.21, 22.77}
Xiong et al. (TAG)	{39.12, 23.48, 5.49, 23.98}
Ours@1	{39.21, 25.37, 6.01, 25.17}
Ours@5	{42.57, 28.26, 6.83, 27.86}
Ours@10	{43.58, 28.95, 7.00, 28.56}
Ours@25	{44.14, 29.42, 7.07, 28.96}
Ours@100	{44.39, 29.65, 7.09, 29.17}

The mAP curve is described as saturating quickly, with most localization gains coming from the first few proposals, which the paper interprets as evidence that proposal precision matters more than sheer quantity in ActivityNet-RTL (Lin et al., 2017). For Task 4 testing, the paper lists comparison baselines on test average mAP—Xiong et al. at $[s_2,e_2]$0 and Zhao et al. at $[s_2,e_2]$1—and claims state of the art driven by proposal quality, but does not list its own test mAP number in the paper (Lin et al., 2017).

The 2024 LVC work introduces a different evaluation regime because offline metrics are described as inadequate for online dense captioning (Blanco-Fernández et al., 2024). It defines streaming metrics computed every $[s_2,e_2]$2 seconds. If $[s_2,e_2]$3 is the caption-similarity score for chunk $[s_2,e_2]$4, then

$[s_2,e_2]$5

and

$[s_2,e_2]$6

where $[s_2,e_2]$7 is the mean false-positive count over processed chunks (Blanco-Fernández et al., 2024). It also defines history-window variants hLS and hwLS (Blanco-Fernández et al., 2024). For temporal localization in the streaming setting, it reports recall and precision across IoU thresholds $[s_2,e_2]$8 (Blanco-Fernández et al., 2024).

Empirically, caption-quality scores under standard offline evaluation are low for online predictions; for example, the paper reports BLEU-4 values increasing from $[s_2,e_2]$9 at $$\text{tIoU} = \frac{\max(0, \min(e_1, e_2) - \max(s_1, s_2))}{\max(e_1, e_2) - \min(s_1, s_2)}.$$0 to $$\text{tIoU} = \frac{\max(0, \min(e_1, e_2) - \max(s_1, s_2))}{\max(e_1, e_2) - \min(s_1, s_2)}.$$1 at $$\text{tIoU} = \frac{\max(0, \min(e_1, e_2) - \max(s_1, s_2))}{\max(e_1, e_2) - \min(s_1, s_2)}.$$2, with METEOR from $$\text{tIoU} = \frac{\max(0, \min(e_1, e_2) - \max(s_1, s_2))}{\max(e_1, e_2) - \min(s_1, s_2)}.$$3 to $$\text{tIoU} = \frac{\max(0, \min(e_1, e_2) - \max(s_1, s_2))}{\max(e_1, e_2) - \min(s_1, s_2)}.$$4 and ROUGE-L from $$\text{tIoU} = \frac{\max(0, \min(e_1, e_2) - \max(s_1, s_2))}{\max(e_1, e_2) - \min(s_1, s_2)}.$$5 to $$\text{tIoU} = \frac{\max(0, \min(e_1, e_2) - \max(s_1, s_2))}{\max(e_1, e_2) - \min(s_1, s_2)}.$$6 (Blanco-Fernández et al., 2024). At $$\text{tIoU} = \frac{\max(0, \min(e_1, e_2) - \max(s_1, s_2))}{\max(e_1, e_2) - \min(s_1, s_2)}.$$7, temporal localization reaches recall average $$\text{tIoU} = \frac{\max(0, \min(e_1, e_2) - \max(s_1, s_2))}{\max(e_1, e_2) - \min(s_1, s_2)}.$$8 and precision average $$\text{tIoU} = \frac{\max(0, \min(e_1, e_2) - \max(s_1, s_2))}{\max(e_1, e_2) - \min(s_1, s_2)}.$$9, while at $\{1,2,4,8,16,32,64\}$0 it reaches recall average $\{1,2,4,8,16,32,64\}$1 and precision average $\{1,2,4,8,16,32,64\}$2 (Blanco-Fernández et al., 2024). By contrast, online LS and wLS at $\{1,2,4,8,16,32,64\}$3 reach BLEU-4 $\{1,2,4,8,16,32,64\}$4 and $\{1,2,4,8,16,32,64\}$5, METEOR $\{1,2,4,8,16,32,64\}$6 and $\{1,2,4,8,16,32,64\}$7, and ROUGE-L $\{1,2,4,8,16,32,64\}$8 and $\{1,2,4,8,16,32,64\}$9, respectively (Blanco-Fernández et al., 2024).

These results support two distinct but related claims in the literature: first, temporal proposal quality dominates localization performance in the 2017 challenge setting; second, offline metrics can underrepresent online performance when causality and timely emission are required (Lin et al., 2017, Blanco-Fernández et al., 2024).

5. Streaming captioning as an extension of the RTL setting

"Live Video Captioning" (Blanco-Fernández et al., 2024) extends the ActivityNet real-time orientation from temporal localization to joint localization and caption generation. A video stream $(4,8,16)$0 is segmented into fixed-length chunks $(4,8,16)$1 of $(4,8,16)$2 frames, with experiments using $(4,8,16)$3 at 30 fps, corresponding roughly to $(4,8,16)$4–$(4,8,16)$5 windows (Blanco-Fernández et al., 2024). Features are computed per chunk, and multi-scale temporal features are built using strided temporal convolutions with stride $(4,8,16)$6 and kernel $(4,8,16)$7 for multiple resolutions $(4,8,16)$8 (Blanco-Fernández et al., 2024).

The model uses a deformable transformer encoder-decoder with multi-scale deformable attention in the style of Deformable DETR (Blanco-Fernández et al., 2024). Decoder queries act as event proposals. For each chunk, the decoder generates $(4,8,16)$9 event queries; a localization head predicts segment center and length together with foreground confidence, a captioning head generates sentence tokens through an LSTM, and an event counter predicts the number of events (Blanco-Fernández et al., 2024). For each chunk, 10 query-level captions are consolidated by a weighted voting scheme, with votes weighted by the head’s confidence scores, to produce the final caption or captions for the chunk (Blanco-Fernández et al., 2024). Once emitted, predictions become immutable (Blanco-Fernández et al., 2024).

Training is based on Hungarian matching between predicted queries and ground-truth segments with a cost combining generalized IoU for localization and focal loss for classification (Blanco-Fernández et al., 2024). The total loss sums localization loss, focal classification loss, counting loss, and caption loss across decoder layers (Blanco-Fernández et al., 2024). The model can use any video encoder; the reported experiments use TSP features (Blanco-Fernández et al., 2024).

This extends ActivityNet-RTL in two respects documented by the source. First, it formalizes strict causality and irreversibility rather than post hoc localization from a full video. Second, it introduces streaming-aware evaluation with explicit false-positive penalties and optional history windows, which are absent from the 2017 proposal-centric challenge protocol (Blanco-Fernández et al., 2024).

6. Latency, efficiency, and systems considerations

The 2017 proposal paper does not report inference speed or model size, but it characterizes the design as RTL-friendly because it uses efficient 1D temporal convolutions over fixed-length sequences of 256 snippets, multi-scale anchors rather than recurrent passes or heavy proposal refinement, and a simple IoU-based boundary replacement with threshold $(1\text{–}64)$0 (Lin et al., 2017). The description further notes that resampling to 256 bounds sequence length and therefore bounds cost (Lin et al., 2017).

The 2024 LVC work reports explicit runtime behavior for a demo system. Average caption generation time is $(1\text{–}64)$1 with approximately $(1\text{–}64)$2 FPS UI throughput on a laptop GPU (Blanco-Fernández et al., 2024). For a 30 FPS source, the recommended chunk length to avoid frame loss is approximately 71 frames, corresponding to about $(1\text{–}64)$3 (Blanco-Fernández et al., 2024). The demo includes a memory parameter $(1\text{–}64)$4 that allows the captioning thread to consider the last $(1\text{–}64)$5 chunks to stabilize current predictions without revisiting past outputs (Blanco-Fernández et al., 2024).

These runtime descriptions show two distinct levels of maturity in the ActivityNet-RTL literature. The earlier work argues for efficiency primarily from architectural form, whereas the later work reports wall-clock latency and interface throughput directly. A plausible implication is that explicit latency reporting became more central once the task shifted from offline challenge submission to interactive streaming deployment.

7. Limitations, metric tensions, and open directions

Several limitations recur across the cited literature. In the 2017 proposal system, features, proposals, and classification are trained separately rather than end-to-end; kernel sizes, strides, dilations, and explicit receptive field calculations are not reported; no boundary regression loss is used in Prop-SSAD training; and runtime is not reported (Lin et al., 2017). The authors also critique the AR–AN evaluation because it weights proposal quantity, arguing that small-AN recall should be emphasized more, given that mAP depends mainly on top proposals (Lin et al., 2017).

In the 2024 streaming setting, the paper argues that standard offline dense-captioning metrics are inadequate because they assume access to the full video, filter predictions heavily by IoU thresholds, and do not capture temporal evolution or false positives in a streaming context (Blanco-Fernández et al., 2024). It shows that modified offline annotations obtained by chopping ground truth into $(1\text{–}64)$6 windows can artificially improve offline metrics, reaching recall average approximately $(1\text{–}64)$7–$(1\text{–}64)$8 and precision average approximately $(1\text{–}64)$9–$a$0, but regards this as impractical and still insufficient because it lacks temporal evolution (Blanco-Fernández et al., 2024). Additional limitations listed by the paper include robustness to domain shifts and long-tail events, complications from longer videos and overlapping events, the absence of explicit anticipation metrics, the challenge of stricter latency budgets such as $a$1, and the fact that irreversibility limits later refinement (Blanco-Fernández et al., 2024).

Taken together, these issues identify a consistent research agenda for ActivityNet-RTL. The literature points toward end-to-end training, stronger emphasis on few-proposal recall, streaming-native metrics, explicit latency accounting, and architectures that preserve causality while maintaining temporal alignment quality (Lin et al., 2017, Blanco-Fernández et al., 2024). This suggests that ActivityNet-RTL is best understood not as a closed benchmark definition but as a continuing line of research centered on causal temporal localization and related online event understanding tasks on ActivityNet.