---
title: Temporal Proposal Generation in Video Analysis
url: https://www.emergentmind.com/topics/temporal-proposal-generation
type: topic
---

# Temporal Proposal Generation in Video Analysis

Temporal proposal generation is the process of hypothesizing candidate temporal intervals within untrimmed videos that are likely to contain action instances. These candidate intervals—temporal proposals—form the essential input for downstream temporal action detection and localization systems, as they reduce the search space from all possible segments to a tractable yet highly-recall set of intervals. Temporal proposal generation must simultaneously achieve high recall across action instances, precise localization of action boundaries, and robust ranking for downstream retrieval. The field comprises a spectrum of methodologies including boundary-based pairing, anchor-based prediction, dense confidence map sampling, proposal-level relational modeling, and more recently, transformer and graph-based context aggregation frameworks. Advances in temporal proposal generation have driven improvements in action localization benchmarks including ActivityNet-1.3, THUMOS14, and HACS Segments.

## 1. Core Problem and Evaluation Metrics

Temporal proposal generation operates on feature sequences $X \in \mathbb{R}^{C \times L}$ derived from untrimmed videos, where $C$ is the clip feature dimension and $L$ is the number of clips per video [2204.02674]. The objective is to output a set of candidate intervals $\Psi_p = \{ (t_{s,k}, t_{e,k}, p_k) \}_{k=1}^{K}$, with $(t_{s,k}, t_{e,k})$ denoting start/end indices and $p_k$ a confidence score.

The standard metrics are:
- **Average Recall at Average Number of proposals (AR@AN):** Defined as the mean recall of ground-truth actions at a fixed average number of predicted proposals per video, for a range of temporal Intersection-over-Union (tIoU) thresholds [1806.02964][1907.09702].
- **Area Under the AR-AN Curve (AUC):** AR plotted across AN, then integrated.
- **Mean Average Precision (mAP):** Used when proposals are combined with classifiers for full action localization [2204.02674].

## 2. Boundary and Confidence Modeling Paradigms

Two dominant paradigms exist for proposal generation: boundary-sensitive pairing and anchor-based (region proposal) scoring.

**Boundary-based pairing:** This method estimates start and end boundary probabilities at each time index and forms proposals by pairing start/end candidates [1806.02964][1907.09702]. Key steps include:
- **Temporal Evaluation Module (TEM):** 1D-CNN predicts $p^s(t)$, $p^e(t)$—boundary probabilities per index.
- **Proposal Evaluation Module (PEM):** Computes scores for candidate intervals, classically using either learned MLPs over pooled feature representations [1806.02964] or boundary-matching confidence maps for all $(i,j)$ pairs [1907.09702].

**Anchor-based and dense map approaches:** Proposals are predicted using anchor intervals at multiple temporal scales and locations, or by efficiently evaluating all pairs through dense tensors:
- **Anchor-based generators:** Predefined temporal anchors (center, duration) at each feature location, with regression to refine boundaries [1811.11524][2003.04145].
- **Boundary-Matching maps:** $\mathbb{R}^{D \times T}$ confidence maps where each entry $(i,j)$ scores a proposal of length $i$ starting at $j$ [1907.09702][2204.02674].
- **Proposal pruning:** Top-K scoring intervals are selected, often followed by non-maximum suppression (NMS) or Soft-NMS to limit redundancy [2204.02674][1806.02964].

## 3. Architectural Advances: Context, Relations, and End-to-End Learning

Recent frameworks extend basic pairing/anchor logic with richer temporal and inter-proposal context modeling:

**Context-Adaptive and Relational Blocks:**
- **Self-/cross-attention mechanisms:** Exampled by the Context-Adaptive Proposal Module in Faster-TAD [2204.02674], which applies transformer decoder and cross-attention layers over proposal features and the full video context, capturing long-range dependencies among proposals and between proposals and the temporal feature sequence.
- **Proposal Relation Block (PRB):** As in BSN++ [2009.07641], encodes inter-proposal dependencies via position- and channel-wise self-attention to improve proposal ranking and reliability.
- **Explicit graph reasoning:** BC-GNN treats proposal boundaries and content as nodes and edges in a graph neural network, refining both via edge/node updates and message passing to yield state-of-the-art recall [2008.01432].

**Label Assignment and Training:**
- **Soft and proximity labels:** Beyond binary positive/negative assignment, frameworks like Faster-TAD use 2M-category soft labeling, accounting for proximity to ground-truth through measured IoU and distance [2204.02674]. Scale-balanced sampling is also employed to ensure robust training across proposal durations [2009.07641].
- **Multi-task objectives:** Losses blend classification (boundary, proposal) and regression (offset) terms, with focal or smooth-L1 losses to manage class imbalance and outlier proposals [2204.02674][2009.07641].

**Auxiliary features and proposal augmentation:**
- **Fake proposals and atomic features:** Artificial proposals with offset boundaries are injected to improve regressor robustness [2204.02674]. Auxiliary atomic-action features extracted from complementary networks are fused to increase proposal discriminability under fine-grained motion [2204.02674].

## 4. Transformer and Graph Architectures

Advanced context modeling is central to recent state-of-the-art:
- **Transformer-based networks:** TAPG Transformer [2105.12043] separates boundary detection and proposal confidence evaluation into two transformer modules (Boundary Transformer, Proposal Transformer) to decouple frame-wise locality from proposal-level dependencies.
- **Unified transformer detectors:** RTD-Net [2102.01894] discards anchors, using a set of proposal queries and a relaxed Hungarian matching scheme with an auxiliary completeness prediction head, producing NMS-free proposals with high accuracy.
- **Hybrid local-global encoders:** ATAG [2103.16024] utilizes an augmented transformer for global context and an adaptive graph convolutional network for local, position-sensitive context. Fused features drive more accurate proposal boundaries, notably under background clutter.
- **Pyramid slot-based local attention:** PRSA-Net [2206.10095] leverages a pyramid region-based slot-attention encoder-decoder, focusing only on local neighborhoods across multiple parallel scales, outperforming global similarity computation approaches.

## 5. Proposal Diversity, Imbalance Mitigation, and Scalability

To counteract intrinsic biases and organization mismatches:
- **Scale-balanced and IoU-balanced sampling:** BSN++ implements a two-stage re-sampling strategy to equalize the influence of proposal duration, improving recall for both short and long actions [2009.07641].
- **Synthetic/fake proposals:** As in Faster-TAD, a fixed fraction of proposals per batch are synthesized by offsetting ground-truth boundaries, exposing the network to a range of errors during training [2204.02674].
- **Parallel and distributed training:** Scalability on large datasets is supported by frameworks such as MTN with high-performance MPI-based ring communication, achieving near-linear speedup for proposal training on multi-GPU clusters [1906.06496].

## 6. Empirical Performance and Benchmarks

Temporal proposal generation methods are primarily evaluated on ActivityNet-1.3, THUMOS14, and HACS Segments, with recall and mAP as principal metrics. Recent results include:
- **Faster-TAD**: ActivityNet-1.3 mAP 40.01%, HACS 38.39%, SoccerNet Action Spotting 54.09%, outperforming prior single-network detectors [2204.02674].
- **PRSA-Net**: On THUMOS14, AR@100 reaching 56.12% and mAP at 0.5 tIoU of 58.7% (UntrimmedNet) and 55.0% (P-GCN), exceeding all prior reported numbers [2206.10095].
- **ATAG**: On ActivityNet1.3 (val), AUC=68.50%, AR@100=76.75%; THUMOS14 (AR@100)=52.21% [2103.16024].
- **BSN++**: ActivityNet-1.3 AUC=68.26%, THUMOS14 AR@100=49.84% [2009.07641].
- **RTD-Net**: THUMOS14 AR@100=49.32%, no NMS needed [2102.01894].
- **TCANet**: ActivityNet-1.3 AUC=68.08%, THUMOS14 AR@100=50.48%, with local-global refinement [2103.13141].
These results indicate a continual increase in proposal recall and precision as context modeling, label assignment, and training protocols evolve.

## 7. Synthesis and Prospective Directions

Contemporary research in temporal proposal generation emphasizes:
- **Multi-level and relational context aggregation** via transformers, graph neural networks, and self-/cross-attention structures to improve both boundary precision and proposal discriminability.
- **Balanced label assignment and diversity-promoting training** to counteract long-tailed duration distributions and boundary misalignment biases.
- **Scalability and efficiency**, including ring-based distributed training and sampling-efficient architectures, to handle modern video corpora at scale.

Ongoing areas of investigation include end-to-end unified detection/classification models, dynamic fusion of agent-centric and environmental context, and explicit modeling of segmental relationships beyond pairwise boundaries. The trend is towards more holistic, context-aware, and self-supervised pretraining pipeline elements to further push recall and localization precision in complex real-world scenarios [2204.02674][2009.07641][2105.12043].

Source: https://www.emergentmind.com/topics/temporal-proposal-generation