---
title: Long-Term Feature Bank for Video Understanding
url: https://www.emergentmind.com/topics/long-term-feature-bank-lfb
type: topic
---

# Long-Term Feature Bank for Video Understanding

A Long-Term Feature Bank (LFB) is a memory mechanism designed for detailed video understanding, enabling models to access supportive information extracted over extensive temporal ranges—often spanning minutes of video. By augmenting conventional short-term video recognition backbones, such as 3D convolutional networks, with an explicit, persistent repository of per-clip features, LFB architectures support fine-grained spatio-temporal reasoning critical for tasks like action localization, video classification, and egocentric video analysis. LFBs have been central to achieving state-of-the-art results in large-scale video understanding benchmarks, with their use prominently established by Wu et al. [1812.05038] and further developments such as low-rank and incremental memory approximations for efficiency [2406.07191].

## 1. Core Definition and Construction

The LFB is instantiated as a time-indexed sequence of deep feature maps, each summarizing the output of a pretrained backbone (typically a 3D-CNN) from regularly sampled short clips or frames:

- For a video of $T$ frames (or clips), indexed by $t=0,…,T-1$, input segments $x_t \in \mathbb{R}^{C \times \tau \times H \times W}$ are encoded via a 3D-CNN $\varphi(\cdot;\theta)$, producing downsampled feature maps $F_t$.
- Candidate actors or regions-of-interest (RoIs) are detected at each time step, yielding bounding boxes $\{b_{t,1},…,b_{t,N_t}\}$, where $N_t$ denotes the number of detections at time $t$.
- RoI pooling (e.g., RoIAlign) is applied to each box, followed by spatial and temporal average pooling, producing per-detection feature vectors $s_{t,j} \in \mathbb{R}^D$.
- This results in $L_t \in \mathbb{R}^{N_t \times D}$, with $L = \{L_0, L_1, …, L_{T-1}\}$ forming the Long-Term Feature Bank for the entire video.

To ensure efficient storage and computation, the feature bank is typically materialized at regular intervals $\Delta t$ (e.g., one clip per second), and stored on disk or CPU memory. The dimension $D$ is usually large (e.g., $D=2048$) to capture rich context required by downstream tasks [1812.05038].

## 2. Feature Retrieval and Integration

At test or training time, the short-term backbone processes a local window (e.g., 32–64 frames), producing current RoI-pooled features $S_t \in \mathbb{R}^{N_t\times D}$. The LFB is queried for a context window of size $w$ centered at $t$:

- The query window is $[t-w, t+w]$, collecting all detections to form $\widetilde{L}_t \in \mathbb{R}^{N \times D}$, with $N=\sum_{t'=t-w}^{t+w} N_{t'}$.

Feature integration is performed with a Feature Bank Operator (FBO). The standard approach uses a non-local attention mechanism:

\[
\begin{aligned}
Q &= S_t W_q, \\
K &= \widetilde{L}_t W_k, \\
V &= \widetilde{L}_t W_v, \\
A &= \mathrm{softmax}\left(Q K^\top/\sqrt{d_k}\right), \\
S'_t &= S_t + A V W_o,
\end{aligned}
\]

where $W_q, W_k, W_v \in \mathbb{R}^{D \times d_k}$ and $W_o \in \mathbb{R}^{d_k \times D}$ are learned projectors and $d_k$ is a reduced probe dimension (e.g., $d_k = 512$) [1812.05038]. Optionally, simpler pooling operators (average/max) on $\widetilde{L}_t$ can be used as a baseline variant.

## 3. Implementation and Computational Considerations

The LFB pipeline is characterized by a clear separation of long-term (video-wide) and short-term (local clip) computation. Feature extraction for the LFB is fully decoupled and can be performed offline, amortizing cost across deployments:

- Storage and I/O requirements are dominated by the number of detections and embedding size, i.e., $\sum_t N_t \cdot D$ floats per video.
- For efficient batching during model training, zero-padding is used so all windows have fixed $N$.
- The memory cost on GPU grows with the window size (due to $\widetilde{L}_t$'s size) but can be bounded by keeping only $2w+1$ time steps in memory at a time.
- For large-scale deployment or longer temporal support, innovations such as low-rank SVD-based memory (e.g., MeMSVD [2406.07191]) compress the LFB to a basis $U_\text{mem} \in \mathbb{R}^{n_c \times d}$ (with $n_c \ll N$) and singular values, dramatically reducing inference and update complexity.

Pseudocode for training and inference illustrates precomputing the LFB, efficient window selection, and attention-based updates [1812.05038]:

```python
# Precompute LFB
LFB = []
for t in range(0, T, Δt):
    f = CNN(load_clip(frames[t-τ//2 : t+τ//2]))
    detections = DETECTOR(frames[t])
    features = [RoIAlign(f, box) for box in detections]
    LFB.append((t, np.stack(features)))

# During inference
for t in test_times:
    S_t = extract_short_term_features(t)
    L̃_t = concatenate_LFB_window(LFB, t, w)
    U_t = FBO(S_t, L̃_t)
    scores = classifier(U_t)
    ...
```

## 4. Empirical Performance and Evaluation

LFB-augmented models yield significant improvements across diverse video understanding benchmarks [1812.05038], with empirical results demonstrating substantial gains over strong short-term baselines:

| Dataset         | Baseline         | +LFB (NL, 60s window) | Δ (mAP / top-1) |
|-----------------|-----------------|-----------------------|-----------------|
| AVA (mAP)       | 22.1            | 25.5                  | +3.4            |
| EPIC-Kitchens   | 19.0            | 22.8                  | +3.8            |
| Charades (mAP)  | 38.3            | 40.3                  | +2.0            |

- On AVA, a 3D CNN (Res50-I3D-NL) achieves 22.1\% mAP, which rises to 25.5\% with LFB. Enhancement with larger backbones and test-time augmentation yields 27.7\% mAP.
- For EPIC-Kitchens, top-1 action accuracy improves from 19.0\% to 22.8\% (+3.8 points).
- For Charades, mAP increases from 38.3\% to 40.3\% with the addition of LFB.
- Results are computed using mAP (mean Average Precision) for detection/classification, and top-k accuracy for egocentric action recognition.

## 5. Comparison with Alternative Memory Mechanisms

The conventional LFB approach relies on attention-based integration, but this introduces substantial computational and memory overhead, especially for large window sizes:

- Attention cost per actor/query is $O(N_\mathrm{mem} \cdot d_u)$, and the entire $N_\mathrm{mem} \times d$ memory bank must be resident.
- To address these costs, MeMSVD [2406.07191] replaces attention with a low-rank SVD basis. Feature memories are compressed to $n_c \ll N_\mathrm{mem}$ principal components. Actor features are projected into and reconstructed from this basis, yielding a linear, parameter-free integration:
  
  \[
  \alpha = h_{t,s} \cdot U_\text{mem}^\top, \qquad h' = \alpha \cdot U_\text{mem}, \qquad h_{t,s} \leftarrow h_{t,s} + h'
  \]
  
- MeMSVD achieves similar or superior accuracy with 10–20× fewer FLOPs in the memory integration, constant runtime with increasing window length, and much lower parameter count.
- Incremental updates to the SVD basis can be performed efficiently ($O(n_c^2)$ per update), enabling near-real-time adaptation to streaming input.
- On AVA and Charades, MeMSVD matches or outperforms standard attention-LFBs (e.g., 43.0\% mAP on Charades with MViTv2-B 16×4, establishing new state-of-the-art performance).

## 6. Experimental Design and Protocols

Key experimental details for LFB-based models include:

- Datasets: AVA (spatio-temporal action labels per person at 1 FPS across 15-minute videos), EPIC-Kitchens (action recognition from egocentric viewpoint with verb–noun labels), and Charades (video-level multilabel action recognition).
- Performance metrics: mean Average Precision (mAP) over classes, and top-1/top-5 accuracy as appropriate.
- LFB window size and granularity: empirically, optimal temporal support occurs around $\pm30$ seconds (window size $\sim$61s), with diminishing returns beyond this.
- Padding and batching: all memory windows are zero-padded to a uniform size for efficient batch processing.
- All LFB computations (including FBO and classifier) are trained end-to-end except for the fixed backbone features in the LFB.

## 7. Context, Adoption, and Limitations

The LFB has become foundational in video recognition, enabling models to reason over much longer temporal horizons than would be possible with memory- or compute-constrained backbones alone. This design extends the horizon of deep action detectors to minutes of context, supporting nuanced spatio-temporal queries and disambiguating temporally extended actions.

Limitations include the cost of storing and retrieving high-dimensional feature banks for large datasets or long videos, as well as the scalability of attention mechanisms with increasing context length. Compressing the memory (e.g., via low-rank SVD) is effective in alleviating these constraints. A persistent limitation is the reliance on accurate region proposals from person/object detectors, with representational quality of the LFB potentially bottlenecked by detection recall or the diversity of backbone features [1812.05038, 2406.07191].

A plausible implication is that as architectures for video understanding continue to mature, the integration of persistent, scalable feature banks—potentially combined with online or streaming memory mechanisms—will remain a core strategy for leveraging context at scale.

Source: https://www.emergentmind.com/topics/long-term-feature-bank-lfb