---
title: 'TCAM: Track and Caption Any Motion'
url: https://www.emergentmind.com/topics/track-and-caption-any-motion-tcam
type: topic
---

# TCAM: Track and Caption Any Motion

Track and Caption Any Motion (TCAM) is a motion-centric video understanding framework that autonomously discovers and spatially grounds multiple motion patterns in video, describing each with natural language in a completely query-free manner. TCAM fuses dense point tracking, vision–language contrastive learning, and spatial grounding via a novel Motion-Field Attention mechanism. By leveraging motion dynamics instead of static appearance, it addresses video understanding tasks in the presence of occlusion, camouflage, and rapid movement, achieving state-of-the-art results on retrieval and grounding benchmarks [2512.10607].

## 1. System Architecture and Algorithmic Components

TCAM processes video inputs through parallel appearance and motion pipelines and unifies their representations for motion discovery and description grounding.

- **Input Representation:** For a video $\mathbf V = \{I_t\}_{t=1}^T$, appearance features are extracted by a frozen CLIP-ViT per frame:
  $$
  \mathbf f_t^{vis} = \mathrm{CLIP\text{-}ViT}(I_t),\quad \mathbf e_{\rm video} = \mathrm{TemporalPool}(\{\mathbf f_t^{vis}\})
  $$
  yielding a 512-dimensional video embedding.

- **Motion Trajectories:** Dense 2D point trajectories are obtained using CoTracker3, represented as $\{p_{j,t}\}$ for $N_t$ tracks.

- **Motion-Field Attention (MFA):** MFA fuses each trajectory $\{p_{j,t}\}$ with visual features into a 512-D motion descriptor $\mathbf m_j$. This involves three MLPs for per-frame position, velocity, and occlusion encoding, followed by a “spatial” transformer (jointly attending over tracks and visual context at each time), a “temporal” transformer (aggregating features over $T$ steps for each track), and a linear CLIP-projection:
  $$
  \mathbf m_j = \mathrm{MFA}(\{p_{j,t}\}, \{\mathbf f_t^{vis}\})
  $$

- **Text Bank and Caption Embedding:** All unique ground-truth descriptions $d_k$ are embedded using $\mathrm{CLIP\_text}(d_k)$.

- **Inference Workflow:** TCAM enables query-free discovery by calculating cosine similarities between $\mathbf e_{\rm video}$ and text bank embeddings, performing percentile thresholding, and cross-attention-based spatial grounding.

The overall pipeline aligns the temporal visual signal to a large text bank, spatially grounds selected descriptions via cross-attention with motion descriptors, and produces per-trajectory (and optionally per-pixel) semantic masks.

## 2. Training Objectives and Vision–Language Alignment

TCAM is trained end-to-end using a joint global and spatial loss:

- **Global Video–Text Alignment Loss ($\mathcal{L}_{\mathrm{global}}$):**
  This multi-positive InfoNCE contrastive objective aligns videos and their set of correct descriptions:
  $$
  \mathcal L_{\mathrm{global}}
  = -\frac{1}{B} \sum_{b=1}^B
  \log \frac{ \sum_{k\in\mathcal P_b} \exp( s_{bk}/\tau ) }{ \sum_{k=1}^{N_b} \exp (s_{bk}/\tau) }
  $$
  where $s_{bk}$ denotes the cosine similarity between the $b$-th video and $k$-th text embedding, and $\tau=0.1$.

- **Fine-Grained Spatial Grounding Loss ($\mathcal{L}_{\mathrm{spatial}}$):**
  Enforces correct spatial association between captions and tracks, comprising:
  - **Diversity**: $\mathcal L_{\mathrm{diversity}} = \max(0, 0.1 - \mathrm{std}(\mathbf r_i))$
  - **Sparsity**: $\mathcal L_{\mathrm{sparsity}} = 0.01 \|\mathbf r_i\|_1$
  - **Alignment**: Margin ranking loss between positive (inside-mask) and negative (outside-mask) tracks:
    $$
    \mathcal L_{\mathrm{alignment}} = \frac{1}{|\mathcal T^+| |\mathcal T^-|} \sum_{j\in\mathcal T^+} \sum_{j'\in\mathcal T^-} \max(0, \gamma - r_{i,j} + r_{i,j'}),\ \gamma=0.2
    $$

- **Total Loss:** Combined as
  $$
  \mathcal{L}_\text{total} = \lambda_1\,\mathcal{L}_\text{global} + \lambda_2\,\mathcal{L}_\text{spatial}
  $$
  with $\lambda_1=0.9$, $\lambda_2=0.1$.

## 3. Query-Free Motion Discovery and Multi-Expression Handling

During inference, TCAM autonomously discovers multiple salient motions and their language descriptions without any external queries.

- **Text Bank Utilization:** A large text bank, precomputed from datasets (e.g., MeViS, YouTube-VOS, ActivityNet), captures the diversity of possible natural language expressions.

- **Description Selection:** The system computes similarities $s_k$ between $\mathbf e_{\rm video}$ and all text bank embeddings, then thresholds (e.g., top-70th percentile) or gathers top-scoring descriptions via multi-head cross-attention. The “multi-head” mechanism discovers $N_d \approx 4$–$5$ relevant expressions per video.

- **Spatial Grounding:** Each retrieved description is grounded to specific motion tracks by cross-attention:
  $$
  \mathbf r_i = \mathrm{CrossAttention}(\mathbf e_i^{text}, \{\mathbf m_j\})
  $$
  Output tracks may be rendered as masks for per-frame, per-description region segmentation.

## 4. Training Protocol, Datasets, and Model Capacity

TCAM is trained and evaluated primarily on the MeViS dataset (2,006 videos, 28.6K motion expressions, 443K segmentation masks), with additional evaluation on HC-STVG.

- **Optimization:** AdamW, initial learning rate $10^{-4}$, OneCycleLR, $\tau=0.1$, for 500 epochs using PyTorch DDP on 4 NVIDIA V100 GPUs.
- **Model Dimensions:** MFA hidden dimension $d=256$ with 16 attention heads; $N_t=24\times24=576$ dense point tracks per video.
- **Loss Weighting:** Spatial grounding loss weight $\lambda=0.1$.

Larger text banks increase discovery diversity and coverage, with 150K expressions yielding 68.5% coverage and 84.7% precision.

## 5. Experimental Results and Benchmarks

TCAM attains high performance on both retrieval and spatial grounding metrics. Key results include:

| Setting / Metric                  | Value                | Dataset                 |
|:-----------------------------------|:---------------------|:------------------------|
| Video→Text R@1                    | 58.4% (no query)     | MeViS                   |
| Spatial grounding $J\!+\!F$        | 64.9%                | MeViS                   |
| Expressions Discovered / video     | 4.8                  | MeViS                   |
| Coverage                          | 68.5%                | MeViS (adaptive)        |
| Precision                         | 84.7%                | MeViS (adaptive)        |
| m\_vIoU                           | 42.3%                | HC-STVG                 |
| m\_tIoU                           | 52.8%                | HC-STVG                 |
| vIoU@0.5                          | 38.7%                | HC-STVG                 |

Ablation studies emphasize the importance of MFA, the temporal encoder, multi-positive loss, and multi-head cross-attention for overall system performance. Removal of any major component yields a significant decrease in both retrieval and grounding accuracy.

## 6. Spatial Grounding and Evaluation Metrics

TCAM grounds each caption to a subset of motion tracks, forming per-description, sparse masks. The following metrics quantify grounding accuracy:

- **Jaccard Index (IoU):** $J = |\hat{M}\cap M_{gt}|/|\hat{M}\cup M_{gt}|$
- **Boundary F-score ($F$):** As per segmentation benchmarks.
- **Combined Grounding Score:** $J\!+\!F$
- **Tube IoU (HC-STVG):** Spatio-temporal intersection metrics (m\_vIoU, m\_tIoU, vIoU@0.5).

These metrics capture mask similarity per frame and across time, reflecting how accurately discovered expressions are spatially localized.

## 7. Limitations and Prospective Developments

TCAM’s reliance on pointwise motion trajectories entails inherited limitations:

- **Robustness to Scene Discontinuities:** Abrupt scene cuts or point-of-view shifts can disrupt trajectory identity, weakening spatial association between discovered motions and language.
- **Point-Tracking Failures:** Errors or dropouts in point tracking directly propagate to failures in semantic grounding.
- **Future Improvements:** Integration of temporal-consistency modules, explicit object re-identification, and end-to-end fine-tuning of CLIP may enhance robustness, especially for disorderly motion or long-range temporal reasoning.

A plausible implication is that future architectures combining dense motion correspondence with open-vocabulary vision–language models and strong temporal association will further improve autonomous motion discovery and compositional video understanding.

---

For a detailed methodological exposition, refer to "Track and Caption Any Motion: Query-Free Motion Discovery and Description in Videos" [2512.10607].

Source: https://www.emergentmind.com/topics/track-and-caption-any-motion-tcam