---
title: Referring Multi-Object Tracking (RMOT) Overview
url: https://www.emergentmind.com/topics/referring-multi-object-tracking-rmot
type: topic
---

# Referring Multi-Object Tracking (RMOT) Overview

Searching arXiv for recent RMOT papers to ground the article in the latest work.
Referring Multi-Object Tracking (RMOT) is a video understanding task in which a model receives a video sequence and a natural-language expression, then localizes and temporally associates all objects whose semantics match that expression. The task was introduced as a general referring understanding setting that extends beyond single-object referring by allowing an arbitrary number of referent object predictions in videos, including cases where the referent set changes over time and may even be empty [2303.03366]. In later work, RMOT is consistently framed as a cross-modal problem that combines object grounding, temporal association, and language-conditioned selection, and it has been broadened into end-to-end MLLM formulations that jointly perform grounding and tracking rather than decomposing them into separate modules [2604.10527].

## 1. Definition and formal scope

In its canonical formulation, RMOT takes as input a video sequence \(\mathcal{V}\) and a language expression \(S\), and produces, for each frame \(t\), a variable-sized set of bounding boxes \(\mathcal{B}_t\) with consistent identities across time, such that each box corresponds to an object matching the semantics of \(S\) at frame \(t\) [2303.03366]. This formulation differs from standard Multi-Object Tracking, which tracks all predefined instances of interest, because RMOT tracks only those objects selected by the language expression. It also differs from single-object referring understanding, because one expression may refer to multiple objects, no objects, or objects that satisfy the description only during part of a sequence [2303.03366].

The literature repeatedly emphasizes that RMOT is not reducible to “MOT plus text.” The language may encode appearance, action, position, motion state, relations, or more complex semantics. Existing work explicitly highlights arbitrary-number prediction, dynamic temporal status, and contiguous temporal validity as core properties of the task, and later benchmarks further introduce no-target expressions, motion-centric expressions, and reasoning-oriented instructions [2303.03366]. This makes RMOT relevant to video editing, human-computer interaction, autonomous driving, surveillance, robotics, and interactive AI systems [2406.05039].

A persistent misconception is that the task can be solved by simply attaching a text encoder to a tracker. The benchmark and method papers collectively argue otherwise: RMOT requires cross-modal grounding, target selection, temporal identity preservation, and robustness to ambiguity, occlusion, and sparse supervision. Later end-to-end work makes this point more strongly by presenting RMOT as the joint problem of grounding and tracking rather than a loose composition of independently optimized modules [2604.10527].

## 2. Benchmarks, datasets, and annotation regimes

Benchmark construction has been central to RMOT research because the task depends on paired trajectory-language supervision. The initial benchmark, Refer-KITTI, was derived from KITTI and provides 18 videos, 818 expressions, and an average of 10.7 objects per expression [2303.03366]. Later work expanded the scale and linguistic diversity of supervision, introduced synthetic-data pipelines, and created domain-specific variants for cross-view, reasoning-based, RGB-thermal, RGBD, omnidirectional, UAV, motion-expression, and end-to-end MLLM settings [2406.05039].

| Benchmark or task | Scale | Distinguishing property |
|---|---:|---|
| Refer-KITTI [2303.03366] | 18 videos, 818 expressions | Original RMOT benchmark |
| Refer-KITTI-V2 [2406.05039] | 21 videos, 9,758 expressions, 617 unique words | LLM-augmented expansion of Refer-KITTI |
| Refer-UE-City [2404.12031] | 14 videos, 714 expressions | Unreal Engine 5 surveillance scenes |
| CRTrack [2412.17807] | 13 scenes, 221 language descriptions, 82,000+ frames | Cross-view RMOT benchmark |
| ReaMOT Challenge [2505.20381] | 1,156 instructions, 423,359 image-language pairs, 869 scenes | Reasoning-based tracking benchmark |
| RefRT [2602.22033] | 388 language descriptions, 1,250 tracked targets, 166,147 L-RGB-T triplets | First RGB-Thermal RMOT dataset |
| DRSet [2602.04692] | 187 scenes, 240 language descriptions | RGBD RMOT with 56 depth-related descriptions |
| ORSet [2603.05384] | 27 scenes, 848 language descriptions, 3,401 annotated objects | Omnidirectional RMOT dataset |
| AerialMind [2511.21053] | 93 UAV videos, 24,600 expressions, 293,100 object instances | First large-scale UAV RMOT benchmark |
| STORM-Bench [2604.10527] | 15K videos, 0.2M expressions, 73.7K tracked objects | Bottom-up annotation for end-to-end RMOT |
| MeViS [2512.10945] | 2,006 videos, 33,072 motion expressions, 8,171 objects | Motion-expression dataset supporting RMOT |

The evolution of these datasets reveals several annotation strategies. Refer-KITTI-V2 starts from 2,719 manual annotations, expands them to 9,758 annotations by prompting large language models, and reaches 617 different words, explicitly addressing class imbalance and limited lexical diversity [2406.05039]. Refer-UE-City uses Unreal Engine 5 to generate surveillance-style scenes with automatically generated 2D bounding boxes and dynamic, action-focused descriptions, reducing manual annotation cost while increasing action-centric supervision [2404.12031]. CRTrack augments DIVOTrack and CAMPUS with descriptions generated with GPT-4o and manually verified, thereby tying synchronized multi-view video to fine-grained language-object associations [2412.17807]. AerialMind introduces COALA, a semi-automated collaborative agent-based labeling assistant, to reduce labor cost while maintaining annotation quality in UAV scenarios [2511.21053]. STORM-Bench adopts a bottom-up annotation pipeline with LLM-assisted object-level caption generation and verification to produce accurate, diverse, and unambiguous referring expressions [2604.10527].

These benchmarks also progressively widen the semantic regime. ReaMOT Challenge explicitly scores reasoning difficulty and includes spatial reasoning, social interaction, behavioral trends, objective-driven intent, and activity recognition [2505.20381]. MeViS shifts supervision toward object motion and temporal reasoning, including multiple, single, and zero-target expressions, in both text and audio [2512.10945]. DRMOT and RT-RMOT introduce depth and thermal cues for spatial semantics and low-visibility conditions, while ORMOT and AerialMind target full-field-of-view and aerial viewpoints rather than conventional ground-camera imagery [2602.04692].

## 3. Methodological paradigms

The methodological history of RMOT begins with end-to-end transformer tracking. TransRMOT is an online transformer-based architecture inspired by DETR and MOTR, with a CNN visual encoder, a pre-trained text encoder, an early-fusion cross-modal encoder, track queries propagated across frames, detect queries for new objects, and a referent head that predicts class, box, and referent score [2303.03366]. This line was extended by TempRMOT, which adds a query-based temporal enhancement module with an \(N \times K\) query memory in a FIFO buffer, a temporal decoder, and track refinement, while also bootstrapping supervision through Refer-KITTI-V2 [2406.05039].

Subsequent end-to-end transformer work concentrated on stronger fusion and better temporal conditioning. TenRMOT performs deep multi-modal feature fusion at both encoding and decoding stages through an Interleaving Cross-modality Encoder, a Language Guided Decoder, and a Query Update Module that explicitly uses temporal prior information of tracked objects [2410.13437]. “Tell Me What to Track” addresses the imbalance between newborn targets and existing targets with Collaborative Query Matching, strengthens multi-scale encoder fusion through an Enhanced Cross-Modal and Multi-Scale Encoder, and injects explicit referring guidance into query tokens through Referring-Infused Query Adaptation [2412.12561]. MLS-Track adds a Semantic Guidance Module that makes queries prompt-aware before decoding and a Semantic Correlation Branch that filters predictions by similarity to the expression [2404.12031]. CDRMT further decomposes language according to “what” and “where” pathways, using Bidirectional Interactive Fusion, Progressive Semantic-Decoupled Query Learning, and Structural Consensus Constraint to preserve modality-specific characteristics, hierarchically inject semantics, and reconstruct language representations from visual features [2503.11496].

A second paradigm is two-stage RMOT. The data describe a family of methods in which tracking and referring are decoupled, allowing off-the-shelf trackers to remain frozen while only the referring module is trained. MEX is a plug-and-play Memory-Efficient Cross-modality module that replaces heavier referring modules, reducing memory consumption by about 50%, running on a single GPU with 4 GB of memory, and improving HOTA on Refer-KITTI [2502.13875]. JustHook revisits the two-stage setting more fundamentally: it introduces a Visual Feature Hook that extracts context-rich local features directly from the original visual backbone and a Parallel Combined Decoder that replaces passive cosine similarity with active contrastive learning in a combined feature space, while also arguing that fixed dual-tower architectures restrict compatibility with other visual and text backbones [2503.07516].

A third paradigm uses LVLMs or MLLMs as the core reasoning engine. ReaTrack is a training-free framework that combines an LVLM for language-conditioned candidate generation with Online SAM2 for temporal tracking and IoU-based matching via the Hungarian algorithm [2505.20381]. RTrack, DRTrack, and ORTrack use multimodal large models or large vision-language models to process RGB-Thermal-Language, RGB-D-Language, and omnidirectional imagery with language, respectively, and couple these detectors with tracking and association modules tailored to their modality [2602.22033]. STORM generalizes this direction into a fully end-to-end MLLM that jointly performs grounding and tracking without external detectors and learns from image grounding, single-object tracking, and RMOT through Task-Composition Learning [2604.10527].

A fourth paradigm centers not on architecture alone but on supervision quality. COAL diagnoses a “sparsity-discriminability paradox” in RMOT and addresses it through Explicit Semantic Injection via DINO-X, Counterfactual Learning via Qwen3-Max, and Hierarchical Multi-Stream Integration, thereby moving beyond isolated structural optimization toward knowledge regularization [2605.14795].

## 4. Task extensions beyond canonical single-view RGB RMOT

Once the base task was established, the field diversified into variants that modify either the visual domain or the linguistic regime. CRMOT extends RMOT from single-view tracking to multiple synchronized views with overlapping fields, with the explicit goal of overcoming invisible appearances and maintaining identity consistency in each cross-view. Its benchmark, CRTrack, and method, CRTracker, combine CrossMOT-style multi-view MOT with APTM-based vision-language attribute prompt matching and a prediction module for association fusion [2412.17807].

ReaMOT extends RMOT in the language dimension. It defines reasoning-based multi-object tracking for complex language instructions with reasoning characteristic, introduces ReaMOT Challenge with three levels of reasoning difficulty, and proposes dedicated metrics and a training-free baseline rather than treating reasoning as a minor prompt-phrasing variation [2505.20381]. MeViS pushes in a related but distinct direction: it emphasizes motion expressions rather than static attributes, supports multiple, single, and zero-target expressions, and benchmarks RMOT alongside RVOS, AVOS, and referring motion expression generation [2512.10945]. TenRMOT also extends the task to Referring Multi-Object Tracking and Segmentation, and Ref-KITTI Segmentation contains 18 videos with 818 expressions and an average of 10.7 masks per expression [2410.13437].

Other extensions alter sensing modalities. RT-RMOT fuses RGB appearance features with the illumination robustness of thermal imagery to enable all-day referring multi-object tracking under nighttime, smoke, and other challenging scenarios [2602.22033]. DRMOT introduces explicit depth as a first-class modality to resolve spatial semantics such as “the person closest to the camera” and improve association under severe occlusion [2602.04692]. ORMOT addresses the limited field of view of conventional cameras by moving to omnidirectional imagery, with the motivation that targets otherwise move out of frame, producing fragmented tracking and poorer support for long-horizon language descriptions [2603.05384]. AerialMind moves RMOT into UAV scenarios characterized by drastic appearance differences, complex spatial relationships, dynamic scene variations, rich and complex referring expressions, and a severe small-object problem [2511.21053].

These extensions indicate that “RMOT” now denotes a family of language-guided tracking problems rather than a single benchmark template. This suggests that methodological claims should be interpreted in the context of the visual regime, language regime, and metric suite used by each paper, rather than as universally transferable rankings.

## 5. Evaluation protocols and empirical tendencies

The dominant evaluation protocol in canonical RMOT uses HOTA, with the decomposition
\[
\mathrm{HOTA} = \sqrt{\mathrm{DetA} \cdot \mathrm{AssA}},
\]
alongside DetA, AssA, DetRe, DetPr, AssRe, AssPr, and LocA [2303.03366]. This metric family remains standard across Refer-KITTI, Refer-KITTI-V2, Refer-UE-City, and many later variants. However, task-specific benchmarks introduce additional metrics when the classical MOT suite is insufficient. CRMOT defines CVRIDF1 and CVRMA as cross-view metrics that aggregate per-language-description IDF1 and matching accuracy [2412.17807]. ReaMOT defines RIDF1, RMOTA, RRcll, and RPrcn to evaluate reasoning-conditioned tracking at the instruction level [2505.20381]. MeViS adapts HOTA into HOTA\(^*\), DetA\(^*\), and AssA\(^*\), and adds N-Acc. and T-Acc. for the generalized no-target setting [2512.10945]. AerialMind introduces HOTA\(_S\) and HOTA\(_M\) to summarize scene robustness and motion resilience under UAV attributes [2511.21053].

Reported results trace a clear progression on mature road-scene benchmarks. TransRMOT reports HOTA 35.54 on Refer-KITTI [2303.03366]. TempRMOT reports 52.21 HOTA on Refer-KITTI and 35.04 on Refer-KITTI-V2 [2406.05039]. TenRMOT reports HOTA 49.77 on Ref-KITTI for box RMOT and 46.00 on Ref-KITTI Segmentation for RMOTS [2410.13437]. “Tell Me What to Track” reports 55.63 HOTA on Refer-KITTI and 37.22 on Refer-KITTI-V2 [2412.12561]. CDRMT reports 49.35 HOTA on Refer-KITTI and 31.99 on Refer-KITTI-V2 [2503.11496]. JustHook reports 42.81 HOTA on Refer-KITTI-V2 with a RoBERTa plus ROPE Swin-T combination and states a 7.77% HOTA improvement over TempRMOT on that benchmark [2503.07516]. COAL reports 53.38 HOTA on Refer-KITTI and 43.46 on Refer-KITTI-V2, surpassing the previous state of the art by 7.28% HOTA on Refer-KITTI-V2 [2605.14795].

On newer task variants, absolute values are often lower, even when methods outperform prior baselines. CRTracker reports 54.88% CVRIDF1 and 35.97% CVRMA on the in-domain CRTrack evaluation [2412.17807]. RTrack reports HOTA 15.53 on RefRT [2602.22033]. DRTrack reports HOTA 33.24 on DRSet [2602.04692]. ORTrack reports HOTA 9.97 in zero-shot evaluation on ORSet [2603.05384]. HETrack reports HOTA 31.46 on AerialMind in-domain and 31.60 cross-domain on UAVDT [2511.21053]. STORM reports HOTA 66.7, AssA 82.6, IDF1 78.3, and MOTA 57.1 on STORM-Bench [2604.10527]. This suggests that benchmark expansion is introducing genuinely harder regimes rather than merely adding more data, and that raw performance numbers should not be compared across datasets without regard to domain, annotation policy, and metric definition.

## 6. Structural challenges, controversies, and outlook

Several recurrent bottlenecks structure the RMOT literature. One is supervision scarcity and ambiguity. The original benchmark papers already note limited paired prompt-instance data and restricted domain coverage, while later work explicitly identifies sparse semantic supervision, shortcut learning, and semantic collapse as central problems [2404.12031]. COAL formalizes this as the sparsity-discriminability paradox: highly homogeneous scenarios require fine-grained discrimination over complex compositional semantics, but available annotations cover only a small fraction of the combinatorial attribute space [2605.14795].

A second issue is the interaction between architectural efficiency and expressivity. One-stage strategies jointly train tracking with referring but suffer from substantial computational overhead, whereas two-stage methods improve efficiency but may take a detour or become constrained by fixed dual-tower alignment [2503.07516]. The same tension appears in resource usage: MEX argues that cross-modality reasoning can be made substantially lighter, reporting over 10M fewer trainable parameters than the compared referring module, about 50% lower memory use, and 1.5x faster inference while still improving HOTA [2502.13875]. This indicates that RMOT progress is not solely a matter of larger backbones.

A third issue is target imbalance and long-tail temporal behavior. “Tell Me What to Track” states that prior studies overlook the imbalanced data distribution between newborn targets and existing targets and proposes collaborative matching to improve newborn-target detection without degrading tracking performance [2412.12561]. Small-object sensitivity is repeatedly identified in UAV and driving scenarios, while cross-view, omnidirectional, depth, and thermal variants all originate from explicit failure modes of single-view RGB RMOT under occlusion, limited field of view, low visibility, or ambiguous spatial semantics [2602.22033].

A fourth issue is robustness and security. “See No Evil” argues that the reliability and robustness of RMOT remain underexplored, identifies adversarial vulnerabilities in both linguistic-visual referring and track-object matching, and shows that FIFO-based temporal memory can preserve corrupted state over multiple subsequent frames. The VEIL framework demonstrates digital and physical perturbations that induce track ID switches and terminations on Refer-KITTI [2509.02028]. This makes robustness a substantive research topic rather than a peripheral deployment concern.

The newest end-to-end MLLM line suggests a different path forward. STORM argues that RMOT data efficiency can be improved by decomposing training into image grounding and object tracking before fine-tuning on RMOT, and reports that performance plateaus with as few as 12,000 RMOT samples in its ablation [2604.10527]. A plausible implication is that future RMOT systems will combine stronger multimodal foundation models, richer benchmark construction, and explicit reasoning or regularization mechanisms, while also addressing efficiency, security, and domain transfer rather than optimizing only HOTA on a single benchmark.

Source: https://www.emergentmind.com/topics/referring-multi-object-tracking-rmot