Referring Multi-Object Tracking (RMOT) Overview

• RMOT is a video understanding task where a model uses a language expression to localize and track an arbitrary number of objects across time, handling cases with variable or absent targets.
• Benchmark datasets like Refer-KITTI and Refer-KITTI-V2 employ diverse annotation strategies, offering rich language-object pairings while highlighting challenges such as sparse supervision and semantic ambiguity.
• Methodologies range from end-to-end transformer architectures and two-stage frameworks to LVLM/MLLM-based approaches, each balancing multi-modal fusion, efficiency, and temporal coherence.

Searching arXiv for recent RMOT papers to ground the article in the latest work. Referring Multi-Object Tracking (RMOT) is a video understanding task in which a model receives a video sequence and a natural-language expression, then localizes and temporally associates all objects whose semantics match that expression. The task was introduced as a general referring understanding setting that extends beyond single-object referring by allowing an arbitrary number of referent object predictions in videos, including cases where the referent set changes over time and may even be empty (Wu et al., 2023). In later work, RMOT is consistently framed as a cross-modal problem that combines object grounding, temporal association, and language-conditioned selection, and it has been broadened into end-to-end MLLM formulations that jointly perform grounding and tracking rather than decomposing them into separate modules (Lu et al., 12 Apr 2026).

1. Definition and formal scope

In its canonical formulation, RMOT takes as input a video sequence $\mathcal{V}$ and a language expression $S$, and produces, for each frame $t$, a variable-sized set of bounding boxes $\mathcal{B}_t$ with consistent identities across time, such that each box corresponds to an object matching the semantics of $S$ at frame $t$ (Wu et al., 2023). This formulation differs from standard Multi-Object Tracking, which tracks all predefined instances of interest, because RMOT tracks only those objects selected by the language expression. It also differs from single-object referring understanding, because one expression may refer to multiple objects, no objects, or objects that satisfy the description only during part of a sequence (Wu et al., 2023).

The literature repeatedly emphasizes that RMOT is not reducible to “MOT plus text.” The language may encode appearance, action, position, motion state, relations, or more complex semantics. Existing work explicitly highlights arbitrary-number prediction, dynamic temporal status, and contiguous temporal validity as core properties of the task, and later benchmarks further introduce no-target expressions, motion-centric expressions, and reasoning-oriented instructions (Wu et al., 2023). This makes RMOT relevant to video editing, human-computer interaction, autonomous driving, surveillance, robotics, and interactive AI systems (Zhang et al., 2024).

A persistent misconception is that the task can be solved by simply attaching a text encoder to a tracker. The benchmark and method papers collectively argue otherwise: RMOT requires cross-modal grounding, target selection, temporal identity preservation, and robustness to ambiguity, occlusion, and sparse supervision. Later end-to-end work makes this point more strongly by presenting RMOT as the joint problem of grounding and tracking rather than a loose composition of independently optimized modules (Lu et al., 12 Apr 2026).

2. Benchmarks, datasets, and annotation regimes

Benchmark construction has been central to RMOT research because the task depends on paired trajectory-language supervision. The initial benchmark, Refer-KITTI, was derived from KITTI and provides 18 videos, 818 expressions, and an average of 10.7 objects per expression (Wu et al., 2023). Later work expanded the scale and linguistic diversity of supervision, introduced synthetic-data pipelines, and created domain-specific variants for cross-view, reasoning-based, RGB-thermal, RGBD, omnidirectional, UAV, motion-expression, and end-to-end MLLM settings (Zhang et al., 2024).

Benchmark or task	Scale	Distinguishing property
Refer-KITTI (Wu et al., 2023)	18 videos, 818 expressions	Original RMOT benchmark
Refer-KITTI-V2 (Zhang et al., 2024)	21 videos, 9,758 expressions, 617 unique words	LLM-augmented expansion of Refer-KITTI
Refer-UE-City (Ma et al., 2024)	14 videos, 714 expressions	Unreal Engine 5 surveillance scenes
CRTrack (Chen et al., 2024)	13 scenes, 221 language descriptions, 82,000+ frames	Cross-view RMOT benchmark
ReaMOT Challenge (Chen et al., 26 May 2025)	1,156 instructions, 423,359 image-language pairs, 869 scenes	Reasoning-based tracking benchmark
RefRT (Yu et al., 25 Feb 2026)	388 language descriptions, 1,250 tracked targets, 166,147 L-RGB-T triplets	First RGB-Thermal RMOT dataset
DRSet (Chen et al., 4 Feb 2026)	187 scenes, 240 language descriptions	RGBD RMOT with 56 depth-related descriptions
ORSet (Chen et al., 5 Mar 2026)	27 scenes, 848 language descriptions, 3,401 annotated objects	Omnidirectional RMOT dataset
AerialMind (Chen et al., 26 Nov 2025)	93 UAV videos, 24,600 expressions, 293,100 object instances	First large-scale UAV RMOT benchmark
STORM-Bench (Lu et al., 12 Apr 2026)	15K videos, 0.2M expressions, 73.7K tracked objects	Bottom-up annotation for end-to-end RMOT
MeViS (Ding et al., 11 Dec 2025)	2,006 videos, 33,072 motion expressions, 8,171 objects	Motion-expression dataset supporting RMOT

The evolution of these datasets reveals several annotation strategies. Refer-KITTI-V2 starts from 2,719 manual annotations, expands them to 9,758 annotations by prompting LLMs, and reaches 617 different words, explicitly addressing class imbalance and limited lexical diversity (Zhang et al., 2024). Refer-UE-City uses Unreal Engine 5 to generate surveillance-style scenes with automatically generated 2D bounding boxes and dynamic, action-focused descriptions, reducing manual annotation cost while increasing action-centric supervision (Ma et al., 2024). CRTrack augments DIVOTrack and CAMPUS with descriptions generated with GPT-4o and manually verified, thereby tying synchronized multi-view video to fine-grained language-object associations (Chen et al., 2024). AerialMind introduces COALA, a semi-automated collaborative agent-based labeling assistant, to reduce labor cost while maintaining annotation quality in UAV scenarios (Chen et al., 26 Nov 2025). STORM-Bench adopts a bottom-up annotation pipeline with LLM-assisted object-level caption generation and verification to produce accurate, diverse, and unambiguous referring expressions (Lu et al., 12 Apr 2026).

These benchmarks also progressively widen the semantic regime. ReaMOT Challenge explicitly scores reasoning difficulty and includes spatial reasoning, social interaction, behavioral trends, objective-driven intent, and activity recognition (Chen et al., 26 May 2025). MeViS shifts supervision toward object motion and temporal reasoning, including multiple, single, and zero-target expressions, in both text and audio (Ding et al., 11 Dec 2025). DRMOT and RT-RMOT introduce depth and thermal cues for spatial semantics and low-visibility conditions, while ORMOT and AerialMind target full-field-of-view and aerial viewpoints rather than conventional ground-camera imagery (Chen et al., 4 Feb 2026).

3. Methodological paradigms

The methodological history of RMOT begins with end-to-end transformer tracking. TransRMOT is an online transformer-based architecture inspired by DETR and MOTR, with a CNN visual encoder, a pre-trained text encoder, an early-fusion cross-modal encoder, track queries propagated across frames, detect queries for new objects, and a referent head that predicts class, box, and referent score (Wu et al., 2023). This line was extended by TempRMOT, which adds a query-based temporal enhancement module with an $N \times K$ query memory in a FIFO buffer, a temporal decoder, and track refinement, while also bootstrapping supervision through Refer-KITTI-V2 (Zhang et al., 2024).

Subsequent end-to-end transformer work concentrated on stronger fusion and better temporal conditioning. TenRMOT performs deep multi-modal feature fusion at both encoding and decoding stages through an Interleaving Cross-modality Encoder, a Language Guided Decoder, and a Query Update Module that explicitly uses temporal prior information of tracked objects (Xiao et al., 2024). “Tell Me What to Track” addresses the imbalance between newborn targets and existing targets with Collaborative Query Matching, strengthens multi-scale encoder fusion through an Enhanced Cross-Modal and Multi-Scale Encoder, and injects explicit referring guidance into query tokens through Referring-Infused Query Adaptation (Huang et al., 2024). MLS-Track adds a Semantic Guidance Module that makes queries prompt-aware before decoding and a Semantic Correlation Branch that filters predictions by similarity to the expression (Ma et al., 2024). CDRMT further decomposes language according to “what” and “where” pathways, using Bidirectional Interactive Fusion, Progressive Semantic-Decoupled Query Learning, and Structural Consensus Constraint to preserve modality-specific characteristics, hierarchically inject semantics, and reconstruct language representations from visual features (Liang et al., 14 Mar 2025).

A second paradigm is two-stage RMOT. The data describe a family of methods in which tracking and referring are decoupled, allowing off-the-shelf trackers to remain frozen while only the referring module is trained. MEX is a plug-and-play Memory-Efficient Cross-modality module that replaces heavier referring modules, reducing memory consumption by about 50%, running on a single GPU with 4 GB of memory, and improving HOTA on Refer-KITTI (Tran et al., 19 Feb 2025). JustHook revisits the two-stage setting more fundamentally: it introduces a Visual Feature Hook that extracts context-rich local features directly from the original visual backbone and a Parallel Combined Decoder that replaces passive cosine similarity with active contrastive learning in a combined feature space, while also arguing that fixed dual-tower architectures restrict compatibility with other visual and text backbones (Li et al., 10 Mar 2025).

A third paradigm uses LVLMs or MLLMs as the core reasoning engine. ReaTrack is a training-free framework that combines an LVLM for language-conditioned candidate generation with Online SAM2 for temporal tracking and IoU-based matching via the Hungarian algorithm (Chen et al., 26 May 2025). RTrack, DRTrack, and ORTrack use multimodal large models or large vision-LLMs to process RGB-Thermal-Language, RGB-D-Language, and omnidirectional imagery with language, respectively, and couple these detectors with tracking and association modules tailored to their modality (Yu et al., 25 Feb 2026). STORM generalizes this direction into a fully end-to-end MLLM that jointly performs grounding and tracking without external detectors and learns from image grounding, single-object tracking, and RMOT through Task-Composition Learning (Lu et al., 12 Apr 2026).

A fourth paradigm centers not on architecture alone but on supervision quality. COAL diagnoses a “sparsity-discriminability paradox” in RMOT and addresses it through Explicit Semantic Injection via DINO-X, Counterfactual Learning via Qwen3-Max, and Hierarchical Multi-Stream Integration, thereby moving beyond isolated structural optimization toward knowledge regularization (Jia et al., 14 May 2026).

4. Task extensions beyond canonical single-view RGB RMOT

Once the base task was established, the field diversified into variants that modify either the visual domain or the linguistic regime. CRMOT extends RMOT from single-view tracking to multiple synchronized views with overlapping fields, with the explicit goal of overcoming invisible appearances and maintaining identity consistency in each cross-view. Its benchmark, CRTrack, and method, CRTracker, combine CrossMOT-style multi-view MOT with APTM-based vision-language attribute prompt matching and a prediction module for association fusion (Chen et al., 2024).

ReaMOT extends RMOT in the language dimension. It defines reasoning-based multi-object tracking for complex language instructions with reasoning characteristic, introduces ReaMOT Challenge with three levels of reasoning difficulty, and proposes dedicated metrics and a training-free baseline rather than treating reasoning as a minor prompt-phrasing variation (Chen et al., 26 May 2025). MeViS pushes in a related but distinct direction: it emphasizes motion expressions rather than static attributes, supports multiple, single, and zero-target expressions, and benchmarks RMOT alongside RVOS, AVOS, and referring motion expression generation (Ding et al., 11 Dec 2025). TenRMOT also extends the task to Referring Multi-Object Tracking and Segmentation, and Ref-KITTI Segmentation contains 18 videos with 818 expressions and an average of 10.7 masks per expression (Xiao et al., 2024).

Other extensions alter sensing modalities. RT-RMOT fuses RGB appearance features with the illumination robustness of thermal imagery to enable all-day referring multi-object tracking under nighttime, smoke, and other challenging scenarios (Yu et al., 25 Feb 2026). DRMOT introduces explicit depth as a first-class modality to resolve spatial semantics such as “the person closest to the camera” and improve association under severe occlusion (Chen et al., 4 Feb 2026). ORMOT addresses the limited field of view of conventional cameras by moving to omnidirectional imagery, with the motivation that targets otherwise move out of frame, producing fragmented tracking and poorer support for long-horizon language descriptions (Chen et al., 5 Mar 2026). AerialMind moves RMOT into UAV scenarios characterized by drastic appearance differences, complex spatial relationships, dynamic scene variations, rich and complex referring expressions, and a severe small-object problem (Chen et al., 26 Nov 2025).

These extensions indicate that “RMOT” now denotes a family of language-guided tracking problems rather than a single benchmark template. This suggests that methodological claims should be interpreted in the context of the visual regime, language regime, and metric suite used by each paper, rather than as universally transferable rankings.

5. Evaluation protocols and empirical tendencies

The dominant evaluation protocol in canonical RMOT uses HOTA, with the decomposition

$$\mathrm{HOTA} = \sqrt{\mathrm{DetA} \cdot \mathrm{AssA}},$$

alongside DetA, AssA, DetRe, DetPr, AssRe, AssPr, and LocA (Wu et al., 2023). This metric family remains standard across Refer-KITTI, Refer-KITTI-V2, Refer-UE-City, and many later variants. However, task-specific benchmarks introduce additional metrics when the classical MOT suite is insufficient. CRMOT defines CVRIDF1 and CVRMA as cross-view metrics that aggregate per-language-description IDF1 and matching accuracy (Chen et al., 2024). ReaMOT defines RIDF1, RMOTA, RRcll, and RPrcn to evaluate reasoning-conditioned tracking at the instruction level (Chen et al., 26 May 2025). MeViS adapts HOTA into HOTA$^*$, DetA$^*$, and AssA$S$0, and adds N-Acc. and T-Acc. for the generalized no-target setting (Ding et al., 11 Dec 2025). AerialMind introduces HOTA$S$1 and HOTA$S$2 to summarize scene robustness and motion resilience under UAV attributes (Chen et al., 26 Nov 2025).

Reported results trace a clear progression on mature road-scene benchmarks. TransRMOT reports HOTA 35.54 on Refer-KITTI (Wu et al., 2023). TempRMOT reports 52.21 HOTA on Refer-KITTI and 35.04 on Refer-KITTI-V2 (Zhang et al., 2024). TenRMOT reports HOTA 49.77 on Ref-KITTI for box RMOT and 46.00 on Ref-KITTI Segmentation for RMOTS (Xiao et al., 2024). “Tell Me What to Track” reports 55.63 HOTA on Refer-KITTI and 37.22 on Refer-KITTI-V2 (Huang et al., 2024). CDRMT reports 49.35 HOTA on Refer-KITTI and 31.99 on Refer-KITTI-V2 (Liang et al., 14 Mar 2025). JustHook reports 42.81 HOTA on Refer-KITTI-V2 with a RoBERTa plus ROPE Swin-T combination and states a 7.77% HOTA improvement over TempRMOT on that benchmark (Li et al., 10 Mar 2025). COAL reports 53.38 HOTA on Refer-KITTI and 43.46 on Refer-KITTI-V2, surpassing the previous state of the art by 7.28% HOTA on Refer-KITTI-V2 (Jia et al., 14 May 2026).

On newer task variants, absolute values are often lower, even when methods outperform prior baselines. CRTracker reports 54.88% CVRIDF1 and 35.97% CVRMA on the in-domain CRTrack evaluation (Chen et al., 2024). RTrack reports HOTA 15.53 on RefRT (Yu et al., 25 Feb 2026). DRTrack reports HOTA 33.24 on DRSet (Chen et al., 4 Feb 2026). ORTrack reports HOTA 9.97 in zero-shot evaluation on ORSet (Chen et al., 5 Mar 2026). HETrack reports HOTA 31.46 on AerialMind in-domain and 31.60 cross-domain on UAVDT (Chen et al., 26 Nov 2025). STORM reports HOTA 66.7, AssA 82.6, IDF1 78.3, and MOTA 57.1 on STORM-Bench (Lu et al., 12 Apr 2026). This suggests that benchmark expansion is introducing genuinely harder regimes rather than merely adding more data, and that raw performance numbers should not be compared across datasets without regard to domain, annotation policy, and metric definition.

6. Structural challenges, controversies, and outlook

Several recurrent bottlenecks structure the RMOT literature. One is supervision scarcity and ambiguity. The original benchmark papers already note limited paired prompt-instance data and restricted domain coverage, while later work explicitly identifies sparse semantic supervision, shortcut learning, and semantic collapse as central problems (Ma et al., 2024). COAL formalizes this as the sparsity-discriminability paradox: highly homogeneous scenarios require fine-grained discrimination over complex compositional semantics, but available annotations cover only a small fraction of the combinatorial attribute space (Jia et al., 14 May 2026).

A second issue is the interaction between architectural efficiency and expressivity. One-stage strategies jointly train tracking with referring but suffer from substantial computational overhead, whereas two-stage methods improve efficiency but may take a detour or become constrained by fixed dual-tower alignment (Li et al., 10 Mar 2025). The same tension appears in resource usage: MEX argues that cross-modality reasoning can be made substantially lighter, reporting over 10M fewer trainable parameters than the compared referring module, about 50% lower memory use, and 1.5x faster inference while still improving HOTA (Tran et al., 19 Feb 2025). This indicates that RMOT progress is not solely a matter of larger backbones.

A third issue is target imbalance and long-tail temporal behavior. “Tell Me What to Track” states that prior studies overlook the imbalanced data distribution between newborn targets and existing targets and proposes collaborative matching to improve newborn-target detection without degrading tracking performance (Huang et al., 2024). Small-object sensitivity is repeatedly identified in UAV and driving scenarios, while cross-view, omnidirectional, depth, and thermal variants all originate from explicit failure modes of single-view RGB RMOT under occlusion, limited field of view, low visibility, or ambiguous spatial semantics (Yu et al., 25 Feb 2026).

A fourth issue is robustness and security. “See No Evil” argues that the reliability and robustness of RMOT remain underexplored, identifies adversarial vulnerabilities in both linguistic-visual referring and track-object matching, and shows that FIFO-based temporal memory can preserve corrupted state over multiple subsequent frames. The VEIL framework demonstrates digital and physical perturbations that induce track ID switches and terminations on Refer-KITTI (Bouzidi et al., 2 Sep 2025). This makes robustness a substantive research topic rather than a peripheral deployment concern.

The newest end-to-end MLLM line suggests a different path forward. STORM argues that RMOT data efficiency can be improved by decomposing training into image grounding and object tracking before fine-tuning on RMOT, and reports that performance plateaus with as few as 12,000 RMOT samples in its ablation (Lu et al., 12 Apr 2026). A plausible implication is that future RMOT systems will combine stronger multimodal foundation models, richer benchmark construction, and explicit reasoning or regularization mechanisms, while also addressing efficiency, security, and domain transfer rather than optimizing only HOTA on a single benchmark.