ORMOT: Omnidirectional Referring Multi-Object Tracking

Updated 5 July 2026

The paper introduces ORTrack, a zero-shot LVLM-driven framework that leverages CLIP-based appearance encoding and Hungarian matching to track language-referred objects in 360° video.
ORMOT is defined as extending RMOT to omnidirectional equirectangular imagery, addressing challenges like geometric distortions, edge-wrapping, and long-horizon event descriptions.
The ORSet dataset and annotation pipeline provide detailed omnidirectional language semantics and continuous tracklets, overcoming field-of-view limitations in traditional RMOT.

Omnidirectional Referring Multi-Object Tracking (ORMOT) is the extension of Referring Multi-Object Tracking (RMOT) to 360° imagery. Given an omnidirectional video sequence of equirectangular frames $\{I_t\}_{t=1}^T$ and a natural language description $L$ , the task is to produce a set of trajectories $\mathcal{T}=\{\tau_k\}_{k=1}^K$ , where each $\tau_k=\{b_t^k\}_{t=t_s}^{t_e}$ represents a continuous tracklet corresponding to a language-referred target. The formulation was introduced to overcome the field-of-view limitation of conventional RMOT datasets, where targets frequently move out of frame, tracks fragment, and contextual evidence required for long-horizon descriptions is lost; it is instantiated by the ORSet dataset and the ORTrack framework (Chen et al., 5 Mar 2026).

1. Definition and distinguishing characteristics

ORMOT inherits the core RMOT objective—track only those objects that satisfy a natural-language description—but changes the visual substrate from conventional perspective video to omnidirectional imagery. In standard MOT, the output is a set of trajectories for all tracked instances in predefined categories. In RMOT, language restricts this output to the subset that matches the description. In ORMOT, the same language-conditioned selection is performed on equirectangular 360° frames, which introduces strong geometric distortions, left/right edge-wrapping, and longer visible temporal spans because the entire surrounding scene remains in view (Chen et al., 5 Mar 2026).

The motivation is fundamentally geometric and temporal. Conventional RMOT datasets use narrow field-of-view cameras; only a sector of the scene is visible, targets often exit the frame, and descriptions involving multi-step events can become unresolvable because earlier context is not observed when later actions occur. The canonical example is the description “People who push open the door and then go up the stairs.” In a conventional setup, the “push open the door” event may be missed, so the model tracks all people going upstairs. In an omnidirectional setup, the full scene stays visible, allowing the system to identify who opened the door and continue to track that person over the subsequent action sequence (Chen et al., 5 Mar 2026).

A common misconception is that ORMOT is simply RMOT on a very wide image. The formulation and dataset construction contradict that view. ORMOT explicitly treats boundary crossing, circumferential orientation cues such as “12 o’clock direction,” field-of-view transitions, and projection-aware ambiguities as first-class language phenomena. This makes the task jointly about identity continuity, language grounding, and omnidirectional geometry rather than mere aspect-ratio scaling (Chen et al., 5 Mar 2026).

2. ORSet dataset and annotation pipeline

ORSet is the first dataset introduced specifically for ORMOT. It is built on top of JRDB, which provides omnidirectional RGB imagery, per-frame bounding boxes for people, and person IDs collected using a mobile robot equipped with a 360° omnidirectional camera in real environments such as campuses, cafes, corridors, and open squares. ORSet adds a language layer aligned to these trajectories and to explicitly omnidirectional semantics (Chen et al., 5 Mar 2026).

The dataset uses equirectangular projection with frame resolution $3760 \times 480$ . It contains 27 scenes in total, split into 17 training scenes and 10 test scenes. Among the 27 scenes, 13 are static-camera sequences and 14 are moving-camera sequences. ORSet includes 848 language descriptions, 175 omnidirectional-specific descriptions, and 3,401 annotated referred objects, with about 2.9 million bounding boxes. The average number of frames per description is 575.3, and the average description length is 8.2 words.

Item	Value
Scenes	27
Train / Test	17 / 10
Static / Moving	13 / 14
Language descriptions	848
Omnidirectional-specific descriptions	175
Annotated objects	3,401
Frame resolution	$3760 \times 480$
Average frames per description	575.3
Average description length	8.2 words

The annotation protocol proceeds in three stages. First, annotators perform semantic content selection and temporal segmentation. They review each video, identify salient appearance, behavior, interaction, and omnidirectional phenomena, and use an algorithmic keyframe selection method based on frame-difference or motion magnitude, smoothing, adaptive peak detection, and peak merging. A GPT-based refinement step then checks that selected segments correspond to meaningful semantics such as direction changes, interactions, occlusions, or entering and exiting the scene.

Second, GPT-4o is used to generate candidate descriptions from sampled keyframes and manually summarized semantic cues. These descriptions may focus on appearance, actions, relationships, or omnidirectional properties such as edge reappearance. Human annotators then match each description to trajectory IDs, correct errors, remove hallucinations, and align each description to precise temporal ranges.

Third, a verification and alignment stage ensures that each final annotation forms a consistent triplet of video clip, track IDs, and language description. The final schema associates each description with a scene, one or more temporal frame ranges, and the per-frame boxes and IDs of the referred targets (Chen et al., 5 Mar 2026).

3. Omnidirectional language semantics

A defining feature of ORMOT is that the language is not limited to appearance and short actions. ORSet introduces explicit omnidirectional-specific descriptors organized into four categories, each motivated by 360° projection and continuous surround-view observation (Chen et al., 5 Mar 2026).

Category	Example
Boundary-crossing motion	“The person who disappears from the left edge and reappears from the right edge.”
Circumferential orientation cues	“The person walking at the 12 o’clock direction.”
Projection-aware semantic disambiguation	“The person skateboarding along a straight road.”
FoV transition marking	“The person leaving the room through the doorway.”

Boundary-crossing motion encodes the fact that, in an equirectangular panorama, a single smooth 3D trajectory can cross the image seam and appear discontinuous in pixel space. Circumferential orientation cues exploit absolute camera-centric directions such as “12 o’clock” or “3 o’clock,” which are specific to 360° coverage and provide globally consistent directional references. Projection-aware semantic disambiguation addresses the curvature induced by equirectangular projection, allowing descriptions to preserve real-world motion semantics even when straight lines appear bent. FoV transition marking distinguishes leaving the physical scene from merely leaving a narrow camera crop.

The language statistics reinforce this design. Frequent terms include appearance words such as “black,” “shirt,” “jacket,” and “backpack,” action words such as “walking,” “carrying,” and “holding,” and 360° cues such as “edge” and “camera.” ORSet reports 332 pure-appearance descriptions, 182 pure-action descriptions, and 46 pure omnidirectional-specific descriptions, together with multiple composite categories. Track lengths are long-tailed: many descriptions cover 0–200 frames, a considerable number cover 200–1000 frames, and some exceed 1000 frames. This makes ORMOT especially suitable for long-horizon temporal grounding (Chen et al., 5 Mar 2026).

4. ORTrack framework

ORTrack is the baseline framework proposed for ORMOT. It is an LVLM-driven, zero-shot pipeline: it does not fine-tune on ORSet, but instead composes a large vision-LLM for language-guided detection with CLIP-based appearance encoding and Hungarian matching for cross-frame association (Chen et al., 5 Mar 2026).

The first stage is language-guided detection. Given a frame $I_t$ and description $L$ , the detector is modeled as

$\{ b_t^i \}_{i=1}^{N_t} = \text{LVLM}(I_t, L) = \text{Align}\big(\psi_v(I_t), \psi_l(L)\big),$

where each detection box is

$b_t^i = (x_t^i, y_t^i, w_t^i, h_t^i).$

Because the detector is an LVLM rather than a closed-set object detector, the system operates in an open-vocabulary manner: the model attempts to localize all instances referred to by the description rather than all instances of a fixed class list.

The second stage is two-stage cropping and feature extraction. For each detection, ORTrack forms an expanded global crop and a tight local crop: $L$ 0 with margin ratio $L$ 1. These crops are encoded by frozen CLIP encoders: $L$ 2 and then fused as

$L$ 3

with fusion weight $L$ 4. The motivation is explicitly omnidirectional: local object crops can be small or distorted in equirectangular panoramas, while nearby context can stabilize identity.

The third stage is appearance-based association. For detections in consecutive frames, ORTrack computes cosine similarity

$L$ 5

and solves the assignment problem

$L$ 6

with the Hungarian algorithm. Unmatched detections start new tracks; tracks missing for more than $L$ 7 frames are terminated.

The default implementation uses Qwen2.5-VL-7B as the LVLM, CLIP-ViT-B/32 as the visual feature encoder, and runs on a single NVIDIA RTX A6000. The framework is therefore not an end-to-end trained ORMOT model; it is an algorithmic composition of pre-trained modules targeted at zero-shot omnidirectional referring tracking (Chen et al., 5 Mar 2026).

5. Benchmark performance, ablations, and error modes

ORSet evaluates ORMOT with the RMOT metric suite inherited from prior work: HOTA, DetA, AssA, DetRe, DetPr, AssRe, AssPr, and LocA. No new metric is introduced specifically for 360° video; instead, the omnidirectional difficulty is embedded in the data distribution and language annotations (Chen et al., 5 Mar 2026).

The main zero-shot comparison shows a substantial gap between ORTrack and adapted perspective-RMOT baselines:

Method	HOTA	DetA	AssA	DetRe	DetPr	AssRe	AssPr	LocA
TransRMOT	2.41	1.40	4.24	1.56	12.10	4.29	79.29	75.05
TempRMOT	2.00	0.45	9.01	0.47	11.01	9.44	80.93	78.88
ORTrack	9.97	6.37	16.15	9.20	16.69	17.35	61.80	79.68

These numbers indicate that the principal gains come from both improved detection and improved association under omnidirectional conditions. The paper explicitly notes that perspective RMOT methods adapted to ORSet struggle because they cannot fully exploit omnidirectional geometry, rely on detection modules not tuned for 360° distortions, and do not use LVLM-based open-vocabulary reasoning.

Ablations isolate several design choices. Among tested LVLMs, Qwen2.5-VL-7B performs best, surpassing Qwen2.5-VL-3B, DeepSeek-VL-7B, LLaVA-NEXT-8B, and InternVL3.5-8B. For association features, LVLM embeddings slightly improve HOTA over CLIP features, but CLIP remains the default due to lower computational cost and better efficiency. Appearance-based association also strongly outperforms motion-based OC-SORT under the same detections:

Variant	HOTA	DetA	AssA
DeepSeek-VL 7B	0.12	0.06	0.29
LLaVA-NEXT 8B	1.76	1.11	3.12
InternVL3.5 8B	0.55	0.49	0.72
Qwen2.5-VL 3B	3.33	2.13	5.84
Qwen2.5-VL 7B	9.97	6.37	16.15

Association / feature setting	HOTA	AssA
LVLM features	10.05	16.42
CLIP features (default)	9.97	16.15
LVLM + OC-SORT	5.05	10.19
ORTrack	9.97	16.15

Qualitative analysis identifies both strengths and bottlenecks. ORTrack succeeds on projection-distorted scenes such as “The person skateboarding along a straight road,” on role-based descriptions such as “The moving waiter wearing a black apron,” on high-level intent descriptions such as “The person who is happy and looking forward to it, waiting for others to ask,” and on boundary-crossing trajectories that cross the equirectangular seam. The failure cases fall into two groups. Detection failures include false positives, missed detections, and failures on small or blurred targets under heavy distortion. Association failures include ID switches when people are close, cross paths, or undergo rapid scale and pose changes. The paper observes that Detection Accuracy is generally lower than Association Accuracy, making detection the current bottleneck (Chen et al., 5 Mar 2026).

6. Position within RMOT and panoramic MOT research

ORMOT emerged after a sequence of RMOT developments that were largely defined on perspective video. Those developments established several architectural families: decoupled track-then-refer pipelines with plug-and-play referring heads and memory-efficient tri-modal fusion; end-to-end multimodal transformers with language-guided queries and temporal query updates; encoder-decoder designs that strengthen cross-modal and multi-scale fusion while addressing newborn-target imbalance; cognitive disentanglement strategies that separate “what” and “where” semantics; and zero-shot RMOT through MLLM-generated captions and CLIP-plus-fuzzy matching (Tran et al., 19 Feb 2025, Xiao et al., 2024, Huang et al., 2024, Liang et al., 14 Mar 2025, Chamiti et al., 12 Apr 2025).

Within that perspective RMOT line, several points are especially relevant to ORMOT. The MEX work shows that factorized tri-modal attention can reduce parameters from 92M to 81M and enable inference on a single 4 GB GPU, which is pertinent because 360° imagery raises token counts and memory pressure. TenRMOT contributes an Interleaving Cross-modality Encoder, a Language-Guided Decoder, and a Query Update Module, all designed to exploit long-term temporal information. The robust language guidance work introduces Collaborative Query Matching, Cross-Modal Encoder refinement, and Referring-Infused Query Adaptation to improve newborn target detection and association. CDRMT argues that RMOT should not treat language as a holistic embedding, and instead separates static object semantics from spatial-motion semantics through progressive query injection. ReferGPT demonstrates that zero-shot RMOT can be built by combining a pre-trained tracker, an MLLM, ego-centric spatial summaries, and text-only similarity matching, without RMOT supervision. A plausible implication is that ORMOT can draw from all of these directions, but must anchor them in omnidirectional geometry rather than directly transplanting perspective assumptions.

Panoramic MOT provides a parallel lineage. Before ORMOT, MMPAT studied online panoramic tracking with a 360° panorama and LiDAR on JRDB, using split-detect-merge image detection, projection-based 2D-3D fusion, and multi-cue association over appearance, 2D motion, and 3D location (He et al., 2021). Although MMPAT has no language component, it establishes an important antecedent: omnidirectional tracking is not purely a language problem, and sensor geometry, panoramic detection, and association across wrap-around views remain foundational. AerialMind extends RMOT to UAV scenarios rather than omnidirectional imagery, but its focus on camera rotation, scale variation, low resolution, and wide-area context makes it a useful adjacent benchmark for reasoning about large field-of-view tracking challenges (Chen et al., 26 Nov 2025).

7. Limitations and open directions

The current ORMOT formulation, as represented by ORSet and ORTrack, is deliberately a first benchmark rather than a final solution. The paper identifies four central limitations: detection robustness remains brittle under omnidirectional distortions and crowding; association still suffers from rapid scale changes, pose changes, and seam effects; ORSet covers 27 scenes and focuses on human tracking in JRDB environments rather than broader domains such as traffic or sports; and LVLM inference on large 360° images is computationally heavy, making real-time deployment difficult (Chen et al., 5 Mar 2026).

The technical directions proposed in the paper follow directly from those bottlenecks. For detection, it points to spherical or ERP-aware backbones, distortion-aware anchors or patches, and leveraging depth or multi-fisheye inputs. For association, it points to richer temporal modeling, explicit modeling of edge-crossing and circular continuity, and integration of motion cues compatible with 360° geometry. For applications, it identifies robotics and mobile navigation, AR/VR, and autonomous driving with 360° sensor setups. For reasoning, it highlights multi-step temporal descriptions, causal relations, and interaction-level tracking.

This suggests a broader research program for ORMOT. One plausible direction is to replace ORTrack’s per-frame LVLM detector with a temporally structured multimodal tracker that preserves the zero-shot vocabulary of large vision-LLMs while adding long-term memory. Another plausible direction is to combine omnidirectional geometry with richer semantic decomposition, so that boundary crossing, “12 o’clock” orientation, and multi-stage event language are represented as structured components rather than free-form phrases. A further implication is that the field may converge on hybrid systems: omnidirectional detection and tracking modules grounded in panoramic or 2D+3D sensing, coupled to LLMs that supply open-vocabulary semantics and long-horizon event descriptions. In that sense, ORMOT is both a new benchmark and a new systems problem at the intersection of panoramic perception, multimodal grounding, and identity-preserving video understanding (Chen et al., 5 Mar 2026).