NOOUGAT: Unified Online-Offline Graph Tracker

• NOOUGAT is a unified graph-based framework for multi-object tracking that adapts its behavior by controlling a key temporal parameter (subclip size T).
• It integrates a hierarchical GNN for within-subclip associations with an Autoregressive Long-term Tracking (ALT) layer for fusing tracklets across subclips.
• The architecture demonstrates improved metrics (AssA, IDF1, HOTA) across benchmarks like DanceTrack and MOT, confirming its scalability and efficiency.

Searching arXiv for the cited paper and closely related MOT references mentioned in the source material. NOOUGAT, short for Neural Online and Offline Unified Graph Architecture for Tracking, is a graph-neural-network-based framework for multi-object tracking (MOT) that is designed to operate across arbitrary temporal horizons, rather than treating online and offline tracking as separate problem classes. In tracking-by-detection MOT, detections $\mathcal{O}=\{o_i\}_{i=1}^n$ are partitioned into object trajectories $\mathcal{T}=\{T_k\}_k$, where each trajectory is an ordered set of detections from the same physical object. NOOUGAT formulates the online–offline distinction as a question of temporal horizon and implements a single learned architecture whose behavior is controlled by the subclip size $T$ and by graph connectivity, enabling strict causal tracking, intermediate buffered tracking, and large-batch offline tracking within one framework (Missaoui et al., 2 Sep 2025).

1. Problem formulation and motivation

In the formal view adopted by NOOUGAT, the difference between online and offline MOT lies in what temporal information is available when associations are produced at time $t$. In online tracking, the tracker may use only detections up to time $t$, $\mathcal{O}_{\le t}=\{o_i:t_i\le t\}$, and must satisfy causality, low latency, and irrevocability of past decisions under strict evaluation. In offline tracking, the tracker receives the entire sequence $\mathcal{O}_{1:C}=\{o_i:t_i\in[1,C]\}$ and can use future frames, perform global optimization, and tolerate high latency. This is summarized by a temporal horizon $H$: online corresponds to $H=0$, offline to $H=C-t$, and intermediate settings to $\mathcal{T}=\{T_k\}_k$0.

The motivation for a unified model arises from limitations on both sides of the literature. Modern online trackers of the SORT family—including SORT, DeepSORT, ByteTrack, OC-SORT, HybridSORT, and DiffMOT—typically combine a hand-crafted motion model, optional ReID features from a separate network, and heuristic association, often via Hungarian matching with tuned weights and cascaded stages. The paper identifies three consequences: frame-by-frame myopia, brittle multi-stage logic, and limited temporal modeling. Offline graph-based trackers such as MPNTrack, SUSHI, and CoNo-Link can reason over larger temporal windows, but often assume bounded clip length and use overlapping sliding windows with heuristic stitching across overlaps.

NOOUGAT is motivated by deployment regimes that occupy the full spectrum between these extremes. The paper lists autonomous driving, CCTV and surveillance, tele-surgery or tele-operation, aerial vehicle tracking, and unconstrained offline analytics as examples with distinct latency budgets. A single model that can operate at arbitrary horizon $\mathcal{T}=\{T_k\}_k$1, is fully learned, and scales to arbitrarily long sequences is presented as both practically desirable and conceptually cleaner.

2. Unified architecture and temporal control

NOOUGAT operates on non-overlapping subclips of length $\mathcal{T}=\{T_k\}_k$2 and combines two components: a hierarchical GNN for within-subclip association and an Autoregressive Long-term Tracking (ALT) layer for cross-subclip fusion (Missaoui et al., 2 Sep 2025). Given a video of $\mathcal{T}=\{T_k\}_k$3 frames, the sequence is partitioned into

$\mathcal{T}=\{T_k\}_k$4

with $\mathcal{T}=\{T_k\}_k$5.

The pipeline begins with a detector, specified as YOLOX, which produces bounding boxes and confidence scores, and a separate ReID network, specified as ResNet50-SBS, which produces appearance embeddings per detection. For MOT17 and MOT20, missing detections are interpolated under the standard protocol. Each subclip is then processed by a hierarchical GNN, producing subclip-local tracklets $\mathcal{T}=\{T_k\}_k$6. These are fused autoregressively:

$\mathcal{T}=\{T_k\}_k$7

and for subsequent subclips,

$\mathcal{T}=\{T_k\}_k$8

The subclip size $\mathcal{T}=\{T_k\}_k$9 serves as the key temporal control parameter. With $T$0, the system operates in frame-by-frame strict online mode. With moderate $T$1, the model buffers several frames and reasons over richer incoming evidence. With large $T$2, such as $T$3, the same framework functions as an offline tracker with extensive temporal context. The paper reports that on DanceTrack, AssA and IDF1 improve steadily as $T$4 increases from 1 to 256, indicating that the architecture scales monotonically with available temporal context.

A plausible implication is that NOOUGAT treats online and offline tracking not as different architectures, but as different operating points of one learned association mechanism.

3. Graph formulation, hierarchical GNN, and ALT layer

NOOUGAT uses the standard graph-based MOT formulation in which a graph $T$5 represents detections or tracklets as nodes and candidate associations as edges. Each edge $T$6 carries a binary label $T$7 indicating whether the connected nodes belong to the same identity. After $T$8 message-passing steps, the edge classifier predicts

$T$9

Within each subclip, NOOUGAT reuses the hierarchical architecture from SUSHI. Level 0 operates on detections and connects adjacent or near-adjacent frames. Higher levels recursively merge short tracklets into longer ones until the full subclip length is covered. Edge and node features include geometry and temporal information, motion, appearance, and detection confidence. Message passing follows standard MPN-style updates:

$t$0

$t$1

with aggregation typically taken as a sum or mean. The weights are shared across hierarchy levels.

Two additions are emphasized. First, the model generalizes Velocity Direction Consistency (VDC) from “track versus detection” to “track versus track.” For two tracklets $t$2 and $t$3, with forward and backward velocity vectors $t$4 and $t$5, VDC is defined as

$t$6

This angle measures directional consistency when two tracklets are expected to meet. Second, following CoNo-Link/SPAM, the model appends bounding-box dimensions and detector confidence to the initial node embeddings so that the GNN can learn when low-confidence detections should be trusted or downweighted.

The ALT layer operates at the tracklet level and connects past trajectories $t$7 with incoming tracklets $t$8. Its node features aggregate geometry, motion, appearance, and detection confidence over a tracklet; its edge features encode relative geometry, temporal gap, motion consistency, appearance similarity, and VDC. The same learned GNN with shared weights is used for both intra-subclip association and inter-subclip fusion. This is a central unification claim: the model does not switch to a separate heuristic stitcher when the temporal horizon grows.

4. Online, intermediate, and offline operating modes

The ALT layer supports two graph connectivity modes. In offline full-graph mode, the node set is $t$9 and the edge set is all pairings in $t$0, permitting past–past, incoming–incoming, and past–incoming interactions. This allows already processed associations to be revised in light of new evidence. In online bipartite mode, edges are restricted to

$t$1

which enforces causal association between active past tracks and current observations only.

In both modes, ALT classifies candidate edges as same-identity or different-identity. The paper further states that for each past track, only the top 10 most similar incoming tracklets, measured by geometry, appearance, and motion, are connected; this sparsification controls complexity. Edges classified as same identity are used to merge incoming tracklets into past trajectories, and unmatched incoming tracklets start new trajectories.

The deployment distinction is therefore controlled by wiring rather than by retraining. In online mode, the configuration is $t$2 with bipartite connectivity. Incoming tracklets degenerate to single-frame detections, earlier decisions are not revised, and appearance is updated with EMA. In offline mode, the reported configuration is $t$3 with full graph connectivity, no causality constraint, and appearance represented by simple averaging over all detections in the track. The paper explicitly states that the same model parameters are used in both regimes; only $t$4, graph connectivity, and appearance aggregation during inference differ.

This design suggests a continuous interpolation between strict causal tracking and full-sequence retrospective association, rather than a binary separation between two incompatible tracker families.

5. Training objectives and inference procedure

Training is formulated as supervised edge classification. For any two detections or tracklets $t$5, the label is $t$6 if they belong to the same ground-truth identity and are temporally consistent, and $t$7 otherwise. Both the hierarchical GNN and ALT are trained jointly with shared weights using Focal Loss with focusing parameter $t$8:

$t$9

The stated purpose is to address class imbalance, since negative edges far outnumber positive ones.

The training schedule follows progressive unfreezing inherited from SUSHI: one hierarchy level is unfrozen every 500 iterations, and ALT training is enabled only after the hierarchy has become reasonably trained. For ALT specifically, the training procedure samples random incoming subclips and random past windows, introduces a skip of up to 8 frames to simulate occlusions, and randomly drops past tracks and detections to simulate track loss and noise. The paper states that this teaches ALT to bridge long temporal gaps and handle partial observations.

Optimization uses Adam with learning rate $\mathcal{O}_{\le t}=\{o_i:t_i\le t\}$0 and weight decay $\mathcal{O}_{\le t}=\{o_i:t_i\le t\}$1. The online version is trained for 100k iterations with batch size 32, and the offline version for 50k iterations with batch size 8. The detector and ReID network are fixed; there is no joint training with detection.

At inference time, the algorithm proceeds by preprocessing detections and ReID embeddings, partitioning the video into non-overlapping subclips of length $\mathcal{O}_{\le t}=\{o_i:t_i\le t\}$2, initializing the trajectory set from the first subclip, and then iterating over later subclips through local hierarchical tracking, ALT graph construction, ALT message passing, and trajectory update. The paper notes that while no particular LP or matching solver is specified after edge classification, one uses the positive edge scores to form a consistent association. The claimed novelty lies in the fact that the edge scores driving matching and stitching are learned by the GNN rather than determined by hand-crafted logic.

6. Benchmarks, quantitative results, and ablations

The experimental evaluation spans DanceTrack, SportsMOT, MOT17, and MOT20 (Missaoui et al., 2 Sep 2025). The paper reports dataset characteristics as follows: DanceTrack has 100 videos and 106k frames with high appearance similarity and non-linear motion; SportsMOT has 240 videos and 150k frames with high-speed motion, frequent occlusions, and emphasis on long-term identities; MOT17 has 14 sequences and 11k frames; MOT20 has 8 sequences, 13k frames, and more than 2M boxes. Metrics include HOTA, DetA, AssA, IDF1, and MOTA, with AssA described as the primary metric of interest.

In online mode, NOOUGAT improves over the listed state of the art using the same YOLOX detections. On DanceTrack test, HybridSORT reports AssA 52.6, IDF1 67.4, HOTA 65.7, whereas NOOUGAT online reports AssA 54.9, IDF1 70.6, HOTA 65.9. On SportsMOT test, DiffMOT reports AssA 65.1, IDF1 76.1, HOTA 76.2, whereas NOOUGAT online reports AssA 74.3, IDF1 85.3, HOTA 81.0. On MOT17 test (private), DiffMOT with extra training data reports AssA 64.6, IDF1 79.3, HOTA 64.5, while NOOUGAT online reports AssA 65.3, IDF1 80.0, HOTA 65.2; the paper also notes that NOOUGAT is extremely lightweight at approximately 27k parameters and uses no extra data. On MOT20 test (private), DiffMOT reports AssA 60.5, IDF1 74.9, HOTA 61.7 and HybridSORT reports IDF1 78.4, while NOOUGAT online reports AssA 67.5, IDF1 80.2, HOTA 64.6.

In offline mode with $\mathcal{O}_{\le t}=\{o_i:t_i\le t\}$3, the same architecture improves further. On DanceTrack test, CoNo-Link reports AssA 50.7, IDF1 64.1, HOTA 63.8, while NOOUGAT offline reports AssA 58.7, IDF1 72.7, HOTA 68.4. On SportsMOT test, NOOUGAT offline reports AssA 83.0, IDF1 92.3, HOTA 85.6, compared with its own online values of 74.3, 85.3, and 81.0. On MOT17 test (private), CoNo-Link reports AssA 67.8, IDF1 83.7, HOTA 67.1, while NOOUGAT offline reports AssA 68.5, IDF1 83.9, HOTA 66.9. On MOT20 test, CoNo-Link reports AssA 68.0, IDF1 81.8, HOTA 65.9, while NOOUGAT offline reports AssA 69.7, IDF1 82.0, HOTA 65.6.

The ablation studies serve a structural role in the paper’s argument. On SportsMOT, increasing the retention window beyond the approximately 30-frame regime used by many SORT-style trackers continues to improve NOOUGAT, indicating learned long-term recovery rather than special-case reconnection heuristics. On DanceTrack validation, replacing ALT with Hungarian matching using geometry, motion, VDC, and appearance with tuned HybridSORT weights yields HOTA 62.1, IDF1 65.8, MOTA 88.8, whereas ALT yields HOTA 64.6, IDF1 69.0, MOTA 87.5. For offline stitching on the same validation set, a SUSHI reproduction with heuristic stitching yields HOTA 67.9, IDF1 72.0, MOTA 88.9, while NOOUGAT yields HOTA 69.8, IDF1 75.2, MOTA 89.0. Another ablation reports that on DanceTrack validation in online mode, geometry alone gives HOTA 59.1 and IDF1 61.9, geometry plus motion gives HOTA 60.4 and IDF1 64.4, and geometry plus motion plus appearance gives HOTA 64.6 and IDF1 69.0. For offline training parameters on DanceTrack validation, baseline NOOUGAT yields HOTA 68.4 and IDF1 73.5, adding VDC yields HOTA 69.5 and IDF1 74.9, and adding past-track augmentation yields HOTA 70.5 and IDF1 76.6.

7. Efficiency, limitations, and position in the literature

NOOUGAT’s scalability rests on two design choices. First, the use of non-overlapping subclips plus ALT means each frame is processed once by the hierarchy, unlike overlapping sliding-window methods that repeatedly process the same frames. Second, the hierarchical GNN reduces node and edge counts at higher levels, enabling subclips as long as 256 frames. Runtime is reported on DanceTrack validation with precomputed YOLOX and ReID features on a V100: approximately 12 FPS in online mode with $\mathcal{O}_{\le t}=\{o_i:t_i\le t\}$4, and approximately 340 FPS in offline mode with $\mathcal{O}_{\le t}=\{o_i:t_i\le t\}$5. The paper states that graph construction accounts for about 48% of online runtime because a new graph is built every frame.

The limitations are correspondingly specific. Online runtime is bottlenecked by Python-level graph construction; the method remains detector-dependent as a tracking-by-detection system; and there is no explicit global optimality guarantee comparable to network flow or multicut approaches because association is performed by GNN classification plus simple decision logic. Future directions identified in the paper include C++/CUDA graph construction, better caching and incremental graph updates, adaptive or learned horizon selection through the $\mathcal{O}_{\le t}=\{o_i:t_i\le t\}$6 parameter, extension to 3D MOT and multi-camera tracking, richer contextual cues, and joint detection-tracking training.

Within the broader MOT literature, NOOUGAT is positioned between three established families. Relative to SORT-style online trackers, it replaces the full hand-crafted association stack with a learned GNN layer while remaining within the tracking-by-detection paradigm. Relative to end-to-end transformer approaches such as TrackFormer, MOTR, MeMOTR, and MOTRv2, it focuses on association and data efficiency rather than joint detection-query tracking. Relative to graph-based offline systems such as MPNTrack, SUSHI, and CoNo-Link, it removes heuristic stitching across overlapping windows by introducing the autoregressive ALT layer over non-overlapping subclips. The paper’s central novelty claim is therefore threefold: a unified online–offline architecture, a learned ALT layer that handles both matching and stitching, and strong association performance across both causal and non-causal regimes (Missaoui et al., 2 Sep 2025).