Generic Object Tracking

• Generic Object Tracking (GOT) is the task of tracking any object specified in the first frame using category-agnostic models and robust benchmark protocols.
• It encompasses methods such as Siamese matching, detector-conditioned approaches, and transformer-based architectures to handle occlusion, scale variation, and deformation.
• Benchmark datasets like GOT-10k emphasize generalization over unseen categories and highlight challenges including rapid motion, clutter, and illumination changes.

Searching arXiv for recent and foundational papers on Generic Object Tracking to ground the article. Generic Object Tracking (GOT) is the task of tracking an arbitrary object instance in a video once it has been specified in the first frame, typically by a bounding box. In the standard formulation, a tracker receives a video sequence and the initial target box, then predicts the target state in each subsequent frame while coping with occlusion, distractors, scale variation, deformation, illumination change, fast motion, and category shift. A recurrent theme across the literature is that GOT is category-agnostic: the model must work for “any object” rather than a predefined semantic class, and strong evaluation protocols often emphasize generalization to unseen categories, as in GOT-10k (Zhang et al., 2019, Abdelaziz et al., 2024). Over time, the field has evolved from correlation-filter and discriminative formulations to Siamese template-matching systems, detector-conditioned trackers, transformer-based one-stream architectures, and, more recently, methods that explicitly incorporate geometric reasoning, objectness priors, or tracking-specific model prediction (Meibodi et al., 31 Jul 2025, Dave et al., 2019, Chen et al., 9 Feb 2026).

1. Definition, problem setting, and benchmark protocols

In the standard single-object setting, a tracker is given a video sequence and a bounding box of the target in the first frame, and it must estimate the target’s bounding box in all subsequent frames. This formulation is explicitly described in work spanning detector-based tracking, Siamese tracking, and lightweight unsupervised systems (Dave et al., 2019, Abdelaziz et al., 2024, Zhou et al., 2023). A closely related segmentation-oriented view replaces the box with a mask and propagates the target at pixel level, but the task remains generic if no target class is assumed and the model is required to track arbitrary initialized objects (Kardoost et al., 2019).

A central benchmark in this area is GOT-10k, introduced in the supplied sources as a “large-scale generic object tracking dataset” with “9,334 training sequences,” “180 validation sequences,” and “180 test sequences,” together with “Non-overlapping categories between train and test splits” (Abdelaziz et al., 2024). The testing protocol requires models to be trained “only on the GOT-10k training split,” and the standard metrics are “AO (Average Overlap),” “SR_{0.5},” and “SR_{0.75}” (Abdelaziz et al., 2024). Another account describes GOT-10k as “a large high-diversity database for generic object tracking in the wild with rich motion trajectory and wide coverage of object classes” (Zhang et al., 2019). This class-disjoint design makes GOT-10k a canonical benchmark for category-agnostic generalization.

Longer-term and broader-distribution evaluations are provided by LaSOT, TrackingNet, OxUvA, OTB, UAV123, AVisT, and VOT. The 2025 survey organizes these benchmarks as standard resources for short-term and long-term GOT, and characterizes the field through “discriminative trackers, Siamese-based trackers, and, more recently, prominent transformer-based approaches” (Meibodi et al., 31 Jul 2025). More recent benchmark work argues that existing datasets still understate the difficulty of “universal visual tracking,” leading to VastTrack, which contains “2,115 classes,” “50,610 sequences,” and “4.20 million frames,” with a “hybrid evaluation protocol” in which “Training and test sets have partial overlap in categories: 561 classes appear in both” (Peng et al., 2024). This suggests a shift from category-agnostic evaluation understood narrowly as train/test disjointness toward a broader notion of realistic open-world generalization.

2. Historical paradigms and methodological taxonomy

The major paradigms in GOT can be grouped into discriminative trackers, Siamese trackers, detector-conditioned trackers, and transformer-based trackers (Meibodi et al., 31 Jul 2025). Discriminative trackers formulate tracking as online target-versus-background learning. This line includes correlation-filter systems and later optimization-based trackers such as ATOM, DiMP, and PrDiMP, which the survey treats as central milestones in the pre-transformer era (Meibodi et al., 31 Jul 2025). In contrast, Siamese trackers learn a similarity function between a template and a search region, initially through cross-correlation, later through region proposal networks, anchor-free heads, and refined fusion modules (Zhang et al., 2020, Meibodi et al., 31 Jul 2025).

A distinct conceptual move appears in detector-conditioned tracking. “Learning to Track Any Object” formulates tracking as “finding the right object in a video” and converts “a category-specific object detector into a category-agnostic, object-specific detector” (Dave et al., 2019). Rather than relying primarily on sliding-window template matching, it reuses “Mask R-CNN with FPN” and conditions detection on an online discriminative template obtained by regularized least squares. This suggests that GOT can be interpreted not only as similarity matching but also as target-conditioned detection with a learned objectness prior.

Transformer-based trackers now occupy a dominant role in the literature. The survey distinguishes “Hybrid” and “Fully transformer-based” trackers, including TransT, ToMP, STARK, MixFormer, OSTrack, SeqTrack, VideoTrack, ARTrack, and ODTrack (Meibodi et al., 31 Jul 2025). A representative technical pattern is the one-stream ViT-based framework used by MixFormer, OSTrack, and ODTrack, in which template and search are jointly encoded and the head decodes center, size, and offset maps (Abdelaziz et al., 2024). Within this regime, the dissertation-style account of recent work argues that generalization and online adaptation remain bottlenecks and frames progress in terms of three capacities: “Target discrimination,” “Robust adaptation,” and “Geometric reasoning” (Chen, 1 Jul 2026).

3. Architectures and tracking formulations

Two broad architectural templates recur throughout the supplied literature. The first is template-conditioned similarity learning. In Ocean, a Siamese backbone extracts template and search features, depth-wise cross-correlation combines them, and an anchor-free head predicts classification and box regression outputs (Zhang et al., 2020). Ocean explicitly argues that anchor-based regression is fragile because it is trained only on anchors with “$IoU \geq 0.6$,” whereas anchor-free regression is trained on every pixel inside the ground-truth box, making the tracker “capable of rectifying inexact predictions” (Zhang et al., 2020). Ocean also introduces a “feature alignment module” to construct “object-aware feature” from predicted boxes, followed by a final score

$p_{cls} = \omega p_{o} + (1 - \omega) p_{r}$

that combines regular-region and object-aware classification (Zhang et al., 2020).

The second template is tracking-by-detection or model prediction. In detector-based GOT, the model computes candidate detections $D_t = \{d_{i,t}\}$ with confidences $c_{i,t}$, optionally reweights them temporally via

$$\tilde{c}_{i,t} \leftarrow \alpha c_{i,t} + (1 - \alpha)\, j_{i,t}, \qquad j_{i,t} = \text{IoU}(d_{i,t}, d^*_{t-1}),$$

and selects $d_t^* = \arg\max_i \tilde{c}_{i,t}$ (Dave et al., 2019). In one-stream ViT trackers, the model instead produces a fused embedding

$$\mathbf{F} \in \mathbb{R}^{B \times (H \times W) \times D}$$

from the joint template-search input, then applies a head that predicts a “Center map,” “Size map,” and “Offset map” (Abdelaziz et al., 2024). This head design has recently been revisited: “A Novel Bounding Box Regression Method for Single Object Tracking” argues that the regression head itself is a bottleneck, and that Inception-based and deformable-Inception heads improve GOT-10k performance when plugged into ODTrack and OSTrack (Abdelaziz et al., 2024).

A third, less conventional template uses explicit motion, segmentation, or low-level video cues. “Robust Visual Tracking by Motion Analyzing” states that Video Object Segmentation and Video Object Tracking are complementary and that a “tensor structure, obtained through Tucker2 tensor decomposition, proves to be effective in describing the target’s motion,” leading to competitive results on “LaSOT,” “AVisT,” “OTB100,” and “GOT-10k” with real-time operation (Leo et al., 2023). Because the paper is unavailable as a PDF in the supplied material, only these claims can be stated directly. At a different point in the design space, the “green object tracker (GOT)” proposes three branches: “a global object-based correlator,” “a local patch-based correlator,” and “a superpixel-based segmentator,” with “tiny model size (<3k parameters)” and “around 58M FLOPs per frame” (Zhou et al., 2023). This demonstrates that GOT research also includes lightweight, interpretable, and non-pretrained systems.

4. Objectness, motion, segmentation, and geometric reasoning

One persistent theme in GOT is the use of priors that are not tied to fixed semantic categories. A particularly early and explicit formulation is “Adaptive Objectness for Object Tracking,” which argues that “the target object in tracking must be an object other than non-object” and adapts BING objectness to each sequence using an adaptive SVM (Liang et al., 2015). The adapted objective is

$$\min_{w} \; \|w - \widehat{w}\|_1 \;+\; C \sum_{i=1}^N \left( \max(0, 1 - y_i w^\top x_i) \right)^2,$$

and the final tracker score becomes

$f_{OT}(c_i) = f_T(c_i) + \lambda f_O(c_i).$

The paper emphasizes that this integration is generic and shows gains across seven trackers on OTB-50 and PTB (Liang et al., 2015). This suggests that objectness acts as a transferable prior orthogonal to the particular tracking model.

Motion and segmentation offer another route to genericity. “Object Segmentation Tracking from Generic Video Cues” formulates semi-supervised video object segmentation as a variational minimal partition problem

$$\min_{\Omega_1, \dots, \Omega_n \subset \Omega} \frac{\lambda}{2}\sum_{i=1}^{n}\textrm{Per}(\Omega_i;\Omega) + \sum_{i=1}^{n}\int_{\Omega_i}h_i(x)\, dx,$$

with data terms driven by color and flow features and confidence-based propagation through forward-backward flow consistency (Kardoost et al., 2019). Although the paper is framed as VOS, the supplied notes explicitly characterize it as “a segmentation-based generic object tracker” because it tracks arbitrary initialized objects without class assumptions (Kardoost et al., 2019). A plausible implication is that segmentation and mask propagation can be understood as a dense spatial refinement mechanism for GOT, especially under deformation and partial occlusion.

The most recent conceptual extension is geometry-aware GOT. “GOT-Edit: Geometry-Aware Generic Object Tracking via Online Model Editing” argues that most GOT systems are “flat” 2D systems and proposes integrating geometry-aware cues extracted from RGB by a “Visual Geometry Grounded Transformer” (Chen et al., 9 Feb 2026). Semantic and geometric features are fused as

$p_{cls} = \omega p_{o} + (1 - \omega) p_{r}$0

then geometry-induced perturbations are projected into a semantic null space: $p_{cls} = \omega p_{o} + (1 - \omega) p_{r}$1 yielding final tracking weights

$p_{cls} = \omega p_{o} + (1 - \omega) p_{r}$2

The supplied material reports that this improves robustness under occlusion and clutter, though it also notes the runtime cost of the geometry branch (Chen et al., 9 Feb 2026). This introduces a more explicit 3D inductive bias into GOT without requiring RGB-D input.

5. Training regimes, adaptation strategies, and efficiency trade-offs

A key axis in GOT is the balance between offline training and online adaptation. Siamese trackers often rely heavily on offline pretraining and only limited online updates. Ocean, for example, is trained on “YouTube-BB,” “ImageNet VID,” “ImageNet DET,” “GOT-10k,” and “COCO,” with the backbone first frozen and later fine-tuned end-to-end (Zhang et al., 2020). “Learning to Track Any Object” similarly uses staged training: first standard Mask R-CNN on COCO, then tracking fine-tuning on “ImageNet-VID” and “YouTube-VOS,” and finally a closed-form discriminative template computed at test time (Dave et al., 2019). This staged design treats offline objectness learning and online target adaptation as complementary.

Other systems push much further toward lightweight or training-free operation. The “green object tracker (GOT)” explicitly avoids offline pretraining and instead learns all sequence-specific models from the first frame, including Saab PCA kernels, feature selection, and a two-stage XGBoost classifier (Zhou et al., 2023). It reports “2,199 parameters” and “57.56M FLOPs” per frame, with performance competitive with unsupervised trackers that require heavy pretraining (Zhou et al., 2023). At the multi-object end, McByte is described as “training-free,” using only pre-trained “YOLOX,” “SAM,” and “Cutie” plus fixed thresholds across SportsMOT, DanceTrack, SoccerNet-tracking 2022, and MOT17 (Stanczyk et al., 2 Jun 2025). Although McByte is MOT rather than single-object GOT, it illustrates a broader trend toward modular tracking systems built from general-purpose pretrained components instead of end-to-end tracking-specific training.

Efficiency-aware backbones also matter in GOT. SkyNet is not itself a tracker but a “hardware-efficient neural network” designed as a backbone for SiamRPN++ and SiamMask (Zhang et al., 2019). When used on GOT-10k, “SiamRPN++” with “SkyNet” achieves “AO: 0.364” at “41.22 FPS,” versus “0.365” at “25.90 FPS” with “ResNet-50,” while “SiamMask” with “SkyNet” reaches “AO: 0.390” at “30.15 FPS,” versus “0.380” at “17.44 FPS” with “ResNet-50” (Zhang et al., 2019). The authors frame this as evidence that hardware-aware backbone design can reduce parameters by “37.20×” in the tracking pipeline while maintaining similar or better GOT-10k accuracy (Zhang et al., 2019). This suggests that architectural efficiency remains a first-class concern alongside raw tracking performance.

6. Benchmarks, empirical trends, and open problems

Empirically, recent GOT work shows both rapid progress and persistent fragility. On GOT-10k, the detector-conditioned “Learning to Track Any Object” reports validation AO “73.0” with temporal heuristic, exceeding “SiamMask: 66.8” and “SiamRPN++: 65.8” under its evaluation setting with external data (Dave et al., 2019). Ocean reports GOT-10k AO “0.592” offline and “0.611” online, with the latter matching DiMP and improving upon SiamRPN++ (Zhang et al., 2020). The head-focused ODTrack study reports “ODTrack-B-Inception” at “AO: 77.3,” “$p_{cls} = \omega p_{o} + (1 - \omega) p_{r}$3: 87.5,” and “$p_{cls} = \omega p_{o} + (1 - \omega) p_{r}$4: 75.5” on GOT-10k, improving over baseline “ODTrack-B” (Abdelaziz et al., 2024). These results indicate that improvements can come not only from new backbones or fusion mechanisms, but also from regression-head design and more effective decoding of joint template-search features.

At the same time, broader benchmark analyses argue that current trackers still fall far short of universal tracking. VastTrack evaluates “25 representative trackers” and shows substantial drops relative to LaSOT and TrackingNet; for example, “SeqTrack” falls from “TrackingNet: 0.855” and “LaSOT: 0.725” to “VastTrack: 0.396” SUC (Peng et al., 2024). The benchmark also reports that “Low resolution (LR)” is the hardest attribute, with best SUC around “0.236” (Peng et al., 2024). This suggests that benchmark saturation on established datasets can obscure unresolved problems in long-tail categories, rare object types, and highly adverse scene conditions.

A further contemporary line of thought argues that human-like tracking will require stronger integration of semantics, adaptation, and geometry. The dissertation-scale account “Rethinking Generic Object Tracking Toward Human-Level Perceptual Intelligence” frames recent methods such as PiVOT, GOT-JEPA, and GOT-Edit as progressively enhancing “target discrimination,” “robust adaptation,” and “geometric reasoning capabilities” (Chen, 1 Jul 2026). This suggests that future GOT systems may increasingly resemble modular perceptual systems rather than monolithic template matchers, combining foundation-model priors, prompting, model-space predictive learning, point-based occlusion reasoning, and geometry-aware editing.

A persistent misconception is that GOT is solved once a tracker performs strongly on one benchmark or one architectural family. The supplied literature repeatedly argues against this. The survey emphasizes that occlusions, similar distractors, and appearance variations remain central obstacles (Meibodi et al., 31 Jul 2025). VastTrack demonstrates large distribution shifts even for strong transformer trackers (Peng et al., 2024). Geometry-aware and model-predictive systems show that complementary cues beyond 2D template matching still yield gains (Chen et al., 9 Feb 2026, Chen, 1 Jul 2026). A plausible implication is that GOT remains an open systems problem: progress depends not only on better feature encoders, but also on principled handling of uncertainty, target state adaptation, geometry, and the interaction between generic priors and sequence-specific evidence.