SOIBench: Semantic Guidance in SOT

• SOIBench is a benchmark designed to assess semantic cognitive guidance by isolating failures caused by similar object interference in single object tracking.
• It automatically mines SOI frames and annotates them with hierarchical natural-language guidance to enhance evaluation of vision-language tracking.
• The benchmark reveals that modern SOT systems relying on shallow appearance matching experience persistent drift when confronted with target-like distractors.

SOIBench is a benchmark for semantic cognitive guidance in single object tracking (SOT), introduced to isolate and evaluate failures caused by Similar Object Interference (SOI), a condition in which non-target objects resemble the designated target closely enough to induce multiple high-confidence peaks in a tracker’s confidence heat-map (Wang et al., 13 Aug 2025). The benchmark is built on the premise that modern SOT systems, including Siamese-based and Transformer-based trackers, rely predominantly on shallow appearance matching and therefore exhibit persistent drift when fine-grained semantic discrimination is required. SOIBench operationalizes this problem by automatically mining SOI frames, annotating them with hierarchical natural-language guidance, and evaluating both vision-language tracking and VLM-assisted correction under controlled SOI-specific settings.

1. Similar Object Interference as a tracking bottleneck

SOI refers to situations in single-object tracking where one or more non-target objects visually resemble the designated target so closely in color, shape, texture, or context that the tracker’s confidence heat-map exhibits multiple high-confidence peaks. These multi-peak responses indicate ambiguous visual evidence and are associated with continuous fine-grained discrimination failures, ultimately producing drift (Wang et al., 13 Aug 2025).

Within the SOIBench formulation, the significance of SOI is not merely that distractors exist, but that they are sufficiently target-like to induce a structured failure mode that conventional aggregate tracking benchmarks do not isolate. Prior benchmarks report overall performance without separating SOI-driven errors from other sources of degradation. This creates a measurement problem: trackers may appear competitive in aggregate while remaining fundamentally weak in the specific regime where semantic disambiguation is required.

The benchmark’s central claim is that SOI is a primary constraint on robust SOT