Tracking-Any-Granularity: Adaptive Multi-Level Tracking

• Tracking-Any-Granularity is a research area focused on adapting tracking units—ranging from points to objects and masks—to the problem’s inherent structure rather than using fixed representations.
• It employs hierarchical schemes, such as clip-to-event-to-video in temporal anomaly detection and point-to-pixel methods in spatial tracking, to tailor inference to specific needs.
• This approach unifies disparate domains including video analysis, JavaScript execution contexts, and dialogue state tracking, challenging one-size-fits-all benchmark models.

Tracking-Any-Granularity denotes a family of research formulations in which the tracked unit is not fixed to a single representation, scale, or supervisory level. Across the literature, the phrase covers markedly different objects of inference: temporal anomaly segments at clip-, event-, and video-level in long-video understanding; function invocations under execution context in JavaScript tracking mitigation; arbitrary 2D points and dense pixels in visual correspondence; masks, boxes, and points in unified video tracking; and even slot-conditioned dialogue-history selection in dialogue state tracking (Zhang et al., 2024, Zholus et al., 8 Apr 2025, Zhang et al., 21 Oct 2025). The common thread is not a single benchmark definition, but a move away from one-size-fits-all units of state, evidence, or localization.

1. Scope and meanings of granularity

In the cited work, “granularity” is domain-dependent. Sometimes it denotes spatial support, sometimes temporal scope, sometimes program-analysis units, and sometimes the breadth of semantic categories rather than a part–whole hierarchy.

Setting	Granularity unit	Representative work
Long-video anomaly understanding	clip / event / video	Holmes-VAU (Zhang et al., 2024)
JavaScript tracking mitigation	function under execution context	NoT.js (Amjad et al., 2024)
Point tracking	arbitrary queried points	TAPNext (Zholus et al., 8 Apr 2025)
Anomaly localization	frame / object / pixel	TAO (Huang et al., 5 Jun 2025)
Promptable segmentation	objects / parts / nested masks	Semantic-SAM (Li et al., 2023); UnSAMv2 (Yu et al., 17 Nov 2025)
Unified video tracking	masks / boxes / points	SAM 2++ (Zhang et al., 21 Oct 2025)
Multi-task, multi-modal video understanding	SOT / MOT / VOS / MOTS instances	SATA (Zhang et al., 22 Nov 2025)
Dialogue state tracking	slot-specific turn selection	DiCoS-DST (Guo et al., 2022)

This heterogeneity is especially clear in works whose titles sound similar but solve different problems. “TAO: A Large-Scale Benchmark for Tracking Any Object” expands multi-object tracking from a handful of categories to a bottom-up vocabulary of 833 categories over 2,907 videos; its main advance is broad semantic coverage in an open-world benchmark, not arbitrary spatial support such as parts, masks, or pixels (Dave et al., 2020). “TrackAny3D” likewise targets category-unified 3D LiDAR SOT with a single shared tracker across categories, but remains box-level single-object tracking rather than arbitrary-granularity tracking in the stronger sense (Wang et al., 26 Jul 2025).

A useful implication is that “Tracking-Any-Granularity” functions less as a canonical task name than as a recurring research ambition: to make the unit of tracking adaptable to the structure of the problem rather than predetermined by legacy benchmarks.

2. Temporal-semantic granularity in anomaly understanding

A temporal interpretation of granularity is explicit in "Holmes-VAU: Towards Long-term Video Anomaly Understanding at Any Granularity" (Zhang et al., 2024). There, “at any granularity” does not mean spatial object tracking; it means understanding anomalous content across multiple temporal scales. The framework is organized at three nested levels: clip-level, event-level, and video-level. Clip-level supports local visual perception; event-level supports temporally localized anomalous episodes with anomaly judgment, description, and explanation; video-level supports whole-video judgment, long-form anomaly description, and start-to-end analysis. The benchmark HIVAU-70k is built from UCF-Crime and XD-Violence and contains 5,443 videos, 11,076 events, 55,806 clips, and over 70,000 multi-granular instruction annotations. Holmes-VAU couples this hierarchy with an Anomaly-focused Temporal Sampler (ATS), which defines a cumulative anomaly-density function

$S_{cumsum}(t) = \sum_{i=1}^t (s_i + \tau),$

so that anomaly-rich temporal regions receive denser frame allocation while low-score regions still contribute context. The result is a system that tracks temporal anomaly structure and event evolution rather than identities or trajectories of objects.

"Track Any Anomalous Object: A Granular Video Anomaly Detection Pipeline" reinterprets granularity differently (Huang et al., 5 Jun 2025). TAO spans frame-level, object-level, and pixel-level anomaly analysis. Its pipeline begins with object boxes, scores them with an object-centric VAD model, filters anomalous candidates through a short-term temporal-consistency rule based on IoU counts, and then converts retained prompts into mask tracks through SAM2. The robust filtering condition is written as

$$\sum_{p=i-k}^{i-1} \mathbb{I}\big(\operatorname{IoU}(b_j,b_p) > h\big) \ge m,$$

with a symmetric forward-looking rule for initializing new anomaly labels. This is not generic arbitrary-granularity tracking; it is a pipeline in which one granularity seeds the next: anomalous boxes become temporally filtered box tracks, and box tracks become pixel-level mask trajectories. On UCSD Ped2, the reported result is Pixel-AUROC $75.11$, Pixel-AP $50.78$, Pixel-AUPRO $72.97$, Pixel-F1 $64.12$, RBDC $83.6$, and TBDC $93.2$ (Huang et al., 5 Jun 2025).

Taken together, these works establish a temporal-semantic reading of granularity: clip-to-event-to-video reasoning in Holmes-VAU, and frame-to-object-to-pixel anomaly localization in TAO. In both cases, the tracked entity is not primarily an object identity, but a structured anomalous process unfolding over time.

3. Spatial granularity: points, pixels, boxes, and dense fields

A spatial reading of Tracking-Any-Granularity is most explicit in point and dense motion tracking. "TAPNext: Tracking Any Point (TAP) as Next Token Prediction" recasts point tracking as sequential masked token decoding (Zholus et al., 8 Apr 2025). A query point $(t,x,y)$ is expanded into a sequence of $T$ point-track tokens, one per frame, and concatenated with video-patch tokens of shape

$$\sum_{p=i-k}^{i-1} \mathbb{I}\big(\operatorname{IoU}(b_j,b_p) > h\big) \ge m,$$0

The backbone combines a temporal SSM with a spatial ViT; coordinates are predicted as discrete distributions over $$\sum_{p=i-k}^{i-1} \mathbb{I}\big(\operatorname{IoU}(b_j,b_p) > h\big) \ge m,$$1 bins with truncated soft-argmax, and visibility is predicted separately. The paper argues that arbitrary visual entities can be represented as collections of tracked points: sparse keypoints, semi-dense grids, dense correspondences, object parts, boundaries, thin structures, and textureless regions. This is a direct spatial interpretation of “any granularity,” because point sets can instantiate different granularities without changing the tracking primitive.

"DELTA: Dense Efficient Long-range 3D Tracking for any video" occupies the opposite end of the spectrum (Ngo et al., 2024). Rather than starting from sparse queries, it tracks every pixel. The output tensor is

$$\sum_{p=i-k}^{i-1} \mathbb{I}\big(\operatorname{IoU}(b_j,b_p) > h\big) \ge m,$$2

with per-pixel trajectory state

$$\sum_{p=i-k}^{i-1} \mathbb{I}\big(\operatorname{IoU}(b_j,b_p) > h\big) \ge m,$$3

DELTA first tracks densely at reduced resolution using joint global-local attention, then upsamples trajectories to full resolution with a transformer-based upsampler. The paper reports dense 3D tracking at scale, over 8x faster than previous methods, and identifies log-depth as the best depth representation. Relative to Tracking-Any-Granularity, DELTA is best viewed as a dense substrate from which sparse points, patches, and region summaries can be derived, rather than as a native query-any-granularity interface.

"NetTrack: Tracking Highly Dynamic Objects with a Net" introduces a finer but still box-centric notion of granularity (Zheng et al., 2024). The final output remains MOT-style object trajectories, yet association is made more robust by sampling multiple points of interest inside a tracked box and matching candidate detections by point containment. The area penalty is

$$\sum_{p=i-k}^{i-1} \mathbb{I}\big(\operatorname{IoU}(b_j,b_p) > h\big) \ge m,$$4

and the fine-grained similarity depends on how many tracked internal points fall inside a candidate box. This is not arbitrary-granularity output, but it shows how sub-box point structure can stabilize box-level tracking under deformation, fast motion, and occlusion.

These systems represent three distinct spatial strategies. TAPNext treats points as a universal primitive; DELTA makes dense per-pixel 3D trajectories the base representation; NetTrack keeps box-level outputs but injects point-level internal evidence. The shared idea is that coarse object support alone is often insufficient.

4. Unified tracking and segmentation systems

Several works aim to unify tasks or representations across masks, boxes, points, and modalities. The clearest statement is "SAM 2++: Tracking Anything at Any Granularity" (Zhang et al., 21 Oct 2025). SAM 2++ defines tracking granularity explicitly as masks for VOS, boxes for SOT, and points for PT. Its Unified Decoder first predicts a mask-form intermediate and then derives task outputs as

$$\sum_{p=i-k}^{i-1} \mathbb{I}\big(\operatorname{IoU}(b_j,b_p) > h\big) \ge m,$$5

The model uses task-specific prompts, a task-adaptive memory mechanism, and the Tracking-Any-Granularity (TAG) dataset, which contains 6,000 videos, 2,200,891 frames, 2,148,716 masks, 2,148,716 boxes, and 2,640,987 points. This is one of the strongest literal realizations of the phrase, because the same online tracker is trained and benchmarked across three spatial support formats.

"Tracking and Segmenting Anything in Any Modality" extends unification in a different direction (Zhang et al., 22 Nov 2025). SATA is strongest on task and modality unification rather than semantic part–whole granularity. It uses DeMoE to separate cross-modal shared knowledge from modality-specific knowledge and TaMOT to cast SOT, VOS, MOT, and MOTS as instance association. Unified representation learning is written as

$$\sum_{p=i-k}^{i-1} \mathbb{I}\big(\operatorname{IoU}(b_j,b_p) > h\big) \ge m,$$6

followed by

$$\sum_{p=i-k}^{i-1} \mathbb{I}\big(\operatorname{IoU}(b_j,b_p) > h\big) \ge m,$$7

The framework supports RGB, RGB-T, RGB-D, and RGB-E, and reports results on 18 benchmarks. Its notion of granularity is task/output granularity: single vs multi-object, box vs mask, tracking vs segmentation.

Segmentation foundation models provide another axis of unification. "Semantic-SAM: Segment and Recognize Anything at Any Granularity" is not a tracker, but it makes one click generate multiple masks spanning objects, parts, and nested regions by duplicating a prompt into $$\sum_{p=i-k}^{i-1} \mathbb{I}\big(\operatorname{IoU}(b_j,b_p) > h\big) \ge m,$$8 level-conditioned queries; each query is

$$\sum_{p=i-k}^{i-1} \mathbb{I}\big(\operatorname{IoU}(b_j,b_p) > h\big) \ge m,$$9

and decoded jointly with image features (Li et al., 2023). "UnSAMv2: Self-Supervised Learning Enables Segment Anything at Any Granularity" adds continuous granularity control to SAM-2. It assigns each pseudo-mask a scalar

$75.11$0

encodes $75.11$1 into a prompt embedding, and conditions a granularity-aware mask token on it (Yu et al., 17 Nov 2025). Since SAM-2 is video-capable, this gives a direct route from promptable segmentation at controlled scale to video propagation, even though the paper itself does not introduce a new tracker.

Language-aligned foundations further broaden what counts as granularity. "PixCLIP" learns any-granularity pixel-text alignment by combining mask-conditioned visual input with long-form text and a three-branch objective $75.11$2 (Xiao et al., 6 Nov 2025). "Griffon" shows that a plain LVLM can localize all objects from free-form text at any granularity by generating outputs in the textual form label-[x1, y1, x2, y2], handling one-vs-one, one-vs-many, many-vs-many, and nonexistent referents without expert detectors or special localization heads (Zhan et al., 2023). These works are not trackers, but they supply the query, segmentation, and localization primitives from which any-granularity tracking systems can be built.

5. Beyond vision: execution context and adaptive context selection

Tracking-Any-Granularity is not confined to visual state spaces. In "Blocking Tracking JavaScript at the Function Granularity," the tracked unit is the JavaScript function under dynamic execution context (Amjad et al., 2024). NoT.js instruments browser execution, records the call stack and scope chain, and builds a graph with five node types: JS function nodes, DOM element nodes, Network nodes, Storage nodes, and Web API nodes. Crucially, one source-level function can generate multiple nodes depending on calling sequence and context, so the operational granularity is closer to function-invocation-under-context than to plain source-function identity. A random forest classifier over structural and contextual features achieves precision 94.3%, recall 98.0%, and F1-score 96.2% on about 361 thousand JS functions, and automatically generates surrogate scripts that block 84% of tracking JS function calls without causing breakage on 92% of websites. Here, “granularity” refers neither to space nor time, but to the unit at which tracking behavior is detected and neutralized.

"Beyond the Granularity: Multi-Perspective Dialogue Collaborative Selection for Dialogue State Tracking" pushes the notion further into language understanding (Guo et al., 2022). DiCoS-DST argues that dialogue history should not be consumed at a single fixed window size for all slots; rather, each slot update should select the relevant turn-level evidence dynamically. For slot $75.11$3, the model ranks dialogue turns using three views—explicit slot-name to dialogue relation, current-turn to dialogue-history relevance, and Implicit Mention Oriented Reasoning—and fuses them before selecting the top $75.11$4 turns for generation. The slot-conditioned selection step

$75.11$5

is followed by top-$75.11$6 turn retrieval and cascaded context refinement. The best setting reaches 61.13 joint accuracy and 98.06 slot accuracy on MultiWOZ 2.2. In this setting, granularity means the effective amount and distribution of dialogue history, not spatial support or category breadth.

These non-visual cases clarify that the core abstraction behind Tracking-Any-Granularity is broader than “track objects at multiple scales.” It is the problem of choosing the right unit of state, context, or intervention for a given inference target.

6. Recurring patterns, boundaries, and unresolved issues

Across the literature, several recurrent design patterns appear. A first pattern is unification by common latent form plus task-specific prompts or readouts. SAM 2++ uses mask-form pre-output for masks, boxes, and points (Zhang et al., 21 Oct 2025); SATA converts SOT, MOT, VOS, and MOTS into instance association with calibrated IDs (Zhang et al., 22 Nov 2025); Griffon serializes different localization tasks into the same autoregressive text format (Zhan et al., 2023). A second pattern is hierarchical supervision or data engines: HIVAU-70k provides clip/event/video supervision (Zhang et al., 2024), TAG provides aligned masks, boxes, and points (Zhang et al., 21 Oct 2025), and UnSAMv2 constructs mask–granularity pairs from unlabeled images (Yu et al., 17 Nov 2025). A third pattern is context-sensitive selection rather than fixed windows or fixed supports: ATS samples anomaly-dense frames (Zhang et al., 2024), NoT.js distinguishes the same function across calling contexts (Amjad et al., 2024), and DiCoS-DST selects dialogue turns per slot (Guo et al., 2022).

The same literature also makes the limits of the phrase explicit. Holmes-VAU tracks temporal anomaly structure rather than object identity (Zhang et al., 2024). TAO supports frame/object/pixel anomaly analysis, but not a universal multi-scale tracker for arbitrary regions or scene-wide events (Huang et al., 5 Jun 2025). TAO the benchmark broadens object vocabulary yet does not formalize arbitrary semantic granularity (Dave et al., 2020). NetTrack remains a box-level MOT system even though it uses sub-box point cues (Zheng et al., 2024). TrackAny3D is category-unified 3D SOT, not arbitrary-granularity tracking (Wang et al., 26 Jul 2025). SATA is stronger on cross-task and cross-modal generalization than on part/object/group hierarchy (Zhang et al., 22 Nov 2025). Semantic-SAM, UnSAMv2, PixCLIP, and Griffon contribute crucial segmentation, alignment, or grounding primitives, but do not themselves solve temporal identity persistence (Li et al., 2023, Yu et al., 17 Nov 2025, Xiao et al., 6 Nov 2025, Zhan et al., 2023).

A plausible synthesis is that Tracking-Any-Granularity has become a convergent research direction rather than a single task definition. In the strongest sense, it would require a system that can initialize, represent, associate, and explain targets across points, parts, objects, masks, regions, events, and contextual abstractions while preserving temporal or logical identity. The surveyed work shows that different subfields have solved different slices of this program: point-based universal primitives, dense per-pixel correspondences, continuous segmentation granularity, multimodal task unification, context-sensitive program analysis, and hierarchical temporal anomaly reasoning. What remains unresolved is a single formulation that unifies these granularities without reducing them to one privileged representation.