AVTrack: Audio-Visual Tracking in Human-centric Complex Scenes

Abstract: Audio-visual speaker tracking aims to localize and track active speakers by leveraging auditory and visual cues, enabling fine-grained, human-centric scene understanding. This capability is essential for real-world applications such as intelligent video editing, surveillance, and human-computer interaction. However, existing datasets are largely limited to simple or homogeneous audio-visual scenes with coarse annotations. Such oversimplified settings bias evaluation toward static audio-visual co-occurrence, rather than rigorously assessing robust spatiotemporal modeling and cross-modal reasoning in complex, dynamic scenes. To address these limitations, we introduce AVTrack, a human-centric audio-visual instance segmentation (AVIS) dataset designed for dynamic real-world scenarios. AVTrack features diverse and challenging conditions, including camera motion, visual occlusions, and position changes. Evaluations of representative AVIS methods on AVTrack reveal substantial performance degradation, establishing AVTrack as a challenging benchmark for robust human-centric audio-visual scene understanding in complex environments. We further provide a simple yet effective baseline to facilitate future research. Project website: https://FudanCVL.github.io/AVTrack/

• The paper presents a novel benchmark (AVTrack) for human-centric audio-visual tracking that challenges models with complex scenarios such as occlusion, background changes, and multi-turn dialogue.
• The methodology leverages dynamic windowing, speaker chunk aggregation, and a dual-stage local and global reasoning process to maintain persistent identity across frames.
• The proposed AVTracker framework achieves a HOTA score of 29.08, outperforming previous models and highlighting its robust performance in challenging real-world settings.

AVTrack: Advancing Human-centric Audio-Visual Tracking in Complex Scenes

Introduction and Motivation

Audio-visual speaker tracking—localizing and temporally associating active speakers using joint auditory and visual cues—is central to high-fidelity, human-centric scene understanding. This research domain underlies diverse applications, including intelligent video editing, HCI, and surveillance. While the literature has made substantial progress in audio-visual instance segmentation (AVIS) [guo2025aviseg], most benchmarking has relied on datasets whose scene dynamics are limited, with a bias toward static, single-instance, and laboratory-controlled settings. Such oversimplified contexts fail to probe the capacity of models for robust, long-range spatiotemporal audio-visual reasoning, especially in realistic human-centric environments characterized by occlusion, dynamic interactions, and complex temporal structure.

To address these limitations, "AVTrack: Audio-Visual Tracking in Human-centric Complex Scenes" (2606.02724) introduces AVTrack—a rigorously curated, test-only benchmark targeting challenging human-centric AVIS scenarios. This benchmark evaluates model generalization and robustness across diverse, non-trivial conditions that push beyond static co-occurrences, emphasizing the need for advanced audio-visual alignment, persistent identity maintenance, and agentic reasoning.

AVTrack Dataset: Composition and Real-world Complexity

AVTrack comprises 871 video clips, each densely annotated at the instance and timeline level, yielding 3,120 tracklets. The dataset is defined by explicit complexity criteria, with each video required to contain at least one of the following: visual occlusion, background switch, camera motion, relative position change, audio-visual inconsistency, multi-turn sounding, instance scale dynamics, or multiple simultaneous human instances. This ensures a comprehensive coverage of the real-world challenges commonly absent in prior benchmarks.

A direct comparison of condition distributions between AVTrack and AVISeg shows that AVTrack features far higher rates of visual occlusion (80.9% vs. 8.6%), scale variation (56.9% vs. 35.3%), background switches (60.5% vs. 5.9%), multi-turn dialogue (56.8% vs. 16.0%), and pronounced camera motion (90.5% vs. 7.1%) (Figure 1).

Figure 1: Comparison of AVTrack and AVISeg data distributions across different challenging conditions; AVTrack consistently presents higher prevalence of the most difficult scenarios for audio-visual tracking.

Video sources are highly heterogeneous, spanning TV series, films, vlogs, animation, and stage performances (Figure 2). This diversity ensures that benchmark performance reflects generalization to in-the-wild human-centric video, rather than overfitting to narrow domains (e.g., scripted dialogue or actor-centric views).

Figure 2: Video source distribution in AVTrack, highlighting multi-domain coverage for stress-testing model robustness.

Importantly, AVTrack is distributed as a test-only set, decoupling model evaluation from dataset-specific training and providing a stable reference for measuring progress in human-centric AVIS.

AVTracker: Modular Baseline for Human-centric Audio-Visual Tracking

To facilitate and baseline future research, the paper introduces AVTracker—a modular, three-stage framework for fine-grained, multi-speaker audio-visual tracking (Figure 3). AVTracker leverages advanced pre-trained models, including Whisper for ASR, ECAPA-TDNN for speaker representation, and large vision-LLMs (Qwen3-VL) for cross-modal reasoning. The segmentation backbone is SAM3, and MossFormer2 handles optional speech separation.

Figure 3: Overview of AVTracker's three-stage framework for human-centric AVIS: (1) Speaker chunk aggregation, (2) local window process, and (3) global window process for persistent identity association.

Key pipeline stages:

• Speaker Chunks Aggregation: Dynamic windowing identifies contiguous segments of speech, merged based on speaker embedding similarity, yielding semantically coherent units for subsequent reasoning (Figure 4).
• Local Window Process: For each speaker chunk, local association fuses transcript content and visual features to detect/person-segment the currently active speaker, producing temporally localized tracklets.
• Global Window Process: Aggregates local tracklets across the entire video, resolving persistent identity via multi-view cross-frame reasoning.

  Figure 4: Dynamic windowing paradigm preserves semantic and temporal continuity, contrasting with fixed-size sliding window approaches.

The framework is designed for plug-and-play extensibility, enabling straightforward integration of stronger backbone models and alternative reasoning modules as they become available.

Empirical Analysis: Benchmarking and Ablation

AVTrack sets a more stringent bar than previous human-centric AVIS datasets. Across multiple baseline models—including VITA, LBVQ, CAVIS (VIS only), AVISM, ACVIS, and the new AVTrackFormer (end-to-end AVIS)—all approaches experience a large drop in performance on AVTrack compared to simpler datasets. Visual-only methods perform especially poorly (HOTA <12), highlighting the necessity for joint audio-visual modeling. AVIS methods, despite leveraging audio information, still report HOTA below 21.

AVTracker achieves HOTA = 29.08 (+8 over the best end-to-end AVIS baselines), also exceeding all competing approaches on detection, association, and identity tracking metrics. Ablation confirms that large-scale reasoning models, effective speech separation (when automatic separation quality is high), and dynamic chunk processing all contribute positively to performance. The modular aggregation of transcript, speaker embeddings, and visual reasoning emerges as central to robust long-term identity tracking.

Qualitative Assessment and Error Analysis

Qualitative analysis demonstrates that AVTracker effectively maintains identity through speaker movement and camera changes, outperforming end-to-end baselines particularly in scenarios with dynamic positioning, multi-turn dialogue, or audio-visual inconsistency (Figure 5).

Figure 5: Qualitative comparison between AVTracker predictions and ground truth; robust identity tracking through challenging position changes, but failures in occlusion and multi-speaker overlap remain.

Failure cases highlight persistent open problems: ambiguous tracklet association during close-range occlusions, merging/splitting identities in crowded scenes, and residual errors under heavy audio-visual misalignment.

AVTrackFormer and Comparative Fusion Architectures

Complementing the modular pipeline, AVTrackFormer implements a more tightly integrated end-to-end AVIS architecture. Unlike AVISM, it features bidirectional audio-visual fusion at the object token level (Figure 6, Figure 7), facilitating richer reciprocal interactions during object tracking. Despite these architectural enhancements, AVTrackFormer trails AVTracker, suggesting current end-to-end paradigms may be limited by inflexible input windowing and suboptimal cross-modal disambiguation in extreme conditions.

Figure 6: AVTrackFormer architecture overview—bidirectional audio-visual fusion between object tokens and audio features for improved temporal association.

Figure 7: Audio-Visual Object-level Fusion Module (AV-OFM) interleaves cross-modal reasoning in an iterative, bidirectional manner.

Implications, Limitations, and Future Research Directions

AVTrack exposes fundamental gaps in the current state-of-the-art for human-centric audio-visual understanding:

• Even the best AVIS methods, including commercial LLMs (e.g., Gemini 2.5 Pro), fall short under real-world complexity (HOTA ≤21 and far below AVTracker).
• Audio-visual association is fragile in the presence of long-term occlusion, multi-turn dialogue, and cross-modal inconsistency. Persistent identity reasoning remains an unsolved engineering and algorithmic challenge.

Future work should focus on:

• Robust audio-visual alignment utilizing semantic bridges (e.g., utterance-level transcript context, multimodal graph structures).
• Integrated agentic reasoning, leveraging explicit memory and self-correction across windows to adaptively refine hypotheses about identity during ambiguous or fragmented input.
• Scalable human-centric data construction for training, emphasizing annotated person–time–location triplets and challenging interactional scenarios.
• Ensembling or orchestration of foundation models for open-world audio-visual scene analysis, coupled with efficient quantization and batched inference for practical deployment.

AVTrack’s benchmark structure and reported per-scenario breakdown (see supplement) can guide architectural enhancements for targeted robustness under the hardest conditions, e.g., audio-visual inconsistency, small-scale instance tracking, and crowded scene re-identification.

Conclusion

AVTrack constitutes a rigorous human-centric benchmark that elevates the requirements for robust multimodal perception beyond static, oversimplified scene contexts. By coupling comprehensive scene complexity with strong annotation fidelity and sophisticated baselines, AVTrack provides the community with a precise diagnostic for progress in real-world audio-visual localization and tracking. These contributions set the stage for advances in persistent identity association, long-horizon agentic reasoning, and accountable, privacy-preserving deployment of human-centric AVIS systems.