SpotSound: Enhancing Large Audio-Language Models with Fine-Grained Temporal Grounding

Abstract: Large Audio-LLMs (ALMs) have recently demonstrated remarkable capabilities in holistic audio understanding, yet they remain unreliable for temporal grounding, i.e., the task of pinpointing exactly when an event occurs within long-form audio. This limitation stems from two factors: training data dominated by clip-level supervision lacking precise timestamps, and benchmarks that fail to simulate real-world scenarios where short events are obscured by dense background sounds. In this paper, we introduce SpotSound, an audio LLM designed for grounding audio events. SpotSound incorporates a novel training objective, specifically designed to suppress hallucinated timestamps for events absent from the input. Additionally, we present SpotSound-Bench, a challenging temporal grounding benchmark where target events occupy less than ~10\% of each clip, creating a rigorous `needle-in-a-haystack' evaluation. Experiments demonstrate that SpotSound achieves state-of-the-art results on temporal grounding benchmarks while maintaining robust performance across general downstream audio-language tasks. Code, models and benchmark are released on https://loiesun.github.io/spotsound/

• The paper introduces an explicit temporal encoding mechanism that interleaves timestamp tokens with audio embeddings for precise event localization.
• The approach leverages a synthetic benchmark alongside real-world datasets to achieve significant improvements in mean IoU and binary event verification.
• A discriminative quadruplet training objective is employed to suppress hallucinations, ensuring robust detection of both event presence and accurate temporal boundaries.

SpotSound: Fine-Grained Temporal Grounding for Large Audio-LLMs

Motivation and Problem Setting

Despite substantial advances in large audio-LLMs (ALMs), current systems lack precise temporal grounding—the ability to localize exactly when in an audio sequence a queried event occurs. Dominant ALMs are typically trained using clip-level, coarse-grained annotations and evaluated on benchmarks where target events are temporally isolated and easily detectable. This undermines performance in realistic settings where fleeting acoustic events are embedded in continuous, complex auditory scenes with dense, dynamic backgrounds.

SpotSound is proposed as an explicit solution to this gap, enabling large-scale ALMs to achieve fine-grained, open-vocabulary temporal localization and robust verification of event existence, even in challenging "needle-in-a-haystack" scenarios.

Model Architecture and Temporal Encoding

SpotSound introduces an explicit temporal encoding mechanism, integrating timestamp tokens directly with audio embeddings. For an input audio stream and free-form language query, SpotSound constructs an interleaved sequence of textual timestamp tokens at a fixed granularity (default 1 s), paired with the corresponding audio tokens, then concatenated with the text query and an instruction indicator.

The resulting sequence structure is:

Figure 1: Overview of the SpotSound architecture and dataset generation, illustrating timestamp-token and audio-token interleaving, and synthetic dataset creation via foreground/background mixing with ground-truth timestamp preservation.

Such explicit marking enables LLMs (Qwen2-Audio or Audio Flamingo 3 backbones) to learn direct associations between event semantics and temporal boundaries, rather than relying on dense or implicit positional encodings which degrade temporal precision.

Additionally, SpotSound employs an autoregressive, instruction-augmented prediction paradigm. The inference proceeds in two stages: first, the model is queried for event existence, producing a binary output ("Yes"/"No"); if present, event localization proceeds as a natural language generation task, producing start and end timestamps for all matching intervals.

Dataset and Benchmark Construction

To train and evaluate temporal grounding, SpotSound leverages both existing datasets (AudioGrounding, Clotho-Moment, UnAV-100, AudioSet Strong Label) and a newly constructed synthetic corpus. The synthetic pipeline randomly inserts labeled foreground events (from AudioSet or VGGSound, with captions generated by DeepSeek-v3 or Qwen2-Audio) into natural background ambience, maintaining precise, timestamped boundaries and maximizing diversity while preserving class priors.

Category and duration statistics confirm that the dataset covers a wide range of event types and time spans:

Figure 2: Category distribution of the synthetic training set, reflecting diverse event coverage and balanced sampling.

Figure 3: Distribution of audio clip durations and event window lengths in SpotSound-Bench, highlighting the challenge of sparse, short events within long-form audio.

A key contribution is SpotSound-Bench, a benchmark of 300 long-form audio clips (mean duration ≈ 53 s) with ground-truth labels for short-duration events (mean event < 10% of clip), designed to rigorously evaluate fine-grained grounding under complex, real-world background conditions.

Training Objectives and Hallucination Suppression

SpotSound applies a discriminative quadruplet training objective to enforce robust verification and hallucination suppression. Each training instance comprises: audio input, a positive query (event present), ground-truth timestamps, and a negative query (event absent). The negative sampling strictly precludes lexical overlap with present events.

The model is supervised to output explicit "No" predictions when queried about absent events. This forces the model to rely on acoustic evidence rather than spurious correlations, substantially reducing the frequency of false positive (hallucinated) responses.

Experimental Results

Temporal Grounding

SpotSound establishes new state-of-the-art performance across multiple temporal grounding benchmarks, including both short-clip and long-form datasets. Comparing SpotSound-Q (Qwen2 backbone) and SpotSound-A (Audio Flamingo 3 backbone) against specialized and general ALMs demonstrates:

• SpotSound-A achieves substantial boosts in mean IoU over prior SOTA: +4.7% (Clotho-Moment), +27.0% (UnAV-100), +2.9% (AudioGrounding), and +20.4% (SpotSound-Bench).
• On SpotSound-Bench, which features extremely low event density, both SpotSound-Q/A outperform all baselines by large margins, demonstrating superior temporal resolution and generalization across distributions.

Qualitative analysis corroborates these findings:

Figure 4: SpotSound correctly localizes events in time (a), identifies negative cases (b), and demonstrates strong improvements in quantitative benchmarks (c) compared to top-tier and proprietary models.

Figure 5: Qualitative comparison on SpotSound-Bench and Clotho-Moment. SpotSound provides temporally precise intervals, while other LLM-based models either misalign semantically or hallucinate events.

SpotSound demonstrates robustness to common failure modes: baseline ALMs often generate time windows even for absent events (hallucination); SpotSound reliably predicts event absence on negative queries.

Event Existence Verification

SpotSound significantly improves the accuracy of event presence detection, as measured on balanced positive/negative query splits, with up to +18.8% (Clotho-Moment) and +8.1% (AudioGrounding) improvements in binary accuracy over strong ALM baselines.

Downstream Sound Event Detection and Ablation

Evaluation on classic SED datasets (TUT 2017, DESED) confirms that SpotSound maintains strong event localization and SED performance despite its fine-grained grounding focus.

Extensive ablations demonstrate that explicit timestamp interleaving is critical—relaxing audio encoder segment limits or simple backbone fine-tuning yield modest improvements, but temporal tokenization alone provides >10% mIoU gains and robustifies the model across all test regimes. Timestamp granularity impacts sensitivity–optimally set at 1s for balanced efficiency and performance.

Limitations and Future Directions

While SpotSound considerably advances temporal grounding for open-vocabulary ALMs, several challenges remain. Temporal precision for extremely short, overlapping, or multi-instance events could be limited by both dataset annotation quality and autoregressive generation truncation. Improved reasoning over polyphonic and densely overlapping events, as well as scaling to ultra-long-form audio and more naturalistic query types, are necessary for robust deployment in real-world surveillance, multimedia indexing, and forensics applications.

Conclusion

SpotSound introduces an explicit, timestamp-interleaved alignment and robust, hallucination-suppressing training protocol for large audio-LLMs, enabling fine-grained, reliable temporal grounding. Together with SpotSound-Bench, these advances allow systematic progress towards ALMs capable of real-world time-sensitive acoustic reasoning. The demonstrated improvements in temporal localization fidelity and negative query verification position SpotSound as a strong baseline for future research on reliable, temporally-aware audio-language understanding.

(2604.13023)