Towards Fine-grained Temporal Perception: Post-Training Large Audio-Language Models with Audio-Side Time Prompt

Abstract: Large Audio-LLMs (LALMs) enable general audio understanding and demonstrate remarkable performance across various audio tasks. However, these models still face challenges in temporal perception (e.g., inferring event onset and offset), leading to limited utility in fine-grained scenarios. To address this issue, we propose Audio-Side Time Prompt and leverage Reinforcement Learning (RL) to develop the TimePro-RL framework for fine-grained temporal perception. Specifically, we encode timestamps as embeddings and interleave them within the audio feature sequence as temporal coordinates to prompt the model. Furthermore, we introduce RL following Supervised Fine-Tuning (SFT) to directly optimize temporal alignment performance. Experiments demonstrate that TimePro-RL achieves significant performance gains across a range of audio temporal tasks, such as audio grounding, sound event detection, and dense audio captioning, validating its robust effectiveness.

• The paper introduces a novel Audio-Side Time Prompt mechanism that interleaves timestamp embeddings with audio tokens to enhance temporal localization.
• It employs adaptive RL post-training using Group Relative Policy Optimization to optimize event-based F1 and mIoU, thereby improving detection accuracy.
• Ablation studies and attention analysis confirm that semantic initialization and adaptive rewards are pivotal for robust, fine-grained temporal perception.

Fine-Grained Temporal Perception Enhancement in LALMs: The TimePro-RL Framework

Background and Motivation

Large Audio-LLMs (LALMs), integrating sophisticated audio encoders with powerful LLMs, have established general audio understanding capabilities and achieved substantial performance across tasks including acoustic scene classification, audio captioning, and audio QA. Despite these advances, LALMs manifest substantial deficiencies in fine-grained temporal perception, particularly in their inability to reliably infer event boundaries—onset and offset timestamps—for temporally grounded tasks such as audio grounding and sound event detection. Previous efforts have relied on time-annotated datasets and time token injection, but explicit modeling of physical temporal cues remains elusive, and supervised fine-tuning (SFT) objectives are misaligned with temporal localization requirements.

Audio-Side Time Prompt: Explicit Temporal Encoding

The paper introduces the Audio-Side Time Prompt (ASTP) mechanism, which encodes timestamps as explicit embeddings interleaved within the audio feature sequence. This embedding process utilizes a semantic initialization strategy that averages subword token embeddings of the numerical timestamp representation, preserving linguistic prior and stabilizing the embedding space. These Timestamp Embeddings are frozen during training, preventing semantic drift and enforcing consistent temporal grounding.

Interleaving timestamp embeddings with audio tokens restructures the input to the LALM, allowing the attention mechanism to directly access ambiguous time references. The autoregressive nature of LALMs is leveraged so that the model can retrieve temporal information from adjacent timestamp embeddings during token generation.

Figure 1: The architecture of TimePro-RL, where Timestamp Embeddings are interleaved with audio features and training proceeds via SFT and RL post-training.

RL-Driven Temporal Optimization

To counteract the inadequacy of SFT, which penalizes small temporal prediction deviations excessively via token-level cross-entropy, the TimePro-RL framework deploys post-training with Group Relative Policy Optimization (GRPO) and task-specific rewards. The main reward metric is Event-based F1 (Eb-F1), augmented with an adaptive mechanism: when Eb-F1 lacks discriminative power within sampled prediction groups, a continuous auxiliary reward such as mean Intersection over Union (mIoU) is fused to restore advantage gradience. This adaptive reward mechanism dynamically shifts the training objective based on real-time variance in reward signal, optimizing both discrete temporal alignment and smooth localization.

Experimental Results

The framework is evaluated across audio grounding (AG), sound event detection (SED), and dense audio captioning (DAC) on FTAR and DESED datasets. Zero-shot baselines (Qwen2-Audio, Qwen2.5-Omni) exhibit low recall and Eb-F1 on temporally precise metrics. SFT-adapted LALMs (including Kimi-Audio, Audio-Flamingo2) achieve moderate improvements, but TimePro-RL post-trained models surpass all baselines substantially. Specifically, TimePro-RL elevates Qwen2.5-Omni’s recall at [email protected] on AG from 34.1 (SFT baseline) to 39.8, and DAC Eb-F1 from 35.2 to 40.7, without compromising METEOR scores that measure linguistic quality—demonstrating robust holistic optimization.

Ablation studies reveal that random initialization of Timestamp Embeddings induces performance regression, whereas semantic initialization achieves notable gains. Only Eb-F1 reward leads to imbalances and linguistic quality degradation in DAC; the adaptive reward recovers METEOR and further enhances temporal localization.

Figure 2: Visualization of Timestamp Embedding attention weights; high concentration at onset and offset boundaries confirms precise temporal grounding capability.

Interpretability via Attention Analysis

Analysis of attention weights in the model’s final layer demonstrates a sharp concentration on Timestamp Embeddings coinciding with event boundaries in the mel-spectrogram. This direct alignment validates that the model learns to utilize timestamp prompts for accurate temporal localization, reinforcing ASTP’s interpretability.

Implications and Future Directions

The TimePro-RL paradigm offers a methodological advancement for audio-language modeling that aligns temporal prediction objectives with fine-grained localization metrics, integrates explicit temporal cues at the audio input level, and leverages context-sensitive RL signals for robust optimization. Practically, this enables deployment in audio-centric tasks where precise event boundary detection is critical: audio event retrieval, real-time surveillance analytics, and temporally aligned multimodal reasoning. Theoretically, the explicit timestamp embedding strategy and adaptive reward-based RL open avenues for similar interventions in other modalities (video, sensor streams) and complex generative tasks involving temporal reasoning.

Ongoing work can extend this framework to multi-hop and chain-of-thought reasoning, where temporal cues are pivotal as intermediate evidence, and to hierarchical event segmentation in long-form audio. A prospective direction is exploring the scalability of timestamp embeddings in longer audio streams, adaptive sampling strategies for group RL, and cross-modal transfer for video-to-audio temporal grounding.

Conclusion

TimePro-RL demonstrates a principled framework for enhancing the fine-grained temporal perception capabilities of LALMs through explicit timestamp prompting and adaptive RL post-training. By optimizing for precise temporal alignment and integrating robust semantic initialization, the approach achieves considerable performance improvements on temporally grounded audio tasks, establishes interpretability through attention analysis, and lays the foundation for further advances in temporal reasoning within multimodal LLM architectures.