- The paper introduces EAR which leverages similarity-based uni-modal label migration and asymmetric pseudo-label generation to improve supervision in weakly annotated settings.
- It employs temporal-aware transformer encoders and soft-constrained cross-modal fusion to maintain modality balance and address event misalignment.
- Empirical results on UnAV-100 and LLP benchmarks show up to 4.2% F-score improvements and enhanced pseudo-label accuracy over previous state-of-the-art methods.
Enhanced Uni-Modal Representations for Weakly Supervised Audio-Visual Video Parsing
Weakly supervised Audio-Visual Video Parsing (AVVP) targets the segmentation and temporal localization of audio, visual, and audio-visual events in long-form videos, using only coarse video-level, modality-agnostic labels during training. The fundamental challenge is the absence of fine-grained annotations demarcating temporal boundaries and event modalities, which forces reliance on either pseudo-label generation or on complex audio-visual fusion architectures. Existing strategies bifurcate into two paradigms: (1) the production of segment-level pseudo-labels using external pre-trained models or event localization datasets; and (2) architectural enhancements that prioritize joint multi-modal representations. Both approaches are fundamentally constrained—they either propagate noise due to weak semantic guidance for uni-modal events or induce sub-optimal fusion that degrades uni-modal discriminative capacity.
The absence of architectural incentives or explicit supervision to preserve uni-modal information leads to ambiguous event detection—especially under temporal and modality misalignment where, for example, an audio-only event (like "barking" off-camera) must be detected without visual corroboration. Prior research has mostly disregarded this intrinsic misalignment and the independent semantic richness of each modality, yielding models with limited generalization and ambiguous attention.
Technical Approach: The EAR Framework
The EAR (Enhanced Uni-Modal Representation) framework introduces an explicit pipeline to address the above limitations by enhancing both pseudo-label generation and the final AVVP architecture through targeted uni-modal modeling.
Similarity-Based Uni-Modal Label Migration
To provide uni-modal supervision during pre-training (which is absent in previous frameworks), EAR introduces a label migration mechanism: leveraging the DAVE dataset with dense event annotations, uni-modal event labels are extrapolated from audio-visual events by calculating intra-modality segment similarity. Segments with high cosine similarity in a modality inherit the event label of their most similar pair, generating soft, confidence-weighted uni-modal supervision signals. These operate as soft labels, reducing noise impact and encoding label uncertainty—a noted limitation in earlier hard-label pseudo-label generation strategies.
Temporal-Aware Pseudo-Label Generator
The pseudo-label generator is constructed with asymmetric temporal modeling, where modality-specific transformer encoders process input features, separately capturing intra-modality dynamics. Event probabilities are obtained by projecting these features through fixed text embeddings from CLIP/CLAP, with multi-modal event prediction computed via asymmetric combinations of static and dynamic modality probabilities. This design ensures that the generator retains explicit uni-modal event perception alongside the ability to model temporal relationships—expanding its utility beyond prior architectures that bias toward multi-modal events.
Soft-Constrained AVVP Model Architecture
During AVVP model training, the framework introduces two key modules:
- Asymmetric Audio/Visual-Driven Fusion: Dynamic audio features query static visual features (and vice versa) via cross-attention. This approach acts as a soft constraint, preventing one modality from overwhelming event representations in the other, crucial for scenario robustness under event misalignment.
- Multi-Event Relationship Modeling: Event co-occurrence dependencies are explicitly modeled with a relationship module, leveraging convolutional layers and adjacency matrices to encode inter- and intra-modality event relationships. This module iteratively propagates dependency signals, refining both uni-modal and cross-modal attention allocation.
Crucially, the overall loss combines uncertainty-weighted, class-balanced BCE, mixup self-supervision, and standard weakly-supervised video-level loss, providing dense training signal in the absence of strong labels.
Empirical Validation and Results
The framework is validated on UnAV-100 (pseudo-labeling) and LLP (AVVP) benchmarks using both standard (VGGish+ResNet) and advanced (CLIP+CLAP) feature extractors. For AVVP, EAR consistently achieves superior segment- and event-level F-scores relative to SOTA methods, including VALOR [9] and UWAV [10], with improvements of up to 4.2% average F-measure over the best prior work. Pseudo-label accuracy for both audio and visual events also sets new benchmarks, with a delta of +3.8% versus previous SOTA methods on the LLP test set.
Ablation continues to confirm:
- The necessity of similarity-based uni-modal label migration for robust pseudo-labels.
- The asymmetric fusion and multi-event relationship modules, when removed, consistently reduce model performance.
- Soft-constrained, uncertainty-weighted pseudo-labels improve training stability and prevent catastrophic overfitting to noisy signals—a failure mode observed in previous frameworks.
Qualitative analysis highlights improved temporal and semantic event localization, especially under strong modality misalignment. Relationship modeling captures realistic event co-occurrences, and the framework demonstrates generalization capacity even with varying backbone architectures.
Implications and Directions
The EAR framework presents a paradigm shift away from exclusive focus on cross-modal fusion or pseudo-labeling. By explicitly operationalizing uni-modal representations at both the pseudo-label generation and model training phase, it directly addresses the core weakly supervised AVVP challenge: maintaining strong, modality-specific event detection under noisy, coarse-grained annotation regimes. The soft-constraint mechanisms are efficient, avoiding the computational overhead of multi-branch models seen in prior work.
Practically, these advancements facilitate deployment in real-world video analysis scenarios where full annotation is non-viable and where event misalignment is the norm. The strong empirical performance with publicly released pseudo-labels positions the framework as a new baseline for downstream audio-visual event understanding and future work on multi-modal representation learning.
Future directions include developing migration strategies less sensitive to similarity thresholds, improving the robustness of label migration (especially for visual modality), and integrating advanced backbone architectures with multi-scale locality analysis for fine-grained event recognition. The architecture's generality and modular design also provide a foundation for extension to joint audio-visual-LLMs.
Conclusion
The EAR framework robustly enhances uni-modal representation and supervision for weakly supervised AVVP, outperforming previous pseudo-label and fusion-based approaches across numerous metrics and backbone settings (2605.08723). The explicit uni-modal modeling combined with soft constraints and multi-event dependency encoding shifts the baseline for event parsing in complex audio-visual streams, with significant implications for the construction of scalable, label-efficient video understanding systems.