Charades-AudioMatter: Audio-Driven Retrieval

• Charades-AudioMatter is a curated test subset from Charades-STA where each instance is selected for having informative, temporally aligned audio that materially aids localization.
• It is constructed using a rigorous multi-stage manual screening process with over 95% inter-annotator agreement, ensuring only audio-relevant queries are retained.
• The associated IMG model employs an importance-aware multi-granularity fusion of audio and visual features, demonstrating significant performance gains on video moment retrieval metrics.

Charades-AudioMatter is a curated test subset of Charades-STA for video moment retrieval in which, for every query–video pair, the audio track is treated as materially informative for localizing the correct temporal moment. It was introduced together with the Importance-aware Multi-Granularity fusion model (IMG) to isolate the contribution of audio in audio–vision–text retrieval, while preserving the underlying Charades-STA training setup. In operational terms, the training split remains the original Charades-STA training set, whereas the test split is replaced by a manually screened subset of instances whose audio is valid, relevant to the query, and temporally aligned with the labeled moment (Lin et al., 6 Aug 2025).

1. Lineage within the Charades family

Charades-AudioMatter inherits its video domain from Charades, the “Hollywood in Homes” dataset of everyday household activity videos recorded in real homes. Charades contains 9,848 annotated videos with an average length of 30 seconds, 267 people from three continents, 27,847 video descriptions, 66,500 temporally localized intervals for 157 action classes, and 41,104 labels for 46 object classes (Sigurdsson et al., 2016). Its defining properties are long untrimmed clips, dense multi-action structure, fine-grained object-centric activities, and realistic domestic environments.

The immediate parent benchmark is Charades-STA, which was built on top of Charades by adding sentence temporal annotations for language-driven temporal localization. In that setting, the task is to take an untrimmed video and a natural-language query sentence and predict the start and end times of the moment described by the sentence. Charades-STA established sentence-to-time grounding on Charades-style indoor videos and became a standard benchmark for temporal sentence grounding and video moment retrieval (Gao et al., 2017).

Charades-AudioMatter preserves that Charades-STA task structure but changes the evaluation emphasis. Standard Charades-STA contains many queries for which audio is irrelevant or weakly informative, such as visually dominated actions. Charades-AudioMatter was created to answer a narrower question: whether a retrieval model can exploit audio when audio is genuinely useful for temporal localization (Lin et al., 6 Aug 2025).

2. Construction and curation procedure

Charades-AudioMatter is constructed from the test split of Charades-STA by a multi-step manual screening pipeline. The resulting test set contains 1,196 samples, while the training set remains exactly the original Charades-STA training set (Lin et al., 6 Aug 2025).

The curation protocol uses six postgraduate students experienced in multimodal learning. Each instance is independently labeled by two annotators, disagreements are resolved by a third annotator, and the final inter-annotator agreement is reported as greater than 95%. This design makes the “audio matters” criterion a human judgment rather than an automatic heuristic.

The first screening stage evaluates the validity of the audio. Samples containing significant background noise or lacking any sound are excluded, because such audio “lacks meaningful information and cannot contribute effectively to VMR.” This is an intentionally coarse filter that removes obviously unusable audio tracks.

The second stage evaluates correlation between audio and query text. Annotators consider both audio and video, but the central criterion is whether the query depends on or is significantly helped by audio. Queries describing static or purely visual actions, such as “sit on the bed,” “look out the window,” or “stand in the kitchen,” are usually excluded because the audio does not provide meaningful cues. Queries associated with characteristic sounds, including “laugh,” “close the door/closet,” “open the cabinet,” “run,” and “pour water,” are retained only when the corresponding sound is actually audible in the specific instance. The decision is therefore instance-specific rather than class-specific.

The third stage enforces temporal alignment between audio and the ground-truth moment. Annotators manually annotate timestamps using only the audio and the query text, compute the IoU between that audio-based timestamp and the official Charades-STA ground-truth moment, and discard the sample if the IoU is below 0.3. This eliminates cases in which audio is relevant in a broad semantic sense but misaligned with the annotated visual moment.

Taken together, these stages enforce three simultaneous conditions: the audio must be present, semantically useful, and roughly co-temporal with the target event. That combination is the defining criterion of Charades-AudioMatter.

3. Data model, annotations, and task definition

Because Charades-AudioMatter is derived from Charades-STA, each sample consists of an untrimmed video, its synchronized audio track, a natural-language query, and inherited temporal boundaries for the target moment. In the formulation used by IMG, the video is represented as $V = \{f_t\}_{t=1}^{T}$, the synchronized audio as $A = \{a_j\}_{j=1}^{T}$, and the text query as $Q = \{w_i\}_{i=1}^{N}$. The target annotation is the ground-truth moment $[f_s, f_e]$ inherited directly from Charades-STA (Lin et al., 6 Aug 2025).

The benchmark task is standard video moment retrieval, also called temporal sentence grounding. Given $V$, $A$, and $Q$, the model predicts start and end indices $(\hat{f}_s, \hat{f}_e)$ of the moment matching the query, with inference constrained by $\hat{f}_s \le \hat{f}_e$. The split convention is asymmetric by design: training is conducted on ordinary Charades-STA, and evaluation is conducted either on the original Charades-STA test set for overall performance or on Charades-AudioMatter for audio-sensitive performance.

The feature setup described for Charades-AudioMatter uses I3D for visual features, pre-trained audio CNNs with PANN on AudioSet for Charades-STA and Charades-AudioMatter, and GloVe-initialized word embeddings. The synchronized audio is split into clips aligned with the corresponding visual frame intervals. This makes the dataset directly usable by audio–vision–text retrieval pipelines without redefining the basic supervision format.

No explicit human annotation is provided for per-sample “audio importance.” Audio-relatedness is enforced at the dataset level through manual selection and IoU-based temporal verification. In IMG, any scalar notion of audio importance is learned on the model side through pseudo-labels derived from branch losses rather than through a separate dataset annotation channel.

Evaluation follows standard VMR metrics. The benchmark reports $R@n@\mu$, where $A = \{a_j\}_{j=1}^{T}$0 and $A = \{a_j\}_{j=1}^{T}$1, corresponding to IoU thresholds of $A = \{a_j\}_{j=1}^{T}$2, $A = \{a_j\}_{j=1}^{T}$3, and $A = \{a_j\}_{j=1}^{T}$4, together with mean IoU (mIoU). On Charades-AudioMatter, these metrics quantify whether audio-aware fusion improves localization when audio is known to be useful.

4. Semantic profile of the selected subset

The curation procedure changes the semantic composition of the test set. Category-wise analysis shows that selected activities are biased toward actions with distinctive or at least potentially informative sounds. Reported selected categories include open (door/cabinet/…) with 241 instances, close (door/closet/…) with 150, put (bag/groceries/…) with 138, run with 90, turn on/off (light/tv/…) with 89, throw (broom/shoes/…) with 66, take (vacuum/food/…) with 56, laugh with 52, eat with 41, wash (hand/glass/…) with 29, drink with 28, walk with 25, cook with 22, pour (water/coffee/…) with 16, sit down with 13, and talk with 10 (Lin et al., 6 Aug 2025).

The unselected side is dominated by actions whose evidence is largely visual or weakly audible. Reported unselected categories include sit (on bed/chair/…) with 218 instances, hold with 147, (un)dress with 111, look with 85, stand with 59, smile with 55, watch with 48, awake with 38, read with 32, take a picture with 30, play (phone/camera/…) with 23, snuggle with (pillow/…) with 20, (fix/adjust) hair with 19, and lay with 18.

The paper emphasizes that activity type alone is insufficient. Even for an action class such as “open the door,” one instance may be retained if the door sound is clearly audible and another may be discarded if the corresponding sound is absent or too faint. The benchmark is therefore not simply a soundful-action taxonomy; it is an instance-level audio-relevance filter over Charades-STA.

Temporal statistics are more stable than semantic statistics. The normalized moment-duration distribution of Charades-AudioMatter is reported to be similar to that of the original Charades-STA, and the subset “maintains comparable diversity and roughly follows the original Charades-STA in duration distribution.” The bias introduced by the curation is therefore primarily semantic—toward audio-dependent moments—rather than a collapse to unusually short or unusually long intervals.

Qualitative examples reported for the subset are consistent with that profile. One example is “closes the window,” where the window is partly hidden by curtains and the closing sound disambiguates the moment. Another is “laugh,” where the visual evidence is subtle and the acoustically salient laughter provides the critical cue. These examples illustrate the intended failure mode for visual-only retrieval on Charades-AudioMatter.

5. IMG and the benchmark’s empirical role

Charades-AudioMatter was introduced alongside IMG, an Importance-aware Multi-Granularity fusion model designed for the proposition that audio is useful for VMR only when it is selectively integrated. IMG first forms text-guided visual and audio features, then predicts a sample-wise audio importance score

$A = \{a_j\}_{j=1}^{T}$5

This Audio Importance Predictor (AIP) is trained without ground-truth importance labels. Instead, it uses pseudo-labels derived from the relative retrieval losses of the audio-only and visual-only branches (Lin et al., 6 Aug 2025).

The model then performs Multi-Granularity Fusion (MGF) at local, event, and global levels. At the local level, the reported fusion rule is

$A = \{a_j\}_{j=1}^{T}$6

and the same importance-weighted pattern is reused across the event- and global-level branches. A Cross-modal Knowledge Distillation term further distills the audio-visual fusion branch into unimodal branches so that inference can still proceed when audio is missing.

On Charades-AudioMatter, this architecture functions as a controlled stress test of selective audio use. Using the same I3D visual backbone for all compared methods, the reported results are: SeqPAN at R1@3 79.30, R1@5 67.17, R1@7 48.96, mIoU 58.74; EAMAT at 78.30, 68.25, 48.88, 58.90; EMB at 77.81, 67.00, 47.96, 58.66; ADPN without audio at 77.89, 64.42, 44.64, 56.98; ADPN with audio at 78.65, 66.75, 49.71, 59.85; IMG without audio at 77.89, 65.92, 47.58, 58.35; and full IMG at 82.74, 71.93, 54.27, 62.76. The gain from IMG without audio to IMG with audio is therefore +6.69 on R1@7 and +4.41 on mIoU (Lin et al., 6 Aug 2025).

The benchmark also supports failure analysis of the importance mechanism itself. When the AIP is forced to ignore audio by setting $A = \{a_j\}_{j=1}^{T}$7, IMG drops from R1@3 82.74, R1@5 71.93, R1@7 54.27, mIoU 62.76 to 80.27, 70.22, 50.96, 59.84. The reported decrease—R1@7 down by 3.31 and mIoU down by 2.92—serves as direct evidence that the curated audio channel is not merely present but operationally useful on this subset.

In that sense, Charades-AudioMatter is not only an evaluation set but also an experimental instrument. It makes the effect of audio measurable under conditions that suppress the common confound in ordinary VMR benchmarks: many samples simply do not reward audio modeling.

6. Biases, limitations, and relation to later audio-aware retrieval

Charades-AudioMatter is intentionally biased. It over-represents soundful actions such as open, close, laugh, run, and pour, and under-represents silent or visually dominated actions such as sit, hold, look, and stand. It is also a test-only subset rather than a fully independent train/validation/test dataset. The paper explicitly notes that this makes it ideal for evaluating audio-aware methods but not representative of typical video distributions in which many queries are visual-only (Lin et al., 6 Aug 2025).

That design trade-off is also the benchmark’s principal significance. It separates two questions that are often conflated in ordinary Charades-STA evaluations: whether a model is good at moment retrieval in general, and whether it can exploit audio when audio is genuinely informative. Charades-AudioMatter addresses the second question directly.

Its appearance also fits a broader shift in Charades-family research from visual-text retrieval toward explicitly audio-visual grounding. SMART, for example, introduces a Shot-aware Multimodal Audio-enhanced Retrieval of Temporal Segments framework for moment retrieval on Charades-STA and QVHighlights, integrates BEATs-based audio features into an MLLM pipeline, and states that many prior methods ignore audio and therefore miss queries that inherently require audio cues such as speech or sirens. SMART does not define or use a dataset explicitly named “Charades-AudioMatter,” but it identifies Charades-STA and QVHighlights as audio-usable benchmarks and presents a direct conceptual foundation for audio-augmented Charades variants (Yu et al., 18 Nov 2025).

Within that trajectory, Charades-AudioMatter occupies a specific niche. It is not a replacement for Charades-STA, nor a general-purpose benchmark for all temporal grounding scenarios. Rather, it is a controlled evaluation subset for the audio-helpful regime of Charades-style moment retrieval: domestic videos, natural-language queries, inherited temporal annotations, and manually verified cases in which the soundtrack contributes materially to localization.