DICCAE: Dynamic Confusion-Aware AV Encoder

• The paper introduces a novel audio-visual fusion framework that aligns representations at a fine-grained, category level while dynamically weighting confusion loss.
• It employs a two-layer encoder architecture with separate modality processing, InfoNCE loss, and cluster-guided self-supervised pre-training to improve feature discrimination.
• Empirical results on VGGSound demonstrate near state-of-the-art performance, validating the approach's effectiveness in handling inter-class confusion and improving class separation.

Searching arXiv for the cited DICCAE and precursor TCIL papers to ground the article in current arXiv records. Dynamic Inter-Class Confusion-Aware Encoder (DICCAE) is an audio-visual fusion encoder for human activity recognition that aligns audio-video representations at a fine-grained, category-level and dynamically adjusts a confusion loss according to inter-class confusion degrees rather than relying only on global modality alignment (Cong et al., 12 Jul 2025). It is motivated by the claim that existing audio-video pre-training paradigms focus on alignment of the overall audio-video modalities without reinforcing the distinction of easily confused classes through cognitive induction and contrast during training. In the reported framework, DICCAE is coupled with a two-layer audio-video fusion training procedure, an update mechanism based on an inter-class confusion matrix, and a cluster-guided audio-video self-supervised pre-training strategy designed to mitigate the scarcity of labeled audio-video data (Cong et al., 12 Jul 2025).

1. Conceptual basis and problem formulation

DICCAE is defined around two departures from standard audio-video pre-training. First, it performs fine-grained category-level alignment rather than only modality-level alignment. Second, it uses dynamic confusion-aware training rather than fixed losses. The stated target is category confusion in human activity recognition, especially for activity pairs that are easily confused in one modality alone, such as fall/lie down and cough/sneeze. The paper frames this in cognitive terms: humans do not understand individual events in isolation, but generalize concepts within classes and compare them to others (Cong et al., 12 Jul 2025).

The central claim is that global cross-modal alignment is too coarse-grained for this regime. Standard objectives can bring corresponding audio and video representations closer in latent space, but they do not explicitly model which categories overlap or require additional discriminative pressure. DICCAE therefore emphasizes inter-class confusion as a training signal in its own right. This suggests a shift from generic cross-modal correspondence learning toward explicit modeling of difficult category boundaries (Cong et al., 12 Jul 2025).

A common misconception is that DICCAE is only an audio-video alignment method with an additional penalty term. The paper instead presents it as a category-aware and confusion-aware framework in which the weighting of the confusion loss is updated dynamically from measured inter-class confusion. The adaptive weighting is treated as structurally important rather than incidental (Cong et al., 12 Jul 2025).

2. Encoder architecture and two-layer fusion framework

DICCAE is presented as a two-layer audio-video fusion training framework. In the first layer, audio and video are processed by separate encoders, producing low-level representations for each modality. These features are used to compute InfoNCE loss, enabling contrastive learning within each modality. In the second layer, audio and video features are combined for each class pair, passed through an MLP, and used to compute both the confusion loss $\mathcal{L}_{CL}$ and the classification loss (Cong et al., 12 Jul 2025).

This division of labor is central to the design. The first layer enforces intra-modal contrastive structure, while the second layer acts on fused representations and category relations. The paper distinguishes this from ordinary AV contrastive learning by stating that the model does not only learn whether audio matches video; it also learns whether two fused category representations should be close or far, with that pressure modulated by confusion-aware weights (Cong et al., 12 Jul 2025).

The end-to-end framework also includes an update set derived from part of the test set, computation of the inter-class confusion matrix after each epoch, and dynamic adjustment of the confusion loss weights for the next training stage. The resulting feedback loop is explicitly described as: train the model, infer on an update set, compute inter-class confusion, normalize confusion values, reweight the confusion loss accordingly, and continue training (Cong et al., 12 Jul 2025).

3. Inter-class confusion modeling and dynamic loss adjustment

The core objective starts from a pairwise confusion loss over category pairs $(x_i, x_j)$ with extracted features $(f_i, f_j)$:

$$\mathcal{L}_{CL} = -y \log F_p(f_i, f_j) - (1 - y) \log(1 - F_p(f_i, f_j)) \tag{1}$$

where $y = 1$ if $i$ and $j$ are the same category and $y = 0$ otherwise. In the paper’s interpretation, same-class pairs are pulled together and different-class pairs are pushed apart; this is described as simulating human “contrast and induction” (Cong et al., 12 Jul 2025).

To estimate which class pairs are most confusing, the method computes an inter-class confusion degree $\mathcal{M}_{ij}$. For each class, it first computes the centroid of all hidden-space features in that class. It then fits an enclosing circle that covers 95% of the feature points in the class, using PCA followed by circle fitting to avoid outlier effects. For a class pair $i,j$, the confusion degree is determined from the overlap implied by the two radii $(x_i, x_j)$0, $(x_i, x_j)$1 and the center distance $(x_i, x_j)$2, with larger values indicating greater overlap and hence greater confusion (Cong et al., 12 Jul 2025).

After each epoch, the confusion scores are normalized to $(x_i, x_j)$3, and the weighted DICCAE loss is formed by summing the pairwise confusion loss terms with these normalized confusion weights:

$(x_i, x_j)$4

The stated effect is that pairs with higher confusion receive larger loss weight, so the model focuses more on difficult class boundaries. The paper explicitly notes that this dynamic weighting stabilizes training and performs better than fixed weights, which can overemphasize contrast and hurt accuracy (Cong et al., 12 Jul 2025).

This dynamic schedule is not merely an optimization detail. The update rule is itself part of the method: at the end of each epoch, confusion degrees are computed on an update set, normalized, and then used as weights in the next training phase. A plausible implication is that DICCAE treats confusion as a nonstationary property of the representation space rather than as a static prior (Cong et al., 12 Jul 2025).

4. Cluster-guided self-supervised pre-training

Because labeled audio-video data are scarce in human activity recognition, the paper adds a cluster-guided audio-video self-supervised pre-training strategy. Multiple AV HAR datasets are merged into a large unlabeled pre-training set, fusion features are extracted with a pre-trained encoder, K-means clustering is run on these features, and the resulting cluster assignments are used as pseudo-labels (Cong et al., 12 Jul 2025).

The pre-training pool combines Audioset, VGGSound, EPIC-KITCHENS-100, and UCF101 into a dataset of approximately 2.3 million samples. The practical clustering setup uses $(x_i, x_j)$5 by default, tries 20 random K-means initializations and selects the best result, and refines cluster labels every 10 epochs using the current DICCAE features. In the reported configuration, the optimizer is Adam, the learning rate is $(x_i, x_j)$6, batch size is 32, the self-supervised phase runs up to 500 iterations, the supervised phase runs up to 30 steps, and learning-rate decay is used for stable convergence (Cong et al., 12 Jul 2025).

The stated rationale is that DICCAE naturally benefits from clustering because it is designed to align features at the category level; clusters can therefore function as provisional categories during pre-training. This suggests that pseudo-label quality and representation quality are coupled through the refinement loop, which is why the paper emphasizes iterative cluster refresh rather than one-shot clustering (Cong et al., 12 Jul 2025).

5. Empirical performance and ablation evidence

The main evaluation benchmark is VGGSound, with top-1 accuracy as the primary metric. The reported results are 60.1% audio-only top-1 accuracy and 65.5% audio-visual top-1 accuracy. The paper states that DICCAE achieves near state-of-the-art performance on VGGSound and matches the best reported AV result in the comparison table (Cong et al., 12 Jul 2025).

Model	Modality	Top-1 accuracy
GBlend	Audio	43.2%
Perceiver	AV	58.9%
Attn AV	AV	60.5%
MBT	AV	61.3%
CAV-MAE	AV	65.5%
AudiovisualMAE	AV	65.0%
DICCAE	AV	65.5%

The ablation study is used to validate the necessity of each module. Removing $(x_i, x_j)$7, the confusion-aware weighting term $(x_i, x_j)$8, the InfoNCE component, the refinement stage, or K-means all reduces performance, with the largest drop occurring when $(x_i, x_j)$9 is removed. This is presented as evidence that the contrastive training framework, confusion modeling, and cluster-guided refinement all contribute materially to the final result (Cong et al., 12 Jul 2025).

Variant	Audio	Audio-Visual
Baseline DICCAE	60.1	65.5
w/o $(f_i, f_j)$0	58.2	63.1
w/o $(f_i, f_j)$1	59.6	64.1
w/o $(f_i, f_j)$2	56.7	61.6
w/o Refinement	58.4	63.3
w/o K-means	57.7	62.2

The paper also reports qualitative evidence on feature-space separation: DICCAE reduces the mean of the confusion-degree distribution by 0.75 and reduces the variance by 4.89. In the paper’s interpretation, this indicates better class separation and less overlap in feature space (Cong et al., 12 Jul 2025).

6. Relation to earlier confusion-aware encoders, misconceptions, and limitations

A relevant antecedent appears in class-incremental learning rather than audio-visual fusion. "Resolving Task Confusion in Dynamic Expansion Architectures for Class Incremental Learning" identifies inter-task confusion (ITC) and old-new confusion (ONC) as major failure modes of dynamic expansion architectures and proposes Task Correlated Incremental Learning (TCIL), which combines multi-level knowledge distillation, attention-based feature fusion, and classifier re-scoring (Huang et al., 2022). Although TCIL is not the same method as DICCAE, it addresses dynamic inter-class/task confusion by making the encoder and decision process explicitly aware of confusion rather than treating forgetting or alignment as the only problem. This suggests that DICCAE belongs to a broader family of confusion-aware encoder designs in which difficult cross-class relations are modeled directly (Huang et al., 2022).

Within that broader context, one misconception is to treat DICCAE as equivalent to global multimodal alignment with a dynamic scalar schedule. The paper’s distinction is sharper: the method is category-aware rather than only modality-aware, confusion-aware rather than only similarity-aware, and dynamically weighted according to how confused classes are during training (Cong et al., 12 Jul 2025). Another misconception is to view the self-supervised stage as generic pseudo-label pre-training. In the reported framework, K-means pseudo-labeling is specifically coupled to category-level alignment and iterative refinement, not used as an isolated preprocessing step (Cong et al., 12 Jul 2025).

The limitations stated or implied in the paper are methodologically significant. The confusion matrix depends on an update set drawn from the test set, which is described as unusual and may raise practical or methodological concerns. The inter-class confusion calculation relies on geometric approximations such as centroids and enclosing circles, which may not capture all forms of multimodal class overlap. The evaluation is presented primarily on VGGSound, so broader generalization is suggested rather than extensively demonstrated in the excerpt. The paper also does not provide a fully explicit formula for the overall combined loss beyond the DICCAE weighting term and the descriptions of InfoNCE and classification loss (Cong et al., 12 Jul 2025).

Taken together, the available evidence frames DICCAE as a confusion-aware audio-video representation learning framework whose distinctive contribution is not merely fusing modalities, but identifying which class boundaries are difficult and increasing optimization pressure on those boundaries. In that sense, its significance lies as much in the formulation of inter-class confusion as in the encoder itself (Cong et al., 12 Jul 2025).