DIMNet: Cross-Modal Representation Learning

• DIMNet is a cross-modal framework that maps voice and face to a shared semantic space through independent modality-specific embeddings.
• The method integrates covariate classifiers (e.g., identity, gender) to supervise embeddings without requiring paired training data.
• It enables cross-modal matching, verification, and retrieval with promising accuracy, though retrieval performance and modality gap issues remain.

The Disjoint Mapping Network (DIMNet) is a cross-modal representation learning framework that approaches biometric matching—specifically voice and face matching—by mapping each modality to a shared space of semantic covariates without directly modeling joint modality relationships. This framework introduces a structural separation in the learning of modality-specific embeddings, establishing a common prediction space via supervised covariate classification. DIMNet explicitly avoids learning cross-modal correspondence in the latent space and instead relies on disjointly-trained mappings that share supervision via auxiliary covariate classifiers, such as identity, gender, or nationality. After training, it enables cross-modal matching, verification, and retrieval through simple similarity measures in the learned embedding space (Wen et al., 2018).

1. Conceptual Framework

Let $\mathcal{V}$ denote the set of voice recordings and $\mathcal{F}$ the set of face images. Both are associated with a set of discrete covariates $\mathcal{C} = \{C_1, C_2, ..., C_m\}$ (e.g., identity, gender, nationality), where each sample $x$ has $C(x) \in \{1, ..., K_C\}$ for covariate $C$. DIMNet defines two modality-specific embedding functions: $$f_v: v \mapsto \mathbf{z}_v \in \mathbb{R}^d, \qquad f_f: f \mapsto \mathbf{z}_f \in \mathbb{R}^d,$$ with separate parameters for each modality. These produce fixed-dimensional vectors in a shared embedding space.

For each covariate $C$, a classifier $g_C: \mathbb{R}^d \rightarrow \Delta^{K_C}$ maps embeddings into a softmax-predicted covariate distribution. Training optimizes all $f_v$, $\mathcal{F}$0, and all $\mathcal{F}$1 to minimize a multi-task cross-entropy loss summed across modalities and covariates.

2. Network Architecture and Embedding Mechanism

The DIMNet architecture consists of three main modules:

• Voice Embedding Network ($\mathcal{F}$2):
  • Input: $\mathcal{F}$3 log-Mel spectrogram (where $\mathcal{F}$4 is the variable sequence length).
  • Architecture: Deep 1D convolutional stack with batch normalization and ReLU nonlinearities, progressively increasing channel dimensionality and temporal downsampling, followed by global temporal average pooling to yield the final embedding.
  • Embedding size: $\mathcal{F}$5.
• Face Embedding Network ($\mathcal{F}$6):
  • Input: $\mathcal{F}$7 RGB face crops (pre-processed for alignment).
  • Architecture: Sequential 2D convolutional blocks with increasing filter counts, global spatial average pooling to obtain a $\mathcal{F}$8 embedding.
• Covariate Classifiers ($\mathcal{F}$9):
  • Structure: For each covariate, a fully connected layer from $\mathcal{C} = \{C_1, C_2, ..., C_m\}$0 to $\mathcal{C} = \{C_1, C_2, ..., C_m\}$1 plus softmax.
  • Parameter sharing: While the embedding networks are disjoint for voice and face, the classifier bank $\mathcal{C} = \{C_1, C_2, ..., C_m\}$2 is shared across both modalities.

During mini-batch training, only the embedding network corresponding to the sample’s modality is updated, while the covariate classifier parameters receive gradients from both modalities.

3. Training Objective and Optimization

DIMNet employs a multi-task cross-entropy loss aggregated over modalities and covariates: $\mathcal{C} = \{C_1, C_2, ..., C_m\}$3 where

$\mathcal{C} = \{C_1, C_2, ..., C_m\}$4

and $\mathcal{C} = \{C_1, C_2, ..., C_m\}$5 are weighting factors (taken as 1.0 in all experiments). Training alternates batches from each modality with stochastic gradient descent, momentum, and learning rate decay.

No explicit cross-modal alignment loss is imposed; all cross-modal correspondence emerges through supervision by the shared covariate classifiers. Critically, DIMNet allows training on unpaired data, as long as covariate labels are available.

4. Inference and Cross-Modal Evaluation Protocols

After training, the modality-specific embedding networks $\mathcal{C} = \{C_1, C_2, ..., C_m\}$6 and $\mathcal{C} = \{C_1, C_2, ..., C_m\}$7 are retained, while all covariate classifiers $\mathcal{C} = \{C_1, C_2, ..., C_m\}$8 are discarded. These encoders embed voices and faces into the shared feature space: $\mathcal{C} = \{C_1, C_2, ..., C_m\}$9 For cross-modal tasks, the cosine similarity in $x$0 determines matching, verification, and retrieval: $x$1

Standard protocols include:

• 1:2 matching: Determining the correct association between one probe and two candidates.
• 1:N matching: Selecting the most likely cross-modal match from a gallery.
• Verification: Judged by thresholding cosine similarity, with EER computed where false accept and false reject rates intersect.
• Retrieval: Computing mean average precision (mAP) across ranked matches (Wen et al., 2018).

5. Empirical Results and Covariate Supervision Impact

Experiments evaluated DIMNet on cross-modal matching of voices and faces using VoxCeleb and VGGFace datasets, employing covariates such as identity, gender, and nationality. Key results include:

• Covariate classifier accuracy on test data reached ≈97.7%/99.4% for gender (voice/face), and ≈74.6%/60.5% for nationality.
• 1:2 matching accuracy (%): DIMNet with identity supervision (DIMNet-I) achieved V→F scores of 83.45 (U) and 70.91 (G), outperforming SVHF-Net [25]. Adding gender to identity yielded minor gains.
• Verification EER (%): DIMNet-I reported 24.95 (U), 34.95 (G); gender-only aligns with the theoretical lower bound for EER given perfect gender prediction.
• Retrieval mAP for identity: ≈4.42%, indicating that, while above random (≈0.08%), the cross-modal space is still diffuse and overlap between face and voice embeddings is incomplete (Wen et al., 2018).

Ablation studies indicated:

• Identity is the most informative covariate.
• Nationality supervision had limited effect, attributed to data imbalance.

6. Advantages, Limitations, and Open Challenges

DIMNet provides several architectural and procedural advantages:

• Enables cross-modal embedding learning without requiring paired sample training.
• Modular structure: easy extension to new covariates, including multi-task scenarios.
• Efficient in data usage compared to metric learning approaches.

Principal limitations are:

• Low retrieval mAP (~4–5%), implying that modality-specific embeddings remain imperfectly aligned.
• The structure does not mitigate modality gap independently; observed that face and voice embeddings form distinct clouds.
• Performance is highly sensitive to covariate distribution and quality.
• Potential improvements include advanced loss functions (e.g., center loss, angular softmax) and larger datasets (Wen et al., 2018).

7. Broader Impact and Extensions

The conceptual disjointness in DIMNet’s mapping networks has inspired related works in continual and incremental learning for multi-modal and multi-domain scenarios. For example, Disjoint Relevance Mapping Networks (DRMN) extend the principle of strict parameter isolation to semantic segmentation with large domain shifts, by associating each modality-specific mask with a non-overlapping subset of network parameters, thereby achieving near-zero catastrophic forgetting and efficient utilization of network capacity (Hegde et al., 2024). This suggests a general applicability of disjoint mapping principles to a wide spectrum of cross-domain and continual learning problems.

An outstanding research direction remains in narrowing the intrinsic modality gap and improving retrieval performance, especially in low-resource and imbalanced covariate regimes. A plausible implication is that hybrid methods combining covariate supervision with explicit cross-modal matching objectives may provide further performance gains.