Joint Mixing Data Augmentation (JMDA)

• JMDA is a class of mix-based augmentation methods that jointly mix multiple samples, inputs, or modalities with a shared coefficient to maintain semantic consistency.
• In multimodal settings like audio-visual scene classification, JMDA synchronizes transformations across modalities, ensuring that cross-modal alignment is preserved.
• Empirical studies show that tuning the mixing coverage and parameters in JMDA can enhance model performance and robustness across diverse benchmarks.

Searching arXiv for the specified paper and closely related work on joint mixing data augmentation. arxiv_search(query="(Wang et al., 2022) OR \"A study on joint modeling and data augmentation of multi-modalities for audio-visual scene classification\" OR \"joint mixup\" multimodal audio visual", max_results=10) arxiv_search(query="\"MixGen: A New Multi-Modal Data Augmentation\" OR (Hao et al., 2022) OR \"Learning Multimodal Data Augmentation in Feature Space\" OR (Liu et al., 2022)", max_results=10) arxiv_search(query="multimodal data augmentation feature space mixup audio visual scene classification", max_results=10) Joint Mixing Data Augmentation (JMDA) denotes a family of augmentation methods that construct synthetic training examples by jointly mixing multiple samples and, when supervision is available, their targets. In the multimodal case, the defining requirement is coordinated transformation across modalities so that semantic coupling is preserved rather than broken. A canonical instance appears in audio-visual scene classification, where synchronized audio and video embeddings are mixed with a shared coefficient and the labels are mixed by the same proportion, yielding a deterministic feature-space joint mixup that preserves cross-modal alignment (Wang et al., 2022). In the broader mix-based augmentation literature, this jointness is commonly expressed as a transformation in a product space—input and label, multiple hidden layers, multiple modalities, or multiple samples at once—rather than as a single scalar interpolation of two images (Cao et al., 2022).

1. Conceptual definition and formal scope

JMDA is best understood as a class of mix-based augmentation strategies that jointly mix more than one object of interest: inputs and labels, multiple samples, multiple modalities, or multiple representational levels. A general two-sample formulation used across the literature is

$$\tilde{x} = \lambda x^{(1)} + (1-\lambda)x^{(2)}, \qquad \tilde{y} = \lambda y^{(1)} + (1-\lambda)y^{(2)},$$

with $\lambda \in (0,1)$, often sampled from a Beta distribution or fixed by design (Cao et al., 2022). Multi-sample generalizations replace the scalar $\lambda$ with simplex weights $\{\lambda_k\}$, yielding

$$\tilde{x} = \sum_{k=1}^{K} \lambda_k x_k, \qquad \tilde{y} = \sum_{k=1}^{K} \lambda_k y_k,$$

typically with $\boldsymbol{\lambda}$ drawn from a Dirichlet distribution (Cao et al., 2022).

In multimodal settings, the same principle is applied per modality. A generic two-modality formulation is

$$\tilde{x}_a = \lambda x_a^{(1)} + (1-\lambda)x_a^{(2)}, \qquad \tilde{x}_v = \lambda x_v^{(1)} + (1-\lambda)x_v^{(2)}, \qquad \tilde{y} = \lambda y^{(1)} + (1-\lambda)y^{(2)}.$$

The shared coefficient is not incidental: it is the mechanism that preserves cross-modal consistency. In this sense, JMDA is distinct from unimodal augmentation applied independently to each modality, because independent perturbations can destroy semantic alignment between paired signals (Wang et al., 2022).

The 2022 survey of mix-based data augmentation does not treat JMDA as a named method, but it repeatedly frames joint input-label mixing, multi-sample mixing, feature-space mixing, cross-domain mixing, and cross-modal mixing as central design patterns of MixDA (Cao et al., 2022). The 2021 survey of mixing augmentation methods similarly places such methods within a unified formalism in which a mixing mask $B$, a mixing ratio $\lambda$, and a label-mixing rule together define the synthetic sample (Lewy et al., 2021). Within that landscape, JMDA occupies the regime in which mixing is coordinated across several coupled variables rather than performed on a single image tensor in isolation.

2. Canonical multimodal instantiation: audio-visual joint mixup

A concrete and fully specified JMDA instance was introduced for audio-visual scene classification in "A study on joint modeling and data augmentation of multi-modalities for audio-visual scene classification" (Wang et al., 2022). The task is audio-visual scene classification on TAU Urban Audio Visual Scenes 2021: given a synchronized 10-second binaural audio clip and video, predict one of ten urban scene classes. For modeling, each clip is split into 1-second non-overlapping segments. Audio segments are converted to log-Mel filter bank features with derivatives, $\mathbf{I}_\text{A}^\text{LMFB}$ of shape $\lambda \in (0,1)$0, and encoded by an FCNN trained from scratch. Video segments use the first and fifteenth frames, resized to $\lambda \in (0,1)$1, processed by a DenseNet pretrained on Places365, and the two frame embeddings are summed to form a segment-level video representation (Wang et al., 2022).

Fusion is performed by early concatenation of the audio and video embeddings,

$\lambda \in (0,1)$2

followed by an MLP with layer sizes $\lambda \in (0,1)$3. Labels are one-hot vectors in $\lambda \in (0,1)$4, and the base training objective is cross-entropy,

$\lambda \in (0,1)$5

The system is trained end-to-end: both extractors and the fusion MLP are fine-tuned jointly, and no part of the network is frozen during AVSC training (Wang et al., 2022).

The JMDA component is an audio-video joint mixup applied at the embedding level. Given two synchronized multimodal samples $\lambda \in (0,1)$6 and $\lambda \in (0,1)$7, with audio embeddings $\lambda \in (0,1)$8, video embeddings $\lambda \in (0,1)$9, and one-hot labels $\lambda$0, the method constructs

$\lambda$1

$\lambda$2

$\lambda$3

with a fixed $\lambda$4. The mixed audio and video embeddings are concatenated and passed to the classifier, and the mixed label is used as a soft target in cross-entropy (Wang et al., 2022).

This design produced the best result when applied to 20% of the training data. On the development set, the baseline joint audio-video model without augmentation achieved 93.2% accuracy; audio augmentation alone raised this to 93.4%, video perturbation alone to 93.7%, and combined audio plus video augmentation to 93.9%. Adding joint mixup on 20% of the training data increased accuracy to 94.2%, while 50% and 100% coverage yielded 93.8% and 94.0%, respectively. The final 94.2% was reported as the best accuracy among all single AVSC systems submitted to DCASE 2021 Task 1b (Wang et al., 2022).

3. Alignment, label semantics, and the choice of mixing space

A central property of JMDA is that the same mixing coefficient is shared across modalities and labels. In the AVSC system, this was motivated explicitly by cross-modal consistency: if only audio or only video were mixed, the resulting example would become cross-modally inconsistent and could confuse the fusion network (Wang et al., 2022). The method therefore enforces

$\lambda$5

with the same $\lambda$6, so that the mixed audio and mixed video correspond to the same synthetic scene.

The AVSC formulation also clarifies a common misconception: JMDA is not tied to raw-input interpolation. In that system, joint mixup is applied after the modality-specific encoders, not on raw LMFB tensors or image pixels. The paper gives two reasons. First, embedding spaces are already aligned to scene semantics. Second, feature-space mixing avoids raw-data artifacts, such as unnatural visuals created by linearly combining unrelated images (Wang et al., 2022). This choice places the AVSC method closer to feature-space JMDA than to classical pixel-level Mixup.

Label handling is likewise task-dependent. The same paper distinguishes between two different operations. Audio mixing for unimodal training mixes two recordings from the same scene class and keeps the label unchanged. By contrast, the audio-video joint mixup uses soft labels and does not impose a same-class restriction on the paired examples. This preserves the classical mixup property that different categories can be interpolated, while still maintaining multimodal alignment through the shared coefficient (Wang et al., 2022).

The empirical results also show that stronger mixing is not automatically better. In AVSC, partial application at 20% outperformed 50% and 100% application. This suggests that a balance between original and synthetic samples can be preferable to replacing too much of the training distribution with mixed examples. The same pattern appears elsewhere: MixGen reports that $\lambda$7 mixed pairs per batch is optimal among the tested ratios, while $\lambda$8 is suboptimal (Hao et al., 2022). Such results do not imply a universal optimum, but they do indicate that JMDA has a coverage hyperparameter in addition to the usual mixing coefficient.

4. Representative instantiations across domains

Method	Domain	Joint mixing operation
Audio-video joint mixup (Wang et al., 2022)	Audio-visual scene classification	Mix audio embeddings, video embeddings, and labels with shared $\lambda$9
MixGen (Hao et al., 2022)	Vision-language pre-training	Linearly interpolate images and concatenate paired texts
LeMDA (Liu et al., 2022)	General multimodal learning	Learn a joint feature-space augmentation operator $\{\lambda_k\}$0 over latent tuples
RandoMix (Liu et al., 2022)	Image and speech classification	Randomly choose among linear and mask-based mixed modes
LSTC-MDA (Ding et al., 18 Sep 2025)	Skeleton-based action recognition	Combine SpatialMix, TemporalMix, and input-level additive mixup

MixGen is a direct input-level multimodal augmentation for vision-language representation learning. Given two image-text pairs $\{\lambda_k\}$1 and $\{\lambda_k\}$2, it constructs

$\{\lambda_k\}$3

with a fixed $\{\lambda_k\}$4 and $\{\lambda_k\}$5 newly generated mixed pairs per batch (Hao et al., 2022). It was evaluated on CLIP, ViLT, ALBEF, and TCL, and the paper reports absolute downstream improvements for ALBEF pre-training on image-text retrieval, visual grounding, visual reasoning, visual question answering, and visual entailment. It also reports that adding MixGen to ALBEF increases training time by only 0.4% (Hao et al., 2022). The distinguishing feature here is asymmetric modality handling: images are linearly mixed, while texts are concatenated rather than interpolated.

LeMDA, "Learning Multimodal Data Augmentation in Feature Space," generalizes the idea further by introducing a learned augmentation network $\{\lambda_k\}$6 acting on the tuple of modality-specific latent features $\{\lambda_k\}$7 (Liu et al., 2022). The task network is factorized at a fusion layer into $\{\lambda_k\}$8 and $\{\lambda_k\}$9, and $$\tilde{x} = \sum_{k=1}^{K} \lambda_k x_k, \qquad \tilde{y} = \sum_{k=1}^{K} \lambda_k y_k,$$0 maps

$$\tilde{x} = \sum_{k=1}^{K} \lambda_k x_k, \qquad \tilde{y} = \sum_{k=1}^{K} \lambda_k y_k,$$1

The method trains $$\tilde{x} = \sum_{k=1}^{K} \lambda_k x_k, \qquad \tilde{y} = \sum_{k=1}^{K} \lambda_k y_k,$$2 adversarially to increase task loss while regularizing it by a consistency term between original and augmented predictions and a VAE prior term. The consistency loss is typically a KL divergence between output distributions, and confidence masking is used so that consistency is enforced only when the model is confident on the original sample (Liu et al., 2022). The paper reports improvements across eight benchmarks and multiple modality combinations, including Hateful Memes accuracy from 0.6939 to 0.7562, Petfinder quadratic kappa from 0.2911 to 0.3537, and SNLI-VE from 0.7916 to 0.7981 (Liu et al., 2022).

RandoMix broadens JMDA in a different direction: instead of fixing one mixing operator, it randomizes over multiple mixed modes (Liu et al., 2022). For a minibatch, it first pairs samples via random permutation, then chooses a method from $$\tilde{x} = \sum_{k=1}^{K} \lambda_k x_k, \qquad \tilde{y} = \sum_{k=1}^{K} \lambda_k y_k,$$3 using user-defined weights, and finally applies the chosen linear or mask-based mixing rule. The paper frames this as a general mixed-sample augmentation that simultaneously addresses robustness and diversity challenges. On CIFAR-100 with WideResNet-28-10, it reports 84.84% top-1 accuracy, compared with 82.61% for Mixup, 82.34% for CutMix, 83.00% for ResizeMix, and 83.71% for PuzzleMix; on ImageNet with ResNet-50 it reports 77.88% top-1, slightly above ResizeMix and PuzzleMix, with cost approximately 1.01 relative to baseline (Liu et al., 2022).

A later domain-specific extension appears in LSTC-MDA for skeleton-based action recognition. There, JMDA is treated as a structured skeleton augmentation strategy comprising SpatialMix and TemporalMix, and the paper extends it with input-level additive mixup,

$$\tilde{x} = \sum_{k=1}^{K} \lambda_k x_k, \qquad \tilde{y} = \sum_{k=1}^{K} \lambda_k y_k,$$4

while restricting additive mixup to same-camera groups to avoid unrealistic view interpolation (Ding et al., 18 Sep 2025). Each of TemporalMix, SpatialMix, and AdditiveMix is applied independently with probability 50%. On NTU RGB+D 60 X-Sub ($$\tilde{x} = \sum_{k=1}^{K} \lambda_k x_k, \qquad \tilde{y} = \sum_{k=1}^{K} \lambda_k y_k,$$5), the paper reports 92.42% for SkateFormer-R, 93.19% with JMDA, 93.26% with JMDA plus add-mix, and 93.36% for the full LSTC-MDA pipeline (Ding et al., 18 Sep 2025).

5. Relation to MixDA theory and differentiable policy learning

Within the broader theory of mix-based data augmentation, JMDA is most naturally interpreted as a multi-axis extension of vicinal interpolation. The MixDA survey states that its generic definition already mixes features and labels jointly,

$$\tilde{x} = \sum_{k=1}^{K} \lambda_k x_k, \qquad \tilde{y} = \sum_{k=1}^{K} \lambda_k y_k,$$6

and it presents multi-sample, multi-layer, graph, and multimodal variants as natural generalizations of this scheme (Cao et al., 2022). On that view, JMDA is not a single algorithm but a family in which the mixing operator acts jointly on several coupled variables and the label transformation remains aligned with the data transformation.

The 2021 overview of mixing augmentation methods provides a complementary taxonomy. It distinguishes pixel-wise mixing, patch-wise mixing, feature-space mixing, and optimization-based multi-sample methods such as RICAP, Puzzle Mix, and Co-Mixup (Lewy et al., 2021). Co-Mixup is especially close in spirit to a general JMDA formulation because it optimizes mixing masks jointly over the whole mini-batch, using saliency, smoothness, and diversity objectives. This suggests that the “joint” in JMDA can refer not only to multimodality but also to batch-level coupling of several mixed samples.

A different line of work makes the mixing policy itself differentiable. "Joint Search of Data Augmentation Policies and Network Architectures" combines Faster AutoAugment and DARTS in a single bilevel optimization in which both augmentation policies and architecture choices are relaxed into differentiable soft mixtures (Kashima et al., 2020). Its augmentation module uses a softmax-weighted combination of primitive operations,

$$\tilde{x} = \sum_{k=1}^{K} \lambda_k x_k, \qquad \tilde{y} = \sum_{k=1}^{K} \lambda_k y_k,$$7

and the outer objective updates augmentation parameters and architecture parameters jointly on validation loss (Kashima et al., 2020). Although that paper does not name JMDA explicitly, it provides a direct template for learning not only which mix-based operators to use but also how strongly to mix and how to coordinate those choices with the architecture.

6. Empirical regularities, misconceptions, and open design questions

Several empirical regularities recur across the JMDA literature. First, preserving semantic or structural consistency is the central constraint. AVSC joint mixup mixes both modalities with the same coefficient specifically to preserve audio-video alignment (Wang et al., 2022). LeMDA moves augmentation into feature space because ordinary unimodal transformations can break multimodal semantics; its motivating SNLI-VE example is that cropping the smoke loop from an image can break an entailment relation with the paired text (Liu et al., 2022). LSTC-MDA restricts additive mixup to same-camera groups because mixing skeletons across views can create unrealistic poses and distribution shift (Ding et al., 18 Sep 2025).

Second, the most effective mixing space is task-dependent. Input-level interpolation works well for MixGen because image blending can be paired with full text concatenation, and the resulting pair can still be used by standard vision-language objectives (Hao et al., 2022). By contrast, AVSC joint mixup and LeMDA both favor feature-space augmentation, either by directly mixing embeddings or by learning a latent-space operator. This does not establish a universal preference, but it does show that JMDA is not reducible to raw-pixel mixup.

Third, different mixed modes encode different robustness biases. RandoMix explicitly attributes linear modes to adversarial and corruption robustness and mask-based modes to occlusion robustness, then combines them in a single randomized controller (Liu et al., 2022). On CIFAR-100, it reports that Mixup reduces FGSM and corruption error but worsens occlusion error, whereas mask-only methods improve occlusion robustness but degrade noise robustness; RandoMix uniquely reduces error across clean accuracy, FGSM, corruption, and occlusion simultaneously (Liu et al., 2022). A plausible implication is that JMDA is often less about finding one universally optimal operator than about coordinating several operators whose inductive biases are complementary.

Finally, the literature rules out several simplifications. JMDA is not synonymous with same-class mixing, because soft-label formulations are explicitly designed to support cross-class interpolation (Wang et al., 2022). It is not guaranteed to improve monotonically with more synthetic samples, because over-mixing can distort the effective training distribution; the AVSC and MixGen results both show nontrivial optima for mixup coverage (Wang et al., 2022, Hao et al., 2022). Nor is it restricted to a fixed, hand-designed transform family, since LeMDA and differentiable joint search methods replace analytic mixing rules with learned augmentation operators or differentiable mixtures of policies (Liu et al., 2022, Kashima et al., 2020).

The open design questions are therefore less about whether to mix and more about what should be mixed jointly, where the mixing should occur, and how semantic validity should be enforced. Existing work provides three broad answers. One answer is synchronization: share the coefficient across modalities or structured components, as in audio-video joint mixup and view-consistent skeleton mixup (Wang et al., 2022, Ding et al., 18 Sep 2025). A second answer is semantic restructuring: choose modality-specific mixing functions that preserve information, as MixGen does by interpolating images but concatenating texts (Hao et al., 2022). A third answer is learned augmentation: optimize a joint operator directly in latent space, regularized by prediction consistency, as in LeMDA (Liu et al., 2022). Together, these establish JMDA as a general strategy for constructing semantically constrained synthetic samples in settings where independent augmentation would violate the structure of the task.