---
title: Dual-Balance Collaborative Experts (DCE)
url: https://www.emergentmind.com/topics/dual-balance-collaborative-experts-dce
type: topic
---

# Dual-Balance Collaborative Experts (DCE)

Dual-Balance Collaborative Experts (DCE) is a framework for **imbalanced Domain-Incremental Learning (DIL)** that addresses two coupled difficulties: **intra-domain class imbalance** and **cross-domain class distribution shifts**. In the formulation introduced in "Addressing Imbalanced Domain-Incremental Learning through Dual-Balance Collaborative Experts" [2507.07100], DCE combines a **frequency-aware expert group** with a **dynamic expert selector** trained on **balanced Gaussian-sampled pseudo-features** derived from historical class statistics. The framework is explicitly designed for **PTM-based, exemplar-free DIL**, where the label space is fixed across domains, task boundaries are known during training, old raw data are unavailable after a domain is finished, and preservation of old knowledge must be reconciled with the possibility that later domains can improve old few-shot classes [2507.07100].

## 1. Problem setting and the meaning of “dual balance”

DCE is defined in a DIL regime with a sequence of domains
\[
\left\{\mathcal{D}^{1}, \mathcal{D}^{2}, \cdots, \mathcal{D}^{B}\right\},
\]
where each domain is
\[
\mathcal{D}^{b} = \left\{\mathcal{X}_b, \mathcal{Y}_b\right\}.
\]
The input-label pairs are written as \(\mathcal{X}_b = \{\mathbf{x}_i\}_{i=1}^{n_b}\) and \(\mathcal{Y}_b = \{y_i\}_{i=1}^{n_b}\), with a **fixed label space** across all domains:
\[
y_i \in \mathcal{Y}, \qquad \mathcal{Y}\ \text{shared across all tasks.}
\]
The paper decomposes the joint distribution as
\[
p_b(x,y)=p_b(y)p_b(x|y),
\]
and emphasizes that domain shift includes both **concept drift** through \(p_b(x|y)\) and **class distribution shift** through \(p_b(y)\) [2507.07100].

Two imbalance phenomena define the target problem. The first is **intra-domain class imbalance**, where sample counts \(n_b^c\) vary strongly across classes within a domain. The second is **cross-domain class distribution shift**, where the class-frequency profile changes across domains. The paper’s central claim is that these two effects create a sharper stability–plasticity tension than ordinary DIL: **old many-shot classes** mainly require protection from forgetting, whereas **old few-shot classes** may benefit from transfer from later domains [2507.07100].

The phrase **dual-balance** refers to two distinct balancing objectives. The first is **intra-domain frequency balance**, implemented through experts with different frequency biases. The second is **cross-domain knowledge balance**, implemented through a selector that adaptively fuses experts so that old many-shot knowledge can be preserved while old few-shot classes can improve. This makes DCE neither a standard MoE in the generic sparse-routing sense nor a standard replay-based continual learner. It is a framework that separates **specialization** from **fusion** in an exemplar-free DIL setting [2507.07100].

A common misconception is that DCE is simply another prompt-based continual learner. The paper instead positions it as a method in which the pre-trained Vision Transformer backbone is adapted early, then stabilized, while domain-specific expertise and cross-domain fusion are handled by separate modules. This suggests that DCE’s defining feature is not prompt tuning alone, but the joint use of **frequency-aware experts** and **statistic-based selector training** [2507.07100].

## 2. System architecture and domain-wise workflow

DCE uses a **pre-trained Vision Transformer encoder**
\[
\theta(x): \mathbb{R}^D \to \mathbb{R}^d
\]
with frozen backbone parameters \(\theta_0\), plus a small set of adaptable parameters \(\theta_1\) such as prompts. The generic PTM-based DIL objective is written as
\[
\min_{\theta_1, h} \sum_{(x, y) \in \mathcal{D}^b} \ell \left(h\left(\theta(x)\right), y\right) + \mathcal{L}_{\theta_1},
\]
where \(h\) is the classifier and \(\mathcal{L}_{\theta_1}\) regularizes the adaptable parameters [2507.07100].

The framework proceeds in **two stages per domain**. In **Stage 1**, DCE trains a **frequency-aware expert group** on the current domain. In **Stage 2**, it computes historical feature statistics, synthesizes balanced pseudo-features by Gaussian sampling, and trains a **dynamic expert selector** on these pseudo-features [2507.07100].

A crucial design choice is that Visual Prompt Tuning is used during the first task to learn prompts that adapt the pre-trained ViT into a DIL-aware representation space. After that, the encoder and prompts are frozen, and later tasks only add new experts and update the selector. The paper presents this as the condition that makes historical feature statistics meaningful across tasks. This suggests that DCE deliberately trades some representational plasticity for a stable feature space in which statistical replay remains coherent over time [2507.07100].

The per-domain procedure can be summarized directly from the algorithmic description:

| Stage | Operation | Data source |
|---|---|---|
| Stage 1 | Train new domain experts with \(\ell_{\text{CE}}+\ell_{\text{Bal}}+\ell_{\text{Rev}}\) | Real current-domain data |
| Stage 2 | Compute \(G_b\), update \(G\), synthesize \(\hat D\), train selector with \(\mathcal{L}_{\text{Select}}\) | Historical statistics and pseudo-features |

At inference time, the pipeline is likewise explicit: compute \(\theta(x)\), feed it to all experts, obtain selector weights, and fuse expert outputs by weighted summation. The selector is therefore **globally shared across all seen domains**, but routing is **instance-adaptive** rather than task-ID based [2507.07100].

## 3. Frequency-aware expert group

The expert group contains **three experts per domain**. For a new domain \(b\), the experts are denoted \(e_b\), \(e_{b+1}\), and \(e_{b+2}\), corresponding to a **many-shot-biased expert**, a **balanced expert**, and a **few-shot-biased expert**. All experts receive the same current-domain data, but each is optimized with a different loss encoding a different frequency prior [2507.07100].

Each expert consists of an **MLP plus a dedicated classifier**. The appendix specifies a **three-layer MLP** with widths
\[
D,\quad D/2,\quad |\mathcal{Y}|.
\]
The shared encoder provides the feature \(\theta(x)\), and the three experts produce logits
\[
v^1 = e_b(\theta(x)), \qquad v^2 = e_{b+1}(\theta(x)), \qquad v^3 = e_{b+2}(\theta(x)).
\]

The first expert uses standard cross-entropy:
\[
\ell_{\text{CE} = -\log \frac{\exp (v^1_{y})}{\sum_{j \in |\mathcal{Y}|} \exp (v^1_{j})}.
\]

The second expert uses **balanced softmax**:
\[
\ell_{\text{Bal} = -\log \frac{\exp(v^2_y + \log p^y_b)}{\sum_{j \in |\mathcal{Y}|} \exp(v^2_j + \log p^j_b)}.
\]

The third expert uses the **inverse distribution loss**. First define
\[
\hat{p}_b(y)=\{\hat{p}_b^i\}_{i=1}^{|\mathcal{Y}|}, \qquad \hat{p}_b^i=\frac{1/p_b^i}{\sum_j^{|\mathcal{Y}|}(1/p_b^j)}.
\]
Then
\[
\ell_{\text{Rev} = -\log \frac{\exp(v^3_y + \log p^y_b - \log \hat{p}^y_b)}{\sum_{j \in |\mathcal{Y}|} \exp(v^3_j + \log p^j_b -\log \hat{p}^j_b )},
\]
which the paper simplifies to
\[
\ell_{\text{Rev} = -\log \frac{\exp(v^3_y + 2 \log p^y_b)}{\sum_{j \in |\mathcal{Y}|} \exp(v^3_j + 2 \log p^j_b)}.
\]

The overall expert-training loss is
\[
\ell_{\text{exp} = \ell_{\text{CE}+\ell_{\text{Bal}+\ell_{\text{Rev}.
\]

The intended effect is a spectrum of frequency biases: the cross-entropy expert is naturally biased toward frequent classes; balanced softmax reduces bias relative to the empirical class prior; the reverse expert imposes a stronger complementary bias toward classes disadvantaged under standard empirical risk minimization. The paper reports that among single experts, \(\ell_{\text{CE}}\) helps many-shot most, \(\ell_{\text{Rev}}\) helps few-shot most, and \(\ell_{\text{Bal}}\) is the most balanced. This is the first of the two balances in DCE: the model does not try to make one classifier uniformly optimal across all frequency regimes, but instead creates specialized experts whose collaboration can later be learned [2507.07100].

For evaluation and dataset characterization, the paper uses explicit frequency thresholds. On Office-Home, classes are categorized using thresholds of **20** and **60** training samples; on DomainNet, the thresholds are **20** and **100**. These thresholds are used for many/medium/few-shot analysis rather than for routing samples to experts during training [2507.07100].

## 4. Dynamic expert selector and balanced Gaussian pseudo-replay

After experts accumulate over domains, DCE introduces a **dynamic expert selector** to combine them. At task \(b\), the selector is an MLP
\[
s(\cdot): \mathbb{R}^d \to \mathbb{R}^{3b},
\]
because there are \(3b\) experts after \(b\) tasks. Given an input feature, the selector outputs a weight vector
\[
\mathbf{w} = [w_i]_{i=1}^{3b},
\]
which measures the importance of each expert for that sample [2507.07100].

The selector is trained after expert training at every task. Its loss is
\[
\mathcal{L}_{\text{Select} = \frac{1}{|\hat{D}|} \sum_{(\tilde{x}, y) \in \hat{D} \ell_{\text{CE}\left( \sum_{i=1}^{3b} s(\tilde{x})_i \cdot e_i(\tilde{x}),\ y \right).
\]
Thus the effective logits are formed by the weighted mixture
\[
\sum_{i=1}^{3b} s(\tilde{x})_i \cdot e_i(\tilde{x}).
\]
The paper describes this as a probabilistic routing mechanism, although it does not explicitly write a softmax normalization over the selector outputs [2507.07100].

The selector cannot use old raw data because the framework is exemplar-free. Instead, DCE stores historical feature statistics
\[
G_b = \{(\mu_b^c, \Sigma_b^c)\},
\]
where \(\mu_b^c\) and \(\Sigma_b^c\) are the empirical mean and covariance for class \(c\) in domain \(b\), computed in the frozen feature space. The global repository is
\[
G = \bigcup_{i=1}^b G_i.
\]

Pseudo-features are synthesized by assuming a class-conditional Gaussian:
\[
(\tilde{x},c) \sim \mathcal{N}(\mu_i^c, \Sigma_i^c).
\]
The synthetic dataset is
\[
\hat{D} = \bigcup_{i=1}^b \bigcup_{c=1}^{|\mathcal{Y}|} \left\{ (\tilde{x},c) \sim \mathcal{N}(\mu_i^c, \Sigma_i^c) \right\}_{k=1}^K.
\]
The defining balancing rule is that the **same number \(K\)** of pseudo-features is generated for every domain-class pair. This is why the paper calls the procedure **balanced Gaussian sampling**: replay weight is equalized across domain-class cells regardless of the original sample counts [2507.07100].

This second balance is central to DCE’s interpretation of imbalanced DIL. The selector is not trained on a replay distribution that reproduces historical long-tail bias; it is trained on a deliberately balanced pseudo-distribution. The paper’s argument is that this allows the selector to preserve old many-shot classes without letting them dominate, while still giving old few-shot classes enough replay mass to learn when later experts are beneficial [2507.07100].

A further technical detail is covariance estimation. Because few-shot classes yield unstable covariance estimates, the paper uses **Oracle Approximating Shrinkage (OAS)**:
\[
\hat{\Sigma}_{c,d} = (1 - \rho)\hat{\Sigma}_{\text{emp} + \rho \hat{\Sigma}_{\text{prior},
\]
with
\[
\hat{\Sigma}_{\text{emp} = \frac{1}{n-1}\sum_{i=1}^n (x_i - \mu_c)(x_i - \mu_c)^\top,
\]
\[
\hat{\Sigma}_{\text{prior} = \mathrm{tr}(\hat{\Sigma}_{\text{emp})/d \cdot I_d,
\]
and
\[
\rho = \frac{\left(1 - \frac{2}{d} \right) \cdot \mathrm{tr}(\hat{\Sigma}_{\text{emp}^2) + \mathrm{tr}(\hat{\Sigma}_{\text{emp})^2}{(n + 1 - \frac{2}{d}) \cdot \left( \mathrm{tr}(\hat{\Sigma}_{\text{emp}^2) - \frac{1}{d} \mathrm{tr}(\hat{\Sigma}_{\text{emp})^2 \right)}.
\]
To reduce storage, the implementation averages class-specific covariances within each domain to obtain a **shared domain-level covariance matrix**, and OAS is only computed when a class has at least \(n \ge 10\) samples in a domain [2507.07100].

## 5. Optimization, inference behavior, and retention–transfer dynamics

The optimization schedule is sequential rather than joint. For each domain \(b\), DCE first optimizes the expert group on real current-domain data, then optimizes the selector on synthetic pseudo-features. If written compactly, the per-domain objective is effectively
\[
\mathcal{L}^{(b)} = \ell_{\text{exp} + \mathcal{L}_{\text{Select},
\]
but the paper is explicit that these are minimized in two stages rather than a single pass [2507.07100].

The first domain is special. When \(b=1\), the prompt parameters \(\theta_1\) and the first three experts are jointly trained. When \(b>1\), \(\theta_0\) and \(\theta_1\) are frozen, and only the new domain’s three experts are trained. After statistics are computed and pseudo-features generated, the selector is trained over the full accumulated expert pool [2507.07100].

This yields a specific preservation mechanism. DCE does **not** use real-data replay, **does not** use distillation, and **does not** rely on parameter-importance regularizers such as EWC or SI. Preservation instead comes from three structural choices: freezing the backbone and prompts after task 1, storing domain-specific knowledge in separate experts, and training a single selector on balanced pseudo-replay [2507.07100].

The paper interprets the resulting behavior through the distinction between **old many-shot** and **old few-shot** classes. Shared-space methods can improve old few-shot classes but tend to forget old many-shot classes more severely. Domain-specific methods retain old many-shot classes better but do not improve old few-shot classes using future domains. DCE is designed to occupy the middle ground by allowing instance-dependent collaboration across all experts while preserving stable specialized experts. This suggests that the selector’s role is not merely routing, but routing under an explicit retention–transfer trade-off [2507.07100].

The paper formalizes this analysis with **Class Performance Drift (CPD)**:
\[
\text{CPD}_b^c = a_B^c-a_b^c,
\]
where \(a_b^c\) is class \(c\)’s accuracy immediately after training domain \(b\), and \(a_B^c\) is the accuracy after the final domain. Positive CPD indicates degradation; negative CPD indicates improvement over time. On DomainNet, the paper reports that shared prompt methods show larger drift on many/medium-shot classes but smaller drift on few-shot classes, domain-specific prompt methods show smaller drift on many/medium-shot classes but worse few-shot improvement, and DCE “maintains moderate CPD across all class types” [2507.07100]. A plausible implication is that DCE’s dual-balance design is most valuable precisely when forgetting and positive backward transfer should not be treated uniformly across classes.

## 6. Empirical results, ablations, related context, and limitations

DCE is evaluated on **Office-Home**, **DomainNet**, **CORe50**, and **CDDB-Hard**, using **five randomized task/domain orders**. The main metrics are final accuracy \(\mathcal{A}_B\) and
\[
\bar{\mathcal{A}=\frac{1}{B}\sum_{b=1}^{B}\mathcal{A}_b,
\]
the average accuracy across incremental stages. For Office-Home and DomainNet, the paper also reports \(\mathcal{A}_{\text{many}}\), \(\mathcal{A}_{\text{med}}\), and \(\mathcal{A}_{\text{few}}\) [2507.07100].

The main benchmark results reported in the paper are summarized below.

| Dataset | \(\bar{\mathcal{A}}\) | \(\mathcal{A}_B\) | Additional breakdown |
|---|---:|---:|---|
| Office-Home | 84.6 | 84.4 | \(88.7/83.2/79.4\) for many/med/few |
| DomainNet | 64.3 | 63.5 | \(65.2/58.6/50.8\) for many/med/few |
| CORe50 | 80.1 | 84.8 | Not reported by frequency group |
| CDDB-Hard | 74.6 | 71.8 | Not reported by frequency group |

On Office-Home, DCE improves over the strongest competing \(\bar{\mathcal{A}}\) baseline by about **+1.2** and over the strongest \(\mathcal{A}_B\) baseline by about **+0.9**. On DomainNet, the gains are about **+5.3** on \(\bar{\mathcal{A}}\) and **+5.6** on \(\mathcal{A}_B\), which is the clearest numerical win reported in the paper. On CORe50 and CDDB-Hard, the reported margins are likewise substantial [2507.07100].

The ablation on the number of experts is particularly important. The paper compares one expert with \(\ell_{\text{CE}}\), two experts with \(\ell_{\text{CE}}+\ell_{\text{Bal}}\), three experts with \(\ell_{\text{CE}}+\ell_{\text{Bal}}+\ell_{\text{Rev}}\), and four experts with an additional more extreme loss
\[
\ell_4=-\log \frac{\exp(v^4_y + 3 \log p^y_b)}{\sum_{j \in |\mathcal{Y}|} \exp(v^4_j + 3 \log p^j_b)}.
\]
The trend reported is that increasing expert diversity improves performance, especially on medium-shot and few-shot classes, but gains beyond three experts are marginal. This is the direct empirical basis for the three-expert design [2507.07100].

DCE also has a clear computational profile. After the first task, the encoder is frozen, and only expert heads and the selector are trained. The appendix states that DCE uses **only one encoder forward pass** in both training and inference, unlike prompt-selection baselines that require two encoder forward passes. The implementation reports **SGD**, **batch size 128**, **cosine decay**, **20 or 30 epochs** for stage 1 depending on dataset, and **10 epochs** for stage 2; the main text says learning rate \(0.001\), while the appendix gives an initial learning rate \(0.01\), a discrepancy the paper itself leaves to be noted [2507.07100].

The framework is exemplar-free in raw-data storage, but it is not constant-memory. Parameter/storage growth is linear in the number of tasks because DCE adds three experts per task and enlarges the selector output dimension accordingly. The paper quantifies the per-task stored parameters as
\[
3 \times D \times \frac{D}{2} \times |\mathcal{Y}| + (D \times D + D \times |\mathcal{Y}|),
\]
with selector size
\[
D \times \frac{D}{2} \times 3b.
\]
A common misunderstanding is therefore that exemplar-free means negligible memory growth; DCE is exemplar-free in raw samples, but not in parameters or statistics [2507.07100].

In the broader collaborative-expert literature, DCE has clear conceptual relatives but a distinct problem target. "Use What You Have: Video Retrieval Using Representations From Collaborative Experts" [1907.13487] establishes collaborative experts as a way to combine multiple pretrained modality-specific streams for text–video retrieval, while "Enhancing Molecular Property Prediction via Mixture of Collaborative Experts" [2312.03292] explicitly addresses expert homogeneity and decision dominance through expert-specific projection and expert-specific loss. DCE differs in that its two balances are defined by **frequency imbalance within domains** and **retention–transfer trade-offs across domains**, rather than by modality sparsity or molecular-task collaboration. This suggests that DCE belongs to the same general expert-collaboration lineage, but its novelty lies in how collaboration is tied to long-tail continual learning rather than generic expert fusion [1907.13487] [2312.03292].

The paper’s limitations are also explicit or directly implied. DCE assumes that **task/domain boundaries are known during training**. It depends on a **stable frozen feature space** after the first task. It adopts a **Gaussian unimodality assumption** for class-conditional PTM features. Covariance estimation for few-shot classes is fragile enough to require OAS, and parameter count grows with the number of domains. These points indicate that DCE is specifically matched to **exemplar-free PTM-based DIL with strong class imbalance and cross-domain frequency shift**, rather than to task-free continual learning or unconstrained replay settings [2507.07100].

Source: https://www.emergentmind.com/topics/dual-balance-collaborative-experts-dce