--- title: Joint Expert Embedding Training (JEET) url: https://www.emergentmind.com/topics/joint-expert-embedding-training-jeet type: topic --- # Joint Expert Embedding Training (JEET) Joint Expert Embedding Training (JEET) is a modular deep learning paradigm designed to maximize parameter efficiency and avoid catastrophic forgetting across multi-domain or multi-task settings, while retaining expert-level specialization for each subtask. In JEET, a single shared embedding or backbone encoder is paired with multiple downstream expert modules which are trained jointly, allowing high-capacity feature extraction to be amortized across specialized decoders (experts) tuned for specific languages, domains, or tasks. The JEET framework has been explored across natural language processing and speech diarization, enabling modular yet unified approaches for complex pipelines [2411.02165][2407.19610]. ## 1. Conceptual Foundations and Motivation JEET emerges from the intersection of modular Mixture-of-Experts (MoE), knowledge distillation, and multi-task learning. Standard MoE architectures assign inputs to expert subnetworks via a gating or routing function, yielding parameter isolation and specialization. Conventional training of multitask pipelines often involves training separate embeddings and decoders for each task, which leads to duplicated parameters and increased risk of catastrophic forgetting when sequentially trained. JEET addresses these by introducing a joint optimization regime in which a common embedding layer is shared among multiple expert decoders, each responsible for a distinct domain or subtask. This configuration enables expert-level performance, improved memory and compute efficiency, and robust knowledge retention across tasks [2407.19610]. In the language modeling context, JEET allows every expert module (e.g., for English, French, German, or Python) to share the same high-dimensional token embeddings, thus reducing redundancy without sacrificing specialization. In speaker diarization, analogous parameter efficiency is achieved by sharing a ResNet-based encoder for extracting frame-level features utilized in parallel by heads for speaker embedding, voice activity detection (VAD), and overlap speech detection (OSD) [2411.02165]. ## 2. Mathematical Formulation and Architectural Elements At the core of JEET is a shared embedding layer or backbone encoder, with expert-specific decoders downstream: - **Language Models:** For vocabulary size $V$ and embedding dimension $d$, the shared embedding matrix $E \in \mathbb{R}^{V \times d}$ transforms token sequences to embedding sequences. For sequence $x = [x_1, \ldots, x_T]$, output is $H = [h_1, ..., h_T]$ with $h_t = E[x_t]$. Predicted logits for expert $j$ are $\ell^{(j)} = \mathrm{TransformerDecoder}_{\theta_j}(H)$ [2407.19610]. - **Router/Gating Mechanism:** A sequence-level classifier (router) determines expert selection. Given pooled embedding $z = \mathrm{Pool}(H) \in \mathbb{R}^d$, scores are $s = U z + b \in \mathbb{R}^M$ for $M$ experts; softmax yields routing probabilities $g_j(x)$. At inference, $j^\star = \arg\max_j g_j(x)$, and only $\mathrm{Expert}_{j^\star}$ operates on $x$. - **Speaker Diarization:** A ResNet-101 encoder computes 8192-dimensional features from audio frames (64-d Mel filterbanks, every 10 ms); these are mapped through a linear layer to 256-dimensional per-frame embeddings. Three heads—speaker embedding, VAD, and OSD—access these embeddings in parallel [2411.02165]. ### Joint Training Objective - **Language Models:** Each expert is distilled from a common teacher (e.g., GPT-2 Medium) using two losses: - Language modeling loss: $L_{\mathrm{LM}} = -\sum_{t=1}^T \log p_S^{(j^\star)}(y_t | x_{