--- title: Dual-view Personalized MoE Architecture url: https://www.emergentmind.com/topics/dual-view-personalized-mixture-of-experts-dpmoe type: topic --- # Dual-view Personalized MoE Architecture Searching arXiv for the cited DPMoE-related papers to ground the article in current records. Searching arXiv for "Dual-view Personalized Mixture-of-Experts" and closely related titles. Dual-view Personalized Mixture-of-Experts (DPMoE) denotes an architectural pattern in which two complementary views are processed through expert mechanisms and then fused by personalized routing. In the narrowest sense, the term appears as the core adaptation module inside L2Rec, where a frozen Transformer backbone receives view-specific, user-conditioned low-rank perturbations for a **semantic** view and a **behavioral** view, followed by adaptive cross-view fusion [2605.26717]. In a broader and more interpretive sense, several adjacent lines of work instantiate DPMoE-like structure without using the exact name: some define the two views as **global/generalized** versus **local/personalized** representations in federated learning, while others organize dual expert systems across semantic and personalized computation stages, or across feature learning and downstream prediction [2402.01350], [2505.24597], [2510.26014]. The common thread is that “dual-view” usually refers not to two modalities alone, but to two distinct sources of information, two computational perspectives, or two specialization roles, combined by a routing mechanism that is adaptive at the user, client, sample, or token level. ## 1. Conceptual definition and scope In L2Rec, DPMoE is defined as a parameter-efficient adaptation mechanism for sequential personalized recommendation that unifies **behavioral signals** and **semantic signals** at the parameter level of a frozen LLM backbone [2605.26717]. The semantic view is constructed from concatenated item descriptions, \[ \mathbf{E}_u^{S} = \text{Embed}([\mathbf{c}_1;\mathbf{c}_2; \ldots;\mathbf{c}_L]) \in \mathbb{R}^{T\times d}, \] whereas the behavioral view compresses each interacted item into a vector sequence, \[ \mathbf{E}_{u}^B = P_{U}\big([\text{pool}(\text{Embed}(\mathbf{c}_i))]_{i=1}^{L}\big) \in \mathbb{R}^{L \times d}. \] DPMoE then produces view-specific perturbations of the same frozen Transformer parameters, yielding \[ \widetilde{\mathbf{W}^S = \mathbf{W} + \Delta \mathbf{W}^S,\qquad \widetilde{\mathbf{W}^B = \mathbf{W} + \Delta \mathbf{W}^B. \] This makes DPMoE a dual-view model in a literal sense: two views are preserved throughout the backbone as two personalized adaptation pathways rather than being forced into a single shared representation at the input level. The broader literature uses the same idea under adjacent names. In pFedMoE, the two views are a **global/generalized representation view** from a shared homogeneous small feature extractor and a **local/personalized representation view** from a client-private heterogeneous backbone; the paper explicitly states that this is “very close to a ‘Dual-view Personalized MoE (DPMoE)’ interpretation” if “dual-view” is understood as “two representation sources / two expertise streams” rather than two modalities [2402.01350]. In NextLocMoE, the architecture is described as a **dual-level MoE**, with one expert system modeling a **location-semantic / functional view** and another modeling a **user-behavior / personalization view** [2505.24597]. In dual-MoE survival analysis, the two perspectives are a **subgroup-aware patient representation view** and a **temporal hazard-evolution view** [2510.26014]. This suggests that DPMoE has become a functional design principle: one branch models common or semantic structure, the other models personalized or dynamically conditioned structure, and the final predictor is obtained by learned fusion rather than fixed interpolation. A persistent misconception is that “dual-view” must imply multimodality. The available literature does not support such a restriction. Some of the clearest DPMoE-like systems are not multimodal at all: they instead derive the two views from federated provenance, temporal conditioning, or distinct computational levels [2402.01350], [2510.26014]. Conversely, multimodal systems may still fail to be DPMoE in a strict sense if they do not include personalized or adaptive expert routing. ## 2. Core architectural patterns The most explicit DPMoE formulation appears in L2Rec. For each adapted Transformer parameter matrix \(\mathbf{W}_j\), the view-specific perturbation is \[ \Delta\mathbf{W}_j = \sum_{i \in \mathcal{E}_{sh} \mathbf{B}_{j,i} \mathbf{A}_{j,i} + \sum_{i \in \mathcal{E}_{v} g_{j,i}\, \mathbf{B}_{j,i} \mathbf{A}_{j,i}, \qquad v\in\{s,b\}, \] where \(\mathcal{E}_{sh}\) are shared experts and \(\mathcal{E}_v\) are view-specific experts [2605.26717]. Shared experts are always active, while semantic-specific and behavioral-specific experts are sparsely selected. The semantic branch and behavioral branch therefore traverse the same frozen backbone, but under different low-rank expert mixtures. The model uses **9 experts total**—**1 shared expert** and **8 view-specific experts**—with **Top-2 experts selected per view**, **LoRA rank \(r=8\)**, scaling \(\alpha=16\), and about **32M trainable parameters**, or about **5% of the backbone** [2605.26717]. The view-specific outputs are then merged by Adaptive Cross-view Fusion (ACF). After residual projections, \[ \tilde{\mathbf{h}^v_u = \mathbf{h}^v_u + P_v(\mathbf{h}^v_u), \qquad v\in\{B,S\}, \] a dynamic gate is computed as \[ \delta = \sigma\!\left(W_g[\tilde{\mathbf{h}^B_u;\tilde{\mathbf{h}^S_u] + b_g\right), \] and the fused user representation is \[ \mathbf{h}^{F}_u = \delta\, \tilde{\mathbf{h}^B_u + (1-\delta)\, \tilde{\mathbf{h}^S_u. \] The same dual-view encoding and fusion are used on the item side. Architecturally, DPMoE is therefore not merely a routing module; it is a two-path adaptation-and-fusion system spanning parameter perturbation, representation formation, and final preference scoring [2605.26717]. Other papers instantiate the same pattern with different expert granularity. PFL-MoE is a two-expert predictive fusion framework in which a **global/shared model** and a **personalized/local model** are mixed by a client-specific gate: \[ \tilde{y} = g \cdot M_G(\theta;x) + (1-g)\cdot M_G(\theta_i;x), \] with \[ g = \operatorname{sigmoid}(G(w_i;x)). \] In the freeze-base instantiation PFL-MF, the two experts are classifier heads sharing one extractor, and the improved PFL-MFE uses the shared feature \(a\) rather than raw input as gate input: \[ g = \operatorname{sigmoid}(G(w_i;a)). \] This is a minimal DPMoE-like design in which the two views are “global generalization” and “local specialization” [2012.15589]. pFedMoE moves the fusion point from output space to representation space. For each client \(k\), a shared homogeneous small feature extractor \(\mathcal{G}(\theta)\) produces a generalized representation, a local heterogeneous extractor \(\mathcal{F}_k^{ex}(\omega_k^{ex})\) produces a personalized representation, and a local gate \(\mathcal{H}(\varphi_k)\) yields instance-specific weights \[ [\alpha_{k,i}^{\mathcal{G},t}, \alpha_{k,i}^{\mathcal{F}_k,t}] = \mathcal{H}(\boldsymbol{x}_i;\varphi_k^{t-1}), \quad \alpha_{k,i}^{\mathcal{G},t}+\alpha_{k,i}^{\mathcal{F}_k,t}=1. \] The fused representation is \[ \boldsymbol{\mathcal{R}_{k,i}^t} = \alpha_{k,i}^{\mathcal{G},t}\cdot \boldsymbol{\mathcal{R}_{k,i}^{\mathcal{G},t} + \alpha_{k,i}^{\mathcal{F}_k,t}\cdot \boldsymbol{\mathcal{R}_{k,i}^{\mathcal{F}_k,t}, \] which is then passed to the local head. This is explicitly a dual-expert MoE over two representation sources [2402.01350]. ## 3. Personalization and routing mechanisms The defining property of DPMoE is not just duality, but personalized routing. In L2Rec, the router uses three signals for the view-specific experts: a **context signal** \[ \mathbf{z}_j^c = \mathbf{x}_j, \] a **user signal** \[ \mathbf{z}_j^u = \mathbf{u}, \] and an **interaction signal** \[ \mathbf{z}_j^f = [\mathbf{u};\mathbf{x}_j]. \] Each is mapped to expert scores by a two-layer MLP, \[ \mathbf{g}_{j}^{(m)} = \mathrm{Softmax}\big(\mathbf{R}_{2,j}^{(m)} \cdot \mathrm{ReLU}(\mathbf{R}_{1,j}^{(m)} \mathbf{z}_{j}^{(m)} + \mathbf{b}_{1,j}^{(m)}) + \mathbf{b}_{2,j}^{(m)}\big), \qquad m \in \{c,u,f\}, \] and the final routing weights are formed by \[ \hat{\mathbf{g}_{j} = \mathbf{g}_j^{c} + (\mathbf{g}_j^{u} + \mathbf{g}_j^{f}) \odot \mathbf{g}_j^{c}, \] followed by Top-\(N\) sparsification [2605.26717]. The stated design intuition is that context remains primary while user-specific information acts as a personalized correction. The most common alternative is sample-wise gating between a generalized branch and a personalized branch. pFedMoE exemplifies this at the data level: the local gating network produces personalized weights for both experts on **each data sample**, and weight analyses show that distributions vary across clients, across classes, and across samples [2402.01350]. This is why the paper characterizes the method as **data-level personalization** rather than merely client-level personalization. NextLocMoE uses a more heterogeneous routing design. Its **Location Semantics MoE** routes current records to function experts using the current embedding and long-term history, \[ \mathbf{r}^{\text{func} = \mathrm{MLP}([\mathbf{e}_c^{(0)}; \mathbf{h}^{\text{hist}]) \in \mathbb{R}^{K_f}, \] then applies fixed top-\(k\) routing to obtain a function-aware spatial representation. Its **Personalized MoE**, inserted into upper Transformer layers, forms per-expert inputs \[ \mathbf{z}_i^{\text{user} = [\mathbf{x};\mathbf{h}^{\text{hist};\mathbf{e}^{\text{user}_i}], \] computes user-expert probabilities \[ p_i^{\text{user} = \mathrm{softmax}(\mathbf{r}^{\text{user})_i, \] and selects experts until cumulative probability exceeds \(\tau\), with \(\tau=0.8\) [2505.24597]. Personalization here is trajectory-conditioned and prior-guided rather than user-ID-based. A stronger user-centric routing regime appears in sparse MoE reward modeling. The reward model uses expert heads \(\{r_{\theta_k}\}_{k=1}^K\) and a router \(\pi_\phi(x)\in \Delta^{K-1}\), with final reward \[ r(x,y)=\sum_{k=1}^K \pi_{\phi,k}(x)\, r_{\theta_k}(x,y). \] Personalization occurs at test time by freezing experts and adapting only the router using a small adaptation set, so that individual preference profiles are expressed as different mixtures over shared latent preference components [2606.04284]. This is not a dual-view design, but it establishes router-only personalization as a reusable DPMoE principle. ## 4. Learning objectives and regularization DPMoE systems differ substantially in where they impose coupling and regularization. In L2Rec, the primary recommendation objective is a contrastive loss on fused representations, \[ \mathcal{L}_{\text{Rec} = -\log \frac{e^{r_{u,i^{+} {e^{r_{u,i^{+} + \sum_{i^{-}\in\mathcal{N} e^{r_{u,i^{-}, \qquad r_{u,i} = \text{sim}(\mathbf{h}^F_u, \mathbf{h}^F_i)/\tau, \] supplemented by a **Bidirectional Preference Contrastive** loss, \[ \mathcal{L}_{\text{BPC} = -\log\frac{e^{r_{u}^{B \to S}{\sum_{u'} e^{r_{u,u'}^{B \to S} -\log\frac{e^{r_{u}^{S \to B}{\sum_{u'} e^{r_{u,u'}^{S \to B} + \lambda \|\tilde{\mathbf{h}^B_u - \tilde{\mathbf{h}^S_u\|_2^2, \] and a load-balancing loss \(\mathcal{L}_{\text{LB}}\), yielding \[ \mathcal{L} = \mathcal{L}_{\text{Rec} + \gamma\, \mathcal{L}_{\text{BPC} + \beta\, \mathcal{L}_{\text{LB}. \] This explicitly addresses cross-view drift even though both views already inhabit the same frozen parameter manifold [2605.26717]. Other dual-expert systems use much simpler task losses. pFedMoE uses only cross-entropy on the fused representation and states that there is **no explicit auxiliary regularization term**; regularization is implicit in normalization layers, federated averaging of the shared extractor, and architecture choices [2402.01350]. PFL-MoE similarly trains the gate by minimizing \[ p_i(w_i)=\mathbb{E}_{(x,y)\sim \mathcal{D}_i}[L_i(\tilde{y},y)], \] after first adapting the personalized branch on local data [2012.15589]. When dual expert groups are meant to remain functionally distinct, explicit decorrelation losses become central. In OneTrackerV2’s Dual Mixture-of-Experts, **T-MoE** models spatio-temporal relations and **M-MoE** models multimodal knowledge; their outputs are regularized by an expert decoupling loss \[ L_{\text{dis} = \cos(y^T, y^M)^2, \] and M-MoE routing is further structured by a router clustering loss \[ L_{\text{cluster} = L_{\text{same} + L_{\text{diff}. \] The total loss also includes a balance loss, though its explicit formula is not provided in the text [2605.03716]. This suggests a general DPMoE design principle: if the two views are intended to specialize, explicit anti-collapse regularization may be required rather than assumed. A different form of expert regularization appears in sparse MoE reward modeling, where local routing entropy, \[ \mathcal L_{\mathrm{ls} = \mathbb E_{x\sim\mathcal D} \left[ \frac{H(\pi_\phi(x))}{\log K} \right], \] global batch balance, \[ \mathcal L_{\mathrm{gb} = \mathbb E_{\mathcal B\sim\mathcal D} \left[ -\frac{H(\bar\pi_\phi)}{\log K} \right], \] and expert diversity on pairwise reward differences, \[ \mathcal L_{\mathrm{div} = \frac{2}{K(K-1)} \sum_{i