Controllable-LPMoE: Fine-Tuning via Local Priors

• The paper introduces a parameter-efficient segmentation method that adapts frozen vision transformers using dynamic, image-conditioned local priors through a Mixture-of-Experts module.
• It details a dual-branch architecture combining a dynamic local priors extractor with a bi-directional interaction adapter to iteratively refine feature representations.
• The approach significantly reduces trainable parameters and training memory while maintaining competitive performance across various binary segmentation benchmarks.

Searching arXiv for the cited papers to ground the article and verify identifiers. Controllable-LPMoE denotes, in its original usage, a dynamic priors-based parameter-efficient fine-tuning paradigm for binary object segmentation that adapts frozen foundation vision transformers by dynamically controlling local priors extracted through a Mixture-of-Experts module (Sun et al., 24 Oct 2025). In related MoE literature, the label has also been used in a broader sense for large-parameter routed systems with explicit control surfaces over activated compute, routing geometry, temporal persistence, expert trajectories, or post-training behavioral edits, rather than relying on unconstrained token-wise routing alone (Jin et al., 16 Dec 2025, Yang et al., 29 Jan 2026, Shen et al., 22 Apr 2026, Ye et al., 20 Apr 2026, Hong et al., 10 Jun 2026). The term therefore refers both to a specific segmentation architecture and, more generally, to a research program in which MoE mechanisms are made externally tunable, auditable, and structurally localized.

1. Definition, problem setting, and scope

The original Controllable-LPMoE framework addresses binary object segmentation with large foundation vision transformers, targeting six task families: camouflaged object detection, salient object detection, polyp segmentation, skin lesion segmentation, shadow detection, and glass detection (Sun et al., 24 Oct 2025). These tasks share a foreground-versus-background output format but differ substantially in appearance statistics, object scale, boundary ambiguity, and local texture structure. The motivating claim is that full-parameter fine-tuning of large backbones such as BEiT-L and UniPerceiver-L incurs substantial computational overhead, while prompt-based adaptation often lacks explicit semantic priors for dense prediction (Sun et al., 24 Oct 2025).

Within that formulation, Controllable-LPMoE adapts a frozen backbone by learning image-conditioned local priors and then injecting them through a dedicated interaction mechanism. The trainable portion comprises a Dynamic Mixed Local Priors extractor, a Bi-Directional Interaction adapter, and a lightweight mask decoder, while the foundation backbone remains frozen (Sun et al., 24 Oct 2025). The resulting design is parameter-efficient rather than sparsity-driven in the routing sense typical of language-model MoEs.

A broader usage emerged in subsequent MoE work. In that literature, “controllable” refers to explicit manipulation of computational sparsity, refusal behavior, switching frequency, routing smoothness, or expert-path semantics. This broader framing treats controllability as a system property of routed models: the model may have many experts or latent routes, but the degree and form of activation are intentionally constrained or steerable (Jin et al., 16 Dec 2025, Shen et al., 22 Apr 2026).

2. Core architecture of the original segmentation framework

Controllable-LPMoE is organized around two branches and a decoder. The task-universal branch is a frozen foundation model, instantiated as BEiT-L or UniPerceiver-L, producing universal features $\{f_u^i\}_{i=1}^{5}$, where $f_u^1$ is the patch-embedding output and $f_u^2,\dots,f_u^5$ are outputs of four equal-sized transformer blocks (Sun et al., 24 Oct 2025). The task-specific branch is the Dynamic Mixed Local Priors extractor, a lightweight convolutional encoder composed of four stages and heterogeneous convolutional experts. A Bi-Directional Interaction adapter bridges the frozen and trainable branches, and a lightweight Transformer-based mask decoder from Mask2Former outputs the binary mask (Sun et al., 24 Oct 2025).

The Dynamic Mixed Local Priors extractor is built from heterogeneous convolutions corresponding to four expert families: depthwise separable convolution, atrous convolution, asymmetric convolution, and wavelet convolution. At each stage, these experts generate diverse local priors, and a gating network dynamically outputs the expert priors required for subsequent fine-tuning (Sun et al., 24 Oct 2025). The stage output is therefore not a fixed prompt or a static adapter response; it is a content-conditioned mixture of local operators.

The interaction module is explicitly bi-directional. First, task-specific priors enrich frozen universal features. Then, updated universal features refine the task-specific branch. This exchange is repeated across four backbone blocks rather than being confined to a single insertion point (Sun et al., 24 Oct 2025). The paper describes two submodules inside this adapter: Cosine-Aligned Deformable Attention, which combines learnable attention with cosine similarity and deformable sampling, and Channel-Oriented Adaptive Scale Enhancement, which restructures task-specific features through channel attention, reverse attention, and gated multi-scale fusion (Sun et al., 24 Oct 2025).

This architecture differs from prompt tuning in two ways emphasized by the paper. First, its trainable signals are image-derived local priors rather than free prompt embeddings. Second, the interaction is iterative and bi-directional, rather than a one-way conditioning of frozen tokens by trainable tokens (Sun et al., 24 Oct 2025). This suggests that the method is best understood as structured feature control rather than prompt insertion.

3. Dynamic local priors, gating, and optimization

At a single stage of the Dynamic Mixed Local Priors extractor, the image $\mathcal{I}_c$ is first down-sampled by a stem operation,

$$\hat{\mathcal{I}}_c = \mathsf{stem}(\mathcal{I}_c).$$

For each expert family $n \in \{1,2,3,4\}$, local features are built through progressively larger receptive fields,

$$\begin{aligned} l_1^n &= \mathcal{C}_1(\mathsf{stem}(\mathcal{I}_c)),\ l_{2k+1}^n &= \mathcal{ZC}_{2k+1}^{n}(l_1^n + l_{2k-1}^n), \quad k=1,2,3, \end{aligned}$$

and are concatenated and compressed into an expert prior,

$$\mathrm{E}_p^n = \mathcal{C}_1\big([l_1^n, l_3^n, l_5^n, l_7^n]\big).$$

The gating network then computes dynamic expert weights

$$w_l^n(\hat{\mathcal{I}}_c) = \mathsf{Softmax}\big(\mathsf{W}_g \hat{\mathcal{I}}_c + \mathsf{b}_g\big),$$

and forms the stage feature

$$f_s^1 = \mathcal{C}_1\big(\hat{\mathcal{I}}_c + \sum_{n=1}^{N} w_l^n \otimes \mathrm{E}_p^n\big), \quad N=4.$$

The same pattern is repeated across four stages, yielding multi-scale task-specific features $f_u^1$0 (Sun et al., 24 Oct 2025).

The Bi-Directional Interaction adapter uses Cosine-Aligned Deformable Attention to align the two branches. In the first interaction, normalized universal features act as queries and normalized task-specific features act as values. The attention weights are

$f_u^1$1

with the cosine term given by

$f_u^1$2

and the feature update is

$f_u^1$3

The reverse interaction updates task-specific features using the refined universal representation,

$f_u^1$4

CASE then decomposes, reweights, and fuses channel-scale experts, finally producing

$f_u^1$5

These mechanisms are the formal basis of the “controllable” local-prior modulation described in the paper (Sun et al., 24 Oct 2025).

Training uses a segmentation objective

$f_u^1$6

with $f_u^1$7 and $f_u^1$8. The binary cross-entropy and Dice terms are

$f_u^1$9

$f_u^2,\dots,f_u^5$0

The paper does not introduce a separate MoE sparsity penalty; control is embedded in the dynamic composition of local priors and in their structured interaction with frozen features (Sun et al., 24 Oct 2025).

4. Empirical performance and efficiency in binary segmentation

The original paper reports that only 23.4M parameters are trainable, compared with 326.7M for full fine-tuning of UniPerceiver-L and 328.0M for full fine-tuning of BEiT-L (Sun et al., 24 Oct 2025). In the efficiency analysis, the UniPerceiver-L full-tuning variant uses 18.63G training memory and 0.907 s per 50 iterations, whereas the parameter-efficient Controllable-LPMoE variant uses 13.75G and 0.819 s; for BEiT-L, the corresponding numbers are 13.32G and 1.032 s for full tuning versus 8.55G and 0.908 s for Controllable-LPMoE (Sun et al., 24 Oct 2025). The paper therefore characterizes the method as reducing trainable parameters by about $f_u^2,\dots,f_u^5$1, reducing training memory by 35–56%, and improving training time per 50 iterations by about 11–14%, with only a minor drop relative to full fine-tuning (Sun et al., 24 Oct 2025).

Quantitatively, the method is reported to outperform 31 state-of-the-art methods across 18 datasets and 6 task families (Sun et al., 24 Oct 2025). On COD10K, the BEiT-L version achieves IoU 0.817, Dice 0.883, and $f_u^2,\dots,f_u^5$2, compared with ZoomXNet at IoU 0.758 and $f_u^2,\dots,f_u^5$3 (Sun et al., 24 Oct 2025). On ECSSD in salient object detection, the BEiT-L version reports IoU 0.927 and Dice 0.955 with $f_u^2,\dots,f_u^5$4; on DUTS-TE it reports IoU 0.861, Dice 0.908, and $f_u^2,\dots,f_u^5$5 (Sun et al., 24 Oct 2025). In medical segmentation, the BEiT-L variant reports CVC-ClinicDB IoU 0.896, Dice 0.935, and $f_u^2,\dots,f_u^5$6, and Kvasir IoU 0.885 and Dice 0.930 (Sun et al., 24 Oct 2025). For shadow detection, the method reports SBU IoU up to 0.861 and Dice 0.917, and on ISTD the UniPerceiver-L version reports IoU 0.941 and Dice 0.965 (Sun et al., 24 Oct 2025). For glass detection, the BEiT-L version reports Trans10k IoU 0.931, Dice 0.961, and $f_u^2,\dots,f_u^5$7, and GDD IoU 0.922, Dice 0.952, and $f_u^2,\dots,f_u^5$8 (Sun et al., 24 Oct 2025).

Ablation results isolate the contributions of the main components. Adding the Dynamic Mixed Local Priors extractor to a frozen UniPerceiver baseline raises CHAMELEON IoU from 0.672 to 0.787, COD10K IoU from 0.652 to 0.752, and NC4K IoU from 0.742 to 0.803 (Sun et al., 24 Oct 2025). Adding Cosine-Aligned Deformable Attention without local priors raises COD10K IoU from 0.652 to 0.754, and combining local priors with CDA raises it to 0.774; adding CASE further raises it to 0.795 (Sun et al., 24 Oct 2025). The paper also reports that using only task-specific-to-frozen or only frozen-to-task-specific interaction underperforms full bi-directional interaction (Sun et al., 24 Oct 2025).

These results establish the specific, vision-oriented meaning of Controllable-LPMoE: efficient adaptation of frozen backbones through dynamically mixed, image-conditioned local priors. They do not, by themselves, imply compute-controllable sparsity or behavioral steering. Those later associations come from adjacent routed-model research.

5. Broader controllable-MoE interpretations in later work

Several later papers describe mechanisms that fit a broader “controllable LPMoE” interpretation, although they address different modalities and control axes.

Control axis	Mechanism	Representative paper
Activated compute	Dynamic Top-p routing with PI feedback	(Jin et al., 16 Dec 2025)
Refusal behavior	Support-gated multidirectional rank-one edits	(Hong et al., 10 Jun 2026)
Temporal persistence	Option-style expert masks with deliberation cost	(Shen et al., 22 Apr 2026)
Routing geometry	Low-rank latent routing with SIPS	(Yang et al., 29 Jan 2026)
Semantic trajectories	Router-visible control/content decomposition	(Ye et al., 20 Apr 2026)

In sparsity control, "Sparsity-Controllable Dynamic Top-p MoE for Large Foundation Model Pre-training" introduces DTop-p MoE, a dynamic Top-p routing mechanism in which a Proportional-Integral controller adjusts the probability threshold so that the running activated-expert sparsity tracks a specified target (Jin et al., 16 Dec 2025). The paper states that standard Top-k routing imposes a uniform sparsity pattern and that fixed-threshold Top-p is adaptive but uncontrolled. DTop-p is presented as a way to decouple total capacity from activated compute while preserving token- and layer-adaptive routing; it further adds dynamic routing normalization so that different layers can learn distinct expert-selection patterns under a global threshold (Jin et al., 16 Dec 2025). This broader usage of controllable LPMoE concerns a hardware-relevant compute knob rather than segmentation adaptation.

In post-training behavior control, "LoMC: Localized Multidirectional Correction for Refusal Suppression in Routed Foundation Models" studies routed MoE and hybrid-MoE models and introduces a support-then-correction framework: first identify a compact edit support, then aggregate prototype correction directions into layer-wise correction directions, and finally apply rank-one layer-wise correction only within the selected support (Hong et al., 10 Jun 2026). The paper defines refusal rate and Target Compliance Rate, emphasizes that TCR measures non-refusal rather than harmfulness, and reports average TCR rising from 8.75% in original models to 96.33% under LoMC while average GCA changes from 63.77% to 64.25% across the reported settings (Hong et al., 10 Jun 2026). In this interpretation, controllability refers to behavior editing under a compact intervention footprint.

In temporal control, "Temporally Extended Mixture-of-Experts Models" frames expert-set selection as an options problem in reinforcement learning (Shen et al., 22 Apr 2026). Each MoE layer maintains a persistent expert mask, and a controller decides when to terminate that mask and which new mask to load. The deliberation cost $f_u^2,\dots,f_u^5$9 acts as a direct switch-rate knob: increasing $\mathcal{I}_c$0 reduces terminations and lengthens expert-mask persistence (Shen et al., 22 Apr 2026). Applied to gpt-oss-20b with low-rank adapters and self-distillation, the method reduces switch rates from over 50% to below 5% while retaining up to 90% of base-model accuracy on MATH, MMLU, and MMMLU (Shen et al., 22 Apr 2026). Here, controllability is temporal rather than spatial.

In routing-geometry control, "L2R: Low-Rank and Lipschitz-Controlled Routing for Mixture-of-Experts" replaces raw high-dimensional linear routing with a shared low-rank routing space and Saturated Inner-Product Scoring,

$\mathcal{I}_c$1

where $\mathcal{I}_c$2 is bounded and saturating and $\mathcal{I}_c$3 compresses anchor norms (Yang et al., 29 Jan 2026). The paper argues that this yields smoother and more stable routing geometry, improves expert specialization, and explicitly controls the Lipschitz behavior of routing functions (Yang et al., 29 Jan 2026). In that broader sense, controllability refers to geometric regularity and stable expert assignment.

Finally, "Polysemantic Experts, Monosemantic Paths: Routing as Control in MoEs" interprets routing itself as a control channel (Ye et al., 20 Apr 2026). Using an SVD of the router matrix, it decomposes each hidden state into a router-visible control signal and an orthogonal router-blind content channel. The paper reports that surface-level features such as language, token identity, and position primarily reside in the content channel, while the control signal changes more rapidly across layers and determines expert choice (Ye et al., 20 Apr 2026). Its central claim is that the natural unit of interpretability in MoEs is the trajectory rather than the individual expert. This suggests a path-based notion of controllable LPMoE in which steering is performed on trajectories or control subspaces rather than on isolated experts.

6. Conceptual significance, misconceptions, and open directions

A common ambiguity is to treat Controllable-LPMoE as if it referred to a single unified architecture across all of these works. The evidence does not support that reading. The exact title "Controllable-LPMoE: Adapting to Challenging Object Segmentation via Dynamic Local Priors from Mixture-of-Experts" names a specific vision-segmentation method built around dynamic local priors, CDA, CASE, and parameter-efficient adaptation of frozen backbones (Sun et al., 24 Oct 2025). The later papers instead use the phrase in an extended or analogous sense: controllable sparsity, controllable post-training edits, controllable switching, controllable routing smoothness, or controllable expert trajectories (Jin et al., 16 Dec 2025, Hong et al., 10 Jun 2026, Shen et al., 22 Apr 2026, Yang et al., 29 Jan 2026, Ye et al., 20 Apr 2026).

A second misconception is that “controllable” always means compute control. In the segmentation paper, control is exerted over image-conditioned local priors and their interaction with frozen features, not over activated FLOPs (Sun et al., 24 Oct 2025). In DTop-p, by contrast, controllability is explicitly a compute knob defined by target average activated experts per token (Jin et al., 16 Dec 2025). In LoMC it is a behavior knob over refusal suppression (Hong et al., 10 Jun 2026). In temporally extended MoEs it is a trade-off between switch rate and retained accuracy (Shen et al., 22 Apr 2026). These are related only at the level of design philosophy.

The papers also delimit several limitations. The original segmentation method is restricted to binary segmentation tasks and involves implementation complexity from deformable attention and MoE gating, while full fine-tuning still slightly outperforms the parameter-efficient variant in absolute metrics (Sun et al., 24 Oct 2025). DTop-p reports experiments up to about 13.6B total parameters and 300B tokens, leaving frontier-scale behavior to future validation (Jin et al., 16 Dec 2025). The temporally extended MoE work argues for memory and latency savings but does not implement a full GPU–CPU offloading engine (Shen et al., 22 Apr 2026). LoMC explicitly notes that TCR is non-refusal rather than harmfulness and that static edits may still affect behaviors beyond refusal (Hong et al., 10 Jun 2026). The path-based control paper cautions that individual experts remain polysemantic, so expert-level interpretability is structurally limited (Ye et al., 20 Apr 2026).

Future directions stated across these works define a broad research agenda for controllable MoE systems. In vision, the segmentation paper identifies extensions to multi-class segmentation, instance segmentation, video segmentation, 3D volumetric segmentation, and multi-task learning (Sun et al., 24 Oct 2025). In sparsity control, proposed directions include multi-objective control, expert-choice variants, heterogeneous experts, and frontier-scale validation (Jin et al., 16 Dec 2025). In behavioral editing, proposed directions include extending the support-then-correction framework beyond refusal to toxicity suppression, stylistic control, and modality preference (Hong et al., 10 Jun 2026). In temporal control, future work includes latency-aware costs, joint options across layers, and pretraining with temporal extension built in (Shen et al., 22 Apr 2026). In routing geometry, a plausible implication is that explicit spectral constraints or additional Lipschitz regularization could further tighten routing control, because the paper already frames SIPS as a structural mechanism for controlling the Lipschitz behavior of the router (Yang et al., 29 Jan 2026). In interpretability, the trajectory-based view suggests that future controllable MoEs may intervene directly in low-dimensional control subspaces rather than editing whole experts (Ye et al., 20 Apr 2026).

Taken together, these results position Controllable-LPMoE as both a concrete segmentation framework and an organizing idea for routed models: a controllable MoE is one in which the locus of adaptation is explicit, localized, and auditable, whether that locus is local visual priors, activated sparsity, edit support, temporal expert masks, routing geometry, or expert trajectories.