Decoupled Proxy Alignment

• Decoupled Proxy Alignment (DPA) is a design pattern that isolates alignment signals into dedicated proxy modules to enable targeted ethical adjustments without retraining the entire model.
• It encompasses methods like parameter transplantation, proxy-controlled decoding, and multimodal pre-alignment to address challenges such as harmful-prompt scenarios and language prior conflicts.
• Empirical results indicate that DPA improves alignment performance with minimal parameter overhead, yielding significant gains on benchmarks such as AdvBench and GSM8K.

Decoupled Proxy Alignment (DPA) denotes a family of alignment strategies in which alignment-relevant information is isolated into a separate proxy—such as a small parameter subset, an auxiliary model, a proxy representation, or a per-objective optimization signal—and then coupled back to a larger capability model in a controlled way. In this sense, DPA appears as parameter transplantation in "Decoupled Alignment for Robust Plug-and-Play Adaptation" (Luo et al., 2024), as proxy-LLM training for multimodal language prior conflict in "Decoupled Proxy Alignment: Mitigating Language Prior Conflict for Multimodal Alignment in MLLM" (Tan et al., 18 Sep 2025), as proxy-controlled decoding in "Proxy-RLHF: Decoupling Generation and Alignment in LLM with Proxy" (Zhu et al., 2024) and "On the Rejection Criterion for Proxy-based Test-time Alignment" (Hammal et al., 17 Apr 2026), and as architectural pre-alignment via a small VLM perceiver in "Deep Pre-Alignment for VLMs" (Yu et al., 14 May 2026). This suggests that DPA is best understood as a design pattern rather than a single canonical algorithm.

1. Terminological scope and conceptual core

The acronym is overloaded across adjacent literatures. "Directional Preference Alignment" uses DPA for a multi-objective reward-conditioning framework in which user preferences are represented as directions $v \in \mathbb{S}^k$ in reward space (Wang et al., 2024). "Understanding the Logic of Direct Preference Alignment through Logic" uses DPA as an umbrella term for offline preference-based methods such as DPO, IPO, SliC, RRHF, CPO, ORPO, and SimPO (Richardson et al., 2024). "Deep Pre-Alignment for VLMs" also uses the acronym DPA, but for "Deep Pre-Alignment" rather than decoupled proxy alignment (Yu et al., 14 May 2026). By contrast, the explicit phrase "Decoupled Proxy Alignment" appears in the multimodal alignment setting of language prior conflict (Tan et al., 18 Sep 2025).

Within the decoupled-proxy reading, the recurring principle is the separation of capability parameters from alignment machinery. The proxy may be a transplanted parameter subset, a small aligned controller, a shared latent proxy, or a geometry-aware combination of per-objective gradients. What remains stable is the decomposition: the large model supplies most of the competence, while the proxy carries the alignment signal. This broader pattern is explicit in parameter-level gate transplantation (Luo et al., 2024), proxy-guided decoding (Zhu et al., 2024, Hammal et al., 17 Apr 2026), proxy-space omnimodal distillation (Gao et al., 12 Apr 2026), and proxy LLMs for multimodal pretraining (Tan et al., 18 Sep 2025).

2. Parameter-localized DPA in LLMs

A concrete and technically precise instance appears in "Decoupled Alignment for Robust Plug-and-Play Adaptation" (Luo et al., 2024). There, alignment is treated as a localized memory space inside an already aligned teacher model. Causal tracing shows that in harmful-prompt scenarios the largest indirect effects occur in middle layers, and within those layers the MLP contribution has higher effect than attention. A finer module-level analysis further decomposes the MLP into up, gate, and down projections and finds that the gate projections $W_{\text{gate}^{(l)}}$ have the largest impact on the last-token representation, especially in middle layers. The operational interpretation is that the gates are the primary carriers of alignment behavior.

The proxy is therefore defined as a subset of gate matrices extracted from an aligned teacher,

$$\mathcal{A} = \Big\{ W_{\text{gate}^{(l)}} \,\big|\, l \in S^* \Big\},$$

where $S^*$ is a minimal set of layers selected by delta debugging. The search space is the set of candidate gate layers $\mathbb{S}$, and a policy $\pi : \mathbb{S} \to \{0,1\}$ marks whether a transplanted subset yields ethical responses to a held-out 5% subset of additional unethical questions. The delta debugging procedure returns the smallest subset $S^* \subseteq \mathbb{S}$ such that $\pi(S^*)=1$.

The resulting hybrid model replaces only the teacher’s gate matrices in layers $S^*$ while leaving attention and the remaining MLP parameters unchanged. At inference, the hybrid runs exactly like a standard transformer, with no runtime guardrail model and no additional forward pass. Empirically, the method was applied to 17 unaligned pre-trained or fine-tuned LLMs across Llama-2, Mistral, and Gemma families, adapting at most 8.11% of parameters. On AdvBench harmful prompts, it improves average Defense Success Rate by 14.41 percentage points and reaches 51.39 percentage points on one Gemma-based SFT variant, while the average perplexity increase is only 1.69 and the average reasoning drop is only 2.59% (Luo et al., 2024).

This formulation is prototypical DPA: alignment is localized into a small reusable proxy module, causally identified, minimally extracted, and transplanted without SFT or RLHF on the student side.

3. Proxy-controlled decoding and test-time DPA

A second major form of DPA decouples alignment at inference time rather than at the parameter level. In "Proxy-RLHF" (Zhu et al., 2024), the base LLM is frozen and performs pure next-token generation, while a separate proxy policy $\pi_\theta(a_i \mid s_i)$ decides whether to accept or reject each candidate token. The environment contains the base model $W_{\text{gate}^{(l)}}$0, the action space is binary $W_{\text{gate}^{(l)}}$1, and rejection causes the model to resample at the same position after removing previously rejected tokens. The effective probability of ultimately selecting a token is thus shaped by a sequence of proxy rejections over more probable alternatives. Alignment is trained with RL on the proxy only; the base LLM is not updated. On the filtered HH helpfulness benchmark with Alpaca-7B, Proxy-RLHF uses 0.03B trainable parameters versus 13.35B for RLHF and 6.74B for DPO, and achieves a GPT-4 pairwise win rate of 63.24% against 61.24% for RLHF and 42.65% for DPO (Zhu et al., 2024).

"On the Rejection Criterion for Proxy-based Test-time Alignment" generalizes this setting with a probabilistic graphical model over a draft token $W_{\text{gate}^{(l)}}$2, a rejection variable $W_{\text{gate}^{(l)}}$3, and the emitted token $W_{\text{gate}^{(l)}}$4 (Hammal et al., 17 Apr 2026). The induced marginal is

$W_{\text{gate}^{(l)}}$5

where $W_{\text{gate}^{(l)}}$6 is the large base model, $W_{\text{gate}^{(l)}}$7 is the small aligned proxy, and $W_{\text{gate}^{(l)}}$8 is the token-dependent rejection distribution. Within this view, nudging, KAD, and implicit-reward methods differ only in the rejection criterion. The paper argues that raw confidence thresholds are ill-motivated because ambiguous phrasing can yield low token probabilities even for good continuations, and proposes the conservative confidence bet,

$W_{\text{gate}^{(l)}}$9

On GSM8K, MATH500, SVAMP, ARC-Challenge, and CommonsenseQA, the conservative confidence bet is competitive with or better than dual KAD, and for OLMo 2 its best reported average accuracy is 58.9 at $$\mathcal{A} = \Big\{ W_{\text{gate}^{(l)}} \,\big|\, l \in S^* \Big\},$$0, compared with 57.7 for dual KAD at $$\mathcal{A} = \Big\{ W_{\text{gate}^{(l)}} \,\big|\, l \in S^* \Big\},$$1 (Hammal et al., 17 Apr 2026).

These test-time methods retain the same structural decomposition as parameter-transplant DPA: capability remains in the base model, while alignment is offloaded to a smaller proxy that either vetoes or redirects generation.

4. Multimodal DPA: proxy language priors and deep pre-alignment

In multimodal LLMs, DPA appears as a response to the mismatch between a base LLM’s inherent language prior and the language prior embedded in image-caption datasets. "Decoupled Proxy Alignment: Mitigating Language Prior Conflict for Multimodal Alignment in MLLM" names this mismatch language prior conflict and proposes a three-stage framework (Tan et al., 18 Sep 2025). Stage 1, Proxy Model Optimization, trains a proxy LLM on text-only captions with LoRA using

$$\mathcal{A} = \Big\{ W_{\text{gate}^{(l)}} \,\big|\, l \in S^* \Big\},$$2

Stage 2 constructs a proxy MLLM by combining the proxy LLM, a frozen CLIP-pretrained ViT-L, and a 2-layer MLP connector, then trains only the connector with Contrastive Modality Optimization. Stage 3 restores the original target LLM, retains the learned connector, and performs instruction tuning with the same CMO weighting. Visual relevance is estimated by the probability difference

$$\mathcal{A} = \Big\{ W_{\text{gate}^{(l)}} \,\big|\, l \in S^* \Big\},$$3

where $$\mathcal{A} = \Big\{ W_{\text{gate}^{(l)}} \,\big|\, l \in S^* \Big\},$$4 is the token distribution with image and $$\mathcal{A} = \Big\{ W_{\text{gate}^{(l)}} \,\big|\, l \in S^* \Big\},$$5 is the text-only distribution, and the weighted loss becomes

$$\mathcal{A} = \Big\{ W_{\text{gate}^{(l)}} \,\big|\, l \in S^* \Big\},$$6

The empirical picture is explicitly multimodal. For Qwen2.5-7B-Instruct on ShareGPT4V-PT, the average score rises from 49.0 for Vanilla to 51.7 for DPA, with notable gains on vision-centric tasks. In an ablation on ShareGPT4V-PT with Qwen2.5, the baseline gives General Avg. 54.3 and Vision Avg. 43.8, PMO only gives 55.4 and 46.1, CMO only gives 55.7 and 47.2, and full DPA gives 55.8 and 47.6. The method is also data-hungry: at very small data sizes $$\mathcal{A} = \Big\{ W_{\text{gate}^{(l)}} \,\big|\, l \in S^* \Big\},$$7, DPA underperforms Vanilla, while at $$\mathcal{A} = \Big\{ W_{\text{gate}^{(l)}} \,\big|\, l \in S^* \Big\},$$8 it begins to outperform it substantially. The training overhead is about 33% in time and about 20% in memory (Tan et al., 18 Sep 2025).

A structurally related architectural variant appears in "Deep Pre-Alignment for VLMs" (Yu et al., 14 May 2026). There, the proxy is not a style-adapted LLM but a small VLM perceiver inserted before the target LLM: $$\mathcal{A} = \Big\{ W_{\text{gate}^{(l)}} \,\big|\, l \in S^* \Big\},$$9 The perceiver uses deep LM blocks to pre-align visual features to the target LLM’s text space before the large LLM sees them. MIR analysis shows that modality gap decreases across perceiver layers and remains lower across the target LLM than in LLaVA-style baselines. On the 4B scale, DPA improves the multimodal average across 8 benchmarks by 1.9 points; on the 32B scale, the gain widens to 3.0 points. It also reduces language capability forgetting by 32.9% at 4B and 21.6% at 32B over three text benchmarks, with gains reported for both Qwen3 and LLaMA 3.2 families (Yu et al., 14 May 2026).

At a finer granularity, phrase-level hallucination mitigation through correct versus hallucinated response pairs and a contrastive-plus-KL objective is conceptually close to DPA, because the alignment pressure is localized to proxy comparisons while a KL term preserves the base model’s general behavior (Sarkar et al., 2024).

5. Proxy spaces, multi-objective proxies, and logical formulations

The proxy in DPA need not be a model or a parameter subset. In omnimodal dataset distillation, "Omnimodal Dataset Distillation via High-order Proxy Alignment" introduces a shared low-rank proxy that replaces explicit pairwise modality modeling (Gao et al., 12 Apr 2026). For each instance, modality embeddings are stacked into $S^*$0, the Gram matrix is $S^*$1, and the proxy is the principal singular structure $S^*$2. Cross-instance similarity is defined in proxy space as $S^*$3. The key theoretical point is eigenvalue selectivity: pairwise objectives activate all singular modes, whereas HoPA’s inner objective depends only on the principal mode, yielding a tighter bound on endpoint discrepancy whose single-mode form does not depend on the number of modalities $S^*$4. Empirically, rank-1 proxies outperform rank-2 proxies, and overall distillation is about $S^*$5 faster than pairwise baselines despite a 39% per-iteration SVD overhead (Gao et al., 12 Apr 2026).

In multi-objective language alignment, the same separation appears as explicit value proxies. "Directional Preference Alignment" models a reward vector $S^*$6 and a user preference direction $S^*$7, then scalarizes via

$S^*$8

The model is trained as a preference-conditioned policy $S^*$9, so the reward dimensions remain explicit and are combined on demand rather than collapsed once and for all (Wang et al., 2024). A different reconciliation mechanism appears in "MGDA-Decoupled," where each objective $\mathbb{S}$0 has its own DPO loss $\mathbb{S}$1, the coefficient-selection geometry is based on $\mathbb{S}$2, and the applied update is the raw combination $\mathbb{S}$3. This is decoupled because the gradients used to compute the mixing weights are not the same vectors that are applied to the model (Vári-Kakas et al., 22 Apr 2026).

A further terminological complication is that "Direct Preference Alignment" uses DPA as an umbrella term for direct offline preference-learning losses. There the general form is

$\mathbb{S}$4

and preference losses are formalized through symbolic preference structures $\mathbb{S}$5 whose weighted model counts recover the log-ratio $\mathbb{S}$6 (Richardson et al., 2024). In this literature, the proxy is not a separate model but a declarative preference semantics.

6. Empirical regularities, constraints, and misconceptions

A recurrent empirical regularity is that DPA improves alignment by relocating alignment effort into a smaller or more specialized component while preserving a larger capability substrate. In parameter-transplant DPA, this appears as higher harmful-prompt refusal with only modest changes in perplexity and reasoning (Luo et al., 2024). In proxy-controlled decoding, it appears as comparable or better alignment with orders-of-magnitude fewer trainable parameters than RLHF or DPO (Zhu et al., 2024). In multimodal DPA, it appears as improved vision-centric performance and reduced language forgetting because the proxy absorbs either language-prior adaptation or modality alignment (Tan et al., 18 Sep 2025, Yu et al., 14 May 2026).

The literature also places clear limits on the pattern. Parameter-transplant DPA requires architectural compatibility and, in practice, same-family teacher–student pairs with matching layer shapes; cross-family transfer is not attempted (Luo et al., 2024). Proxy-based test-time alignment depends critically on the rejection rule; confidence thresholds alone can be linguistically ill-motivated because ambiguity lowers token confidence without implying poor quality, and there is an expressiveness boundary for single-proxy rejection schemes (Hammal et al., 17 Apr 2026). The explicit multimodal DPA of language prior conflict is sensitive to CMO clipping bounds, requires enough data for proxy estimation and visual relevance to be reliable, and increases training time and memory (Tan et al., 18 Sep 2025). HoPA incurs SVD overhead, especially when backpropagating through SVD, even though its global scaling is better than pairwise alternatives (Gao et al., 12 Apr 2026). Deep Pre-Alignment adds non-zero computation and depends on language-pretrained perceiver LM blocks; random initialization of the perceiver LM causes severe degradation (Yu et al., 14 May 2026).

A common misunderstanding is to treat DPA as a single algorithm. The available work does not support that reading. The same structural motif recurs at several levels: a small gate subset can serve as an alignment memory space, a proxy model can control or replace tokens at decoding time, a proxy LLM can absorb dataset language priors before multimodal tuning, a perceiver can pre-align image features before they enter a large LLM, and a low-rank proxy space or per-objective gradient geometry can decouple alignment from combinatorial interactions. The unifying idea is not a fixed loss or architecture, but a modular separation between capability and alignment, together with an explicit interface for recombining them (Luo et al., 2024, Tan et al., 18 Sep 2025, Zhu et al., 2024, Gao et al., 12 Apr 2026).

Under that broader interpretation, Decoupled Proxy Alignment identifies a distinct trajectory in alignment research. Rather than repeatedly fine-tuning the entire model against a monolithic scalar objective, it seeks compact loci of alignment signal—submodules, controllers, proxy spaces, or objective decompositions—that can be audited, searched, transplanted, or conditioned with lower marginal cost and with less disruption to the underlying model.