Shortcut-Rerouted Adapter Training

• The paper’s main contribution shows that providing explicit auxiliary paths for nuisance factors prevents the main adapter from internalizing unwanted attributes.
• Shortcut-Rerouted Adapter Training uses a frozen text-to-image backbone, a trainable identity encoder, and frozen shortcut modules (e.g., LoRA or ControlNet) to separate target features from confounders.
• Empirical results indicate improved identity preservation and reduced pose/expression leakage compared to conventional reconstruction-based adapter training.

Shortcut-Rerouted Adapter Training is a training strategy for text-to-image personalization adapters in which the model is given explicit auxiliary paths for nuisance factors during training so that the main adapter is not incentivized to internalize them. In the formulation introduced in "Preventing Shortcuts in Adapter Training via Providing the Shortcuts" (Goyal et al., 23 Oct 2025), the target attribute is typically facial identity or full-body identity, while pose, facial expression, lighting, background, composition, camera/viewpoint, body pose, and dataset/domain style are treated as confounding factors. The auxiliary modules are active only during training and are removed during inference. The central claim is deliberately paradoxical: if adapter training suffers from shortcut learning, one effective way to prevent shortcuts is to provide the shortcuts (Goyal et al., 23 Oct 2025).

1. Problem setting and design principle

The method is motivated by a structural property of single-image personalization. A reference image contains not only the intended target attribute but also incidental attributes. Under a standard reconstruction objective, an adapter that minimizes reconstruction loss can encode both the target factor $T$ and confounding factors $C$, yielding a "copy-and-paste" behavior in which the adapter leaks pose, expression, or appearance conditions from the reference image into later generations. The paper frames this as a disentanglement problem induced by the objective itself: the adapter is rewarded for reconstructing the entire image, not only the stable attribute that should be transferred (Goyal et al., 23 Oct 2025).

The formal setup begins from an observed image $X$ generated from a target factor and confounders,

$X \sim p(X \mid T, C),$

where $T$ denotes what the adapter should learn and $C$ denotes nuisance variables. The desired behavior is that the frozen generator $G$ conditioned on the adapter $\mathcal{A}$ should approximate

$G(\mathcal{A}(X)) \approx p(\cdot \mid T).$

Under standard reconstruction-based adapter training, however, the objective

$$\mathbb{E}_{(X)}\big[\mathcal{L}\big(G(\mathcal{A}(X)), X\big)\big]$$

encourages $C$0 to encode everything present in $C$1, including both $C$2 and $C$3 (Goyal et al., 23 Oct 2025).

Shortcut-Rerouted Adapter Training modifies the optimization pressure rather than attempting to impose disentanglement abstractly. It introduces an auxiliary shortcut module $C$4 that directly supplies confounding information to the generator during training: $C$5 with training objective

$C$6

At inference, the shortcut module is removed and generation reverts to

$C$7

The intended effect is that confounding factors are "explained away" by $C$8, leaving the main adapter under less pressure to memorize them (Goyal et al., 23 Oct 2025).

2. Formal mechanism and division of representational labor

Architecturally, the setup consists of a frozen base text-to-image backbone $C$9, a trainable main adapter $X$0 that encodes the target attribute from the reference image, and one or more auxiliary shortcut modules $X$1 that carry nuisance information during training. The paper states that $X$2 is a pre-trained and frozen module. This establishes a strict division of labor: the main adapter learns the target factor, and the shortcut module explains the confounds (Goyal et al., 23 Oct 2025).

Two primary instantiations of $X$3 are studied. In SR-LoRA, the shortcut path is a lightweight LoRA trained to absorb dataset-specific distributional characteristics such as style, lighting, and low-level appearance biases. In SR-CN, the shortcut path is a pretrained ControlNet that receives automatically extracted pose/expression maps for facial identity or body pose structure for full-body identity. In both cases the auxiliary module is active only during training and removed during inference (Goyal et al., 23 Oct 2025).

The paper also specifies a stage-level asymmetry. The shortcut modules are not jointly optimized with the main adapter during the main training stage. For SR-LoRA, the LoRA shortcut is pretrained first and then frozen while the identity adapter is trained. For SR-CN, the ControlNet remains pretrained/frozen according to the paper’s general definition of $X$4 as "pre-trained and frozen." This suggests that Shortcut-Rerouted Adapter Training is not merely multi-condition training, but a training-time intervention in which nuisance channels are deliberately externalized (Goyal et al., 23 Oct 2025).

One notational caveat is explicit in the source. The SR-CN stage is written with the objective

$X$5

even though the earlier formalism uses $X$6. The paper notes that this appears to be a notational simplification or inconsistency rather than a change in the actual setup, because the adapter is fed the reference image and trained to encode identity from it (Goyal et al., 23 Oct 2025).

3. Concrete instantiations and training workflow

In the reported experiments, the frozen backbone is FLUX.1 [Dev], implemented in Diffusers, with a DiT backbone and a Conditional Flow Matching objective. The main adapter is an IP-Adapter-style identity encoder. The paper does not provide a lower-level architectural equation for the IP-Adapter internals or the exact Flow Matching loss formula used by FLUX. It also does not present an explicit LoRA weight-update formula such as $X$7, nor an explicit ControlNet residual equation, even though these components are used operationally (Goyal et al., 23 Oct 2025).

The three principal variants can be summarized as follows.

Variant	Shortcut module	Routed confounders
SR-LoRA	pretrained lightweight LoRA	dataset/domain shift, style, lighting, low-level appearance biases
SR-CN (face)	pretrained ControlNet	head pose and facial expression
SR-CN (full-body)	pretrained ControlNet	body pose structure

For a face-identity adapter with SR-LoRA, the procedure is: collect a personalization training dataset of human images; train a lightweight LoRA on that finetuning dataset to absorb dataset-specific distributional characteristics such as style, lighting, and low-level appearance biases; freeze the base FLUX model; freeze the pretrained shortcut LoRA; extract the adapter input face crop from each image using facial landmarks and alignment; use the image and caption in the standard diffusion/flow reconstruction training pipeline with both the main adapter and frozen shortcut LoRA active; optimize only the main identity adapter $X$8; and remove the LoRA shortcut at inference (Goyal et al., 23 Oct 2025).

For a face-identity adapter with SR-CN, the procedure is: freeze the base FLUX model; use a pretrained ControlNet as the shortcut module; extract the reference face crop for the identity adapter; automatically derive pose/expression conditioning maps from each training image using pose estimation and landmark detection; feed identity information through the main adapter and pose/expression maps through ControlNet; reconstruct the training image with the frozen generator conditioned on text, the main adapter, and ControlNet; optimize the main adapter while the ControlNet remains pretrained/frozen; and remove ControlNet at inference (Goyal et al., 23 Oct 2025).

For a full-body identity adapter with SR-CN, the procedure is: prepare full-body crops using segmentation and background removal; treat full-body identity as the target factor, including body type, clothing, and limb proportions; extract body pose structure from each training image using a body pose estimator; route body pose through the ControlNet shortcut during training; train the body IP-Adapter to encode identity traits while structural pose is handled externally; and remove the ControlNet at inference (Goyal et al., 23 Oct 2025).

The training corpus is an internal large-scale set of a few million high-quality human images, filtered to retain single-subject photos and remove low-quality, NSFW, or watermarked content. Images are bucketed by aspect ratio, and auxiliary modalities such as landmarks, segmentation masks, and text embeddings are cached. Captions are generated by Qwen2.5-14B for text and InternViT-300M-V2.5 for vision-assisted captioning. Training uses PyTorch and HuggingFace Diffusers with AdamW, learning rate $X$9, global batch size 32, over 250K iterations on $X \sim p(X \mid T, C),$0 A100 80GB GPUs. Inference is standardized with IP scale 1.0, classifier-free guidance 3.5, 28 steps, and $X \sim p(X \mid T, C),$1 resolution. The identity encoder backbone is openai/clip-vit-large-patch14 (Goyal et al., 23 Oct 2025).

4. Evaluation protocol and empirical findings

The evaluation protocol targets identity preservation, prompt following, and prior preservation. The reported metrics are FaceNet Id. $X \sim p(X \mid T, C),$2, LLM Id. $X \sim p(X \mid T, C),$3, LLM Expr. $X \sim p(X \mid T, C),$4, EMOCA Sim. $X \sim p(X \mid T, C),$5, Head Pose $X \sim p(X \mid T, C),$6, Body Pose $X \sim p(X \mid T, C),$7, and Prior (LPIPS) $X \sim p(X \mid T, C),$8. The baselines for face adapters are InfU, PuLID, and a vanilla IP-Adapter trained by the authors without shortcut rerouting. The baselines for full-body adapters are InstantX and vanilla IPA (Goyal et al., 23 Oct 2025).

For face adapters, the quantitative results are: InfU with LLM Id. 3.3824, FaceNet Id. 0.7402, LLM Expr. 3.7664, EMOCA Expr. 0.5420, Head Pose 17.7139, Prior LPIPS 0.4490; PuLID with 4.2826, 0.7742, 3.5899, 0.4890, 17.5345, 0.4584; IPA with 4.7929, 0.7150, 3.0714, 0.3470, 16.1199, 0.4800; SR-LoRA IPA with 4.7194, 0.6708, 3.4286, 0.4580, 13.2701, 0.4330; and SR-CN IPA with 4.7941, 0.7118, 3.6934, 0.5800, 12.6755, 0.3937. These results support two reported effects: both SR-LoRA and SR-CN improve prior preservation relative to vanilla IPA, and SR-CN improves controllability over expression and head pose while keeping LLM Id. essentially tied with the best (Goyal et al., 23 Oct 2025).

For full-body adapters, the reported results are: InstantX with LLM Id. 2.9930, FaceNet Id. 0.3533, LLM Expr. 3.4736, EMOCA Expr. 0.4687, Head Pose 25.97, Body Pose 186.7454, Prior LPIPS 0.5075; IPA with 4.5986, 0.5733, 3.3000, 0.3466, 20.70, 167.4000, 0.4566; and SR-CN IPA with 4.6510, 0.5857, 3.5263, 0.4794, 18.05, 137.6888, 0.4133. In this setting SR-CN improves every reported metric over vanilla IPA, including identity, expression, head pose, body pose, and Prior LPIPS (Goyal et al., 23 Oct 2025).

The ablation story in the main text distinguishes shortcut type rather than offering a full combinatorial sweep. For face adapters, the paper compares vanilla IPA, SR-LoRA IPA, and SR-CN IPA. It also discusses additional combinations such as SR-LoRA+CN, SR-LoRA+CN+BG, and a background shortcut module in the appendix/qualitative section. The reported qualitative interpretation is that the LoRA shortcut mitigates distribution shift and quality degradation, the ControlNet shortcut preserves pose priors and reduces pose/expression leakage, the background shortcut suppresses illumination leakage, and combining them better isolates identity by simultaneously routing away multiple confounders (Goyal et al., 23 Oct 2025).

5. Relation to adjacent adapter rerouting literatures

Shortcut-Rerouted Adapter Training occupies a specific niche within a broader family of modular adaptation methods. Its defining property is training-time rerouting of nuisance information through auxiliary modules that are later removed. This contrasts with post hoc composition methods over task-specialized modules. In continual learning for LLMs, L2R trains each new LoRA adapter in isolation, deactivates previous adapters during new-task learning, and only afterward learns layerwise router networks that compose task-specific adapters using a small episodic memory. Its weighted-output variant is

$X \sim p(X \mid T, C),$9

with Gumbel-sigmoid gates, but the paper explicitly separates module learning from route learning; rerouting is postponed to a post hoc composition stage rather than used as a training-time shortcut path (Araujo et al., 2024).

A second adjacent line is training-free LoRA pool routing. LoRAuter routes a query through task representations rather than directly through adapter characteristics. It constructs task embeddings from small validation sets, retrieves top-$T$0 tasks by cosine similarity, maps them to adapters, and composes them with an input-aware weighted output-space mixture,

$T$1

without retraining the adapters themselves. This is rerouting in a semantic task space, not shortcut provision during adapter training (Dhasade et al., 29 Jan 2026).

A third neighboring paradigm is activation-space control rather than conventional PEFT. ASA uses a linear softmax router over intermediate activations, domain-specific probes, and a single-shot activation intervention

$T$2

where $T$3 combines a global steering vector with a routed domain-specific vector. The paper describes ASA as an inference-time, training-free mechanism with lightweight routing, but it is not adapter training in the conventional sense (Wang et al., 4 Feb 2026).

Other nearby works focus on where or how adaptation should be constrained once trainable modules exist. DomLoRA argues that adaptation signal is highly concentrated in one shallow FFN down-projection identified by PAGE and places a single LoRA adapter there, using about $T$4 of vanilla LoRA’s trainable parameters (Zhang et al., 7 May 2026). Shortcut Guardrail trains a LoRA debiasing module at deployment time so that classifier representations become less sensitive to top-saliency tokens, thereby rerouting prediction-relevant computation away from token-level shortcuts (Li et al., 14 Apr 2026). SART, by contrast, treats shortcut reasoning as a gradient-level optimization pathology and alters training through ShortcutScore-based reweighting and gradient surgery rather than through auxiliary nuisance routes (Cao et al., 21 Mar 2026). Taken together, these works suggest that "shortcut-rerouted" can denote distinct mechanisms—training-only nuisance channels, post hoc adapter composition, task-space retrieval, activation steering, or gradient rerouting—but only the text-to-image method uses auxiliary shortcut modules that are intentionally removed at inference (Goyal et al., 23 Oct 2025).

6. Scope, limitations, and common misconceptions

A recurrent misconception is that Shortcut-Rerouted Adapter Training is simply a richer inference-time conditioning stack. In fact, the auxiliary shortcut modules are scaffolding used during training and are removed during inference. The retained object is the main adapter $T$5, which is expected to remain more responsive to text control because it was trained while the confounders were externally supplied (Goyal et al., 23 Oct 2025).

A second misconception is that the method establishes formal disentanglement. The paper presents an optimization-pressure argument and empirical evidence, but it does not present a formal causal proof or a representation analysis demonstrating factor separation inside the adapter. The supported claim is narrower: SR-LoRA and SR-CN improve several metrics over vanilla IP-Adapter in face and body personalization on the reported internal dataset and FLUX-based setup (Goyal et al., 23 Oct 2025).

The method’s scope is also constrained by what can be extracted or approximated as a confounder. Body pose is straightforwardly routed through ControlNet using a body pose estimator, and facial pose/expression are plausibly routed through automatically derived pose/expression maps. Distribution shift can be approximated by a LoRA trained on the finetuning dataset. More abstract nuisances, such as subtle identity-correlated style, camera optics, or composition, are harder to isolate. The paper therefore notes that rerouting may help less, or require more elaborate auxiliary modules, when target and confounder are inherently intertwined or when the confounder-extraction pipeline is noisy (Goyal et al., 23 Oct 2025).

The empirical evidence is likewise concentrated in a specific regime. Experiments focus on encoder-based adapter training, specifically IP-Adapter, with FLUX.1 [Dev] as the frozen backbone. The authors state that the method should apply more broadly, including LoRA training for style adapters or other modular fine-tuning setups, but this is not experimentally established in the paper. The strongest evidence is therefore for IP-Adapter plus either a LoRA or ControlNet shortcut in face and full-body personalization (Goyal et al., 23 Oct 2025).

A plausible implication is that the method’s broader significance lies less in any one auxiliary architecture than in its training principle: when a reconstruction objective encourages an adapter to absorb both target and nuisance attributes, the nuisance attributes can be rerouted into explicit, frozen side paths so that the main adapter specializes on what should remain after those confounds are explained away. Within the current evidence base, that principle is demonstrated most concretely for identity personalization, pose/expression leakage, and dataset/domain shift (Goyal et al., 23 Oct 2025).