Cross-Decoder Gradient Blocking Explained

• Cross-decoder gradient blocking is defined as interventions that stop or distort gradients at the decoder interface, enabling controlled training behavior.
• It is applied in YOCO architectures via stop-gradient mechanisms and in watermarking defenses with adversarial gradient shaping, ensuring modularity and security.
• These techniques maintain forward-pass performance while blocking or altering backward signals, though they may introduce trade-offs in learning dynamics and adaptation.

Cross-decoder gradient blocking denotes the blocking, stopping, or deliberate distortion of gradients that traverse a decoder interface. In the available literature, the phrase arises in two technically distinct settings. In YOCO, a decoder–decoder language-model architecture, the relevant interface is the mapping from the self-decoder output $M=X^{L/2}$ to the shared global key–value cache $(\hat{K},\hat{V})$ consumed by the cross-decoder; the paper identifies this as the architectural locus where one might implement gradient blocking, but does not itself do so (Sun et al., 2024). In box-free model watermarking, Decoder Gradient Shield (DGS) and Decoder Gradient Shields (DGSs) explicitly protect a watermark decoder by manipulating the gradients that cross from the decoder into an external watermark remover, while preserving forward decoder behavior for legitimate verification (An et al., 28 Feb 2025, An et al., 17 Jan 2026). Taken together, these works show that cross-decoder gradient blocking can mean either stop-gradient style isolation between stacked decoder components or adversarial gradient shaping at a decoder API boundary.

1. Scope of the concept

In YOCO, the architecture is split into a self-decoder and a cross-decoder. The self-decoder produces a hidden representation $M$, and the cross-decoder reads a single shared global cache

$$\hat{K} = \operatorname{LN}(M)W_K,\qquad \hat{V}=\operatorname{LN}(M)W_V$$

through cross-attention. The paper states that this mapping is the place “where one might implement ‘cross-decoder gradient blocking’,” but also states that YOCO “does not mention any explicit gradient blocking, stop-gradient, or freezing between self-decoder and cross-decoder” and that training is “standard end-to-end backpropagation through the entire computation graph” (Sun et al., 2024).

In the watermarking literature, the expression is operational rather than hypothetical. A protected model $\mathbb{M}$, watermark encoder $\mathbb{E}$, and watermark decoder $\mathbb{D}$ are exposed through APIs, while an attacker trains a remover $\mathbb{R}$ by backpropagating through $\mathbb{D}$. The gradient

$\frac{\partial\mathcal{L}_{\text{Removal}}{\partial \mathbb{R}(Y)} =2(Z-W_0)^\top \frac{\partial Z}{\partial \mathbb{R}(Y)},\qquad Z=\mathbb{D}[\mathbb{R}(Y)]$

is the cross-decoder training signal that DGS is designed to block or mislead (An et al., 28 Feb 2025).

A plausible synthesis is that cross-decoder gradient blocking names a family of interface-level interventions on decoder-mediated supervision paths: either by enforcing $(\hat{K},\hat{V})$0 at an internal boundary, or by replacing truthful decoder Jacobians with controlled, task-preserving but optimization-hostile surrogates.

2. The internal decoder boundary in YOCO

YOCO is a decoder–decoder architecture with $(\hat{K},\hat{V})$1 blocks in total: the first $(\hat{K},\hat{V})$2 blocks form the self-decoder and the remaining $(\hat{K},\hat{V})$3 blocks form the cross-decoder. Both parts are causal and used autoregressively, so the overall model behaves like a decoder-only Transformer (Sun et al., 2024).

The self-decoder applies efficient self-attention (ESA) and SwiGLU: $(\hat{K},\hat{V})$4 ESA is instantiated as either sliding-window attention (SWA) or multi-head gated retention (MHGR). The architectural purpose is to maintain constant inference KV memory in the self-decoder, while still enabling a global memory for upper layers.

After $(\hat{K},\hat{V})$5 layers, the model computes shared global key–value tensors from $(\hat{K},\hat{V})$6: $(\hat{K},\hat{V})$7 Every cross-decoder layer then uses its own query projection but the same $(\hat{K},\hat{V})$8: $(\hat{K},\hat{V})$9 The decisive design point is that $M$0 are shared across all cross-decoder layers and cached only once (Sun et al., 2024).

This interface has two distinct roles. Forward-pass-wise, it is the mechanism behind “You Only Cache Once”: KV memory goes from $M$1 to $M$2, and prefill can early exit after the self-decoder because the cross-decoder depends on the prompt only through the shared cache. Backward-pass-wise, it is the only route by which language-model loss from the upper half of the stack reaches the lower half through cross-attention. The early-exit prefill optimization is explicitly a “forward-only system optimization” and “does not change the computation graph during training,” so it is not itself gradient blocking (Sun et al., 2024).

3. Mathematical meaning of blocking at a decoder interface

Under standard YOCO training, the loss $M$3 depends on the final hidden states $M$4, and gradients propagate not only through the cross-decoder’s own parameters but also through the shared cache into the self-decoder. For a single cross-decoder layer,

$M$5

and the induced gradients satisfy

$M$6

Since

$M$7

these terms backpropagate into $M$8, $M$9, $$\hat{K} = \operatorname{LN}(M)W_K,\qquad \hat{V}=\operatorname{LN}(M)W_V$$0, and then the entire self-decoder stack (Sun et al., 2024).

The paper gives a precise mathematical interpretation of blocking at this boundary. If one defines

$$\hat{K} = \operatorname{LN}(M)W_K,\qquad \hat{V}=\operatorname{LN}(M)W_V$$1

and feeds $$\hat{K} = \operatorname{LN}(M)W_K,\qquad \hat{V}=\operatorname{LN}(M)W_V$$2 into the cross-decoder, then the loss arising from cross-decoder outputs enforces

$$\hat{K} = \operatorname{LN}(M)W_K,\qquad \hat{V}=\operatorname{LN}(M)W_V$$3

In consequence, no gradient from cross-decoder layers reaches $$\hat{K} = \operatorname{LN}(M)W_K,\qquad \hat{V}=\operatorname{LN}(M)W_V$$4, $$\hat{K} = \operatorname{LN}(M)W_K,\qquad \hat{V}=\operatorname{LN}(M)W_V$$5, $$\hat{K} = \operatorname{LN}(M)W_K,\qquad \hat{V}=\operatorname{LN}(M)W_V$$6, or self-decoder parameters $$\hat{K} = \operatorname{LN}(M)W_K,\qquad \hat{V}=\operatorname{LN}(M)W_V$$7 through cross-attention. The only remaining path to self-decoder updates would have to come from “some other loss or path,” such as an auxiliary loss directly applied to $$\hat{K} = \operatorname{LN}(M)W_K,\qquad \hat{V}=\operatorname{LN}(M)W_V$$8 or self-decoder outputs (Sun et al., 2024).

This is the canonical internal form of cross-decoder gradient blocking: the forward signal crosses the boundary, but the backward signal does not.

4. Variants inside decoder–decoder architectures

The YOCO paper does not propose or evaluate gradient blocking, but its architecture exposes several natural intervention points (Sun et al., 2024).

A hard stop-gradient at shared KV detaches both $$\hat{K} = \operatorname{LN}(M)W_K,\qquad \hat{V}=\operatorname{LN}(M)W_V$$9 and $\mathbb{M}$0 after projection: $\frac{\partial\mathcal{L}_{\text{Removal}}{\partial \mathbb{R}(Y)} =2(Z-W_0)^\top \frac{\partial Z}{\partial \mathbb{R}(Y)},\qquad Z=\mathbb{D}[\mathbb{R}(Y)]$3 This makes the cross-decoder learn to use a fixed memory representation. The stated conceptual advantages are “Modularity” and “Stability,” while the stated downsides are “Under-training $\mathbb{M}$1,” “Mismatch with inference,” and “Lost end-to-end adaptation.” The text further states that such blocking “would likely hurt the scaling curves (Figure 1) and long-context behaviors (Figure 2)” unless compensated by separate training.

A second variant blocks gradients into $\mathbb{M}$2 but not into $\mathbb{M}$3: $\mathbb{M}$4 Here the cross-decoder can still adapt the key–value projections while the self-decoder remains fixed. The paper presents this as suitable for training a cross-decoder “on top for a new domain/task without changing the memory encoder.”

A third variant blocks gradients only into lower self-decoder layers by freezing or zeroing gradients below a cutoff $\mathbb{M}$5. A fourth combines blocking with separate losses: $\mathbb{M}$6 This allows $\mathbb{M}$7 to train the cross-decoder while $\mathbb{M}$8 trains the self-decoder directly (Sun et al., 2024).

The paper explicitly relates these interventions to standard Transformer practice. Blocking at the YOCO boundary is analogous to freezing lower layers in a Transformer or training only upper layers, but the structure is “conceptually closer to” freezing an encoder in an encoder–decoder model, because all upper layers read the same blocked shared memory rather than receiving direct per-layer lower-stack activations. This structural asymmetry makes the effect more global than ordinary layer freezing (Sun et al., 2024).

5. Decoder Gradient Shield as external cross-decoder gradient blocking

In box-free watermarking, the cross-decoder boundary is not internal to one model; it lies between a private watermark decoder $\mathbb{M}$9 and an external watermark remover $\mathbb{E}$0. The protected pipeline consists of a victim model $\mathbb{E}$1, a watermark encoder $\mathbb{E}$2, and a watermark decoder $\mathbb{E}$3. With $\mathbb{E}$4 frozen, $\mathbb{E}$5 and $\mathbb{E}$6 are trained jointly using

$\mathbb{E}$7

where

$\mathbb{E}$8

and

$\mathbb{E}$9

An attacker queries the APIs, obtains watermarked images $\mathbb{D}$0, and trains a remover $\mathbb{D}$1 with

$\mathbb{D}$2

where

$\mathbb{D}$3

The exposed decoder gradients are the vulnerability: an unprotected $\mathbb{D}$4 gives the attacker a learning signal that allows $\mathbb{D}$5 to learn the inverse of $\mathbb{D}$6 (An et al., 28 Feb 2025).

DGS inserts a protection layer inside the decoder API and replaces the raw output with a transformed output $\mathbb{D}$7. The core closed-form mapping is

$\mathbb{D}$8

where $\mathbb{D}$9 is positive definite and $\mathbb{R}$0. The final API rule is

$\mathbb{R}$1

This keeps outputs near $\mathbb{R}$2 for watermarked inputs but transforms the gradient by $\mathbb{R}$3, thereby reorienting and rescaling the signal seen by the attacker (An et al., 28 Feb 2025).

The later DGSs paper generalizes this into output-, input-, and layer-level shields (An et al., 17 Jan 2026).

Variant	Insertion point	Stated mechanism
DGS-O	Decoder output	Closed-form transform $\mathbb{R}$4
DGS-I	Decoder input	Orthogonality-based perturbation $\mathbb{R}$5
DGS-L	Intermediate layer	Perturb $\mathbb{R}$6 before the sub-decoder

For DGS-O, the Jacobian satisfies

$\mathbb{R}$7

so the attacker receives a systematically biased gradient rather than merely a noisy one. For DGS-I and DGS-L, the perturbation is chosen orthogonal to the removal-loss gradient so that the first-order Taylor term vanishes for benign behavior, while higher-order interference contaminates backpropagated gradients (An et al., 17 Jan 2026).

6. Empirical behavior, misconceptions, and limitations

A central misconception is that YOCO itself introduces cross-decoder gradient blocking. It does not. The paper states that training is end-to-end, and its reported performance—favorable scaling, strong long-context behavior, and near-perfect needle retrieval at 1M context length—comes from that unblocked training regime rather than from any stop-gradient mechanism (Sun et al., 2024). The text further states that heavy blocking “would likely reduce” Transformer equivalence and long-context performance unless additional pretraining or auxiliary objectives are introduced.

A second misconception is that DGS is merely random gradient noise. The papers distinguish it from naive gradient sign flipping or additive Gaussian noise. DGS-O implements a structured transformation $\mathbb{R}$8 in the simplified linearized sense, with small eigenvalues for norm reduction and, in the DGSs formulation, randomness injection through per-query sampling of the diagonal entries of $\mathbb{R}$9. The claim is not simply that gradients are obscured, but that they are reoriented, rescaled, and made unsuitable for convergence (An et al., 28 Feb 2025, An et al., 17 Jan 2026).

The watermarking results are explicit. Without defense, removal losses converge to extremely low values, and decoded outputs become nearly blank. With DGS-O, the attacker’s removal loss stays near $\mathbb{D}$0 rather than converging to $\mathbb{D}$1, and the defender’s internal true loss increases and settles at a high value. Under DGS-I and DGS-L, the loss decreases initially but then rises and stabilizes near $\mathbb{D}$2. The reported defense success rate is $\mathbb{D}$3 under all tested settings, while PSNR and MS-SSIM remain high (An et al., 17 Jan 2026).

The practical overhead is also quantified. DGS-O has complexity $\mathbb{D}$4 per query and runtime about $\mathbb{D}$5–$\mathbb{D}$6 ms; DGS-I has complexity $\mathbb{D}$7 and runtime about $\mathbb{D}$8 s; DGS-L at layer $\mathbb{D}$9 has complexity $\frac{\partial\mathcal{L}_{\text{Removal}}{\partial \mathbb{R}(Y)} =2(Z-W_0)^\top \frac{\partial Z}{\partial \mathbb{R}(Y)},\qquad Z=\mathbb{D}[\mathbb{R}(Y)]$0 and runtimes about $\frac{\partial\mathcal{L}_{\text{Removal}}{\partial \mathbb{R}(Y)} =2(Z-W_0)^\top \frac{\partial Z}{\partial \mathbb{R}(Y)},\qquad Z=\mathbb{D}[\mathbb{R}(Y)]$1 s for shallow, mid, and deep placements (An et al., 17 Jan 2026).

The limitations are equally specific. DGS addresses attacks that depend on the true decoder’s gradients; if an attacker trains a completely surrogate decoder without using those gradients, the shield “won’t apply.” The 2025 paper further notes possible circumventions through gradient sign flipping, attempted inversion

$\frac{\partial\mathcal{L}_{\text{Removal}}{\partial \mathbb{R}(Y)} =2(Z-W_0)^\top \frac{\partial Z}{\partial \mathbb{R}(Y)},\qquad Z=\mathbb{D}[\mathbb{R}(Y)]$2

or gradient-free attacks, but reports that simple recovery attempts perform poorly in experiments (An et al., 28 Feb 2025). This suggests that cross-decoder gradient blocking is strongest when the attack relies on exposed or estimable decoder Jacobians and weakest when the attacker can bypass that interface entirely.

A plausible general implication is that the term now covers two complementary design principles: interface isolation in stacked decoder architectures, and interface poisoning in decoder APIs. In both cases, the objective is identical: permit the forward use of decoder outputs while constraining the backward information available across the decoder boundary.