Context Guided Transformer (CGT)

Updated 7 July 2026

Context Guided Transformer (CGT) is a design that explicitly separates the decoder’s source and target contexts for controlled evidence fusion.
It employs a gated fusion mechanism using a dimension-wise sigmoid gate, with regularization guided by PMI-based supervision.
Empirical studies in machine translation reveal that regularized context gating improves BLEU scores and balances context contribution with minimal overhead.

Searching arXiv for the primary paper and closely related uses of the term to ground the article. Context Guided Transformer (CGT) denotes a Transformer design in which distinct internal context streams are explicitly identified and their contribution to prediction is dynamically controlled, rather than being left to implicit residual mixing. In machine translation, the canonical CGT-style formulation is the decoder modification introduced in “Regularized Context Gates on Transformer,” which treats the target-prefix representation and the source-conditioned representation as separate evidence sources, combines them through a dimension-wise gate, and regularizes that gate with automatically generated supervision derived from pointwise mutual information (PMI) (Li et al., 2019). In this usage, “context” does not mean discourse context; it refers specifically to the source and target evidence streams inside the decoder.

1. Decoder-level definition of context

The CGT interpretation begins from the standard autoregressive translation factorization

$P(\mathbf{y}\mid \mathbf{x}) = \prod_{i=1}^{|\mathbf{y}|} P(y_i \mid y_{<i}, \mathbf{x}) = \prod_{i=1}^{|\mathbf{y}|} P(y_i \mid h_i^L),$

where $h_i^L$ is the decoder representation at target position $i$ in the top decoder layer $L$ (Li et al., 2019).

Within this formulation, the paper makes a precise distinction between two decoder-side contexts at layer $l$ and position $i$ . The first is the target context, defined as information derived only from the target prefix $y_{<i}$ . The second is the source context, defined as information derived from the encoded source sentence $\mathbf{x}$ , conditioned on the target-context query. This separation is central because, in a standard Transformer decoder, masked self-attention, encoder–decoder attention, residual addition, and normalization jointly entangle source and target information, leaving no explicit mechanism to say that a particular next word should rely more on source evidence or more on target-side history.

The paper motivates this decomposition by arguing that context-control failures can arise when the decoder over-trusts target-side continuation patterns and produces fluent but inadequate translations. Its illustrative example is a Chinese phrase translated as “play golf” instead of “play soccer,” attributed to excessive reliance on target context rather than source evidence. This framing is specific to machine translation, where faithfulness to the source and fluency from the target-side LLM are naturally competing pressures (Li et al., 2019).

2. Context decomposition inside the Transformer decoder

The main technical step is to define source and target contexts recursively inside each decoder layer. Let $y_{<i}^{\,l-1}$ be the previous-layer target-prefix representation, with base case $y_{<i}^{\,0}$ equal to target embeddings. Then, for target position $h_i^L$ 0,

$h_i^L$ 1

$h_i^L$ 2

Here, $h_i^L$ 3 denotes multi-head attention using $h_i^L$ 4 as query and $h_i^L$ 5 as key/value, $h_i^L$ 6 is layer normalization, and $h_i^L$ 7 is the residual network. Under this definition, $h_i^L$ 8 is the masked-self-attention output over prior target-side states, and $h_i^L$ 9 is the cross-attention output obtained by querying the encoder top-layer representation $i$ 0 with $i$ 1 (Li et al., 2019).

This decomposition is important because it replaces the informal notion of “context” with two operationally identifiable streams inside the decoder stack. In recurrent NMT, analogous quantities are comparatively easy to isolate as the decoder hidden state and the attention context vector. In the Transformer, by contrast, the architecture is more structured and the streams are tightly coupled across sublayers. A CGT, in this sense, is not merely a Transformer with an extra gate; it is a Transformer in which the relevant evidence streams are first made explicit and only then recombined.

3. Gated fusion and architectural modification

Once $i$ 2 and $i$ 3 are defined, the decoder is turned into a CGT by inserting a gate before the feed-forward sublayer. The gate at layer $i$ 4, position $i$ 5, is

$i$ 6

where $i$ 7 is a feed-forward network, $i$ 8 denotes concatenation, and $i$ 9 is the sigmoid. The gate is computed jointly from source and target contexts, and it is dimension-wise rather than scalar (Li et al., 2019).

The gated fusion is

$L$ 0

where $L$ 1 is element-wise multiplication. If a gate dimension is near $L$ 2, that dimension is driven more by source context; if it is near $L$ 3, it is driven more by target context. The paper interprets each gate component as inducing a probabilistic preference for whether the next generated word is mainly contributed by the source sentence or by the target-side prefix.

Architecturally, this replaces the ordinary additive mixing of source and target streams with a highway-style gated combination. The paper explicitly remarks that the model reduces to the standard Transformer if

$L$ 4

is replaced with the ungated sum

$L$ 5

The resulting CGT is therefore best understood as a decoder in which source/target fusion is controlled rather than implicit.

4. Regularized gate learning and PMI supervision

A major empirical finding is that inserting gates alone is not sufficient. Gates learned from scratch tend to develop a context imbalance bias toward target-side context. The paper quantifies this with a mean gate value of $L$ 6 for unconstrained context gates on MT08, below $L$ 7, indicating more frequent preference for target context than source context (Li et al., 2019).

To reduce this bias, the training objective introduces a latent binary label $L$ 8 for each target word $L$ 9, where $l$ 0 means the word is more attributable to the source sentence and $l$ 1 means it is more attributable to the target prefix. The full objective is

$l$ 2

This hinge-like regularizer does not force exact gate values; it only encourages each gate to lie on the correct side of $l$ 3.

The supervision signal $l$ 4 is generated automatically using PMI. For word pair $l$ 5,

$l$ 6

where $l$ 7 and $l$ 8 are unigram counts, $l$ 9 is co-occurrence count, and $i$ 0 is the normalization constant over all word pairs. The paper computes PMI in a bilingual scenario between target and source words, and in a monolingual scenario between a target word and earlier target words. The final label is

$i$ 1

This means a target token is treated as source-driven if its strongest source-side lexical association exceeds its strongest target-prefix association. The paper reports that a max strategy works about as well as averaging and retains the max formulation. Translation loss and gate-regularization loss are optimized jointly, with the best reported setting $i$ 2 and fairly robust performance for values from $i$ 3 to $i$ 4 (Li et al., 2019).

5. Empirical behavior, diagnostics, and limitations

The evaluation covers NIST Chinese $i$ 5English, WMT14 English $i$ 6German, IWSLT14 German $i$ 7English, and IWSLT17 French $i$ 8English, all with BPE segmentation and implementation on THUMT (Li et al., 2019). The main result is that Context Gates alone yield only modest gains, whereas Regularized Context Gates produce consistent improvements averaging about +1.0 BLEU over the baseline Transformer.

Dataset	Transformer	Regularized gates
ZH $i$ 9EN MT05/06/08	46.9 / 47.4 / 38.3	47.7 / 48.3 / 39.7
EN $y_{<i}$ 0DE	27.4	28.1
DE $y_{<i}$ 1EN	32.2	33.0
FR $y_{<i}$ 2EN	36.8	38.3

The paper also performs a forced-decoding error analysis. A translation error is defined when the gold token has lower probability than the model’s argmax prediction, and a context selection error is defined when the PMI-based source/target attribution of the gold token differs from that of the model prediction. On ZH $y_{<i}$ 3EN MT08, the reported values are: Transformer FER 40.5 and CER 13.8; Context Gates FER 40.5 and CER 13.7; Regularized Context Gates FER 40.0 and CER 13.4. The paper interprets this as evidence that roughly one-third of translation errors are related to context-selection errors, and that regularized gates reduce both overall translation errors and context-selection errors (Li et al., 2019).

Interpretability evidence also comes from gate statistics. On MT08, the unconstrained gate has mean 0.38 and variance 0.10, whereas the regularized version has mean 0.51 and variance 0.13. The stated interpretation is that regularization changes gate behavior toward a more balanced source/target allocation rather than merely adding capacity. Layerwise analysis further shows that regularizing any single decoder layer helps a little, but regularizing all decoder layers works best, with nearly no computation overhead. The paper also notes that plain Transformer context gates do not notably improve long-sentence translation, whereas the regularized version improves both short and long sentences.

The limitations are explicit. Raw context-gate insertion alone is weak; the source/target supervision $y_{<i}$ 4 is a PMI heuristic rather than ground truth; the improvement in explicit context-selection error is real but not huge; and the model introduces extra parameters, with the ZH $y_{<i}$ 5EN system growing from 74M to 92M parameters. These observations make the paper’s main lesson narrower than “gating helps”: it is regularized source/target gating that defines the effective CGT behavior in this formulation.

The term “Context Guided Transformer” is not reserved for a single architecture. In the narrow machine translation sense, it most naturally refers to explicit source/target disentangling plus gated fusion inside the Transformer decoder (Li et al., 2019). In the broader literature, however, closely related phrases denote a wider design family in which context is made explicit and then used to steer transformer computation.

Several later works fit that broader pattern while differing mechanistically. “Context-Guided Spatio-Temporal Video Grounding” uses a DETR-like multimodal Transformer whose decoder is guided by stage-wise mined instance context from appearance and motion features (Gu et al., 2024). “The Context-Ready Transformer” makes each token context-ready before entering the main block by combining the previous position’s block output with the current token embedding, yielding a recurrent, past-only pre-contextualization scheme rather than source/target gating (Godavarti, 25 Jun 2026). “Contextually Guided Transformers via Low-Rank Adaptation” uses a dedicated context-summary stream to generate low-rank weight updates for later layers, effectively encoding context into the model’s weights rather than into prompt tokens (Zhmoginov et al., 6 Jun 2025).

At the same time, acronym usage is highly ambiguous. CGT can also mean “Cell Graph Transformer” in nuclei classification (Lou et al., 2024), “Counterfactual Graph Transformer” in traffic flow prediction (Yang et al., 2023), “Cross-modal Clinical Graph Transformer” in ophthalmic report generation (Li et al., 2022), or “Contextual Graph Transformer” for engineering document information extraction (Reddy et al., 4 Aug 2025). Related but nonidentical phrases also appear in non-Transformer or hybrid settings, such as the CNN-based “Guided Context Gating” module for retinal fundus images, which explicitly is not a Context Guided Transformer architecture (Cherukuri et al., 2024).

A plausible implication is that “CGT” is best treated as a descriptive architectural label rather than a uniquely identifying acronym. In that descriptive sense, the defining idea is explicit context identification followed by guided fusion or modulation. The machine-translation CGT of (Li et al., 2019) remains a particularly clear instance because it formalizes “context” as the target self-attention state and the source cross-attention state, inserts a vector gate

$y_{<i}$ 6

and regularizes that gate with PMI-derived source/target labels. That combination makes it a canonical example of controlled context fusion inside a Transformer decoder.