Adaptive Context Enrichment

• Adaptive Context Enrichment is a mechanism that dynamically fuses contextual information via self-attention, enabling models to integrate reconstruction and editing cues without destructive feature replacement.
• It operates by concatenating Key-Value pairs from parallel editing and reconstruction paths, preserving background structure and semantic details while guiding dynamic token fusion.
• The approach is selectively deployed in vital layers and timesteps, balancing computational cost with improved performance across applications like video editing, translation, and adaptive agent memory.

Adaptive Context Enrichment denotes a family of mechanisms that adapt how contextual information is exposed to a model at inference or decision time. In its most explicit formulation, it is the central guidance mechanism of ContextFlow for training-free video object editing with Diffusion Transformers (DiTs): instead of hard-replacing internal features, it enriches the editing path’s self-attention context by concatenating Key-Value pairs from parallel reconstruction and editing trajectories, allowing the pre-trained attention mechanism to fuse background and edited content dynamically (Chen et al., 22 Sep 2025). The same expression, or closely related context-adaptive formulations, also appears in clinical trial design, conversational question answering, agent memory, document translation, and personalized rewriting, where it denotes dynamic selection, restructuring, or weighting of contextual information under task-specific constraints (Maleyeff et al., 10 Mar 2026, Perera et al., 22 Sep 2025, Vijayvargiya et al., 24 Sep 2025).

1. ContextFlow and the canonical formulation

In ContextFlow, Adaptive Context Enrichment (ACE) is introduced for training-free video object editing with DiTs, with the immediate goal of supporting object insertion, swapping, and deletion while preserving original background structure and temporal consistency (Chen et al., 22 Sep 2025). The method is positioned against earlier Plug-and-Play feature injection schemes developed largely for U-Nets, which reconstruct the source video and then hard-replace features in the editing trajectory. ContextFlow identifies two failure modes of that strategy in DiTs: contextual conflict in attention, and overwriting of the model’s generative flexibility (Chen et al., 22 Sep 2025).

The contextual conflict arises because editing queries encode the new semantic concept, whereas hard-replaced keys and values may encode only the source-video background. In the paper’s example, queries corresponding to “a Pikachu is floating on the sea” are forced to attend to reconstruction features that encode “sea surface with no Pikachu,” which can suppress the edit or produce artifacts (Chen et al., 22 Sep 2025). The second problem is architectural: replacement discards the editing path’s own context, and in DiTs—described as having more homogeneous layers than U-Nets—naive layer-wise replacement can destabilize spatial and semantic processing (Chen et al., 22 Sep 2025).

ACE is therefore framed as a non-destructive alternative. ContextFlow distinguishes between “what to edit” and “where to edit”: ACE specifies the contextual information that should influence each token, while a separate vital-layer analysis determines the most effective DiT blocks for applying that enrichment (Chen et al., 22 Sep 2025). This division is central to the framework’s design and explains why ACE is not merely a replacement rule, but a context-selection mechanism embedded in self-attention.

2. Dual-path self-attention enrichment

ContextFlow builds a dual-path denoising process on a pre-trained image-to-video DiT, Wan 2.1 I2V. Both paths start from the same high-fidelity inverted noise anchor $\mathbf{z}_1$, and the DiT weights are shared; only the conditions differ (Chen et al., 22 Sep 2025). The reconstruction path is conditioned on the original first frame and a null text prompt, with the objective of faithfully reconstructing the source video and supplying high-fidelity source context. The editing path is conditioned on the edited first frame and target text prompt, with the objective of synthesizing the edited video while respecting motion and background (Chen et al., 22 Sep 2025).

Without ACE, a standard self-attention block in the editing path is

$$\text{SelfAttn}^{\text{edit}}_{t,l} = \text{softmax}\left( \frac{ Q^{\text{edit}}_{t,l}(K^{\text{edit}}_{t,l})^\top }{ \sqrt{d} } \right) V^{\text{edit}}_{t,l},$$

where $Q,K,V \in \mathbb{R}^{N \times d}$ denote queries, keys, and values over $N$ spatio-temporal tokens with feature dimension $d$ (Chen et al., 22 Sep 2025).

ACE modifies only the context available to the editing queries. It forms augmented keys and values by concatenating the editing-path and reconstruction-path Key-Value sets along the token dimension,

$$K_{\text{aug}} = \text{Concat}\big[K^{\text{edit}}_{t,l},\, K^{\text{res}}_{t,l}\big], \qquad V_{\text{aug}} = \text{Concat}\big[V^{\text{edit}}_{t,l},\, V^{\text{res}}_{t,l}\big],$$

and then computes

$$\text{SelfAttn}^{\text{ACE}}_{t,l} = \text{softmax}\left( \frac{ Q^{\text{edit}}_{t,l} K_{\text{aug}}^\top }{ \sqrt{d} } \right) V_{\text{aug}}.$$

Each editing query thus sees both its own evolving generation context and the reconstruction context that encodes stable background, camera motion, and original object placement (Chen et al., 22 Sep 2025).

The mechanism is termed adaptive because the fusion is not governed by explicit masks, fixed rules, or new gating parameters. Both $(K^{\text{edit}},V^{\text{edit}})$ and $(K^{\text{res}},V^{\text{res}})$ remain available, and the pre-trained attention weights determine, per token and per timestep, how much to draw from each source (Chen et al., 22 Sep 2025). In insertion, background-area queries can attend more strongly to reconstruction context while inserted-object queries rely more on editing context; in swapping, the model can preserve pose and motion from reconstruction while changing appearance via the edit path; in deletion, queries at the removed object’s location can reuse reconstruction background tokens to fill in scenery without reintroducing the object (Chen et al., 22 Sep 2025).

3. Temporal scheduling, inversion, and vital-layer selection

ACE in ContextFlow is not deployed in isolation. The full framework has three components: high-order Rectified Flow inversion, Adaptive Context Enrichment, and vital-layer analysis via the Guidance Responsiveness Metric (Chen et al., 22 Sep 2025). The inversion stage establishes a nearly reversible latent anchor by using a second-order Rectified Flow solver rather than first-order DDIM inversion. The solver updates latent states as

$$z_{t_{i+1}} = z_{t_i} + (t_{i+1}-t_i)\,v_\theta(z_{t_i}, t_i) + \frac{1}{2}(t_{i+1}-t_i)^2 v_\theta^{(1)}(z_{t_i}, t_i),$$

and inversion is applied on VAE latents of the source video under the original first frame and null text prompt, producing $$\text{SelfAttn}^{\text{edit}}_{t,l} = \text{softmax}\left( \frac{ Q^{\text{edit}}_{t,l}(K^{\text{edit}}_{t,l})^\top }{ \sqrt{d} } \right) V^{\text{edit}}_{t,l},$$0 as a high-fidelity noise anchor (Chen et al., 22 Sep 2025). The paper argues that this removes ambiguity between inversion error and edit effects, which is especially important for precise object-level editing.

The temporal deployment of ACE is also restricted. ACE is active only in the first half of denoising, at timesteps $$\text{SelfAttn}^{\text{edit}}_{t,l} = \text{softmax}\left( \frac{ Q^{\text{edit}}_{t,l}(K^{\text{edit}}_{t,l})^\top }{ \sqrt{d} } \right) V^{\text{edit}}_{t,l},$$1 in the normalized schedule (Chen et al., 22 Sep 2025). Early steps are used to influence coarse structure, semantics, layout, and motion integration; later steps are left to native DiT dynamics for detail refinement, reducing over-constraint (Chen et al., 22 Sep 2025).

The complementary question—where to inject ACE—is handled by the Guidance Responsiveness Metric,

$$\text{SelfAttn}^{\text{edit}}_{t,l} = \text{softmax}\left( \frac{ Q^{\text{edit}}_{t,l}(K^{\text{edit}}_{t,l})^\top }{ \sqrt{d} } \right) V^{\text{edit}}_{t,l},$$2

computed by comparing one-step denoising outputs with and without ACE applied only at layer $$\text{SelfAttn}^{\text{edit}}_{t,l} = \text{softmax}\left( \frac{ Q^{\text{edit}}_{t,l}(K^{\text{edit}}_{t,l})^\top }{ \sqrt{d} } \right) V^{\text{edit}}_{t,l},$$3 (Chen et al., 22 Sep 2025). High $$\text{SelfAttn}^{\text{edit}}_{t,l} = \text{softmax}\left( \frac{ Q^{\text{edit}}_{t,l}(K^{\text{edit}}_{t,l})^\top }{ \sqrt{d} } \right) V^{\text{edit}}_{t,l},$$4 indicates that the layer is responsive to contextual enrichment. Empirically, the DiT exhibits three responsive zones: shallow layers around $$\text{SelfAttn}^{\text{edit}}_{t,l} = \text{softmax}\left( \frac{ Q^{\text{edit}}_{t,l}(K^{\text{edit}}_{t,l})^\top }{ \sqrt{d} } \right) V^{\text{edit}}_{t,l},$$5–$$\text{SelfAttn}^{\text{edit}}_{t,l} = \text{softmax}\left( \frac{ Q^{\text{edit}}_{t,l}(K^{\text{edit}}_{t,l})^\top }{ \sqrt{d} } \right) V^{\text{edit}}_{t,l},$$6, mid layers around $$\text{SelfAttn}^{\text{edit}}_{t,l} = \text{softmax}\left( \frac{ Q^{\text{edit}}_{t,l}(K^{\text{edit}}_{t,l})^\top }{ \sqrt{d} } \right) V^{\text{edit}}_{t,l},$$7–$$\text{SelfAttn}^{\text{edit}}_{t,l} = \text{softmax}\left( \frac{ Q^{\text{edit}}_{t,l}(K^{\text{edit}}_{t,l})^\top }{ \sqrt{d} } \right) V^{\text{edit}}_{t,l},$$8, and deep layers around $$\text{SelfAttn}^{\text{edit}}_{t,l} = \text{softmax}\left( \frac{ Q^{\text{edit}}_{t,l}(K^{\text{edit}}_{t,l})^\top }{ \sqrt{d} } \right) V^{\text{edit}}_{t,l},$$9–$Q,K,V \in \mathbb{R}^{N \times d}$0 (Chen et al., 22 Sep 2025). Task dependence is explicit: insertion is most responsive in early layers, swapping in deep layers, and deletion in both mid and deep layers (Chen et al., 22 Sep 2025). ContextFlow therefore applies ACE only in the top-$Q,K,V \in \mathbb{R}^{N \times d}$1 responsive layers, with $Q,K,V \in \mathbb{R}^{N \times d}$2 reported as optimal, during the first half of timesteps (Chen et al., 22 Sep 2025).

A recurrent misconception is that adaptive enrichment means “inject context everywhere.” ContextFlow states the opposite: injecting ACE at every layer is computationally heavy and conceptually harmful because it over-constrains the DiT and degrades generation (Chen et al., 22 Sep 2025). The method’s adaptivity therefore lies not only in token-wise fusion, but also in selective temporal and layer-wise deployment.

4. Task behavior, temporal coherence, and operating characteristics

ContextFlow evaluates ACE on object insertion, swapping, and deletion. For insertion, the edited first frame contains a new object, while the reconstruction path retains original motion and background. The paper reports that ACE at shallow, early-responsive layers yields realistic object integration, correct trajectory and interactions, and preserved background across frames; Table 1 reports higher CLIP-I and DINO-I for identity than AnyV2V and VACE, strong alignment and aesthetic scores, and PSNR/SSIM for background that are comparable or better than baselines (Chen et al., 22 Sep 2025).

For swapping, the reconstruction path preserves pose, geometry, and motion, while ACE is concentrated at deep layers where the Guidance Responsiveness Metric peaks. The reported outcome is that the swapped object moves coherently with the original trajectory while adopting the new identity, with best or near-best CLIP-Score, CLIP-I, and DINO-I and strong temporal smoothness and dynamics (Chen et al., 22 Sep 2025). For deletion, ACE is applied at mid and deep layers so that semantic removal and plausible background filling can be coordinated; the paper reports high smoothness and dynamics, good reconstruction quality on unedited regions measured by PSNR/SSIM, and fewer artifacts than AnyV2V and VACE, which may hallucinate wrong fills or leave shadows (Chen et al., 22 Sep 2025).

Temporal coherence is attributed to three design choices: both paths share the same inverted noise $Q,K,V \in \mathbb{R}^{N \times d}$3, the reconstruction path acts as a temporal anchor whose Key-Value tensors encode original motion and global spatio-temporal structure, and ACE allows background tokens to attend heavily to reconstruction context across frames (Chen et al., 22 Sep 2025). Qualitative comparisons in the paper are described as showing smooth object motion, stable background geometry, and absence of frame-to-frame flicker or geometry drift observed in naive DiT-guided editing (Chen et al., 22 Sep 2025).

The framework remains training-free, but its implementation cost is substantial. It operates by running the pre-trained DiT twice in parallel, extracting intermediate $Q,K,V \in \mathbb{R}^{N \times d}$4, concatenating Keys and Values at selected layers and timesteps, and feeding them back through the same attention mechanism (Chen et al., 22 Sep 2025). Dual-path sampling doubles forward cost; the reported memory footprint is approximately 120 GB VRAM, using two A800 GPUs at roughly 60 GB per GPU, for 81 frames at 480p (Chen et al., 22 Sep 2025). Inversion and generation both use 50 steps with the RF-Solver, and restricting ACE to top-$Q,K,V \in \mathbb{R}^{N \times d}$5 layers and the first half of timesteps reduces overhead compared with applying it everywhere (Chen et al., 22 Sep 2025).

5. Broader research landscape

Outside video editing, related formulations appear across several research areas. In document-level neural machine translation, a light predictor selects explicit context options—previous sentence, next sentence, both, or none—rather than always using fixed windows or full context; the method is trained with Gumbel–Softmax-weighted losses and reports gains of up to 1.99 BLEU points (Zhang, 2021). In text classification, Adaptive Region Embedding uses a metanetwork to generate a context matrix for each region, replacing vocabulary-indexed local context units with instance-specific Adaptive Context Units; it improves on 7 of 8 datasets and reduces context-unit parameters to less than 5% of the earlier LCU formulation (Xiang et al., 2019). In context-aware language modeling, adaptation is extended beyond input concatenation to hidden-layer additive and multiplicative adaptation and output-layer low-rank plus hashed bias terms, improving perplexity and classification across Reddit, Twitter, and SCOTUS corpora (Jaech et al., 2017). In reinforcement learning, the Decision Adapter uses a hypernetwork to generate adapter weights from context, producing a context-aware policy that is more robust to distractor variables than concatenation and several alternative methods (Beukman et al., 2023). In domain generalization and test-time adaptation, CONTXT steers hidden features with additive transforms of the form $Q,K,V \in \mathbb{R}^{N \times d}$6 and reports average lifts of roughly 8–10% on target domains together with controllable sentiment and persona steering in LLMs (Delanois et al., 6 Apr 2026). A different weight-space formulation appears in Bridge-Mode Connectivity, where context is identified with a continuous path coordinate and model parameters are sampled along a low-loss curve or plane tuned to risk profiles, corruption severity, or context-dependent distribution shift (Drenkow et al., 2022).

A second cluster of work treats adaptive enrichment as context management rather than internal feature modulation. In on-device agents, adaptive context management uses a structured Context State Object, token-efficient tool schemas, and just-in-time schema passing; the reported outcome is more than a 6-fold reduction in initial system prompt context and a 10- to 25-fold reduction in context growth rate, depending on interaction verbosity (Vijayvargiya et al., 24 Sep 2025). In conversational question answering, Adaptive Context Management composes context from unmodified recent turns, summaries of older turns, and entity extraction from the oldest turns, and reports improved F1, ROUGE-L, ROUGE-1, and BLEU over a pipeline baseline across six backbones (Perera et al., 22 Sep 2025). In the agentic-memory literature, one ACE framework treats contexts as evolving playbooks of bullets updated by Generator, Reflector, and Curator modules, reporting +10.6% on agents and +8.6% on finance while reducing latency and rollout cost (Zhang et al., 6 Oct 2025). Another ACE, the Adaptive Context Elasticizer, stores both raw and abstract forms of each trajectory step and dynamically assigns each step the elastic type Raw, Abs, or Drop at each decision step, consistently outperforming truncation and summarization baselines across ReAct, DeepAgent, WebThinker, and MiroFlow (Liao et al., 30 Jun 2026).

A third cluster uses the terminology in statistical decision systems. In Bayesian adaptive enrichment trials, context is a biomarker-defined subgroup, and enrichment means learning an effective subspace $Q,K,V \in \mathbb{R}^{N \times d}$7 where posterior treatment benefit exceeds a threshold; with historical borrowing via a normalized power prior anchored on summary measures, the design reports improved power, earlier stopping, and reduced expected sample size relative to non-borrowing designs (Maleyeff et al., 10 Mar 2026). A related design uses Bayesian model averaging with free-knot B-splines to identify tailoring variables and treatment-sensitive subspaces from mixed binary and continuous biomarkers (Maleyeff et al., 2024). Another adaptive enrichment design uses a joint model of longitudinal ctDNA trajectories and time-to-event data, showing that biomarker-informed interim statistics improve subgroup identification and power while protecting familywise error in the strong sense (Burdon et al., 2023). In personalized story rewriting, context-aware narrative enrichment is contrasted with style-only adaptation: a pilot study reports that style-only rewriting yields only a 2.3% satisfaction gain, whereas context-enhanced rewriting yields 24.5%, motivating a benchmark, reward model, and GRPO-based rewriting policy (Cui et al., 27 May 2026).

This cross-domain usage does not imply a single shared algorithm. It does suggest, however, that the term increasingly denotes mechanisms that make contextual information selective, structured, reversible, or dynamically fused, rather than fixed, uniform, or destructively compressed.

6. Misconceptions, limitations, and significance

A common misconception is that adaptive enrichment simply means “adding more context.” Several papers argue the opposite. The document-level NMT work states that inappropriate context may harm translation performance (Zhang, 2021). ContextFlow states that injecting ACE at every layer is computationally heavy and conceptually harmful, and therefore selects only vital layers and early timesteps (Chen et al., 22 Sep 2025). In long-horizon agents, truncation and one-shot summarization are criticized not because they use too little context in the abstract, but because they are inflexible and irreversible; ACE-style elasticization keeps the raw information recoverable and changes only how it is exposed at a given step (Liao et al., 30 Jun 2026).

A second misconception is that adaptive enrichment is equivalent to summarization. Several systems reject that identification explicitly or in effect. ContextFlow does not summarize; it enriches self-attention context by concatenating reconstruction and editing Key-Value sets (Chen et al., 22 Sep 2025). The on-device agent framework does summarize conversational history into a Context State Object, but combines that with just-in-time schema passing and permanent-versus-ephemeral KV-cache management (Vijayvargiya et al., 24 Sep 2025). The agentic-context-engineering framework warns against context collapse caused by repeatedly rewriting an entire context and instead advocates incremental bullet-level updates (Zhang et al., 6 Oct 2025). These examples suggest that summarization is only one possible operator inside a broader family of adaptive context mechanisms.

The limitations are equally domain-specific. In ContextFlow, dual-path inference doubles forward cost and requires approximately 120 GB VRAM for 81-frame 480p generation (Chen et al., 22 Sep 2025). In Bayesian adaptive enrichment, optimistic historical bias can inflate Type I error, and the mapping from marginal historical summaries to subgroup-specific treatment parameters is not identifiable from summaries alone (Maleyeff et al., 10 Mar 2026). In StoryLens, the reward model may overfit benchmark-specific preference distributions and hallucination risks remain if reward and evaluation signals are imperfect (Cui et al., 27 May 2026). In agentic context engineering, adaptation quality depends on the Reflector and on the quality of execution feedback, and some tasks may not benefit from large, detailed contexts (Zhang et al., 6 Oct 2025). In the Adaptive Context Elasticizer, abstraction quality and auxiliary-model overhead remain practical concerns, even though the method is training-free (Liao et al., 30 Jun 2026).

Taken together, the literature supports a narrower and technically more precise understanding of Adaptive Context Enrichment than the phrase might suggest. It is not merely the use of context, and not merely longer context. In ContextFlow, it is a self-attention-level mechanism for non-destructive fusion of reconstruction and editing context in DiT-based video editing (Chen et al., 22 Sep 2025). In adjacent areas, it refers to structured selection, dynamic modulation, reversible exposure, or statistically principled enrichment of relevant context. This suggests that the enduring research question is not whether context should be used, but how models should expose, preserve, and condition on it without sacrificing fidelity, efficiency, or adaptability.