---
title: In-Context Generation Techniques
url: https://www.emergentmind.com/topics/in-context-generation
type: topic
---

# In-Context Generation Techniques

to=arxiv_search.search  大发时时彩怎么 无码不卡高清免费 code  {"query":"all:\"in-context generation\" OR title:\"In-Context Generation\"","max_results":10,"sort_by":"submittedDate","sort_order":"descending"} საქმე
to=arxiv_search.search  北京赛车前json code  {"query":"ti:\"In-Context\" AND (abs:generation OR ti:generation)","max_results":10,"sort_by":"submittedDate","sort_order":"descending"}
to=arxiv_search.search  东臣ീയ code  {"query":"id_list:[\"2409.01552\",\"2402.02212\",\"2312.03584\",\"2412.01824\",\"2407.07356\",\"2506.09630\",\"2502.16414\",\"2506.00507\",\"2311.09263\",\"2206.08082\",\"2504.20690\",\"2602.01609\",\"2506.15742\",\"2605.26356\"]","max_results":14}
In-context generation denotes a class of generation procedures in which a model is conditioned on examples, retrieved evidence, or reference signals supplied in its context window and then produces a new output without parameter updates. In the data-generation perspective, a large language model can be viewed as implicitly modeling a family of data-generation functions $f_\theta$; at inference, it may either select a previously learned function that explains the demonstrations (“skill recognition”) or adapt to the demonstrations to induce a new function (“skill learning”) [2402.02212]. Recent work uses this pattern across black-box LLM prompting, synthetic tabular data generation, machine translation, question generation, image editing, image synthesis, and video imitation [2409.01552][2502.16414][2407.07356].

## 1. Conceptual and theoretical foundations

A systematic formulation treats pre-training as learning a mixture over latent data-generation functions,
$$
p(t_{t+1}\mid t_{1:t})\approx\int_{\Theta}p_{\theta}(t_{t+1}\mid t_{1:t})\,p(\theta)\,d\theta,
$$
where each $\theta\in\Theta$ defines a conditional distribution or, in classification language, a function $f_\theta$ from input to label [2402.02212]. Under skill recognition, the model performs implicit Bayesian inference over $\theta$ using the demonstrations $D=\{(x_i,y_i)\}_{i=1}^n$ and concentrates on a latent concept $\theta^*$ that best explains them. Under skill learning, the model behaves as an on-the-fly learner, implicitly selecting
$$
f^*=\arg\min_{f\in\mathcal F}\sum_{i=1}^{n}\ell(f(x_i),y_i),
$$
and then applying $f^*(x_{\mathrm{test}})$ to the query [2402.02212].

This perspective has been sharpened by analyses of linear self-attention. In retrieval-augmented generation, one recent formulation introduces a unified linear predictor $y=W_1x_1+W_2x_2$, where $x_1$ is a query-side feature and $x_2$ is a retrieval-derived feature, and shows that one linear self-attention layer can implement one gradient-descent step on the corresponding linearized RAG objective [2605.26356]. The result is exact in the constructed linear regime: one forward update changes only the $y$ slot of the query token by $\Delta y=\Delta W_1x_1+\Delta W_2x_2$ [2605.26356]. The same work also shows the boundary of the analogy: the correspondence remains stable under controlled linear extensions, but becomes feature-distribution dependent under nonlinear architectures, especially under skewed or heavy-tailed feature distributions [2605.26356].

Taken together, these formulations treat in-context generation not as a special-purpose prompt trick but as forward-pass adaptation. This suggests that the central technical question is how context should be constructed, compressed, regularized, or audited so that the induced adaptation is useful rather than misleading.

## 2. Context construction in language models

A major line of work replaces manually curated demonstrations with generated or optimized context. Self-Generated In-Context Learning (SG-ICL) uses the same autoregressive model as a demonstration generator: given a test instance and a candidate label, the model samples demonstrations with temperature $\tau=0.5$, uses $N=k=8$ generated pairs, and then predicts with an inference template over the generated pool [2206.08082]. On SST-2, SST-5, RTE, and CB, SG-ICL consistently improves over zero-shot learning, has markedly lower variance than randomly selected gold demonstrations, and is “generally worth approximately 0.6 gold training samples”; equivalently, 8 self-generated demonstrations match roughly 5 gold demonstrations [2206.08082].

Auto-ICL generalizes this idea by having the model autonomously generate either demonstrations, instructions, or both, in a first stage and answer with them in a second stage [2311.09263]. In generating mode, the reported average accuracy is 68.1, compared with 63.9 for Zero-Shot-CoT and 38.3 for Zero-Shot; in retrieving mode, Auto-ICL reaches 75.7, exceeding Few-Shot, APE, Instruction Induction, and Auto-CoT on the reported averages [2311.09263]. The same study reports that instruction-only context is strongest in retrieving mode, whereas demonstration+instruction is best in generating mode [2311.09263].

ProGen adds feedback from a task-specific model. It iteratively grows a synthetic dataset, scores examples by a robust influence function using Reverse Cross-Entropy on a synthetic validation set, and feeds the top-$M$ “helpful” examples back as in-context demonstrations for the next generation round [2210.12329]. On five text classification datasets, ProGen improves average zero-shot accuracy from 82.94 to 86.51 for DistilBERT and from 75.56 to 80.99 for an LSTM, and matches or exceeds ZeroGen with only 1% of its synthetic dataset size [2210.12329].

A more explicit optimization of context appears in Li et al.’s two-stage framework for black-box LLMs. The method leaves the original prompt $x$ intact, learns a policy $\pi_\theta(x'|x)$ that generates a semantically aligned derived prompt $x'$, queries a fixed response model $\mathcal M$ on $x'$, and then wraps the $(x',y')$ pair into a single-shot in-context demonstration for the original prompt [2409.01552]. Training maximizes
$$
\mathcal L(\theta)=\mathbb E\big[\mathcal R(x',y')-\beta\log \tfrac{\pi_\theta(x'|x)}{\pi_{\mathrm{ref}}(x'|x)}\big],
$$
implemented with a ReMax-style policy gradient while keeping the response model immutable [2409.01552]. The inference template explicitly asks the model to “emulate” the way the derived-prompt response answers its question while replying to the original prompt, thereby anchoring the final answer in $x$ rather than replacing it [2409.01552]. On Vicuna Eval with GPT-4, “OURS vs. Original Prompt” wins 90.0% of the time and “OURS vs. BPO” 88.8%; on Self-Instruct Eval with GPT-4, the corresponding win rates are 76.2% and 71.4%; on GPT-3.5, the method maintains 70%+ win rate over the original prompt and 65%+ over BPO across benchmarks [2409.01552].

## 3. Structured-data and task-specific generation

In structured-data generation, in-context generation is used both as a substitute for fine-tuning and as a target for prompt optimization. TabGen-ICL formulates tabular synthesis with a fixed LLM and an iterative residual-aware selector: at iteration $t$, it chooses a subset
$$
R_t=\arg\min_{S\subseteq\mathcal D_{\mathrm{real}},\,|S|=n} d(\mathcal D_{\mathrm{real}},\mathcal Y_t\cup S),
$$
where $\mathcal Y_t$ is the set of generated rows so far and $d$ alternates between Jensen–Shannon divergence and Kolmogorov–Smirnov distance [2502.16414]. The selected examples are JSON-serialized into the prompt, and the loop progressively narrows the gap between generated and real distributions. Across five real-world tables, TabGen-ICL reduces the error rate by 3.5%–42.2% on fidelity metrics relative to random selection [2502.16414].

The same dependence on context creates a fairness risk. In LLM-based tabular generation, few-shot prompts consist of $k$ demonstrations $x_i=(a_i,f_i,t_i)$, and a bias parameter
$$
\pi=P_P(t=c\mid a=a^*)
$$
controls the label imbalance for a protected subgroup in those demonstrations [2506.09630]. The reported empirical result is that even mild in-context bias leads to global statistical distortion: as $\pi$ increases, the generated $P_G(t\mid a)$ tracks it nearly linearly, the effect strengthens with larger context size, and all tested models leak in-context imbalances [2506.09630]. In the adversarial setting, an attacker controlling a fraction $\pi$ of the in-context records can drive downstream fairness violations; at $\pi=0.4$, a Random Forest trained on the synthetic data exhibits $\mathrm{SPD}>0.2$, while utility degrades only modestly and fidelity remains high, with TVC $>0.79$ and JSD $<0.31$ in the reported example [2506.09630].

For machine translation in low-resource settings, Demonstration Augmentation for Translation (DAT) generates a candidate pool $X=\{x_1,\dots,x_m\}$ of source-side examples, filters it with an MMR objective balancing relevance and diversity, generates target sides zero-shot, and then translates the query with the resulting few-shot prompt [2506.00507]. In the reported setup, $m=10$, $k=4$, and $\lambda=0.5$ [2506.00507]. On English to Nepali, Khmer, Pashto, Zulu, and Swahili with Llama-3.1-70B, DAT outperforms zero-shot in 4 out of 5 languages and avoids the severe backfire observed for fixed human pairs, including a 21.6 COMET-point drop on Khmer for the few-shot fixed-pair baseline relative to zero-shot [2506.00507].

Task-specific generation also includes educational and commonsense settings. In automatic question generation from educational passages, GPT-4 with ICL and a Hybrid model combining ICL and retrieval both outperform baseline models; among automated metrics, ICL($k=7$) obtains ROUGE-L 55.95, METEOR 34.62, ChRF 60.48, and BERTScore 75.92, while the Hybrid model is best on all reported human measures except Answerability [2501.17397]. In commonsense generation, a two-step diversification wrapper first produces default outputs, then asks the model to generate sentences different from its previous outputs when diversity is low; on CommonGen with GPT3.5-turbo, this raises the harmonic mean of diversity and BERTScore from 39.6 for default ICL to 72.7 for the proposed ICD selection, while substantially lowering self-BLEU-4 from 72.4 to 21.0 [2404.16807].

## 4. Visual and multimodal in-context generation

In image generation, one influential design principle is to separate contextual appearance from structural control. Context Diffusion augments a latent diffusion UNet with a visual-context encoder, a frozen CLIP text encoder, a query-control encoder derived from ControlNet, and modified cross-attention that attends jointly to text embeddings $h^c$ and visual-context embeddings $h^V$ [2312.03584]. Prompt dropout replaces the text prompt with the empty string with probability $\tfrac12$, forcing the model to rely on visual context [2312.03584]. The reported user study shows especially large gains when only context is present: in-domain, the method wins 80.2% versus 4.5% for Prompt Diffusion under context-only conditioning; out-of-domain, the corresponding figures are 63.7% versus 22.8% [2312.03584].

X-Prompt extends in-context generation to a purely auto-regressive vision-language model by compressing each in-context example into a small set of learnable “X-Prompt” tokens through cross-attention [2412.01824]. If $X_{IE}\in\mathbb R^{L\times d}$ is an example sequence and $X_{XP}\in\mathbb R^{S\times d}$ are the learnable compression tokens, the model forces information flow through $X_{XP}$ and blocks direct attention from raw example tokens to target tokens [2412.01824]. This makes the total context length
$$
T=K\cdot S+|X_{\mathrm{target}}|,
$$
with a practical window of up to 5,120 tokens [2412.01824]. In zero-shot unseen tasks, the reported gains are large: low-light enhancement PSNR improves from 9.14 to 17.00, derain PSNR from 7.92 to 18.10, object addition $\mathrm{CLIP}_{dir}$ from $-0.031$ to 0.092, and unseen depth-color palette RMSE from 0.745 to 0.390 [2412.01824].

Video In-context Learning applies the same principle to tokenized video. A decoder-only LLaMA-style Transformer is trained self-supervised on 16-frame clips tokenized by a pretrained VQ-GAN into 4,096 tokens plus bos/eos, with no explicit demo/query structure during training [2407.07356]. At inference, demonstration clips and query frames are concatenated into one causal prefix, and the model autoregressively samples future frames [2407.07356]. With the 1.1B model, in-class demonstrations raise probing accuracy from 29.6% to 36.7% and V-Acc by 1.8 points, while PSNR and FID improve with model scale [2407.07356]. The paper characterizes the resulting behavior as zero-shot imitation from demonstration videos [2407.07356].

Latent-space flow and diffusion transformers now provide a unified setting for in-context image generation and editing. FLUX.1 Kontext uses simple sequence concatenation of text and image latents, offsets the “time” coordinate of each context image in factorized 3D RoPE, and trains a rectified-flow transformer with a conditional flow-matching objective in latent space [2506.15742]. On KontextBench, a benchmark with 1,026 image-prompt pairs across local editing, global editing, character reference, style reference, and text editing, FLUX.1 Kontext[pro] and [max] rank at or near the top in human ELO evaluations; on five successive edits, AuraFace cosine similarity averages 0.908 for FLUX.1 Kontext[pro], versus 0.774 for Runway Gen-4 and 0.416 for GPT-4o-High [2506.15742]. The reported inference time is 3–5 seconds for 1024×1024 images on a single A100 GPU [2506.15742].

## 5. Efficiency, token management, and forward-only adaptation

As contextual generation scales, sequence length becomes the main systems bottleneck. In Diffusion Transformers, in-context generation concatenates noisy latent tokens $\mathcal G^t$ with a fixed reference sequence $\mathcal C$, giving self-attention cost $O((|\mathcal G|+|\mathcal C|)^2\cdot d)$ [2602.01609]. ToPi addresses this with training-free token pruning. It first computes a layerwise Context Sensitivity Score
$$
S^{(\ell)}=\mathbb E\Big[\sum_{h\in\mathcal H}\sum_{i\in\mathcal G}\sum_{j\in\mathcal C}A^{(\ell)}_{h,i,j}\Big]
$$
on a calibration set, selects the top-$M$ representative layers, and then scores each context token by a value-weighted attention influence metric [2602.01609]. Pruning is updated only at anchor timesteps through a fidelity-constrained objective that preserves at least a fraction $\tau$ of total influence [2602.01609]. On Flux.1-Kontext and Qwen-Image-Edit, ToPi yields about 1.21×–1.33× speedup, recovers within $<0.2$ dB of full-context PSNR, adds $<1\%$ latency overhead, and removes over 50% of reference tokens on average while preserving at least 85% of the context “information mass” [2602.01609].

Instructional image editing exposes a parallel efficiency–precision problem. In-Context Edit treats a pretrained DiT inpainting model as a black box by forming a side-by-side in-context image in which the source image occupies the left half and the target half is masked; the associated IC prompt describes the original image on the left and the instructed edit on the right [2504.20690]. A training-free version already benefits from the IC prompt alone, improving CLIP-I from 0.681 to 0.794 and GPT from 0.14 to 0.24 in the reported ablation [2504.20690]. The paper then adds a LoRA-MoE hybrid, in which the output of the frozen base layer is augmented by a sparse mixture of low-rank experts, and an early-filter inference-time scaling method that scores partial denoising trajectories with Qwen-VL-72B [2504.20690]. The early filter improves SC by 19% and overall VIE-Score by 16% over single-seed outputs [2504.20690].

Forward-only adaptation also appears in retrieval-augmented generation. RAG-GD keeps the retriever and LLM backbone frozen, trains a base retrieval adapter $W_0^{\mathrm{ret}}$, and then meta-trains a predictor $g_\phi$ that maps a few-shot RAG support set to low-rank updates approximating what $K$ steps of SGD would have done to the retrieval interface [2605.26356]. At inference, the update is produced in one small forward pass rather than test-time backpropagation [2605.26356]. On Qwen 2.5 B with E5 retrieval, the reported average EM/F1 improves from 34.16/42.54 to 36.71/45.11, and the method approaches test-time gradient adaptation at much lower per-query cost [2605.26356].

## 6. Failure modes, safeguards, and open problems

A recurring concern is that contextual generation can improve surface quality while drifting semantically or statistically. Li et al.’s derived-prompt framework addresses semantic drift with two explicit safeguards: a KL penalty keeps the learned derived-prompt policy close to its reference initialization, and the final inference template always asks the model to answer the original prompt rather than the derived one [2409.01552]. In tabular generation, by contrast, the few-shot examples themselves may be the attack surface; prompt audit, balanced prompt design, fairness-guided exemplar selection, post-generation debiasing, and model-internal defenses are proposed as mitigation strategies for in-context bias propagation [2506.09630].

Several modality-specific limitations remain. Context Diffusion reports that very fine-grained local edits can still fail and that, when visual context and text disagree, the model tends to favor the context image [2312.03584]. X-Prompt notes that the base Chameleon VQ-VAE compresses at 16× and loses fine detail, and that generalization degrades across completely unrelated tasks [2412.01824]. FLUX.1 Kontext reports minor artifact accumulation and occasional instruction non-compliance after 6–7 edits in multi-turn workflows [2506.15742]. In machine translation, progressive accumulation of synthetic demonstrations improves retrieval-based reuse but does not fully match dynamic on-the-fly generation [2506.00507]. In educational question generation, example selection remains sensitive, and retrieved passages may be semantically related yet contextually irrelevant [2501.17397].

The broader research agenda remains explicitly open. The data-generation survey identifies the mechanistic origin of skill learning, the causal linkage between the pre-training function class and learnable in-context functions, the extension of the framework to chain-of-thought reasoning and self-critique, and unified probabilistic frameworks that cover both recognition and learning as central future directions [2402.02212]. A plausible implication is that progress in in-context generation will depend less on any single prompting heuristic than on a joint theory of context selection, context compression, implicit optimization, and failure analysis across modalities.

Source: https://www.emergentmind.com/topics/in-context-generation