Prompt Grafting: Controlled Knowledge Injection

• Prompt Grafting is a technique that injects targeted, compositional knowledge into frozen pretrained models by manipulating prompts in distinct stages.
• In text-to-image synthesis, it first establishes a generic spatial layout followed by precise content grafting to ensure clear object separation.
• For protein models, it employs multi-block continuous prompts to improve interaction predictions and binding affinity without modifying model weights.

Prompt Grafting (PG) is a technique for controlling information injection in large pretrained models—either for compositional text-to-image synthesis or protein representation learning—by manipulating input prompts in a staged, structured manner. PG achieves targeted outcomes such as object separation or conformation-aware embeddings by decoupling prompt-driven knowledge over time or attention, without retraining or modifying underlying model weights (Pan et al., 25 Jan 2026, Zhang et al., 2022).

1. Fundamental Concepts and Motivation

Prompt Grafting addresses limitations in neural architectures where explicit compositionality or task-specificity is required but not inherently supported by frozen representations. In diffusion-based text-to-image models, adjacent object entanglement persists even with advanced text encoding; objects such as “rice” and “soup” visually fuse due to ambiguous boundaries in training data (Pan et al., 25 Jan 2026). In protein modeling, universal pretrained models generate fixed embeddings that obscure conformation-dependent dynamics, undermining accuracy on interaction tasks (Zhang et al., 2022). PG introduces separate prompt phases—spatial or semantic—whose fusion at specific times or locations achieves controlled downstream representations.

2. Methodology: Prompt Grafting in Diffusion Models

PG for text-to-image generation operates in two discrete stages:

Stage 1: Layout Prompt Formation

A layout prompt substitutes content tokens with generic, spatially separable objects (e.g., “plate,” “bowl”) and specifies arrangement (“on the left,” “on the right”). The model runs early denoising steps (first 10–20 % of total timesteps) conditioned on this layout, reliably forming distinct regions due to object boundaries.

Stage 2: Target Prompt Grafting

Upon stabilization of the spatial layout, as measured by CLIP image–text similarity plateauing, the system switches (grafts) to the true content prompt (e.g., names of specific foods). Denosings proceed with these real tokens, preserving prior regionalization.

The sampling loop, guidance, and dynamic graft detection mechanisms are captured by:

for t = T_total, ..., 1: if t > t_graft: c = c_layout else: c = c_target epsˆ = UNet-guided noise update x = x + γ_t * epsˆ

Here, $c(t)$ is a time-gated conditioning:

$$c(t) = \begin{cases} c_{\text{layout}}, & \text{if } t > T_{\text{graft}} \ c_{\text{target}}, & \text{otherwise} \end{cases}$$

Classifier-free guidance integrates a negative prompt to discourage undesirable merging:

$$\hat{\epsilon} = \epsilon_\theta(x_t, t; \varnothing) + w[\epsilon_\theta(x_t, t; c(t))-\epsilon_\theta(x_t, t; \varnothing)] - w[\epsilon_\theta(x_t, t; c_{neg})-\epsilon_\theta(x_t, t; \varnothing)]$$

3. Methodology: Prompt Grafting in Protein Representation Models

PG in protein modeling utilizes multi-block continuous prompts. For sequence $S=(s^1,...,s^n)$ and learnable prompt vectors $P_{seq}$ (for sequence) and $P_{IC}$ (for interaction-conformation), input embeddings are:

• $X_{in} = [E_{tok}(s^1),...,E_{tok}(s^n)]$
• $X_{full} = [P_{seq}; P_{IC}; X_{in}]$

A custom attention mask ensures only input tokens receive prompt information, blocking prompt-prompt and source-prompt cross-attention. Only the prompt vectors are updated through back-propagation; pretrained Transformer weights are held fixed.

Multi-task objectives govern loss:

$L = L_C + \lambda L_I$

where $L_C$ is the masked language modeling loss (for sequence prompt) and $L_I$ is the binary cross-entropy loss for protein-protein interaction (for IC prompt).

4. Implementation Details

Text-to-Image PG

• Model: Stable Diffusion v3; inference-only, no fine-tuning.
• Steps: 100 with DDIM or DPM++ sampling; guidance scale $w=12$; negative prompt “empty plate”.
• Platform: HuggingFace Diffusers + PyTorch; ≥1 NVIDIA A100/V100 GPU.

Protein Model PG

• Architecture: Pretrained Transformer (e.g., ESM-1b) with frozen weights.
• Prompts: Variable lengths $m, k \ll n$, concatenated as input.
• Training: Adam/SGD; only prompt vectors updated.

5. Experimental Results

Quantitative Outcomes in Text-to-Image Synthesis (Pan et al., 25 Jan 2026)

Dataset/Method	F1	BLIP (%)	FID
SD v3 baseline	0.490	99.4	40.5
SD v3 + SC only	0.508	99.2	47.8
SD v3 + PG only	0.500	99.5	43.7
SD v3 + SC + PG (full)	0.537	99.6	49.0

On UEC-256: | Dataset/Method | F1 | BLIP (%) | FID | |-----------------------------|---------|----------|-------| | SD v3 baseline | 0.056 | 99.5 | 70.6 | | SD v3 + SC only | 0.081 | 99.5 | 64.4 | | SD v3 + PG only | 0.149 | 99.7 | 60.8 | | SD v3 + SC + PG (full) | 0.165 | 99.7 | 65.0 |

PG produces distinct object regions more reliably than baselines, generalizes to non-food objects, and supports intentional merging by manipulating layout prompt regions.

Quantitative Outcomes in Protein Modeling (Zhang et al., 2022)

• PPI classification F1 improvements:
  • SHS27k: 68.12 → 71.24 (+3.12, with Seq+IC)
  • SHS148k: 75.16 → 79.55 (+4.39, with Seq+IC)
  • STRING-Human: 86.66 → 87.82 (+1.16, with Seq+IC)
• SAbDab binding affinity (Spearman’s $\rho$): Seq only 0.48 → IC only 0.51 → Seq+IC 0.55.
• CASP12 native contact (P@L/2): 0.43 → +IC prompt 0.41 (drop, indicating knowledge incompatibility).
• ICProtein contact precision: 0.29 → +IC prompt 0.37 (+8%).

Sequential prompt preserves sequence-driven tasks; IC prompt is crucial for conformation-driven tasks. Combining prompts yields additive benefits for complex tasks; incompatible prompt-task pairs degrade performance marginally.

6. Analytical Perspectives and Extensions

Ablation studies indicate that dynamically-determined grafting timesteps, particularly those guided by CLIP similarity convergence, optimize F1 and existence rates over fixed-step approaches (Pan et al., 25 Jan 2026). User-controlled entanglement is achieved by specifying the number or arrangement of generic regions in the layout prompt, directly modulating separation or fusing behaviors.

PG’s architecture-agnostic and training-free properties enable its extension across domains: new prompts can be learned and grafted for arbitrary downstream objectives, such as subcellular localization or open-vocabulary mixtures. Potential improvements include learnable gating ($g(t)$), vision-language layout predictors, and retrieval-coupled scheduling for richer compositional relations.

Failure modes include rare class generation (owing to limited pretraining distribution), and inadequacies for relational prompts requiring hierarchy (“soup poured over salad”).

7. Broader Significance

Prompt Grafting introduces a principled, interpretable adapter paradigm for frozen models. By temporally or structurally gating explicit prompt content—whether via layout-first denoising in generative diffusion or disentangled attention blocks in sequence encoders—PG enables targeted compositionality and knowledge injection. This methodology generalizes across modalities, drastically reduces fine-tuning cost, and preserves pretraining capabilities by confining updates to slim, prompt-specific parameter sets. It fosters modularity, facilitates user control over entanglement, and elucidates the roles of explicit versus implicit conditioning in high-capacity model architectures (Pan et al., 25 Jan 2026, Zhang et al., 2022).