InstructDiff: Instruction-Following Diffusion

Updated 6 July 2026

InstructDiff is a diffusion-based paradigm that uses natural language instructions to transform pretrained models into generalist problem solvers.
It recasts diverse computer vision and video tasks as conditional image or video translations, unifying outputs across heterogeneous domains.
The approach employs dual guidance scales and multimodal conditioning strategies to ensure both structural fidelity and instruction adherence.

Searching arXiv for papers related to “InstructDiff” and instruction-following diffusion models. Search query: all:("InstructDiffusion" OR "InstructCV" OR "VIDiff" OR "Diff-Instruct" OR "Uni-Instruct" OR "InstructRL4Pix" OR "Generating Illustrated Instructions" OR "I^2G") “InstructDiff” commonly denotes a diffusion-centric paradigm in which a pretrained diffusion backbone is steered by natural-language instructions so that it functions as a generalist problem solver rather than merely a generative prior. In this formulation, heterogeneous tasks are recast as conditional image-to-image or video-to-video translation, with the instruction specifying both the task identity and the desired transformation (Geng et al., 2023, Gan et al., 2023). Within vision, this paradigm is exemplified by InstructDiffusion, InstructCV, VIDiff, and related systems that unify understanding tasks such as segmentation and keypoint detection with generative tasks such as editing and enhancement under a single instruction-following interface (Geng et al., 2023, Xing et al., 2023). The label is also used more loosely for adjacent lines of work on procedural illustration generation and diffusion distillation, and it is distinct from the unrelated large-language-model data selection framework titled “InstructDiff” (Bi et al., 22 May 2025, Luo et al., 2023, Su et al., 30 Jan 2026).

1. Definition and conceptual scope

In its diffusion-model sense, InstructDiff is the broad paradigm of taking a pretrained diffusion model and steering it with natural language instructions so that it becomes a generalist problem solver, rather than just a generative prior (Gan et al., 2023). The defining move is to cast diverse tasks into an instruction-following image-manipulating or video-translating process whose output space is a flexible pixel space or latent video space, rather than a task-specific output head (Geng et al., 2023, Xing et al., 2023).

This formulation appears in multiple concrete systems. InstructDiffusion frames computer vision tasks as instruction-following image manipulation in pixel space, always outputting a 3-channel image whose semantics depend on the instruction and downstream post-processing (Geng et al., 2023). InstructCV similarly casts multiple computer vision tasks as text-to-image generation problems, where the instruction specifies the task and the generated image visually encodes the answer (Gan et al., 2023). VIDiff extends the same philosophy to video, treating many video problems as conditional video-to-video translation from a source video $V_s$ and an instruction $\mathbf{c}$ to a target video $V_t$ (Xing et al., 2023).

A central conceptual property is that task identity is encoded implicitly by the instruction text rather than by a fixed task token. In VIDiff, for example, all tasks are trained as triplets $\langle V_s, V_t, \mathbf{c}\rangle$ , and the instruction $\mathbf{c}$ encodes “what task and how to do it” (Xing et al., 2023). In InstructCV and InstructDiffusion, the same image-to-image diffusion backbone is reused across segmentation, detection, depth estimation, classification, editing, and enhancement, with natural-language instructions as the sole task specifier (Gan et al., 2023, Geng et al., 2023).

The term is not fully standardized. “InstructDiffusion” is the specific 2023 generalist vision model (Geng et al., 2023). “Diff-Instruct” is a separate 2023 framework for data-free knowledge transfer from pretrained diffusion models to arbitrary differentiable generators via Integral Kullback–Leibler divergence (Luo et al., 2023). “Uni-Instruct” is a 2025 unification of one-step diffusion distillation methods under an expanded $f$ -divergence framework (Wang et al., 27 May 2025). A further ambiguity is that “InstructDiff” is also the title of a 2026 LLM supervised fine-tuning data-selection framework unrelated to diffusion modeling (Su et al., 30 Jan 2026). In contemporary vision usage, however, “InstructDiff” most often refers to instruction-following diffusion systems and the broader modeling interface they instantiate (Geng et al., 2023, Gan et al., 2023).

2. Architectural pattern and diffusion formulation

The canonical InstructDiff architecture is a latent diffusion model conditioned jointly on an input visual signal and an instruction. InstructDiffusion is built on Stable Diffusion v1.5, using a VAE encoder $\mathcal{E}$ , a UNet-like denoiser $\epsilon_\theta$ , and a text encoder whose embeddings are injected through cross-attention (Geng et al., 2023). InstructCV adopts the same base design and applies instruction tuning to a Stable Diffusion v1.5 backbone, again combining image conditioning and text conditioning in latent space (Gan et al., 2023). VIDiff extends a Stable Diffusion-style latent diffusion model to video by inflating 2D convolutions to 3D convolutions and inserting temporal attention layers over frame sequences (Xing et al., 2023).

The core supervision pattern is conditional noise prediction on triplets $(s_i, c_i, t_i)$ or $\langle V_s, V_t, \mathbf{c}\rangle$ . In InstructDiffusion, for each triplet $\mathbf{c}$ 0, the target image $\mathbf{c}$ 1 is encoded to latent space and corrupted with Gaussian noise; the model then predicts the added noise while conditioned on the source image and the instruction (Geng et al., 2023). In InstructCV, the conditional objective is

$\mathbf{c}$ 2

with $\mathbf{c}$ 3 the input image, $\mathbf{c}$ 4 the instruction, and $\mathbf{c}$ 5 the visual encoding of the target label (Gan et al., 2023). In VIDiff, the noisy target latent and source latent are concatenated channel-wise,

$\mathbf{c}$ 6

and denoising is conditioned on both the source-video latent and the instruction embedding (Xing et al., 2023).

A recurring architectural design is separate control over fidelity to the input visual condition and fidelity to the instruction. In InstructCV, classifier-free guidance is decomposed into image and text guidance scales,

$\mathbf{c}$ 7

so that $\mathbf{c}$ 8 emphasizes structural alignment with the input and $\mathbf{c}$ 9 emphasizes instruction faithfulness (Gan et al., 2023). VIDiff uses an analogous two-scale design with $V_t$ 0 for video similarity and $V_t$ 1 for instruction faithfulness (Xing et al., 2023).

This architectural pattern supports a general interpretation of InstructDiff as conditional translation in latent space. The input visual signal provides structural context; the instruction embedding specifies the transformation; diffusion provides the flexible generative mechanism for producing a target that may be a segmentation mask, keypoint markup, restored image, stylized image, or edited video (Geng et al., 2023, Xing et al., 2023).

3. Unified task interface in image and video domains

The central empirical claim of InstructDiff-style systems is that tasks with radically different conventional output spaces can be unified by converting their outputs into images or videos. InstructDiffusion always produces an image, and what changes across tasks is the instruction and the interpretation of the output pixels (Geng et al., 2023). InstructCV likewise converts labels $V_t$ 2 into image-space encodings $V_t$ 3, so the model solves tasks by generating visualized outputs rather than logits, coordinates, or class IDs (Gan et al., 2023).

In the image setting, InstructDiffusion covers keypoint detection, semantic and referring segmentation, image editing, and image enhancement (Geng et al., 2023). Keypoint detection is recast as drawing small opaque colored circles at chosen keypoints; segmentation is recast as applying a semi-transparent colored overlay to the labeled region; editing and enhancement remain direct image-to-image transformations (Geng et al., 2023). InstructCV covers segmentation, object detection, monocular depth estimation, and classification, again by mapping each target to an RGB image. Detection is represented as the original image with a drawn rectangle, depth as a grayscale image produced by

$V_t$ 4

and classification as a color-coded image block whose interpretation is recovered by post hoc decoding (Gan et al., 2023).

In the video setting, VIDiff explicitly treats heterogeneous tasks as instances of

$V_t$ 5

Its task set includes video recolorization, inpainting, dehazing, deblurring, language-guided video object segmentation, and instruction-guided video editing (Xing et al., 2023). A distinctive feature is that discriminative tasks such as referring video object segmentation are cast as generative segmentation: the model generates a target video consisting of semi-transparent segmentation masks or recolored object regions, thereby avoiding separate classifier or segmentor heads (Xing et al., 2023).

The same unification extends to procedural visualization. “Generating Illustrated Instructions” formulates the goal-to-procedure problem as generating a sequence of step texts $V_t$ 6 and a sequence of step images $V_t$ 7 conditioned on a goal $V_t$ 8, with goal faithfulness, step faithfulness, and cross-image consistency as distinct desiderata (Menon et al., 2023). $V_t$ 9 studies procedural text $\langle V_s, V_t, \mathbf{c}\rangle$ 0 to visual sequence generation, modeling

$\langle V_s, V_t, \mathbf{c}\rangle$ 1

and introduces a pairwise discourse coherence model to improve consistency across instructional steps (Bi et al., 22 May 2025).

This body of work suggests that the unifying interface is not the downstream task ontology but the generative rendering of a target visual state. A plausible implication is that InstructDiff-style systems are most naturally defined by a common conditional generation mechanism rather than by a common symbolic output vocabulary.

4. Instruction modalities and conditioning strategies

Natural-language instruction is the default control channel, but multimodal instruction has become an important extension. InstructDiffusion and InstructCV use textual instructions as their primary conditioning interface, often with template diversification or paraphrasing to broaden the instruction manifold seen during training (Geng et al., 2023, Gan et al., 2023). In InstructCV, a T5-based paraphrasing LLM is used to generate multiple paraphrases of task templates, and robustness to unseen phrasings improves markedly relative to training only on fixed templates (Gan et al., 2023).

VIDiff generalizes conditioning from text-only to multi-modal instruction. Text instructions are encoded with CLIP-Text (ViT-L/14), image instructions are encoded with CLIP-Vision, and a small MLP projects the image embedding into the text embedding space before channel-wise concatenation,

$\langle V_s, V_t, \mathbf{c}\rangle$ 2

The CLIP encoders are frozen and only the MLP is trained (Xing et al., 2023). This supports text-only, image-only, and text-plus-image editing, including reference-style transfer.

Procedural systems further refine instruction structure. $\langle V_s, V_t, \mathbf{c}\rangle$ 3 separates global goal $\langle V_s, V_t, \mathbf{c}\rangle$ 4 from step texts $\langle V_s, V_t, \mathbf{c}\rangle$ 5, encodes the goal with CLIP ViT-L, and parses long steps into clauses using a constituency parser, with clause embeddings produced by OpenCLIP ViT-G and concatenated for semantic completeness (Bi et al., 22 May 2025). “Generating Illustrated Instructions” uses an LLM to transform arbitrary user input into a normalized goal and up to six concise step headlines, then conditions a stacked diffusion model on the goal embedding, step embeddings, and step-positional encodings $\langle V_s, V_t, \mathbf{c}\rangle$ 6 (Menon et al., 2023).

Conditioning can also be refined at the level of spatial control and reward shaping. InstructRL4Pix starts from an instruction-based image editor initialized from InsPix2Pix and fine-tuned on MagicBrush, but then adds reinforcement learning guided by cross-attention maps. Its reward combines attention-map similarity,

$\langle V_s, V_t, \mathbf{c}\rangle$ 7

with an image-preservation regularizer

$\langle V_s, V_t, \mathbf{c}\rangle$ 8

and optimizes the diffusion policy with PPO so that edits localize more accurately to the target object region (Li et al., 2024).

In DiT-based multi-instruction editing, conditioning itself becomes the object of disentanglement. “Disentangling Instruction Influence in Diffusion Transformers for Parallel Multi-Instruction-Guided Image Editing” derives instruction-specific masks from head-wise differences in self-attention maps and uses them to localize each instruction’s effect during a single denoising process (Liu et al., 7 Apr 2025). This replaces prompt concatenation without control by explicit attention masking and latent blending, enabling parallel rather than sequential execution of multiple instructions.

5. Representative systems and empirical results

The empirical literature on InstructDiff-style models is organized around a small set of representative systems. The table summarizes their domain, task interface, and notable reported behavior.

System	Domain	Characteristic contribution
InstructDiffusion (Geng et al., 2023)	Image	Single diffusion backbone for understanding and generative tasks in pixel space
InstructCV (Gan et al., 2023)	Image	Instruction-tuned Stable Diffusion for segmentation, detection, depth, and classification
VIDiff (Xing et al., 2023)	Video	Unified instruction-following latent video diffusion with multi-modal instructions
StackedDiffusion (Menon et al., 2023)	Procedural image sequences	Joint generation of step images in a stacked latent for cross-step consistency
$\langle V_s, V_t, \mathbf{c}\rangle$ 9 (Bi et al., 22 May 2025)	Procedural image sequences	Pairwise discourse coherence and parser-based encoding for long instructional text

InstructDiffusion demonstrates that a single Stable Diffusion v1.5–based model can perform keypoint detection, semantic and referring segmentation, image editing, and low-level enhancement through an instruction-plus-image interface (Geng et al., 2023). On COCO val2017 keypoints it reports AP 71.2, on HumanArt 51.4, and on AP-10K 15.9, while semantic segmentation results include 53.17 mcIoU on COCO-Stuff, 72.55 on VOC, and 59.00 on PC-59 (Geng et al., 2023). Its editing results are roughly on par with specialized editors in CLIP-Sim and aesthetic score on replace, remove, and add benchmarks, while enhancement quality is explicitly bounded by the VAE reconstruction ceiling (Geng et al., 2023).

InstructCV presents a closely related but independently developed instruction-tuned text-to-image diffusion generalist (Gan et al., 2023). It reports RMSE 0.275 on NYUv2 depth, mIoU 52.3 on ADE20K segmentation, and [email protected] 49.1 on COCO detection for its fixed-prompt model, while also showing strong out-of-distribution behavior such as 69.8 mIoU on FSS-1000 open-vocabulary segmentation (Gan et al., 2023). It also demonstrates that paraphrase-trained instructions substantially improve robustness to unseen phrasings relative to fixed templates (Gan et al., 2023).

VIDiff is the principal video analogue. It is trained on short clips of 16 frames at $\mathbf{c}$ 0, but introduces an autoregressive Long Video Translation scheme in which later clips reuse the last $\mathbf{c}$ 1 generated frames of the previous clip as the first $\mathbf{c}$ 2 frames of the next source clip (Xing et al., 2023). On video editing benchmarks, it is described as one-model-for-all-videos, no per-video tuning, no DDIM inversion, and reports the best PickScore (20.73), best CLIPScore (31.15) for text alignment, and the lowest inference time of approximately 0.54 minutes on A100 hardware (Xing et al., 2023). On DAVIS recolorization it reports best FID 63.96, best Colorfulness 32.84, and best LPIPS 0.196 among the compared methods (Xing et al., 2023).

Instruction-following diffusion has also been applied to stepwise instructional content. “Generating Illustrated Instructions” reports that StackedDiffusion strongly outperforms baseline approaches and state-of-the-art multimodal LLMs, and that in 30% of cases users even prefer it to human-generated articles (Menon et al., 2023). Under fixed goal-and-step text, it reports goal faithfulness 74.3, step faithfulness 61.5, cross-image consistency 50.7, and FID 39.5, outperforming per-step text-to-image baselines especially in step faithfulness and cross-step consistency (Menon et al., 2023). $\mathbf{c}$ 3 instead emphasizes evaluation: classic metrics such as FID and CLIPScore are argued to be inadequate for procedural text alignment, and the paper introduces caption-based similarity scoring and distribution-level KL and $\mathbf{c}$ 4 discrepancy metrics, reporting consistently lower KL and Chi-square than several baselines across HT-Step, CaptainCook4D, and WikiAll (Bi et al., 22 May 2025).

Parallel multi-instruction editing in DiT backbones represents another empirical branch. IID improves FluxEdit from L1 0.1048 to 0.0731 and OmniGen from 0.1325 to 0.1115 on MagicBrush, while also improving CLIP-I and DINO similarity (Liu et al., 7 Apr 2025). InstructRL4Pix, by contrast, improves localization and preservation: on MagicBrush it reports L1 0.0608, L2 0.0189, SSIM 0.7978, and PSNR 24.26, outperforming InstructPix2Pix and supervised MagicBrush fine-tuning on those image quality metrics, though with lower CLIP-T (Li et al., 2024).

The phrase “InstructDiff” also appears in a theoretically distinct line on diffusion distillation. Diff-Instruct uses a pretrained diffusion model as an instructor for arbitrary differentiable generators and grounds training in the Integral Kullback–Leibler divergence

$\mathbf{c}$ 5

which compares teacher and student distributions along a shared diffusion process (Luo et al., 2023). This is not an instruction-following model in the human-language sense, but it is part of the broader “instruct” vocabulary around diffusion. Uni-Instruct later unifies Diff-Instruct, DMD, SIM, SiD, $\mathbf{c}$ 6-distill, SDS, and VSD as special cases of an expanded $\mathbf{c}$ 7-divergence family, and reports one-step generation FID 1.46 on CIFAR-10 unconditional, 1.38 on CIFAR-10 conditional, and 1.02 on ImageNet- $\mathbf{c}$ 8 conditional (Wang et al., 27 May 2025).

A second ambiguity is terminological. The 2026 paper titled “InstructDiff” introduces a domain-adaptive data selection framework for LLM supervised fine-tuning based on differential entropy, warmup calibration, bi-directional NLL filtering, and entropy-based ranking (Su et al., 30 Jan 2026). Its subject is unrelated to diffusion-based instruction following. In encyclopedia usage, this title collision is important because it makes “InstructDiff” polysemous across communities.

Across the instruction-following diffusion literature, several limitations recur. Resolution and length remain constrained: VIDiff is trained at fixed $\mathbf{c}$ 9 and 16 frames, and long-video support relies on a heuristic autoregressive scheme rather than native long-context temporal modeling (Xing et al., 2023). InstructCV and InstructDiffusion operate at $f$ 0 and are slower than task-specific feed-forward models; for structured tasks such as detection or classification, representing outputs as images induces brittle post-processing and inferior performance relative to specialized token or head-based models (Gan et al., 2023, Geng et al., 2023). Low-level restoration quality is capped by the VAE bottleneck in latent diffusion (Geng et al., 2023, Xing et al., 2023).

Instruction scope is also a limitation. In VIDiff, instructions are primarily short imperative phrases and richer compositional or multi-step instructions are underexplored (Xing et al., 2023). InstructCV’s instruction diversity is generated from a small set of templates and does not cover complex logical constraints (Gan et al., 2023). Procedural systems such as $f$ 1 improve long-form handling with parser-based encoding, but still report degradation on abstract or implicit actions and on long sequences where local pairwise coherence does not guarantee global consistency (Bi et al., 22 May 2025).

A common misconception is that instruction following by itself guarantees robust task execution. The broader alignment literature argues against this assumption: “On the Paradoxical Interference between Instruction-Following and Task Solving” shows that adding self-evident constraints can degrade task-solving performance in LLMs, with failed cases allocating significantly more attention to constraints compared to successful ones (Qi et al., 29 Jan 2026). Although this work is not about diffusion, it suggests that more explicit instruction control does not automatically imply better underlying task competence. A plausible implication is that InstructDiff-style systems also require explicit evaluation of task correctness, not just instruction adherence or visual plausibility.

7. Research directions and historical significance

Historically, InstructDiff marks a shift from task-specific visual heads and symbolic output spaces toward a common generative interface in which language specifies operations over pixels or latent visual states. InstructDiffusion states this transition most explicitly, presenting a generic framework that aligns computer vision tasks with human instructions by casting them into image manipulation rather than pre-defining categories and coordinates (Geng et al., 2023). InstructCV reaches a similar conclusion from the text-to-image side, showing that instruction tuning can convert a text-to-image diffusion model into a vision generalist (Gan et al., 2023). VIDiff extends the same interface to video, including both understanding and editing tasks (Xing et al., 2023).

Recent work indicates several concrete expansion paths. One is richer modality control: VIDiff already combines text and image instructions (Xing et al., 2023), and its own discussion suggests future expansion to audio, trajectories, and depth. Another is stronger structural reasoning for long-form instruction, as in parser-guided clause segmentation and pairwise discourse coherence in $f$ 2 (Bi et al., 22 May 2025). A third is finer edit localization and compositionality, pursued through attention-based reinforcement learning in InstructRL4Pix and instruction-specific attention masking in IID (Li et al., 2024, Liu et al., 7 Apr 2025).

Another direction concerns efficiency and distillation. Diff-Instruct and Uni-Instruct show that pretrained diffusion models can supervise one-step generators or other differentiable students without access to the original training data (Luo et al., 2023, Wang et al., 27 May 2025). While these works are not instruction-following systems, they address a practical bottleneck for InstructDiff-style models: the high inference cost of iterative denoising. This suggests a convergence between the instruction-following and distillation threads, where generalist instruction interfaces may eventually be paired with one-step or few-step backbones.

In that broader sense, InstructDiff is less a single model than a modeling interface: source visual context plus instruction in, transformed visual output out. Its significance lies in showing that segmentation masks, keypoints, restorations, stylizations, object insertions, procedural illustrations, and edited videos can all be treated as manifestations of the same conditional generative process (Geng et al., 2023, Xing et al., 2023). This suggests a common research program for diffusion-based generalists: unify task specification in language, unify output in visual space, and let diffusion serve as the shared inference mechanism.