Image Editing as Programs (IEAP)

Image Editing As Programs (IEAP) refers to the conceptual and practical framework in which image editing workflows are structured as explicit, programmable operations or sequences of operations—so-called “programs”—that integrate user intent, fine-grained compositionality, and interpretability. In the era of diffusion models and advanced neural generative techniques, IEAP addresses the need for editing systems that not only support a wide variety of user instructions (from text, images, multimodal pairs, or GUI interactions) but also offer modularity, generalization, transparency, and robust multi-step editing. The systematic analysis of “Image Editing As Programs” is now an active subdomain within image synthesis and editing, as evidenced by comprehensive reviews such as "Image Editing with Diffusion Models: A Survey" (Wang et al., 17 Apr 2025 ).

1. Task and Instruction Taxonomy

Image editing is precisely defined as the modification of existing images to realize user-specified intentions. The literature organizes image editing tasks along several key axes:

• Image Type: Differentiating between natural images (photorealistic or artistic) and feature images (e.g., edges, pose, depth maps).
• Editable Component: Partitioned into visual content (objects, backgrounds) and visual expression (structure, style, lighting, texture).
• Manipulation Operation: Canonical actions include Add (inserting elements), Delete (removal), Change (altering attributes or style), and Combine (compositing across images).
• Instruction Mode:
  • Text prompts (free-form or templated),
  • Visual/feature images (edges, sketches, depth, pose as control signals),
  • Multimodal/in-context examples (e.g., image pairs illustrating the desired transformation),
  • Drag-based or pointwise interactive input.

These factors combine to produce a rich taxonomy of editing scenarios, spanning object manipulation, global style changes, subject-driven edits, layout transformation, relighting, outpainting, and texture adjustment.

2. Methods Classification

State-of-the-art approaches are organized into three primary classes reflecting the architectural and computational strategy:

A. Inversion-Based Methods

These leave the base generative model unaltered, performing edits by manipulating internal variables—typically during the denoising process of diffusion models. The two main subgroups are:

• Information Preservation: Techniques inject controlled noise and denoise within a selected time window (e.g., SDEdit), or invert null-text embeddings to optimize MSE between noise latents (e.g., Null-text Inversion).
• Information Introduction: Editing is performed by modifying attention maps (Prompt-to-Prompt), manipulating masks for regional control (Blended Diffusion, Differential Diffusion), or injecting features directly (PnP, MasaCtrl).

These methods are widely adopted for their transparency and modularity—key requirements for principled programmatic editing.

B. Fine-Tuning-Based Methods

Here, the diffusion model is partially or wholly re-trained to accommodate editing operations, with two major flavors:

• Training-Time Fine-tuning: Models are trained on large paired datasets to directly perform editing in the desired regime (e.g., InstructPix2Pix incorporates both original images and textual instructions in conditioning).
• Test-Time Fine-tuning: Customization for new subjects or attributes is performed per-inference via focused adaptation (e.g., Textual Inversion, DreamBooth, Imagic).

These methods achieve high-quality effects but at the cost of dataset dependence and reduced flexibility.

C. Adapter-Based Methods

Adapters are trainable modules affixed to frozen base models, enabling multi-modal or targeted conditioning (e.g., ControlNet for edge/pose/depth, IP-Adapter for reference images, T2I-Adapter for multiple control signals). They facilitate plug-and-play modularity, supporting multitask compositionality and efficient knowledge transfer without incurring full model retraining.

3. Evaluation Metrics and Protocols

Rigorous benchmarking is foundational for progress in IEAP. The field employs a spectrum of evaluation protocols:

• Quality Metrics: FID (Fréchet Inception Distance), KID (Kernel Inception Distance), LPIPS, and DINO—evaluating realism, fidelity, and semantic consistency.
• Consistency/Instruction Faithfulness: CLIP Score quantifies image-text alignment; task-specific measures (SSIM, mAP/OKS, MSE, mIoU) are used for structural, detection, and segmentation accuracy.
• Human and MLLM-based Evaluation: Automated assessment with vision-LLMs (e.g., GPT-4V) and subjective human ratings capture nuance and user relevance.
• Benchmark Design: Datasets such as EditBench, EditVal, TEdBench, I2EBench, and EditEval employ multi-axis, weighted score aggregation to provide robust, multidimensional performance profiles.

Benchmarks and protocols emphasize generalizability, the ability to handle open-instruction sets, and transparency across evaluation axes.

4. Dataset Construction Strategies

Data availability is a critical factor for the advancement of IEAP. Current approaches use the following:

• Extraction-Based: Utilizes existing datasets or video sequences, extracting feature maps (edges, depth, segmentation, pose) or image pairs via segmentation, bounding box perturbation, frame tracking, or domain-specific selection (e.g., subject-driven multi-shot datasets, object composition from object-centric videos).
• Generation-Based: Employs synthetic pipelines by coupling LLMs with powerful T2I models to generate diverse (input image, instruction, edited image) triplets. Filters (CLIP-based, offensive/quality control, human-aligned scoring via MLLMs) are applied post-synthesis to ensure data robustness and relevance.

These methods accelerate dataset scale and diversity, helping overcome subject matter and compositional bottlenecks associated with manual annotation.

5. Programmatic Editing Paradigm and Future Directions

The emerging vision is to systematize image editing as a form of program execution, blending the strengths of symbolic and neural techniques:

• Compositional/Sequential Editing: Models are being designed to treat multi-step, multi-turn editing as the execution of an explicit or latent "program"—sequencing modular operations, parameterized by user instructions.
• Universal and Multitask Models: There is increasing emphasis on architectures (often adapter- or prompt-based) capable of handling all major editing operations (object, style, layout, compositional, etc.) without task switching or reconfiguration.
• Interactive and Multimodal Workflows: Integrating richer input modalities (vision-language, sketch, region, in-context exemplars), supporting stepwise or dialog-driven program assembly, and enabling real-time or near-real-time editing feedback.
• Program Synthesis and Symbolic Integration: There is growing exploration of neuro-symbolic language design, enabling users to specify sophisticated batch operations (e.g., ImageEye’s DSL) or enable programmable control logic suitable for automation and large-scale media pipeline integration.

Critical Challenges

Future research directions include robust noise scheduling to mitigate detail loss, scalable data construction for broader domains, adapter design for truly multi-modal/multi-turn scenarios, improved evaluation with semantic- and instruction-aware metrics, and tighter unification of vision (detection, segmentation) and editing models for multitask learning.

The survey concludes that contemporary progress in diffusion-based and neural image editing is moving resolutely toward the “image editing as programs” paradigm: edits are interpreted, parameterized, and executed as a modular and compositional program. This shift supports greater controllability, transparency, and creative breadth, and is expected to catalyze new advances in generalization, automation, and multimodal interactive editing over the coming years.