IEA: Amateur-Friendly Conversational Image Editing Agent via Three Stages of Multitask Alignment

Abstract: Current image editing software often hinges on fixed filters or expert tuning, leaving a gap between amateur users' intent and outcomes. Creations by generative models may contain artifacts, implausible details, or stylistic drift away from photorealism and offer little insight into why an edit was made. We propose IEA, a conversational Image Editing Agent that learns to operate parameterized tools in an explicit, interpretable action space. IEA is trained via a three-stage multitask pipeline: (1) SFT on distilled expert edits, (2) GRPO with rewards for likeness improvement, tool usefulness, and intent summarization, and (3) large-scale synthetic fine-tuning to jointly master image editing, refinement, and user intent summarization. By manipulating 16 editing tools step by step, IEA produces transparent edit traces that can be inspected and debugged. In quantitative experiments, it attains a lower pixel distance on the edit task and a higher ROUGE-L on the summary task than strong baselines. In user studies, it ranks best among tool-calling methods for instruction following while surpassing generative methods in overall perceptual quality. Our results validate interpretable, tool-centric VLMs as a reliable path to human instruction-guided image retouching.

• The paper introduces IEA, a framework that employs a three-stage multitask alignment to translate natural language instructions into precise, parameterized image edits.
• It utilizes explicit tool invocation and reinforcement learning to achieve state-of-the-art edit precision and instruction summarization compared to generative models.
• The framework demonstrates superior auditability and control by bridging the gap between amateur usability and professional-grade image editing tools.

IEA: A Conversational, Amateur-Friendly Image Editing Agent via Multitask Alignment

Motivation and Problem Statement

The paper "IEA: Amateur-Friendly Conversational Image Editing Agent via Three Stages of Multitask Alignment" (2606.08016) addresses critical gaps in current image editing paradigms. Existing professional suites offer precise control but are inaccessible to amateurs due to steep learning curves, while "one-click" or purely generative tools lack fine-grained interpretability, limiting user-driven customization and failing to bridge the intention-outcome gap. Generative editing with diffusion or autoregressive models introduces unpredictable artifacts and is computationally expensive, while most visual LLMs (VLMs) lack explicit, interpretable action traces.

The core motivation is to provide an agentic framework, grounded in parameterized editing tools, that enables natural language-driven, transparent, controllable, and user-intent-aligned image editing accessible to non-experts. The authors propose a vision-language agent (IEA) that orchestrates stepwise editing via explicit parameter controls, supporting instruction following, intent summary, and iterative refinement through a three-stage alignment process.

Figure 1: User can easily interact with IEA to edit the image in the styles they prefer, or simply refine in general expertise. IEA can also learn user intents based on their previous attempts.

IEA Architecture and Training Pipeline

The IEA framework is built upon an explicit, interpretable action space with 16 parameterized editing tools, such as exposure, contrast, saturation, and sharpness, closely modeling professional photo editors. The model is trained to predict a sequence of tool calls (in JSON format), providing transparent edit logs for every image manipulation. The pipeline comprises three aligned stages:

Figure 2: Overview of the IEA framework: Data Pipelines and Three-Stage Training Process.

Stage 1: Supervised Policy Initialization

IEA is bootstrapped via supervised fine-tuning (SFT) on expert-edited datasets. Starting from GIER, semantic gap-bridging is performed by leveraging GPT-4.1 to infer precise numeric tool parameters and chain-of-thought rationales, compensating for the lack of direct parameter labels in expert data. A lightweight, heuristic, parameter search further bridges domain discrepancies, ensuring the system's parameterization closely matches expert edits. In this stage, two formulations are introduced:

• Image-Edit: Predict tool parameters and rationales from an original image and user instruction.
• Image-Summary: Given an original and edited image, summarize the user’s intent as a concise instruction.

  Figure 3: Example of Image-Edit task.

  

  Figure 4: Example of Image-Summary Task.

Stage 2: Policy Optimization with Reinforcement Learning

The model is refined using Group Relative Policy Optimization (GRPO), incorporating rollout-based update dynamics and carefully crafted reward functions:

• Likeness Improvement ($R_L$): Measures normalized improvement in perceptual distance to expert reference through the average of L1/L2 metrics.
• Tool Usefulness ($R_U$): Enforces that each tool invocation yields a measurable, non-redundant improvement by ablation.
• Summary Alignment ($R_A$): Trains a reward model to assess the semantic and specific congruence of the model’s summarized instruction against ground truth.

Both $R_L$ and $R_U$ are visualized for clarity.

Figure 5: Illustration of likeness improvement reward $R_L$ and usefulness reward $R_U$.

Stage 3: Policy Generalization via Synthetic Data

The final stage addresses coverage sparsity and intent variation through large-scale synthetic data generation. Expert programs (parameter sets) are sampled, paired with template- and LLM-generated natural instructions, and rendered into before/after image pairs. This supports:

• Full coverage of tool space and tool interactions
• Exposure to diverse instruction styles (casual to expert)
• Supervision of refinement via related edit pairs and derived refinement instructions

The Image-Refine task is thus added: given current parameters, output, and user's feedback, produce refined parameter sets.

Figure 6: Example of Image-Refine Task.

Quantitative Evaluation

Extensive evaluation leverages two key axes: edit precision and instruction alignment, compared across both generative approaches (e.g., diffusion models) and tool-calling methods (e.g., baselines using LLMs and VLMs).

IEA achieves state-of-the-art results in both Image-Edit and Image-Summary tasks:

• Pixel Distance (Edit Precision): IEA-Stage-3 attains a distance of 0.103 to expert references, outperforming all baselines.
• ROUGE-L (Instruction Summarization): IEA-Stage-3 achieves 0.258, significantly above prior baselines.
• Reward Metrics ($R_L$, $R_U$, $R_A$): Positive, high-performing values, with IEA-Stage-3 maximizing both likeness and usefulness, and near doubling $R_U$0 over the initial stage.

These results confirm strong transfer and fine-grained alignment from semi-supervised distillation to reinforcement-based and synthetic generalization.

Qualitative and User Studies

A comparative user study on 50 real-world editing scenarios evaluates instruction-following and perceptual quality. IEA consistently receives the highest average rank for instruction adherence among tool-calling models and outperforms all generative and tool-based competitors in perceived image quality.

Figure 7: Task A: Instruction Following Assessment.

Figure 8: Comparison of image editing results. Images are cropped and resized to 512-by-512 pixels for clear presentation. Readers can zoom in to examine the details.

Case studies confirm that IEA provides measured, context-sensitive edits that closely follow user intent, excel at subtle manipulations, and preserve photorealism. Generative models, while capable of filling in or hallucinating content, frequently introduce unwanted artifacts or deviate stylistically.

Reward Model Reliability

A dedicated, lightweight reward model was trained to judge instruction-summary consistency. The IEA-Summary-RM demonstrates a mean absolute error of 1.14 and accuracy of 66%, outperforming small and frontier LLMs for this discriminative task.

Figure 9: Distribution of predicted reward vs.\ ground-truth scores.

System Demo and User Interface

The IEA demo presents a conversation-based editing interface, supporting both instruction-based and refinement-based workflows, as well as intent summarization.

Figure 10: Interface of Image Editing Agent.

User feedback is further collected and assessed through tailored UI in large-scale preference studies.

Figure 11: Screenshot of the user study interface. For a given original image and instruction (top), participants were asked to rank the edited results from different methods (bottom) by dragging and dropping them in order of preference or compliance.

Implications and Outlook

IEA's explicit, tool-centric design offers advantages for auditability, user-specific refinement, and generalization beyond opaque LLM or diffusion-based pipelines. It demonstrates that VLMs, when properly multitask-aligned through a mix of expert, RLHF, and synthetic data, can robustly translate natural language instructions into parameterized, high-quality edits while supporting iterative workflows.

Practically, this supports the deployment of controllable, transparent image editing agents in consumer and professional applications, lowering barriers for non-experts. Theoretically, IEA advances understanding of how agentic VLMs can leverage symbolic action spaces for high-fidelity visual reasoning and personalization.

Further avenues include extending from global to local/semantic editing tools, scaling real-user feedback integration, improving backend rendering fidelity, and expanding evaluation to more diverse and longitudinal user studies.

Conclusion

IEA introduces a robust, amateur-accessible paradigm for conversational image editing via a hybrid VLM-agent framework. Through a meticulously structured alignment pipeline, IEA surpasses both generative and tool-based baselines in edit fidelity and instruction adherence, substantiating the interpretability and reliability advantages of parameter-centric image retouching mediated by VLMs. This work provides a practical and extensible platform for future research on controllable, user-aligned vision-language agents.