---
title: 'IMAGAgent: Multi-turn Image Editing Framework'
url: https://www.emergentmind.com/topics/imagagent
type: topic
---

# IMAGAgent: Multi-turn Image Editing Framework

Searching arXiv for the specified topic and closely related agentic image-generation/editing work.
IMAGAgent is a multi-turn image editing agent framework that casts image editing as a closed-loop, constraint-aware process organized around a “plan–execute–reflect” mechanism. It is designed for settings in which an initial image is repeatedly edited through a sequence of user instructions, with each intermediate output serving as the input to the next turn. The framework addresses three failure modes identified in existing multi-turn editing systems—error accumulation, semantic drift, and structural distortion—by combining instruction decomposition, dynamic tool scheduling, and multi-expert reflective correction within a unified pipeline [2603.29602].

## 1. Problem setting and formal definition

IMAGAgent targets **multi-turn image editing**, where a system receives an initial image and then iteratively applies editing instructions over successive turns. In this regime, each intermediate result becomes the starting point for the next edit, so the system must satisfy the current instruction while preserving the semantics and structure established so far. The paper characterizes this as a long-horizon setting in which later edits depend on earlier ones and small errors can propagate across turns [2603.29602].

The framework is formalized as a mapping
\[
\mathcal{I}^* = \Phi(\mathcal{I}_0, \mathcal{T}_0),
\]
where \(\mathcal{I}_0\) is the initial image, \(\mathcal{T}_0\) is the user instruction, and \(\mathcal{I}^*\) is the final edited image. To support iterative control, IMAGAgent maintains a **dynamic historical context**
\[
\mathcal{C} = \{ (\mathcal{R}_k, \mathcal{I}_k, \mathcal{F}_k) \}_{k \ge 1} \cup \{ \mathcal{I}_0 \},
\]
in which each completed editing attempt stores an orchestration plan \(\mathcal{R}_k\), an output image \(\mathcal{I}_k\), and feedback \(\mathcal{F}_k\) [2603.29602].

The motivating diagnosis is explicit. Existing paradigms often treat each instruction in isolation, lack a persistent context representation, and do not evaluate whether an edit succeeded before moving forward. The consequence is a characteristic triad of failures: **error accumulation**, because early mistakes are not corrected; **semantic drift**, because prior constraints are not explicitly represented; and **structural distortion**, because local edits are not subjected to robust visual verification [2603.29602].

## 2. Closed-loop architecture

The system is organized into three main modules: **Constraint-Aware Planning**, **Tool-Chain Orchestration**, and **Multi-Expert Collaborative Reflection**. These modules form an iterative control loop rather than a one-shot editing pipeline [2603.29602].

| Module | Function | Main output |
|---|---|---|
| Constraint-Aware Planning | Decomposes complex instructions into atomic sub-tasks | \(\mathcal{A} = \{t_1, t_2, \dots, t_n\}\) |
| Tool-Chain Orchestration | Builds and executes a toolchain for the current sub-task | Candidate edited image \(\mathcal{I}_i^{(j+1)}\) |
| Multi-Expert Collaborative Reflection | Critiques the candidate edit and decides accept/retry | \(\mathcal{F}_i^{(j+1)} = \{F_{pos}, F_{neg}, S_i^{(j+1)}\}\) |

The planner \(\Psi\) receives the current image and instruction and produces a task sequence
\[
\mathcal{A} = \{t_1, t_2, \dots, t_n\} = \Psi(\mathcal{I}_0, \mathcal{T}_0).
\]
For each sub-task \(t_i\), an agent LLM generates an execution plan
\[
\mathcal{R}_i^{(j)} = \text{Agent}(\mathcal{I}_{i-1}, t_i, \mathcal{C}),
\]
which is then executed as
\[
\mathcal{I}_i^{(j+1)} = \text{Execute}(\mathcal{R}_i^{(j)}, \mathcal{I}_{i-1}).
\]
The resulting candidate is scored by a committee of VLM critics, and the fused reflection output determines whether the system accepts the edit or retries the same sub-task [2603.29602].

The acceptance rule is thresholded. If the reflection score \(S_i^{(j+1)}\) exceeds the success threshold \(\tau_{sr}\), the candidate is accepted and the system advances to the next sub-task. Otherwise it re-edits, up to an iteration limit \(\tau_{it}\). In the reported experiments, \(\tau_{sr}=7\) and \(\tau_{it}=3\) [2603.29602].

A common misconception is that IMAGAgent is a new end-to-end editor. It is not. The paper defines it as an **orchestration framework** built from off-the-shelf models and hand-designed prompts; no new model is trained end-to-end, and no gradient-based optimization or reinforcement learning is described [2603.29602].

## 3. Constraint-aware planning

The planning module is intended to convert a high-level natural-language instruction into a sequence of **atomic edit instructions** that are executable, visually grounded, and ordered by dependency. The planner is implemented with **Qwen-VL-MAX**, which takes the image and instruction and returns a JSON-like array of sub-task strings [2603.29602].

The planner is governed by three explicit constraints:

- **Target singularity**: each sub-task operates on a single entity or coherent group.
- **Semantic atomicity**: a sub-task cannot be meaningfully decomposed further without changing semantics or adding redundancy.
- **Visual perceptibility**: each sub-task must produce a tangible visual change.

These constraints are supplemented by temporal dependency rules and clarity requirements. If one edit depends on an object introduced by another, the creation step must precede the dependent step. The output sequence is therefore not merely segmented but also reordered into a causally consistent editing plan [2603.29602].

The paper gives a representative decomposition example for the instruction:

```text
[
  "The colour of the teacup is changed to black.",
  "The person's right hand makes a yay pose.",
  "The environment is changed to a green meadow",
  "Add a teapot",
  "delete the clouds in the sky",
  "add a rainbow in the sky"
]
```

This example illustrates the planner’s role in separating heterogeneous edit types—attribute modification, pose editing, environment replacement, object insertion, object removal, and object addition—into units that downstream detection, segmentation, and editing tools can address [2603.29602].

The ablation results indicate that this constraint regime is not incidental. Removing target singularity, semantic atomicity, and visual perceptibility lowers MTEditBench performance to DINO 0.722, CLIP-I 0.833, and CLIP-T 0.229, with the paper explicitly noting that instruction alignment degrades when decomposition is not well constrained [2603.29602].

## 4. Tool-chain orchestration

Once a sub-task is defined, IMAGAgent dynamically constructs an execution path conditioned on the current image, the current sub-task, and the historical context. The controller is **GLM-4.1V-9B-Thinking**, which reasons over the current state and emits a Python function `ToolGenerate(img)` implementing the selected toolchain [2603.29602].

The tool inventory is heterogeneous and includes image retrieval, detection, segmentation, inpainting, and editing:

- **Editing / inpainting**: `Inpainting`, `InpaintingByIpAdapter`, `ImageEditByPipe`, `ImageEditByAPI`
- **Detection & segmentation**: `GroundingDINO_SAM2`
- **Image retrieval**: `GetTargetImage`
- **Underlying editing models**: **Doubao-Seedream-4.0**, **Qwen-Image-Edit**, **Stable-Diffusion-XL-Base-1.0**

`GroundingDINO_SAM2` returns `target_box`, `maxscore`, `box_image`, `original_mask`, `white_mask`, and `cutOut_img`; internally it uses **Grounding DINO Base** for detection and **SAM3** for segmentation. `GetTargetImage(target: str) → Image` retrieves a reference image consistent with a textual description using **CLIP-ViT-Large-Patch14** [2603.29602].

The orchestration policy is prompt-based rather than learned. The controller checks prior edits and prior failures through \(\mathcal{C}\), including negative prompts such as `"background changed"` or `"dog"`, and can bias future tool selection accordingly. This history-sensitive behavior is central to preserving previously established image content [2603.29602].

The reported implementation stack includes **Qwen-VL-MAX** as planner, **GLM-4.1V-9B-Thinking** as orchestrator, **SAM3** for segmentation, **Grounding-DINO-Base** for detection, **CLIP-ViT-Large-Patch14** for retrieval, and **Doubao-Seedream-4.0** as the main image editing API. The system runs on a single NVIDIA A100 GPU, with local hosting except for API-based proprietary models. The paper estimates a per-task cost of approximately **0.0174 USD** under its reported API pricing and a \(512 \times 512\) resolution basis [2603.29602].

## 5. Reflection, feedback, and adaptive correction

After each candidate edit is generated, IMAGAgent performs **multi-expert collaborative reflection**. Three independent VLM experts critique the candidate using the pre-edit image, the edited image, and the current sub-task instruction, producing
\[
f_k = \{ f_{pos}^{(k)}, f_{neg}^{(k)}, s^{(k)}\}, \quad k=1,2,3,
\]
where \(f_{pos}^{(k)}\) is positive feedback, \(f_{neg}^{(k)}\) is a negative prompt string, and \(s^{(k)} \in [0,10]\) is a score that combines semantic alignment, perceptual quality, aesthetic assessment, and logical consistency [2603.29602].

These critiques are fused by an aggregator LLM:
\[
\mathcal{F}_i^{(j+1)} = \{F_{pos}, F_{neg}, S_i^{(j+1)}\} = \text{Aggregator}(f_1, f_2, f_3).
\]
The aggregator is **DeepSeek-V3.2**. The expert committee consists of **Qwen-VL-MAX**, **Doubao-Seed-1.6-Vision**, and **Doubao-Seed-1.6**. The fused negative prompts are passed directly into downstream editing tools, while positive prompts are used to preserve content that was judged correct [2603.29602].

The feedback mechanism is operational rather than merely diagnostic. In the paper’s “remove power sockets” example, the initial edit removes the sockets but distorts the wall and introduces an unintended bottle. Reflection then produces negative prompts such as `"bottle"` and `"background changed"` together with a positive instruction to remove the sockets while keeping the wall unchanged. On the second execution, the orchestrator selects a more precise detection-and-inpainting path, yielding a corrected result with preserved wall structure [2603.29602].

The reported refinement statistics quantify the importance of this loop. Approximately **68.32%** of steps satisfy the quality threshold on the first attempt, and approximately **27%** initially fail but are successfully corrected within three iterations. The authors note that without the loop those failures would remain and would contribute to severe multi-turn degradation. The multi-expert reflection stage increases runtime by approximately **1.4×** relative to open-loop baselines [2603.29602].

The reflection module also resolves a second misconception: IMAGAgent is not only a planner with tool use. Its distinctive property is that planning, execution, and critique are coupled through a persistent memory of failed attempts and corrective prompts, so the system can revise its own behavior at the level of both tool selection and prompt design [2603.29602].

## 6. Benchmarks, results, and ablations

Because existing multi-turn editing benchmarks were considered shallow, the paper introduces **MTEditBench**, constructed from the **VINCIE** training corpus through heuristic instruction segmentation, random shuffling, VLM-based segmentation and validation, filtering for sequences with at least **4 turns**, and manual verification. The final benchmark contains **1,000 long-horizon sequences**, each with at least four turns, spanning object addition, removal, and attribute editing [2603.29602].

Evaluation uses three automatic metrics:

- **DINO** for visual consistency / identity
- **CLIP-I** for semantic similarity between original and edited images
- **CLIP-T** for alignment between edited image and editing instruction

On **MTEditBench**, IMAGAgent reports average scores of **DINO 0.766**, **CLIP-I 0.875**, and **CLIP-T 0.248**. The paper emphasizes that degradation over turns is substantially slower than for comparison systems: for example, **OmniGen** drops in DINO from **0.800** at turn 1 to **0.480** at turn 5, whereas IMAGAgent drops from **0.803** to **0.721** over the same span [2603.29602].

On **MagicBrush**, which contains more than **10k triplets from 5,313 sessions** and has a maximum of **3 turns**, IMAGAgent reports **DINO 0.806**, **CLIP-I 0.898**, and **CLIP-T 0.282**, outperforming all non-proprietary baselines across metrics and turns. GPT‑4o attains stronger CLIP-T but is proprietary [2603.29602].

Ablation results further localize the gains. Removing reflection reduces MTEditBench performance to **DINO 0.689**, **CLIP-I 0.815**, and **CLIP-T 0.221**. Removing historical context reduces performance further to **DINO 0.646**, **CLIP-I 0.772**, and **CLIP-T 0.233**, with the paper specifically highlighting the large CLIP-I drop as evidence that historical context is critical for identity and semantic consistency. Replacing the multi-expert committee with a single expert yields **DINO 0.742**, **CLIP-I 0.854**, and **CLIP-T 0.237**, indicating a consistent advantage for collaborative reflection [2603.29602].

The user study covers **50 cases** with **20 expert raters** and reports that IMAGAgent is ranked top in instruction following, visual realism, and overall quality. The code is available at `https://github.com/hackermmzz/IMAGAgent.git` [2603.29602].

## 7. Position within agentic image systems and limitations

IMAGAgent belongs to a broader arXiv trend that treats image systems as agents rather than single-pass generators or editors. In adjacent work, **ImAgent** is a training-free unified multimodal agent for test-time scalable image generation and editing that integrates reasoning, generation, and self-evaluation inside a single backbone [2511.11483]. **Qwen-Image-Agent** frames image generation around a “Context Gap” and organizes plan, reason, search, memory, and feedback as context construction before rendering [2606.26907]. By contrast, IMAGAgent is explicitly specialized for **multi-turn image editing**, and its core novelty lies in coupling constraint-aware task decomposition, heterogeneous tool scheduling, and multi-expert visual reflection within a persistent historical context [2603.29602].

Within the paper’s own positioning, IMAGAgent differs from several prior categories. Relative to **single-turn diffusion editing** systems such as SDEdit, DiffEdit, InstructPix2Pix, and Imagic, it adds multi-step planning over a single instruction together with explicit feedback and self-correction. Relative to **unified multi-task image generators/editors** such as OmniGen, ACE++, and VAREdit, it replaces an open-loop “instruction → edit” paradigm with dynamic toolchain orchestration and reflection-driven correction. Relative to **in-context and autoregressive multi-turn editing** systems such as VINCIE and ICEdit, it adds an explicit visual evaluation stage with a multi-expert VLM committee [2603.29602].

The framework’s limitations are also clear. It is currently restricted to **static images**; extension to **video** and **3D** is presented as non-trivial because of temporal consistency and multi-view geometric coherence. Performance depends heavily on proprietary foundation models such as Qwen, GLM, Doubao, and DeepSeek. Inference cost and latency increase because of the closed-loop reflection stage. Finally, the control logic is **heuristic / prompt-based** rather than a learned policy, so there is no explicit guarantee of optimal decomposition or tool selection [2603.29602].

A plausible implication is that IMAGAgent is best understood as a systems architecture rather than a single editing model. Its contribution is procedural: it formalizes long-horizon image editing as a stateful sequence of planned sub-tasks, executable toolchains, and reflective corrections. In that sense, it establishes a concrete blueprint for agentic editing systems in which semantic preservation and structural stability are enforced through orchestration rather than assumed to emerge from a single forward pass [2603.29602].

Source: https://www.emergentmind.com/topics/imagagent