---
title: 'Lego-Edit: Modular Image & Robotic Editing'
url: https://www.emergentmind.com/topics/lego-edit
type: topic
---

# Lego-Edit: Modular Image & Robotic Editing

Lego-Edit denotes two distinct but conceptually related research usages. In one usage, it is the name of a general instruction-based image editing framework that treats editing as workflow synthesis over model-level tools by a multimodal large language model. In another usage, suggested by recent robotic LEGO-construction literature, it denotes a broader technical agenda in which creating, modifying, correcting, assembling, and reversing LEGO structures are reduced to discrete brick operations, validated in simulation, and executed by robots. Across both usages, the “LEGO” metaphor refers to compositionality: complex behavior is assembled from smaller, typed units rather than produced by a single monolithic model [2509.12883][2309.11010].

## 1. Scope and naming

In the image-editing literature, Lego-Edit is a specific framework introduced for open-domain instruction-based image editing. Its defining claim is that real-world instructions are “immensely diverse,” and that a planner should compose specialized tools rather than rely on a single end-to-end editor. The system therefore separates a reasoning agent, called the **Builder**, from a library of **Bricks** and an **Executor** that runs the resulting workflow graph [2509.12883].

In robotic manipulation and assembly research, the term does not yet denote a single standardized platform. Instead, multiple papers explicitly analyze how a system like “Lego-Edit” could be built from existing components: demonstration-driven task extraction, digital-twin verification, robust assembly and one-piece disassembly, fingertip perception for precise local correction, and graph- or instruction-based structural representations. This suggests a broader, prospective meaning of Lego-Edit as a LEGO-structure editing stack rather than a finished unified system [2309.11010].

A common source of confusion is that these two usages are not about the same artifact. The image-editing framework operates on images and natural-language editing instructions. The robotic usage concerns physical LEGO structures, assembly traces, brickwise manipulation, and reversible build plans. The overlap is architectural rather than domain-specific: both favor modular decomposition, typed intermediate outputs, and explicit execution graphs over single-shot generation [2509.12883][2309.11010].

## 2. Lego-Edit as an instruction-based image editing framework

The image-editing framework formalizes instruction-based editing as planning and execution over typed tools. Given an image-instruction pair $(I,q)$, the Builder observes state $s=(I,q)$ and generates a reasoning trace and a JSON workflow according to policy $\pi_\theta(g \mid s)$. The workflow is a tool-invocation graph $g=(V,E)$, where nodes are selected tool instances and edges represent data dependencies. The Executor parses this graph and produces the edited image as $I' = D(g, I)$ [2509.12883].

Its tool library is divided into predictive tools and editing tools. Predictive tools include **RES** for text-prompted segmentation, **SOS** for salient-object segmentation, **ADD-PRED** for predicting a plausible region for object insertion, **CAP-PRED** for captioning and optional image expansion, **INVERSE** for mask or image inversion/subtraction, and **BBOX** for converting a mask to a bounding-box mask. Editing tools include **FLUX-FILL** for masked generation, **FASTINPAINT** and **FLUX-INPAINT** for removal-oriented inpainting, **RCM** for color or material replacement, **STYLE** for style conversion, **ENV** for environment modification, **POSE** for posture or expression changes, **COMPOSE** for combining masks or images, and **RESIZE** for scaling a valid mask or image region [2509.12883].

A central design decision is that the Bricks are **model-level tools**, not coarse pre-authored task workflows. This lets the Builder synthesize multi-step pipelines such as `RES \rightarrow RCM` for localized recoloring, `RES \rightarrow FASTINPAINT \rightarrow INPAINT` for object removal, `CAP-PRED \rightarrow STYLE` for style transfer with semantic preservation, and `RES \rightarrow ADD-PRED \rightarrow FASTINPAINT \rightarrow INPAINT \rightarrow FILL` for object replacement. The system enforces parameter traceability: every parameter supplied to a tool must come from the initial image, the user text, a previous tool output, or the Builder’s own language decomposition of the instruction [2509.12883].

The editing tools themselves are specialized rather than jointly multitask-trained. Lego-Edit adopts ICEdit’s setup built on **FLUX-1 Fill** and trains five specialized **LoRA adapters** at rank 32 for **INPAINT**, **POSE**, **ENV**, **STYLE**, and **RCM**. The Builder is based on **MiMo-VL-7B**. Predictive modules include **EVF-SAM** for RES, **U2Net** for SOS, and **Qwen2-VL-2B** for ADD-PRED and CAP-PRED. This specialization is explicitly motivated as a way to avoid “task confusion” [2509.12883].

## 3. Progressive reinforcement learning, evaluation, and extensibility

The Builder is trained in three stages. In Stage 1, supervised fine-tuning uses examples containing image $I$, instruction $q$, ground-truth reasoning trace, and ground-truth workflow, with target sequence $l=[Think_{\text{GT}}, g_{\text{GT}}]$ and loss
\[
L_{\text{SFT}} = -\sum_{t=1}^{T} \log p_\theta(l_t \mid I, q, l_{<t}).
\]
This stage teaches workflow syntax, tool interfaces, and initial domain-specific planning behavior [2509.12883].

Stages 2 and 3 use **Group Relative Policy Optimization (GRPO)**. For sampled workflows $\{g_1,\dots,g_G\}$ with rewards $r_j$, the relative advantage is
\[
A_j = \frac{r_j - \mathrm{mean}(\{r_1,\ldots,r_G\})}{\mathrm{std}(\{r_1,\ldots,r_G\})}.
\]
Stage 2 combines a **valid reward**,
\[
R_{\text{valid}} =
\begin{cases}
0, & \text{if } D \text{ successfully executes } g \\
-1, & \text{otherwise},
\end{cases}
\]
with a **similarity reward** based on hierarchical graph matching against an expert workflow. Stage 3 removes the need for ground-truth workflows and replaces the similarity term with an **effectiveness reward**
\[
R_{\text{effect}} = 1 - 0.5 \cdot (N_{\text{add}} + N_{\text{remove}}),
\]
where an MLLM critic decomposes the generated workflow into editing chains, abstracts them into meta-edit descriptions, and judges whether chains should be added or removed to better satisfy the instruction [2509.12883].

The training corpus scales across supervision levels: 500 image-text pairs with instructions, thoughts, and workflows for Stage 1; 20K pairs with instructions and workflows for Stage 2; and 50K pairs with instructions only for Stage 3. All are sourced from **OmniEdit**. The Stage-3 critic is based on **Qwen2.5-VL-72B**. Training uses AdamW with learning rate $1\times10^{-5}$, weight decay 0.01, batch size 8, input image size $448\times448$, one epoch per stage, and **DeepSpeed ZeRO-3** on $8\times$ NVIDIA H20 [2509.12883].

Empirically, Lego-Edit reports state-of-the-art results on both **GEdit-Bench** and **ImgEdit-Bench / ImgBench**. On GEdit-Bench it achieves
\[
G_O = 6.64,\quad G_{PQ} = 7.45,\quad G_{SC} = 5.99,
\]
and on ImgEdit-Bench it achieves best overall score $3.50$, with particular strength on **Hybrid Editing** at $3.18$. The Builder variant trained with GT-free RL reports **Simple pass: 100\%** and **Complex pass: 99.0\%**. Runtime on GEdit-Bench with one H20 GPU is reported as **3.5 s** Builder latency, **2.7 s** slowest tool latency, and **\sim 7.2 s** total pipeline latency [2509.12883].

A notable property is tool extensibility. The framework reports that newly introduced tools can be used without additional fine-tuning of the Builder. The paper illustrates this with **reflection removal**: an initial workflow using `RES + INPAINT` fails, user feedback leads the Builder to switch to `SOS + INVERSE + INPAINT`, and a newly introduced dedicated tool **RRF** can then be adopted immediately. This is presented as evidence that the Builder learns tool-interface reasoning rather than memorizing a closed workflow library [2509.12883].

## 4. Robotic LEGO editing through demonstration and simulation

The most direct robotic foundation for a physical Lego-Edit system is **Simulation-aided Learning from Demonstration (SaLfD)**. The framework asks a human to construct a customized LEGO object on a baseplate, observes the demonstration with an RGB-D camera, extracts the sequence of brickwise operations, verifies candidate operations in a digital twin, and then executes the learned plan on a robot. Disassembly is obtained automatically by reversing the assembly sequence. The target structure is formalized as a set of bricks
\[
B=\{b_1, b_2, \dots, b_n\},
\]
with each brick encoded as
\[
b_i=[p;\ id;\ \omega],
\]
where $p$ is position, $id$ is brick type, and $\omega$ is orientation. The overall objective is to generate a structure $B$ identical to target $B^t$ by minimizing
\[
\min L(B^t, B),
\]
and the brickwise recovery objective is decomposed as
\[
\min_{b_i} L(b_i, b_i^t) = L_p(p_i,p_i^t) + L_i(id_i,id_i^t) + L_w(\omega_i,\omega_i^t).
\]
The output of learning is therefore an ordered symbolic-geometric task sequence rather than a continuous control policy [2309.11010].

The hardware setup uses a **FANUC LR Mate 200iD/7L** robot and a **RealSense L515 RGB-D camera** mounted on the end effector, with one \(48\times48\) construction plate and one \(48\times48\) storage plate. Keyframe detection uses a CNN with **3 convolution layers**, each followed by ReLU and max-pooling, and then **3 fully connected layers**. Operation extraction from consecutive RGB-D keyframes uses a second CNN whose outputs are brick ID, binary orientation, and position. Position is modeled with Gaussian uncertainty around mean
\[
\mu_p=[\mu_x,\mu_y,\mu_z],
\]
and candidate confidence is
\[
f(b_i') = f_p(p,\mu_p)\cdot f_{id}\cdot f_{\omega}.
\]
Rather than committing to a single operation, the model returns a ranked candidate list [2309.11010].

Verification is performed in a **ROS Gazebo** digital twin. The simulator executes the highest-ranked candidate and compares the simulated end state with the real observed end state using a learned similarity score $s$ from another CNN. If
\[
s > \delta_s,
\]
the candidate is accepted; otherwise the next candidate is tried. If no candidate exceeds the threshold, the final choice is
\[
\arg\max_{p,id,\omega} s.
\]
This makes simulation a consistency filter for noisy RGB-D perception rather than a global symbolic planner [2309.11010].

This pipeline is directly relevant to LEGO editing because an edit can be represented as a localized sequence of brick operations \([p,id,\omega]\), inserted into or removed from an assembly trace. The paper explicitly notes that replacing a brick can be represented as disassembling the old brick at position $p$ and then assembling a new one at $p$, and that partial disassembly could be realized by reversing a suffix or subtrace of the assembly plan. However, these are extrapolations from the framework rather than implemented features [2309.11010].

Quantitatively, SaLfD substantially improves exact recovery over learned extraction without simulation verification. Across objects such as **AI**, **RI**, **Human**, **Chair**, **Spiral**, **Bridge**, **Pyramid**, and **Temple**, SaLfD reaches **100\%** on **AI** and **Pyramid**, **96.7\%** on **RI**, **95.2\%** on **Temple**, and **86.3\%** on **Bridge**, consistently outperforming LfD without simulation verification. For plan generation time, **AI \{10 bricks\}** takes **66.2 s** with SaLfD versus **168.6 s** by handcrafting and **302.3 s** by manual simulation authoring, while **Temple \{23 bricks\}** takes **133.5 s** with SaLfD versus **333.4 s** and **790.5 s**. The earlier precursor paper represents the learned task as a temporal task graph
\[
G=[g_1,g_2,\dots,g_T],\qquad
g_i=\{id_i,~p^s_i,~\omega^s_i,~p^a_i,~\omega^a_i\},
\]
and similarly derives disassembly by reversing the learned assembly graph [2309.11010][2305.15667].

## 5. Manipulation substrates: end-of-arm tooling and fingertip perception

A physical Lego-Edit system requires reliable brick insertion and one-piece disassembly, and this is addressed by a hardware-software co-design centered on a passive **end-of-arm tool (EOAT)** inspired by the LEGO brick separator. The EOAT reduces the manipulation problem from uncertain grasp-and-insert to an insert-and-twist primitive. The robot first inserts the target brick into the EOAT so that the tool frame becomes kinematically attached to the brick, then executes a twist about a task-defined pivot. Assembly and disassembly are parameterized mainly by twist angle $\theta$, axis offsets $d_x,d_z$, and task duration $T$; parameter tuning is posed as
\[
\min_{T,dx,dz,\theta} L(\theta,dx,dz,T,U),
\]
with cost
\[
L(\theta,dx,dz,T,U)=
\begin{cases}
-\alpha T + \beta \theta + \gamma F + \eta \sum_T |u_t|, & \text{if manipulation succeeded} \\
\infty, & \text{otherwise}.
\end{cases}
\]
The parameters are optimized by **covariance matrix adaptation evolution strategy (CMA-ES)** directly on real robots [2309.02354].

The manipulation benchmark covers **\(1\times2\), \(1\times4\), \(2\times2\), and \(2\times4\)** bricks, heights from **1 to 10 layers**, solid and hollow support structures, and **25 positions** on a \(48\times48\) LEGO plate. Before learning, assembly success is consistently **100\%** and disassembly success is generally above **80\%**. After safe learning, both assembly and disassembly reach **\[100\% / 100\%\]** across all tested scenarios in the table. The learned parameters also transfer from **FANUC LR-mate 200iD/7L** to **Yaskawa GP4** “without further tuning.” The resulting system demonstrates sustainable repeated build-break-rebuild cycles, which is directly aligned with insertion, deletion, and repeated reconfiguration operations in a Lego-Edit workflow [2309.02354].

A second low-level advance is **Eye-in-Finger**, which embeds a low-cost, high-resolution endoscope camera directly in the LEGO-handling tool. This targets the cluttered, partially assembled regime where global or wrist-mounted cameras are heavily occluded. The perception pipeline uses a fine-tuned **YOLOv8-seg** model to segment visible studs and estimates planar misalignment from partially occluded masks by fitting circles with cost
\[
C(r) = \alpha \cdot \left(1 - \frac{|Mask \cap Circle|}{|Mask|} \right) + \beta \cdot \left|\mathbb{E}_{area}(z) - \pi r^2\right|,
\]
with empirically set coefficients
\[
\alpha = 3.5 \times 10^4,\qquad \beta = 1.
\]
Tilt after placement is estimated from LED-ring reflection using
\[
\begin{bmatrix} \theta_x \\ \theta_y \end{bmatrix}
=
\begin{bmatrix}
\arctan\left(\frac{d_x}{f_x}\right) \\
\arctan\left(\frac{d_y}{f_y}\right)
\end{bmatrix}.
\]
The method supports a **Peek \(\rightarrow\) Pick Up \(\rightarrow\) Place \(\rightarrow\) Inspection** loop rather than direct visual servoing during the final motion with a held brick [2503.06848].

Experimentally, Eye-in-Finger improves standard deviation error in **X** from **0.183 mm** to **0.024 mm** and in **Y** from **0.198 mm** to **0.019 mm**, with angular standard deviation improvements from **0.33^\circ** to **0.158^\circ** in yaw, **1.70^\circ** to **0.125^\circ** in pitch, and **3.63^\circ** to **0.139^\circ** in roll. Most notably, it increases calibration-error tolerance from about **0.4 mm in open loop to up to 2.0 mm with fingertip visual feedback**, and achieves **100\%** pickup and **100\%** place success up to **2.0 mm** injected error. This makes it a natural local correction and verification layer for editing an existing build, provided the coarse target estimate is already within the local field of view [2503.06848].

## 6. Structural representations and executable edit programs

Several papers provide representational substrates for Lego-Edit even when they do not implement editing directly. **Break and Make** introduces **LTRON**, an interactive 3D simulator in which an agent inspects, disassembles, and reconstructs LEGO assemblies using legal low-level actions over connection points. An assembly is represented as
\[
a = \{b_1, b_2, \hdots b_n\},\qquad
b_i = (s_i, c_i, R_i, t_i),
\]
with shape, color, rotation, and translation. Evaluation uses brick-inventory $F1_b$, full-assembly $F1_a$, assembly edit distance **AED**, and connectivity $F1_e$, making structural correctness explicit rather than purely visual [2207.13738].

**Learning to Build by Building Your Own Instructions** extends this line by replacing long-horizon latent memory with an explicit instruction stack. During Break, the agent disassembles a model and periodically saves images; during Make, it rebuilds the model by matching the current scene to the top instruction image and then popping that image once the state is reached. This converts reconstruction into short-horizon transitions between intermediate states. The resulting **InstructioNet** uses a modified Vision Transformer with two images—current scene and top instruction image—and a phase token, and outperforms LSTM and Studnet baselines on RC-2, RC-4, RC-8, and RC-Vehicles. The paper also shows a selective modification setting in which one color is replaced by another, with RC-2 performance of \(F1_a=0.95\) on the altered-color task, suggesting that explicit intermediate-state memory can support constrained modification as well as exact reconstruction [2410.01111].

**BrickNet** supplies a more symbolic program representation. It converts LDraw assemblies into a typed connectivity graph whose edges encode full local \(SE(3)\) transforms through connector semantics, then serializes a spanning tree as an executable build sequence. Its connector families are **Stud**, **Hinge**, **Axle**, **Ball**, and **Fixed**. The key claim is that “each edge defines the full local SE(3) transform between two paired parts,” so the full structure can be represented by a spanning tree of the graph. In unconditional generation, graph-backed sequences dramatically improve connectivity validity under nucleus sampling: for models from **0.6B** to **14B**, graph-backed validity is **94.1–97.0** successful build steps versus **31.8–49.9** for pose-based serialization, though collision-free validity remains much lower at **16.0–19.1** for graph-backed generation. This suggests that graph-backed serialization is highly promising for attachment-valid local edits, but still requires explicit geometric validation for collision-free editing [2604.22984].

A complementary graph-based line is **DGMLG**, which models LEGO assemblies as directed, labeled graphs whose nodes are bricks and whose edge labels are relative spatial offsets. Message passing is defined by
\[
\mathbf{a}_v = \sum\limits_{u:(u,v)\in E} f_e(\mathbf{h}_u, \mathbf{h}_v, \mathbf{s}_{u,v}),\qquad
\mathbf{h'}_v = f_n(\mathbf{a}_v, \mathbf{h}_v),
\]
and graph embedding by
\[
\mathbf{h}_G = \sum\limits_{v \in V}\mathbf{g}_v^G\odot\mathbf{h}_v^G.
\]
Its constrained variant **DGMLG-Re** reaches **100\%** validity, whereas the unconstrained generator reaches **25\%** validity but better distributional metrics. This is not an editor, but it provides a natural conditional-completion view of editing: preserve part of the graph, delete or perturb a local region, and regrow a valid replacement under graph constraints [2012.11543].

Finally, **String Diagrams for Assembly Planning** offers a compositional, hierarchical plan language in which subassemblies are wires, join operations are boxes, sequential composition is denoted by \(\cdot\), and parallel composition by \(\otimes\). For a LEGO connectivity graph \(G=(E,V)\), the paper proves the existence of a symmetric monoidal category whose morphisms are construction plans over subsets of \(V\). The practical value for Lego-Edit is the black-box treatment of subplans: a modified subassembly can plausibly be re-planned locally and then reinserted into a higher-level plan with the same interface, even though the paper itself addresses forward assembly planning rather than editing or disassembly [1909.10475].

## 7. Limitations, misconceptions, and open directions

Neither usage of Lego-Edit constitutes a universal editor. The image-editing framework is constrained by its current toolkit: the paper explicitly states that capability is limited by what the tool library can express, that robustness can still improve, and that failure cases can stem from weak individual tools such as inadequate segmentation for reflections. Its generalization mechanism depends on the quality of the critic in GT-free RL and on the continued modularity of the tool interfaces [2509.12883].

The robotic LEGO line is also not yet a full arbitrary-scene editing system. SaLfD is best suited to “demonstrate a complete build, then replay it,” not to infer abstract edit intent such as “change roof color” or “move this subassembly two studs left.” The framework assumes a bounded, discretized workspace, binary orientation, a fixed set of brick types, a predefined storage arrangement, and a largely environment-specific perception setup. It does not formulate explicit optimization over stability, reachability, ordering constraints, or inventory shortages beyond what the demonstration implicitly provides, and its disassembly mechanism is the reverse of assembly rather than a general selective disassembly planner. The earlier demonstration-driven temporal task graph similarly verifies operability in a digital twin but does not repair infeasible plans through symbolic replanning [2309.11010][2305.15667].

Structural understanding platforms expose additional bottlenecks. Break-and-Make reports that the hardest difficulty is “spatially-precise long term memory, and a large interactive action space,” not simple brick detection. InstructioNet reduces that burden through explicit instruction pages, but performance drops sharply under camera motion, and expert-generated instruction stacks substantially outperform model-generated ones on RC-Vehicles. BrickNet, by contrast, shows that graph-backed serialization nearly solves connectivity validity over long generations but leaves global collision avoidance largely unsolved. A plausible implication is that a practical physical Lego-Edit system will need both explicit intermediate-state memory and explicit graph-backed local constraints, rather than either alone [2207.13738][2410.01111][2604.22984].

The broader research trajectory points toward a composite system rather than a single method. One plausible synthesis is to combine demonstration-based task extraction, digital-twin verification, EOAT or Eye-in-Finger manipulation, explicit graph-backed or instruction-stack state, and localized plan repair with stronger checks for support, accessibility, and selective disassembly. That synthesis is not yet implemented as a single platform, but the component papers already define most of its technical primitives [2309.11010][2309.02354][2503.06848][2604.22984].

Source: https://www.emergentmind.com/topics/lego-edit