Lego-Edit: Modular Image & Robotic Editing

• Lego-Edit is a compositional framework that enables complex edits through modular workflows, operating both in image editing and robotic assembly domains.
• In the image-editing usage, a reasoning Builder orchestrates a library of specialized predictive and editing tools to execute multi-step instructions with high precision.
• In robotics, Lego-Edit leverages demonstration-based learning, digital-twin verification, and discrete brick operations to achieve efficient assembly and reversible disassembly.

Lego-Edit denotes two distinct but conceptually related research usages. In one usage, it is the name of a general instruction-based image editing framework that treats editing as workflow synthesis over model-level tools by a multimodal LLM. In another usage, suggested by recent robotic LEGO-construction literature, it denotes a broader technical agenda in which creating, modifying, correcting, assembling, and reversing LEGO structures are reduced to discrete brick operations, validated in simulation, and executed by robots. Across both usages, the “LEGO” metaphor refers to compositionality: complex behavior is assembled from smaller, typed units rather than produced by a single monolithic model (Jia et al., 16 Sep 2025, Liu et al., 2023).

1. Scope and naming

In the image-editing literature, Lego-Edit is a specific framework introduced for open-domain instruction-based image editing. Its defining claim is that real-world instructions are “immensely diverse,” and that a planner should compose specialized tools rather than rely on a single end-to-end editor. The system therefore separates a reasoning agent, called the Builder, from a library of Bricks and an Executor that runs the resulting workflow graph (Jia et al., 16 Sep 2025).

In robotic manipulation and assembly research, the term does not yet denote a single standardized platform. Instead, multiple papers explicitly analyze how a system like “Lego-Edit” could be built from existing components: demonstration-driven task extraction, digital-twin verification, robust assembly and one-piece disassembly, fingertip perception for precise local correction, and graph- or instruction-based structural representations. This suggests a broader, prospective meaning of Lego-Edit as a LEGO-structure editing stack rather than a finished unified system (Liu et al., 2023).

A common source of confusion is that these two usages are not about the same artifact. The image-editing framework operates on images and natural-language editing instructions. The robotic usage concerns physical LEGO structures, assembly traces, brickwise manipulation, and reversible build plans. The overlap is architectural rather than domain-specific: both favor modular decomposition, typed intermediate outputs, and explicit execution graphs over single-shot generation (Jia et al., 16 Sep 2025, Liu et al., 2023).

2. Lego-Edit as an instruction-based image editing framework

The image-editing framework formalizes instruction-based editing as planning and execution over typed tools. Given an image-instruction pair $(I,q)$, the Builder observes state $s=(I,q)$ and generates a reasoning trace and a JSON workflow according to policy $\pi_\theta(g \mid s)$. The workflow is a tool-invocation graph $g=(V,E)$, where nodes are selected tool instances and edges represent data dependencies. The Executor parses this graph and produces the edited image as $I' = D(g, I)$ (Jia et al., 16 Sep 2025).

Its tool library is divided into predictive tools and editing tools. Predictive tools include RES for text-prompted segmentation, SOS for salient-object segmentation, ADD-PRED for predicting a plausible region for object insertion, CAP-PRED for captioning and optional image expansion, INVERSE for mask or image inversion/subtraction, and BBOX for converting a mask to a bounding-box mask. Editing tools include FLUX-FILL for masked generation, FASTINPAINT and FLUX-INPAINT for removal-oriented inpainting, RCM for color or material replacement, STYLE for style conversion, ENV for environment modification, POSE for posture or expression changes, COMPOSE for combining masks or images, and RESIZE for scaling a valid mask or image region (Jia et al., 16 Sep 2025).

A central design decision is that the Bricks are model-level tools, not coarse pre-authored task workflows. This lets the Builder synthesize multi-step pipelines such as RES \rightarrow RCM for localized recoloring, RES \rightarrow FASTINPAINT \rightarrow INPAINT for object removal, CAP-PRED \rightarrow STYLE for style transfer with semantic preservation, and RES \rightarrow ADD-PRED \rightarrow FASTINPAINT \rightarrow INPAINT \rightarrow FILL for object replacement. The system enforces parameter traceability: every parameter supplied to a tool must come from the initial image, the user text, a previous tool output, or the Builder’s own language decomposition of the instruction (Jia et al., 16 Sep 2025).

The editing tools themselves are specialized rather than jointly multitask-trained. Lego-Edit adopts ICEdit’s setup built on FLUX-1 Fill and trains five specialized LoRA adapters at rank 32 for INPAINT, POSE, ENV, STYLE, and RCM. The Builder is based on MiMo-VL-7B. Predictive modules include EVF-SAM for RES, U2Net for SOS, and Qwen2-VL-2B for ADD-PRED and CAP-PRED. This specialization is explicitly motivated as a way to avoid “task confusion” (Jia et al., 16 Sep 2025).

3. Progressive reinforcement learning, evaluation, and extensibility

The Builder is trained in three stages. In Stage 1, supervised fine-tuning uses examples containing image $I$, instruction $q$, ground-truth reasoning trace, and ground-truth workflow, with target sequence $l=[Think_{\text{GT}}, g_{\text{GT}}]$ and loss

$$L_{\text{SFT}} = -\sum_{t=1}^{T} \log p_\theta(l_t \mid I, q, l_{<t}).$$

This stage teaches workflow syntax, tool interfaces, and initial domain-specific planning behavior (Jia et al., 16 Sep 2025).

Stages 2 and 3 use Group Relative Policy Optimization (GRPO). For sampled workflows $\{g_1,\dots,g_G\}$ with rewards $s=(I,q)$0, the relative advantage is

$s=(I,q)$1

Stage 2 combines a valid reward,

$s=(I,q)$2

with a similarity reward based on hierarchical graph matching against an expert workflow. Stage 3 removes the need for ground-truth workflows and replaces the similarity term with an effectiveness reward

$s=(I,q)$3

where an MLLM critic decomposes the generated workflow into editing chains, abstracts them into meta-edit descriptions, and judges whether chains should be added or removed to better satisfy the instruction (Jia et al., 16 Sep 2025).

The training corpus scales across supervision levels: 500 image-text pairs with instructions, thoughts, and workflows for Stage 1; 20K pairs with instructions and workflows for Stage 2; and 50K pairs with instructions only for Stage 3. All are sourced from OmniEdit. The Stage-3 critic is based on Qwen2.5-VL-72B. Training uses AdamW with learning rate $s=(I,q)$4, weight decay 0.01, batch size 8, input image size $s=(I,q)$5, one epoch per stage, and DeepSpeed ZeRO-3 on $s=(I,q)$6 NVIDIA H20 (Jia et al., 16 Sep 2025).

Empirically, Lego-Edit reports state-of-the-art results on both GEdit-Bench and ImgEdit-Bench / ImgBench. On GEdit-Bench it achieves

$s=(I,q)$7

and on ImgEdit-Bench it achieves best overall score $s=(I,q)$8, with particular strength on Hybrid Editing at $s=(I,q)$9. The Builder variant trained with GT-free RL reports Simple pass: 100\% and Complex pass: 99.0\%. Runtime on GEdit-Bench with one H20 GPU is reported as 3.5 s Builder latency, 2.7 s slowest tool latency, and \sim 7.2 s total pipeline latency (Jia et al., 16 Sep 2025).

A notable property is tool extensibility. The framework reports that newly introduced tools can be used without additional fine-tuning of the Builder. The paper illustrates this with reflection removal: an initial workflow using RES + INPAINT fails, user feedback leads the Builder to switch to SOS + INVERSE + INPAINT, and a newly introduced dedicated tool RRF can then be adopted immediately. This is presented as evidence that the Builder learns tool-interface reasoning rather than memorizing a closed workflow library (Jia et al., 16 Sep 2025).

4. Robotic LEGO editing through demonstration and simulation

The most direct robotic foundation for a physical Lego-Edit system is Simulation-aided Learning from Demonstration (SaLfD). The framework asks a human to construct a customized LEGO object on a baseplate, observes the demonstration with an RGB-D camera, extracts the sequence of brickwise operations, verifies candidate operations in a digital twin, and then executes the learned plan on a robot. Disassembly is obtained automatically by reversing the assembly sequence. The target structure is formalized as a set of bricks

$\pi_\theta(g \mid s)$0

with each brick encoded as

$\pi_\theta(g \mid s)$1

where $\pi_\theta(g \mid s)$2 is position, $\pi_\theta(g \mid s)$3 is brick type, and $\pi_\theta(g \mid s)$4 is orientation. The overall objective is to generate a structure $\pi_\theta(g \mid s)$5 identical to target $\pi_\theta(g \mid s)$6 by minimizing

$\pi_\theta(g \mid s)$7

and the brickwise recovery objective is decomposed as

$\pi_\theta(g \mid s)$8

The output of learning is therefore an ordered symbolic-geometric task sequence rather than a continuous control policy (Liu et al., 2023).

The hardware setup uses a FANUC LR Mate 200iD/7L robot and a RealSense L515 RGB-D camera mounted on the end effector, with one $\pi_\theta(g \mid s)$9 construction plate and one $g=(V,E)$0 storage plate. Keyframe detection uses a CNN with 3 convolution layers, each followed by ReLU and max-pooling, and then 3 fully connected layers. Operation extraction from consecutive RGB-D keyframes uses a second CNN whose outputs are brick ID, binary orientation, and position. Position is modeled with Gaussian uncertainty around mean

$g=(V,E)$1

and candidate confidence is

$g=(V,E)$2

Rather than committing to a single operation, the model returns a ranked candidate list (Liu et al., 2023).

Verification is performed in a ROS Gazebo digital twin. The simulator executes the highest-ranked candidate and compares the simulated end state with the real observed end state using a learned similarity score $g=(V,E)$3 from another CNN. If

$g=(V,E)$4

the candidate is accepted; otherwise the next candidate is tried. If no candidate exceeds the threshold, the final choice is

$g=(V,E)$5

This makes simulation a consistency filter for noisy RGB-D perception rather than a global symbolic planner (Liu et al., 2023).

This pipeline is directly relevant to LEGO editing because an edit can be represented as a localized sequence of brick operations $g=(V,E)$6, inserted into or removed from an assembly trace. The paper explicitly notes that replacing a brick can be represented as disassembling the old brick at position $g=(V,E)$7 and then assembling a new one at $g=(V,E)$8, and that partial disassembly could be realized by reversing a suffix or subtrace of the assembly plan. However, these are extrapolations from the framework rather than implemented features (Liu et al., 2023).

Quantitatively, SaLfD substantially improves exact recovery over learned extraction without simulation verification. Across objects such as AI, RI, Human, Chair, Spiral, Bridge, Pyramid, and Temple, SaLfD reaches 100\% on AI and Pyramid, 96.7\% on RI, 95.2\% on Temple, and 86.3\% on Bridge, consistently outperforming LfD without simulation verification. For plan generation time, AI {10 bricks} takes 66.2 s with SaLfD versus 168.6 s by handcrafting and 302.3 s by manual simulation authoring, while Temple {23 bricks} takes 133.5 s with SaLfD versus 333.4 s and 790.5 s. The earlier precursor paper represents the learned task as a temporal task graph

$g=(V,E)$9

and similarly derives disassembly by reversing the learned assembly graph (Liu et al., 2023, Liu et al., 2023).

5. Manipulation substrates: end-of-arm tooling and fingertip perception

A physical Lego-Edit system requires reliable brick insertion and one-piece disassembly, and this is addressed by a hardware-software co-design centered on a passive end-of-arm tool (EOAT) inspired by the LEGO brick separator. The EOAT reduces the manipulation problem from uncertain grasp-and-insert to an insert-and-twist primitive. The robot first inserts the target brick into the EOAT so that the tool frame becomes kinematically attached to the brick, then executes a twist about a task-defined pivot. Assembly and disassembly are parameterized mainly by twist angle $I' = D(g, I)$0, axis offsets $I' = D(g, I)$1, and task duration $I' = D(g, I)$2; parameter tuning is posed as

$I' = D(g, I)$3

with cost

$I' = D(g, I)$4

The parameters are optimized by covariance matrix adaptation evolution strategy (CMA-ES) directly on real robots (Liu et al., 2023).

The manipulation benchmark covers $I' = D(g, I)$5, $I' = D(g, I)$6, $I' = D(g, I)$7, and $I' = D(g, I)$8 bricks, heights from 1 to 10 layers, solid and hollow support structures, and 25 positions on a $I' = D(g, I)$9 LEGO plate. Before learning, assembly success is consistently 100\% and disassembly success is generally above 80\%. After safe learning, both assembly and disassembly reach $I$0 across all tested scenarios in the table. The learned parameters also transfer from FANUC LR-mate 200iD/7L to Yaskawa GP4 “without further tuning.” The resulting system demonstrates sustainable repeated build-break-rebuild cycles, which is directly aligned with insertion, deletion, and repeated reconfiguration operations in a Lego-Edit workflow (Liu et al., 2023).

A second low-level advance is Eye-in-Finger, which embeds a low-cost, high-resolution endoscope camera directly in the LEGO-handling tool. This targets the cluttered, partially assembled regime where global or wrist-mounted cameras are heavily occluded. The perception pipeline uses a fine-tuned YOLOv8-seg model to segment visible studs and estimates planar misalignment from partially occluded masks by fitting circles with cost

$I$1

with empirically set coefficients

$I$2

Tilt after placement is estimated from LED-ring reflection using

$I$3

The method supports a Peek $I$4 Pick Up $I$5 Place $I$6 Inspection loop rather than direct visual servoing during the final motion with a held brick (2503.06848).

Experimentally, Eye-in-Finger improves standard deviation error in X from 0.183 mm to 0.024 mm and in Y from 0.198 mm to 0.019 mm, with angular standard deviation improvements from 0.33^\circ to 0.158^\circ in yaw, 1.70^\circ to 0.125^\circ in pitch, and 3.63^\circ to 0.139^\circ in roll. Most notably, it increases calibration-error tolerance from about 0.4 mm in open loop to up to 2.0 mm with fingertip visual feedback, and achieves 100\% pickup and 100\% place success up to 2.0 mm injected error. This makes it a natural local correction and verification layer for editing an existing build, provided the coarse target estimate is already within the local field of view (2503.06848).

6. Structural representations and executable edit programs

Several papers provide representational substrates for Lego-Edit even when they do not implement editing directly. Break and Make introduces LTRON, an interactive 3D simulator in which an agent inspects, disassembles, and reconstructs LEGO assemblies using legal low-level actions over connection points. An assembly is represented as

$I$7

with shape, color, rotation, and translation. Evaluation uses brick-inventory $I$8, full-assembly $I$9, assembly edit distance AED, and connectivity $q$0, making structural correctness explicit rather than purely visual (Walsman et al., 2022).

Learning to Build by Building Your Own Instructions extends this line by replacing long-horizon latent memory with an explicit instruction stack. During Break, the agent disassembles a model and periodically saves images; during Make, it rebuilds the model by matching the current scene to the top instruction image and then popping that image once the state is reached. This converts reconstruction into short-horizon transitions between intermediate states. The resulting InstructioNet uses a modified Vision Transformer with two images—current scene and top instruction image—and a phase token, and outperforms LSTM and Studnet baselines on RC-2, RC-4, RC-8, and RC-Vehicles. The paper also shows a selective modification setting in which one color is replaced by another, with RC-2 performance of $q$1 on the altered-color task, suggesting that explicit intermediate-state memory can support constrained modification as well as exact reconstruction (Walsman et al., 2024).

BrickNet supplies a more symbolic program representation. It converts LDraw assemblies into a typed connectivity graph whose edges encode full local $q$2 transforms through connector semantics, then serializes a spanning tree as an executable build sequence. Its connector families are Stud, Hinge, Axle, Ball, and Fixed. The key claim is that “each edge defines the full local SE(3) transform between two paired parts,” so the full structure can be represented by a spanning tree of the graph. In unconditional generation, graph-backed sequences dramatically improve connectivity validity under nucleus sampling: for models from 0.6B to 14B, graph-backed validity is 94.1–97.0 successful build steps versus 31.8–49.9 for pose-based serialization, though collision-free validity remains much lower at 16.0–19.1 for graph-backed generation. This suggests that graph-backed serialization is highly promising for attachment-valid local edits, but still requires explicit geometric validation for collision-free editing (Kulits et al., 24 Apr 2026).

A complementary graph-based line is DGMLG, which models LEGO assemblies as directed, labeled graphs whose nodes are bricks and whose edge labels are relative spatial offsets. Message passing is defined by

$q$3

and graph embedding by

$q$4

Its constrained variant DGMLG-Re reaches 100\% validity, whereas the unconstrained generator reaches 25\% validity but better distributional metrics. This is not an editor, but it provides a natural conditional-completion view of editing: preserve part of the graph, delete or perturb a local region, and regrow a valid replacement under graph constraints (Thompson et al., 2020).

Finally, String Diagrams for Assembly Planning offers a compositional, hierarchical plan language in which subassemblies are wires, join operations are boxes, sequential composition is denoted by $q$5, and parallel composition by $q$6. For a LEGO connectivity graph $q$7, the paper proves the existence of a symmetric monoidal category whose morphisms are construction plans over subsets of $q$8. The practical value for Lego-Edit is the black-box treatment of subplans: a modified subassembly can plausibly be re-planned locally and then reinserted into a higher-level plan with the same interface, even though the paper itself addresses forward assembly planning rather than editing or disassembly (Master et al., 2019).

7. Limitations, misconceptions, and open directions

Neither usage of Lego-Edit constitutes a universal editor. The image-editing framework is constrained by its current toolkit: the paper explicitly states that capability is limited by what the tool library can express, that robustness can still improve, and that failure cases can stem from weak individual tools such as inadequate segmentation for reflections. Its generalization mechanism depends on the quality of the critic in GT-free RL and on the continued modularity of the tool interfaces (Jia et al., 16 Sep 2025).

The robotic LEGO line is also not yet a full arbitrary-scene editing system. SaLfD is best suited to “demonstrate a complete build, then replay it,” not to infer abstract edit intent such as “change roof color” or “move this subassembly two studs left.” The framework assumes a bounded, discretized workspace, binary orientation, a fixed set of brick types, a predefined storage arrangement, and a largely environment-specific perception setup. It does not formulate explicit optimization over stability, reachability, ordering constraints, or inventory shortages beyond what the demonstration implicitly provides, and its disassembly mechanism is the reverse of assembly rather than a general selective disassembly planner. The earlier demonstration-driven temporal task graph similarly verifies operability in a digital twin but does not repair infeasible plans through symbolic replanning (Liu et al., 2023, Liu et al., 2023).

Structural understanding platforms expose additional bottlenecks. Break-and-Make reports that the hardest difficulty is “spatially-precise long term memory, and a large interactive action space,” not simple brick detection. InstructioNet reduces that burden through explicit instruction pages, but performance drops sharply under camera motion, and expert-generated instruction stacks substantially outperform model-generated ones on RC-Vehicles. BrickNet, by contrast, shows that graph-backed serialization nearly solves connectivity validity over long generations but leaves global collision avoidance largely unsolved. A plausible implication is that a practical physical Lego-Edit system will need both explicit intermediate-state memory and explicit graph-backed local constraints, rather than either alone (Walsman et al., 2022, Walsman et al., 2024, Kulits et al., 24 Apr 2026).

The broader research trajectory points toward a composite system rather than a single method. One plausible synthesis is to combine demonstration-based task extraction, digital-twin verification, EOAT or Eye-in-Finger manipulation, explicit graph-backed or instruction-stack state, and localized plan repair with stronger checks for support, accessibility, and selective disassembly. That synthesis is not yet implemented as a single platform, but the component papers already define most of its technical primitives (Liu et al., 2023, Liu et al., 2023, 2503.06848, Kulits et al., 24 Apr 2026).