Terminal-Lego: Terminal-Agent Task Pipeline

• Terminal-Lego is a paradigm that bridges terminal interfaces with structured LEGO worlds, enabling discrete, verifiable state transitions across virtual and physical domains.
• The system employs a Docker-verified pipeline that transforms over 36K StackOverflow questions into environment-validated terminal tasks using rigorous testing and validation measures.
• It supports diverse applications—from code-to-artifact transformations and robotic RL platforms to agent-based control—emphasizing environment-grounded supervision and precise terminal manipulation.

Terminal-Lego denotes, most explicitly, a “scalable pipeline that converts large-scale StackOverflow-grounded issues into Docker-verified tasks spanning 90+ domains, establishing a controlled substrate of environment-verified agentic terminal tasks for training and evaluating terminal agents” (Yang et al., 2 Jun 2026). In adjacent literature, the same expression also functions as a technical label for systems in which text-based programs, terminal-like APIs, or agent trajectories culminate in LEGO artifacts, LEGO simulators, or LEGO robots. This broader usage suggests that Terminal-Lego is best understood not as a single software package, but as a family of interfaces in which symbolic interaction—source code, shell commands, BDI actions, or low-level assembly primitives—is bound to a structured LEGO world, whether virtual, physical, or environment-verified (Winter et al., 2016, Walsman et al., 2022, Dittert et al., 2024).

1. Scope and principal meanings

Across the literature, Terminal-Lego appears in several technically distinct but structurally related forms.

Manifestation	Terminal interface	Source
Docker-verified terminal-agent substrate	Terminus-2 headless terminal with environment-verified tasks	(Yang et al., 2 Jun 2026)
Code-to-artifact system	SML programs run “from a terminal or IDE” and exported to LEGO or voxel viewers	(Winter et al., 2016, Winter, 2014)
BDI robot terminal	Jason agents linked to LEGO Mindstorms NXT bricks through LEGOAgArchitecture	(Jensen, 2010)
Interactive assembly simulator	Gym-style environment with Pick, Assemble, Disassemble, and camera actions	(Walsman et al., 2022)
Real-world RL platform	PyBricks and TorchRL expose LEGO robots as gym-like environments over BLE	(Dittert et al., 2024)
Terminal manipulation benchmark	Fingertip visual servoing for the “last few mm/degrees” of LEGO alignment	(2503.06848)

The narrowest definition is the 2026 terminal-agent pipeline. The broader technical usage is inferential but well supported by the surrounding papers: Bricklayer is described as essentially a “terminal-to-LEGO” system; Jason-based LEGO agents treat the brick as a terminal or node for BDI reasoning; LTRON exposes LEGO assembly through an OpenAI Gym interface; and BricksRL makes LEGO robots programmable from Python scripts and terminal-launched training loops. A plausible implication is that Terminal-Lego names a recurrent interface pattern: a discrete, inspectable environment in which symbolic control and verifiable state transitions are central.

2. Terminal-Lego as a Docker-verified task-generation pipeline

In its most explicit form, Terminal-Lego is a six-phase pipeline built for terminal-agent post-training and evaluation (Yang et al., 2 Jun 2026). It begins with StackOverflow crawling over approximately 36k questions with accepted answers across 98 tags covering about 90+ domains. Tag allocation uses the formula

$$\text{tag\_target} = \min\left(\left\lfloor \frac{w_i}{\sum_j w_j} \cdot \text{TARGET\_COUNT} \cdot 1.5 \right\rfloor,\ \text{remaining}\right).$$

Questions are filtered to require an accepted answer and deduplicated by question ID.

Task construction is performed by a cascade centered on Claude Opus 4.6. The cascade generates, in order, instruction.md, an environment JSON, solution/solve.sh, a difficulty label, tests (test.sh and test_outputs.py), a test-review loop, and a Dockerfile. The instruction stage requires Linux-terminal tasks rooted at /app/task_file/, with explicit input paths, output paths, and success criteria. The solution script includes a mandatory canary string. Tests are designed to verify only the post-solution state: test_outputs.py must not run solve.sh, and the review loop checks for brittle path comparisons, syntax errors, inconsistent values, and other failure modes before regeneration.

Docker round-trip validation is then executed end to end: build the image, run a container, copy in solve.sh, execute it, copy in the tests, run test.sh, and read /logs/verifier/reward.txt. Only tasks with reward 1 are retained. The validation funnel begins with 36,846 candidates and retains 15,389 passed tasks, a 41.8% pass rate. Recorded failures include 8,859 test-or-solution failures, 4,254 build failures, 8,317 timeouts, 18 runtime errors, and 9 missing solutions. The resulting corpus is renumbered as task_00000, task_00001, and so forth, with a generation_summary.json describing difficulty statistics and mappings.

What distinguishes this pipeline from generic synthetic benchmark generation is the insistence on environment closure. Each task contains an instruction, environment, reference solution, tests, and Dockerfile that have already survived the same execution path later used for teacher rollout and student evaluation. This makes Terminal-Lego a controlled substrate for matched-task comparisons among terminal agents rather than merely a repository of prompts.

3. Environment-grounded supervision and the pedagogical paradox

The central empirical result associated with Terminal-Lego is the “pedagogical paradox” (Yang et al., 2 Jun 2026). Under matched-task distillation, Claude Opus 4.6 attains the highest standalone Terminal-Bench 2.0 score, 69.4, yet students fine-tuned on its 8.1k solved trajectories are the weakest among the compared teachers. DeepSeek-V3.2 attains only 39.3 standalone, but its trajectories produce the strongest students: Qwen3-8B reaches 10.5% and Qwen3-32B reaches 20.6% on Terminal-Bench 2.0.

The paper attributes this to Environment-Grounded Supervision, or EGS. In this formulation, a trajectory is pedagogically useful when it explicitly exposes the inspect–act–verify loop through harness-visible shell interactions. A teacher should inspect files and directories, act by editing or running programs, and then verify the resulting state. The key quantitative proxy is the Targeted Observation Ratio,

$$\mathrm{TOR} = \frac{ |\{a \in \mathcal{A} : \exists o \in \mathcal{O},\ o \prec a \land \mathrm{align}(o, a)\}| }{ |\mathcal{A}| }.$$

Here, $\mathcal{A}$ denotes action commands, $\mathcal{O}$ denotes observation commands such as cat, ls, find, stat, file, which, grep, head, wc, and diff, $o \prec a$ means the observation precedes the action, and align(o,a) requires path alignment.

The teacher comparison is striking. Claude Opus 4.6 solved trajectories have TOR 2.5%; Qwen3.5-Plus has 6.5%; GLM-5 has 7.3%; and DeepSeek-V3.2 has 13.4%. Student performance increases with TOR rather than with teacher standalone score. DeepSeek failed trajectories are also informative: 2.5k failed trajectories with TOR 14.1% still train a Qwen3-32B student to 16.1%, exceeding the 15.4% obtained from 8.1k passed Claude trajectories. The paper further shows that high-TOR subsets outperform low-TOR subsets on matched tasks, and that masking observation-only turns during training sharply reduces performance. For Qwen3-32B, masking observation turns in DeepSeek trajectories reduces Terminal-Bench 2.0 score from 20.6% to 13.8%; masking targeted observations hurts more than masking untargeted observations.

The same logic appears in teacher prompting. When Claude Opus 4.6 is instructed to inspect the environment “as thoroughly as possible” before modifying files, its TOR rises from 2.5% to 6.6%; the resulting Qwen3-32B student improves from 15.4% to 19.5%; and Claude’s own inference pass rate on 15k Terminal-Lego tasks rises from 88.7% to 95.4%. The paper’s broader claim is therefore methodological: terminal-agent post-training depends less on teacher outcome quality than on whether the harness makes environment-facing reasoning visible and learnable.

4. Program-to-artifact systems: Bricklayer as terminal-to-LEGO

Long before the 2026 pipeline, Bricklayer provided a concrete realization of Terminal-Lego in the sense of text-based artifact construction (Winter et al., 2016). Bricklayer is a freely available online educational ecosystem whose programs are written in SML and, when executed, create 2D and 3D artifacts viewable in LEGO Digital Designer, LDraw, Minecraft clients, Brickr, or STereoLithography viewers. Its core workflow is explicit: write a program in SML, run it from a terminal or IDE, generate an artifact in a virtual 2D or 3D grid, and export that artifact to LDD, LDraw, Brickr, Minecraft through RaspberryJuice, or STL. The paper describes Bricklayer as essentially a “terminal-to-LEGO” system already.

The computational model is coordinate-centric. A Bricklayer artifact can be read as a function

$f : \mathbb{Z}^n \to C,$

with $n=2$ for 2D, $n=3$ for 3D, and $C$ a set of brick types. Coding levels structure access to this space: Levels 1–3 expose 2D primitives, Levels 4–5 introduce 3D primitives, and advanced users gain predicates, iterators, and brick functions. The system’s pedagogical “low-threshold infinite ceiling” design begins with single-brick placement and progresses toward full functional abstractions, fractals, Minecraft, LEGO, and 3D printing. A typical pattern is build2D, put2D, and show2D, with traverseWithin later enabling region-wise application of coordinate-to-brick functions.

The earlier Bricklayer paper provides the foundational API account (Winter, 2014). Its abstractions are organized into four SML structures: Predicate, BrickFunction, BasicNavigation, and AdvancedNavigation. The key types are point, predicate : point -> bool, and brick_function : point -> brick_type. Generic traversal hides iteration: Predicate.show traverses a cube and places a single brick type wherever the predicate is true, while BrickFunction.show allows the brick type itself to vary by point. BasicNavigation adds explicit space allocation with build, side-effectful access and update, and region traversals such as traverseXYZ, traverseXY, and traverseWithin. AdvancedNavigation exposes first-class virtual spaces and 3D turtle graphics suitable for L-systems, cellular automata, and Turing machines.

Bricklayer also defines the limits of this modality. LDD is described as struggling with more than 25K pieces, sometimes up to about 60K but not reliably; LDraw handles up to about 250K bricks; and Minecraft artifacts can reach about 450K blocks. Bricklayer’s internal model is voxel-like and “legoized,” so physically robust multi-stud LEGO builds require a post-processing step through Brickr. Even so, the system establishes a canonical Terminal-Lego template: text code, functional abstraction over integer coordinates, an intermediate virtual world, and multiple export paths to visualization or fabrication.

5. Embodied terminal interfaces: agent terminals, RL platforms, and last-centimeter alignment

A second cluster of work makes LEGO itself the terminal hardware of an agent or control stack. In Jason-based multi-agent systems, each logical AgentSpeak agent is paired with a physical LEGO Mindstorms NXT robot, and the NXT brick functions as the agent’s terminal or node (Jensen, 2010). The architecture extends only Jason’s environment interface: LEGOAgArchitecture overrides perceive() and act(), while a custom belief base UniqueBelsBB keeps only the most recent percept for sensor schemas such as light(port,_), sound(port,_), obstacle(port,_), and touching(port,_). Actions such as forward, backward, rotate, reverse, speed, stop, and exit are translated into LeJOS motor commands and sent over Bluetooth. The design is synchronous rather than asynchronous, which preserves Jason cycle semantics but introduces a high delay in the reasoning cycle; as a result, sensor-dependent robots must move quite slowly.

BricksRL shifts the same idea into reinforcement learning (Dittert et al., 2024). It couples PyBricks on LEGO hubs with TorchRL on a laptop through Bluetooth Low Energy, using a client.py script on the hub, a PyBricksHubClass on the host, and BaseEnv subclasses that define action_spec, observation_spec, reset, step, and task-specific reward functions. The platform demonstrates TD3, SAC, and DroQ on tasks such as RunAway-v0, Spinning-v0, Walker-v0, WalkerSim-v0, RoboArm-v0, RoboArmSim-v0, and RoboArm-mixed-v0. The end-to-end BricksRL loop runs at about 11 Hz, although the paper reports that 2 Hz produced faster and more stable Walker training, analogous to frame skipping. Real-world training times are typically under 120 minutes on a normal laptop, and the system supports non-LEGO sensors, exemplified by a USB webcam integrated into the RoboArm-mixed environment. In this setting, Terminal-Lego becomes a Python-and-terminal workflow for defining a robot, an environment, and an RL algorithm over standardized specs and TensorDict interchange.

Eye-in-Finger reframes “terminal” as the last few millimeters of assembly and disassembly (2503.06848). The end-effector embeds a low-cost USB endoscope camera inside a LEGO-specific tool head whose contact geometry snaps onto studs, ensuring that perception and mechanical interaction share the same terminal geometry. The perception stack uses a fine-tuned YOLOv8-seg model for knob segmentation, a circle-reconstruction procedure for partially occluded studs, and a fine-tuned YOLOv11 detector for LED reflection used in tilt estimation. The system’s discrete visual-servoing loop follows a Peek–Pick Up–Place–Inspect structure. Quantitatively, it increases the tolerance of calibration error from 0.4 mm to up to 2.0 mm, achieves X/Y standard errors of about 0.024 mm and 0.019 mm compared with about 0.183 mm and 0.198 mm for a wrist-mounted Realsense D405, and reports pitch and roll standard errors of 0.125° and 0.139°. Here Terminal-Lego is not a shell interface but a terminal manipulation benchmark: accuracy and robustness are concentrated at the end-effector contact site.

6. Simulation, graph generation, and assembly-state representations

LTRON and “Break and Make” provide a simulator-centered interpretation of Terminal-Lego (Walsman et al., 2022). The task is split into a Break phase, in which an agent visually inspects and disassembles a previously unseen target assembly, and a Make phase, in which the environment is reset and the agent must rebuild the assembly from scratch using only the history of observations and the same low-level action primitives. An assembly is represented as a set of brick instances

$a = \{b_1, b_2, \dots, b_n\},$

with

$$\mathrm{TOR} = \frac{ |\{a \in \mathcal{A} : \exists o \in \mathcal{O},\ o \prec a \land \mathrm{align}(o, a)\}| }{ |\mathcal{A}| }.$$0

where $$\mathrm{TOR} = \frac{ |\{a \in \mathcal{A} : \exists o \in \mathcal{O},\ o \prec a \land \mathrm{align}(o, a)\}| }{ |\mathcal{A}| }.$$1 is a shape ID, $$\mathrm{TOR} = \frac{ |\{a \in \mathcal{A} : \exists o \in \mathcal{O},\ o \prec a \land \mathrm{align}(o, a)\}| }{ |\mathcal{A}| }.$$2 a color ID, $$\mathrm{TOR} = \frac{ |\{a \in \mathcal{A} : \exists o \in \mathcal{O},\ o \prec a \land \mathrm{align}(o, a)\}| }{ |\mathcal{A}| }.$$3 a rotation, and $$\mathrm{TOR} = \frac{ |\{a \in \mathcal{A} : \exists o \in \mathcal{O},\ o \prec a \land \mathrm{align}(o, a)\}| }{ |\mathcal{A}| }.$$4 a translation. LTRON uses polygon meshes from LDraw, connection metadata from LDCad, collision checking, and two workspaces: a table workspace rendered at $$\mathrm{TOR} = \frac{ |\{a \in \mathcal{A} : \exists o \in \mathcal{O},\ o \prec a \land \mathrm{align}(o, a)\}| }{ |\mathcal{A}| }.$$5 and a hand workspace rendered at $$\mathrm{TOR} = \frac{ |\{a \in \mathcal{A} : \exists o \in \mathcal{O},\ o \prec a \land \mathrm{align}(o, a)\}| }{ |\mathcal{A}| }.$$6. Actions include Disassemble, Assemble, Pick, Rotate Brick, Rotate Camera, Switch Phase, and End Episode. The simulator is exposed through an OpenAI Gym interface and supports headless EGL rendering.

The dataset side of LTRON is equally important. The Open Model Repository source contains about 1,727 fan-made reproductions, roughly 1,790 shapes after canonicalization, and 98 colors. These models are converted into connected components and greedily sliced into compact 2-, 4-, and 8-brick scenes; the paper reports 136,072 train and 2,000 test scenes for 2-brick OMR slices, 61,514 train and 2,000 test for 4-brick, and 28,094 train and 2,000 test for 8-brick. Evaluation uses brick-level $$\mathrm{TOR} = \frac{ |\{a \in \mathcal{A} : \exists o \in \mathcal{O},\ o \prec a \land \mathrm{align}(o, a)\}| }{ |\mathcal{A}| }.$$7, pose-sensitive assembly-level $$\mathrm{TOR} = \frac{ |\{a \in \mathcal{A} : \exists o \in \mathcal{O},\ o \prec a \land \mathrm{align}(o, a)\}| }{ |\mathcal{A}| }.$$8, assembly edit distance, and edge-level $$\mathrm{TOR} = \frac{ |\{a \in \mathcal{A} : \exists o \in \mathcal{O},\ o \prec a \land \mathrm{align}(o, a)\}| }{ |\mathcal{A}| }.$$9. On Random Construction, StudNet-B reaches $\mathcal{A}$0, $\mathcal{A}$1, $\mathcal{A}$2, and AED $\mathcal{A}$3 on 2-brick scenes, but performance drops sharply by 8 bricks; on OMR slices, StudNet-B reaches only $\mathcal{A}$4 and $\mathcal{A}$5 at 8 bricks. The technical significance is that Terminal-Lego can be instantiated as a partially observable MDP over low-level LEGO manipulation, but long-horizon planning, occlusion, and large part vocabularies remain severe bottlenecks.

A complementary representation appears in “Building LEGO Using Deep Generative Models of Graphs” (Thompson et al., 2020). There, the terminal object is a directed, labeled graph

$\mathcal{A}$6

whose nodes are bricks and whose directed edges encode physical support relations and 2D offsets. DGMLG is an autoregressive deep generative model of graphs derived from DGMG: it repeatedly decides whether to add a node or terminate, whether and how to add edges incident to the new node, which existing node to connect to, and which $\mathcal{A}$7 offset to assign. Termination is explicit in the Add node module, so the final LEGO structure is a terminal graph in the literal sense of a completed, absorbing state of the generative process. The paper evaluates generated graphs with GIN Accuracy, Fréchet Distance, Kernel Distance, Precision, Recall, Density, Coverage, novelty, and validity. DGMLG yields 60.5% GIN Accuracy and 25% valid graphs; the restricted DGMLG-Re yields 50.5% GIN Accuracy and 100% valid graphs. This line of work suggests that Terminal-Lego can also be formalized as sequential graph construction whose terminal condition is learned, verified, and evaluated independently of any physical interface.

7. Limitations and prospective directions

The literature identifies a consistent set of technical constraints. In Bricklayer, the principal bottlenecks are viewer limits, voxel granularity, and the fact that real LEGO builds require post-processing through Brickr for structural soundness (Winter et al., 2016). In Jason-based LEGO agents, the chief limitation is the high delay in the reasoning cycle when the agent is combined with a robot, making percept-dependent control slow and motivating asynchronous alternatives (Jensen, 2010). Eye-in-Finger has only a small field of view and still requires an approximate initial estimate of target location and height; its LED-reflection tilt method depends on the shiny LEGO surface and built-in illumination (2503.06848). BricksRL operates at about 11 Hz end to end, and the paper emphasizes hardware imprecision, backlash, and sensor noise as intrinsic constraints of LEGO robots (Dittert et al., 2024). LTRON exposes combinatorial explosion, partial observability, and long-horizon credit assignment, while its learned agents still struggle at 8-brick OMR reconstructions (Walsman et al., 2022). DGMLG is restricted to 2×4 bricks in its experiments and enforces primarily geometric rather than structural validity; physically fragile graphs can still be admissible (Thompson et al., 2020). The terminal-agent pipeline itself is not presented as complete: TOR is described as a practical proxy rather than a proof of logical necessity, and the reported student study is limited to 8B and 32B models (Yang et al., 2 Jun 2026).

At the same time, the papers converge on several future directions. Bricklayer points toward richer brick libraries, optimization routines, tighter viewer integration, and more direct build-instruction export; LTRON points toward hierarchical planners, better expert data, and scalable architectures for larger assemblies; BricksRL points toward harder locomotion and manipulation tasks, imitation learning, offline RL, and improved sim-to-real; Eye-in-Finger suggests combining global perception with local refinement and extending the smart-tool design beyond LEGO; and the 2026 terminal-agent work shifts emphasis toward “Harness Engineering,” where environment design, observability, and verification structure determine whether trajectories teach robust routines or only brittle action sequences. Taken together, these works indicate that Terminal-Lego is less a single benchmark than a technical paradigm: discrete environments, explicit state transitions, and verifiable terminal interactions organized so that symbolic agents can inspect, act, and verify against a LEGO-mediated world.