Tmax: A simple recipe for terminal agents
Abstract: Terminal-using agents have quickly become the most popular downstream application of LMs. Despite their prevalence, relatively little academic work has examined RL-based training of these models, likely due to difficult benchmarks, a lack of data, and a lack of simple baseline recipes. We present Tmax, the strongest open RL recipe for terminal agents to date, bringing open data recipes closer to the frontier. While simple, our recipe achieves 27\% on Terminal-Bench 2.0 with only 9B parameters, outperforming much larger models from prior work. Concretely, we generate data using a novel taxonomy, combining difficulty control, personas, and verifier diversification, which allows us to cheaply generate large amounts of terminal environments for RL and SFT training. We open-source our terminal dataset, which is over 2.5x larger than previously released terminal-agent datasets. We then train open-weight models using RL with our data, using a simple, outcome-only recipe. We release our data, models, and code as a strong baseline for future open academic work on terminal agents at https://github.com/hamishivi/tmax.
Summary
- The paper introduces TMAX, a novel recipe combining a synthetic RL dataset (TMAX-15K) and outcome-based RL fine-tuning to train terminal agents.
- The methodology leverages compositional sampling across nine axes to ensure diverse, balanced, and challenging task generation for terminal automation.
- Empirical results show that using DPPO with an FP32 LM head improves training stability and generalization, setting a new standard for open-weight models under 32B.
TMAX: A Simple Recipe for Training Terminal Agents
Overview
The paper "Tmax: A simple recipe for terminal agents" (2606.23321) introduces TMAX, a systematic methodology for constructing and training high-performance, open-weight terminal agents through a combination of principled dataset synthesis and outcome-based RL fine-tuning. The work addresses a core gap in the landscape of agentic LLMs—the lack of scalable, balanced datasets and robust RL recipes tuned for the complexities of terminal automation at scale. TMAX delivers a synthetic RL dataset (TMAX-15K) of unprecedented diversity and difficulty, post-training recipes that robustly benefit from RL, and open-source code artifacts, collectively advancing the open research frontier in agentic terminal tasks.
Compositional Synthetic Data Generation
A salient contribution is the TMAX-15K dataset which comprises 14,600 RL environment instances generated using a compositional pipeline. The pipeline samples hierarchically across nine well-curated axes (domain, skills, primitive skills, persona, language, task complexity, command complexity, fixture, verifier), affording explicit, fine-grained control over both task diversity and difficulty. Unlike prior work that depends on extensive teacher validation, TMAX leverages "soft filtering" during RL, enabling efficient, scalable data creation while preventing task-mode collapse or redundancy.
Figure 1: TMAX data generation pipeline, detailing the hierarchical sampling and efficient environment instantiation process.
Figure 2: Domain distribution for TMAX versus prior datasets, highlighting TMAX's balanced coverage across all axes.
The use of graded verifiers (continuous rewards), persona conditioning, and multimodal fixtures (e.g., tasks involving images, binaries, and audio processed via terminal tools) broadens the generalization capabilities of trained models without requiring model multi-modality, in line with real CLI application demands. Analytic metrics confirm TMAX-15K's superior diversity (domain balance: 0.998) and persistent task difficulty (lowest pass@8 among all datasets; see Figure 3).
Reinforcement Learning Post-Training Recipe
For RL fine-tuning, the work systematically investigates and advocates for Divergence Proximal Policy Optimization (DPPO) over GRPO, supported by FP32 precision on the LM head and large parallel rollout group sizes for training stability. This configuration is empirically demonstrated to curtail logprob mismatches and collapse in high-turn, long-horizon RL settings (see Figure 4 and Figure 5), a known challenge for terminal agents.
Figure 6: DPPO with FP32 LM head reduces logprob deviation, increasing RL stability.
Figure 5: DPPO versus GRPO—DPPO substantially limits training collapse on challenging RL data.
Further, the training effectively explores the complexity inherent in TMAX-15K, as evidenced by higher average step counts (Figure 7) and increased reasoning/tool-calling length per turn (Figure 8) as RL progresses.
Figure 7: Training on TMAX sustains higher per-task step counts, underscoring persistent task difficulty.
Empirical Results and Benchmarks
On Terminal-Bench 2.0—a de facto evaluation for agentic terminal tasks—TMAX achieves the strongest published results among open-weight models under 32B. The TMAX-9B checkpoint yields 27.2%, surpassing prior art even at larger model sizes (Table 16), and rivaling smaller closed offerings (e.g., Claude Haiku 4.5).
Figure 9: TMAX occupies the hardest band on pass@k metrics, maintaining difficulty even with multiple rollouts.
The RL recipe demonstrates robust generalization:
- Task generalization: Gains in non-terminal-agentic evaluations (e.g., +5.5 points on SWE-Bench-Verified, substantial gains on AIME).
- Harness generalization: RL improvements transfer across multiple evaluation harnesses, not just the training harness.
- Model family generalization: Qwen 3.5 and Qwen 3 8B both improve markedly, with largest absolute gains at moderate model scales (see Table 3).
Analysis of Training Stability and Practical Considerations
Despite successes, the TMAX training protocol reveals several stability hazards. For large multiturn RL (20+ steps per episode), synchronous/asynchronous batching, and harness design are nontrivial factors. The use of FP32 LM heads and DPPO is pivotal for stability, but high-resource infrastructure remains necessary, challenging the accessibility for smaller labs. Additionally, tuning for older SFT datasets can degrade performance on advanced models like Qwen 3.5, indicating the need for teacher calibration in SFT mix design.
Reward hacking is rare but observed, with models sometimes attempting to "stub out" tool requirements rather than genuinely solving complex tasks; however, such behaviors do not yield positive reward signal and are filtered by the RL pipeline.
Limitations and Future Work
A central limitation is the synthetic nature of the data—although the data generation is diverse and balanced, future progress may require leveraging data synthesis to improve on or transcend the generation model itself. Training instability also constrains the attainable upper bound of agentic performance, suggesting that architectural and infrastructural advances (larger context windows, more robust harnesses, and scalable sandboxing) are necessary for maximizing capability. Scaling to extremely large, industry-scale agents will remain challenging for academic groups until open RL infrastructure and orchestrators mature.
Conclusion
TMAX sets a new standard for open, reproducible research in terminal agent RL. Through rigorous compositional data generation and an optimized post-training pipeline, the approach defines a robust, scalable baseline for future academic and industrial work. Its release of high-quality data and code aims to catalyze progress on the open agentic RL frontier, where there is substantial headroom for improvements in both architecture and RL algorithmic design.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Explain it Like I'm 14
What is this paper about?
This paper is about teaching AI models to be better “terminal agents.” A terminal is the command-line window where you type commands to control a computer. A terminal agent is an AI that can read a problem, plan steps, and use the terminal to solve it—like installing software, fixing code, processing data, or running experiments.
The authors introduce TMAX, a simple, open recipe to train these agents so they work well even when the model is not huge. They share three things: a big training dataset (TMAX‑15K), a straightforward training method using reinforcement learning (RL), and open-source models and code so others can build on their work.
What questions were they trying to answer?
- Can we create a large, varied set of realistic terminal problems that teach an AI to handle tough, multi-step tasks?
- Can a simple, clear RL training recipe make smaller open models perform well as terminal agents?
- Will the skills learned from terminal training also help on other tasks and with different tool setups?
How did they do it? (Methods in plain language)
1) Building a big, varied practice set (TMAX‑15K)
Think of training like practice for a sport. The team built 14,600 “practice drills” (tasks) for the AI. Each drill is a small computer environment inside a container (like a clean, isolated mini-computer), created with a Docker image. The AI must solve a goal by using terminal commands.
To make these drills:
- They used a strong external AI to write tasks and code up the environments.
- They mixed and matched across nine “axes” (like domain, skills, complexity, and a “persona” for variety) so tasks don’t all look the same.
- They controlled difficulty carefully. Two separate “complexity” dials make tasks range from a few commands to long, multi-step workflows, and from simple shell use to combining bash, coding, and system services.
- They added different “fixtures” (extra files like images, audio, or binaries) the AI can inspect using command-line tools, so the model learns to use the terminal to analyze real inputs.
- They used smarter “verifiers” (checkers) that can give partial credit or thresholds (not just exact string matches), which helps training progress smoothly.
Instead of expensive human/teacher checks, they rely on “soft filtering”: during training, tasks that never yield any positive reward simply don’t affect the model, keeping the pipeline scalable.
2) Training the agent with reinforcement learning (RL)
Reinforcement learning is like coaching with feedback. The AI tries a solution; if it works (passes the checker), it gets a positive signal and adjusts to do better next time.
Key ideas they used:
- A stable RL method (called DPPO) that helps avoid training “crashes” on long, multi-step tasks.
- Technical care to keep the model’s calculation consistent during training and testing (they use a high-precision output layer and an inference setup called vLLM).
- A simple “harness” (the agent’s toolbox and prompting style) based on mini-SWE-agent, which lets the model think over multiple turns and call tools safely.
They trained open models at sizes like 2B, 4B, 9B, and 27B parameters (you can think of parameters as tiny adjustable knobs in the model’s brain—more usually means bigger capacity).
What did they find, and why does it matter?
Here are the main takeaways:
- Their 9B model (TMAX‑9B) is very strong for its size. On a tough standard test called Terminal‑Bench 2.0, it scores about 27%. That’s better than many larger open models and comparable to some smaller closed models. In short: strong small model, simple recipe.
- Their dataset helps more than others. Training on TMAX‑15K beats training on previous open datasets for terminal agents on multiple benchmarks. The balanced coverage across nine domains (like software engineering, security, data processing, and system administration) and calibrated difficulty seem to make the model more capable.
- The skills transfer. After RL training on terminal tasks, the model also improves on:
- SWE‑Bench Verified (real-world code bug fixing),
- AIME (math questions).
- This suggests the training teaches general problem-solving and tool-use skills, not just how to “game” one specific setup.
- It works across sizes and setups. Their method improves different-sized models and still helps when the agent uses different tool harnesses and prompts—not just the one used during training.
They also openly release the dataset, code, and models, so researchers can reproduce results and try new ideas.
Why is this important?
- Practical AI helpers: Many real AI tools need to install packages, run scripts, test code, and manage systems. Teaching models to handle terminals well makes them more useful in the real world.
- Strong baselines for research: TMAX gives a clear, reproducible starting point. That speeds up progress because others can focus on new ideas instead of reinventing the setup.
- Better small models: Not everyone can afford giant models. Showing that a 9B model can perform well helps make advanced tools more accessible.
- General skills, not just memorization: The fact that the model improves on other tasks (like coding fixes and math) means it learned deeper strategies—planning, using tools effectively, and sticking with longer tasks.
A note on challenges
Training terminal agents is tricky. Long, multi-step tasks can make RL unstable. The authors found ways to make it steadier (like DPPO, larger rollout groups, and careful precision settings), but there’s still room to improve the stability and speed of training.
In one short list: what TMAX contributes
- A big, balanced, difficulty‑aware terminal dataset (TMAX‑15K) with 14,600 tasks.
- A simple, open RL recipe that trains strong small models (like TMAX‑9B) and beats prior open approaches.
- Evidence that training on terminal tasks boosts skills across different tasks, tools, and model sizes.
- Fully open data, models, and code so others can build on this work.
Knowledge Gaps
Knowledge gaps, limitations, and open questions
Below is a single, focused list of concrete gaps that remain unresolved and could guide future work:
- Teacher dependence: The data pipeline relies on a strong, closed generator (Gemini-3-Pro); it remains untested whether student models can surpass the teacher on both Terminal-Bench and the generated TMAX tasks. Design experiments quantifying the teacher–student gap and how it varies with data scale and RL steps.
- Axis ablations: No ablation isolates the contribution of each generation axis (difficulty controls, personas, graded verifiers, multimodal fixtures). Run factorial ablations to measure per-axis impact on downstream performance and stability.
- Data quality audit: The “soft filtering” approach avoids upfront validation, but the true rates of non-executable tasks, flaky environments, and misleading instructions are unreported. Conduct a full executability and integrity audit with precise failure taxonomy and prevalence.
- Verifier robustness: The reliability of unit tests and graded verifiers (false positives/negatives, reward hacking susceptibility) is unmeasured. Fuzz-test verifiers, quantify exploitability, and harden reward checks.
- Continuous rewards: Graded verifiers are introduced but not evaluated for learning dynamics. Compare continuous vs binary rewards on sample efficiency, stability, and final accuracy in long-horizon settings.
- Persona effects: Persona diversification is untested. Measure whether personas improve robustness, instruction following, and cross-domain transfer, and identify harmful or bias-inducing personas.
- Multimodal fixtures impact: The prevalence of image/audio/video/binary fixtures and their influence on learning is unreported. Provide per-fixture performance breakdowns and test whether multimodal LMs or better terminal tooling materially help.
- Difficulty calibration validity: Difficulty buckets are not validated against human or external difficulty estimators. Calibrate complexity labels with human ratings or performance curves from multiple models.
- Data scaling laws: No analysis of how performance scales with dataset size, diversity, or complexity buckets. Establish scaling laws for task count, domain balance, and complexity vs returns.
- Domain weighting: Uniform domain sampling may misalign with real-world usage. Evaluate reweighting and curriculum strategies that mirror deployment distributions or cost constraints.
- Contamination checks: Only 13-gram overlap is reported. Add semantic and code-structure similarity checks, and assess potential indirect contamination via the teacher model.
- Synthetic–real mix: The pipeline is synthetic-only. Test mixing real-world terminal logs/repos and quantify gains vs risks (noise, brittleness, licensing).
- SFT warm-start diagnostics: SFT harms highly post-trained bases (Qwen 3.5), but helps smaller bases (Qwen 3). Develop diagnostics to predict when SFT helps vs hurts and design SFT mixtures tailored to over-post-trained models.
- Reward shaping and credit assignment: Outcome-only RL is used; intermediate rewards (e.g., command exit codes, partial unit-test credit, tool success signals) are unexplored. Compare reward designs for long-horizon credit assignment.
- RL stability root causes: Collapse after ~300 steps is observed but not isolated. Conduct controlled studies on sequence length, episode turn limits, KL schedules, value baselines, and optimizer/clip settings to identify primary drivers.
- Algorithm comparisons: DPPO outperforms GRPO here, but alternatives (e.g., V-trace, UPGO, advantage normalization, entropy bonuses, truncation schedules) are not tested. Benchmark across algorithms for stability and sample efficiency.
- Group size vs cost: Larger group size (32) improves stability but its cost–benefit trade-off is unquantified. Map performance and stability vs group size and rollout budgets.
- Training–inference drift: The FP32 head reduces logprob mismatches, but a general method to detect and mitigate drift across architectures/backends is missing. Develop automated drift diagnostics and mitigation recipes.
- Harness generalization factors: Gains across a few harnesses are shown, but the causal harness features (prompt structure, tool availability, memory format) that transfer are unknown. Systematically vary harness components to isolate what matters.
- Backend sensitivity: Daytona vs Podman differences affect scores, yet variance sources (I/O throughput, network, scheduler) are unquantified. Standardize a reproducible evaluation backend and report infra-adjusted metrics.
- Timeouts vs compute policy: Models increase token usage over training without adapting to per-task timeouts. Develop adaptive inference-time compute policies that trade off reasoning length with latency constraints.
- Context/tool-call limits: Effects of context length and max tool calls (64) are not ablated. Quantify performance and stability improvements vs increased limits and their compute costs.
- Cost–performance accounting: End-to-end cost (GPU-hours, dollars) vs returns for data generation, SFT, and RL is not reported. Provide cost curves and marginal gains to guide practitioners’ budgeting.
- Safety and misuse: Security personas/tasks are included without safety audits. Evaluate harmful capability acquisition, sandbox escape risks, and integrate guardrails/red-teaming into training/evaluation.
- Reproducibility of fixtures: Determinism and longevity of Docker builds and shipped artifacts (version pinning, mirrors) are not assessed. Measure rebuild success rates over time and publish pinning/mirroring policies.
- Verifier flakiness: Cross-run/backend flakiness (nondeterminism, intermittent timeouts) is not measured. Track flakiness metrics and implement seeding/retry protocols.
- Sampling bias from active filtering: Zero-variance and active sampling filters may skew the training distribution toward “easier/high-variance” tasks. Quantify distribution shift introduced by filters and its effect on generalization.
- Error analysis: No per-category failure analysis is provided (e.g., environment setup vs package management vs service orchestration). Publish failure modes to target data/algorithmic fixes.
- Teacher baselines: No head-to-head comparison with the generator on TMAX tasks and Terminal-Bench is reported. Measure where students match/surpass the teacher and analyze task characteristics enabling outperformance.
- Evaluation breadth: Transfer is shown on SWE-Bench Verified and AIME only. Expand to additional agentic/non-agentic suites (e.g., HumanEval+, Codeforces, NL2Bash variants) and quantify data efficiency of transfer.
- Ethical/legal: Licensing and redistribution of binaries/packages and security-related artifacts are not addressed. Audit licenses and define usage/redaction policies for dataset release.
- Beyond CLI: Web, GUI, and networked tool usage is not covered. Investigate extending the RL recipe to multi-tool, multi-environment agents.
- Curriculum learning: No staged training along difficulty axes is attempted. Test curricula from simple to complex and quantify effects on stability and convergence speed.
- Teacher-free/self-play generation: The viability of student-in-the-loop or self-play generation to reduce dependence on closed models is untested. Prototype iterative generation with evolving students.
- Long-run training operations: Early-warning signals and recovery (e.g., model averaging, checkpoint ensembles) for instability are absent. Develop monitoring and recovery strategies enabling runs beyond 500 steps.
- Benchmarking protocol: A standardized protocol to compare datasets under matched base models, harnesses, context limits, and compute budgets is missing. Propose and adopt a community baseline protocol.
Practical Applications
Immediate Applications
Below are actionable ways to apply the paper’s findings, methods, and assets (data, models, training recipe) today, with sector tie-ins, candidate tools/products/workflows, and feasibility notes.
- [Software/DevTools] Ship a small-footprint “Terminal Coding Copilot” inside IDEs and CLIs
- What: Integrate TMAX-9B (or TMAX-4B for lighter use) into VS Code/JetBrains or native terminals to automate environment setup, project scaffolding, dependency resolution, test running, log triage, and long-horizon CLI workflows.
- How: Use the released mini-SWE-agent-style harness; enable interleaved thinking and tool calls; containerize tasks during risky operations.
- Tools/products: “TMAX Copilot for Terminal,” a VS Code extension, or a shell plugin (zsh/bash/fish) with container-backed execution using Podman/Apptainer.
- Dependencies/assumptions: Container runtime available; network and package mirrors accessible; org-specific guardrails and permissioning; acceptance of ~9B model latency/throughput constraints.
- [Software/DevOps/MLOps] Automate CI triage and ephemeral environment reproduction
- What: Use TMAX agents to recreate failing CI jobs locally in containers, run stepwise diagnostics, apply patch suggestions, and re-run targeted tests.
- How: Adapt the Harbor/Daytona evaluation patterns to spin up reproducible sandboxes for each CI incident; drive the same harness used for RL rollouts.
- Tools/products: “ReproLab,” a CI bot that opens a container, replays steps, proposes fixes, and pushes PRs.
- Dependencies/assumptions: Access to build artifacts/logs; container orchestration budget; guardrails to prevent unsafe commands on shared runners.
- [Data Engineering] Self-serve CLI automation for ETL debugging and data checks
- What: Direct a terminal agent to locate broken pipelines, validate schema/row-level expectations, pin package versions, and regenerate partitions.
- How: Use graded verifiers (metric thresholds) to define success (e.g., null rate ≤ 0.5%, partition completeness ≥ 99%).
- Tools/products: “ETL Doctor” CLI agent with continuous verification using the paper’s graded verifier patterns.
- Dependencies/assumptions: Read-only credentials initially; well-specified verifier thresholds; ephemeral sandboxes to avoid data corruption.
- [Security] Red-team task generation and agent evaluation using persona diversification
- What: Generate realistic adversarial CLI tasks (e.g., log evasion detection, payload analysis), then evaluate internal agents or SOC playbooks.
- How: Use TMAX’s persona axis (e.g., “red-team operator”) and multi-modal fixtures (e.g., binaries, pcap files) accessed via terminal tooling.
- Tools/products: “PersonaGen for SecOps,” a task packer that emits Dockerized evals with graded/verifier gates (adversarial-corpus, fuzz-equivalence).
- Dependencies/assumptions: Clear scope and isolation; export controls and responsible use; verifiers designed to penalize unsafe actions.
- [Education] CLI tutor and labs with calibrated difficulty
- What: Teach Linux, Git, Python tooling, data wrangling, and compilers via curriculum-controlled tasks (5–60 command workflows).
- How: Map the paper’s task and command complexity axes to course modules; evaluate with graded verifiers instead of exact-match answers.
- Tools/products: “TMAX Lab Kits,” containerized exercises with instant feedback and step-by-step hints from a local TMAX-2B/4B.
- Dependencies/assumptions: Classroom compute or cloud credits; instructor-approved task bank; content moderation for shared images/files.
- [Academia/Research] Strong, reproducible baseline for agentic RL studies
- What: Use the open TMAX-15K environments, rollouts, logprobs, and DPPO recipe to reproduce and extend RL results for terminal agents.
- How: Start from Qwen 3.5/3.6 checkpoints; track stability with FP32 LM head and group size 32; swap harnesses to test generalization claims.
- Tools/products: “AgentRL Baseline Suite,” a cookiecutter repo bundling vLLM, Harbor, Podman/Apptainer, DPPO, and interleaved thinking templates.
- Dependencies/assumptions: Modest GPU cluster (e.g., 2 nodes train, 6 inference in the paper); ops for sandbox reliability; license compliance.
- [Quality/Benchmarking] Internal procurement and vendor evaluation of coding agents
- What: Evaluate third-party agents under Terminal-Bench and TMAX-15K-derived suites with consistent timeouts and sandbox backends.
- How: Adopt Harbor/Daytona-style harness with multi-run averaging and standardized prompts to avoid “harness-fitting.”
- Tools/products: “Agent RFP Bench,” a gated test harness for vendors with Pareto curves (accuracy vs. params/latency) like Fig. 1 methodology.
- Dependencies/assumptions: Budget for sandbox compute; reproducible backend selection (local Podman vs. hosted Daytona may affect timeouts).
- [Operations/IT] Safe, sandboxed command execution for helpdesk workflows
- What: Route ambiguous or risky CLI requests (driver installs, log scraping, service restarts) to an agent inside containers before issuing real changes.
- How: Use graded verifiers to certify the plan (dry-run success metrics) before escalation to production.
- Tools/products: “PreFlight CLI,” a change simulator with policy gating and human-in-the-loop approval.
- Dependencies/assumptions: Strong RBAC; immutable base images per domain; change management integration.
- [Content/Media Ops] Non-multimodal media handling via terminal tools
- What: Use terminal-only agents to OCR images/PDFs (tesseract), transcode or transcribe audio/video (ffmpeg), and run QA checks end-to-end.
- How: Leverage TMAX’s multi-modal fixtures pattern without a multimodal LLM—agent calls system tools and parses outputs.
- Tools/products: “MediaOps Agent,” a batch processor of ingest pipelines with pass/fail thresholds (e.g., bitrate, subtitle coverage).
- Dependencies/assumptions: Tool availability in images; quality verifiers defined; storage/IO quotas.
- [Model Engineering] RL stability and training diagnostics adopted into existing pipelines
- What: Add FP32 head, TV-divergence masking (DPPO), large group sizes, and logprob-drift monitors to stabilize agentic RL runs.
- How: Extend current PPO/GRPO stacks with the paper’s DPPO settings; monitor rollout step counts and per-turn token growth as health signals.
- Tools/products: “RL Stability Toolkit for LLMs,” shipping code to instrument mismatch spikes, collapse warning, and active sampling.
- Dependencies/assumptions: Access to trainer and inference logs; tolerance for slightly higher inference cost due to FP32 head.
- [Policy/Governance] Immediate guidance for decontamination and reproducible containerized evals
- What: Adopt 13-gram overlap checks, per-task timeouts, multi-run averages, and domain balance metrics when reviewing internal/third-party agents.
- How: Mirror paper’s decontamination protocol; publish harness details and backend choice in evaluation reports.
- Tools/products: “Agent Eval Governance Checklist,” templating contamination checks, domain/skill balance, and timeout parity.
- Dependencies/assumptions: Organizational buy-in; ability to release evaluation metadata; alignment with IP policies.
Long-Term Applications
These require further research, scaling, or engineering to reach robust, safe deployment.
- [Software/Platform] Autonomous “Harness-Agnostic” terminal agent platforms
- Vision: Agents that generalize across diverse harnesses, prompts, and tool ecosystems without performance drops (“no harness-fitting”).
- Path: Broaden training across multiple harnesses and container backends; meta-RL or domain randomization; richer verifier suites.
- Dependencies/assumptions: More stable long-horizon RL; scalable sandbox orchestration; standardized tool-call schemas.
- [Enterprise Engineering] Org-specialized terminal agents trained via curriculum/difficulty shaping
- Vision: Agents learn org-specific stacks (mono-repos, bespoke CLIs) with compositional task generation tuned to internal complexity profiles.
- Path: Extend TMAX axes with proprietary domains/skills; auto-curriculum that raises graded thresholds as mastery improves.
- Dependencies/assumptions: Secure synthetic generation using a strong teacher (frontier model or top-tier internal LLM); privacy-preserving data pipelines.
- [Security/Trust] Safety-verified terminal agents with adversarial verifiers and fail-safe policies
- Vision: Agents that are formally constrained by graded/adversarial verifiers, with provable guardrails against destructive commands.
- Path: Combine static/dynamic policy enforcers, capability whitelists, and multi-condition verifiers; RL with penalties for policy breaches.
- Dependencies/assumptions: Rich verifier libraries for side-effect detection; certified sandbox isolation; organizational risk appetite.
- [MLOps/Automation] Self-managing ML infrastructure agents
- Vision: Agents that set up training runs, monitor resource contention, auto-mitigate slowdowns, and tune containers based on observed load.
- Path: Incorporate the paper’s observations on infra-induced instability into reward shaping; add signals for queueing/IO/contention.
- Dependencies/assumptions: Deep integration with schedulers (K8s/SLURM); fine-grained telemetry; robust rollbacks.
- [Education at Scale] Personalized CLI curricula via persona-conditioned generation
- Vision: Courseware that adapts tasks to student background (personas), scaffolding from basic shell through complex multi-service stacks.
- Path: Extend persona axis and graded difficulty to track mastery; cross-course analytics for skill-type balance and learning gains.
- Dependencies/assumptions: Ethical persona design; institutional LMS integration; content moderation; measurement validity.
- [Benchmarking Ecosystem] Sector-specific terminal benchmarks with graded success
- Vision: Healthcare, finance, and telecom-specific CLI suites with metric-threshold verifiers (HIPAA redaction rates, SOX log-audit coverage).
- Path: Co-develop domains and verifiers with regulators/industry bodies; standardize reporting akin to Terminal-Bench.
- Dependencies/assumptions: Regulatory alignment; synthetic-but-representative data; strict decontamination and privacy constraints.
- [Agent Foundations] Stable, long-horizon RL for multi-turn tool use
- Vision: Algorithms and infrastructure that avoid collapse after 300+ steps and scale to longer contexts and more tools.
- Path: New trust-region variants beyond DPPO, curriculum over turn-count, KL schedules that preserve exploration, richer active sampling.
- Dependencies/assumptions: More compute; improved rollout throughput; better mismatch mitigation between trainer and inference.
- [DevSecOps] End-to-end change planning with pre-merge graded verifiers
- Vision: Agents that propose complex infra changes (service meshes, data migrations) with verifiable pre-merge success metrics and rollback plans.
- Path: Formalize verifier composition (multi-protocol checks, fuzz-equivalence on critical binaries); integrate with GitOps flows.
- Dependencies/assumptions: High-fidelity staging environments; auditor-approved verifier definitions; cultural shift to “verify-before-merge.”
- [Content/Media] Policy-compliant synthetic data and QA factories
- Vision: Large synthetic corpora for OCR/transcription/QA generated by persona-conditioned tasks with acceptance thresholds.
- Path: Automate task generation and graded acceptance; couple with human audit loops for bias/safety.
- Dependencies/assumptions: Strong teachers for initial synthesis; governance for synthetic data provenance; legal review.
- [Policy/Standards] Compliance frameworks for agentic coding systems
- Vision: Standards for disclosure of backends (Harbor/Daytona), timeouts, decontamination, and graded verifier specs in public benchmarks and RFPs.
- Path: Multi-stakeholder working groups; harmonize with software supply-chain security and AI transparency guidelines.
- Dependencies/assumptions: Industry/academic consensus; resource commitments to maintain shared task banks and verifiers.
- [Personal Computing] Safe local OS maintenance agents for non-experts
- Vision: Consumer agents that diagnose disk space, repair broken packages, configure drivers, and set up dev environments safely.
- Path: Ship curated task packs with strict verifiers and rollback; always run in local containers/VMs with limited host access.
- Dependencies/assumptions: Lightweight containerization on consumer OSes; UX for approvals; robust fallbacks when tools are unavailable.
Notes on cross-cutting assumptions and dependencies
- Synthetic generation quality: Many training benefits hinge on access to a strong generator (e.g., Gemini-3-Pro) and careful persona/axis design.
- Sandbox reliability/costs: Container backends (Podman/Apptainer, or hosted Daytona) shape throughput, timeout behavior, and cost; budget and ops maturity matter.
- RL stability: DPPO, FP32 LM head, and large group sizes mitigate but don’t eliminate collapse; longer runs and larger contexts will need further research.
- Safety/permissions: Production use should enforce strict RBAC, command allow-lists, graded/verifier gates, and human-in-the-loop review for high-impact actions.
- Licensing/IP: Ensure data and model licenses permit fine-tuning and deployment; maintain decontamination practices for fair evaluation.
Glossary
- Active sampling: A batching strategy that selects prompts or groups to ensure complete batches during RL training. "and use active sampling to ensure full batches"
- Adversarial-corpus: A verifier that accepts clean outputs and rejects malicious ones in security-oriented tasks. "adversarial-corpus (accept clean, reject malicious)"
- AIME: The American Invitational Mathematics Examination; used here as a non-agentic evaluation benchmark. "as well as non-agentic evalua- tions such as AIME."
- Apptainer: A container runtime used for sandbox management in experiments. "either a Podman or Apptainer backend for managing sandboxes."
- Asynchronous RL: Reinforcement learning where rollouts and updates occur asynchronously to increase throughput. "a fully asynchronous RL infrastructure"
- Balance score: A uniformity metric (based on entropy) used to assess how evenly a dataset covers categories. "We additionally compute a balance score for each categorical axis"
- Base image: A pre-built container image shared by tasks within a domain, extended with task-specific dependencies. "tasks within the same domain share the same base image"
- Continuous-valued rewards: Rewards that take real values rather than binary pass/fail, enabling graded feedback. "go beyond simple binary correctness checks to continuously-valued rewards for certain tasks."
- Contamination protocol: A standard procedure for detecting overlap between training data and evaluation benchmarks. "following standard contamination protocol"
- Curriculum: A training approach that schedules tasks from easier to harder to aid learning. "or induce a curriculum."
- Daytona: A managed sandbox backend used for Terminal-Bench evaluations. "Daytona sandboxes often perform installs faster than our locally-hosted runs"
- De-contamination: The process of verifying that training data does not contain benchmark content. "De-contamination We measure overlap"
- Docker image: A built container artifact encapsulating the environment for a task. "by building a Docker image per task."
- Dockerfile: A specification file that defines how to build a Docker image for a task environment. "into a Dockerfile, unit-test verifier, source files, and task instructions."
- DPPO (Divergence Proximal Policy Optimization): An RL algorithm that masks tokens based on divergence to improve stability; a variant of GRPO. "using Divergence Proximal Policy Optimization (DPPO) (Qi et al., 2026)"
- ffmpeg: A command-line multimedia toolkit used by agents to inspect audio/video artifacts via the terminal. "e.g., OCR, audio transcription, ffmpeg"
- Frontier model: A state-of-the-art external model used to synthesize challenging tasks. "relying on a strong frontier model to gen- erate useful environments for us."
- Fuzz-equivalence: A verifier that checks bit-exact equivalence against an oracle, often under input fuzzing. "fuzz-equivalence (bit-exact match against an or- acle)"
- GRPO: Group Relative Policy Optimization, an RL algorithm for LMs that can be unstable in long-horizon settings. "which is a variant of GRPO (Shao et al., 2024)"
- Harbor: A framework for evaluating and optimizing agents and models in container environments. "We use Harbor (Harbor Framework Team, 2026)"
- Interleaved thinking: An agent prompting style that preserves thought content across turns to improve performance. "often called 'interleaved thinking'"
- Inference-time reasoning scaling: The effect where allowing more tokens at inference improves reasoning quality. "This is reminiscent of inference-time reasoning scaling in single-turn math settings"
- KL penalty: A regularizer that penalizes divergence between policies to stabilize RL training. "a small KL penalty during training"
- Logprobs: Log-probabilities of tokens produced by the model, used for divergence checks and losses. "difference between inference (vLLM) and trainer (HuggingFace) logprobs"
- Mask tokens: An RL stabilization technique that ignores tokens during training when distributions diverge. "which masks tokens when inference and training logprobs deviate."
- Metric-threshold: A verifier that accepts outputs meeting a numeric criterion (e.g., accuracy). "metric-threshold (e.g., accuracy ≥ 0.95)"
- mini-SWE-agent: A lightweight terminal-agent harness used for rollouts and evaluation. "we use a simple harness based on mini-SWE-agent"
- Multi-protocol: A verifier that tests behavior through multiple protocol-level requests to services. "multi-protocol (protocol-level requests against a service)."
- Multi-service compose stack: A task fixture bundling multiple services in a single composed stack. "or multi- service compose stack (Table 10)."
- Multiturn: Refers to tasks requiring many agent turns or actions rather than single exchanges. "the highly multiturn nature of terminal tasks"
- OCR: Optical Character Recognition; used from the terminal to read text from images. "e.g., OCR, audio transcription, ffmpeg"
- Open-instruct: An open-source framework extended here for terminal-agent RL training. "based on open-instruct (Lambert et al., 2025)."
- Open-weight: Models whose weights are publicly released for use and finetuning. "We then train open-weight models"
- Pareto curve: The set of non-dominated trade-offs (e.g., performance vs. size) in a comparison plot. "dominates the Pareto curve"
- Pass@k: A success metric indicating whether a task is solved within k attempts. "pass@1 is 42% (vs. 41-92% for prior datasets), and it has the lowest pass@4 (50%) and pass@8 (53%)"
- Persona: A user profile used to condition task generation for greater diversity. "Persona diversification."
- Podman: A daemonless container engine used to run isolated sandboxes locally. "either a Podman or Apptainer backend"
- PPO: Proximal Policy Optimization, a baseline RL algorithm for policy updates. "using a simple harness and PPO over limited con- text lengths (16k tokens)."
- Rollouts: Generated trajectories of agent actions and observations used for training or evaluation. "For rollouts, we use the same mini-SWE-agent-based harness"
- Sandboxing infrastructure: Systems that isolate and manage containerized environments for tasks. "running sandboxing infrastructure can be expensive and slow"
- Seed tasks: Initial examples that guide synthetic task generation via expansion. "seed tasks"
- SFT (Supervised Fine-Tuning): Supervised post-training of a model before RL to improve stability or performance. "validates their approach only using SFT training"
- Soft filtering: Training-time removal of low-signal samples (e.g., zero-reward groups) instead of expensive pre-validation. "applies effec- tive soft filtering."
- Taxonomy-plus-seed recipe: A data synthesis approach combining category taxonomies with seed tasks. "a similar taxonomy-plus-seed recipe"
- Teacher generation: Using a stronger model to generate or validate training data. "validate task quality and correctness through expensive teacher generation"
- Terminal-Bench: A benchmark suite for evaluating terminal agents on realistic, hard tasks. "Terminal-Bench 2.0"
- Terminal-Bench Lite: A lighter-weight, high-signal subset/variant of Terminal-Bench used for faster iteration. "Terminal- Bench Lite (OpenThoughts-Agent team, 2026)."
- Terminus-2: A terminal interaction harness that requires raw keystroke inputs. "We find that Terminus-2 harness (Merrill et al., 2026) is more brittle"
- Token-level loss: An RL objective computed at the token level rather than only on sequences. "We use a token level loss (Yu et al., 2025)"
- Total variation (TV) divergence: A distance measure between distributions used to decide when to mask tokens. "total variation (TV) divergence."
- Unit-test verifier: Automated tests used to check if a task has been solved correctly. "unit-test verifier"
- vLLM: A high-throughput inference engine used to serve LLMs during rollouts. "we use vLLM (Kwon et al., 2023) for rollouts"
Collections
Sign up for free to add this paper to one or more collections.
