JIT-Agent: Scaling Harness Intelligence via Just-in-Time Harness Evolution
Abstract: Agent capability is not determined by the model alone. The agent harness, encompassing memory management, planning strategy, action protocol, and tool/skill orchestration, can dominate the contribution of the underlying foundation model. Yet harness design remains manual, task-specific, and fundamentally unscalable. We present JIT-Agent, a harness intelligence model trained to synthesize task-adaptive agent harnesses on the fly for arbitrary off-the-shelf agentic LLMs. We formalize the agent harness as a composable, machine-generatable artifact governed by a fixed four-module protocol, and train JIT-Agent to customize harnesses for a given task at hand, repair harnesses for stable and reliable execution, and self-evolve by distilling performance signals from an expanding archive of prior harness configurations. Equipped with JIT-Agent as a harness helper, DeepSeek-V4-Flash surpasses GPT-5.6 on DeepSearchQA (+9.1) and OdysseyBench (+4.3), while the already strong GLM-5.2 gains up to +20.2 points. Across controlled evaluations, JIT-Agent-generated harnesses are performance-competitive with mature agent runtimes such as OpenCode and Claude Code and consistently improve multi-scale model families of DeepSeek V4, Mimo-V2.5, and Qwen3.6. To our knowledge, JIT-Agent is the first model purpose-built for just-in-time harness generation, establishing harness intelligence as a trainable, transferable, and compounding dimension of agent capability orthogonal to model scaling.
First 10 authors:
Summary
- The paper demonstrates the JIT-Agent meta-agent, which generates, repairs, and evolves special harnesses for given LLM tasks, improving average performance by 7.7-8.8 points on key deep learning benchmark tests.
- JIT-Agent splits agent architectures into contrasted four-modules: memory, planning, action, and capability orchestration, enabling greater adaptivity, reliability, and evolvability of agent behavior
- The study highlights that the harness customization procedure significantly reduces latency as well as cost, by up to 36% on specific model–performance benchmarks, by exploiting ongoing task-conditioned synthesis across tasks and across model families.
The paper frames agent capability as a property of the model–harness pair, rather than of model parameters alone. Its central claim is that memory management, planning, action execution, and capability orchestration are first-order determinants of performance, and that these components can be generated dynamically for individual tasks. “JIT-Agent: Scaling Harness Intelligence via Just-in-Time Harness Evolution” (2608.25593) consequently proposes a meta-agent that synthesizes, repairs, and evolves executable agent harnesses around frozen, off-the-shelf LLM backbones.
Problem formulation and conceptual contribution
Most existing approaches to agent-system optimization are ahead-of-time (AOT): a persistent scaffold is manually designed or optimized over an experience distribution and then reused across subsequent tasks. This assumption is appropriate when task distributions and deployment environments are stable, but it is poorly matched to heterogeneous workloads in which research, planning, coding, browsing, and workspace manipulation impose different requirements on state representation and control flow.
JIT-Agent instead implements a Model-as-a-Harness paradigm. Given a task specification, protocol, capability registry, and retrieved examples of prior harnesses, the model emits an executable harness specialized to that task. The resulting harness wraps a separate agentic LLM, which remains the executor. This separation is important: JIT-Agent does not primarily improve the backbone’s weights or directly solve the task; it changes the computational organization through which the backbone observes state, forms directives, invokes tools, and advances execution.
The paper defines “harness intelligence” through three properties:
- Adaptivity: matching the operational scaffold to the task and backbone.
- Reliability: producing executable code and recovering from synthesis failures.
- Evolvability: using execution outcomes to improve subsequent harness proposals.
The design is deliberately narrower than a production runtime. The authors acknowledge that systems such as Claude Code, Codex, and DeepSeek Harness expose richer mechanisms. The objective is therefore not to reproduce complete commercial runtimes, but to establish whether a compact, typed, generative harness space can produce systematic gains.
A unified executable harness space
JIT-Agent represents every harness as a four-module tuple:
where is memory, is planning, is action, and is capability orchestration. At runtime, these modules interact in the order memory planning capability selection action.
Memory maps immutable event history and mutable controller state into a context view. Planning converts the task and this view into a local directive. Capability orchestration selects a task-relevant subset of available tools or skills. Action then updates controller state and emits either an executable call or a terminal output. A shared execution kernel interprets tool calls and appends resulting observations to the trajectory.
This factorization makes otherwise heterogeneous agent architectures comparable. ReAct is represented as full-history memory, no explicit planner, a ReAct action loop, and a full capability registry. Plan-and-Execute introduces a roadmap planner; recursive agents introduce decomposition, context isolation, recursive execution, and routing. The representation thus captures both conventional sequential loops and more structured graph-based or recursive systems.
HarnessFactory operationalizes the design space by reimplementing 13 scaffolds, including ReAct, Plan-and-Execute, ReSum, Flash-Searcher, General Agentic Memory, MemoBrain, AggAgent, OAgent, AgentFold, HiAgent, DeepAgent, ROMA, and AOrchestra. These implementations form an initial bank of executable harnesses. The bank serves both as a source of demonstrations for synthesis and as an archive of incumbent designs during evolutionary optimization.
The crucial architectural distinction is that JIT-Agent does not merely combine pre-existing modules. It instantiates new module implementations under a common protocol. Consequently, task specialization can alter the data structures, execution topology, verification policy, and state transitions of the harness itself.
Figure 1: JIT-Agent instantiates task-specific memory, planning, action, and capability modules under a shared executable protocol.
Three-stage training procedure
The training objective evaluates a generated harness through the trajectory induced by a frozen executor, measuring task reward together with latency and monetary cost. Generation occurs in an unconstrained program space, after which static validation, protocol checks, and runtime execution determine whether a candidate is usable.
Stage I: Task-conditioned customization
Stage I uses a stronger teacher to generate protocol-compliant harnesses conditioned on the task, capability registry, protocol, and three task-type-matched reference harnesses from the seed bank. Only candidates that pass validation and execution checks enter the accepted corpus.
The stage combines supervised fine-tuning with preference optimization. Supervision teaches the generator to emit structurally valid implementations. Preference learning favors a candidate only when it improves reward, does not worsen latency or cost, and strictly improves at least one efficiency dimension. This is more restrictive than scalar reward optimization: a candidate that increases accuracy by substantially increasing cost is not automatically preferred.
The design therefore treats effectiveness and efficiency as partially decoupled objectives. This choice is reflected throughout the paper’s evaluation, where token consumption and API cost are reported alongside benchmark scores.
Stage II: Bounded repair
Invalid generations are not discarded. Their compiler errors, interface mismatches, tool-call failures, and runtime exceptions are converted into structured repair trajectories. A teacher proposes patches, which are deterministically applied to the current harness, and the revised implementation is revalidated.
Training retains only failures that become executable within two repair rounds. This bounded regime makes the learned behavior deployment-relevant: JIT-Agent is trained to correct locally recoverable defects rather than redesign arbitrary failed programs. The restriction is also a limitation. It excludes failures requiring substantial architectural restructuring, so the reported reliability mechanism does not establish robustness to open-ended synthesis errors.
Figure 2: The training pipeline progresses from customization to bounded repair and frontier-based evolutionary improvement.
Stage III: Evo-GDPO
Stage III trains online evolution through Evolutionary Group-Decoupled Policy Optimization (Evo-GDPO). At each round, JIT-Agent retrieves incumbent harnesses from the archive, samples a group of candidates, executes them under the same backbone, budget, and evaluation seeds, and compares them with the current incumbent.
The reward channels are separated into task reward, latency advantage, and cost advantage. Reward receives primary weight. Latency and cost bonuses activate only when the candidate preserves or exceeds incumbent task reward. Each channel is normalized independently before aggregation, preventing the numerical scale of one metric from dominating the others. A PPO-style clipped objective, together with a KL penalty against the Stage-II checkpoint, updates the generator.
Archive updates are conservative: a candidate is retained only if it reaches the current reward frontier and strictly improves at least one frontier dimension. This mechanism prevents indiscriminate accumulation of low-quality harnesses and gives streaming inference a retrieval corpus of non-dominated designs.
The resulting procedure differs from ordinary group-relative policy optimization. Candidates are not rewarded merely for outperforming other samples in the current group; they must improve upon prior archive states. The optimization target is therefore cumulative frontier advancement.
Empirical evaluation
The evaluation covers nine benchmarks spanning deep research, daily work, long-horizon planning, and workspace execution. The primary backbones are GLM-5.2 and DeepSeek-V4-Flash, with additional tests on Qwen3.6 and Mimo-V2.5 variants. The study compares JIT-generated harnesses with vanilla backbone configurations, ReAct, and fixed agent runtimes including Claude Code, Codex, OpenCode, Hermes, and NanoBot.
The strongest result is the consistency of within-backbone improvement. Across all 18 matched comparisons for GLM-5.2 and DeepSeek-V4-Flash, replacing the default scaffold with a JIT-generated harness improves benchmark performance.
| Backbone | Vanilla average | JIT average | Absolute gain |
|---|---|---|---|
| GLM-5.2 | 74.1 | 81.8 | +7.7 |
| DeepSeek-V4-Flash | 66.7 | 75.5 | +8.8 |
The largest task-specific improvements occur in settings requiring persistent state and constraint tracking. DeepSeek-V4-Flash improves by 24.8 points on DeepPlanning-Shopping, from 59.1 to 83.9. GLM-5.2 improves by 20.2 points on DeepPlanning-Travel, from 62.8 to 83.0. DeepSeek-V4-Flash also gains 11.9 points on xBench-DeepSearch and 8.9 points on DeepSearchQA.
These results support the paper’s claim that the intervention is not merely prompt variation. The harness changes context curation, decomposition, state persistence, verification, and action coordination. However, the evaluation does not isolate the contribution of every generated module on every benchmark; the aggregate gains establish the value of the full harness-generation procedure more directly than the causal importance of individual components.
Figure 3: JIT-generated harnesses improve backbone agents across research, daily work, planning, and workspace benchmarks.
The JIT-equipped systems attain the best reported result in eight of the nine benchmark columns. JIT-Agent with GLM-5.2 ranks first on seven benchmarks, including DeepSearchQA at 93.9, AgentIF at 69.9, and PinchBench at 93.3. JIT-Agent with DeepSeek-V4-Flash leads DeepPlanning-Shopping at 83.9. DeepPlanning-Travel is the sole benchmark not led by a JIT-equipped system: GLM-5.2 with JIT-Agent reaches 83.0, 1.9 points below GPT-5.6.
The paper’s stronger comparative claim is that harness adaptation can recover a substantial portion of the advantage normally attributed to backbone scaling. This interpretation is plausible under the matched comparisons, but the end-to-end leaderboard should not be read as a pure model-size comparison: the systems differ in training provenance, tool environments, backbone implementations, and likely evaluation infrastructure.
Controlled comparison with fixed harnesses
The most informative experiment holds the backbone fixed and varies only the harness. On DeepSeek-V4-Flash, JIT-Agent achieves 85.1 on DeepSearchQA and 82.0 on xBench-DS, exceeding the strongest fixed-harness alternatives by 4.7 and 4.0 points, respectively. On Qwen3.6-Flash, it reaches 70.0 on xBench-DS and 58.3 on AgentIF, improving over the strongest fixed harnesses by 7.0 and 2.9 points.
JIT-Agent does not dominate every task in raw accuracy. Claude Code exceeds it on DeepSeek-V4-Flash AgentIF by 3.1 points, while NanoBot exceeds it on Qwen3.6-Flash DeepSearchQA by 3.9 points. These exceptions are consequential because they demonstrate that JIT synthesis is not uniformly superior to mature fixed runtimes. Its advantage is conditional on task structure and operating point.
The efficiency results are more consistent. JIT-Agent uses the fewest tokens and lowest API cost in all six controlled settings. Relative to the cheapest fixed harness for each setting, it reduces cost by 14.9% to 54.1%, with an average reduction of 36.0%.
| Backbone and task | JIT performance | JIT tokens | JIT cost | Comparison |
|---|---|---|---|---|
| DeepSeek-V4-Flash, DeepSearchQA | 85.1 | 400K | $0.066 | Best fixed: 80.4,$0.131 | |
| DeepSeek-V4-Flash, xBench-DS | 82.0 | 212K | $0.039 | Best fixed: 78.0,$0.075 | |
| Qwen3.6-Flash, AgentIF | 58.3 | 394K | 00.170 |
On DeepSeek-V4-Flash xBench-DS, the generated harness simultaneously improves performance from 78.0 to 82.0 while reducing cost from 10.039. On Qwen3.6-Flash AgentIF, it improves the best fixed-harness score from 55.4 to 58.3 while using less than half the cost of the cheapest fixed alternative. Thus, the reported gains are not explained by longer trajectories or larger token budgets.
Figure 4: JIT-generated harnesses shift several backbone–harness pairs toward stronger cost–performance Pareto frontiers.
The Pareto analysis makes the trade-offs explicit. On DeepSearchQA, DeepSeek-V4-Flash with JIT-Agent improves over NanoBot by 4.7 points while cutting cost by 49.6%. For Qwen3.6-Flash, JIT-Agent sacrifices 3.9 points relative to NanoBot but reduces cost by 51.8%, yielding a distinct low-cost operating point. On AgentIF with Qwen3.6-Flash, JIT-Agent strictly dominates the fixed alternatives in the reported comparison.
This pattern supports the authors’ narrower efficiency claim: task-adaptive scaffolds can improve orchestration selectivity rather than simply increasing interaction volume. It does not establish that JIT-generated harnesses are globally optimal; it establishes that they occupy favorable measured operating points within the evaluated candidate set.
Transfer across model families
The paper evaluates six backbones from DeepSeek-V4, Qwen3.6, and Mimo-V2.5, comparing JIT-generated harnesses with ReAct under matched conditions. Across 24 comparisons, JIT-Agent improves performance in every case, with an average gain of 7.6 points.
Average gains differ by model family:
| Model family | Average JIT gain |
|---|---|
| DeepSeek V4 | +10.2 |
| Qwen 3.6 | +4.0 |
| Mimo 2.5 | +8.6 |
DeepSearchQA shows the largest average gain at 15.2 points, including a 22.2-point improvement for Mimo-V2.5-Pro and a 19.0-point improvement for DeepSeek-V4-Flash. DeepPlanning-Shopping improves by 7.5 points on average, with the same 24.8-point gain for DeepSeek-V4-Flash observed in the broader evaluation.
Figure 5: JIT-generated harnesses outperform ReAct across paired model families and variants.
The consistency across model pairs supports transferability of harness intelligence. JIT-Agent was trained from Qwen3.6-27B but is deployed around other model families, indicating that the learned mapping concerns operational organization rather than a narrow compatibility with its training backbone. The result depends, however, on the shared protocol and compatible tool interfaces. Transfer to arbitrary models, APIs, multimodal executors, or substantially different action semantics remains untested.
Streaming harness evolution
Static inference generates task-specific harnesses independently. Streaming inference retains successful harnesses in an archive and retrieves them for subsequent tasks. The paper compares the two modes over task streams in DeepPlanning-Shopping, DeepPlanning-Travel, and OfficeBench.
Streaming JIT ends with higher cumulative accuracy on all three streams. The advantage emerges progressively as execution feedback accumulates, while API cost and tool-call trajectories remain task-dependent and broadly similar in scale.
Figure 6: Streaming JIT improves cumulative accuracy over independent static generation without a uniform increase in cost or tool use.
The result provides evidence for compounding archive-based improvement, but its interpretation requires care. Streaming inference does not update JIT-Agent’s parameters during deployment; it updates only the harness bank. Therefore, the observed improvement is attributable to retrieval and conservative retention of executable harness designs, not online gradient adaptation. The stream order and task similarity are also material assumptions: the benefit may be smaller when successive tasks are unrelated or when archive retrieval is poorly calibrated.
Qualitative structure of generated harnesses
The visualizations show that JIT-Agent changes executable topology and typed state, not merely textual instructions.
For a cross-application contact-processing task, the generated Palimpsest harness constructs a DAG of discovery, schema inspection, filtering, normalization, workbook creation, verification, and email delivery. Bounded-width graph execution ensures that artifact dependencies and commit order are explicit.
Figure 7: Palimpsest uses DAG planning, bounded graph execution, and artifact-indexed memory for a multi-application production task.
For a multi-hop identity question, Trapdoor synthesizes a bounded delegation capability. A parent orchestrator launches a research subagent with private memory, research-specific tools, and a five-step budget, then writes extracted facts into a fact graph. This is materially different from the artifact-oriented DAG: uncertainty and branching evidence collection induce recursive delegation instead of fixed dependency execution.
Figure 8: Trapdoor uses bounded recursive delegation and fact-graph memory for multi-hop research.
The additional harnesses extend this pattern. Origami folds active context while retaining complete subtask trajectories; Turnstile blocks final answering until evidence requirements are satisfied; Gearbox changes exposed tools and memory schemas through a shared phase register; Pegboard represents research as an evidence matrix; Appraiser renders only high-value observations; Abacus transfers computation into typed state; Player Piano performs deterministic file-by-file verification; and Mulligan regenerates failed actions locally while preserving successful history.
These examples substantiate the paper’s claim that the protocol constrains interfaces rather than behavior. They also expose a key engineering premise: the utility of generated harnesses depends on the correctness and security of the runtime that executes arbitrary synthesized modules.
Limitations and open questions
The paper’s principal assumption is that the relevant harness space can be represented by four protocol-compatible modules. This factorization is useful and expressive, but it is not shown to be complete. Production runtimes may require additional dimensions, including permissions, concurrency control, model routing, observability, safety policies, human escalation, and environment-specific transaction semantics.
The training corpus is also selectively filtered. Stage I retains executable teacher generations, and Stage II retains only failures repaired within two rounds. This improves training signal quality but introduces selection bias toward tasks and errors that are syntactically or locally repairable. The reported reliability therefore does not characterize failure modes requiring new tools, new abstractions, or substantial architectural changes.
The evaluation relies on benchmark subsets for several matched experiments: DeepSearchQA uses 100 examples, while the other three ReAct comparisons use 50 examples. The paper reports strong and consistent effects, but confidence intervals, statistical significance tests, seed sensitivity, and detailed per-task variance are not provided in the supplied content. The magnitude of individual gains—especially 20-point-plus improvements—therefore warrants replication under larger, independently controlled test sets.
The frontier objective also depends on measured reward, latency, and cost. Benchmark reward may not capture factuality, safety, artifact validity, or user utility uniformly across domains. In addition, archive retention can preserve systematic errors if evaluation feedback is noisy or if a harness exploits benchmark-specific regularities. A remaining question is whether Evo-GDPO improves out-of-distribution task performance or primarily optimizes the protocol and benchmark distribution used during training and evaluation.
Finally, the comparison with proprietary or mature runtimes is operationally difficult to interpret without fully standardized tool registries, model versions, system prompts, retry policies, and infrastructure. The controlled backbone comparisons are stronger evidence for harness effects than the headline comparisons against frontier models.
Conclusion
JIT-Agent presents a coherent formulation of harness intelligence as a learned capability for task-conditioned synthesis, bounded repair, and archive-based evolution. Its four-module protocol converts harness generation from unconstrained program synthesis into typed executable construction, while Evo-GDPO supplies a mechanism for optimizing reward, latency, and cost against incumbent designs.
The empirical evidence is substantial within the paper’s evaluation regime: JIT-generated harnesses improve all 18 matched GLM-5.2 and DeepSeek-V4-Flash comparisons, yield average gains of 7.7 and 8.8 points respectively, improve six additional backbone–benchmark comparisons over ReAct by an average of 7.6 points, and reduce controlled API cost by an average of 36.0%. The results support the paper’s central conclusion that operational scaffolding is an independent and transferable source of agent capability. The unresolved issue is how well this conclusion survives richer runtime requirements, broader distributions, and failures outside the bounded protocol and repair regimes used here.
Paper Prompts
Sign up for free to create and run prompts on this paper.
Top Community Prompts
Explain it Like I'm 14
1. What is the paper about?
This paper introduces JIT-Agent, a system designed to improve AI agents by changing not only the LLM, but also the way the model works.
A LLM is like the “brain” of an AI agent. However, the agent also needs a harness: a set of rules and tools that tells it how to remember information, make plans, use tools, and check its work.
The paper’s main idea is:
Instead of using one permanent harness for every task, let an AI create a special harness for each task when it is needed.
This is why the system is called Just-in-Time, or JIT.
For example, searching the web, writing computer code, and planning a large project are very different activities. Each one may work better with a different way of remembering information, planning steps, and using tools.
2. What questions does the research ask?
The researchers focus on three main questions:
- Can an AI create a useful harness for a specific task? In other words, can it choose the right memory system, planning method, tools, and action process?
- Can it repair a harness when something goes wrong? A generated harness might contain errors, use a tool incorrectly, or fail while running. Can the system diagnose and fix these problems?
- Can it improve its harnesses over time? If the system learns that one harness works better than another, can it save that knowledge and create even better harnesses in the future?
The researchers call this overall ability harness intelligence. They define it as the ability to:
- adapt a harness to a task,
- make sure the harness works reliably, and
- improve the harness using feedback.
3. How did the researchers approach the problem?
What is an AI harness?
Imagine giving a student a difficult assignment. The student’s success depends not only on their intelligence, but also on the support system around them:
- notes that help them remember important facts,
- a plan for completing the assignment,
- tools such as a calculator or search engine,
- rules for checking whether the answer is correct.
An AI harness provides similar support to a LLM.
The paper divides a harness into four main parts:
| Harness part | Simple meaning |
|---|---|
| Memory | Decides what information from earlier steps should be remembered |
| Planning | Breaks the task into goals and decides what should happen next |
| Action | Controls the agent’s step-by-step behavior |
| Capability orchestration | Chooses which tools, programs, or skills the agent can use |
These parts work together in a loop. The agent remembers what has happened, makes a small plan, chooses useful tools, takes an action, observes the result, and then repeats the process.
HarnessFactory
The researchers built HarnessFactory, a common software system for creating and testing different harness designs.
They recreated 13 existing styles of AI agents, including systems based on:
- simple reasoning-and-action loops,
- planning before acting,
- special memory systems,
- multiple agents working together,
- recursive systems that create smaller sub-agents.
This gave JIT-Agent many examples of possible harness designs to study.
The three training stages
JIT-Agent was trained in three stages.
Stage 1: Learning to customize
First, a stronger “teacher” AI created harnesses for different tasks. JIT-Agent studied these examples.
This taught JIT-Agent to connect the task with a suitable design. For instance:
- a research task might need strong memory for evidence,
- a coding task might need access to files, tests, and a terminal,
- a planning task might need a detailed list of smaller goals.
The researchers also compared harnesses based on three measures:
- reward: how well the task was completed,
- latency: how long it took,
- cost: how much computing or money it used.
This is similar to training a student not only to get the right answer, but also to solve the problem quickly and efficiently.
Stage 2: Learning to repair
Some generated harnesses did not work. They might have had:
- programming errors,
- mismatched interfaces,
- failed tool calls,
- problems during execution.
Instead of throwing these failures away, the researchers used them as repair examples. JIT-Agent learned to read the error report and make a small correction.
The system was trained to fix a problem within at most two repair attempts in this stage.
Stage 3: Learning to evolve
Finally, JIT-Agent compared newly created harnesses with older successful harnesses stored in an archive.
A new harness was kept if it:
- performed at least as well as the old one,
- used less time or money when possible, or
- achieved a better task result.
This is similar to natural selection: successful designs are kept, while weaker designs are replaced. Over time, the archive becomes a collection of increasingly strong harnesses.
The training method used for this stage, called Evo-GDPO, gives separate scores for task performance, speed, and cost. This prevents one measurement from unfairly overpowering the others.
4. What did the researchers find?
According to the paper, JIT-Agent improved the performance of several different LLMs and tasks.
The most important reported results include:
- DeepSeek-V4-Flash with JIT-Agent performed better than GPT-5.6 on several benchmarks.
- It reportedly improved DeepSearchQA by 9.1 points.
- It improved PinchBench by 8.7 points.
- It improved OdysseyBench by 4.3 points.
- On the already powerful GLM-5.2 model, it produced gains of up to 20.2 points on some evaluations.
- JIT-Agent-generated harnesses performed similarly to established agent systems such as OpenCode and Claude Code.
- Improvements appeared across several model families, including DeepSeek V4, Mimo-V2.5, and Qwen3.6.
These results suggest that improving the way an AI operates can be almost as important as making the underlying LLM larger or more powerful.
The paper’s central message is that:
A strong model can perform much better when it is given the right operating system for the task.
5. Why are these results important?
Many efforts to improve AI focus on training a bigger LLM. This paper argues that this is only part of the story.
Two identical models could perform very differently if one is given:
- better memory,
- a better planning method,
- more suitable tools,
- stronger error checking,
- or a better way to organize its actions.
JIT-Agent tries to automate this design process. Instead of human engineers manually building one agent system at a time, JIT-Agent can generate a suitable design for each new task.
This could make AI systems:
- more flexible,
- more reliable,
- better at difficult multi-step work,
- cheaper or faster to operate,
- and easier to adapt to new situations.
6. Possible impact and limitations
If the approach works as claimed, future AI assistants could automatically choose how they should operate. A single assistant might use one harness for researching a science topic, another for fixing software, and a third for organizing a business project.
The system could also become better through experience by saving successful harness designs and learning from mistakes.
However, the paper describes an early system rather than a complete solution. There are still important questions:
- How well will the method work on tasks very different from its training examples?
- Can the generated harnesses be trusted with important or dangerous tools?
- Will the system’s improvements remain reliable outside the tested benchmarks?
- How much computing time and money are needed to generate and test many harnesses?
- Could a poorly designed harness cause an otherwise capable model to make worse decisions?
The researchers also use a relatively simple four-part harness structure. Real-world systems such as coding assistants may have many more features, so future research may need richer designs.
Conclusion
The paper presents JIT-Agent, an AI system that creates, repairs, and improves the support structure surrounding another AI model.
Its main lesson is that AI ability does not come only from the LLM itself. It also depends on the model’s memory, planning, tools, and action rules.
By generating a custom harness for each task, JIT-Agent aims to make AI agents more capable and efficient. The reported experiments suggest that this approach can significantly improve different LLMs, offering a new way to advance AI besides simply making models larger.
Knowledge Gaps
Knowledge Gaps, Limitations, and Open Questions
The paper establishes the JIT-Agent framework but leaves the following issues unresolved:
- Incomplete methodological specification: The provided paper text ends during the description of Stage III, leaving deployment-time evolution, archive-update procedures, stopping criteria, and the complete inference algorithm unspecified.
- Limited characterization of the training data: The paper does not report the sizes, task distributions, domain composition, difficulty levels, or contamination controls for the Stage I–III datasets.
- Dependence on teacher-generated supervision: Stage I and Stage II rely on a stronger teacher model, but the paper does not quantify how teacher quality, prompting, sampling temperature, or teacher–student model mismatch affect JIT-Agent performance.
- Unclear contribution of each training stage: The reported gains are not sufficient to determine how much performance comes independently from customization, repair, and Evo-GDPO. Comprehensive ablations removing each stage and testing alternative training orders are needed.
- Unexamined effect of the four-module factorization: The assumption that memory, planning, action, and capability orchestration adequately represent all relevant harness behavior is not systematically validated against harnesses containing richer mechanisms such as delegation, verification, asynchronous execution, environment modeling, or multi-agent coordination.
- Restricted harness design space: The fixed protocol may prevent JIT-Agent from discovering useful architectures that do not fit the prescribed module interfaces or dependency order. The paper does not measure the performance loss caused by this representational constraint.
- Weak evidence for arbitrary backbone transfer: Although several model families are evaluated, the paper does not establish transfer to substantially different architectures, training objectives, context lengths, modalities, open-source versus proprietary models, or models that use incompatible tool-call formats.
- Limited evaluation of task generalization: It remains unclear whether generated harnesses generalize to unseen task types, novel combinations of tools, adversarial task specifications, or tasks outside the distributions used to construct the seed bank and training corpus.
- Potential task-type leakage: Stage I samples reference harnesses from task-type-matched subsets of the seed bank. The paper does not clarify whether task-type labels, benchmark identities, or benchmark-specific harness patterns leak information that artificially simplifies synthesis.
- Insufficient baseline comparability: Comparisons with OpenCode, Claude Code, GPT-5.6, and other runtimes may involve differences in model access, tool availability, prompts, execution budgets, or hidden engineering. Matched evaluations with identical backbones, tools, seeds, and budgets are needed.
- No comparison with simpler adaptive strategies: The paper does not adequately compare JIT-Agent with lightweight alternatives such as selecting among the 13 seed harnesses, retrieving the best prior harness, prompt-based harness editing, random search, evolutionary search, or per-task test-time scaling.
- Unclear statistical reliability: The paper reports headline point gains but does not provide confidence intervals, significance tests, per-task variance, failure rates, or the number of repeated trials needed to establish that improvements are robust.
- Reward-function sensitivity: The preference and Evo-GDPO objectives use hand-designed reward, latency, and cost weights. The paper does not study how sensitive results are to these weights, indicator thresholds, normalization schemes, clipping parameters, or the definition of the reward frontier.
- Questionable treatment of multi-objective trade-offs: Efficiency rewards activate only when candidate reward matches or exceeds the incumbent. This may discard useful designs that accept a small quality reduction for a substantial cost or latency improvement, leaving the practical Pareto trade-off unexplored.
- Latency and monetary-cost measurement limitations: The paper does not specify whether latency and cost include JIT-Agent generation, repair attempts, archive retrieval, validation, executor calls, tool execution, and failed candidates. Without full accounting, reported cost–performance improvements may be incomplete.
- Potential reward-hacking behavior: The framework optimizes benchmark reward and may learn harnesses that exploit evaluator artifacts, tool quirks, or stopping conditions. Robustness to hidden tests, evaluator randomization, and adversarial validation is not examined.
- Short repair horizon: Stage II retains only repairs that succeed within two rounds. This excludes failures requiring architectural changes or longer debugging sequences and provides no evidence that the two-round limit is appropriate across domains.
- Unresolved repair safety issues: The paper does not evaluate whether repair operations can introduce subtle semantic errors, remove necessary safeguards, alter tool permissions, create infinite loops, or produce harnesses that are executable but unsafe.
- Insufficient analysis of invalid-generation behavior: Invalid harnesses receive minimum task reward, but the paper does not report invalid-generation rates before and after training, the distribution of failure types, or which diagnostics are most useful for recovery.
- Archive quality and diversity are underexplored: The harness bank retains frontier-improving designs, but the paper does not analyze whether this causes mode collapse, redundant harness accumulation, loss of diversity, or poor coverage of rare task structures.
- Risk of online archive contamination: Test-time evolution can use execution feedback and prior harnesses, but the paper does not define safeguards against cross-task contamination, benchmark leakage, or retaining information that should not transfer between users or environments.
- Stability of continual evolution is unknown: The paper does not establish whether repeated online updates produce monotonically improving performance, catastrophic regressions, oscillation between harness styles, or degradation under distribution shift.
- No long-term deployment evaluation: The claimed compounding benefit of harness evolution is not demonstrated over long sequences of tasks, changing tool registries, changing backbone models, or nonstationary user objectives.
- Backbone–harness interaction is insufficiently analyzed: The paper treats the executor as frozen during harness generation, but does not identify which model capabilities are prerequisites for benefiting from a generated harness or when a stronger harness cannot compensate for a weak backbone.
- No analysis of generation overhead: JIT synthesis, validation, repair, and candidate comparison may impose substantial inference-time overhead. The paper does not report end-to-end time and cost relative to using a fixed harness.
- Scalability to large capability registries is unclear: The experiments do not establish how generation quality, tool-selection accuracy, latency, and cost change as the number of tools, APIs, MCP servers, or skills grows.
- Tool and environment reliability are not isolated: It is unclear whether observed failures originate from the harness, the executor, tool errors, network variability, or benchmark environments. Controlled fault-injection experiments are needed to separate these effects.
- Limited robustness testing: The framework is not evaluated under malformed tool outputs, unavailable tools, partial observability, delayed feedback, transient network failures, contradictory observations, or malicious tool responses.
- Security and permission boundaries are unspecified: A model that generates executable harnesses could expose unintended tools, bypass access controls, leak context, or create unsafe execution paths. The paper does not provide a security model or permission-safety evaluation.
- Reproducibility is incomplete: Although code and model links are listed, the paper does not provide enough information about prompts, teacher configurations, training compute, random seeds, benchmark versions, infrastructure, or exact harness implementations to reproduce the reported results reliably.
- Human utility is not measured: The evaluation focuses on benchmark reward, latency, and cost, without assessing maintainability, interpretability, debuggability, auditability, or user trust in automatically generated harnesses.
- No analysis of harness complexity: The paper does not report code size, module complexity, number of control-flow branches, number of tool calls, or debugging burden, making it difficult to assess whether performance gains justify increased operational complexity.
- Evaluation is predominantly task-success oriented: The framework’s effects on factuality, calibration, citation quality, safety, consistency, privacy, and user preference—especially in deep-research and daily-work settings—remain unresolved.
- The claim of orthogonality to model scaling is not established: Improvements across several model families do not demonstrate statistical independence between model scale and harness intelligence. A controlled scaling study is needed to determine whether JIT-Agent provides additive, multiplicative, or diminishing returns as backbone capability increases.
- No principled criterion for when JIT is preferable to AOT: The paper motivates JIT construction for heterogeneous tasks but does not identify the task diversity, workload size, distribution drift, or overhead threshold at which JIT synthesis outperforms a carefully optimized fixed harness.
- Open question about recursive self-improvement: The framework evolves harnesses but keeps JIT-Agent itself fixed during deployment. It remains unknown whether allowing the generator, protocol, module library, or evaluator to evolve would improve capability or create instability and safety risks.
- Unresolved theoretical guarantees: The paper provides an optimization formulation but no guarantees regarding validity, convergence, monotonic frontier improvement, regret, transfer across tasks, or bounded degradation during online evolution.
Practical Applications
Immediate Applications
- Task-adaptive enterprise AI assistants — software, customer operations, and knowledge work
- Deploy JIT-Agent as a middleware layer around existing agentic LLMs to generate a task-specific combination of memory management, planning, action control, and tool orchestration.
- Practical workflows include customer-support resolution, internal knowledge retrieval, document preparation, spreadsheet analysis, and multi-step administrative tasks.
- A support-ticket agent could generate a retrieval-heavy harness for ambiguous tickets, while a data-entry task could use a shorter, lower-cost action loop.
- Dependencies and assumptions: The target LLM must reliably follow the generated protocol; enterprise tools must expose stable APIs; tool permissions, data isolation, and audit logging must be enforced. The paper evaluates benchmark tasks rather than production business processes, so domain-specific reliability still requires validation.
- Adaptive coding and software-engineering agents — software development
- Use JIT-generated harnesses to select an appropriate coding workflow for each repository task: serial terminal execution for small fixes, explicit planning and testing for larger changes, or recursive delegation for complex repositories.
- Potential products include an IDE plugin or CI service that automatically generates a repository-specific agent runtime containing filesystem access, test execution, patch validation, issue tracking, and rollback procedures.
- The repair stage can be used to recover from malformed tool calls, interface mismatches, compilation failures, or runtime exceptions.
- Dependencies and assumptions: The harness must run in a sandbox with controlled credentials and network access. Generated code and tool policies require human review or automated security checks before execution. Benchmark gains do not establish that the system is safe for unrestricted code deployment.
- Deep-research and web-search assistants — academia, consulting, intelligence, and journalism
- Generate research-specific harnesses that maintain evidence memory, decompose questions into subproblems, conduct parallel searches, track source provenance, and verify claims before producing an answer.
- A research platform could select different harnesses for literature reviews, market analysis, policy monitoring, or fact-checking based on the task specification.
- The archive of prior harnesses could preserve effective search and verification workflows for recurring research domains.
- Dependencies and assumptions: Search APIs, document access, citation extraction, and source-quality evaluators must be available. Human verification remains necessary because the paper reports task-level performance improvements, not guaranteed factuality or resistance to biased and malicious sources.
- Cost- and latency-aware agent routing — cloud software and AI infrastructure
- Use the reward, latency, and monetary-cost signals in Evo-GDPO to choose a harness–model pair rather than scaling the foundation model indiscriminately.
- An inference gateway could route simple tasks to a lightweight model with a compact harness and reserve larger models or recursive workflows for difficult tasks.
- Potential tooling includes a harness registry, automated evaluation dashboard, frontier tracking system, and deployment controller that compares candidate harnesses under common seeds and budgets.
- Dependencies and assumptions: Costs and latency must be measured consistently across providers and workloads. The optimization objective must reflect business priorities; otherwise, a harness may reduce cost at the expense of quality, safety, or user experience.
- Automated harness testing and repair — software quality assurance
- Apply the Stage-II repair mechanism to agent runtimes that fail static validation, interface checks, tool invocation tests, or runtime execution.
- A practical workflow would generate a harness, run protocol and integration tests, feed structured diagnostics back to the repair model, and permit only bounded revisions before escalation to an engineer.
- This could reduce manual debugging of prompts, memory policies, planner logic, and tool schemas.
- Dependencies and assumptions: Failures must produce informative diagnostic reports, and the two-round repair limit must be expanded for failures requiring architectural redesign. Repairs should be checked against regression tests to prevent silent degradation.
- Reusable agent-runtime benchmarking — academia and industry research
- HarnessFactory’s common four-module interface can support controlled comparisons of memory, planning, action, and capability-orchestration strategies across different backbone models.
- Researchers can evaluate whether a performance gain arises from model weights, harness design, tool access, or their interaction.
- A benchmark platform could standardize harness interfaces and report quality, latency, cost, validity rate, and recovery rate.
- Dependencies and assumptions: The four-module factorization must be expressive enough for the target agents. More sophisticated production runtimes may not map cleanly to the compact protocol, and comparisons require identical tools, budgets, seeds, and evaluation criteria.
- Education and instructional assistants — education
- Generate different harnesses for tutoring, formative assessment, lesson planning, grading assistance, and research support.
- For example, a tutoring harness could emphasize persistent student memory and Socratic planning, whereas an assessment harness could enforce rubric-based verification and prohibit unsupported answer generation.
- Dependencies and assumptions: Student data requires strong privacy protections, and educational decisions require educator oversight. The paper does not evaluate pedagogical effectiveness, age appropriateness, accessibility, or fairness.
- Personal productivity assistants — daily life
- Use task-conditioned harnesses for calendar management, travel planning, email triage, household purchasing, and document organization.
- The capability-orchestration module could expose only the tools relevant to a particular request, reducing unnecessary actions and limiting the risk of accidental tool use.
- A personal assistant could maintain a local archive of successful workflows while adapting its memory and planning strategy to recurring user preferences.
- Dependencies and assumptions: User authorization, reversible actions, confirmation steps, local data protection, and reliable integrations are essential. Online evolution should not modify high-impact behaviors without explicit user approval.
- Policy prototyping and public-sector workflow automation — government
- Government agencies could use the framework to prototype agents for document classification, regulatory research, benefits-case preparation, procurement analysis, or public-information response.
- A policy-specific harness could enforce source citation, procedural checklists, role-based permissions, and human approval before an external action.
- Dependencies and assumptions: Public-sector deployment requires explainability, records retention, procurement compliance, accessibility, and bias testing. The reported benchmark improvements are insufficient evidence for autonomous decisions involving legal rights or public benefits.
Long-Term Applications
- Self-improving agent platforms — general-purpose AI infrastructure
- JIT-Agent could become a control-plane service that continuously synthesizes, evaluates, repairs, and retires harnesses for a large population of agents.
- The harness archive could evolve into an organization-wide library indexed by task type, model family, tools, quality, latency, cost, and safety outcomes.
- Such a platform would shift agent engineering from manually maintaining fixed runtimes to managing a learned population of executable scaffolds.
- Dependencies and assumptions: Long-term deployment requires robust evaluation under distribution shift, prevention of archive contamination, version control, rollback, reproducibility, and defenses against reward hacking. Online evolution must not optimize benchmark scores while degrading real-world safety.
- Cross-model and cross-domain agent portability — software, robotics, and industrial automation
- Because the proposed harness is intended to be transferable across off-the-shelf backbones, a single harness-generation service could adapt workflows to models with different capabilities, context limits, costs, and tool-following behavior.
- In robotics or industrial systems, the harness could select between reactive control, hierarchical planning, sensor-memory strategies, and tool or skill delegation depending on the task.
- Dependencies and assumptions: Transferability beyond the evaluated model families must be demonstrated. Physical deployment additionally requires deterministic control bounds, real-time guarantees, simulation testing, hardware safety certification, and protection against unsafe generated actions.
- Healthcare decision-support orchestration — healthcare
- A future system could generate task-specific harnesses for clinical literature review, patient-history summarization, differential-diagnosis support, care-plan preparation, and administrative coordination.
- Memory could organize longitudinal records, planning could decompose clinical questions, capability orchestration could select validated medical databases, and action control could require clinician confirmation.
- Dependencies and assumptions: This application requires clinical validation, privacy-preserving infrastructure, medical-device and health-regulatory approval, calibrated uncertainty, reliable provenance, and strict separation between decision support and autonomous treatment. The paper provides no clinical evidence.
- Scientific discovery and autonomous research laboratories — academia, energy, chemistry, and materials
- JIT-generated harnesses could adapt an agent to literature search, hypothesis generation, experimental design, simulation, data analysis, and iterative laboratory control.
- The archive could retain successful research workflows and select harnesses based on reproducibility, experimental cost, time, and scientific yield.
- In energy research, possible applications include materials screening, battery optimization, grid-analysis workflows, and simulation-driven design.
- Dependencies and assumptions: Reliable experiment APIs, laboratory automation, validated simulators, safety interlocks, provenance tracking, and human scientific review are required. Reward definitions must capture scientific validity rather than merely fast or plausible outputs.
- Autonomous multi-agent organizations — enterprise operations and project management
- The recursive and delegation-oriented harness patterns described in the paper could support teams of specialized agents for planning, research, coding, verification, procurement, and reporting.
- JIT-Agent could generate organizational workflows dynamically—for example, assigning independent agents to gather evidence and a separate verifier to reconcile disagreements.
- Dependencies and assumptions: Multi-agent scaling can increase cost, latency, coordination failures, and correlated errors. Deployment requires authority boundaries, conflict resolution, auditability, identity management, and formal approval policies for consequential actions.
- Robust autonomous systems in changing environments — robotics, logistics, and field operations
- A harness could be regenerated when environmental conditions, available tools, mission objectives, or hardware capabilities change.
- Potential uses include warehouse robots that adapt planning to congestion, field-service agents that adjust to unavailable equipment, and disaster-response systems that reconfigure tool access as communications or sensors fail.
- Dependencies and assumptions: Real-time JIT generation may be too slow or unpredictable for safety-critical control. A practical architecture would likely generate policies at a supervisory level while relying on verified low-level controllers, simulations, fallback behaviors, and human command authority.
- Financial analysis and regulated decision workflows — finance and insurance
- Future systems could generate harnesses for portfolio research, fraud investigation, underwriting support, stress testing, and regulatory reporting.
- A finance-specific harness might emphasize structured evidence memory, independent verification, restricted data access, and cost-aware use of market and compliance tools.
- Dependencies and assumptions: Financial deployment requires explainable outputs, immutable audit trails, market-data licensing, fairness testing, access controls, and compliance with applicable regulations. Optimization based only on reward, latency, and cost could be inadequate for systemic risk and legal accountability.
- Public policy and institutional decision simulation — policy and social science
- Harness archives could support controlled comparison of alternative policy-analysis workflows, such as evidence synthesis, stakeholder-impact analysis, scenario generation, and sensitivity testing.
- Researchers could study how memory, planning, and verification policies affect the quality and reproducibility of AI-assisted policy advice.
- Dependencies and assumptions: Policy applications require representative data, transparent evaluation criteria, protection against political or demographic bias, and explicit distinction between simulation and evidence-based prediction. The paper’s benchmark results do not establish validity for social forecasting or policy recommendations.
- Standardized markets for agent skills and harnesses — software ecosystems
- The four-module protocol could enable third-party marketplaces for verified memory modules, planners, action policies, tools, and domain-specific skills.
- Organizations might purchase or share harness components with metadata describing supported models, tools, compliance properties, latency, cost, and evaluation results.
- Dependencies and assumptions: This requires stable interfaces, provenance and licensing standards, sandboxing, cryptographic signing, compatibility testing, and defenses against malicious skills or prompt/tool injection. The paper establishes a conceptual composable protocol but does not yet demonstrate an ecosystem or production-grade security model.
Glossary
- Ahead-of-Time (AOT): A design approach in which an artifact is optimized before the specific future task is encountered. “many of these methods share an Ahead-of-Time (AOT) assumption”
- Agent harness: The operational software layer that manages an agent’s memory, planning, actions, and tools around a LLM. “The capability of an LLM agent is jointly determined by two tightly coupled factors: the foundation model that produces reasoning and actions, and the agent harness that situates this model in a closed-loop execution environment”
- Agentic LLM: A LLM configured to pursue objectives by reasoning, invoking tools, and taking actions. “an arbitrary agentic LLM executes under that harness”
- Agentic scaffold: A structured runtime framework that organizes an agent’s reasoning, memory, actions, and capabilities. “13 representative contemporary agentic scaffolds”
- Advantage: A reinforcement-learning signal representing how much better an action or candidate performs relative to a reference or baseline. “Evo-GDPO normalizes the three signals separately before merging them”
- Amortize: To distribute the cost of repeated computation or search by learning a reusable procedure. “We instead amortize harness search into JIT-Agent”
- API: An application programming interface through which software components expose callable functionality. “The registry contains callable tools (e.g., bash tools, APIs, and MCPs”
- Backbone executor: The fixed underlying model that executes a generated harness. “A frozen executor then runs each harness for validation and utility measurement”
- Closed-loop execution: An interaction process in which an agent takes actions, receives observations, updates its state, and acts again. “the agent harness that situates this model in a closed-loop execution environment”
- Composability: The property that software components can be combined and recombined through compatible interfaces. “Composability is equally central to JIT generation”
- Context isolation: The separation of contextual information among agents or subtasks to prevent unwanted interference. “keep agent contexts isolated in $\mathbf{M}_{\mathrm{subproblem}$”
- Controller state: The mutable internal state used by a system to track and manage an ongoing execution. “where is the maintained controller state”
- DAG planning: Planning with a directed acyclic graph whose nodes represent tasks or subgoals and whose edges represent dependencies. “Wide-search tasks may benefit from parallel evidence exploration”
- Declarative LM pipeline: A language-model workflow specified by desired operations or relationships rather than by explicit control-flow code. “the optimization scope has expanded from prompts and declarative LM pipelines”
- Disjoint union: A mathematical union in which otherwise identical elements from different sets remain distinguishable by their originating set. “ is the disjoint union of executable calls and terminal outputs”
- Distillation: Training a model to reproduce or absorb behavior, knowledge, or performance patterns from another model or data source. “self-evolve by distilling performance signals from an expanding archive of prior harness configurations”
- Evolvability: The ability of a system to improve itself using execution feedback and previously accumulated designs. “evolvability, namely how to turn execution feedback into stronger future harnesses”
- Evolutionary Group-Decoupled Policy Optimization (Evo-GDPO): A policy-optimization method that separately normalizes reward, latency, and cost signals while encouraging candidates to surpass existing designs. “Third, Stage III introduces Evolutionary Group-Decoupled Policy Optimization (Evo-GDPO)”
- Executable artifact: A generated software object that can be run successfully in its target environment. “harness construction becomes an optimization problem over executable artifacts rather than a fixed engineering choice”
- Execution kernel: The shared runtime component that interprets actions, invokes tools, and produces observations. “where $\operatorname{Exec}:\mathcal{U}\times2^{\mathcal{C}_{\tau}\rightarrow\mathcal{O}$ is the shared execution kernel”
- Execution trajectory: A chronological sequence of states, actions, and observations produced during agent operation. “Stage II converts failed generations into bounded repair trajectories”
- Foundation model: A broadly pretrained model that serves as the underlying computational basis for specialized applications. “the foundation model that produces reasoning and actions”
- Harness intelligence: The learned ability to construct, repair, and improve the operational scaffold through which an agent acts. “Harness intelligence is the capacity to construct and refine the operational scaffold through which a model acts”
- Harness frontier: The set of currently best-performing harness configurations across dimensions such as reward, latency, and cost. “keep pushing the harness frontier forward”
- Harness synthesis: The automatic generation of an executable agent harness from a task specification and available components. “just-in-time harness synthesis provides gains beyond model scaling alone”
- Indicator function: A function that equals one when a condition holds and zero otherwise. “ is the indicator function”
- Instance-dependent: Varying according to the particular problem instance rather than only its general domain. “the appropriate harness is not only domain-dependent, but instance-dependent”
- Just-in-Time (JIT): A strategy that generates or adapts a system specifically when a task is encountered. “We instead turn to a Just-in-Time (JIT) view of harness construction”
- Logistic sigmoid: A function mapping real-valued inputs to values between zero and one, commonly used to model probabilities. “Here is the logistic sigmoid”
- Long-horizon task: A task requiring many sequential reasoning and action steps before completion. “This keeps the corpus focused on realistic, locally recoverable failures instead of those requiring wholesale redesign”
- Meta-agent: An agent whose role includes designing, directing, or improving other agents or their operating frameworks. “a compact meta-agent for just-in-time harness generation and evolution”
- Model-as-a-Harness: A paradigm in which a trained model generates the operational harness used by another agentic model. “We ask whether another possibility is viable: Model-as-a-Harness”
- Multi-rollout aggregation: Combining results from multiple independent executions to improve decision-making or reliability. “Multi-rollout aggregation”
- Online policy improvement: Updating a model’s behavior using feedback collected during ongoing or test-time interaction. “online policy improvement in Stage III”
- Pareto frontier: The set of solutions for which no objective can be improved without worsening at least one other objective. “retaining frontier-improving designs”
- Protocol-compliant: Conforming to the required schemas, interfaces, lifecycle rules, and execution semantics of a formal protocol. “protocol-compliant harnesses”
- ReAct: An agent paradigm that interleaves reasoning and action, typically by generating thoughts followed by tool calls or outputs. “Canonical ReAct can be written as”
- Reference-anchored objective: A training objective that compares a current model’s outputs with those of a fixed reference model. “We optimize the reference-anchored objective”
- Reinforcement-learning policy: A model or strategy that determines actions based on observations and optimization through reward signals. “the rollout-policy snapshot”
- Rollout: One simulated or actual execution of an agent on a task. “Running a harness with induces a closed-loop trajectory”
- Runtime exception: An error raised while a program is executing rather than during its compilation or static analysis. “including compiler errors, interface mismatches, tool-call failures, and runtime exceptions”
- Self-evolution: The process by which an agent system uses feedback from its own operation to generate improved future versions. “JIT-Agent is the unified continuation of this progression”
- Static validation: Checking a program or artifact for structural or syntactic errors without fully executing it. “some generated harnesses fail static or runtime validation”
- Supervised fine-tuning (SFT): Training a model to imitate labeled example outputs. “The first is a standard supervised fine-tuning objective over accepted teacher generations”
- Test-time optimization: Improving a model or its execution strategy during evaluation or deployment rather than only during initial training. “This is a powerful paradigm when the deployment distribution is stable and homogeneous”
- Tool orchestration: The process of selecting, sequencing, and coordinating external tools or skills for an agent. “skill orchestration, which discovers, retrieves, composes, and delegates reusable procedures”
- Trajectory: An ordered record of an agent’s states, emitted actions, and resulting observations. “The repair objective conditions on the entire history”
- Typed interface: A formally specified component boundary that restricts inputs and outputs to defined types or schemas. “explicit interfaces make harness components independently substitutable and optimizable”
- Utility function: A quantitative function that evaluates the desirability of an outcome according to one or more objectives. “where evaluates the trajectory induced by the model--harness pair using task reward, latency, and monetary cost”
- Zero-shot: Performing a task without task-specific examples or additional adaptation. “an arbitrary off-the-shelf agentic LLM executes under that harness”