Papers
Topics
Authors
Recent
Search
2000 character limit reached

NeoHorse-1: Towards Recursive Self-Improvement via Agentic Post-Training with Routing Harness

Published 8 Sep 2026 in cs.CL | (2609.08183v1)

Abstract: Recursive self-improvement (RSI) requires a concrete mechanism through which an AI system observes its capabilities and converts that evidence into the next round of learning. We present NeoHorse-1, a family of agent-native models developed to explore this path through agentic post-training. Our system combines a heterogeneous model pool with intelligent routing, recording the predicted capability demand, selected service tier, and subsequent interaction for each user turn. These records are converted into training examples that preserve interleaved reasoning, tool calls, and harness context, and are admitted through structural validation, six-dimensional semantic evaluation, and subscene-level labeling. Routing signals organize supervised fine-tuning into a three-stage curriculum and extend to routing-guided on-policy distillation, where a teacher supervises student-generated responses under the same progression. Capability-guided allocation then converts evaluation feedback into the next training mixture, closing an evaluation-selection-update loop in which what the system learns to do shapes what it learns from next. Across eleven benchmarks covering harness-based agents, tool use, coding, and instruction following, post-training raises the macro-average from 58.94 to 64.87 at 4B and from 65.60 to 69.04 at 9B, substantially narrowing the aggregate gap between the post-trained 4B model and the 9B base model. NeoHorse-1 provides an initial prototype of this feedback-driven process and a path toward harness-mediated RSI across successive iterations.

Summary

  • The paper presents a novel systems formulation for recursive self-improvement (RSI) that uses a routing harness to measure capabilities, generate data, and control training-distribution, improving 4B and 9B models with gains of 5.93 and 3.44 macro-average points in interactive tasks, coding, and tool use.
  • NeoHorse-1 introduces a three-stage training loop where capability demand, inferred by the routing harness, influences subsequent post-training data, demonstrating that agentic post-training can enhance model performance, particularly in tasks involving interactive execution and code generation.
  • The methodology employs a trajectory-centered data construction, achieving structural validity through causal ordering and tool-call/result correspondence, and semantic characterization via goal attainment and error recovery, but does not establish definitive causal claims for the individual components of this approach

NeoHorse-1 presents a concrete systems formulation of recursive self-improvement (RSI) in which the deployment harness is not merely an execution wrapper but an instrument for capability measurement, data generation, and training-distribution control. The central claim is that a routing harness records three forms of information simultaneously: the interaction trajectory, the capability demand inferred by the router, and the outcome of the attempted execution. These signals support an evaluation–selection–update loop in which the model’s observed deficiencies influence the composition of subsequent post-training data. The paper instantiates this design in 4B and 9B models and evaluates it across agentic execution, tool use, coding, and instruction following (2609.08183).

System objective and RSI formulation

The paper defines RSI operationally rather than as autonomous redesign of the entire training process. A model is returned to a heterogeneous routing environment after post-training; its subsequent interactions produce new trajectories, routing records, and outcome evidence; these records are then filtered, characterized, and allocated into the next training mixture. The proposed loop therefore improves model parameters through repeated use of the same harness-mediated data infrastructure.

This framing is narrower than formulations in which an agent modifies its own architecture, optimizer, reward function, or research procedure. NeoHorse-1 does not demonstrate indefinite recursive improvement, nor does it automate the full training pipeline. Instead, it establishes a mechanism by which deployed behavior can influence later model updates. The distinction is important: the paper’s empirical evidence validates one post-training iteration, while the recursive property remains a proposed extension of the demonstrated pipeline.

The routing harness contains a heterogeneous model pool and assigns user turns to capability tiers C0 through C3. Routing decisions are conditioned on the current request, dialogue history, prior routing decisions, and execution state. The system stores the raw prediction, policy-adjusted decision, and actually served tier separately. This separation prevents the served model identity from being treated as a direct difficulty label, since routing may also reflect availability, user overrides, risk policies, and service constraints.

The complete process is summarized below.

Figure 1

Figure 1: Routing-guided agentic training converts heterogeneous harness interactions into training data and uses capability feedback to update the next training mixture.

The resulting design is closely related to trajectory-based agent tuning, on-policy distillation, and harness-native routing, but its distinctive contribution is their integration into a feedback-controlled data pipeline. Trajectory supervision provides behavioral content, routing scores provide an ordering signal, and outcome-based evaluation determines which regions of the capability space receive additional training allocation.

Trajectory-centered data construction

NeoHorse-1 treats the execution trajectory, rather than a static instruction–response pair, as the fundamental data object. A trajectory contains user requests, assistant reasoning, tool calls, tool observations, recovery attempts, harness-injected context, and terminal outcomes. The authors define three linked granularities:

  • Trajectory: the complete harness execution.
  • User turn: a request and the assistant/tool sequence that follows it until the next user request or termination.
  • Subscene: adjacent user turns sharing a local goal.

User turns are used as serialized training units, while subscenes support semantic analysis and capability allocation. This decomposition preserves provenance between the full execution history, the supervised sequence, and the semantic unit used to characterize the task.

The training representation retains visible historical responses, tool calls, tool results, system instructions, tool specifications, and harness context. Earlier hidden reasoning is omitted, whereas reasoning in the current user turn is retained when available. The current turn receives prediction loss on assistant reasoning, tool calls and arguments, visible responses, and end-of-response tokens. User messages, tool results, system messages, and historical context remain available to causal attention but are excluded from the loss.

Figure 2

Figure 2: User-turn supervision retains the execution context while applying loss only to assistant target spans in the current turn.

This masking strategy is technically consequential. It trains the model to generate actions and responses under the same contextual conditions in which they were originally produced, rather than flattening tool interactions into independent text targets. At the same time, it does not directly optimize environment outcomes. Correctness therefore depends on the quality of the recorded trajectories and on the filtering and evaluation procedures applied before training.

The corpus combines approximately 10510^5–10610^6 harness-generated trajectories with public instruction, reasoning, coding, tool-use, and preference data. The reported description emphasizes unified serialization, deduplication, tokenizer freezing, and evaluation decontamination, but the paper does not provide a complete public accounting of the resulting token count or a full breakdown of the mixture in the supplied text. Consequently, the exact contribution of harness data relative to auxiliary public data cannot be inferred from the main benchmark comparison alone.

Structural validation and semantic characterization

The data pipeline separates structural validity from semantic quality. Structural validation reconstructs user requests, assistant responses, tool calls, observations, and terminal events. It checks message readability, causal ordering, tool-call/result correspondence, identifier consistency, execution-branch closure, and terminal-state ownership. Trajectories are categorized as complete, partially recoverable, or quarantined. Recoverable trajectories contribute only causally closed sub-trajectories.

This distinction avoids conflating replayability with task success. A structurally complete trajectory can still contain incorrect tool selection, an invalid artifact, an unrecovered failure, or an unsupported claim. The semantic stage therefore evaluates six dimensions:

  1. Goal attainment.
  2. Instruction adherence.
  3. Tool use.
  4. Evidence consistency.
  5. Error recovery.
  6. Termination.

Each dimension receives PASS, WARN, FAIL, or NOT_EVALUATED, with evidence coverage stored independently. Missing evidence is not converted into a positive judgment. Long interactions are evaluated in segments and aggregated at the turn level, allowing intermediate errors followed by successful recovery to be distinguished from terminal failure.

Subscene characterization provides a second layer of organization. Each subscene is represented through Scene, Goal, and Outcome attributes. Scene captures task type, domain, context, and whether the user is requesting information or execution. Goal records acceptance criteria and cross-turn relations such as continuation, modification, interruption, and resumption. Outcome records the verifiable result rather than merely whether execution terminated.

Figure 3

Figure 3: Subscene characterization separates the user’s situational context, intended goal, and verifiable execution outcome.

This representation supports capability-guided allocation because it preserves multiple explanatory axes for failure. A low-quality trajectory can be associated with a difficult task, poor instruction adherence, incorrect tool use, insufficient recovery, or failure to satisfy the acceptance criteria. However, the paper does not report a quantitative ablation of the six-dimensional semantic evaluator or the subscene taxonomy. Their contribution is therefore justified architecturally and through downstream results, but not isolated experimentally.

Routing-guided curriculum learning

The routing signal is used as a curriculum variable rather than as a loss weight. Each example receives either a hard score based on its assigned tier or a soft score computed from the router’s normalized support over C0–C3. The hard score preserves the discrete routing decision; the soft score distinguishes examples assigned to the same tier by incorporating uncertainty or support for neighboring tiers.

Training proceeds through three approximately equal-sized stages. Higher-scored examples are introduced progressively, while some lower-scored examples are reserved for later stages. Each example is used once per pass, and the optimizer and learning-rate schedule continue across stages without reset.

Figure 4

Figure 4: The routing curriculum shifts the training mixture toward higher estimated capability demand while preserving lower-demand coverage.

The paper’s rationale is that the router’s predicted demand is a better ordering proxy than the identity of the model actually served. A high-capability model may have been selected because of policy or availability rather than task difficulty, while a weaker model may successfully handle a demanding request. This is a sensible distinction, but it leaves an important assumption: router scores must be sufficiently calibrated and stable to function as a useful curriculum. The paper treats them as relative demand estimates, not validated difficulty measures, and does not report calibration metrics or a systematic comparison against random ordering, loss-based ordering, or outcome-based difficulty ordering in the main results.

The curriculum changes when examples are presented, not what targets they contain. Thus, its effect is inseparable from optimization dynamics and the particular mixture of trajectory types. The paper’s evidence supports the overall post-training recipe, but not a definitive causal claim that routing-based ordering itself is responsible for the observed gains.

Routing-guided on-policy distillation

The second optimization component addresses the distribution gap between recorded trajectories and deployed behavior. Standard trajectory SFT trains on fixed assistant responses, whereas deployment conditions on prefixes generated by the student. NeoHorse-1 therefore applies on-policy distillation (OPD): the student generates responses from recorded contexts, and a fixed teacher supplies token-level distributions on those student-generated prefixes.

The starting contexts are scheduled using the same three-stage routing curriculum. At each stage, the student generates one response per context. The teacher evaluates the corresponding prefixes, and the student is optimized using a response-normalized reverse-KL objective. Only generated response tokens receive loss; prompts and padding do not. To reduce storage and scoring cost, the student’s top-KK candidate tokens are retained and all remaining probability mass is represented by an additional bin.

Figure 5

Figure 5: Routing-guided OPD schedules recorded starting contexts by capability demand while teacher supervision follows student-generated prefixes.

This combination separates two functions that are often conflated. Routing determines which interaction states the student encounters and when; OPD determines how the teacher supervises the student on its own evolving behavior. The design is particularly appropriate for agentic tasks because failures often arise after the model departs from the teacher trajectory. Supervision on student-generated prefixes can therefore target states that ordinary trajectory imitation would not visit.

Nevertheless, the supplied results do not isolate the incremental contribution of OPD relative to routing-guided SFT. No ablation is reported for SFT alone, OPD without routing, alternative KL directions, or different top-KK approximations. The paper consequently establishes OPD as part of the system design, but not as an independently validated source of the final performance improvements.

Benchmark results

NeoHorse-1 is evaluated on ten benchmarks spanning harness-based agents, tool use, coding, and instruction following. The reported macro-average increases from 58.94 to 64.87 for the 4B model and from 65.60 to 69.04 for the 9B model. These correspond to absolute gains of 5.93 and 3.44 points, respectively. The larger relative improvement at 4B is consistent with greater headroom in the smaller base model.

Model Base macro-average NeoHorse-1 macro-average Absolute gain
4B 58.94 64.87 +5.93
9B 65.60 69.04 +3.44

At 4B, NeoHorse-1 exceeds Qwen3.5-4B on every benchmark for which both results are available. Its strongest differences occur in agentic and coding evaluations: the reported scores are 88.46 on one agentic benchmark, 77.33 on another, 34.41 on a more demanding execution task, 44.68 on a further interactive evaluation, 96.95 on HumanEval, and 59.43 on LiveCodeBench v6. It also reaches 65.33 and 88.35 on the two instruction-following evaluations. The result is not confined to one task family, although the largest practical gains are concentrated in interactive execution and code generation.

At 9B, NeoHorse-1-9B improves over Qwen3.5-9B on most shared evaluations. It reaches 67.43, 42.25, 90.82, 82.25, 40.15, and 48.73 across the six agentic metrics, 98.17 on HumanEval, 65.14 on LiveCodeBench, 66.33 on IFEval, and 89.09 on IFBench. Relative to the 9B base, the largest gains are concentrated in harness-based interaction and selected tool-use tasks, while instruction-following performance remains approximately stable and declines slightly on one metric.

The 4B model’s aggregate score of 64.87 exceeds the 9B base model’s 65.60 only narrowly, but NeoHorse-1-4B matches or exceeds the 9B base on several individual benchmarks. This supports the paper’s narrower claim that agentic post-training can compensate for part of the performance difference associated with parameter scale. It does not establish that post-training eliminates the value of scale: NeoHorse-1-9B remains stronger overall and retains clear advantages on long-horizon interaction, execution feedback, and difficult coding tasks.

Figure 6

Figure 6: NeoHorse-1 improves substantially over the corresponding base models across six agentic benchmarks at both the 4B and 9B scales.

The evaluation methodology contains several qualifications. Some results are taken from official reports rather than produced under the paper’s own evaluation pipeline. Different benchmarks use different harnesses, and VitaBench uses DeepSeek-V4-Flash for both simulation and judging because the originally recommended models were unavailable. These choices are disclosed, but they complicate direct cross-benchmark comparisons and make the macro-average partly dependent on heterogeneous evaluation conditions.

Behavioral evidence from agent traces

The trajectory analyses provide a qualitative account of the benchmark gains. In a project-scheduling task, Qwen3.5-4B identifies relevant files but fails to inspect an email containing an updated dependency constraint. It consequently plans from stale information, produces an invalid schedule, and writes the artifact to an unintended location. NeoHorse-1-4B retrieves the additional evidence, incorporates the changed dependency, recomputes and verifies the schedule, and saves the output to the required path.

The case illustrates that the claimed improvement is not simply better natural-language completion. It concerns evidence acquisition, constraint tracking, verification, and artifact delivery. These capabilities are directly represented in the trajectory format and are therefore plausible targets of agentic post-training.

The paper reports a stronger scale effect under repeated feedback. In a WorkBuddy code-repair task, NeoHorse-1-4B stops after one implementation attempt without establishing an effective test-and-repair loop. NeoHorse-1-9B performs an edit–test–inspect–repair cycle and continues until the verifier passes. In a PinchBench data-analysis task, the 4B model repeatedly attempts to install or emulate an unavailable pandas dependency. The 9B model recognizes the environmental constraint and switches to Python’s standard csv and mathematical libraries.

The latter trajectory is associated with reductions of approximately 70.8% in model requests, 76.7% in execution time, and 83.6% in token usage. These are strong operational results because they indicate that the larger model’s advantage is not merely greater persistence. It can terminate unproductive branches and select a lower-cost alternative strategy. The implication is that model scale remains important for feedback interpretation and strategy revision even after trajectory-based post-training.

The case studies also expose residual errors. In a ticket-reporting task, NeoHorse-1-9B correctly identifies 50 eligible records and 19 tickets, excludes a next-day update, and preserves source-row references, but its status-summary table contains two priority labels where status labels are required. In a leakage-auditing task, it follows the documented five-minute tolerance and preserves the required output schema, whereas the baseline uses a two-second tolerance and produces incompatible output. In the Gomoku case, NeoHorse-1-9B completes a 26-click replay with 26 visible stones and correct turn state, while Qwen3.5-9B produces invalid board indices on every click.

Figure 7

Figure 7

Figure 7: Under the same 26-click replay, the baseline remains empty while NeoHorse-1-9B preserves the expected 13 black and 13 white stones.

These examples support improved execution grounding, but they do not show universal correctness. The ticket-reporting error is particularly relevant because it demonstrates that successful retrieval and temporal reasoning can coexist with schema-level inconsistency. The paper appropriately limits its interpretation: better grounding does not imply complete artifact validation.

Data-source and supervision scaling analyses

A controlled comparison evaluates routing-harness trajectories against the public Toucan tool-agent dataset under the same routing-guided training configuration. Both runs use the same initialization, curriculum, optimizer, random seed, packing method, and approximately matched training budgets.

Training source LCB HumanEval IFBench BFCL V4 τ2\tau^2-Bench Average
Toucan 49.14 87.80 56.33 54.77 73.54 64.32
Routing-harness data 53.14 96.34 61.33 57.20 84.85 70.57
Difference +4.00 +8.54 +5.00 +2.43 +11.31 +6.26

Routing-harness data improves every benchmark in this comparison and raises the five-benchmark average by 6.26 points. The largest improvements occur on HumanEval and τ2\tau^2-Bench, suggesting that the advantage may derive from richer execution context, better task diversity, or more faithful environment-grounded supervision.

The attribution remains underdetermined. The two sources differ in trajectory composition, tool interfaces, generation procedures, and potentially quality distributions, even though their training configuration is matched. The experiment establishes that routing-harness data is more effective in the tested setup; it does not identify which specific property causes the difference.

A nested-data scaling experiment further increases unique routing-harness supervision while holding initialization, optimization, packing, and the number of passes fixed. The five-benchmark development average rises from 69.31 for the base model to 71.45 at the largest reported data scale. This monotonic trend supports the claim that additional high-quality interaction supervision remains useful over the measured range.

The gain is modest relative to the full base-to-NeoHorse improvement, and the reported experiment does not establish whether performance eventually saturates, whether repeated examples would substitute for unique trajectories, or whether the trend is preserved when data allocation is driven by the model-deficiency profile rather than fixed quality ranking.

Limitations and open questions

The principal limitation is that the RSI loop is demonstrated only for a single evaluation–selection–update cycle. The paper proposes that improved checkpoints can generate better trajectories, which can then improve later checkpoints, but it does not measure multi-iteration accumulation, degradation, or feedback saturation. The central recursive claim is therefore architectural and procedural rather than empirically established across successive generations.

The routing score is treated as a capability-demand proxy without a reported calibration study. Its reliability may vary across domains, model pools, deployment policies, and changing harness configurations. The paper correctly distinguishes predicted demand from the actually served model, but it leaves open whether routing scores remain comparable as the serving stack evolves.

A second limitation is experimental attribution. The final system combines trajectory SFT, routing-guided curriculum learning, OPD, structured data filtering, public auxiliary data, and capability-guided allocation. The results do not provide a complete factorial ablation separating these components. In particular, the incremental effects of the curriculum and OPD mechanisms remain unresolved.

Evaluation coverage is also narrower than the RSI framing. The benchmark suite emphasizes agentic execution, coding, tool use, and instruction following. The paper explicitly notes that broader tasks served by the harness were not evaluated. Several benchmark results involve external reports or heterogeneous simulator and judge configurations, which limits strict comparability.

Finally, the qualitative cases show that improved interaction control does not remove schema, reconciliation, or output-consistency errors. The remaining question is whether the same feedback loop can allocate sufficient supervision to such low-level but consequential failures without sacrificing broad capability coverage.

Conclusion

NeoHorse-1 formulates harness-mediated RSI as a practical post-training loop built from trajectory collection, routing-based demand estimation, structured outcome evaluation, curriculum scheduling, and on-policy distillation. Its 4B and 9B models obtain macro-average gains of 5.93 and 3.44 points, respectively, with the strongest improvements concentrated in interactive execution, tool use, and coding. Controlled data-source comparisons further favor routing-harness trajectories over public synthetic agent data.

The paper’s strongest contribution is the integration of deployment instrumentation with training-data allocation. Its empirical results validate substantial single-iteration improvements and provide evidence that trajectory quality and environmental feedback matter. They do not yet establish recursive improvement across repeated generations. The decisive open question is whether the proposed evaluation–selection–update loop continues to yield net gains after improved models alter the distribution of tasks, failures, and routing decisions that supply the next round of supervision.

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Explain it Like I'm 14

1. Paper overview

This paper introduces NeoHorse-1, a family of AI models designed to become better at working as agents. An AI agent is a system that can do more than answer questions: it can plan, use tools, write code, search for information, and interact with software.

The paper explores a first step toward recursive self-improvement. This means creating a loop in which an AI system:

  1. Does tasks and records what happens.
  2. Studies what it did well and badly.
  3. Uses that information to train a better version of itself.
  4. Sends the improved version back to do more tasks.

The researchers use a system called a routing harness to manage this process. The harness is like a traffic controller: it sends easy tasks to smaller or simpler models and difficult tasks to stronger models, while recording the results.

2. Main objectives and research questions

The researchers wanted to find out whether information collected while AI agents are working could be used to train better agents.

In simple terms, they asked:

  • Can records of an agent’s actions, tool use, mistakes, and successes become useful training examples?
  • Can a routing system estimate how difficult a task is?
  • Can those difficulty estimates help organize training from easier examples to harder ones?
  • Can a teacher AI guide a student AI while the student is producing its own answers?
  • Can evaluation results show which abilities the model needs to improve next?
  • Does this training improve performance on agent tasks, coding, tool use, and instruction following?
  • Can this process eventually become a repeating self-improvement loop?

The paper does not claim to have achieved a fully independent, self-improving AI. Instead, it presents an early prototype of the machinery needed for such a system.

3. Research methods and approach

Collecting agent experiences

The system records complete trajectories. A trajectory is the full history of an agent trying to complete a task, including:

  • The user’s request
  • The agent’s reasoning
  • Tool calls, such as searching or running code
  • Results returned by those tools
  • Mistakes and recovery attempts
  • The final outcome

The researchers divide these long histories into smaller user turns. A user turn starts with a request and includes the agent’s responses and tool actions until the next request or the end of the task.

They also group related turns into subscenes. A subscene is like a chapter in a conversation where the agent is working toward one local goal.

Checking the data

Before using these experiences for training, the researchers check them in several ways.

First, they use structural validation. This is similar to checking whether a form has been filled out correctly. They look for problems such as:

  • Missing responses
  • Tool calls without matching results
  • Events appearing in the wrong order
  • Confusing or incomplete endings

Second, they use semantic evaluation. This means judging what the interaction actually achieved. They examine six areas:

  1. Did the agent achieve the goal?
  2. Did it follow the instructions?
  3. Did it use tools correctly?
  4. Did its claims match the available evidence?
  5. Did it recover from mistakes?
  6. Did it finish properly?

Each area can receive a result such as PASS, WARN, or FAIL.

Using routing as a difficulty guide

The routing system places tasks into four levels:

Level Simple explanation
C0 Easy, limited-risk tasks
C1 Normal everyday tasks
C2 Tasks needing several steps or tool use
C3 Very difficult or high-reliability tasks

The researchers use these levels as an estimate of how much ability a task requires. Importantly, the level is treated as a guide for ordering examples, not as a perfect measure of difficulty.

Training with a curriculum

The researchers use curriculum learning, which is similar to teaching a student math by starting with basic problems and gradually introducing harder ones.

Training happens in three stages:

  1. The model sees mostly easier examples.
  2. It sees a mixture of easier and more demanding examples.
  3. It receives more difficult examples while still keeping some easier examples for balance.

This training method is called supervised fine-tuning, or SFT. In everyday language, the model studies examples of good agent behavior and adjusts its internal settings to imitate them.

The training examples include more than final answers. They can contain reasoning, tool calls, tool results, and visible responses. This helps the model learn the entire process of completing a task.

On-policy distillation

The paper also uses on-policy distillation. Imagine a student trying to solve a problem in its own way while a teacher watches. Instead of only showing the student old examples, the teacher observes the student’s current attempt and gives guidance.

Here:

  • The student is the model being trained.
  • The teacher is a stronger or fixed model.
  • The student generates its own responses.
  • The teacher shows which next words or actions would be more likely to be useful.

This is important because an AI may behave differently when it generates its own response than when it simply copies a recorded example.

Choosing future training data

After testing the model, the researchers create a capability profile showing where it is strong and weak. They then change the next training mixture to include more examples from weak areas.

This creates the intended feedback loop:

1
2
3

Run tasks → record experiences → evaluate abilities
      ↑                              ↓
Train an improved model ← choose data for weak areas

4. Main findings

The researchers tested NeoHorse-1 at two sizes:

  • A smaller model with about 4 billion parameters
  • A larger model with about 9 billion parameters

They evaluated the models on ten benchmarks covering:

  • Agent tasks
  • Tool use
  • Coding
  • Instruction following
  • Multi-step interaction

The average scores improved after agentic post-training:

Model Before training After training Improvement
NeoHorse-1-4B 58.94 64.87 +5.93
NeoHorse-1-9B 65.60 69.04 +3.44

The biggest improvements appeared in tasks involving:

  • Harness-based agents
  • Tool use
  • Executing several steps
  • Checking results
  • Recovering from errors

The smaller 4B model also became much closer, overall, to the larger 9B base model. This suggests that carefully chosen training experiences can help a smaller model perform more like a larger one on some tasks.

Why these results matter

The results suggest that an AI agent’s work history is valuable training material. The model can learn not only from whether its final answer was right, but also from:

  • How it planned
  • Which tools it chose
  • Whether it checked its work
  • How it reacted to errors
  • How it completed a task over several steps

The results also suggest that a routing system can do more than choose which model answers a request. Its records can help decide what the models should learn next.

5. Implications and possible impact

NeoHorse-1 presents a possible way to make AI development more automatic and continuous. Instead of humans manually collecting every training example, deployed agents could generate useful experiences while doing real tasks.

In the future, a system based on this idea might:

  • Notice that it often fails at a certain type of task.
  • Collect or create more examples of that task.
  • Train an updated model.
  • Test the updated model.
  • Repeat the process.

This could make AI agents better at practical activities such as coding, research, customer service, and using computer software.

However, the paper describes an early prototype, not a completely self-improving AI. Humans still design the training process, select the benchmarks, provide the teacher models, and decide how the data is used. The system also needs further testing to determine whether improvements continue over many training rounds and whether the model avoids learning incorrect or unsafe behaviors.

Overall, the paper’s main message is that the records created when AI agents work can become a powerful feedback system. By observing actions, measuring weaknesses, and choosing better training data, AI models may gradually become more capable and more useful.

View Paper Prompt  View All Prompts 

Knowledge Gaps

Knowledge gaps, limitations, and open questions

  • The proposed recursive self-improvement loop is not demonstrated across multiple completed training–deployment iterations; the paper reports a single post-training cycle rather than sustained recursive improvement.
  • It remains unclear whether the reported gains compound over successive iterations or eventually saturate, regress, or amplify undesirable behaviors.
  • The paper does not isolate the contribution of routing-guided data allocation from ordinary data scaling, repeated training, model quality improvements, or increased exposure to agentic trajectories.
  • The causal effect of the three-stage routing curriculum is unresolved because there is no full ablation comparing it with random ordering, conventional difficulty-based curricula, reverse ordering, or no curriculum.
  • The relative value of hard routing scores versus soft score-weighted ordering is not quantified.
  • The reliability and calibration of routing-estimated capability demand are not established; the paper does not report whether higher routing scores consistently predict task difficulty, failure probability, or training value.
  • It is unclear how routing scores transfer across model versions, deployments, harnesses, domains, and changing service-tier definitions.
  • The effects of policy-adjusted routing decisions, user overrides, service availability, and multi-model aggregation on the training signal are not separately evaluated.
  • The paper does not determine whether routing signals encode genuine capability demand or confounding factors such as task length, domain, risk level, prompt style, or infrastructure constraints.
  • The capability-guided allocation procedure is described conceptually, but its exact optimization rule, sampling proportions, update frequency, and stability properties are insufficiently specified for independent reproduction.
  • The method for converting a deficiency profile into a new training mixture is not compared against simpler baselines such as uniform resampling, loss-based sampling, failure-only sampling, or uncertainty-based allocation.
  • The paper does not evaluate whether reallocating data toward underperforming regions causes catastrophic forgetting or reduces performance on already-strong capabilities.
  • The training corpus is described only approximately as containing 10510^5–10610^6 trajectories; detailed counts by source, domain, routing tier, quality status, outcome, and model/harness are not provided.
  • The mixture and weighting of proprietary harness-generated data versus public datasets are not fully disclosed, limiting reproducibility and making it difficult to attribute performance gains to particular data sources.
  • The paper does not report how public datasets and harness trajectories differ in formatting, quality, task distribution, or contamination risk.
  • The effectiveness of retaining current-turn reasoning while omitting earlier-turn reasoning is not tested against alternative context policies, including retaining all reasoning, omitting all reasoning, or selectively summarizing prior reasoning.
  • It remains unknown whether training on recorded reasoning and tool-call traces improves actual execution competence or mainly increases imitation of trajectory style.
  • The study does not assess whether the models reproduce inaccurate, unsafe, or strategically misleading reasoning present in source trajectories.
  • Structural validity is distinguished from task success, but the paper does not report how frequently structurally valid trajectories contain incorrect tool choices, fabricated evidence, invalid arguments, or silently failed tasks.
  • The semantic judge’s accuracy, inter-rater agreement, calibration, and susceptibility to model-generated evidence are not quantified.
  • The six semantic quality dimensions are treated as independent, but their correlations, redundancy, and relative importance for downstream performance remain unexplored.
  • The paper does not evaluate the impact of annotation errors, low-confidence labels, WARN cases, or NOT_EVALUATED cases on training outcomes.
  • The thresholds and policies for admitting, reviewing, recovering, or quarantining trajectories are not accompanied by sensitivity analyses.
  • The benefits of training on informative failures and recovery trajectories are not isolated from the benefits of training on successful trajectories.
  • The proposed subscene-level labeling scheme has not been validated for consistency across annotators, domains, languages, or long multi-goal conversations.
  • The extent to which the evaluation suite is representative of real deployment traffic is unknown; most reported tasks are benchmarked environments rather than naturally occurring user interactions.
  • The evaluation is limited to text-based tasks, leaving multimodal perception, GUI interaction, speech, and other deployment settings unexamined.
  • The paper does not establish whether gains generalize to unseen tools, harnesses, APIs, environments, or domains not represented in the training trajectories.
  • The reported benchmarks do not fully test long-horizon reliability, persistent memory, changing environments, adversarial users, or nonstationary tool behavior.
  • The evaluation does not systematically measure inference cost, latency, energy use, token consumption, or tool-call efficiency, despite the method being motivated by routing and service allocation.
  • Macro-average benchmark gains may conceal regressions on individual tasks, domains, failure modes, or capability tiers; per-category variance and statistical significance are not sufficiently characterized in the provided text.
  • The paper does not report confidence intervals, repeated-seed results, or statistical tests needed to determine whether the improvements are robust.
  • Comparisons with models based on official reports or blog posts are not necessarily matched for prompts, inference settings, harnesses, or evaluation procedures, limiting the strength of cross-model conclusions.
  • The effects of sampling temperature, thinking mode, output limits, penalty settings, and harness configuration are not studied through controlled sensitivity experiments.
  • The contribution of supervised fine-tuning and routing-guided on-policy distillation is not separately quantified through a complete factorial ablation.
  • The on-policy distillation procedure uses a fixed teacher, but the paper does not investigate teacher quality, teacher–student size ratios, teacher diversity, teacher refresh strategies, or teacher failure modes.
  • The consequences of using reverse KL with top-KK candidate tokens plus an aggregated residual bin are not compared with forward KL, full-vocabulary distillation, alternative divergences, or different values of KK.
  • The OPD objective supervises student-generated prefixes but does not address exposure to low-quality, unsafe, or tool-invalid student trajectories beyond the teacher’s token-level distribution.
  • It remains unclear whether routing-guided context scheduling improves OPD because of curriculum ordering specifically or simply because it changes the task mixture and rollout distribution.
  • The paper does not evaluate whether the student learns to exploit teacher distributions without acquiring reliable environment-grounded behavior.
  • Tool execution during on-policy distillation is insufficiently specified, leaving open whether the student interacts with live environments, replayed observations, or static contexts and how tool outcomes affect learning.
  • The models are evaluated primarily at 4B and 9B parameters, so the scaling behavior of the approach at smaller, larger, or substantially different architectures remains unknown.
  • The claim that the 4B model narrows the gap to the 9B base model is aggregate-level; the paper does not establish whether this reflects broad capability improvement or concentration of gains in a subset of benchmarks.
  • The method’s robustness to distribution shift, changing routing policies, new tools, and evolving user behavior is not experimentally established.
  • The paper does not analyze safety, security, privacy, or misuse risks arising from storing and training on deployment trajectories that may contain sensitive user inputs, tool outputs, credentials, or executable actions.
  • No mechanism is presented for preventing recursive feedback loops from reinforcing systematic evaluator errors, routing biases, spurious correlations, or deployment-specific artifacts.
  • The proposed evaluation–selection–update loop may optimize toward the selected benchmarks and deficiency metrics; its resistance to Goodhart effects and benchmark overfitting remains unresolved.
  • The paper does not define stopping criteria, rollback mechanisms, or safeguards for later recursive iterations that produce regressions or increasingly undesirable behavior.
  • The computational and operational cost of continuously collecting, validating, judging, labeling, reallocating, and retraining on trajectories is not reported.
  • The method’s dependence on proprietary or rapidly changing harness components, such as OpenSquilla and versioned service tiers, may limit reproducibility and portability to other serving infrastructures.
  • The paper does not compare routing-guided post-training with direct reinforcement learning from executable outcomes, preference optimization, rejection sampling, or joint model–harness optimization under matched data and compute budgets.
  • It remains open whether the routing harness itself can be automatically improved, rather than merely supplying data and scores to improve the underlying model.
  • The relationship between improved benchmark execution and genuine recursive self-improvement is conceptually asserted but not formally defined or measured; in particular, the paper does not quantify whether the system improves its ability to improve future training procedures.
View Paper Prompt  View All Prompts 

Practical Applications

Immediate Applications

  • Cost- and capability-aware AI routing for production services (software, customer support, enterprise AI) Organizations can deploy a routing harness that classifies each request into capability tiers such as routine, general-purpose, multi-step, or high-reliability. The system can then select smaller models for low-risk requests and stronger or multi-agent configurations for complex tasks. This can reduce inference cost and latency while preserving quality. Potential tools/workflows: model cascades, ticket triage, adaptive chatbot backends, escalation systems, and reliability-aware API gateways. Dependencies: reliable routing predictions, stable tier definitions, monitoring for misrouted high-risk requests, and explicit policies for privacy, latency, and cost.
  • Agentic customer-service automation (retail, telecom, airlines, financial services) The demonstrated improvements in multi-turn tool use and task completion can support agents that retrieve account information, modify reservations, process returns, or troubleshoot services through controlled APIs. The trajectory format provides a way to train agents on complete interaction histories rather than isolated question–answer pairs. Dependencies: authenticated tools, transactional safeguards, human escalation, audit logs, and domain-specific evaluation. The benchmark gains do not by themselves establish safety for unsupervised financial or customer-affecting actions.
  • Coding assistants that execute and verify software tasks (software engineering, DevOps) NeoHorse-style training can be applied to coding agents that edit repositories, run tests, inspect build errors, and revise implementations. Preserving tool calls, observations, recovery attempts, and terminal outcomes is directly useful for learning repository-level workflows rather than only code completion. Potential products: IDE agents, pull-request assistants, automated bug-fixing systems, and CI failure-resolution agents. Dependencies: sandboxed execution, repository permissions, deterministic test environments, protection against destructive commands, and verification of generated patches.
  • Structured trajectory logging and replay for AI operations (MLOps, enterprise software, academia) The paper’s representation of trajectories as linked user turns, subscenes, tool calls, observations, outcomes, and routing records can be implemented as an observability standard for agent systems. Teams can replay failures, inspect causal tool-call chains, compare model versions, and identify where an interaction broke down. Potential tools: trajectory databases, agent trace viewers, replay systems, quality dashboards, and data-lineage platforms. Dependencies: consistent serialization, schema versioning, privacy-preserving storage, and accurate association between tool calls and results.
  • Automated quality assurance for tool-using agents (software, robotics, business-process automation) The structural validation pipeline can serve as a deployable gate that detects malformed conversations, orphaned tool results, duplicate identifiers, missing responses, unresolved calls, and ambiguous terminal states. The six semantic dimensions—goal attainment, instruction adherence, tool use, evidence consistency, recovery, and termination—can provide a practical evaluation checklist. Dependencies: deterministic checks should be used wherever possible; model-based judges require evidence-grounding, calibration, coverage tracking, and human review for consequential decisions.
  • Adaptive data curation for post-training (AI research and model development) Developers can use routing demand, task attributes, execution quality, and outcomes to construct training mixtures targeted at observed weaknesses. Instead of uniformly adding more data, a training pipeline can increase coverage for underperforming combinations such as high-demand tool use, error recovery, or particular application domains. Potential workflow: collect trajectories → validate and decontaminate → label scene, goal, and outcome → evaluate capability gaps → reallocate the next training mixture. Dependencies: sufficient trajectory volume, representative deployment data, protection against feedback-loop bias, and strict separation between training and evaluation sets.
  • Routing-guided curriculum learning for existing models (academia, industrial fine-tuning) The paper provides an immediately reusable training strategy: order examples using predicted capability demand, gradually introduce more demanding interactions, and retain lower-demand examples for coverage. This can be applied to supervised fine-tuning without changing the underlying loss function. Dependencies: routing scores must correlate sufficiently with task demand; otherwise the curriculum may encode serving policy or availability rather than difficulty. The approach should be compared against random, heuristic, and outcome-based curricula.
  • Capability dashboards for model and policy management (AI governance, enterprise operations) The prediction–action–outcome separation can support dashboards showing which task types are routed to which service tiers, where failures occur, how much recovery costs, and whether routing decisions are justified by outcomes. This can inform model selection, capacity planning, and service-level agreements. Dependencies: calibrated metrics, stable tier semantics across model versions, representative traffic, and controls preventing sensitive user information from appearing in reports.
  • Human-in-the-loop escalation and review (healthcare administration, legal services, finance, public services) Routing and structured quality signals can trigger escalation when an agent detects high capability demand, failed verification, poor evidence consistency, or repeated recovery attempts. This is particularly suitable for systems that already require human approval before consequential actions. Dependencies: clearly defined escalation thresholds, qualified reviewers, response-time guarantees, and a policy that treats uncertainty or missing evidence as a reason for review rather than success.
  • Research and teaching infrastructure for agentic AI (academia and education) The open model, repository, data organization, and evaluation methodology can be used to reproduce experiments on trajectory SFT, curriculum learning, on-policy distillation, routing, and agent evaluation. Universities can build course laboratories around replayable tool-use environments and compare models under common harnesses. Dependencies: availability and licensing of the referenced models and datasets, adequate compute, reproducible harness versions, and careful handling of benchmark contamination.

Long-Term Applications

  • Sustained harness-mediated recursive self-improvement (AI research) The proposed evaluation–selection–update loop could become a continual improvement system in which deployed interactions reveal capability gaps, the data allocator selects new training material, and updated models are returned to the harness for another iteration. This would move beyond one-time post-training toward an operational data flywheel. Dependencies: evidence that improvements persist across multiple iterations; safeguards against performance collapse, distribution drift, reward hacking, benchmark overfitting, and self-reinforcing errors; and independent evaluations that are not generated by the improving system itself.
  • Continual adaptation to changing enterprise workflows (business software, operations) A production agent could learn from newly introduced tools, changed APIs, updated policies, and evolving user behavior. Routing records and subscene labels could identify new capability regions requiring training, enabling more rapid adaptation than periodic manual dataset construction. Dependencies: continual-learning methods, strict rollback mechanisms, data governance, protection against poisoning, and validation before updated models are exposed to real users.
  • Joint optimization of models, routing policies, and agent harnesses (AI infrastructure) Future systems could optimize not only model weights but also tool selection, context management, service-tier policies, recovery strategies, and aggregation mechanisms. The paper’s separation of predicted demand, policy-adjusted route, served model, and outcome provides the instrumentation needed for such joint optimization. Dependencies: a well-defined objective balancing quality, cost, latency, safety, and fairness; stable interfaces; causal attribution between harness changes and model changes; and methods that prevent the system from optimizing easily measured metrics at the expense of actual task success.
  • On-policy distillation for self-generated, domain-specific workflows (healthcare, science, engineering, finance) A stronger teacher could supervise student responses on prefixes generated by the student itself, including domain-specific tool calls and recovery states. This may help close the gap between training on expert trajectories and deployment, where the model encounters its own mistakes and unusual states. Dependencies: access to a trustworthy teacher, compatible tokenization and tool templates, high compute cost, reliable handling of unsafe student actions, and validation that distillation does not merely reproduce teacher biases or errors.
  • Autonomous research and engineering agents (scientific discovery, software, robotics) With additional advances, the loop could train agents to design experiments, write and test code, analyze results, revise hypotheses, or improve their own scaffolding. In robotics, analogous trajectory records could connect perception, planning, control actions, environmental feedback, and recovery behavior. Dependencies: grounded environments, reliable outcome verification, simulation-to-real transfer for robotics, domain experts, safety constraints, and substantially stronger long-horizon planning than established by the current text-only results.
  • High-reliability agents in regulated sectors (healthcare, finance, law, public administration) The structured quality dimensions and evidence-grounded evaluation could support agents that perform constrained professional workflows, such as preparing clinical documentation, reconciling financial records, or assembling legal research. Capability-aware routing could reserve the strongest models and additional verification for sensitive cases. Dependencies: regulatory approval, explainability, auditability, access controls, validated domain datasets, liability allocation, human sign-off, and evidence that benchmark performance transfers to real operational environments.
  • Personalized daily-life assistants with adaptive service tiers (consumer technology) The methods could support assistants that handle simple reminders and information requests locally while escalating complex planning, travel, shopping, or software-operation tasks to stronger models or multi-step agents. Subscene-level tracking could preserve goals across interruptions, modifications, and resumptions. Dependencies: user consent, persistent but privacy-preserving memory, dependable integrations with calendars and services, protection against unauthorized transactions, and robust handling of ambiguous user goals.
  • Fairness- and risk-aware allocation of computational resources (policy, public-sector AI, cloud infrastructure) Routing logs could be extended to study whether high-quality service is allocated equitably across languages, user groups, domains, or geographic regions. Policy makers and platform operators could require reporting on route decisions, failure rates, escalation rates, and resource disparities. Dependencies: fairness definitions appropriate to the application, representative data, privacy-preserving measurement, transparency about tier policies, and safeguards against using capability estimates as proxies for protected characteristics.
  • Standardized benchmarks for recursive and harness-level improvement (academia and policy) The paper points toward evaluations that measure not only final answers but also tool use, verification, recovery, termination, routing quality, data-allocation decisions, and performance across successive improvement cycles. Such benchmarks could distinguish genuine capability growth from test-specific optimization. Dependencies: independent task generation, contamination-resistant evaluation, reproducible harnesses, long-term study protocols, and metrics that capture cost, latency, reliability, and safety alongside accuracy.
  • Automated discovery of training and deployment failures (AI safety and governance) A mature version of the pipeline could detect when a model is systematically failing in a particular scene, goal type, outcome state, or capability tier, then recommend additional data, a new tool constraint, a routing change, or human intervention. This would turn operational telemetry into a semi-automated safety and quality-management system. Dependencies: causal diagnosis rather than correlation alone, independent monitoring, adversarial testing, secure data pipelines, and human approval for changes that affect training or deployment behavior.
View Paper Prompt  View All Prompts 

Glossary

  • Agentic post-training: Training an AI model using interaction trajectories, tool use, and execution feedback. “Our system couples a heterogeneous model pool with intelligent routing”
  • Agentic routing: Dynamically selecting a model or sub-agent for a request based on its requirements and interaction state. “Adding agentic routing allows this layer to select models according to the request and the evolving interaction state”
  • Causal attention: An attention mechanism in which each token can attend only to preceding tokens, preventing access to future information. “With causal attention, each assistant response can use earlier actions and tool results in the sequence, but not later ones.”
  • Capability-guided data allocation: Adjusting the composition of training data according to measured strengths and weaknesses of a model. “The quality dimensions of Section~\ref{sec:data-quality}, semantic attributes of Section~\ref{sec:data-labeling}, and routing signals of Section~\ref{sec:routing-data} organize deployment trajectories by user intent, capability demand, execution quality, and outcome.”
  • Capability demand: The estimated level of model ability required to complete an interaction successfully. “The harness router operates at the user-turn level and estimates capability demand from the current request, recent dialogue, previous routing decisions, and available execution state”
  • Coarsened distribution: A probability distribution simplified by grouping multiple outcomes into shared categories or bins. “yielding coarsened distributions P~θ,r,t\widetilde{P}_{\theta,r,t} and Q~r,t\widetilde{Q}_{r,t} over K+1K+1 bins.”
  • Curriculum learning: A training strategy that introduces examples in an ordered progression, often from easier to more difficult cases. “Curriculum learning is a training strategy that presents examples according to an estimated notion of difficulty”
  • Decontamination: Removing training examples that overlap with evaluation data to prevent test-set leakage. “the same matching infrastructure screens every training candidate against our evaluation suites”
  • Deduplication: Identifying and removing identical or near-identical data records. “The corpus is deduplicated at exact and near-duplicate granularity”
  • Direct preference optimization: A method for training LLMs directly from preference comparisons without separately fitting a reward model. “Llama 3 extends this recipe with synthetic multi-step tool-use data and iterative SFT, rejection sampling, and direct preference optimization”
  • Distribution gap: The difference between the data distribution used for training and the distribution encountered during deployment or evaluation. “On-policy distillation (OPD) reduces this distribution gap”
  • Execution trajectory: A recorded sequence of model actions, tool interactions, observations, and outcomes during task completion. “Post-training data for an agentic model is not adequately represented by static instruction--response pairs.”
  • Harness: An execution framework that manages an agent’s context, tools, environment interactions, and feedback. “A harness is the execution layer that manages an agent's context, tools, and interaction with its environment”
  • Heterogeneous model pool: A collection of models with differing capabilities, sizes, or specializations. “Building on the harness-native data flywheel of Agentic Routing~\citep{liu2026agenticrouting}, our system design combines a heterogeneous model pool with multiple harnesses”
  • Inference configuration: The set of parameters and serving choices controlling a model’s generation process. “Models, pricing, and inference configurations may change across deployments”
  • Interleaved thinking: A pattern in which reasoning and tool calls alternate as the model responds to an interaction. “This pattern is referred to as \emph{interleaved thinking}”
  • Long-horizon interaction: An interaction requiring many sequential steps, decisions, or tool operations before completion. “Search-R1, ReTool, and RAGEN study this paradigm for search, tool use, and long-horizon interaction”
  • Loss mask: A binary indicator specifying which tokens contribute to the training loss. “We use a binary token-level loss mask mi,tm_{i,t}”
  • Macro-average: An average calculated by first computing scores for individual tasks or categories and then giving each equal weight. “post-training lifts the macro-average score of the 4B model from 58.94 to 64.87”
  • On-policy distillation: Distillation in which the student generates its own inputs or responses while receiving supervision from a teacher. “On-policy distillation (OPD) provides teacher supervision on these student-generated prefixes.”
  • Orphan observation: A tool observation that lacks a corresponding initiating tool call or valid causal connection. “The same stage detects missing responses, orphan observations, duplicated or conflicting tool-call identifiers”
  • Post-training: Model training performed after initial pretraining, typically involving supervised fine-tuning, preference optimization, or distillation. “NeoHorse-1's post-training methods provide the learning component of this design.”
  • Prediction--action--outcome record: A structured record linking a system’s prediction, selected action, and resulting task outcome. “Aligning these fields yields a prediction--action--outcome record”
  • Recursive self-improvement: An iterative process in which an AI system uses experience or evaluation to improve itself or its improvement process. “Recursive self-improvement (RSI) denotes an iterative process in which an AI system uses experience, evaluations, or generated artifacts to improve its model, scaffold, or improvement procedure”
  • Rejection sampling: A sampling procedure that generates candidate outputs and retains only those meeting a desired criterion. “Llama 3 extends this recipe with synthetic multi-step tool-use data and iterative SFT, rejection sampling, and direct preference optimization”
  • Reverse Kullback–Leibler divergence: A divergence measure that compares a student distribution to a teacher distribution in the direction DKL(P∥Q)D_{\mathrm{KL}}(P\Vert Q). “we minimize the response-normalized reverse KL”
  • Routing-guided curriculum: A curriculum whose ordering is determined by capability estimates produced by a routing system. “We use routing estimates of this demand to organize SFT examples into a curriculum.”
  • Routing-guided on-policy distillation: On-policy distillation in which routing scores determine the order of contexts used for student generation. “Routing-guided on-policy distillation.”
  • Semantic event stream: A normalized, ordered representation of meaningful events in an interaction. “For structurally usable trajectories, we construct a normalized semantic event stream”
  • Semantic judge: A model or evaluation component that interprets trajectory evidence to assess task-level properties. “Cases that require task-level interpretation are evaluated by a semantic judge”
  • Service tier: A routing category representing a particular level of model capability, cost, or reliability. “It assigns each turn to one of four service tiers”
  • Subscene: A group of adjacent user turns sharing a local goal within a larger interaction. “A subscene groups adjacent user turns that share a local goal”
  • Supervised fine-tuning (SFT): Training a pretrained model to reproduce labeled target outputs. “Trajectory-based supervised fine-tuning (SFT) provides a practical route for transferring planning and tool use into model parameters.”
  • Teacher logits: The teacher model’s pre-softmax token scores used to supervise a student model. “the teacher supplies token-level logits on student-visited states”
  • Token-level divergence: A divergence computed separately between model probability distributions at each generated token position. “Each response contributes its average token-level divergence”
  • Trajectory-based supervision: Training supervision derived from complete interaction sequences rather than isolated input–output pairs. “Trajectory-based supervised fine-tuning (SFT) provides a practical route for transferring planning and tool use into model parameters.”
  • User turn: A user request together with the assistant responses and tool interactions that follow it until the next request or termination. “We define a user turn as a user request together with the assistant responses and tool interactions that follow it”
  • Zero-shot generalization: The ability to perform tasks or handle conditions not directly represented in training examples. “IFBench~\citep{pyatkin2025generalizing} focuses on generalization to diverse and previously unseen constraints”
View Paper Prompt  View All Prompts 

Tweets

Sign up for free to view the 2 tweets with 53 likes about this paper.

Sign Up for Free