NeoHorse-1: Agent-Based RSI Models

• NeoHorse-1 are language models developed to explore recursive self-improvement (RSI) through post-training with a routing harness, utilizing heterogeneous model pooling, trajectorial recording, structured control, and fine-tuning methods.
• The models, NeoHorse-1-4B and NeoHorse-1-9B, are built on Qwen3.5-4B and Qwen3.5-9B, and are designed to optimise performance by generating actions within an agentic execution interface
• Performance improvements were demonstrated in harness-based agents, tool use, multi-step interactions, coding, and environmental feedback tasks, though stability across multiple RSI iterations was not established.

NeoHorse-1 is a family of agent-native LLMs developed to investigate recursive self-improvement (RSI) through agentic post-training with a routing harness. The system combines a heterogeneous model pool, capability-demand routing, execution-trajectory recording, structural and semantic quality control, routing-guided supervised fine-tuning (SFT), routing-guided on-policy distillation (R-OPD), and capability-guided allocation of subsequent training data. Its central premise is that deployed agent interactions can function simultaneously as training experience and as measurements of capability demand and deficiency. The reported models, NeoHorse-1-4B and NeoHorse-1-9B, are based on Qwen3.5-4B and Qwen3.5-9B, respectively (Team et al., 8 Sep 2026).

1. Conceptual foundations and system objectives

NeoHorse-1 frames RSI as an operational feedback process rather than as an unspecified capability of a model. The proposed cycle is:

$$\text{deployment} \rightarrow \text{routing and interaction records} \rightarrow \text{quality evaluation and labeling} \rightarrow \text{training-mixture construction} \rightarrow \text{model update} \rightarrow \text{re-deployment}.$$

The system treats an agent trajectory as both a source of behavioral supervision and an observation of which capabilities were demanded, successfully executed, or inadequately supported. The intended mechanism therefore connects deployment-time execution with post-training data selection.

The paper describes NeoHorse-1 as an initial prototype rather than a definitive demonstration of autonomous RSI. The reported experiments implement one principal post-training cycle and demonstrate performance improvements, but do not establish sustained gains over multiple recursive iterations. In particular, the reported system does not recursively improve its own training algorithm, routing harness, or model pool.

NeoHorse-1 is organized around four operational components:

1. Heterogeneous model pool: models and service configurations with different capabilities and costs.
2. Routing harness: an execution layer that selects a model or service configuration for each request.
3. Trajectory recording: persistent records of reasoning, tool calls, observations, failures, recovery attempts, and outcomes.
4. Evaluation and allocation: structured signals identifying capability regions that should receive additional or rebalanced supervision.

The harness-mediated design is significant because the model generates actions within an environment rather than only isolated text responses. An execution may contain reasoning, tool calls, observations, revised reasoning, verification, and a final artifact. This makes the harness an agent-native learning interface: it preserves the conditions under which the model acted and the consequences of those actions.

2. Routing harness and capability organization

A routing harness manages system instructions, conversation state, tools, tool specifications, tool calls, returned observations, environment interaction, task termination, model selection, retries, recovery, verification, judging, logging, and trajectory storage. NeoHorse-1 uses multiple harnesses and execution environments, including OpenSquilla. OpenSquilla is used for QwenClawBench and PinchBench, while other benchmarks use native harnesses or frameworks.

The model pool is divided into four relative service tiers:

Tier	Operational description	Typical demand
C0	Bounded, low-risk requests	Simple interactions
C1	General-purpose default requests	Ordinary assistant tasks
C2	Multi-step reasoning and execution	Tool-mediated tasks
C3	Maximum capability or reliability	High-demand or high-risk tasks

Routing operates at the user-turn level. The router estimates capability demand from the current request, recent dialogue, previous routing decisions, and available execution state. The system distinguishes among three routing quantities:

• Predicted demand: the router’s raw capability-demand prediction.
• Policy-adjusted choice: the tier selected after deployment policy, availability, cost, risk, or user constraints.
• Served tier: the tier or service configuration that actually handled the request.

This distinction prevents the served model from being interpreted as a clean difficulty label. A request may be routed to a stronger model because of temporary service constraints, a user override, previous failure, or deployment policy rather than intrinsic task complexity.

The routing record is linked to the complete trajectory, enabling analysis of the relationship:

$$\text{capability demand} \rightarrow \text{service allocation} \rightarrow \text{execution quality} \rightarrow \text{outcome}.$$

Routing signals are subsequently used to order training examples and construct a three-stage curriculum. The paper treats routing as an allocation signal rather than an objective, invariant measure of difficulty. Router miscalibration, policy-dependent tier semantics, and changing demand as models improve are therefore important limitations.

3. Interaction data, serialization, and quality control

NeoHorse-1 organizes experience at three linked granularities: trajectories, user turns, and subscenes.

A trajectory is a complete harness execution containing user requests, assistant responses, reasoning, tool calls, tool results, environment observations, recovery attempts, terminal events, and final outcomes. A user turn begins with a user request and ends at the next user request or task termination; tool results and harness-injected messages do not initiate new turns. A subscene groups adjacent user turns that share a local goal and may include continuation, modification, interruption, resumption, or multiple related requests.

The primary corpus contains on the order of $10^5$–$10^6$ harness-generated trajectories. It is supplemented with public data covering instruction following, dialogue, reasoning, tool use, code, agent interaction, and preferences. Named sources include OASST2, Aya, OpenThoughts3, OpenR1-Math, ToolACE, APIGen, Nemotron-SWE, HelpSteer2, and UltraFeedback. The exact number of trajectories and tokens is intended to be recorded in a training manifest after serialization, deduplication, and tokenizer freezing.

A major design choice is preservation of interleaved reasoning and tool calls. For the current user turn, the serialized sequence retains the current request, current-turn reasoning when available, current-turn tool calls and arguments, tool observations, visible assistant responses, end-of-response tokens, relevant historical dialogue, earlier visible assistant outputs, earlier tool calls and results, system instructions, tool specifications, and harness-provided context. Earlier hidden reasoning is omitted, while earlier visible responses and tool interactions remain available as context.

The training sequence therefore preserves an execution context rather than reducing the example to an isolated question–answer pair. Using the Qwen3.5 chat template and tool-call format, the causal language-model objective conditions each token only on preceding tokens:

$p_\theta(x_t\mid x_{<t}).$

Structural admission

Structural validation is rule-based. It reconstructs and checks user requests, assistant responses, tool calls, tool observations, terminal events, causal event order, payload readability, supported message structures, request–response presence, tool-call/result matching, execution branches, and unique tool-call identifiers.

The pipeline detects missing responses, orphan observations, duplicated or conflicting tool-call identifiers, unresolved internal calls, and ambiguous terminal branches. Each trajectory receives one of three operational outcomes:

• Internally complete: proceeds directly.
• Partially recoverable: only causally closed sub-trajectories are retained.
• Quarantined: ambiguous or unusable supervision is excluded.

Structural validity ensures that a trajectory can be reliably serialized and replayed. It does not establish that the agent selected the correct tools or completed the task successfully.

Semantic evaluation

Structurally usable trajectories are evaluated along six dimensions:

1. Goal attainment.
2. Instruction adherence.
3. Tool use.
4. Evidence consistency.
5. Error recovery.
6. Termination.

Each dimension receives PASS, WARN, FAIL, or NOT_EVALUATED, with coverage stored separately. Missing evidence or an interrupted judge call is not treated as a positive result. Deterministic failure checks identify missing final responses, unresolved tool calls, and unrecovered terminal errors. Other cases are evaluated by a semantic judge restricted to evidence explicitly present in the trajectory. Long trajectories may be evaluated in segments and aggregated at turn level, preserving the distinction between intermediate failure and eventual recovery.

Subscene labeling

Subscene-level characterization uses three complementary axes:

• Scene: what the user is doing and in what context, including task type, application domain, use context, and asking versus doing.
• Goal: what the user expects to achieve and how success is judged, including acceptance criteria and relations such as new, continued, modified, resumed, or ambiguous.
• Outcome: the verifiable result, distinguishing actual satisfaction, partial completion, failure, and termination without successful task completion.

Structural facts, deterministic labels, routing metadata, loss masks, and reasoning-retention policy remain separate from semantic scene, goal, and outcome labels.

4. Post-training methodology

Routing-guided supervised fine-tuning

For each serialized user-turn sequence, the loss mask marks only assistant target spans in the current user turn:

• reasoning, when retained;
• serialized tool calls and arguments;
• visible assistant responses;
• end-of-response tokens.

System instructions, tool specifications, historical messages, user messages, tool results, harness context, and padding receive zero loss. The resulting masked SFT objective is:

$$\mathcal{L}_{\mathrm{SFT}} = - \frac{ \sum_{i\in\mathcal{B}} \sum_{t=2}^{T_i} m_{i,t} \log p_\theta(x_{i,t}\mid x_{i,<t}) }{ \sum_{i\in\mathcal{B}} \sum_{t=2}^{T_i} m_{i,t} }.$$

This trains the model to reproduce interleaved reasoning–action–response behavior while conditioning on the original execution context.

Routing supplies either a hard tier score or a soft expected-tier score. For an assigned tier $k_i\in\{0,1,2,3\}$ and normalized routing support vector $\pi_{i,k}$, the curriculum score is:

$$s_i = \begin{cases} k_i, & \text{hard ordering},\ \displaystyle\sum_{k=0}^{3} k\,\pi_{i,k}, & \text{soft ordering}. \end{cases}$$

The score orders examples; it does not reweight the token-level SFT loss. It is a whole-example ordering proxy and does not imply that every action within a trajectory has identical difficulty.

Training uses three approximately equal-sized stages:

• Stage 1: lower routing-demand examples.
• Stage 2: intermediate and increasingly demanding examples.
• Stage 3: higher-demand examples while retaining some lower-demand coverage.

The optimizer and learning-rate schedule continue across stages without being reset. Some lower-scored examples are reserved for later stages so that the final stage is not composed exclusively of high-demand interactions.

Routing-guided on-policy distillation

SFT trains on recorded assistant responses, whereas deployment conditions the model on its own generated prefixes. R-OPD addresses this distribution gap by allowing a student to generate responses from recorded contexts while a fixed teacher supplies token-level distributions on the states visited by the student.

A rollout student generates reasoning, tool calls, and visible text from recorded pre-response contexts. The teacher evaluates next-token distributions conditioned on the recorded context plus the student-generated preceding tokens. The rollout checkpoint is periodically refreshed but remains fixed during optimization on a collected response batch.

The teacher and student distributions are compressed to the student’s top-$K$ candidates plus one bin containing all remaining vocabulary probability. The paper uses a response-normalized reverse-KL objective:

$$\text{capability demand} \rightarrow \text{service allocation} \rightarrow \text{execution quality} \rightarrow \text{outcome}.$$0

Routing determines which contexts are encountered and in what progression; on-policy generation determines the prefixes on which supervision is applied; and the teacher provides dense token-level guidance. The three-stage routing curriculum is applied to the context distribution used for R-OPD.

Capability-guided allocation

After evaluation, results are aggregated by scene and task attributes, goal types, outcome states, quality dimensions, routing tiers, benchmark or environment, and execution behavior. This produces a deficiency profile. Underperforming capability regions receive increased supervision while broad coverage is retained.

The paper does not specify a single explicit optimization equation for allocation. Its mechanism is instead described as a feedback process:

$$\text{capability demand} \rightarrow \text{service allocation} \rightarrow \text{execution quality} \rightarrow \text{outcome}.$$1

Positive trajectories provide verified supervision. Failures identify where additional or rebalanced data is needed rather than being optimized solely through a separate failure loss.

5. Evaluation–selection–update loop

The proposed recursive loop is:

$$\text{capability demand} \rightarrow \text{service allocation} \rightarrow \text{execution quality} \rightarrow \text{outcome}.$$2

Here, $$\text{capability demand} \rightarrow \text{service allocation} \rightarrow \text{execution quality} \rightarrow \text{outcome}.$$3 is the current model checkpoint or model-pool configuration; $$\text{capability demand} \rightarrow \text{service allocation} \rightarrow \text{execution quality} \rightarrow \text{outcome}.$$4 is the interaction-trajectory set; $$\text{capability demand} \rightarrow \text{service allocation} \rightarrow \text{execution quality} \rightarrow \text{outcome}.$$5 contains structured quality, routing, semantic, and outcome records; $$\text{capability demand} \rightarrow \text{service allocation} \rightarrow \text{execution quality} \rightarrow \text{outcome}.$$6 is the allocation decision; $$\text{capability demand} \rightarrow \text{service allocation} \rightarrow \text{execution quality} \rightarrow \text{outcome}.$$7 is the next training mixture; and $$\text{capability demand} \rightarrow \text{service allocation} \rightarrow \text{execution quality} \rightarrow \text{outcome}.$$8 is the updated model.

The end-to-end workflow comprises:

1. Request and routing: capability-demand prediction, policy-adjusted tier selection, service allocation, and recording of the served tier.
2. Harness execution: reasoning, visible responses, tool calls, observations, retries, recovery, termination, and final artifacts.
3. Trajectory recording: linking execution events to user turns, subscenes, routing records, outcomes, model versions, and harness versions.
4. Structural admission: classifying trajectories as complete, partially recoverable, or quarantined.
5. Semantic evaluation: assigning six-dimensional quality records with coverage.
6. Subscene labeling: recording scene, goal, acceptance criteria, and outcome.
7. User-turn serialization: preserving historical and harness context while masking only current-turn assistant targets.
8. Routing-guided SFT: ordering examples by routing demand.
9. Routing-guided OPD: sampling pre-response contexts, generating student rollouts, and applying teacher supervision.
10. Evaluation and allocation: measuring performance, identifying deficiencies, and constructing the next mixture.
11. Re-deployment: returning the updated model to the heterogeneous serving pool.

The design’s RSI claim is therefore architectural and operational: the deployed system generates evidence that determines what subsequent model updates should learn.

6. Reported results, limitations, and significance

The evaluation suite covers harness-based agents, tool use, coding, and instruction following. The listed benchmarks are QwenClawBench, WorkBuddy Bench, PinchBench, VitaBench, BFCL V4, $$\text{capability demand} \rightarrow \text{service allocation} \rightarrow \text{execution quality} \rightarrow \text{outcome}.$$9-Bench, HumanEval, LiveCodeBench v6, IFEval, and IFBench.

Model	Base macro-average	Post-training macro-average	Improvement
NeoHorse-1-4B	58.94	64.87	+5.93
NeoHorse-1-9B	65.60	69.04	+3.44

The gains are concentrated particularly in harness-based execution, tool use, multi-step interaction, coding, and recovery under environmental feedback. Instruction-following scores are more stable, especially for the stronger 9B base.

In a project-scheduling case, NeoHorse-1-4B retrieves a missing email constraint, recomputes the schedule, verifies it, and saves the artifact correctly, whereas Qwen3.5-4B uses outdated information. In a WorkBuddy code-repair task, NeoHorse-1-9B performs an edit–test–inspect–repair loop, while NeoHorse-1-4B stops after an incomplete implementation. In a PinchBench dependency-constrained data-analysis task, NeoHorse-1-9B abandons an unavailable pandas dependency and switches to standard-library tools. Relative to the 4B model, that execution reduces model requests by approximately 70.8%, execution time by 76.7%, and token usage by 83.6%.

A matched comparison between public synthetic tool-agent data from Toucan and routing-harness trajectories used the same Qwen3.5-4B initialization, routing curriculum, optimizer, seed, packing, and comparable training budgets. Across five benchmarks, routing-harness data produced an average of 70.57 compared with 64.32 for the public agent data, a difference of +6.26. A scaling experiment using nested subsets of a quality-ranked trajectory pool raised the five-benchmark development average from 69.31 to 71.45 over the evaluated supervision range.

These results support the claim that routing-harness trajectories can provide transferable supervision and that additional high-quality, capability-relevant agent data can improve aggregate performance. They do not, however, isolate the contribution of every system component. The paper does not provide a complete factorial ablation separately evaluating structural validation, semantic evaluation, subscene labeling, hard versus soft routing scores, each curriculum stage, SFT versus R-OPD, or allocation feedback across multiple RSI iterations.

Important limitations include the following:

• Single-cycle evidence: the reported experiments do not demonstrate compounding gains across multiple recursive generations.
• Routing dependence: tiers are policy-dependent allocation signals rather than objective difficulty measures.
• Label uncertainty: semantic judges may be noisy, incomplete, or interrupted.
• SFT imitation risk: recorded trajectories may contain flawed tool strategies.
• Teacher dependence: R-OPD depends on teacher quality and teacher–student compatibility.
• Distribution coarsening: top-$10^5$0 distillation discards part of the teacher distribution.
• Coverage trade-offs: over-sampling deficient regions may reduce broad capability coverage.
• Evaluation scope: the benchmark suite emphasizes agentic, coding, tool-use, and instruction-following capabilities and does not cover every capability served by the harness.
• Contamination and calibration: sustained deployment requires permanently held-out evaluation, versioned routers and harnesses, and longitudinal tracking of capability regions.

The strongest conclusion supported by the reported evidence is that a routing harness can connect deployment, capability observation, data selection, and model updating in an operational post-training system. The stronger proposition—that such a system produces stable, autonomous, and indefinitely compounding RSI—remains unestablished. Multiple iterations would require returning each checkpoint to the live model pool, versioning routing and harness components, collecting fresh trajectories, preserving a held-out evaluation suite, estimating deficiencies in a calibrated manner, reallocating training data under explicit rules, and comparing each generation against both the preceding model and a no-feedback control.