ORO Subnet 15 (SN15) Overview

• ORO Subnet 15 (SN15) is a decentralized Bittensor subnet designed to continuously generate multi-turn agent trajectories for commerce benchmarks.
• It employs innovative mechanisms such as a race-based agent diversity protocol, external LLM reasoning judges, and leak-cluster-guarded filtering to ensure robust outputs.
• SN15 supports post-training of commerce agents via supervised fine-tuning and group-relative reinforcement learning, with on-chain signed provenance for reproducibility.

ORO Subnet 15 (SN15) is a Bittensor subnet specifically architected to serve as an agentic-commerce trajectory substrate by continuously executing the ShoppingBench benchmark in a decentralized, incentive-aligned agent arena. The system addresses critical limitations in conventional agentic training data generation, mitigating mode collapse from synthetic sources and contamination from unfiltered production logs. SN15 leverages a race mechanism for agent diversity, an external LLM reasoning judge for per-trajectory oversight, and a leak-cluster-guarded, rotating problem suite to ensure robust, anti-memorized evaluation. The resulting trajectory corpus is filtered by a multi-stage structural-quality pipeline, enabling effective post-training of commerce agents such as Qwen3-4B via SFT and group-relative reinforcement learning. SN15 establishes reproducibility and provenance through on-chain signed submissions, and its full suite of mechanics, data, and training artifacts are released open source (Bansal et al., 8 Jun 2026).

1. System Architecture

SN15 operates as a persistent, continuously running Bittensor subnet dedicated to generating multi-turn agentic trajectories for the ShoppingBench benchmark. Its architecture comprises two primary participant roles:

• Miners: Each miner deploys an "agent"—a model with autonomous tool-use capabilities plus its execution harness.
• Validators: Each validator executes the submitted agent on a rotating set of ShoppingBench problems, issuing two trajectory-level scores: (a) the deterministic ShoppingBench outcome score $O(τ)\in[0,1]$ (covering price, service, SKU, attribute), and (b) a reasoning-quality coefficient $c(τ)\in[0,1]$ from an external LLM judge.

Emissions (token rewards) are distributed each block in direct proportion to these validator consensus scores. All miner submissions and validator reports are hotkey-signed and recorded on-chain, offering full cryptographic provenance for every trajectory. The underlying ShoppingBench API toolset, including eight specific tools atop a static ≈2.7M product Lucene index, is exposed to all agents.

2. Agent Arena Design

2.1 Race Mechanism

SN15's agent diversity is driven by an "incentive-aligned diversity" protocol:

• Qualification: New miners pass static-analysis checks barring forbidden constructs and hardcoded tables.
• Continuous Race: Agents compete on a shared problem pool. The daily best-performing miner is embargoed (source locked) until the next weekday noon, enforcing a discover-then-share incentive structure.
• Submission Cooldown: Each miner may resubmit their agent only every 12 hours, ensuring sustained diversity and reducing overfitting.

This mechanism distributes the policy search across a broad contributor base, stimulating the emergence of distinct agentic behaviors.

2.2 LLM Reasoning Judge

Trajectory evaluation is bifurcated:

• Deterministic Outcome Scoring: $O(τ)$ reflects direct ShoppingBench rule adherence.
• LLM Judge: An external LLM $J$ inspects each trajectory’s raw "think"+tool-trace, providing the reasoning coefficient $c(τ)$ with supporting citations of evidence.
• Supervision Signal: Training consumes the product $c(τ)\cdot O(τ)$ as process-level supervision, externalizing verification from the miner and decoupling outcome from process quality.

2.3 Leak-Cluster-Guarded Problem Suite

To control memorization and paraphrase contamination:

• Problems are clustered into "leak clusters" (groupings by paraphrase or reordered attributes).
• These clusters are partitioned per rotation into training, held-out, and "never-touch" static evaluation pools, ensuring

$$(\text{train\_clusters}_i) \cap (\text{eval\_clusters}_i) = \varnothing$$

• Three versioned static suites (v1, v2, v3) serve for formal, anti-memorized evaluation.

3. Structural-Quality Filtering Pipeline

Transformation of raw SN15 trajectories into a high-grade supervised corpus proceeds in four explicit stages:

3.1 Reasoning-Coefficient Gate

Reject any trajectory $\tau$ with $c(τ) < \theta_c$ ($\theta_c$ default: 0.2).

3.2 Format-Validity Gate

Trajectories failing any of five strict invariants are hard-rejected:

1. Every tool call is paired with a corresponding response.
2. Exactly one recommend_product call.
3. Recommendation arguments parse as valid JSON product IDs.
4. Max token length does not exceed 14,336.
5. Trace terminates with either recommend_product or terminate (not an “assistant think”).

3.3 Deduplication and Structural Ranking

For each problem $c(τ)\in[0,1]$0, surviving trajectories $c(τ)\in[0,1]$1 are ranked lexicographically by their vector of structural signals $c(τ)\in[0,1]$2 (including depth of tool use, search-query reformulations, explicit verification steps, and step regularity). Only the top $c(τ)\in[0,1]$3 ($c(τ)\in[0,1]$4) per problem are retained.

3.4 Agentic-vs-Sub-task Axis Split

• Axis-A (Agentic): LM emits all tool calls; execution harness only dispatches.
• Axis-B (Sub-task): Tool calls emitted deterministically by harness; LM only classifies/narrates.

Only Axis-A (agentic) traces are kept, ensuring genuine model-initiated behavior. The overall decision function is:

$c(τ)\in[0,1]$5

4. Post-Training Recipe

The post-training pipeline closely matches the published ShoppingBench SFT-then-GRPO protocol, applying multiple stages with targeted substitutions:

4.1 Supervised Fine-Tuning (SFT)

• One epoch LoRA SFT on the filtered corpus.
• Shared parameters: Qwen3-4B, LoRA rank 16, SDPA, $c(τ)\in[0,1]$6 max tokens, gradient checkpointing, LR sweep $c(τ)\in[0,1]$7.

4.2 Rejection-Sampled re-SFT

• For each training problem, generate $c(τ)\in[0,1]$8 rollouts (T=0.9, top$c(τ)\in[0,1]$9=0.95, max 8 turns).
• Score each; retain only those with bucket-aware score 1.0.
• Concatenate 50/50 with Stage 1 corpus, repeat SFT.

4.3 Teacher-SFT

• $O(τ)$0 Sonnet 4.6 successful trajectories on $O(τ)$1 problems, voucher buckets upsampled $O(τ)$2.
• Concatenate with prior corpus, one more LoRA SFT epoch.

4.4 KTO Preference Refinement

• KTO with unpaired desirable/undesirable labels; $O(τ)$3, LR $O(τ)$4, one epoch.
• Early stopping at $O(τ)$5 emission rate $O(τ)$6.
• No net ASR lift, but reshapes per-bucket probability distribution.

4.5 Dr. GRPO Turn-Level RL

• Variant v19 employs blended, teacher-grounded reward: per step $O(τ)$7 for $O(τ)$8, $O(τ)$9; $J$0.
• Optimization via group-relative policy gradient, returns $J$1.

5. Evaluation Metrics and Results

Key performance measures:

• Agent Success Rate (ASR):

$J$2

• pass@k: For $J$3 samples, $J$4 successes,

$J$5

Summary of held-out evaluation (75-problem, production-strict):

Model Variant	ASR (%)	pass@1 (%)	pass@8 (%)
Qwen3-4B base (paper)	18.0	-	-
SFT (paper, synthetic GPT-4.1)	43.6	-	-
SFT+GRPO (paper bar)	48.7	-	-
SN15 SFT-only stack (Stages 1–4)	42.7	34.8	53.3

• The SFT-only stack matches SFT synthetic-data baseline within one-problem noise.
• Large pass@8 to pass@1 gap (≈18.5pp, 53.3–34.8) signifies latent capacity, partially recoverable via Dr. GRPO process RL.
• Dr. GRPO v19 increased in-training rule score $J$6, reduced hallucinations from $J$7 over $J$8 steps; exact-match not fully converged to 48.7%.

A plausible implication is that further rounds of RL with refreshed rollouts and finer reward granularity could close the observed performance gap.

6. Data, Compute, and Release Artifacts

Data and Compute Footprint

• Raw firehose: 12,000–27,000 trajectories/day.
• Agentic axis-A trajectories: ≈1,000–2,000/day.
• Structural filter output: $J$9 usable trajectories (≈1 day’s axis-A).
• Training: Each SFT stage (1–4) runs one epoch on a single A100 GPU (4–6h total). GRPO executed on vLLM 0.9.2/H200; 20 update steps within 12h wall-clock.

Released Artifacts

All resources are open-sourced to support replication and further research, including:

• SN15 arena mechanics (race, judge, and leak-cluster-rotation implementations)
• Full structural-quality filter code/config
• Corpus splits (train/eval, leak-cluster-guarded)
• PrimeIntellect rendering and loss-masking libraries
• All training scripts and hyperparameter recipes for stages 1–5

7. Context and Impact

SN15 establishes an agent-arena design for generating rigorously judged, agentic multi-turn trajectories, with anti-memorized held-out evaluation and explicit structural-quality filtering. This corpus enables the post-training of compact commerce agents from Bittensor-scale decentralized trace generation, closing key gaps in prior practice with synthetic or contaminated datasets. The system’s open release of code, data, and evaluation regimes constitutes a reproducible substrate for benchmarking and advancing agentic commerce workflows (Bansal et al., 8 Jun 2026).