---
title: ORO Subnet 15 (SN15) Overview
url: https://www.emergentmind.com/topics/oro-subnet-15-sn15
type: topic
---

# ORO Subnet 15 (SN15) Overview

ORO Subnet 15 (SN15) is a Bittensor subnet specifically architected to serve as an agentic-commerce trajectory substrate by continuously executing the ShoppingBench benchmark in a decentralized, incentive-aligned agent arena. The system addresses critical limitations in conventional agentic training data generation, mitigating mode collapse from synthetic sources and contamination from unfiltered production logs. SN15 leverages a race mechanism for agent diversity, an external LLM reasoning judge for per-trajectory oversight, and a leak-cluster-guarded, rotating problem suite to ensure robust, anti-memorized evaluation. The resulting trajectory corpus is filtered by a multi-stage structural-quality pipeline, enabling effective post-training of commerce agents such as Qwen3-4B via SFT and group-relative reinforcement learning. SN15 establishes reproducibility and provenance through on-chain signed submissions, and its full suite of mechanics, data, and training artifacts are released open source [2606.10064].

## 1. System Architecture

SN15 operates as a persistent, continuously running Bittensor subnet dedicated to generating multi-turn agentic trajectories for the ShoppingBench benchmark. Its architecture comprises two primary participant roles:

- **Miners:** Each miner deploys an "agent"—a model with autonomous tool-use capabilities plus its execution harness. 
- **Validators:** Each validator executes the submitted agent on a rotating set of ShoppingBench problems, issuing two trajectory-level scores: (a) the deterministic ShoppingBench outcome score $O(τ)\in[0,1]$ (covering price, service, SKU, attribute), and (b) a reasoning-quality coefficient $c(τ)\in[0,1]$ from an external LLM judge.

Emissions (token rewards) are distributed each block in direct proportion to these validator consensus scores. All miner submissions and validator reports are hotkey-signed and recorded on-chain, offering full cryptographic provenance for every trajectory. The underlying ShoppingBench API toolset, including eight specific tools atop a static ≈2.7M product Lucene index, is exposed to all agents.

## 2. Agent Arena Design

### 2.1 Race Mechanism

SN15's agent diversity is driven by an "incentive-aligned diversity" protocol:

- **Qualification:** New miners pass static-analysis checks barring forbidden constructs and hardcoded tables.
- **Continuous Race:** Agents compete on a shared problem pool. The daily best-performing miner is embargoed (source locked) until the next weekday noon, enforcing a discover-then-share incentive structure.
- **Submission Cooldown:** Each miner may resubmit their agent only every 12 hours, ensuring sustained diversity and reducing overfitting.

This mechanism distributes the policy search across a broad contributor base, stimulating the emergence of distinct agentic behaviors.

### 2.2 LLM Reasoning Judge

Trajectory evaluation is bifurcated:

- **Deterministic Outcome Scoring:** $O(τ)$ reflects direct ShoppingBench rule adherence.
- **LLM Judge:** An external language model $J$ inspects each trajectory’s raw "think"+tool-trace, providing the reasoning coefficient $c(τ)$ with supporting citations of evidence.
- **Supervision Signal:** Training consumes the product $c(τ)\cdot O(τ)$ as process-level supervision, externalizing verification from the miner and decoupling outcome from process quality.

### 2.3 Leak-Cluster-Guarded Problem Suite

To control memorization and paraphrase contamination:

- Problems are clustered into "leak clusters" (groupings by paraphrase or reordered attributes).
- These clusters are partitioned per rotation into training, held-out, and "never-touch" static evaluation pools, ensuring 
  $$(\text{train\_clusters}_i) \cap (\text{eval\_clusters}_i) = \varnothing$$
- Three versioned static suites (v1, v2, v3) serve for formal, anti-memorized evaluation.

## 3. Structural-Quality Filtering Pipeline

Transformation of raw SN15 trajectories into a high-grade supervised corpus proceeds in four explicit stages:

### 3.1 Reasoning-Coefficient Gate

Reject any trajectory $\tau$ with $c(τ) < \theta_c$ ($\theta_c$ default: 0.2).

### 3.2 Format-Validity Gate

Trajectories failing any of five strict invariants are hard-rejected:

1. Every tool call is paired with a corresponding response.
2. Exactly one `recommend_product` call.
3. Recommendation arguments parse as valid JSON product IDs.
4. Max token length does not exceed 14,336.
5. Trace terminates with either `recommend_product` or `terminate` (not an “assistant think”).

### 3.3 Deduplication and Structural Ranking

For each problem $p$, surviving trajectories $T_p$ are ranked lexicographically by their vector of structural signals $S(τ)$ (including depth of tool use, search-query reformulations, explicit verification steps, and step regularity). Only the top $k_p$ ($k_p∈\{1,2\}$) per problem are retained.

### 3.4 Agentic-vs-Sub-task Axis Split

- **Axis-A (Agentic):** LM emits all tool calls; execution harness only dispatches.
- **Axis-B (Sub-task):** Tool calls emitted deterministically by harness; LM only classifies/narrates.

Only Axis-A (agentic) traces are kept, ensuring genuine model-initiated behavior. The overall decision function is:
$$
D(τ) = \mathbb{I}[c(τ) \geq \theta_c] \times \mathbb{I}[\text{valid\_format}(τ)] \times \mathbb{I}[\text{rank\_within\_top}_k(τ)] \times \mathbb{I}[\text{axis}(τ)=A]
$$

## 4. Post-Training Recipe

The post-training pipeline closely matches the published ShoppingBench SFT-then-GRPO protocol, applying multiple stages with targeted substitutions:

### 4.1 Supervised Fine-Tuning (SFT)

- One epoch LoRA SFT on the filtered corpus.
- Shared parameters: Qwen3-4B, LoRA rank 16, SDPA, $14\,336$ max tokens, gradient checkpointing, LR sweep $\{1e-5, 3e-5, 1e-4\}$.

### 4.2 Rejection-Sampled re-SFT

- For each training problem, generate $k=8$ rollouts (T=0.9, top$_p$=0.95, max 8 turns).
- Score each; retain only those with bucket-aware score 1.0.
- Concatenate 50/50 with Stage 1 corpus, repeat SFT.

### 4.3 Teacher-SFT

- $≈210$ Sonnet 4.6 successful trajectories on $498$ problems, voucher buckets upsampled $\times 3$.
- Concatenate with prior corpus, one more LoRA SFT epoch.

### 4.4 KTO Preference Refinement

- KTO with unpaired desirable/undesirable labels; $\beta=0.02$, LR $5e-6$, one epoch.
- Early stopping at $\mathrm{recommend\_product}$ emission rate $\geq 0.9$.
- No net ASR lift, but reshapes per-bucket probability distribution.

### 4.5 Dr. GRPO Turn-Level RL

- Variant v19 employs blended, teacher-grounded reward: per step $r_t = (1-\lambda)0 + \lambda m_t$ for $t<T$, $r_T = O(\tau)$; $\lambda=0.5$.
- Optimization via group-relative policy gradient, returns $R_t = \sum_{t^\prime = t}^T r_{t^\prime}$.

## 5. Evaluation Metrics and Results

Key performance measures:

- **Agent Success Rate (ASR):**
  $$
  \mathrm{ASR} = \frac{\#\text{successful trajectories}}{\#\text{attempted trajectories}}
  $$
- **pass@k:** For $N$ samples, $n$ successes,
  $$
  \mathrm{pass}@k = 1 - \prod_{i=0}^{k-1}\frac{N-n}{N-i}
  $$

**Summary of held-out evaluation (75-problem, production-strict):**

| Model Variant                                | ASR (%) | pass@1 (%) | pass@8 (%) |
|----------------------------------------------|---------|------------|------------|
| Qwen3-4B base (paper)                        | 18.0    | -          | -          |
| SFT (paper, synthetic GPT-4.1)               | 43.6    | -          | -          |
| SFT+GRPO (paper bar)                         | 48.7    | -          | -          |
| SN15 SFT-only stack (Stages 1–4)             | 42.7    | 34.8       | 53.3       |

- The SFT-only stack matches SFT synthetic-data baseline within one-problem noise.
- Large pass@8 to pass@1 gap (≈18.5pp, 53.3–34.8) signifies latent capacity, partially recoverable via Dr. GRPO process RL.
- Dr. GRPO v19 increased in-training rule score $0.02 \rightarrow 0.42$, reduced hallucinations from $14 \rightarrow 0$ over $20$ steps; exact-match not fully converged to 48.7%.

A plausible implication is that further rounds of RL with refreshed rollouts and finer reward granularity could close the observed performance gap.

## 6. Data, Compute, and Release Artifacts

### Data and Compute Footprint

- **Raw firehose:** 12,000–27,000 trajectories/day.
- **Agentic axis-A trajectories:** ≈1,000–2,000/day.
- **Structural filter output:** $<8,000$ usable trajectories (≈1 day’s axis-A).
- **Training:** Each SFT stage (1–4) runs one epoch on a single A100 GPU (4–6h total). GRPO executed on vLLM 0.9.2/H200; 20 update steps within 12h wall-clock.

### Released Artifacts

All resources are open-sourced to support replication and further research, including:

- SN15 arena mechanics (race, judge, and leak-cluster-rotation implementations)
- Full structural-quality filter code/config
- Corpus splits (train/eval, leak-cluster-guarded)
- PrimeIntellect rendering and loss-masking libraries
- All training scripts and hyperparameter recipes for stages 1–5

## 7. Context and Impact

SN15 establishes an agent-arena design for generating rigorously judged, agentic multi-turn trajectories, with anti-memorized held-out evaluation and explicit structural-quality filtering. This corpus enables the post-training of compact commerce agents from Bittensor-scale decentralized trace generation, closing key gaps in prior practice with synthetic or contaminated datasets. The system’s open release of code, data, and evaluation regimes constitutes a reproducible substrate for benchmarking and advancing agentic commerce workflows [2606.10064].

Source: https://www.emergentmind.com/topics/oro-subnet-15-sn15