Bittensor Agent Arenas as a Trajectory Primitive: Distilling a Shopping Agent from ShoppingBench Subnet Traces
Abstract: Small-model agentic post-training is bottlenecked less by the algorithm than by the trajectory substrate it consumes. Leading recipes (RLVR, group-relative RL, rejection-sampled re-SFT) all need multi-turn traces carrying per-trajectory supervision, and the two existing sources fall short: frontier-synthesised data inherits the synthesizer's biases and collapses the long tail, while unfiltered production logs are unjudged and contaminated by shortcut behaviour. We argue that an incentive-aligned agent arena can be engineered to manufacture such trajectories, and demonstrate this on ORO Subnet 15 (SN15), a Bittensor deployment of the ShoppingBench agentic-commerce benchmark. SN15's race mechanism, LLM reasoning judge, and rotating leak-cluster-guarded problem suite yield a corpus with three properties: incentive-aligned diversity, per-trajectory judging, and anti-memorised held-out evaluation. We introduce a structural-quality filter that converts the raw firehose into a trainable corpus by keeping agentic trajectories (the model itself emits the tool calls) and rejecting sub-task trajectories (the model only classifies or narrates over a deterministic search loop), then post-train Qwen3-4B with a recipe matched to the published ShoppingBench SFT-then-GRPO pipeline. On a leak-cluster-guarded held-out partition scored production-strict, the model lifts from the published Qwen3-4B base of 18.0% ASR to 42.7%, within single-problem noise of the synthetic-data SFT-only baseline (43.6%), while training on a fraction of a single day of subnet output. The supervised stack leaves a large pass@8 to pass@1 gap (53.3% vs 34.8%); a per-step teacher-grounded Dr. GRPO reward converts that headroom into process improvement, and we identify the sub-task firehose as the primary lever for closing the gap to the 48.7% SFT+GRPO bar. We release the filter, the corpus splits, and the arena mechanics.
Summary
- The paper introduces an innovative agent arena pipeline using incentive-aligned SN15 traces to distill shopping agents.
- It employs strict structural filtering and a multi-stage post-training process, achieving a 42.7% ASR on held-out benchmarks.
- The work highlights a data-centric shift toward agentic behavior learning, addressing synthetic data bias and corpus scale limitations.
Bittensor Agent Arenas as a Trajectory Primitive: Distilling a Shopping Agent from ShoppingBench Subnet Traces
Overview and Motivation
The development of agentic small LMs for tool use is dominated not by the incremental progress of RL algorithmic choices but by the composition and quality of supervised trajectory data. Existing pipeline bottlenecks include (i) the reliance on synthetic trajectory corpora with known coverage collapse and bias, and (ii) the curation difficulties of unfiltered production logs rife with shortcut exploitation and lack of process-level annotations. This work ("Bittensor Agent Arenas as a Trajectory Primitive: Distilling a Shopping Agent from ShoppingBench Subnet Traces" (2606.10064)) introduces a third substrate: incentive-aligned, per-trajectory-judged, and memorization-robust agent traces generated via a Bittensor subnet arena deployment of ShoppingBench (termed SN15). By formalizing the arena mechanisms and introducing a pipeline for strict structural filtering, the paper post-trains Qwen3-4B under a controlled SFT-to-GRPO regime, offering direct comparisons against both synthetic SFT and production logs.
Agent Arena Design and Data Generation Pipeline
SN15 is architected as an economic competition among independent miners, each submitting candidate agent policies to a structured ShoppingBench problem suite rotated under tight anti-leakage controls. The process incorporates:
- Qualifying and racing phases: with submission cooldowns, embargoed code sharing, and regular leaderboard resets to foster genuine policy diversity.
- Bi-axial trajectory judging: deterministic outcome grading along price, service, SKU, and attribute axes; plus an LLM-based reasoning-quality coefficient providing granular process-level assessment.
- Provenance and traceability: every submission, score, and judge annotation is cryptographically signed by distinct miner and validator identities, enabling precise lineage.
A standout mechanism is the "leak-cluster guard" for constructing held-out test sets, which eliminates paraphrase or attribute-drift leakage from training to evaluation problems.
(Figure 1)
Figure 1: End-to-end pipeline converting SN15 agent arena traces to structurally judged, filtered SFT corpora for agent post-training.
Structural-Quality Filtering and Corpus Properties
To bridge the gap between raw, highly heterogeneous SN15 trajectories and a trainable SFT substrate, a multi-stage structural filter is deployed:
- A strict reasoning-coefficient gate drops traces failing the LLM judge's process-quality threshold.
- A format-validity gate ensures schema alignment, tool-call/response matching, single recommend-product terminal event, and length quotas.
- A structural-quality ranking selects for maximal agentic behavior: high think-tool-call depth, query reformulation, explicit verification steps, and regular decision flow.
- Only axis-A (agentic) traces are retained—where tool-use logic is encoded in the LM outputs, not the surrounding Python harness, ensuring that models learn end-to-end online policies rather than merely intent classification or narration.
The leak-cluster-guarded held-out partition is constructed to guarantee that evaluation genuinely measures generalization rather than surface-memorization gains.
Post-Training Pipeline and Experimental Results
The agent distillation is conducted through five key stages:
- LoRA SFT on the filtered corpus,
- Rejection-sampled re-SFT with rollouts per training-set problem, selecting only those achieving perfect outcome score,
- Continued SFT on a Sonnet 4.6 teacher corpus (as a step-wise OPD surrogate),
- KTO preference optimization for further policy refinement,
- Dr.~GRPO turn-level RL with a blended outcome-plus-tool-match reward, encouraging correct tool selection at each step.
The distilled agent (Qwen3-4B base → ORO SFT stack) achieves an average success rate (ASR) of 42.7% on the 75-problem leak-cluster-guarded held-out set (see Figure 2)—a 24.7-point gain over the Qwen3-4B base (18.0%) and on par with the ShoppingBench synthetic SFT-only baseline (43.6%), despite using a far smaller and more diverse corpus.
Figure 2: Overall ASR on the 75-problem leak-cluster-guarded held-out set, showing ORO SFT stack bridging the gap between Qwen3-4B base and synthetic SFT baselines.
Notably, the supervised SFT/RL stack leaves substantial latent capability (pass@8: 53.3%, pass@1: 34.8%), indicating considerable extractable headroom for future RL extraction optimization.
Preference Optimization and Policy Redistribution
KTO preference refinement, while not increasing overall ASR, redistributes performance within constraint buckets (e.g., voucher ASR increases, product ASR decreases), reflecting that policy smoothing via KTO operates on local probability mass but is insufficient for global performance expansion without corresponding gains in agentic policy diversity.
Figure 3: KTO preference refinement redistributes probability mass and success rates within ShoppingBench constraint buckets but does not increase overall ASR.
Analysis of Limitations
A dominant constraint is the nature of the data retained for training. The structurally-filtered corpus utilizes only axis-A "agentic" trajectories, which form a small minority of the total production firehose—most competitive SN15 agents on the leaderboard use axis-B ("sub-task") patterns, where decision logic is implemented in the Python harness and the LM acts as a classifier or narrator. This choice affirms agentic behavior learning but severely limits corpus scale. As a result, the presented results should be read as a strict lower bound on what is achievable via SN15 agent arena traces.
Other limitations include format/prompt sensitivity in benchmarking, the correlational (not gold) nature of the LLM judge signals, and incomplete RL convergence under budget and rollout drift constraints. Notably, Dr.~GRPO variants, in their current form, have not closed the ASR gap to the published SFT+GRPO bar (48.7%).
Broader Implications and Future Work
The paper demonstrates that Bittensor-style agent arenas can manufacture, as a side effect of decentralized, contest-driven system design, a new class of trajectory data blending real diversity, structured supervision, and anti-memorization held-out benchmarking. This addresses critical pathologies in both synthetic and production-log training regimes and positions agent arena architecture as a general primitive for agent post-training data.
Practically, the SN15 pipeline offers a reproducible, open artifact (filters, splits, mechanics) that can be extended for other agentic benchmarks and further refined with new feedback compositions and incentive re-weighting to stimulate more (and higher-quality) agentic traces. Theoretically, it signals a data-centric shift in agent modeling, where the emphasis is on dynamic, contest-mediated, LLM-judged process data rather than batch static/recursive synthetic traces, potentially avoiding modal collapse [shumailov2024collapse].
A key open development is harnessing the axis-B sub-task trace volume, either by upweighting the arena's incentive structure toward agentic (axis-A) submissions or by synthetic transformation of axis-B traces into agentic ones using a strong external LM, similar in spirit to self-evolution or role reversal strategies [liu2024toolace]. The paper identifies clear directions for improving RL credit assignment, reducing rollout drift, and integrating denser, per-decision reward signals.
Conclusion
This work substantiates Bittensor agent arenas as a viable primitive for agentic LM post-training, introducing a robust, incentive-compatible, and structurally filterable trajectory substrate. The resulting distilled agent achieves SFT-only parity with synthetic baselines using a fraction of the data, validates arena-sourced trajectories for practical small-model distillation, and lays the groundwork for scalable and generalizable data-driven agent development. The axis-A/axis-B distinction and pass@ residuals now set the next agenda for corpus expansion and RL optimization.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Explain it Like I'm 14
What is this paper about?
This paper is about teaching a small AI model to be a good online shopping assistant. Instead of relying on fake conversations created by a big AI, the authors build a “game-like arena” where many different teams’ shopping agents compete, get scored by a judge, and earn rewards. The traces of how these agents think and act become high‑quality training data. Using this data, the authors train a small open model to find products that match a shopper’s request much better than before.
What questions were the researchers trying to answer?
They focused on three simple questions:
- Can we design a competitive “agent arena” that naturally produces lots of useful, graded practice attempts from many different agents?
- Can we filter those attempts so we keep the truly “agent-like” ones (where the AI plans and calls tools itself) for training?
- If we train a small model on those attempts, will it become a better shopping assistant, close to models trained on synthetic data?
How did they do it?
1) Build a fair, motivating “arena” to collect good attempts
Think of a school science fair meets a sports league:
- Many teams (“miners”) submit shopping agents to a public arena on a network called Bittensor. The specific arena is called SN15.
- Agents compete daily on shopping tasks (like “find a black waist bag under $20 from a 4.5+ star shop”).
- Each agent’s try is scored in two ways: 1) Did it meet the rules (price, shipping/service, product attributes, etc.)? 2) How good was its reasoning? A separate AI “judge” reads the whole step‑by‑step process and gives a quality score.
- The arena rotates in fresh, hidden test problems and uses “leak clusters” to stop memorization (if a problem appears with different wording, it still counts as the same test). That keeps evaluation honest.
This setup encourages diverse solutions because teams get rewarded for finding new strategies, not just copying the leader.
2) Turn the “firehose” of attempts into clean training data
The arena produces a lot of multi‑step traces of how agents think and call tools (like searching a catalog, checking a voucher, then recommending a product). The authors:
- Normalize all traces into a common format.
- Keep only well‑formed attempts with good reasoning scores from the judge.
- Remove near‑duplicates and rank by structural quality (e.g., did the agent verify its steps?).
- Most important: keep only “agentic” traces, where the LLM itself decides and calls the tools. They discard “sub-task” traces where a Python script does all the planning and the AI just labels things or narrates. Why? Because to teach an agent to plan, you need examples where the AI actually plans.
3) Train a small model with a simple, two-stage approach
They start with a 4‑billion‑parameter model (Qwen3‑4B) and:
- Stage 1: Supervised fine-tuning (SFT) — teach by showing good examples from the filtered traces.
- Stage 2: Practice with feedback (reinforcement learning) — let the model try multi‑step solutions and nudge it based on how well each step and the final outcome match a strong “teacher” solution.
They also tried a preference‑tuning step (KTO) to gently push the model toward better behaviors, and experimented with different RL reward signals to make step‑by‑step learning stable and useful.
4) Test on a strict, truly held‑out set
They evaluate on a special set of problems that were never used for training, guarded against paraphrase leaks. The main score is ASR (Average Success Rate): the percent of problems where the final recommendation satisfies all rules.
What did they find, and why is it important?
- A big jump in accuracy for a small model: On the held‑out test, the small model went from 18.0% ASR (the untrained base) to 42.7% after training on the arena data. That’s almost as good as a 43.6% baseline trained on synthetic data made by a large AI, but here it’s achieved using real, judged, competitive traces.
- There’s untapped potential: When the model is allowed multiple tries per problem (pass@8), it succeeds 53.3% of the time, compared to 34.8% on the first try (pass@1). This gap shows the model already “knows how” more often than it actually “lands it” on the first attempt—so better step‑by‑step training can still help.
- Better process, not just results: Their reinforcement learning with dense, step‑level rewards reduced silly mistakes like “hallucinating” product IDs and improved intermediate steps, even if the final exact-match score didn’t climb yet. That’s a sign the training is moving in the right direction.
- A cleaner data recipe: The arena produces three special data qualities that are hard to get elsewhere:
- Incentive‑aligned diversity: Many independent teams are rewarded for new strategies, so data stays varied.
- Per‑attempt judging: Each full multi‑step attempt gets a reasoning quality score, not just a right/wrong label.
- Strong held‑out testing: Rotating, guarded tasks make scores more trustworthy.
These points matter because for “agent” AIs (which plan and use tools), good training isn’t just lots of data—it’s the right kind of data with clear feedback on the whole process.
What does this mean going forward?
- A new way to grow strong small agents: Carefully designed “agent arenas” can manufacture the kind of rich, graded practice attempts small models need—without relying only on big models to fake conversations.
- A virtuous cycle: The distilled model can re‑enter the arena as a competitor, raising the bar. That creates a loop that keeps producing better data and better agents over time.
- Clear next steps to improve: The authors see two big levers: 1) Make the step‑by‑step learning signals even denser and train longer, so the model turns its pass@8 potential into first‑try success. 2) Use more of the arena’s data. Today, most arena traces are “sub-task” style (scripts plan, AI labels). The team plans to convert those into true “agent-like” examples or tweak the arena scoring so more agent-like traces are produced at the source.
In short, this work shows that if you build the right “game” with fair rules, diverse players, and a good judge, you can collect the exact kind of examples that teach small AIs to think and act better—starting with shopping, but potentially useful for many other tool‑using tasks.
Knowledge Gaps
Below is a single, actionable list of the paper’s unresolved knowledge gaps, limitations, and open questions that future work could address.
- Data/corpus: Quantify the exact axis-A vs axis-B composition of the firehose over time and across miners; report proportions by bucket (product/shop/voucher), constraint types, and tool usage.
- Data/corpus: Validate the accuracy of the axis-A/axis-B classifier used in the structural filter (false positives/negatives), and release detection heuristics to enable independent replication.
- Data/corpus: Ablate the structural-quality filter signals (depth, query reformulation, verification step, shape regularity) to measure their causal impact on downstream ASR; tune thresholds/weights.
- Data/corpus: Evaluate whether selecting only 1–2 trajectories per problem suppresses diversity; compare against keeping more trajectories or sampling for maximal diversity.
- Data/corpus: Incorporate failed-but-high-quality (judge-approved) trajectories as explicit negatives for preference learning and study their contribution to robustness.
- Data/corpus: Assess long-tail coverage of subnet-generated data versus synthetic corpora (rare attribute combinations, tight price bands, compound constraints) and quantify gains/losses.
- Data/corpus: Develop and test concrete methods for transforming axis-B traces into high-fidelity agentic (axis-A) trajectories (e.g., LLM rewriting grounded in the real tool timeline), and compare strategies.
- Data/corpus: Measure the effect of including rewritten axis-B data on ASR and pass@k; evaluate for overfitting or policy imitation effects.
- Judge/verifier: Calibrate the LLM reasoning judge against human labels; report inter-rater agreement, bias analyses, and calibration curves for the judge coefficient.
- Judge/verifier: Quantify the training benefit of judge-coefficient gating via ablations (with/without gate, varying thresholds); measure impact on both ASR and process metrics.
- Judge/verifier: Test adversarial robustness of the judge (evidence-citation gaming, spurious rationales); design and evaluate anti-gaming defenses.
- Judge/verifier: Explore using the judge’s step- or span-level signals directly for token-level RL rewards; compare to teacher-grounded tool-match rewards.
- Arena/economics: Empirically substantiate “incentive-aligned diversity” with concrete metrics (e.g., policy/trajectory novelty, tool-path divergence) and show sensitivity to embargo length/cooldowns/emission weights.
- Arena/economics: Analyze validator reliability (inter-validator variance), consensus stability, and potential collusion; introduce auditing and anomaly detection for scores and judge outputs.
- Arena/economics: Extend anti-cheating beyond static analysis (e.g., detect obfuscated answer tables, latent product-ID lookups) and report residual attack surface.
- Held-out/leak: Rigorously validate leak-cluster construction (precision/recall on paraphrase and semantic duplicates, product-ID/retailer leakage); publish clustering criteria and thresholds.
- Held-out/leak: Quantify how sensitive results are to harness variants (4-tool vs 7-tool), system-prompt nudges, and renderer changes; report cross-harness transfer performance with confidence intervals.
- Held-out/leak: Provide statistical significance (CIs, hypothesis tests) for ASR differences on the 75/90-problem sets to avoid over-interpretation of small deltas.
- Training/RL: Run longer Dr. GRPO training with the teacher-grounded per-step reward and publish learning curves (ASR, pass@k, rule-axis scores) to test convergence to higher pass@1.
- Training/RL: Compare reward designs—outcome-only, tool-match, judge-derived, and rule-axis-specific shaping—in controlled ablations to isolate which signals drive exact-match gains.
- Training/RL: Address importance-sampling drift by evaluating periodic on-policy refresh schedules, behavior cloning warm starts, or fully on-policy PPO-style baselines.
- Training/RL: Explore curricula that progressively tighten success criteria (e.g., neighborhood navigation → exact product match) and measure conversion of pass@8 headroom into pass@1.
- Training/RL: Assess whether teacher-grounded rewards cause over-imitation; test multiple teachers/ensembles and penalize overreliance to preserve diversity.
- Training/RL: Investigate why KTO yielded no net lift (β sensitivity, class imbalance, sampling temperature); determine conditions where KTO is beneficial and how to avoid over-optimization.
- Training/RL: Evaluate scaling to larger bases (7B, 14B) with the same substrate to test the claim that substrate and post-training recipe dominate parameter count.
- Process metrics: Standardize and report process metrics (latency, tool-call count, recovery from dead ends, hallucination rate) alongside ASR to assess practical agent quality.
- Generalization: Test transfer from index-based ShoppingBench to dynamic environments (live websites, browser agents) and to other agentic benchmarks (OSWorld, SWE-Bench); quantify degradation and adaptation needs.
- Robustness: Measure stability under catalog updates (new SKUs/prices) and define continual learning or refreshing strategies; monitor catastrophic forgetting.
- Selection bias: Analyze whether outcome filtering (e.g., keeping only score-1.0 rollouts in re-SFT) biases the policy toward “easy” problems and reduces exploration.
- Prompting: Evaluate robustness to changes in system/user prompts and few-shot exemplars; quantify sensitivity and mitigate via prompt-augmentation during training.
- Tooling: Ablate the added helper tools vs baseline 4-tool setup to understand their contribution to learning and evaluation comparability.
- Reproducibility: Release full arena code, evaluator prompts, and RL training scripts (not just filters/splits) with seeds and compute budgets to enable third-party replication.
- Cost/scale: Quantify end-to-end costs (judge compute, filtering, RL iterations) and propose cost-effective pipelines; the cited ~1/1000 conversion rate in Trac’s full oracle suggests inefficiency to address.
- Governance/IP: Clarify data licensing/usage rights for miner-generated trajectories and validators’ outputs to ensure legally safe redistribution and reuse.
- Feedback loop: Study the effect of distilled agents re-entering the arena on population diversity and convergence (risk of homogenization); design mechanisms to preserve exploration.
Practical Applications
Immediate Applications
Below are actionable uses that can be deployed now, leveraging the paper’s data-generation arena, filtering pipeline, and small-model post‑training recipe.
- Distilled small-model shopping assistants for marketplaces (e-commerce)
- What: Deploy a Qwen3-4B–class agent fine-tuned via SN15-style judged trajectories to handle constrained product queries (price, shipping/service, SKU, attributes) with tool calls.
- Tools/products/workflows: ShoppingBench-compatible tool surface; structural-quality filter to curate agentic logs; SFT + outcome-filtered re-SFT; optional turn‑level RL with tool-match reward.
- Assumptions/dependencies: Access to a static or frequently refreshed product index and deterministic tool APIs; capacity to log tool-call traces; basic evaluation harness.
- Catalog QA and policy compliance sweeps (e-commerce, retail ops)
- What: Use the agent to systematically verify SKU attributes, price bands, shipping terms, and voucher arithmetic across a catalog; flag violations.
- Tools/products/workflows: Attribute-match, shop-info, voucher arithmetic tools; batch evaluation jobs; judge-backed triaging of findings.
- Assumptions/dependencies: Accurate, queryable catalog snapshot; clear compliance rules.
- Voucher/discount and total-cost optimization (e-commerce, daily life)
- What: Automate optimal voucher combinations and “effective price” calculations at checkout recommendations.
- Tools/products/workflows: Voucher arithmetic tool; recommendation policies that include shipping/service constraints.
- Assumptions/dependencies: Up-to-date voucher metadata; index parity with live pricing.
- Provenance-aware, auditable agent logging (regulated industries, platform ops)
- What: Adopt SS58-like signed identifiers and append-only evaluation logs for traceable, auditable agent outputs.
- Tools/products/workflows: Cryptographically signed submissions; append-only evaluation store.
- Assumptions/dependencies: Identity and key management; secure storage; internal policy alignment.
- Per-trajectory LLM “Reasoning Judge” to score process quality (software, support, finance research assistants)
- What: Deploy an LLM judge that reads multi-turn traces and assigns process-quality coefficients with evidence citations; gate SFT data by score.
- Tools/products/workflows: Judge prompts emphasizing evidence citation; outcome-independent process scoring; integration with data pipelines.
- Assumptions/dependencies: LLM inference budget; calibration checks to prevent judge drift.
- Structural-quality filter for production logs → trainable agentic datasets (software, support, RAG agents)
- What: Convert internal tool-use logs into SFT-ready corpora by enforcing format invariants, deduplication, structural-quality ranking, and axis‑A (agentic) selection.
- Tools/products/workflows: The released filter (format-validity gate; reasoning-coefficient gate; dedup/structural ranking; axis-A vs axis-B split).
- Assumptions/dependencies: Availability of tool-call traces; minimal schema normalization work.
- Internal “mini-arenas” to diversify strategies (industry labs, product teams)
- What: Run recurring, incentivized head-to-head evaluations among teams/models on rotating task suites to produce diverse, judged trajectories.
- Tools/products/workflows: Race/qualifying loop; held‑out rotation; consensus scoring; weekly “promote-and-embargo” cadence (without tokens if desired).
- Assumptions/dependencies: Neutral evaluation ops; transparent rules and reward allocation.
- Leak-cluster–guarded benchmark construction (academia, industry eval teams)
- What: Build rotating held‑out partitions with paraphrase/attribute-order leak clusters to preserve real generalization surfaces.
- Tools/products/workflows: Problem clustering via semantic similarity + structured attribute normalization; frozen seeds; public manifesting of splits.
- Assumptions/dependencies: Sufficient problem pool to stratify; clustering quality.
- Turn-level RL with teacher-grounded tool-match rewards (software agents, coding assistants, web-browsing agents)
- What: Use Dr. GRPO–style training with blended outcome + tool-match rewards to convert pass@k headroom into pass@1 gains.
- Tools/products/workflows: Teacher rollout generation (frontier or strong internal model); per-step reward shaping; stability tuning (B, LR, β).
- Assumptions/dependencies: Access to teacher trajectories; RL infra; careful KL anchoring and importance-sampling management.
- Buyer- and seller-facing shopping copilots (daily life, SMB retail)
- What: Lightweight assistants that find products meeting constraints and explain trade-offs; seller tools that suggest attribute completions and compliant bundles.
- Tools/products/workflows: Chat surfaces; indexed-catalog connectors; explanation templating grounded in observations.
- Assumptions/dependencies: Catalog/API access; UX integration.
- Curriculum and ablation research using released corpus and mechanics (academia)
- What: Study substrate-vs-recipe effects, compare SFT vs RL variants, and test verifier designs on the provided splits and arena descriptions.
- Tools/products/workflows: Reuse SN15 mechanics; reproduce held-out construction; open SFT/RL configs.
- Assumptions/dependencies: Compute; access to the released code/splits.
Long-Term Applications
These use cases require additional research, scaling, policy/ethics frameworks, or infrastructure maturation.
- Domain-specific agent arenas as trajectory factories (healthcare triage, legal research, education tutoring, robotics task planning, energy/ops)
- What: Replicate SN15’s economic and evaluation mechanics to generate judged, diverse, multi-turn traces per domain.
- Tools/products/workflows: Task-specific tool surfaces (e.g., clinical guidelines queries, statute retrieval, robotics APIs); LLM judges with domain rubrics; rotating held‑outs.
- Assumptions/dependencies: Strong safety/review protocols (esp. healthcare/legal/robotics); high-quality tool instrumentation; governance and auditability.
- Axis‑B → Axis‑A “Agentic Rewriter” pipelines (cross-domain)
- What: Automatically transform deterministic, sub-task logs (axis‑B) into agentic traces (axis‑A) with natural-language reasoning interleaved with real tool calls.
- Tools/products/workflows: LLM rewriters grounded in actual tool timelines and outcomes; evidence-citation; quality gates.
- Assumptions/dependencies: Access to underlying code + logs; careful hallucination controls; cost-effective rewriter inference.
- Reweighting evaluation to reward agentic richness (arena operators, internal comps)
- What: Introduce score components that value agent-emitted tool calls and explicit verification steps, shifting the population toward learnable agent policies.
- Tools/products/workflows: Scoring update and ablation studies; anti-gaming static analysis; post-change monitoring.
- Assumptions/dependencies: Community buy-in; stability of rankings; guardrails against superficial “reasoning padding.”
- Decentralized data markets for judged agent trajectories (AI infrastructure, policy)
- What: Tokenized or credit-based marketplaces where miners/validators contribute and audit high-quality, per-trajectory-judged data.
- Tools/products/workflows: On-chain provenance, emissions, dispute resolution; anti-Sybil mechanisms; privacy-preserving logging.
- Assumptions/dependencies: Robust governance; regulatory clarity on data/IP; fraud and bias mitigation.
- Standardized, evidence-citing reasoning-judge frameworks (industry consortia, regulators)
- What: Open, auditable judge specifications with calibration protocols and appeals to reduce bias and ensure fairness across domains.
- Tools/products/workflows: Public judge prompt suites; inter-model calibration; human-in-the-loop spot checks.
- Assumptions/dependencies: Community standards; cost sharing; periodic audits.
- Continuous self-play/curriculum arenas (software, ops automation, education)
- What: Automated difficulty scaling and population-based training where agents iteratively generate and solve increasingly complex tasks.
- Tools/products/workflows: Task generators; solver-evaluator splits; curriculum gates tied to process metrics.
- Assumptions/dependencies: Reliable verifiers; compute budgets; safeguards against reward hacking.
- Browser/live-web adaptation of index-based agents (e-commerce, general web agents)
- What: Extend from deterministic index evaluation to live browsing while retaining reproducibility through snapshotting and DOM invariants.
- Tools/products/workflows: Headless browsers; page-state snapshotting; DOM-diff–based rewards; anti-brittleness heuristics.
- Assumptions/dependencies: Legal/ToS constraints; robustness to layout drift; heavier infra.
- Compliance and algorithmic accountability via trajectory auditing (policy, finance/healthcare)
- What: Regulators adopt per-trajectory process judgments and provenance logs as audit artifacts for consequential AI systems.
- Tools/products/workflows: Evidence-linked reasoning logs; regulator-facing dashboards; risk scoring along rule axes (e.g., price/service/SKU/attribute analogs).
- Assumptions/dependencies: Legal frameworks; privacy-preserving storage; standardization across vendors.
- Small-model distillation as a service for vertical agents (SMBs, enterprises)
- What: SaaS that ingests a customer’s tool-use logs, applies structural filters and judged refinement, and outputs a domain agent tuned to their APIs.
- Tools/products/workflows: ETL for tool-call logs; filter + rewriter; SFT + RL pipeline; ongoing arena-based refresh.
- Assumptions/dependencies: Data-sharing agreements; MLOps maturity; cost control for judges and RL.
- Secure cross-organization benchmark sharing with leak-cluster guards (academia, industry consortia)
- What: Institutions exchange benchmarks and judged traces while maintaining true held‑outs using leak-cluster tooling.
- Tools/products/workflows: Open-source leak-cluster libraries; reproducible split manifests; shared governance on rotation cadence.
- Assumptions/dependencies: Trust and alignment on definitions; anonymization where needed.
- Multi-judge ensembles (LLM + human) for safety-critical process grading (healthcare, legal, robotics)
- What: Combine automated judges with expert human review to validate process quality before training and deployment.
- Tools/products/workflows: Disagreement detectors; escalation workflows; adjudication metrics.
- Assumptions/dependencies: Access to experts; budget for human review; careful sampling strategies.
- Multi-agent open competitions to source novel strategies (scientific literature triage, legal doc analysis, fraud detection)
- What: Arenas tailored to discovery tasks where diversity of strategies is the primary asset; reward novel, verifiable reasoning paths.
- Tools/products/workflows: Novelty metrics; per-trajectory evidence requirements; rotating ground-truth seeds.
- Assumptions/dependencies: High-quality ground truth; anti-plagiarism/duplicate detection; incentive alignment.
Notes on Feasibility and Cross-Cutting Dependencies
- Deterministic, queryable tool surfaces dramatically simplify reproducibility; moving to dynamic environments increases engineering and legal overhead.
- LLM judges provide correlational, not gold, supervision; for high-stakes domains, add human review and stronger verifiers.
- The quality and diversity of trajectories hinge on incentive design (cooldowns, promotions, embargoes); poor designs collapse diversity.
- Axis‑B dominance in production logs is common; unlocking value requires either metric reweighting or robust axis‑B→axis‑A rewriting.
- Continuous evaluation with rotating, leak-guarded held‑outs is essential to prevent test overfitting and paraphrase leakage across all sectors.
Glossary
- Agent arena: A competitive environment engineered to elicit and evaluate multi-turn agent trajectories under defined incentives. "Trajectories produced by an incentive-aligned agent arena (left) flow through a data pipeline..."
- Agentic: Refers to models or training that focus on autonomous, multi-step decision-making with tool use. "Small-model agentic post-training is bottlenecked less by the choice of algorithm than by the trajectory substrate it consumes."
- ASR (Average Success Rate): The fraction of problems where the final recommendation satisfies all rule constraints. "the model lifts from the published Qwen3-4B base of ASR to "
- Axis-A trajectories: Traces where the LLM emits tool calls and thus encodes the agent’s policy. "axis-A trajectories are those in which the LLM itself emits the tool calls"
- Axis-B trajectories: Traces where external code issues tool calls and the model only classifies, scores, or narrates. "axis-B trajectories are those in which the surrounding Python emits tool calls and the LLM is used only as a classifier, scorer, or narrator"
- Bittensor: A permissionless network of independent subnets running continuous evaluation competitions with on-chain incentives. "Bittensor \citep{bittensor2024whitepaper} is a permissionless network of independent subnets, each running its own continuous evaluation competition."
- Dr. GRPO: A variant of group-relative policy optimization applying per-token, turn-level reinforcement learning updates. "The final stage applies a Dr.~GRPO \citep{shao2024deepseekmath, liu2025drgrpo} turn-level reinforcement-learning pass"
- Emissions: On-chain token rewards distributed to participants based on validator-aggregated scores. "Emissions are the token reward stream the chain distributes to miners at each block, with the per-miner share determined by the validator consensus over recent scores."
- External verifier: An independent grading mechanism that evaluates trajectories separate from the synthesizer/teacher. "a verifier external to the synthesizer that grades each trajectory before it enters the training set."
- GRPO: Group Relative Policy Optimization; an RL method that uses group-relative advantages to shape policy updates. "the published SFT+GRPO bar ()"
- Group-relative RL: Reinforcement learning that compares rollouts within a group to compute relative advantages for updates. "group-relative RL with external verifiers"
- Importance-sampling drift: Mismatch introduced when updating a policy using rollouts generated by a frozen behavior policy. "addresses the importance-sampling drift between the frozen behaviour policy that seeds the rollouts and the policy currently being updated"
- Index-based agentic commerce: Agents operate over a static, queryable product index rather than live web pages. "ShoppingBench is an example of index-based agentic commerce"
- KTO (Kahneman–Tversky Optimization): A preference-optimization method that learns from unpaired desirable/undesirable labels. "We use KTO \citep{ethayarajh2024kto} for the preference-refinement pass."
- Leak-cluster guard: A grouping and partitioning strategy to prevent paraphrased duplicates from leaking into held-out sets. "with a leak-cluster guard that groups paraphrased or attribute-reordered variants of the same underlying problem"
- LoRA (Low-Rank Adaptation): A parameter-efficient fine-tuning technique injecting low-rank adapters into model weights. "One-epoch LoRA SFT on the structural-filter corpus produces the SFT base."
- Miner: A participant who submits an agent/model to a subnet for evaluation and potential on-chain rewards. "A miner is an independent participant who submits a candidate model or agent to the subnet for evaluation."
- On-policy distillation: Distillation using rollouts from the current policy, often with per-step KL constraints to a teacher. "The published recipe runs step-wise on-policy distillation against a frontier teacher at this point."
- pass@: The success rate when evaluating up to k sampled attempts per problem. "The supervised stack leaves a large pass@$8$ to pass@$1$ gap ( vs )"
- Production-strict scoring: Evaluation with strict, deployment-like rules and harness configurations to mirror real use. "On a leak-cluster-guarded held-out partition scored production-strict, the model lifts..."
- Reasoning-quality coefficient: A per-trajectory score from an LLM judge assessing the quality of the agent’s reasoning process. "A reasoning-quality coefficient is then assigned by an LLM judge we built"
- Rejection-sampled re-SFT: A training loop that samples multiple rollouts and keeps only those meeting strict success criteria for further SFT. "rejection-sampled re-SFT"
- RLVR (Reinforcement Learning from Verifier Rewards): RL that optimizes against signals provided by an external verifier rather than supervised labels. "The leading recipes (RLVR, group-relative RL, rejection-sampled re-SFT)..."
- SFT (Supervised Fine-Tuning): Fine-tuning a model on labeled trajectories to imitate desirable behavior. "Stage 1: SFT on the structural-filter corpus."
- SS58 hotkey: A cryptographic key format used to sign submissions and evaluations on Bittensor subnets. "Every interaction with the subnet, both submission and evaluation, is cryptographically signed by an SS58 hotkey"
- Subnet (Bittensor): An independently run, task-specific network within Bittensor that evaluates and rewards models. "We built ORO Subnet 15 (SN15) on Bittensor with exactly this goal."
- Tool-match reward: A reward term that compares the agent’s chosen tools and arguments to a teacher’s at each step. "replaced the outcome-only reward with a blended outcome-plus-tool-match reward"
- Trajectory substrate: The structured supply of multi-turn, judged traces that post-training algorithms consume. "Small-model agentic post-training is bottlenecked less by the choice of algorithm than by the trajectory substrate it consumes."
- Validator: An operator who executes and scores submitted agents on the task surface and reports results on-chain. "A validator is an independent operator who runs miner submissions against the subnet's task surface, scores the outputs, and reports the scores back to the chain."
Collections
Sign up for free to add this paper to one or more collections.
