PILOT Technical Report

Abstract: Existing agentic approaches for recommendation system optimization remain fundamentally reactive: they adjust parameters in response to observed metric changes but lack the ability to proactively design controlled experiments, personalize strategies at the user-segment level, or accumulate reusable experimental methodology across tasks. We present PILOT (Proactive Insight Learner for Online Tree-Experiments), an LLM-agent framework that organizes three roles within a constrained control loop where deterministic services enforce all safety, statistical, and permission boundaries: (1) an Experiment Manager that drives the full experiment lifecycle -- task intake, observation governance, anomaly recovery, and postmortem -- by selecting only from a rule-generated legal-command envelope; (2) a Search Planner that proposes candidate decision trees for user-segment-level personalization, invoked only when the Manager requests planning; and (3) a Memory Curator that asynchronously distills experiment outcomes into strategy-level domain knowledge and provenance-tracked methodology, failure-isolated from the main loop. The Manager makes the agent proactive, the Planner enables population-level personalization beyond global tuning, and the Curator turns every completed task into a learning opportunity for the next. Deployed on Taobao's platform with 5 experimental buckets, PILOT is compared against ROAM(Reactive Optimization with Agent-driven Moves), a free-exploration agent without lifecycle governance or structured hypothesis testing. PILOT achieves up to +1.40% IPV, +1.60% Core IPV, +0.96% transaction count, and +1.50% transaction amount, improving over ROAM's best results (+1.00% IPV, +0.90% Core IPV, +0.60% transaction count, +1.13% transaction amount) while raising search efficiency from 53.3% to 93.3% (+40 pp), with no human intervention throughout the experimental cycle.

• The paper introduces PILOT, a three-role LLM architecture combining lifecycle management, segment-level PolicyTree search, and memory curation within deterministic safety and statistical controls.
• Experiments on Taobao’s Homepage Guess-You-Like system showed PILOT achieved up to +1.40% IPV, +1.60% core IPV, and 93.3% search efficiency versus 53.3% for ROAM.
• The framework prevents unsafe or unsupported decisions through frozen policies, pre-registered confirmation tests, immutable decision certificates, human approval for irreversible actions, and governed cross-task memory.

Motivation and problem statement

Industrial recommendation platforms run online experimentation as a continuous control loop—splitting buckets, deploying strategies, and watching metrics—but this loop has traditionally been driven by human experimenters and does not scale with the number of concurrent experiments and user segments. The paper identifies three structural limitations of existing LLM-agent coordinators that act on the recommendation system itself rather than on the recommendation list (Wang et al., 10 Feb 2026, Wu et al., 21 Apr 2026, Zhang et al., 27 Mar 2026, Mu et al., 15 Jun 2026, Lao et al., 25 Jun 2026): reactive response (agents adjust parameters after observing metric changes rather than proposing and validating hypotheses before deployment), global-only strategies (optimization applies uniformly to a whole bucket with no user-segment personalization), and no methodology accumulation (memory captures task content but is not distilled into reusable experimental methodology that compounds across tasks).

PILOT (Proactive Insight Learner for Online Tree-Experiments) addresses these limitations by organizing three LLM roles within a constrained control loop in which deterministic services enforce all safety, statistical, and permission boundaries. The design principle throughout is a strict separation of authority: LLM roles hold judgment authority (choosing among admissible options), while deterministic services hold enforcement authority (generating the option space, validating every choice, committing state).

Framework overview

The control loop proceeds as follows: an event triggers the Manager Guard to compute which actions are currently legal, derived purely from a state machine and frozen policy without LLM involvement; the Experiment Manager picks one action from that list; the Guard validates the pick against state, parameter bounds, and outstanding certificates; and the State Committer—the sole component allowed to write state—applies it atomically. Six deterministic services bound the loop: the Manager Guard, State Committer, Statistics Engine (immutable decision certificates), Contract Builder (pre-registered outcome criteria), Candidate Validator, and Action Enumerator.

The three LLM roles map one-to-one onto the identified limitations:

• Experiment Manager: drives the full lifecycle—task intake, playbook assembly, observation governance, anomaly recovery, scale validation, postmortem—rather than responding to alerts after the fact.
• Search Planner: proposes candidate decision trees that personalize strategy bundles at the user-segment level; invoked only when the Manager explicitly requests planning.
• Memory Curator: asynchronously distills settled outcomes into two stores—strategy evidence and methodology experience—and is failure-isolated so its failure degrades knowledge accumulation, not the running experiment.

Experiment Manager

Task intake converges unstructured natural-language objectives into a versioned, frozen task specification $\mathcal{T}$ through four completion states, with high-risk fields (randomization spec, guardrails, approval matrix) requiring explicit structured user input rather than silent inference. A playbook $\mathcal{P}$ is then assembled from user instructions plus methodology proven reliable on past tasks. A fixed authority hierarchy governs all subsequent decisions: frozen safety/statistical policy overrides the task specification, which overrides the playbook, which overrides approved methodology, provisional hints, and finally agent reasoning.

Lifecycle progression distinguishes four time layers—a look (one data read plus judgment), a wave (a batch of co-launched Challengers), an epoch (a window in which the Champion, estimator, and decision policy are frozen), and a search iteration (a complete plan-launch-observe-decide cycle). Observation governance rests only on data-health signals such as freshness, maturity, sample size, and integrity; treatment-lift direction stays hidden until a close is warranted, ruling out cherry-picked windows or "almost significant" early stops. Rounds terminate in one of three certificates—promote, reject, or continue—with guardrail breaches producing forced responses where the Manager's role narrows to confirming or escalating.

Two design choices deserve emphasis. First, search-period data $D_{\text{search}}$ cannot confirm the final Champion: confirmation requires a single pre-registered test on freshly collected held-out data $D_{\text{confirm}}$, and a non-significant result cannot be rescued by re-looking or folding in search data. Second, anything hard to reverse—deploying candidates, expanding traffic or budget, editing $\mathcal{T}$—always requires human approval.

Search Planner

The search operates over PolicyTrees: rooted trees mapping every user to exactly one leaf-bound strategy bundle, with internal nodes of three types (categorical split, numeric split, and stable MurmurHash3-based hashSplit enabling persistent sub-population assignment). Structural constraints from the SearchUniverse—maximum depth, leaf count, minimum global leaf share $w_{\min}$, maximum distinct features—are set by the user and not adjustable by the LLM; they guarantee both deployability and per-leaf statistical power.

The optimization maximizes total treatment effect against a fixed baseline tree $T_0$ via a sequence of trees, each differing from its predecessor by exactly one atomic action drawn from five kinds: split, prune, hashExpand, collapse, and updateStrategy. The Action Enumerator deterministically constructs the finite set of legal actions $\mathcal{A}(T^*)$ for each Champion; the Planner may only reference IDs already present in it.

Pre-screening runs feature quality diagnosis (pre-treatment availability, missing rate, temporal stability, coverage), candidate boundary generation scored by exposure-weighted posterior variance, and historical de-prioritization against Memory. Optional exploration traffic uses hashSplit probes to measure per-population gain contributions weighted by global share, preventing narrow high-lift segments from dominating rankings.

A central methodological contribution is the replacement of bottom-up trajectory summarization—common to prior frameworks (Wu et al., 21 Apr 2026, Lao et al., 25 Jun 2026, Mu et al., 15 Jun 2026)—with top-down hypothesis-first proposal. The paper argues bottom-up summarization suffers from high stochasticity across runs, uncontrollable accumulation direction, and no built-in confidence measure, leaving single-run heuristics indistinguishable from well-confirmed ones. Under PILOT's scheme, each PlannerProposal commits a falsifiable hypothesis with target population, expected mechanism, explicit rejection conditions, evidence references, and a conditional plan for both confirmed and refuted outcomes. Proposals are forbidden from containing effect estimates, significance levels, traffic recommendations, or promotion decisions—these belong exclusively to the deterministic machinery. The exploit–explore–falsify trichotomy values falsification proposals because negative results permanently prune hypothesis branches.

Memory Curator

The Curator maintains two stores. Strategy Evidence records directed bundle contrasts (from_bundle_id → to_bundle_id) with population scope, separating immutable per-round EvidenceItems from evolving cross-task ScopedClaims, and tagging evidence as causal (pre-specified randomized comparison) or diagnostic (post-hoc slices, probes). Diagnostic evidence can motivate follow-up candidates but never advances confidence. Methodology Experience indexes operational recommendations by operation type (wait, retry, guarantee volume, observe, scale up, rollback) and scenario predicate, sourced from immutable PostmortemRecords.

Both stores share a fork–merge architecture: each task forks an isolated branch invisible to concurrent tasks, and merges back only after terminal settlement. Confidence follows a governed lifecycle—draft → supported → approved—where a single decisive experiment can produce only a draft entry regardless of sample size; advancement requires consistent confirmation from independent tasks under exactly compatible retrieval scopes. Contradictory evidence triggers re-evaluation, with overturned claims explicitly marked superseded rather than overwritten, and replacement claims must independently traverse the full lifecycle. Memory remains advisory at every confidence level: it can never substitute for a live DecisionCertificate or expand permissions. This separation between within-task statistical significance and cross-task confidence prevents one-off results from prematurely becoming reusable knowledge.

Experimental results

Deployment ran on Taobao Homepage Guess-You-Like across 5 experimental buckets against a fixed baseline, comparing PILOT against ROAM, a free-exploration agent configuration lacking lifecycle governance and structured hypothesis testing over the same 8-dimensional search space. The task objective was +1.0% IPV within 7 days with cosine-decayed budget checkpoints.

Metric	ROAM best	PILOT best
IPV	+1.00%	+1.40%
Core IPV	+0.90%	+1.60%
Transaction count	+0.60%	+0.96%
Transaction amount	+1.13%	+1.50%

By day 4, four of five PILOT buckets independently cleared the +1.0% target with none turning net-negative, versus only one of five under ROAM. Search efficiency—an aggregate of three binary indicators per bucket (target exceeded, stable read with ≤1 reconfiguration, net search progress)—rose from 53.3% to 93.3% (+40pp). The case study traces concrete behavior: leaf-strategy retunes on days 2 (buckets 21, 24) and a structural pivot activating a previously-zero paging branch on day 4 (bucket 25), with the task depositing six memory entries including a super-additive ctr+ipv gain claim on its third cross-task confirmation.

Limitations and open questions

The authors are explicit that the headline comparison is before/after rather than randomized. Because checkpoint governance and the search-space change landed together, the reported gains cannot yet separate the contribution of lifecycle control from that of hypothesis-driven candidate proposal. A planned $2\times2$ ablation—isolating the Planner-like LLM lane and the Manager-like lifecycle control in four cells—is specified but gated on a calibrated per-action observation model and a comparable cross-bucket scoring protocol, and its results are not reported. The observational evidence is accordingly labeled suggestive rather than causal. Additional caveats: results cover a single platform and a five-day reported window within a seven-day objective; the memory system's compounding benefit is asserted architecturally but not quantified across many sequential tasks; and the confidence lifecycle's approval thresholds are described as "governed" without full specification of the numerical criterion.

Conclusion

PILOT demonstrates that LLM agents can move recommendation-system optimization from reactive metric response to proactive, hypothesis-driven experimentation when embedded in a constrained control loop with deterministic enforcement of safety, statistical, and permission boundaries. Its three-role architecture—lifecycle management, segment-level PolicyTree search, and provenance-tracked memory curation—delivered up to +1.40% IPV and a 40-percentage-point search-efficiency improvement over free exploration on Taobao with no human intervention during the experimental cycle. The principal open question the work leaves is causal attribution: whether the lifecycle governance or the hypothesis-driven planner contributes more to these gains awaits the controlled ablation the paper designs but does not yet report.