Separate the effects of checkpoint governance and search-space personalization

Determine the independent causal contributions of checkpoint governance and the decision-tree search-space change to PILOT’s observed improvement over ROAM by executing the planned randomized 2×2 ablation design.

Background

PILOT and ROAM differ simultaneously in governance and search-space structure: PILOT uses checkpoint governance and hypothesis-driven candidate proposals over user-segment decision trees, whereas ROAM performs freer reactive exploration. The reported comparison is therefore observational rather than randomized, so the measured improvement cannot establish which component produced the gain.

The paper specifies a planned 2×2 ablation with deterministic-only, Planner-like, Manager-like, and combined configurations. This design is intended to isolate lifecycle-control gains from candidate-proposal gains, but its execution is conditional on a calibrated per-action observation model and a comparable cross-bucket scoring protocol.

References

Because checkpoint governance and the search-space split landed together, this cannot yet separate the two effects---but it confirms that \pilot{}'s structured approach measurably increased productive use of the five-bucket budget over ROAM's free exploration.

— PILOT Technical Report  (2608.18637 - Lin et al., 19 Aug 2026) in Section 5.2.2, “Observational Evidence” (with the planned design specified in Section 5.2.1)