Generalization of calibrated directed search

Determine whether the calibration advantage of conformal acquisition and the performance advantage of greedy conformal acquisition over conformal-optimistic tree search generalize across neural architecture families and constrained search testbeds.

Background

The paper’s directed-search evaluation is conducted on a single constrained ResNet-like per-layer architecture space with a strong proxy, a 256 kB memory budget, and three random seeds. In that setting, greedy conformal acquisition outperforms random search across the tested evaluation budgets, while conformal-optimistic Monte Carlo tree search provides only an early-budget advantage and ultimately performs no better than random search.

Because the evidence comes from one family and one constrained testbed, the authors explicitly identify uncertainty about whether these findings are transferable to other architecture families and search environments. The open problem concerns both the generality of the online-calibration benefit and the observed superiority of greedy acquisition over tree search.