Create and explore imaginative 3D worlds with our new product Dreamfield (opens in a new tab)

Compare teacher-action supervision objectives in ALFWorld and WebShop

Determine whether unit-weight behavior cloning provides different training outcomes from the signed sampled-token on-policy-distillation surrogate for teacher-written actions in ALFWorld and WebShop.

Background

For ALFWorld and WebShop, Divergence-Triggered Dynamic Distillation uses the existing sampled-token on-policy-distillation surrogate for both student- and teacher-written actions. Teacher-written tokens are weighted by the signed difference between the teacher’s and rollout student’s log probabilities, so the resulting supervision is not equivalent to unit-weight behavior cloning.

The paper explicitly notes that this weighted surrogate is not an unbiased estimator of either KL direction and reports that it has not been compared with unit-weight behavior cloning in these environments. Establishing this comparison would clarify whether the observed agentic results depend on the chosen teacher-action weighting scheme or on teacher takeover itself.

References

We have not compared it against unit-weight behavior cloning within ALFWorld or WebShop.

— DTDD: Divergence-Triggered Dynamic Distillation for Reliable On-Policy Supervision  (2610.19302 - He et al., 13 Oct 2026) in Appendix, Section “Interactive Training Protocol,” subsection “ALFWorld and WebShop,” paragraph “Teacher-action supervision”