Reduce Numbat’s Training-Memory Overhead

Reduce the approximately 1.9-fold training-memory overhead of numbat relative to the reference implementation at the reported configuration, which is attributed to AMP shadow parameter copies and allocator high-water behavior.

Background

The reported YOLOv8m-class COCO training run uses 13.5 GiB of memory per GPU in numbat, compared with approximately 7 GiB for the reference implementation at the same configuration. Although incremental activation freeing substantially reduces numbat’s memory use from 18.5 GiB, a residual factor of approximately 1.9 remains. The paper attributes this gap to mixed-precision shadow parameter copies and allocator high-water behavior, but does not resolve it, identifying reduction of this overhead as an open engineering item.

References

Peak training memory is 13.5\,GiB/GPU after incremental activation freeing ($-26.8\%$ from 18.5), vs. $\approx$7\,GiB for the reference at the same configuration; the residual factor $\approx$1.9$\times$ is attributed to AMP shadow parameter copies and allocator high-water behavior, and is an open engineering item.

— Numbat: Building and Verifying a Self-Contained Machine-Learning Stack  (2609.10632 - Tran et al., 9 Sep 2026) in Section 5, Section 5.2 “Throughput, memory, energy” (paragraph beginning “Memory.”)