Brand: Managing Training Data with Batched Random Access

Abstract: This paper propose Brand, a comprehensive memory management system for deep learning training (DLT) where the memory capacity is much smaller than the size of the training datasets. Brand starts with a bold design choice that data files are always read from disk in batch, named chunk. Based on this assumption, we propose efficient data access protocol in both single-node setting and distributed environment with multiple nodes. The protocol minimizes the wasted data read due to larger granularity, enables efficient inter-node prefetching, while still ensuring randomness required by DLT. The experimental results indicate that Brand can significantly accelerate data fetching in DLT, achieving up to a 4.57x improvement in end-to-end training compared to PyTorch.