GPUMemNet: GPU Memory Estimation for CARMA

• GPUMemNet is a novel ML-based GPU memory estimator in CARMA designed to predict pre-launch memory requirements for safe task collocation.
• It formulates memory estimation as a classification over discretized bins using architecture-specific features to balance underestimation and overestimation risks.
• Empirical results show that GPUMemNet nearly eliminates OOM failures while optimizing GPU utilization compared to prior estimators like Horus and FakeTensor.

Searching arXiv for GPUMemNet and closely related GPU-memory estimation / unified-memory work. GPUMemNet is the GPU-memory estimation component introduced within CARMA, a collocation-aware resource management system for deep learning training workloads on shared GPUs. In the formulation reported by CARMA, GPUMemNet is a lightweight, ML-based estimator that predicts the GPU memory requirement of an arriving training task before placement, so that the scheduler can decide whether collocation is feasible without inducing out-of-memory failures. Its role is deliberately narrow: it is not a general-purpose GPU memory profiler, not a runtime phase predictor, not a continuous telemetry-driven controller, and not a full memory-management stack. Rather, it is a pre-launch, metadata-driven estimator whose output is consumed by CARMA’s admission and placement logic (Yousefzadeh-Asl-Miandoab et al., 26 Aug 2025).

1. Scope, definition, and design objective

GPUMemNet is defined in CARMA as “a novel ML-based GPU memory estimator framework for DL training tasks,” introduced to support task-level GPU collocation in enterprise-style training clusters (Yousefzadeh-Asl-Miandoab et al., 26 Aug 2025). The immediate scheduling problem is twofold: collocation can improve utilization, but it can also trigger out-of-memory crashes for a newly arriving task and introduce resource interference among colocated jobs. GPUMemNet addresses the first problem—memory feasibility—while CARMA’s utilization-aware collocation policies address the second.

The estimator is therefore best understood as an admission-time predictor. CARMA parses the incoming job, extracts model and task features, invokes GPUMemNet to estimate the task’s memory need, and combines that estimate with observed per-GPU free memory and utilization. The default policy is MAGM, “Most Available GPU Memory,” under an interference guard based on SM activity, with the default threshold given as $\text{SMACT} \le 80\%$ (Yousefzadeh-Asl-Miandoab et al., 26 Aug 2025). This arrangement places GPUMemNet squarely in the scheduler control plane rather than the execution path of a running model.

The paper is explicit about what GPUMemNet is not. It is not formulated as a predictor of a full memory-usage trajectory, per-iteration allocator reservation, or online memory phase changes. It is also not described as being re-run periodically during execution. CARMA instead relies on runtime monitoring of memory usage and utilization for later placement decisions concerning future arrivals, while residual placement failures are handled by a recovery mechanism rather than by online GPUMemNet correction (Yousefzadeh-Asl-Miandoab et al., 26 Aug 2025).

2. Problem formulation and prediction target

GPUMemNet’s target is the GPU memory requirement of a training task as needed for placement-time feasibility checks. The labels are not continuous byte counts. Instead, the framework formulates estimation as classification over discretized memory intervals. The paper gives the label construction explicitly: intervals such as 0–1 GB, 1–2 GB, and 2–3 GB are mapped to class indices 0, 1, and 2, respectively (Yousefzadeh-Asl-Miandoab et al., 26 Aug 2025).

This classification formulation is motivated by an observed staircase growth pattern in GPU memory usage as architectural parameters change. The argument presented in CARMA is that scalar regression is less suitable because smooth regressors struggle with discontinuities, plateau regions provide little gradient, and boundaries create unstable error spikes. Classification over bins is described as more stable under these conditions (Yousefzadeh-Asl-Miandoab et al., 26 Aug 2025). A plausible implication is that GPUMemNet is optimized more for safe coarse feasibility decisions than for byte-level allocator modeling.

The binning granularity is architecture-family dependent. For the more complex CNN and Transformer datasets, the paper reports 8 GB bins; for MLPs, it reports 1 GB and 2 GB ranges (Yousefzadeh-Asl-Miandoab et al., 26 Aug 2025). This matters operationally because CARMA consumes the output as an approximate demand estimate during placement. The paper also notes a tradeoff here: coarse-grained bins can become conservative and can “sideline opportunities for finer-grained collocation,” particularly on lighter traces (Yousefzadeh-Asl-Miandoab et al., 26 Aug 2025).

A central consequence of the formulation is asymmetry in error costs. Underestimation can produce out-of-memory failures, whereas overestimation wastes collocation opportunities. CARMA argues that GPUMemNet is designed to balance these two failure modes more effectively than the analytical Horus estimator and PyTorch FakeTensor, and it characterizes GPUMemNet as providing estimates closest to actual GPU memory consumption while almost never underestimating on the reported real-model comparisons (Yousefzadeh-Asl-Miandoab et al., 26 Aug 2025).

3. Feature representation and architecture awareness

GPUMemNet is explicitly architecture-aware. CARMA argues that GPU-memory behavior differs sufficiently across DL architecture families that “not a single but a set of specialized models” should be developed. Accordingly, the paper constructs separate datasets for MLPs, CNNs, and Transformers and trains family-specific classifiers rather than a single universal estimator (Yousefzadeh-Asl-Miandoab et al., 26 Aug 2025).

The common input features are:

• number of linear layers,
• number of batch-normalization layers,
• number of dropout layers,
• batch size,
• number of parameters,
• activations,
• activation encoding using sine/cosine features.

The activation encoding choice is notable: activation categories are represented with two continuous inputs via sine/cosine features rather than one-hot vectors (Yousefzadeh-Asl-Miandoab et al., 26 Aug 2025).

GPUMemNet also uses a structured sequential representation of the network, expressed as tuples of

$$(\text{layer type}, \text{number of activations}, \text{number of parameters})$$

for the layers in the model. The stated purpose is to capture not only total counts but also the distribution of parameters and activations across layers, since memory usage depends on where variables must be retained during computation (Yousefzadeh-Asl-Miandoab et al., 26 Aug 2025). For CNNs, one additional feature is included: the number of convolutional layers.

The paper is equally explicit about omitted inputs. GPUMemNet is not described as consuming GPU model identity, framework version, online runtime telemetry, execution traces, dataset identity, mixed-precision flags, gradient checkpointing flags, distributed topology, or allocator-specific metadata as estimator inputs. The reported training and evaluation environment is fixed—PyTorch 2.4.1 with CUDA 12.2 on NVIDIA A100 GPUs—so the estimator is effectively conditioned on a particular software and hardware stack, but not on those factors as explicit features (Yousefzadeh-Asl-Miandoab et al., 26 Aug 2025).

This design makes GPUMemNet a metadata-driven cold-start estimator. It operates before launch using parsed task characteristics alone. The offline dataset labels come from monitored one-minute training runs, but the online estimator does not require a warmup execution of the arriving job in the cluster’s critical path (Yousefzadeh-Asl-Miandoab et al., 26 Aug 2025).

4. Model families, training methodology, and reported accuracy

GPUMemNet is a framework rather than a single fixed neural architecture. CARMA studies two estimator families: MLP ensembles and Transformer-classifier ensembles (Yousefzadeh-Asl-Miandoab et al., 26 Aug 2025).

For the MLP ensemble, each base model is a randomly structured feedforward network with 1 to 8 hidden layers, decreasing numbers of neurons per hidden layer, hidden width decaying exponentially from a maximum of 8 to a minimum of 4, ReLU activations, and batch normalization in all hidden layers. The ensemble output is produced by averaging the outputs of multiple such MLPs. The exact ensemble size is not reported (Yousefzadeh-Asl-Miandoab et al., 26 Aug 2025).

For the Transformer-based estimator, the model includes an MLP-based input embedding layer, positional encodings, and a stack of Transformer encoder blocks. The reported hyperparameter choices are $d \in \{4, 6\}$, 2–3 encoder layers, 1 attention head, feedforward size fixed at 4, and dropout rates between 0.0 and 0.3 (Yousefzadeh-Asl-Miandoab et al., 26 Aug 2025). The encoded sequence representation is concatenated with the auxiliary structured features and then passed to a final MLP ensemble for classification. In effect, this is a hybrid sequence encoder plus classifier.

Training uses cross-entropy loss and Adam. Evaluation uses 3-fold cross-validation with stratified sampling. The paper states that each fold uses 70% of the split for training and 30% for validation, while the test set is held out from a separate 30% split (Yousefzadeh-Asl-Miandoab et al., 26 Aug 2025). No learning rate, batch size, number of epochs, weight decay, or calibration method is reported.

The reported estimator-level results are summarized below.

Dataset	Best reported result	Note
MLP	Transformer estimator, 2 GB bins: accuracy 0.98, F1 0.97	Highest reported MLP result
CNN	MLP estimator, 8 GB bins: accuracy 0.83, F1 0.83	Chosen later for CARMA
Transformer	MLP estimator, 8 GB bins: accuracy 0.88, F1 0.88	Chosen later for CARMA

The full reported results are more granular. On the MLP dataset, the MLP estimator achieves accuracy 0.95 and F1 0.93 for 1 GB bins, and accuracy 0.97 and F1 0.96 for 2 GB bins. The Transformer estimator achieves accuracy 0.97 and F1 0.96 for 1 GB bins, and accuracy 0.98 and F1 0.97 for 2 GB bins. On the CNN dataset, the MLP estimator reports accuracy 0.83 and F1 0.83, whereas the Transformer estimator reports 0.81 and 0.81. On the Transformer dataset, the MLP estimator reports 0.88 and 0.88, whereas the Transformer estimator reports 0.86 and 0.86 (Yousefzadeh-Asl-Miandoab et al., 26 Aug 2025).

An important practical outcome is that, although the Transformer-based estimator appears more expressive, the paper states that later CARMA experiments use the MLP-based estimators because they obtained higher accuracy for CNNs and Transformers in those settings (Yousefzadeh-Asl-Miandoab et al., 26 Aug 2025). This suggests that sequential layer modeling was not uniformly dominant under the reported data regime.

5. Integration into CARMA’s placement and recovery loop

CARMA’s end-to-end control path gives GPUMemNet its operational meaning. Users submit jobs through a SLURM-like interface; tasks enter a FIFO queue; a parser extracts model and task features; monitoring gathers per-GPU memory usage and utilization over a one-minute window using DCGM and nvidia-smi; GPUMemNet estimates the arriving task’s GPU memory requirement; and the scheduler applies its collocation policy to decide whether the task can be collocated and on which GPU (Yousefzadeh-Asl-Miandoab et al., 26 Aug 2025).

The default memory-side feasibility logic is paired with MAGM, which chooses among GPUs satisfying memory and utilization conditions by selecting the one with the most free memory. CARMA also uses utilization-aware policies, including the $\text{SMACT} \le 80\%$ condition, to cap interference (Yousefzadeh-Asl-Miandoab et al., 26 Aug 2025). GPUMemNet therefore supplies one constraint in a joint decision process: memory sufficiency for safe launch.

The paper does not present a formal GPUMemNet predictor equation or a confidence-aware safe-placement inequality. The scheduling rule is algorithmic. In the oracle experiments, where true memory demand is known, the paper states the closest thing to an explicit safe-placement condition:

$$\text{available GPU memory} \ge \text{actual task memory need} + 2\text{GB}.$$

The 2 GB margin is motivated by fragmentation risk (Yousefzadeh-Asl-Miandoab et al., 26 Aug 2025). In the estimator-driven setting, robustness is instead obtained from discretized prediction, utilization caps, and a recovery path.

That recovery path is necessary because prediction is not treated as perfect and because fragmentation is outside GPUMemNet’s scope. CARMA periodically checks task error files; if an OOM crash is detected, the task is moved into a higher-priority recovery queue and rescheduled using the exclusive policy (Yousefzadeh-Asl-Miandoab et al., 26 Aug 2025). The paper gives a fragmentation example—9 GB reported free but split into 5 GB and 4 GB chunks, insufficient for an 8 GB contiguous allocation—to illustrate why a demand estimator cannot by itself guarantee safe launch.

The online overhead of GPUMemNet is reported as small. Across 100 runs, parser time is at most 2.6 ms, estimator inference is at most 16 ms on an NVIDIA A100 40 GB GPU and 32 ms on an AMD EPYC CPU, and the authors characterize both as negligible relative to CARMA’s one-minute monitoring interval and to DL training job durations (Yousefzadeh-Asl-Miandoab et al., 26 Aug 2025). This low latency is central to GPUMemNet’s scheduler-facing design.

6. Empirical behavior, limitations, and relation to adjacent GPU-memory work

At estimator level, GPUMemNet is compared against Horus and PyTorch FakeTensor. The paper’s qualitative conclusion is that GPUMemNet provides the closest estimations to actual GPU memory consumption and almost never underestimates. Horus is reported as capable of both under- and overestimation, with synthetic-MLP misestimations reaching up to 395 GB, while FakeTensor often underestimates TIMM-model memory and in other cases can overestimate by up to 1.8 TB (Yousefzadeh-Asl-Miandoab et al., 26 Aug 2025). The paper does not report MAE, MAPE, RMSE, calibration curves, or uncertainty intervals.

At system level, GPUMemNet’s effect appears through CARMA. On the 90-task trace with MAGM, integrating any estimator nearly eliminates OOMs: Horus without an SMACT precondition yields 1 OOM, FakeTensor 0, and GPUMemNet 1; with $\text{SMACT} \le 80\%$, all three yield 0 OOMs (Yousefzadeh-Asl-Miandoab et al., 26 Aug 2025). On the heavier 60-task trace, MAGM + GPUMemNet + MPS + $\text{SMACT} \le 80\%$ gives the best reported CARMA configuration, with 1 OOM and a 26.7% reduction in total trace time versus exclusive execution (Yousefzadeh-Asl-Miandoab et al., 26 Aug 2025). At overall system scale, CARMA reports GPU utilization over time increased by 39.3%, end-to-end execution time decreased by about 26.7%, and GPU energy use reduced by about 14.2%; those are CARMA-level results in which GPUMemNet is one component alongside collocation policy and recovery (Yousefzadeh-Asl-Miandoab et al., 26 Aug 2025).

The limitations are explicit. GPUMemNet does not model fragmentation, hardware heterogeneity, framework heterogeneity, or runtime memory-phase variation. Its inputs exclude mixed precision, checkpointing, ZeRO or distributed strategy, and allocator settings. Its training and evaluation are tied primarily to NVIDIA A100 40 GB GPUs, CUDA 12.2, and PyTorch 2.4.1 (Yousefzadeh-Asl-Miandoab et al., 26 Aug 2025). The GPT-2 case illustrates a coverage failure: the largest misestimate occurs on GPT-2 because it includes 1D convolutional layers absent from the synthetic Transformer dataset. The paper therefore argues that the open dataset should be extended over time (Yousefzadeh-Asl-Miandoab et al., 26 Aug 2025).

This positioning distinguishes GPUMemNet from adjacent GPU-memory research. A learning-based framework for CPU-GPU Unified Virtual Memory oversubscription management predicts next-page deltas and couples prediction to prefetch and pre-eviction policies in order to reduce page thrashing under oversubscription; it targets page-level runtime control, not pre-launch task-memory estimation (Long et al., 2022). A systems paper on integrated CPU/GPU platforms for autonomous workloads builds a hand-crafted analytical model of cudaMalloc, cudaMallocManaged, and cudaHostAlloc overheads and uses per-task policy switching plus overlap-aware scheduling; it is mechanistic rather than learned and focuses on policy selection under unified-memory pressure rather than DL task collocation (Bateni et al., 2020). A recent characterization of AMD MI300A Unified Physical Memory shows that integrated CPU/GPU systems can remove explicit host-device duplication and reduce memory costs by up to 44%, but it addresses allocator behavior, TLB reach, page faults, and coherence in an integrated APU setting rather than server-scale pre-launch estimation for discrete-GPU training clusters (Wahlgren et al., 18 Aug 2025).

Taken together, these comparisons clarify GPUMemNet’s niche. It is neither a general GPU memory manager nor a runtime predictor of memory traffic. Its contribution lies in packaging architecture-family-specific, metadata-driven, low-latency classification of task memory demand into a scheduler-compatible form. That narrowness is also its practical strength: GPUMemNet gives CARMA a pre-launch estimate that is more informed than static thresholds and lighter weight than trace-driven simulation, enabling collocation decisions that are conservative enough to reduce OOM risk without abandoning GPU sharing altogether (Yousefzadeh-Asl-Miandoab et al., 26 Aug 2025).