Optimizing Memory Allocation in Distributed Clusters with Predictive Modeling

Abstract: In modern distributed systems, efficient resource allocation is a vital aspect to maintain scalability, reduce operational costs, and ensure fast execution even across heterogeneous workloads. Predictive models for resource usage are essential tools for optimizing allocation and preventing system bottlenecks. Predictive memory allocation has asymmetric costs as a key challenge: underallocation causes failures while overallocation wastes memory. We propose a regression method based on a LightGBM and XGBoost ensemble trained to predict high conditional quantiles. To further account for the high cost of underallocations we add a multiplicative safety factor. With our method we are able to reduce the number of under-allocated jobs from 4.17% to 2.89% and average overallocation from 148% to 44.51% on a real-world dataset of build jobs provided by SAP. We further explore the pareto frontier between optimization for underallocation and for overallocation.

• The paper introduces a quantile regression ensemble method leveraging LightGBM and XGBoost to strategically mitigate both underallocation and overallocation in memory management.
• It employs extensive feature engineering and Bayesian optimization to reduce underallocation from 4.17% to 2.89% and cut overallocation waste by over 70% compared to baseline methods.
• By exploring the Pareto frontier, the approach enables parameterizable and adaptive allocation policies that can be tuned to varying risk tolerances in distributed clusters.

Predictive Quantile Regression Ensembles for Memory Allocation Optimization in Distributed Clusters

Problem Motivation and Context

Efficient resource allocation in distributed cluster environments is critical for operational scalability and cost reduction, particularly in industrial continuous integration workflows where workloads are heterogeneous and memory demands variable. Memory allocation presents asymmetric risk: underallocation risks catastrophic job failure and wasted computation, while overallocation induces operational inefficiency and resource wastage. Manual allocation strategies and conventional bin-packing algorithms have typically traded off these competing costs suboptimally. Recent advances in ML-based performance modeling have opened new avenues for prediction-driven resource management—yet solutions must specifically address the high-impact asymmetry in allocation errors.

Quantile Regression Ensemble Modeling

The paper proposes a two-layered ensemble regression scheme using LightGBM and XGBoost, both optimized for upper conditional quantile prediction ($\alpha \in [0.90, 0.99]$). This quantile-centric approach, paired with multiplicative safety scaling ($s \in [1.00, 1.15]$), generates allocations targeted to be above peak requirements in $\alpha$ fraction of cases, thereby directly reducing underallocation frequency. Model outputs are aggregated via per-row maximum to further reinforce allocation conservatism.

Feature engineering leverages extensive temporal decomposition and workload characterization, leading to derived predictors such as lag-1 memory histories and rolling 95th percentiles specific to job profiles. Bayesian optimization via Optuna's TPE balances underallocation (weighted by empirical penalty) and overallocation, with cross-validation ensuring generalization.

Exploration of the Pareto Frontier

The trade-off between underallocation and overallocation is visualized as a Pareto frontier parameterized by ($\alpha, s$) pairs for the LightGBM+XGBoost ensemble.

Figure 1: Pareto frontier for the LightGBM+XGBoost ensemble, illustrating the trade-off between memory underallocation rate and overallocation ratio as ($\alpha, s$) vary.

Three operational allocation policies are characterized along the frontier:

• Balanced: Achieves competitive cost minimization (2.89% underallocation, 44.51% overallocation).
• Low Waste: Aggressive policy targeting memory efficiency (25.5% overallocation, 12.6% underallocation).
• Low Underallocation: Highly conservative policy (0.20% underallocation, 78.2% overallocation), still dramatically reducing waste versus baseline (148%).

This parameterizable spectrum supports adaptive deployment aligned with risk tolerance or business requirements in real-world cluster management.

Empirical Evaluation and Baseline Comparison

The evaluation leverages SAP’s large-scale CI build trace dataset, extracting 40 engineered features from raw telemetry. Historical predictors dominate in importance, emphasizing the necessity for temporal context in memory consumption modeling.

The ensemble's balanced configuration demonstrates strong improvements: underallocation is reduced from 4.17% (baseline) to 2.89%, while overallocation is cut by over 70% (from 148% baseline to 44.51%). Job-wise allocation distributions further reveal a significant decrease in resource waste across the spectrum of jobs compared to the SAP manual allocation baseline.

Figure 2: Job distribution by allocation quality for the ensemble method versus manual allocation baseline, highlighting reduced waste and improved safety.

Inference latency ($2 \times 10^{-5}$ s/job) suggests suitability for real-time prediction in production-grade environments.

Architectural and Methodological Implications

A critical finding is that the conservative quantile objective and aggregation strategy drive allocation performance more than specific algorithmic details. Both LightGBM and XGBoost, separately and combined, outperform deep learning-based approaches for these tabular datasets, further affirming the utility of tree-based models in structured ML resource management.

The paper’s quantile ensemble strategy is broadly applicable in operational contexts where allocation safety is non-negotiable and efficiency a continual priority. Model deployment can be flexibly tuned to business-critical trade-offs without behavioral instability.

Prospects for Future Research

Potential future directions include:

• Hybrid allocation policies: Employing differentiated models for jobs with exceptionally high predicted memory needs.
• Reinforcement learning integration: Real-time allocation via online learning, dynamically adapting to workload and cluster variability.
• Expansion to multi-resource optimization: Extending quantile ensemble methodologies to simultaneously optimize CPU, I/O, and memory.

Such extensions could further leverage quantile regression ensembles for scalable, adaptive, and safe resource allocation in evolving distributed systems.

Conclusion

The paper delivers a rigorous framework for memory allocation optimization in distributed clusters, leveraging quantile regression ensembles and robust feature engineering to substantially reduce both underallocation risk and overallocation waste. The operational flexibility enabled by Pareto frontier exploration and parameterizable allocation policies sets a foundation for practical deployment and future research. Empirical evidence confirms the efficacy of conservative ensemble strategies, with implications for both the methodology of ML-driven resource prediction and the architectural design of distributed cluster management systems.