Workload composition smooths aggregate power demand while sustaining short-horizon ramps in AI data centers

Abstract: AI is driving rapid growth in electricity demand, yet the grid-facing power dynamics of AI data centers remain poorly understood. Here we show that, in shared-GPU systems, the composition of batch and inference workloads decouples aggregate power variability from short-horizon ramping. As the inference share rises, variability becomes U-shaped, whereas ramping becomes hump-shaped, particularly under higher loading. The magnitude and turning points of these patterns also depend on system loading. Using a trace-calibrated framework linking workload arrivals, queueing, scheduling, and GPU power, we show that the underlying mechanism is asymmetric. At intermediate workload mixes, queued batch jobs fill capacity left idle by fluctuating inference demand, reducing aggregate power variability. However, short-horizon ramping remains elevated because inference-side fluctuations propagate more directly into realized power. AI data centers should therefore be understood as dynamic systems whose workload composition shapes their grid impact.

• The paper demonstrates that dynamic workload composition in AI data centers decouples overall power variability from rapid short-horizon ramps by leveraging batch job queue buffering.
• It employs trace-calibrated synthetic modeling and empirical GPU workload data to reveal nonlinear, non-additive power behaviors in hybrid systems.
• The analysis highlights practical implications for grid stability and AI system design, urging revised operational and planning paradigms.

Workload Composition Smooths Aggregate Power Demand while Sustaining Short-Horizon Ramps in AI Data Centers

Introduction

This work investigates the grid-facing power dynamics in AI data centers operating under hybrid workload regimes, where both batch and inference jobs co-exist and share GPU resources. Unlike traditional analytical approaches that model data center loads as exogenous or simple aggregations of GPU energy use over long timescales, this analysis leverages large-scale, trace-calibrated synthetic modeling combined with empirical input data to characterize the interaction between workload composition, queueing, and GPU scheduling. The central contribution is a detailed quantification and mechanistic explanation of how the composition of batch and inference work reshapes power variability and ramping—decoupling overall variability from short-horizon ramps and leading to non-additive, nonlinear behaviors in shared-GPU environments.

Nonlinear Power Variability and Ramping in Hybrid Systems

Empirical results show that hybrid AI data centers, where both batch and inference jobs run on the same GPUs, exhibit distinctly nonlinear grid-level power dynamics that cannot be predicted by linear interpolation between batch-only and inference-only behaviors. Notably, as the fraction of inference workload increases, total power variability forms a U-shaped curve with respect to inference share, whereas short-horizon ramping follows a hump-shaped curve. Importantly, these two system properties decouple, enabling configurations with reduced overall variability while maintaining high ramping rates on short timescales.

Figure 1: Nonlinear relationships between power variability/ramping and inference share, including decomposition of overall power into batch and inference components in hybrid configurations.

The model is built upon three major data sources:

• MIT SuperCloud GPU batch workload traces with job-level power profiles,
• Azure LLM inference traces totaling 44 million requests,
• Direct LLM serving measurements mapping token throughput to GPU demand and power.

In the constructed simulation, GPU capacity and utilization are fixed, with only workload mix between batch and inference varied. Analysis of these traces demonstrates that at intermediate inference shares, queued batch jobs can fill capacity left idle by bursty inference demand. This buffers aggregate variability more effectively than either workload alone, but does not equivalently mitigate rapid power ramps, as inference-side fluctuations propagate directly into the power signal due to their latency-sensitivity and short queueing times. As a result, optimal smoothing (minimum variability) and optimal ramping (minimum ramp rate) are achieved at different workload compositions.

Queue-Buffered Batch Mechanisms

Batch workloads exhibit strong queue-mediated smoothing of power swings. The batch scheduling mechanism, including checkpointing and job-start prioritization, absorbs much of the burstiness in job arrivals—decoupling arrival fluctuations from actual power usage. The link between changes in arrivals and realized running jobs is weak, and only strengthens at the level of executing jobs to GPU use to power.

Figure 2: Relationships in batch-only systems across 4-hour changes—arrivals, running jobs, and GPU/power utilization—reveal strong absorption of arrival-side burstiness by queues.

Sensitivity analysis shows that the strength of this smoothing is modulated by utilization and batch job flexibility (determined by checkpoint interval length). Higher utilization (and shorter checkpoints) enhance the capacity for the batch queue to absorb incoming fluctuation, reducing variability without disproportionately increasing ramping rates. This mechanism is directly responsible for the reduction of aggregate variability in the hybrid regime, where queued batch jobs opportunistically fill valleys left by fluctuating inference demand.

Inference-Dominated Short-Horizon Ramping

Inference workloads, designed for latency constraints, maintain little to no buffer, transmitting workload burstiness nearly directly to realized power. Across a range of utilization levels and serving configurations, ramping is only moderately mitigated by increased request durations, larger batch sizes, or capacity saturation. At high utilization, arrival-power correlation weakens as requests are queued or rate-limited due to resource saturation, but this effect is secondary compared to the directness of inference-induced ramps.

Figure 3: Inference-only systems display strong association between 15-minute fluctuations in request arrivals and realized power, especially at moderate utilization, with partial smoothing at high utilization due to request duration and saturation.

These findings indicate that in hybrid environments, short-horizon ramping is dominated by inference workload fluctuations. As the inference share grows, ramping rates initially increase, peaking at intermediate-to-high inference shares before flattening or even declining due to resource saturation limiting real-time response.

Practical and Theoretical Implications

The separation between long-term variability and short-horizon ramping in hybrid AI data centers has direct implications for grid integration, operational reliability, reserve procurement, and emissions accounting. Systems that appear "smooth" in aggregate (due to batch buffering) can simultaneously be challenging to follow on short timescales, demanding greater grid flexibility or more responsive ancillary service deployment. Traditional planning paradigms that depend on annualized energy summaries or simple static load integration will systematically miss these nonlinearities, leading to misestimation of infrastructure needs or risk exposure.

Furthermore, these results establish that resource-sharing policies and workload flexibility parameters (e.g., checkpoint interval tuning, queue admission control) are levers for shaping not only mean data center power, but also the higher-order temporal profile that matters for grid stability.

From a broader AI systems design perspective, this analysis highlights the necessity of workload-aware scheduling not just for compute throughput or cost, but also for external power profile shaping—a new axis of resource and system optimization.

Future Directions

Modeling in this work focuses on direct load as seen by the electrical grid, omitting facility-level controls such as intelligent cooling, local storage, and backup generation, all of which may further buffer or reshape short-term variability and ramps. Future research should directly integrate these control layers into grid-facing models. Additionally, with growing multi-tenant and geographically distributed AI infrastructures, time-zone-aware arrival smoothing, and more complex admission/routing policies, systemic effects may further evolve.

Dynamically adjusting workload composition, changing checkpoint policy adaptively based on grid conditions, or introducing demand response incentives at the resource manager or job scheduler level are also promising avenues for shaping aggregate flexibility and supporting large-scale grid integration of AI data center loads.

Conclusion

This analysis demonstrates that workload composition, queueing, and scheduling jointly modulate the grid impact of modern AI data centers in a nonlinear, non-additive fashion. Queue-buffered batch jobs can smooth aggregate variability, but this does not ensure low short-horizon ramping, as inference-driven power fluctuations persist and often dominate on minute-to-hour timescales. These findings necessitate new grid planning practices and AI system design principles that explicitly consider the temporal structure of data center power demand as shaped by internal workload dynamics (2604.10769).