DECOFFEE: Decentralized Reinforcement Learning for Time-critical Workload Offloading and Energy Efficiency across the Computing Continuum

Abstract: The rapid proliferation of latency-sensitive and battery-constrained Internet-of-Things (IoT) applications has intensified the need for intelligent workload placement mechanisms across the Edge-Cloud computing continuum. In such environments, far-edge nodes must dynamically decide whether to execute workloads locally or offload them to neighboring nodes or the cloud, while accounting for execution delay, energy consumption, and strict timeout constraints. However, workload placement in large-scale distributed infrastructures is a highly dynamic and non-convex optimization problem due to stochastic arrivals, heterogeneous computing capacities, and time-varying network conditions. This paper proposes DECOFFEE, a decentralized reinforcement learning framework for time-critical workload offloading and energy-efficient operation across the computing continuum. The proposed multi-agent learning scheme jointly optimizes system delay, energy consumption, and workload drop rate through adaptive placement decisions. Each edge agent operates as an autonomous learning entity that derives an optimal policy from local system observations and predicted network conditions. The workload placement process is formulated as parallel Markov Decision Processes and solved using a Double Dueling Deep Q-Network (DQN) architecture enhanced with Long Short-Term Memory (LSTM) forecasting to anticipate future load conditions. Extensive simulations demonstrate that DECOFFEE and its variants consistently outperform conventional rule-based and heuristic placement strategies, achieving significant reductions in delay, energy consumption, and workload drop rate under varying traffic and network conditions.

• The paper introduces DECOFFEE, a decentralized DRL framework that optimizes workload placement in a three-tier computing continuum.
• It employs a Double Dueling DQN with integrated LSTM forecasting to proactively predict and route atomic, time-critical workloads.
• Extensive simulations demonstrate reduced execution delays, lower energy consumption, and minimized workload drops across heterogeneous environments.

Decentralized DRL for Time-Critical Workload Offloading in the Computing Continuum

System Architecture and Modeling

The DECOFFEE framework operates within a three-tier computing continuum architecture involving IoT devices, Edge Agents (EAs), and a Cloud Agent (CA). Workloads generated by IoT endpoints are routed through Radio Units to EAs, which must decide between three actions: (i) local computation, (ii) horizontal offloading to a neighboring EA, or (iii) vertical offloading to the CA. Each EA and the CA manage workload stacks with local and public queues implementing FIFO scheduling for both local and offloaded workloads. The connectivity matrix $\boldsymbol{G}$ characterizes wired/wireless inter-agent links and governs feasible horizontal offloading paths.

Figure 1: Three-tier computing continuum architecture depicting EA workload routing options and decision modalities.

Workloads are atomic, non-partitionable, and associated with stringent deadline constraints. Precise delay and energy consumption models track queuing, processing, transfer, and execution costs across each decision modality. The full queuing and placement structure inside EAs and the CA enables efficient traceability of workload states.

Figure 2: Internal placement and storage structure within EAs and CA, detailing DECOFFEE-driven workload stack management.

Decentralized DRL Formulation

DECOFFEE formulates workload placement as a collection of parallel Markov Decision Processes (MDPs), one per EA, operating under local and partial global observability. Each agent’s state space encapsulates workload attributes, queue lengths, and LSTM-predicted future load. The action space spans local execution, vertical (CA) offloading, and horizontal offloading to any connected EA. The cost function jointly penalizes execution delay, energy consumption, and workload drops via deadline violations, allowing flexible weighting between latency and energy metrics.

Figure 3: Delay profiles for local computation, horizontal offloading, and vertical offloading, annotated with constituent latency components.

DECOFFEE leverages a Double Dueling Deep Q-Network (DQN) architecture with an integrated LSTM-based forecasting module for proactive load prediction, counteracting suboptimality from delayed feedback and partial observability. State vectors are continuously enriched with short-term LSTM predictions, supporting anticipatory placement actions.

Figure 4: DRL interaction cycle describing EA-agent state composition, decision loop, and telemetry fusion.

Figure 5: DECOFFEE agent architecture combining LSTM forecasting and Double Dueling DQN for policy optimization.

Training Protocol and Computational Complexity

The DECOFFEE agents use episodic training, leveraging delayed reward assignment based on actual workload completion, experience replay buffers, and double Q-learning with periodic target network synchronization. Complexity analysis confirms efficient inference suitability: lightweight feed-forward DQN evaluation at decision time and moderate batch-wise backpropagation during training.

Numerical Evaluation and Empirical Results

Extensive simulation demonstrates DECOFFEE’s performance in diverse environments. The selection and tuning of DRL hyperparameters—learning rate, discount factor, and delay/energy trade-off weights—critically affect convergence speed and final agent performance.

Sensitivity analyses highlight robust behavior across variable workload arrival rates, scaling agent counts, CPU resource allocations, and communication link bandwidths. With growing traffic or agent count, DECOFFEE maintains low drop rates and competitive energy consumption when appropriately weighted for latency or energy awareness.

Figure 6: Edge-Cloud topology given by connectivity matrix $\boldsymbol{G}$ with dense and sparse agent configurations.

Figure 7: DECOFFEE performance for drop rate and energy under varied workload arrival probabilities and delay/energy weighting.

Figure 8: Drop rate and energy consumption trends as agent count and awareness coefficients increase.

Figure 9: Effects of CPU capacity and awareness weights on average delay and energy for executed workloads.

Figure 10: Distribution of placement decisions as delay/energy weighting varies under sparse and dense workload traffic.

Impact of LSTM Forecasting

Nominal inclusion of LSTM forecasts yields significant cost reduction and lower drop rates in regimes with moderate-to-high traffic, with diminished returns under extreme load or light traffic. Forecasting enables agents to anticipate congestion and strategically adjust placement actions.

Figure 11: Average cost and drop rate CDFs contrasting DECOFFEE with and without LSTM under varying traffic.

Comparative Analysis

DECOFFEE variants (delay-aware, energy-aware, balanced) are benchmarked against rule-based schemes (random, local-only, cloud-only, edge-only, round-robin) and delay-heursitic baselines (MLEO). Strong empirical results show DECOFFEE consistently achieves lower execution delays, reduced energy consumption, and robust drop rate performance, especially for balanced and delay-aware configurations. Under aggressive workload/time constraints, DA-DECOFFEE produces minimal drop rates outperforming even advanced latency estimators.

Figure 12: Delay and energy comparisons across nine workload placement schemes as arrival probability increases.

Figure 13: Drop rates under stricter workload deadlines and varying horizontal bandwidth for nine competitors.

Practical and Theoretical Implications

DECOFFEE’s decentralized, model-free, multi-agent DRL paradigm delivers scalable and adaptive workload placement, avoiding excessive overheads or reliance on full-system knowledge. Integration of LSTM forecasting directly addresses temporal non-stationarity and partial observability, and enables predictive workload orchestration. The framework adapts dynamically to heterogeneous environments, scaling with agent count, handling variable latency and energy requirements, and providing flexible trade-off tuning.

On the theoretical front, DECOFFEE’s formulation sets a precedent for decentralized multi-objective optimization in the computing continuum, reframing global placement problems into parallel factored MDPs with weak coupling through telemetry and forecasting.

Future Directions

There are several natural extensions: (1) federated or collaborative multi-agent learning for knowledge sharing and privacy; (2) support for differentiated workload classes with heterogenous QoS demands; (3) inclusion of network-level congestion/comms reliability and carbon-aware metrics; (4) validation on real-world Edge-Cloud testbeds and integration with orchestration frameworks for operational deployment.

Conclusion

DECOFFEE enables autonomous, decentralized, and predictive workload placement across heterogeneous computing continuum environments, consistently optimizing delay, energy, and reliability metrics. Its empirical superiority and architectural flexibility position it as a generalizable solution for future Edge-Cloud infrastructures facing dynamic, time-critical workload orchestration requirements (2604.24507).