Optimizing Co-flows Scheduling and Routing in Data Centre Networks for Big Data Applications

Abstract: This paper optimizes the scheduling and routing of the co-flows of MapReduce shuffling phase in state-of-the-art and proposed Passive Optical Networking (PON)-based Data Centre Network (DCN) architectures. A time-slotted Mixed Integer Linear Programming (MILP) model is developed and used for the optimization with the objective of minimizing either the total energy consumption or the completion time. The DCN architectures include four state-of-the-art electronic switching architectures which are spine-leaf, Fat-tree, BCube, and DCell data centres. The proposed PON-based DCN architectures include two designs that utilize ports in Optical Line Terminal (OLT) line cards for inter and possibly intra data centre networking in addition to passive interconnects for the intra data centre networking between different PON groups (i.e. racks) within a PON cell (i.e. number of PON groups connected to a single OLT port). The first design is a switch-centric design that uses two Arrayed Waveguide Grating Routers (AWGRs) and the second is a server-centric design. The study also considers different traffic patterns defined according to the distribution of map and reduce tasks in the servers and data skewness.