Efficient and fair: Information-agnostic online coflow scheduling by combining limited multiplexing with DRL

In shared data center networks, communications among users can be modeled as coflows, each comprising of a group of parallel data transmission flows. Efficient and fair scheduling of coflows is critical for improving both system performance and user satisfaction at the application level. Existing coflow scheduling methods maximizing efficiency (coflow completion time, CCT) and fairness (service isolation) simultaneously require prior knowledge of coflow (flow) size that is however not known before completion of coflow execution in reality, which limits their applicability. For information-agnostic scheduling, known results focus either solely on efficiency or fairness, but not both due to the hardness of achieving the desired compromise between them. In this paper, we first present an information-aware non-preemptive coflow scheduling algorithm, and show its provable long-term isolation guarantee under reasonable assumptions. We then adapt this algorithm to information-agnostic online coflow scheduling by combining limited multiplexing with Deep Reinforcement Learning (DRL) framework to achieve long-term isolation guarantee toward fair network sharing and lower average weighted CCT simultaneously. The simulation results show that our algorithm outperforms the state-of-the-art results of both fairness-optimal scheduling (NC-DRF) by 4.92in terms of average weighted CCT and performance-optimal scheduling (Aalo) in the metric of maximum normalized CCT. This fully demonstrates the superiority of our method in simultaneous optimization of efficiency and fairness for information-agnostic coflow scheduling.