A Fundamental Tradeoff between Computation and Communication in Distributed Computing (1604.07086v2)

Abstract: How can we optimally trade extra computing power to reduce the communication load in distributed computing? We answer this question by characterizing a fundamental tradeoff between computation and communication in distributed computing, i.e., the two are inversely proportional to each other. More specifically, a general distributed computing framework, motivated by commonly used structures like MapReduce, is considered, where the overall computation is decomposed into computing a set of "Map" and "Reduce" functions distributedly across multiple computing nodes. A coded scheme, named "Coded Distributed Computing" (CDC), is proposed to demonstrate that increasing the computation load of the Map functions by a factor of $r$ (i.e., evaluating each function at $r$ carefully chosen nodes) can create novel coding opportunities that reduce the communication load by the same factor. An information-theoretic lower bound on the communication load is also provided, which matches the communication load achieved by the CDC scheme. As a result, the optimal computation-communication tradeoff in distributed computing is exactly characterized. Finally, the coding techniques of CDC is applied to the Hadoop TeraSort benchmark to develop a novel CodedTeraSort algorithm, which is empirically demonstrated to speed up the overall job execution by $1.97\times$ - $3.39\times$, for typical settings of interest.

• The paper introduces the Coded Distributed Computing (CDC) scheme, showing an inverse relationship between computation load and communication load.
• The authors derive an information-theoretic lower bound that exactly matches the CDC performance, rigorously characterizing the optimal tradeoff.
• Real-world benchmarks like Hadoop TeraSort demonstrate CDC’s value, achieving speedups up to 3.39 times over conventional methods.

Computation-Communication Tradeoff in Distributed Computing

This paper introduces a comprehensive framework to address a pivotal challenge in distributed computing: optimizing the tradeoff between computation and communication. The authors consider a distributed computing scenario motivated by structures like MapReduce, where the computation of certain functions is divided across multiple nodes.

Key Contributions

1. Coded Distributed Computing (CDC):
   • The paper proposes a novel CDC scheme, showing that the computation load and communication load are inversely proportional. By increasing the computation load during the Map phase by a factor $r$, the communication load during the Shuffle phase can be reduced by the same factor.
2. Optimal Tradeoff Characterization:
   • An information-theoretic lower bound for the communication load, which matches the CDC scheme's achieved load, is introduced. This rigorously characterizes the optimal tradeoff between computation and communication.
3. Application to Real-World Frameworks:
   • The paper examines CDC’s application to the Hadoop TeraSort benchmark, demonstrating significant empirical gains. The proposed CodedTeraSort algorithm achieves a speedup ranging from 1.97 to 3.39 times over conventional TeraSort in typical settings.

Numerical and Theoretical Insights

• Numerical Results:
  • The paper includes a numerical evaluation of the communication loads for various scenarios, comparing the results to uncoded methods. The results showcase a marked decrease in communication load due to CDC, emphasizing its scalability as the network size grows.
• Converse Proofs:
  • Tight converse proofs are presented, ensuring that the CDC scheme not only achieves but defines the lower bound of communication load, validating the theoretical framework.

Practical and Theoretical Implications

• Enhanced Efficiency:
  • For data-intensive applications, reducing communication overhead is crucial. CDC offers a framework to leverage increased computational power for reduced communication, optimizing overall runtime.
• Scalable Architecture Design:
  • With increasing network sizes and distributed nodes, CDC holds potential for broader implementation, ensuring scalability without compromising performance.
• Future Developments:
  • The research opens avenues for further exploration in heterogeneous networks, optimization with straggling nodes, and multi-stage computation tasks. Extending coding strategies to edge and fog computing can leverage distributed resources more efficiently.

Conclusion

The authors provide an exemplary confluence of theory and application, presenting Coded Distributed Computing as an optimal paradigm for distributed computational frameworks. By precisely delineating the computation-communication tradeoff, this research lays a foundation for future explorations into more efficient, scalable, and practical distributed computing systems.