Loading...

A fundamental tradeoff between computation and communication in distributed computing

Li, S ; Sharif University of Technology | 2018

947 Viewed
  1. Type of Document: Article
  2. DOI: 10.1109/TIT.2017.2756959
  3. Publisher: Institute of Electrical and Electronics Engineers Inc , 2018
  4. Abstract:
  5. How can we optimally trade extra computing power to reduce the communication load in distributed computing? We answer this question by characterizing a fundamental tradeoff between computation and communication in distributed computing, i.e., the two are inversely proportional to each other. More specifically, a general distributed computing framework, motivated by commonly used structures like MapReduce, is considered, where the overall computation is decomposed into computing a set of “Map” and “Reduce” functions distributedly across multiple computing nodes. A coded scheme, named “coded distributed computing” (CDC), is proposed to demonstrate that increasing the computation load of the Map functions by a factor of r (i.e., evaluating each function at r carefully chosen nodes) can create novel coding opportunities that reduce the communication load by the same factor. An information-theoretic lower bound on the communication load is also provided, which matches the communication load achieved by the CDC scheme. As a result, the optimal computation-communication tradeoff in distributed computing is exactly characterized. Finally, the coding techniques of CDC is applied to the Hadoop TeraSort benchmark to develop a novel CodedTeraSort algorithm, which is empirically demonstrated to speed up the overall job execution by 1.97× - 3.39×, for typical settings of interest. © 2017 IEEE
  6. Keywords:
  7. Coded multicasting ; Coded terasort ; Computation-communication tradeoff ; Distributed computing ; MapReduce ; Distributed computer systems ; Electrical engineering ; Electronic mail ; Encoding (symbols) ; Information theory ; Multicasting ; Benchmark testing ; Communication load ; Distributed computing frameworks ; Distributed database ; Information-theoretic lower bounds ; Map-reduce ; Optimal computation ; Multiprocessing systems
  8. Source: IEEE Transactions on Information Theory ; Volume 64, Issue 1 , 2018 , Pages 109-128 ; 00189448 (ISSN)
  9. URL: https://ieeexplore.ieee.org/document/8051074