Loading...

Data locality and VM interference aware mitigation of data skew in hadoop leveraging modern portfolio theory

Nabavinejad, S. M ; Sharif University of Technology | 2018

434 Viewed
  1. Type of Document: Article
  2. DOI: 10.1145/3167132.3167150
  3. Publisher: Association for Computing Machinery , 2018
  4. Abstract:
  5. Data skew, which is the result of uneven distribution of data among tasks in big data processing frameworks such as MapReduce, causes significant variation in the execution time of tasks and makes their placement on computing resources more challenging. Moreover, with the proliferation of big data processing in the cloud, the interference among virtual machines co-located on the same physical machine exacerbates the aforementioned variation. To tackle this challenge, we propose Locality and Interference aware Portfolio-based Task Assignment (LIPTA) approach. LIPTA leverages the modern portfolio theory to mitigate the variation in execution time of tasks while considering the interference of virtual machines and locality of input data. It selects and assigns groups of tasks (the portfolio) to each machine such that variation of their total execution time is reduced due to portfolio effect. Experimental results using real-world workload logs demonstrate the effectiveness of our LIPTA approach. It can reduce the total execution time of workloads by up to 46.7% compared with several variation-oblivious approaches. © 2018 ACM
  6. Keywords:
  7. Data skew ; Portfolio theory ; Cloud computing ; Computation theory ; Network security ; Virtual machine ; Computing resource ; Data locality ; Execution time ; Interference-aware ; Modern portfolio theories ; Portfolio theories ; Task assignment ; Big data
  8. Source: 33rd Annual ACM Symposium on Applied Computing, SAC 2018, 9 April 2018 through 13 April 2018 ; 2018 , Pages 175-182 ; 9781450351911 (ISBN)
  9. URL: https://dl.acm.org/citation.cfm?id=3167150