Loading...

Total Cost Reduction for Big Data Processing by Smart VM Storage Selection and Data Placement

Mousavi, Hossein | 2016

1362 Viewed
  1. Type of Document: M.Sc. Thesis
  2. Language: Farsi
  3. Document No: 48873 (19)
  4. University: Sharif University of Technology
  5. Department: Computer Engineering
  6. Advisor(s): Goudarzi, Maziyar
  7. Abstract:
  8. The fastest growing of data in the fields of economic, cultural and scientific quick analysis of this data is one of the needs of users. The inability of the traditional programming model for processing large data makes MapReduce programming model to be introduced. MapReduce model, processing big data in shorter time available. Users to analyze large data processing, rent necessary resources from infrastructure of cloud as one of the most popular are provided. Due to cloud resources acquired and run-time program on selected configurations, the cost of these resources are paid according to their use. So if resources are not proportionate with the needs of the program will be wasting money and time. Allocating resources intelligently makes choosing Rated Resources and thus reduce costs due to the limitations of the user. In this research we intend to store data in the form of intelligent on the hard disks(HDDs) and solid state drives(SSDs). In this way, taking into account the user constraint such as deadline of the program execution, reduce the cost of rented resources. To select the resources that are provided in the form of a virtual machine, the machine’s storage like hard disk and solid state disk will be chosen. Due to the different behavior of each task and accessing data, input data of the tasks stored on suitable virtual machine. This makes the use of virtual machines is reduced and eventually lowered the cost of renting them. Another advantage of using cloud environments that enables dynamic change by more or less the resources to virtual machines. So as to reduce the cost of resources can be dynamically changed during program execution. According to the capabilities of the cloud and big data analysis tasks in a job number, type and data-placment for virtual machines is determined to reduce the final cost. Now, with the initial analysis and matching model using a heuristic algorithm, the tasks of each disk is detected. Finally, with respect to the the execution of tasks and deadlines user, for each task select low-cost virtual machine. The results show that heuristic algorithms with an average error of 7.6% compared to the optimal mode reduces cost and 19.5% better than non-intelligent algorithm
  9. Keywords:
  10. Cloud Computing ; Map Reduce Processing ; Solid State Disk Drive ; Virtual Machine ; Big Data Proccessing

 Digital Object List

 Bookmark

...see more