標題: Impact of MapReduce Policies on Job Completion Reliability and Job Energy Consumption
作者: Lin, Jia-Chun
Leu, Fang-Yie
Chen, Ying-ping
資訊工程學系
Department of Computer Science
關鍵字: Intermediate-data replication;job completion reliability;job energy consumption;MapReduce
公開日期: 1-May-2015
摘要: Recently, MapReduce has been widely employed by many companies/organizations to tackle data-intensive problems over a large-scale MapReduce cluster. To solve machine/node failure which is inevitable in a MapReduce cluster, MapReduce employs several policies, such as input-data replication and intermediate-data replication policies. To speed up job execution, MapReduce allows reduce tasks to early fetch their required intermediate data. However, the impact of these policy combinations on the job completion reliability (JCR for short) and job energy consumption (JEC for short) of a MapReduce cluster was not clear, where JCR is the reliability with which a MapReduce job can be completed by the cluster, whereas JEC is the energy consumed by the cluster to complete the job. Therefore, in this study, we analyze the JCR and JEC of a MapReduce cluster on four policy combinations (POCs for short) derived from two typical intermediate-data replication policies and two typical reduce-task assignment policies. The four POCs are further compared in extensive scenarios, which not only consider jobs at different scales with various parameters, but also give a MapReduce cluster two extreme parallel execution capabilities and diverse bandwidths. The analytical results enable MapReduce managers to comprehend how these POCs impact the JCR and JEC of a cluster and then select an appropriate POC based on the characteristics of their own MapReduce jobs and clusters.
URI: http://dx.doi.org/10.1109/TPDS.2014.2374600
http://hdl.handle.net/11536/124662
ISSN: 1045-9219
DOI: 10.1109/TPDS.2014.2374600
期刊: IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS
Volume: 26
起始頁: 1364
結束頁: 1378
Appears in Collections:Articles