標題: Analyzing job completion reliability and job energy consumption for a heterogeneous MapReduce cluster under different intermediate-data replication policies
作者: Lin, Jia-Chun
Leu, Fang-Yie
Chen, Ying-Ping
資訊工程學系
Department of Computer Science
關鍵字: MapReduce;Hadoop;Job completion reliability;Job energy consumption;Intermediate data;Replication
公開日期: 1-五月-2015
摘要: Recently, MapReduce has been a popular distributed programming framework for solving data-intensive applications. However, a large-scale MapReduce cluster has inevitable machine/node failures and considerable energy consumption. To solve these problems, MapReduce has employed several policies for replicating input data, storing/replicating intermediate data, and re-executing failed tasks. In this study, we concentrate on two typical policies for storing/replicating intermediate data, and derive the job completion reliability (JCR for short) and job energy consumption (JEC for short) of a MapReduce cluster when the two policies are individually employed. The two policies are further analyzed and compared given various scenarios in which jobs with different input data sizes, numbers of reduce tasks, and other parameters are run in a MapReduce cluster with two extreme parallel execution capabilities. From the analytical results, MapReduce managers are able to comprehend how the two policies influence the JCR and JEC of a MapReduce cluster.
URI: http://dx.doi.org/10.1007/s11227-014-1286-7
http://hdl.handle.net/11536/124654
ISSN: 0920-8542
DOI: 10.1007/s11227-014-1286-7
期刊: JOURNAL OF SUPERCOMPUTING
Volume: 71
起始頁: 1657
結束頁: 1677
顯示於類別:期刊論文