標題: | Analyzing job completion reliability and job energy consumption for a heterogeneous MapReduce cluster under different intermediate-data replication policies |
作者: | Lin, Jia-Chun Leu, Fang-Yie Chen, Ying-Ping 資訊工程學系 Department of Computer Science |
關鍵字: | MapReduce;Hadoop;Job completion reliability;Job energy consumption;Intermediate data;Replication |
公開日期: | 1-五月-2015 |
摘要: | Recently, MapReduce has been a popular distributed programming framework for solving data-intensive applications. However, a large-scale MapReduce cluster has inevitable machine/node failures and considerable energy consumption. To solve these problems, MapReduce has employed several policies for replicating input data, storing/replicating intermediate data, and re-executing failed tasks. In this study, we concentrate on two typical policies for storing/replicating intermediate data, and derive the job completion reliability (JCR for short) and job energy consumption (JEC for short) of a MapReduce cluster when the two policies are individually employed. The two policies are further analyzed and compared given various scenarios in which jobs with different input data sizes, numbers of reduce tasks, and other parameters are run in a MapReduce cluster with two extreme parallel execution capabilities. From the analytical results, MapReduce managers are able to comprehend how the two policies influence the JCR and JEC of a MapReduce cluster. |
URI: | http://dx.doi.org/10.1007/s11227-014-1286-7 http://hdl.handle.net/11536/124654 |
ISSN: | 0920-8542 |
DOI: | 10.1007/s11227-014-1286-7 |
期刊: | JOURNAL OF SUPERCOMPUTING |
Volume: | 71 |
起始頁: | 1657 |
結束頁: | 1677 |
顯示於類別: | 期刊論文 |