標題: | Impacts of Task Re-Execution Policy on MapReduce Jobs |
作者: | Lin, Jia-Chun Leu, Fang-Yie Chen, Ying-ping 資訊工程學系 Department of Computer Science |
關鍵字: | MapReduce;job completion reliability;job turnaround time;job energy consumption;Poisson distribution;universal generation function |
公開日期: | 五月-2016 |
摘要: | MapReduce is a popular distributed programming framework for large-scale data processing. To prevent MapReduce jobs from being interrupted by node failures that occur frequently in a MapReduce cluster consisting of a set of commodity machines/nodes, the most well-known MapReduce implementation, i.e. Hadoop, adopts a task re-execution policy (TR policy). When a map/reduce task of a job crashes, the TR policy assigns another node to reperform the task. However, the impact of the TR policy on MapReduce jobs in terms of reliability, job turnaround time (JTT) and energy consumption are not clear, particularly when jobs have different features, e.g. different filtering percentages, different input-data sizes, and different numbers of reduce tasks. In this paper, we formally analyze the job completion reliability (JCR) of a job based on Poisson distributions, and then derive the expected JTT and job energy consumption (JEC) based on the universal generation function. Extensive analyses are further conducted to explore the impact of the TR policy on JCR, JTT and JEC of jobs with different features. The results show that employing the TR policy can dramatically improve JCR for a large MapReduce job. Moreover, if the JCR of a job is highly improved by the TR policy, the expected JTT and JEC will not be significantly prolonged and increased, respectively. |
URI: | http://dx.doi.org/10.1093/comjnl/bxv105 http://hdl.handle.net/11536/133789 |
ISSN: | 0010-4620 |
DOI: | 10.1093/comjnl/bxv105 |
期刊: | COMPUTER JOURNAL |
Volume: | 59 |
Issue: | 5 |
起始頁: | 701 |
結束頁: | 714 |
顯示於類別: | 期刊論文 |