標題: Impact of MapReduce Task Re-execution Policy on Job Completion Reliability and Job Completion Time
作者: Lin, Jia-Chun
Leu, Fang-Yie
Chen, Ying-ping
Munawar, Waqaas
資訊工程學系
Department of Computer Science
關鍵字: MapReduce;job completion reliability;job completion time;Poisson distribution;universal generation function
公開日期: 1-Jan-2014
摘要: MapReduce has been a worldwide accepted framework for solving data-intensive applications. To prevent MapReduce jobs from being interrupted by node failures which occur frequently in a large-scale MapReduce cluster, current MapReduce implementations, e.g., Hadoop, employ a task re-execution policy (TR policy for short) for MapReduce jobs, i.e., when a map/reduce task of a job fails due to node failure, this policy reperforms the task on another node. However, the impact of the TR policy on job completion reliability and job completion time have not been studied from a theoretical viewpoint, especially when the job is given different characteristics, e.g., different input data sizes, different numbers of reduce tasks, and different intermediate data sizes. In this study, we derive the job completion reliability (JCR for short) of a MapReduce job based on Poisson distributions and analyze the expected job completion time (JCT for short) based on the universal generation function. We use nine settings of task re-execution factor (TR factor for short) to explore the impact of the TR policy on the JCR and JCT of jobs. The results show that the TR policy can effectively improve JCR without significantly prolonging JCT. But there is no single TR factor with which all jobs can achieve a high JCR.
URI: http://dx.doi.org/10.1109/AINA.2014.87
http://hdl.handle.net/11536/128537
ISBN: 978-1-4799-3629-8
ISSN: 1550-445X
DOI: 10.1109/AINA.2014.87
期刊: 2014 IEEE 28TH INTERNATIONAL CONFERENCE ON ADVANCED INFORMATION NETWORKING AND APPLICATIONS (AINA)
起始頁: 712
結束頁: 718
Appears in Collections:Conferences Paper