Machine Learning-Based Configuration Parameter Tuning on Hadoop System

Abstract

Apache Hadoop system is a software framework with the capability to process large-scale datasets across a cluster of distributed machines using MapReduce programming model. However, there are two main challenges for system administrators to manage the Hadoop system; (1) system administrators are difficult to tune the parameters appropriately since the behaviors and characteristics of large-scale distributed systems are too complicated; (2) there are dozens of configuration parameters affecting the system performance which makes the configuration parameters tuning task becomes troublesome. In this paper, we focus on optimizing the Hadoop MapReduce job performance by tuning configuration parameters, and then we propose an analytical method to help system administrators choose approximately optimal configuration parameters depending on the characteristics of each application. Our approach has two key phases: prediction and optimization phase. The prediction phase is to estimate the performance of a MapReduce job, whereas the optimization phase is to search the approximately optimal configuration parameters strategically by invoking the predictor repeatedly. In our evaluation results, our work can help system administrators to improve the performance about 2X to 8X better than traditional methods.