标题: | BigExplorer: A Configuration Recommendation System for Big Data Platform |
作者: | Yeh, Chao-Chun Zhou, Jiazheng Chang, Sheng-An Lin, Xuan-Yi Sun, Yichiao Huang, Shih-Kun 資訊工程學系 Department of Computer Science |
关键字: | big data platform;machine learning;configuration optimization |
公开日期: | 1-一月-2016 |
摘要: | With the complexity big data platform architectures, data engineer provides the infrastructure with computation and storage resource for data scientist and data analyst. With those supports, data scientists can focus their domain problem and design the intelligence module (e.g., prepare the data, select/train/tune the machine learning modules and validate the result). However, there is still a gap between system engineer team and data scientists/engineers team. For system engineers, they don't have any knowledge about the application domain and the propose of the analytic program. For data scientists/engineers, they don't know the configuration of the computation system, file system and database. Some application performance issues are related with system configurations. Data scientist and data engineer do not have information and knowledge about the system properties. In this paper, we propose a configuration layer with the current big data platform (i.e., Hadoop) and build a configuration recommendation system to collect data, pre-process data. Based on the processed data, we use semi-automatic feature engineer to provide features for data engineers and build the performance model with three different machine learning algorithms (i.e., random forest, gradient boosting machine and support vector regression). With the same two benchmarks (i.e., wordcount and terasort), our recommended configuration archives remarkable improvement than rule of thumb configuration and better than their improvements. |
URI: | http://hdl.handle.net/11536/146723 |
ISSN: | 2376-6816 |
期刊: | 2016 CONFERENCE ON TECHNOLOGIES AND APPLICATIONS OF ARTIFICIAL INTELLIGENCE (TAAI) |
起始页: | 228 |
结束页: | 234 |
显示于类别: | Conferences Paper |