标题: BigExplorer: A Configuration Recommendation System for Big Data Platform
作者: Yeh, Chao-Chun
Zhou, Jiazheng
Chang, Sheng-An
Lin, Xuan-Yi
Sun, Yichiao
Huang, Shih-Kun
資訊工程學系
Department of Computer Science
关键字: big data platform;machine learning;configuration optimization
公开日期: 1-一月-2016
摘要: With the complexity big data platform architectures, data engineer provides the infrastructure with computation and storage resource for data scientist and data analyst. With those supports, data scientists can focus their domain problem and design the intelligence module (e.g., prepare the data, select/train/tune the machine learning modules and validate the result). However, there is still a gap between system engineer team and data scientists/engineers team. For system engineers, they don't have any knowledge about the application domain and the propose of the analytic program. For data scientists/engineers, they don't know the configuration of the computation system, file system and database. Some application performance issues are related with system configurations. Data scientist and data engineer do not have information and knowledge about the system properties. In this paper, we propose a configuration layer with the current big data platform (i.e., Hadoop) and build a configuration recommendation system to collect data, pre-process data. Based on the processed data, we use semi-automatic feature engineer to provide features for data engineers and build the performance model with three different machine learning algorithms (i.e., random forest, gradient boosting machine and support vector regression). With the same two benchmarks (i.e., wordcount and terasort), our recommended configuration archives remarkable improvement than rule of thumb configuration and better than their improvements.
URI: http://hdl.handle.net/11536/146723
ISSN: 2376-6816
期刊: 2016 CONFERENCE ON TECHNOLOGIES AND APPLICATIONS OF ARTIFICIAL INTELLIGENCE (TAAI)
起始页: 228
结束页: 234
显示于类别:Conferences Paper