BigExplorer: A Configuration Recommendation System for Big Data Platform
檔案
本條目沒有附加檔案。
日期
2016-01-01
類型
系所
期刊名稱
2016 CONFERENCE ON TECHNOLOGIES AND APPLICATIONS OF ARTIFICIAL INTELLIGENCE (TAAI)
卷期與頁次
頁 228–234
ISSN
2376-6816
語言
英文
摘要
With the complexity big data platform architectures, data engineer provides the infrastructure with computation and storage resource for data scientist and data analyst. With those supports, data scientists can focus their domain problem and design the intelligence module (e.g., prepare the data, select/train/tune the machine learning modules and validate the result). However, there is still a gap between system engineer team and data scientists/engineers team. For system engineers, they don't have any knowledge about the application domain and the propose of the analytic program. For data scientists/engineers, they don't know the configuration of the computation system, file system and database. Some application performance issues are related with system configurations. Data scientist and data engineer do not have information and knowledge about the system properties. In this paper, we propose a configuration layer with the current big data platform (i.e., Hadoop) and build a configuration recommendation system to collect data, pre-process data. Based on the processed data, we use semi-automatic feature engineer to provide features for data engineers and build the performance model with three different machine learning algorithms (i.e., random forest, gradient boosting machine and support vector regression). With the same two benchmarks (i.e., wordcount and terasort), our recommended configuration archives remarkable improvement than rule of thumb configuration and better than their improvements.
所屬典藏
會議論文引用此條目
Yeh, C.-C., Zhou, J., Chang, S.-A., Lin, X.-Y., Sun, Y., & Huang, S.-K. (2016). BigExplorer: A Configuration Recommendation System for Big Data Platform. 2016 CONFERENCE ON TECHNOLOGIES AND APPLICATIONS OF ARTIFICIAL INTELLIGENCE (TAAI), 228–234. https://ir.lib.nycu.edu.tw/handle/11536/146723
引用文字依 APA 第 7 版格式自動產生,使用前請再核對。
分享
永久網址:https://ir.lib.nycu.edu.tw/handle/11536/146723