A Hadoop based Web log Analysis System

Abstract

In recent years, cloud computing has been an important issue in the Field of research. Cloud computing employs distributed storage and distributed computing technology to achieve a large number of stored data, as well as fast data analysis and processing. As the rapid development of Internet technology, digital data showing explosive growth, the face of massive data processing, the traditional text software and relational database technology has been facing a bottleneck, presented the results are not very satisfactory. For this problem, the concept of cloud computing is a more appropriate choice. In this paper, based on the architecture of Hadoop with HDFS (Hadoop Distributed File System) and Hadoop NlapRecluce software framework and Pig Latin language, we design and implement an enterprise Web log analysis system. Experimental results, by analyzing daily Web log records, we get Application Server traffic trends, performance of program statistical reports, and performance reports of different intervals and different actions of program by user request. The main purpose of this system is to assist system administrators to quickly capture and analyze data hidden in the massive potential value, thus providing an important basis for business decisions.