Browsing by Author "Lin, Jia-Chun"

Now showing 1 - 9 of 9

Analyzing job completion reliability and job energy consumption for a heterogeneous MapReduce cluster under different intermediate-data replication policies
(2015-05-01)
Lin, Jia-Chun
;
Leu, Fang-Yie
;
Chen, Ying-Ping
;
資訊工程學系
;
Department of Computer Science
Recently, MapReduce has been a popular distributed programming framework for solving data-intensive applications. However, a large-scale MapReduce cluster has inevitable machine/node failures and considerable energy consumption. To solve these problems, MapReduce has employed several policies for replicating input data, storing/replicating intermediate data, and re-executing failed tasks. In this study, we concentrate on two typical policies for storing/replicating intermediate data, and derive the job completion reliability (JCR for short) and job energy consumption (JEC for short) of a MapReduce cluster when the two policies are individually employed. The two policies are further analyzed and compared given various scenarios in which jobs with different input data sizes, numbers of reduce tasks, and other parameters are run in a MapReduce cluster with two extreme parallel execution capabilities. From the analytical results, MapReduce managers are able to comprehend how the two policies influence the JCR and JEC of a MapReduce cluster.
Deriving Job Completion Reliability and Job Energy Consumption for a General MapReduce Infrastructure from Single-Job Perspective
(2013)
Lin, Jia-Chun
;
Leu, Fang-Yie
;
Lee, Ming-Chang
;
Chen, Ying-ping
;
資訊工程學系
MapReduce as a master-slave infrastructure consists of two master-side severs and a large number of slave-side working nodes. In this paper, we derive a job completion reliability (JCR for short) model from a single-job perspective for a general MapReduce infrastructure in which no redundancy scheme is adopted on the master side, and a cold-standby scheme is employed on the slave side. Without loss of generality, the JCR model is derived based on a Poisson distribution. In addition, we calculate the corresponding job energy consumption (JEC for short). Through the simulation and analytical results, MapReduce managers and service providers can comprehend how this infrastructure behaves and how to improve the infrastructure so as to achieve a more reliable and energy-efficient MapReduce environment.
Hybrid Job-Driven Scheduling for Virtual MapReduce Clusters
(2016-06)
Lee, Ming-Chang
;
Lin, Jia-Chun
;
Yahyapour, Ramin
;
資訊工程學系
;
Department of Computer Science
It is cost-efficient for a tenant with a limited budget to establish a virtual MapReduce cluster by renting multiple virtual private servers (VPSs) from a VPS provider. To provide an appropriate scheduling scheme for this type of computing environment, we propose in this paper a hybrid job-driven scheduling scheme (JoSS for short) from a tenant\'s perspective. JoSS provides not only job-level scheduling, but also map-task level scheduling and reduce-task level scheduling. JoSS classifies MapReduce jobs based on job scale and job type and designs an appropriate scheduling policy to schedule each class of jobs. The goal is to improve data locality for both map tasks and reduce tasks, avoid job starvation, and improve job execution performance. Two variations of JoSS are further introduced to separately achieve a better map-data locality and a faster task assignment. We conduct extensive experiments to evaluate and compare the two variations with current scheduling algorithms supported by Hadoop. The results show that the two variations outperform the other tested algorithms in terms of map-data locality, reduce-data locality, and network overhead without incurring significant overhead. In addition, the two variations are separately suitable for different MapReduce-workload scenarios and provide the best job performance among all tested algorithms.
Impact of MapReduce Policies on Job Completion Reliability and Job Energy Consumption
(2015-05-01)
Lin, Jia-Chun
;
Leu, Fang-Yie
;
Chen, Ying-ping
;
資訊工程學系
;
Department of Computer Science
Recently, MapReduce has been widely employed by many companies/organizations to tackle data-intensive problems over a large-scale MapReduce cluster. To solve machine/node failure which is inevitable in a MapReduce cluster, MapReduce employs several policies, such as input-data replication and intermediate-data replication policies. To speed up job execution, MapReduce allows reduce tasks to early fetch their required intermediate data. However, the impact of these policy combinations on the job completion reliability (JCR for short) and job energy consumption (JEC for short) of a MapReduce cluster was not clear, where JCR is the reliability with which a MapReduce job can be completed by the cluster, whereas JEC is the energy consumed by the cluster to complete the job. Therefore, in this study, we analyze the JCR and JEC of a MapReduce cluster on four policy combinations (POCs for short) derived from two typical intermediate-data replication policies and two typical reduce-task assignment policies. The four POCs are further compared in extensive scenarios, which not only consider jobs at different scales with various parameters, but also give a MapReduce cluster two extreme parallel execution capabilities and diverse bandwidths. The analytical results enable MapReduce managers to comprehend how these POCs impact the JCR and JEC of a cluster and then select an appropriate POC based on the characteristics of their own MapReduce jobs and clusters.
Impact of MapReduce Task Re-execution Policy on Job Completion Reliability and Job Completion Time
(2014-01-01)
Lin, Jia-Chun
;
Leu, Fang-Yie
;
Chen, Ying-ping
;
Munawar, Waqaas
;
資訊工程學系
MapReduce has been a worldwide accepted framework for solving data-intensive applications. To prevent MapReduce jobs from being interrupted by node failures which occur frequently in a large-scale MapReduce cluster, current MapReduce implementations, e.g., Hadoop, employ a task re-execution policy (TR policy for short) for MapReduce jobs, i.e., when a map/reduce task of a job fails due to node failure, this policy reperforms the task on another node. However, the impact of the TR policy on job completion reliability and job completion time have not been studied from a theoretical viewpoint, especially when the job is given different characteristics, e.g., different input data sizes, different numbers of reduce tasks, and different intermediate data sizes. In this study, we derive the job completion reliability (JCR for short) of a MapReduce job based on Poisson distributions and analyze the expected job completion time (JCT for short) based on the universal generation function. We use nine settings of task re-execution factor (TR factor for short) to explore the impact of the TR policy on the JCR and JCT of jobs. The results show that the TR policy can effectively improve JCR without significantly prolonging JCT. But there is no single TR factor with which all jobs can achieve a high JCR.
Impacts of Task Re-Execution Policy on MapReduce Jobs
(2016-05)
Lin, Jia-Chun
;
Leu, Fang-Yie
;
Chen, Ying-ping
;
資訊工程學系
;
Department of Computer Science
MapReduce is a popular distributed programming framework for large-scale data processing. To prevent MapReduce jobs from being interrupted by node failures that occur frequently in a MapReduce cluster consisting of a set of commodity machines/nodes, the most well-known MapReduce implementation, i.e. Hadoop, adopts a task re-execution policy (TR policy). When a map/reduce task of a job crashes, the TR policy assigns another node to reperform the task. However, the impact of the TR policy on MapReduce jobs in terms of reliability, job turnaround time (JTT) and energy consumption are not clear, particularly when jobs have different features, e.g. different filtering percentages, different input-data sizes, and different numbers of reduce tasks. In this paper, we formally analyze the job completion reliability (JCR) of a job based on Poisson distributions, and then derive the expected JTT and job energy consumption (JEC) based on the universal generation function. Extensive analyses are further conducted to explore the impact of the TR policy on JCR, JTT and JEC of jobs with different features. The results show that employing the TR policy can dramatically improve JCR for a large MapReduce job. Moreover, if the JCR of a job is highly improved by the TR policy, the expected JTT and JEC will not be significantly prolonged and increased, respectively.
PAReS: A Proactive and Adaptive Redundant System for MapReduce
(2015-09-01)
Lin, Jia-Chun
;
Leu, Fang-Yie
;
Chen, Ying-Ping
;
資訊工程學系
;
Department of Computer Science
Recently, MapReduce has been a key and popular technology for tackling data-intensive applications. But its two master servers in current MapReduce implementations have a single-point-of-failure problem, which may interrupt MapReduce operations and filesystem services. In this paper, we propose a hybrid takeover scheme, called PAReS (Proactive and Adaptive Redundant System). To further improve the master servers\' service qualities, PAReS also employs three service-quality improvement mechanisms, including a proactive synchronization and replication method, a mutual monitoring algorithm, and an adaptive warm-up mechanism. The extensive experiments show that PAReS enhances service quality at acceptable energy consumption level and synchronization cost as compared with four state-of-the-art schemes.
Proactive Distributed Digital Evidence Preservation with Tamper Resistance, Perfect Secrecy and High Survivability
(2014-11-01)
Lin, Jia-Chun
;
Shieh, Shiuhpyng
;
Hsu, Chia-Wei
;
資訊工程學系
;
Department of Computer Science
To reduce the chance of being compromised, digital evidence must be preserved in a secure way when it is generated. The preservation must assure confidentiality, integrity, and survivability of the digital evidence. Some generic storage systems have been proposed. However, they cannot meet the critical requirements for digital evidence preservation. This paper introduces a secure distributed digital evidence preservation system (DEPS) which can preserve digital evidence generated by mission critical servers. With the novel and lightweight (n, n, n-1) secret recovery scheme we proposed, DEPS can efficiently divide each block of a digital evidence file into n shares, compress the first n-1 shares, and then construct a share-message with a sequence number and a fingerprint for each share. Only all the n shares combined together suffice for recovering the block. An adversary can neither obtain any information about the original block from a share-message, nor can he modify digital evidence without being discovered. By using intelligent replication, DEPS can dynamically duplicate and distribute each share-message to a set of distributed repositories to achieve high survivability. The security and performance analyses showed that DEPS is adequate for preserving digital evidence.
Scheduling MapReduce Tasks on Virtual MapReduce Clusters from a Tenant\'s Perspective
(2014)
Lin, Jia-Chun
;
Lee, Ming-Chang
;
Yahyapour, Ramin
;
資訊工程學系
;
Department of Computer Science
Renting a set of virtual private servers (VPSs for short) from a VPS provider to establish a virtual MapReduce cluster is cost-efficient for a company/organization. To shorten job turnaround time and keep data locality as high as possible in this type of environment, this paper proposes a Best-Fit Task Scheduling scheme (BFTS for short) from a tenant\'s perspective. BFTS schedules each map task to a VPS that can finish the task earlier than the other VPSs by predicting and comparing the time required by every VPS to retrieve the map-input data, executeTS sch the map task, and become idle in an online manner. Furthermore, BFedules each reduce task to a VPS that is close to most VPSs that execute the related map tasks. We conduct extensive experiments to compare BFTS with several scheduling algorithms employed by Hadoop. The experimental results show that BFTS is better than the other tested algorithms in terms of map-data locality, reduce-data locality, and job turnaround time. The overhead incurred by BFTS is also evaluated, which is inevitable but acceptable compared with the other algorithms.