針對 Hadoop 在嵌入式異質多核心平台之低功耗設計流程

Abstract

Hadoop是一種廣泛採用的分佈式處理架構。 Hadoop的設計方向是針對每個計算節點由傳統CPU配上記憶體架構的系統組成,且具有良好的跨平台性。不過,這樣的架構不適合在嵌入式異質多核心平台上有效率的利用。主要的挑戰是由於Hadoop的環境和嵌入式異質系統之間資料的收集和管理方式不同。為了解決上述的問題,本文提出了一種工作流程,使Hadoop的應用程序,能夠有效地利用分佈式嵌入式異質多核心系統。在不修改Hadoop資料儲存規則的情況下,能夠讓異質多核心平台更有效率的計算,以高效率的方式來收集和管理細粒度的資料。使用主成分分析(PCA)作為範例,結合所提出的設計方法在Tegra K1集群上執行資料大小16K×16K的矩陣,進而提升了6.4倍的效能,並證明出嵌入式異質多核心群集的功耗效率比相較傳統的PC集群更高。

Hadoop is a widely adopted distributed processing framework. The Hadoop framework achieves good portability by assuming each computing node a conventional CPU-based system with local memory. However, the current flow of this framework cannot effectively take full advantage of an embedded heterogeneous many-core platform. The main challenge stems from the mismatch of data collection and management paradigms between the Hadoop environment and embedded heterogeneous systems. To address the above design concerns, this paper proposes a workflow that enables Hadoop applications to efficiently leverage the distributed embedded heterogeneous many-core systems. By taking the same data layout of conventional Hadoop applications, the proposed flow introduces efficient manners to collect and manage the fine-grained data chunks. Using Principle Component Analysis (PCA) as an application driver, the proposed Hadoop design on a Tegra K1 cluster has achieved 6.4x performance enhancement when running the PCA analysis on an input matrix of 16K × 16K data. The proposed design also demonstrated much better energy efficiency when compared with the designs on conventional PC-based clusters.