使用Hadoop實作之雲端監控系統

Abstract

隨著科技的進步，監控系統越來越普及化，預計在未來每個角落都會有IP camera的架設，但是這代表著會有更大量的影像必須被儲存，且監控系統的架構會變的更龐大；然而，目前現有的傳統監控系統無法處理大量的資料，如果使用NVR系統或許可以處理大量的資料，但是如果透過NVR，每新增一個NVR都必須透過手動的方式一個一個增加，代表新增NVR的擴充動作並不容易，這又需增加管理上的工作量跟成本；此外，影像儲存的工作可能有時會因為系統當機或停電的例外情形而中斷，造成影像儲存的動作將無法繼續進行，導致遺失重要片段，對於此情形，目前亦沒有一個很好的方法能夠處理。

因此，我們這次研究的目的，除了要利用Hadoop的特性來管理大量的資料和擴充儲存節點，並且要提供一個備援機制來處理影像儲存發生例外中斷的情形。

Hadoop可將影像分散儲存在幾個data node，且這些影像透過Hadoop都可以很容易取得，Hadoop也容易新增data node，這可以解決我們前面提到NVR新增的困難，另外Hadoop擁有中央叢集的特性，龐大的系統透過這個特性來管理，即使結合我們的備援機制也不至於讓整個監控系統太過複雜，我們可以讓中央節點來分配備援節點或者指定備援節點來接手影像儲存的工作。

經過我們實作的結果，我們發現讓影像儲存在Hadoop是可行的，且data node擴充容易，因此可隨時增加影像儲存的動作，這樣可以很容易儲存大量的影像，而另外我們的備援機制在系統產生例外情形時，接手儲存的動作，不但控管容易且只會有少量的frame loss。

最後的結論我們證明，使用Hadoop對於大量影像儲存的工作是可行的，而且可以結合備援機制的實行，這樣能確保更完整的連續性影像儲存工作，解決儲存系統問題而造成的遺失片段現象。

Information technology has been widely used to solve many problems in many domains. Many cameras have been installed at the corner of streets for maintaining public order or monitoring traffic status. The recorded video data is stored to the traditional surveillance system or NVR in public department for future inquiry. However, the traditional surveillance system could not process and store large amount of data generated from large number of cameras. Moreover, user must to increase NVR by manual. Thus, scaling up traditional surveillance system increases the cost and complexity of management. In addition, traditional surveillance system is not capable of handling exception break situations such as blackout and system crash while storing streaming data to storage nodes.

Thus, the purpose of this thesis is to propose a cloud-based surveillance system to solve these problems. The proposed system integrates Hadoop to store large amount of streaming and provides a backup mechanism to handle the exception break situations.

By integrating HaDoop File System(HDFS), the approach of this thesis can easily scale up the system for processing corresponding large amount of data generated from cameras. The cluster central of Hadoop is integrated with a novel backup mechanism for handling exception break situations. The evaluation results show that the proposed system can easily process the increasing number of video streams and lost only a few frames while handling the exception break situation.

The result suggested that replacing NVR to the proposed cloud-based surveillance system under Hadoop with our backup mechanism is practicable.