改善ETSI M2M架構下高速與大量資料的處理

Abstract

在ETSI的標準裡，每筆從感測器收集到的資料，在它們被處理之前都會先存於平台的資料庫裡頭。換言之，那些收集到的資訊不只會佔用網路的頻寬，也同時會持有部分的平台儲存資源。此外，在ETSI裡，那些感測數據並不只是像傳統的SQL架構的方式做儲存，而是以更複雜的資源樹的架構做儲存，用來描述層級式的屬性以達成更好的資料管理；也因此需要夠多的資源來管理這樣的資料結構。然而，那些收集到的資料，對於應用來說，並不是所有的資訊都是有用的；事實上，有些資訊其實是多餘且累贅的，而不應分配過多的資源予它們。 在此論文研究裡，我們提出先行處理這些串流資訊，接著決定資料的保存與否。主要的理念是根據資料間不同的本質去進行全然不同地處理。像是，在最一開始，我們可以試著過濾出多餘、無用或是錯誤的資訊；而對於那些不全然有用的資訊，在它們被送入伺服器之前，我們應小心地精簡化，並取出主要的數據加以儲存和更進一步地處理。至於那些十分重要的資訊，我們應馬上處理它們且盡快地採取相對應的行動措施，然後如果有需要則必須完整地將它們儲存在資料庫裡。 藉由過濾及先行處理那些資訊，可節省大量的資源。透過減少物聯網感測器所收集到的資訊其資料傳輸、資料儲存、資料管理及處理成本，進而達到增進物聯網平台的效率及高度使用率的目標，而不浪費資源成本在無用的資料上。 此研究將會著重於如何使得在ETSI M2M標準兼容的物聯網平台 ─ OpenMTC裡進行高速且大量的資料處理。為了比較我們與傳統在處理資料的方法上之差異，將會使用工廠管理作為展示在成本和效率方面的結果範例。在成本的分析上，我們將會測量各方法所需的儲存空間與資料傳輸量。而對於效能分析，我們將會觀察兩種方法在中央處理器與記憶體上使用量的不同之處。我們將會展示我們處理資料的方法可以大幅地改善在ETSI M2M架構裡對於高速且大量的資料處理。

In the ETSI standard, each data collected by sensors would be stored in database before they are processed. In other words, those collecting data will not only occupy bandwidth of the internet, but also hold some storage resources in the platform. Furthermore, in ETSI the sensing data would not be stored like traditional format as SQL-like structure, but become more complicate one - resource tree which describes hierarchical attributes for better data management. Consequently, it takes more efforts to manage IoT data stored in the IoT platform. However, for those collected data, not all of them are really useful for the applications. Some of them, actually, are redundant and doesn’t need to be allocated too much resource. In this thesis research, we propose to process streaming data first then determine whether the data should be kept or not. The idea is to treat data differently based on their different nature. For example, we can filter redundant, useless and fallacious data out in the very beginning. Then for those not completely useful data, we can refine them carefully and take key values out for storage and further processing before they are sent to the server. As for important data, we can process them and take immediate action as soon as possible, then fully store them in the database if needed. By filtering and pre-processing those data, it can save lots of resources by reducing data transmission, data storage and data management and processing overhead for those data collected by IoT sensors, the goal is to make an IoT platform more efficient and highly utilized without wasting the resources on useless data. This research will focus on how to enable big velocity and large volume data processing in an ETSI M2M standard compliant IoT platform - OpenMTC. To compare the differences between our approach and traditional approach of handling data, a use case from factory management will be used to demonstrate the results in terms of cost and efficiency. For the cost analysis, we will measure the storage space and the data transmission volume required for each approach. For the efficiency analysis, we will observe the difference between these two approaches in terms of their cpu and memory usage. We are going to demonstrate our approach of handling data can largely improve big velocity and large volume data processing in an ETSI M2M architecture.