使用統計法最佳化快取來降低網路傳輸

Abstract

防毒軟體公司使用雲端科技- SPN(Smart Protection Network) 雲端截毒系統來做檔案掃毒。當使用這套系統的客戶越多，傳送及接收到雲端主機的資料也會越多，很容易產生巨大的網路傳輸量。在大多數的產品當中，都會設計本地端快取機制來減輕網路傳輸量，避免無謂的查詢浪費頻寬。然而，快取的儲存空間是有限的，在大多數的時間裡，這些快取會因為巨量的查詢而被清除，被更新的查詢內容取代。用傳統產生快取的方法比較沒有效率。

為了減輕網路傳輸的使用量，本論文提出一個解決方法，藉由利用資料探勘技術以及集群的概念，透過收集目前回饋的SPN資料內容作分析，讓這些資料依照相似度做分群，然後預先傳輸這些資料到客戶端，達到減輕網路傳輸量的目的。

在設計的雛型系統中，這套新的快取設計確實可以減少網路傳輸量超過20%，檔案掃描的時間也可以減少超過12%。就單一產品而言，因為網路傳輸必須付給CDN公司的費用，每個月就可以減少20% ~ 60%。

The company use the cloud technology - SPN(Smart Protect Network) to do file scanning. The overwhelming increase of population will lead to an increase network traffic usage. In most products, local cache mechanism is implemented for the purpose of reducing the network traffics. However, the database designed to store cache is limited, lot of times, cache will get purged when there are lots of queries sent and replaced by other newest queries. It is not an efficient by using traditional methodology to build or forming cache.

In order to reduce the network traffic usage, the paper propose a solution, which utilize data mining technique and clustering concept, by gathering the current feedback data we have from our SPN, we are able to form these data in groups with similarity, and by deploying these data to client side, to achieve the reduction of traffic usage.

In prototype, this design can really reduce network traffic more than 20%. And the speed of file scanning time can faster more than 12%. For single product, the payment to the CDN company for network traffic, it can save 20% ~ 60% each month.