嵌入式網路通訊裝置評比技術與工具之研發---子計畫一：嵌入式網路通訊裝置應用效能評比技術與工具之研發

Abstract

嵌入式系統平台已被大量應用在商用及家用產品上如PDA和WiFi Phone，但是對於嵌入式系統網路裝置在使用者端的應用效能上卻缺乏可用的測試工具，廠商在開發產品之後無法精確得到其效能，當產品有關於效能方面的問題也無從得知可以改善的方式，為此，開發「嵌入式系統網路通訊裝置應用效能評比工具」，除了要提供黑箱測試數據外，更應設計對於開發者可進行加強以及偵錯的灰箱及白箱測試，協助開發者清楚了解產品在使用者端所遇到的缺點以及相關改進的方法。 本計畫欲開發的項目，是針對目前熱門的嵌入式系統網路通訊產品，進行相關網路應用服務效能評比(如: Throughput, Latency, Session Capacity, Session Rate, Voice Quality)，利用NBL現有的工具以及Open Source軟體，以Repeatable、Scalable、Automatic、Integrated等四個條件為前提，進行1. 測試計畫及方法之研發, 2. 對測試設備無依賴性之測試工具研發, 3. 整合型測試工具研發，配合總計畫提供的共通測試平台，提供各種不同層級的背景流量，更能貼近實際上使用者端使用的環境，在測試工具的開發規劃裡，預計進行三項測試工具的開發，其中包括: 應用層呑吐量與連線量測試工具、SSL VPN連線量測試工具以及語音品質整合測試工具，這些都是目前所所缺乏的應用效能測試工。本子計畫預計分三年進行，第一年將就各類型的嵌入式系統網路通訊產品裝置以及應用服務所使用的協定進行研究，同時進行測試以及收集測試成果，並且在使用現有測試工具的同時了解其API使用的原理及方法；第二年將就研究的成果進行測試計畫以及測試工具軟體開發，初步進行待測物測試(包含黑箱、灰箱及白箱測試)，同時驗證開發工具的正確性和可靠性，並開放原始碼；第三年將測試項目及參數標準化，將整合測試自動化，針對各項產品進行測試並公佈測試報告，並將測試工具產品化、客製化。

Embedded System has been used for business and home appliance widely. However, there is no congruent tool for benchmarking embedded network and communication devices for user-experience performance. Developers have no idea to improve products’ performance according to the feedback from users. So, it is necessary to develop tools and technologies to benchmark the embedded network and communication devices for user-experience performance. Besides the black-box testing technologies and tools, we should provide gray-box and white-box tests for developers to enhance and debug the devices. This proposal will focus on the popular embedded system network products to benchmark the related network application functions like Throughput, Latency, Session Capacity, Session Rate, Voice Quality. There are three objects of this subproject: 1. Research and Development of Test Plans, 2. Development of Test Utility Independent Test Tools and 3. Development Integrated Test Tools by using the available instruments and open source software on the understanding that repeatable, scalable, automatic and integrated. The grand project provides the common testing environment for us in generating background traffic to make the test network becomes real and trustworthy. In the plan of test utilities development, we target on three testing tools, which are「Real Application Throughput and Session Capacity」, 「SSL VPN Tunnel Capacity」, and 「Integrated Voice Quality」. For the first year, we will study the embedded system network devices and the related network protocols. At the same time, benchmark the applications, gather the report data and learn the APIs provided by the instruments, and design new test plans. After the first year, we will execute new designed test plans and start to develop our test tools. We will begin to do the basic black-box, gray-box and white-box test, by using the prototype of our new tools. At once, we have to validate and open it. Finally, we shall normalize the test items and parameters, automate the test plan, benchmark the products and open the test report, and then turn our tools into product quality packages in the third year.