嵌入式裝置自動測試機制之準確度、效率與重複使用度研究

Abstract

自動化圖形使用者界面測試透過模擬使用者事件以及自動驗證圖形用戶界面的變化以確認Android應用程式是否符合規格。傳統的record-replay 測試工具能自動產生測試項目，但不會自動驗證測試的結果。本篇論文實作的工具SPAG-C (Smart Phone Automated GUI testing tool with Camera) 是延續以前論文所做的SPAG，除了能節省測試所需的時間外，能在不損失測試準確性的前提下，提高test oracle的重複使用性。在record階段，SPAG 檢查從手機frame buffer 獲得的畫面以及將驗證命令加入測試項目。與SPAG不同的地方在於，SPAG-C使用外部的視訊鏡頭捕捉手機畫面。在replay階段，SPAG-C驗證測試結果的方式為自動比較圖片，但SPAG只會比較字串。而且為了使SPAG-C能重複使用於不同的設備以及保有更好的圖片捕捉同步性，我們開發了一個使用外部的視訊鏡頭和Web服務的新架構去搭配test oracle。實驗結果顯示，用SPAG-C記錄測試項目的速度跟SPAG一樣快，但測試項目卻更加準確。而在保有相同的測試項目準確性下，SPAG-C記錄測試項目的速度比SPAG快50％至75％。因為SPAG-C針對不同手機的重複使用性，SPAG-C能將測試時間從幾天縮短到幾小時。

Automated GUI testing consists of simulating user events and validating the changes in the GUI in order to determine if an Android application meets specifications. Traditionally, record-replay testing tools facilitate the test case writing process but not the verification process. In this work, we develop SPAG-C (Smart Phone Automated GUI testing tool with Camera), an extension of our previous work called SPAG, to reduce the time required to record test cases and increase reusability of the test oracle without compromising test accuracy. In the record stage, SPAG captures screenshots from device’s frame buffer and write verification commands into the test case. Unlike SPAG, SPAG-C captures the screenshots from an external camera instead of frame buffer. In the replay stage, SPAG-C automatically performs image comparison while SPAG simply performs a string comparison to verify the test results. In order to make SPAG-C reusable for different devices and allowing better synchronization at the time of capturing images, we develop a new architecture that uses an external camera and Web services to decouple the test oracle. Our experiments show that recording a test case using SPAG-C’s automatic verification is as fast as SPAG’s but more accurate. In particular, SPAG-C is 50% to 75% faster than SPAG in achieving the same test accuracy. With reusability, SPAG-C reduces the testing time from days to hours for heterogeneous devices.