基於RGB-D影像資訊之即時指拼法辨識系統

Abstract

溝通是人機互動中非常重要的一環，本論文提出一個以色彩深度影像為基礎的指拼法辨識系統，分為手掌區域偵測、手勢特徵擷取及指拼法辨識三個部分。首先是手掌區域偵測部分，先使用膚色偵測與聯通物件法找出膚色區域的輪廓，再利用距離轉換法決定出膚色區域的特徵點，進而將手臉分離得出手掌區域。接著是手勢特徵擷取，包括手勢形狀及手勢紋理兩種特徵，先由手掌區域的特徵點找出手掌骨架，並決定出掌心及指尖位置、手掌方向及各手指向量，作為手勢形狀特徵，本論文針對無法經由手勢形狀特徵予以辨識的指拼法手勢，提出以局部二質化模式處理手掌區域之灰階影像，產生手勢紋理特徵後再加以辨識。最後，利用不同的類神經網路分類器進行指拼法辨識。從實驗結果可知，本系統對於美國手語中大部分的指拼法，可以達到八成以上的辨識率，為一有效的即時辨識系統。

Communication is a very important part for human-computer interaction. This thesis provides a fingerspelling recognition system with high accuracy rate based on RGB-D image. The system are separated into three parts, including ROI selection, hand feature extraction, and fingerspelling recognition. For the ROI selection, the regions of hand and face are first obtained by skin color detection and connect component labeling (CCL), and then the hand, the ROI, is determined by the feature point extraction based on distance transform. Followed is the hand feature extraction which consists of the hand structure and the hand texture. From the feature points of ROI, the locations of palm and fingertips, palm direction, and finger vectors are formed as the hand structure. In addition to the hand structure, this thesis adopts the LBP operator to generate the hand texture to deal with the fingerspelling not recognizable by the hand structure. Finally, the extracted hand features are sent into the fingerspelling recognition system, which is built with several different neural network classifiers. The experimental results show that this system is an effective real-time recognition system whose accuracy is higher than 80% for most of the fingerspelling in ASL.