使用彩色深度感測器之虛擬攝影棚自動導播系統

Abstract

隨著影視工業的發展，虛擬攝影棚越來越廣泛地被使用在電視或電影的製作上。然而，傳統的虛擬攝影棚大多使用藍綠幕來進行去背，並且需要由導播在幕後操控導播機。本研究的目的是發展出一套能夠於自然場景下自動達到去背效果的導播系統，同時可以經由使用者的手勢直接執行操控指令，並與虛擬物件進行互動。 由於此系統設計是使用Kinect感測器來輔助專業的攝影棚攝影機，因此我們必須將Kinect感測器的深度資訊對位至另一台高解析度彩色攝影機。本系統主要分為三個部分，第一部分是相機校正與對位，第二部分的去除背景，第三則是手勢辨識。我們使用張正友校正法計算出的相機參數來對位兩台攝影機。接著使用Kinect感測器提供的深度值及人體骨架資訊，並對高解析度攝影機的彩色影像做處理取得更細部的輪廓，互相搭配後切割出前景人物區域。另外，人體骨架中手部關節點的三維座標會被記錄下來，對這些資料經過處理後，利用SVM分類器訓練我們的動態手勢樣本，而後於系統中進行辨識。 根據我們的實驗結果，此系統不須透過藍綠幕，即可有效地在自然場景下去除背景，並且運用深度資訊，達到真實人體前景和虛擬三維物件之間的擬真互動與遮蔽效果。此外，我們的系統提供相當高的自由度讓使用者可以經由手勢自行觸發指令或與虛擬物件，達到高度自動化的目的。

Virtual studios are widely used for television programs or movies. However, in the traditional virtual studio, a director is necessary to control the switcher when using typical program director systems and these systems are mostly based on the bluescreen technology. In this study, we developed a program director system that can be used in natural scenes and can automatically remove the background. It can also carry out instructions and interact with virtual objects through the user’s gestures directly. In the proposed system, Kinect sensor is used to assist the professional studio camera. Therefore, we need to align the depth information of Kinect sensor with another high-resolution color camera. Our work contains three parts. The first part is camera calibration and registration. The second part is background removal. The third one is hand gesture recognition. Zhang's calibration method was used to calibrate and register two cameras. To segment the actor foreground, we utilized the depth information, human skeleton data provided by Kinect sensor and processed the color image captured by the high-resolution camera to get a more detailed contour of the foreground. In addition, three-dimensional coordinates of hand joints of the human skeleton were recorded and preprocessed for hand gesture recognition. Eight types of dynamic hand gesture were trained using the multi-class SVM classifier. According to our experimental results, the proposed algorithm can achieve effective background removal in natural scenes instead of in front of a green or blue screen. Moreover, the immersive interaction and occlusion effect of the human foreground and virtual objects were achieved with the depth information. Our system can provide more freedom for the actor to interact with virtual 3-D objects and trigger instructions using gestures.