立體景物重組之影像合成與攝影機對位

Abstract

背景組合被廣泛的用在電影和電視節目，而結合兩組3D視訊成一組視訊是一項非常有挑戰性的工作。原始的視訊通常由不同位置的兩個攝影機所拍攝而得，例如，這兩組攝影機可能會朝不同的方向移動，它們的方向(姿態)也可能不盡相同。我們的目標在於組合兩組具有不同攝影機方向與移動參數的RGBD視訊。其關鍵技術在於攝影機的移動估測、方向估測，以及用來產生移動補償與方向補償後合成背景之視角合成技術。深度輔助視角合成技術，在此應用是一個關鍵且對於影像品質有巨大的影響。因此我們提出一個強化過的，應用在視角合成的反向映射技術，並且採用ICP演算法以及1D地板模型來計算攝影機移動 / 方向參數。 在本論文，我們使用很普及的超像素方法來強化先前所提出的反向深度映射方法。有許多現成的超像素生成技術，而其中一個很受歡迎的有效方法是稱為SLIC的超像素方法。我們使用SLIC技術，再加上ㄧ些修改來符合我們的需求。超像素技術是用來處理反向映射後所發生的物件內部破洞問題。 我們使用ICP演算法來估測相機移動參數。ICP是一個在電腦圖學很普及的技術，它通常用來建構3D模型。ICP輸入兩組3D點雲並且疊代計算兩組之間的轉換矩陣。在此應用，我們從一個視訊的兩個連續畫面來產生3D點集合。在我們的實驗，我們計算兩個畫面(兩個攝影機位置)之間的位移向量。我們也提出一個畫面之間攝影機移動補償技術來減少計算時間。此外，我們調整背景攝影機的方向 (姿態) 來匹配前景攝影機。我們結合所有以上所提的技術來合成一個高品質且看起來很自然的虛擬視角影像，即使原始兩個攝影機的方向和移動都不相同。

Scene composition is a method widely used in movie and TV production. Merging two sets of 3D videos into one is a very challenging task. The original two video sequences are often taken by different cameras at different locations. For example, these two cameras may have different movement, and their orientations (poses) can also be different. Our focus is compositing two sets of RGB-D videos with different camera orientations and motion parameters. The key techniques are camera motion estimation, camera orientation estimation and the view synthesis technique used to produce the synthesized motion-compensated and/or orientation-compensated background video. The depth-assisted view synthesis technique is a key component in this process and has a strong impact on the final video quality. We thus propose a refined backward warping technique for view synthesis and adopt the ICP algorithm and 1-D floor model to calculate the camera motion/orientation parameters. In this thesis, we use the popular superpixels notion to refine the previously proposed backward depth warping method. There are many superpixels generation techniques and one of the popular and effective technique is the so-called SLIC (Simple Linear Iterative Clustering) superpixels. We use the SLIC technique with some modifications to match our needs. The superpixels technique is used to deal with non-occlussion holes problem that occurs in backward depth warping. We adopt the ICP (Iterative Closest Points) algorithm to estimate the camera motion parameters. ICP is a popular technique in computer graphics, which is commonly used to construct 3D models. ICP takes in two sets of 3D-point clouds as inputs and iteratively calculates the transformation matrix between these two sets. In our application, we generate the 3D point sets from two consecutive frames of a video sequence. In our experiment, we calculate the translation vector between two frames (two camera locations). We also propose an inter-frame camera motion estimation technique to reduce the computing time. Furthermore, we adjust the background camera orientation (pose) to match that of the foreground camera. We combine all the above techniques together to synthesize a good quality nature-look virtual view even when the original two cameras were mismatches in orientation and in motion.