基於正逆向深度映射演算法下的稀疏多相機虛擬視點合成

Abstract

近年來，互動式多視角視頻 (Interactive Multiple View Video) 在多媒體領域上已經是一個熱門的研究方向之一。因為擁有廣泛應用的潛力、以及大量的市場需求，國際標準會議遂制訂立體視訊編碼 (Three-Dimension Video Coding，簡稱 3DVC) 相關的標準競賽。

然而目前的立體視訊編碼只針對於密集且平行相機陣列系統，其相鄰相機間的距離不會超過10公分。本篇論文係打破此一距離上的限制，以稀疏且非平行相機陣列為重心，發展一套稀疏多相機虛擬視點合成系統。本篇論文的目標在於藉由已知的稀疏影像，合成出虛擬視角。一開始會先分析目前現有合成系統中的正向景深映射，並探討映射後所產生的各種瑕疵現象成因，而提出一套景深修復方法來做改進。

接著我們根據現有的自由視點架構，一共改進四個地方，分別是景深圖升取樣方法、逆向景深圖映射演算法、金字塔型影像修補技術、以及瑕疵抑制方法。景深圖對於升取樣中，我們經由實驗發現複本插值法的效果比一般插值法還佳。而目前現有的正向景深圖映射演算法會產生大量的瑕疵現象，利用一些幾何關係，我們發展一套逆向景深圖映射演算法，可大幅降低映射後的瑕疵現象。再來是虛擬視點合成後所產生的遮蔽區域，傳統上所使用的影像修補技術對於雜訊過於敏感，因此我們提出金字塔型影像修補技術，以彌補其缺陷。最後本論文則提供一個簡單的演算法，抑制合成影像的瑕疵，使人眼主觀感受上能更加生動自然。

以上演算法皆會利用三台稀疏陣列相機所拍攝的影像來做測試。實驗結果展示出每項技術對於最後的虛擬視點合成，其主觀品質皆有不錯的改善。

Recently, multiview video technology receives a lot of attention. Because of its potential wide applications and large market, the so-called free-view system or virtual view system becomes a standardization item (3DVC) of the international MPEG committee.

The current focus of MPEG 3DVC project is on the parallel and dense camera array system. The distance between two nearby cameras is about of less than 10 cm. In contrast, our focus in this study is on the so-called sparse multi camera systems, of which the cameras are located farther away. Our target is to synthesize a virtual view based on the recorded sparse camera pictures. We first study the causes of the depth map artifacts produced in the process of forward warping. And then to reduce these artifacts and the artifacts in the texture image synthesized stage, we propose a few refinement tools described below.

Four techniques have been developed and presented in this thesis. They are depth map up-sampling, backward depth map warping, pyramid-based hole filling, and post artifact reduction. At the up-sampling stage, we find empirically that the duplicated interpolation produces better depth map as compared to the other interpolation methods. The ordinary forward warping produces many types of artifacts. We propose an instrumental backward warping algorithm on the depth map. It is able to reduce most artifacts in depth warping due to the more accurate geometric relations. The occlusion regions can be eliminated by using our proposed pyramid-based hole filling method. One of its nice features is that it can suppress the noise when filling in the occlusion area. Finally, with the aid of artifact reduction techniques, the synthesized virtual view is more vivid and natural.

All the above techniques have been tested on the test images captured by a set of sparsely located 3-camera array. The results show that every of them can prove visible subjective quality improvement on the synthesized virtual view images.