多視點視訊產生、傳輸與分析---總計畫及子計畫二：多視點視訊表述及景深圖壓縮研究

Abstract

使用多架攝影機擷取影像，傳送壓縮的資料，之後在接收端合成虛擬視點影像， 這樣的系統成為新興的研究主題。上述系統也稱作「自由視點」或是「多視點」方案， 現在已列為MPEG 標準的標準化項目。過去這個題目上已經有一些研究資料，但整體 而言，這方面的研究還不夠完整。確切來說，這個題目涵蓋了相當多的議題，像是多 攝影機校正、深度估測、深度編碼、視點合成和多視點資料分析。 本計畫將研究多視點系統中兩個重要的研究課題:多視點影像資料表述和景深資訊 編碼為。第一個課題為適於自由視點合成的多視點資料格式或資料結構。第二個課題 為深度(或視差)資訊的壓縮。一般而言，深度資訊有不同於色彩影像資訊的特性。統計 資料顯示深度圖的邊界部份數值變化非常劇烈，邊界部份的誤差會強烈的影響到合成 影像的視覺品質。因此對於邊界處，我們提出各類不同樣式的模版去改善深度資訊壓 縮後的品質。2011 年11 月MPEG 標準會議的競賽結果顯示，多視點(或左右兩視點) 影像加上相對應的深度圖的表述方式，似乎已成為多視點影像最有效且主流的表述方 法。不過縱使已接受此種架構，仍然有許多有待研究的問題。舉例來說，如何在色彩 影像資訊和深度資訊去分配位元數。另一個重要問題則是要如何利用深度資訊去增進 色彩影像編碼的效率。這個子計畫為「多視點影像產生、傳輸和分析」整體計劃的一 部份。在未來三年，對於上述所提到的兩個主題，我們擬分析已有的最佳技術並提出 我們的解決方案。

Capturing videos by multiple cameras, transmitting the compressed data, and then synthesizing the virtual-viewpoint video at the receiver become an emerging new research topic. This is the so-called “free-view” or “multiview” scenario and is now a standardization item in the MPEG standard. Although there are already a few studies on this subject, overall, this subject is less explored. Indeed, there are many issues involved such as multi-camera rectification, depth estimation, depth coding, view synthesis, and multiview data analysis. This project, Multiview Video Data Representation and Depth Map Coding, investigates two important topics in the multiview system. The first topic is the data format or the data structure of presenting multi-view data for free viewpoint synthesis. The second topic is depth map (or disparity map) compression. The depth data has different characteristics from the color image data. The collected statistics show that the depth map boundaries have sharp transitions and have strong impact on the synthesized video visual quality. We thus suggest a pattern (template) model of depth boundary to enhance the compressed depth map quality, particularly, around the boundaries. Based on the MPEG contest results in November 2011, the multiview (say, tow-view) plus depth (MVD) approach seems to be most effective and robust representation of the multiview videos. There are many remaining open questions even if this framework is accepted, for example, how to distribute the bits between the texture (color image) data and the depth data. Another open question is the use of depth data to improve texture coding efficiency. This sub-project is a part of the “Multiview Video Generation, Transmission and Analysis” integrated project. In the next 3 years, we plan to investigate the state-of-art techniques on the above two topics and propose our solutions.