多視點視訊產生、傳輸與分析---子計畫四：在無線寬頻網路上感官式自由視角視訊之即時瀏覽技術與編碼研究

Abstract

最近數年，自由視角電視由於相關設備與訊號處理技術的演進，相關的研究有快速的成長，在MPEG 也相當積極進行自由視角電視標準化的動作。自由視角電視系統在網路上傳遞視訊資料到用戶端，也 讓其可以自由選擇所欲觀賞視角的視訊。由於用戶端選擇的視角可能是沒有實際擷取視訊的角度，而 需要在該視角進行視訊合成。除此之外，文獻上對於流暢之即時視角轉換相關的研究較為缺乏，而在 無線寬頻網路有限而且不穩定的頻寬資源上傳輸多視角視訊也有相當的挑戰。我們規劃以三年的時 間，延續先前多視角景深估計演算法的成果與感官視訊串流相關之經驗，發展更完善之視角合成演算 法，並根據自由視角使用者轉換視角的情況，在即時瀏覽的前提下研究不同複雜程度之合成演算法排 程決策之制定，使自由視角系統在裝置所能負擔的資源之下發揮最佳觀賞經驗。當自由視角電視在網 路上傳遞視訊時，我們規劃以可調式編碼方式壓縮多視角視訊，並發展對應之感官式合成品質預估演 算法與不對稱保護之分層通道編碼，將網路可用頻寬資源進行頻寬資源分配的最佳化，以使自由視角 視訊的使用經驗品質獲得長足的進步。

With the recent progress of displays, capture devices, and coding technologies, the related research topics about free viewpoint TV (FTV) technologies have emerged rapidly. In addition, the MPEG started the international standardization activities of FTV in 2004. Multi-view video streams are usually transmitted to clients in free viewpoint TV systems over various networks and clients are given the freedom to change viewpoints as if they were there. It is possible for the desired viewpoints not being captured actually where in this case those views need to be synthesized. Since the development of FTV is still in the early stage, there are not many feasible free viewpoint navigation technologies available in the literature for real-time and smooth transition of viewpoint changes, and multi-view video streaming over wireless broadband networks with limited and error-prone channel conditions also poses great challenges for realization of free viewpoint TV systems. In this 3-years project proposal, we intent to develop better view synthesis algorithms based on our results in multi-view depth estimation; more importantly, we plan to design efficient scheduling algorithms to determine appropriate view synthesis tools to generate enough viewpoints during the transition of various viewpoint changes such that smooth and real-time FTV quality of experience can be fulfilled at given computation resource and acceptable delay. To transmit FTV multi-view streams over error-prone networks, the multi-view streams will be coded by scalable coding tools and we will propose perceptual video quality estimation algorithms for views that will be synthesized at client sides. Together with the proposed development of layered channel coding tools that shall have accurate protection probability models, a better strategy to allocate bandwidth resource among transmitted scalable multi-view streams and unequal error protection rates can be made accordingly such that the quality of FTV service experiences can be satisfactory.