應用於3D視訊多媒體之多核心微型通訊系統研究---子計畫五：高畫質多視角立體視訊核心技術研究(I)

Abstract

隨著高畫質電視的蓬勃發展與成熟，下一世代的家庭娛樂隨著消費性立體顯示器的 推陳出新，開始轉為著重發展於能更令人類身歷其境的立體視訊。然而，立體視訊相較 於平面的二維視訊面臨更嚴峻的挑戰。第一，傳統平面視訊無法在立體顯示器上產生立 體效果。現有轉換立體視訊方法多無法以即時方式進行，且立體效果十分有限。第二， 良好立體視訊的產生，需要高準確度的深度視訊估測(depth estimation) 、虛擬視點視訊 合成引擎 (view synthesis)，會耗費相當大量的複雜計算資源。第三，如同平面的二維視 訊走向高畫值，立體視訊在高畫值的影像大小下，不單是計算資源大增，也伴隨著相當 大量的資料傳輸，造成匯流排或記憶體頻寬難以負荷。 針對以上的立體視訊轉換與其所帶來的高運算複雜度和高記憶體存取，本計畫將發 展一個可產生立體視訊的設計，其設計主要分為三部分，並且分置於三年執行。第一部 分，針對輸入為單視點的視訊處理，將現有的傳統二維視訊轉換為立體視訊。第二部分， 針對輸入為二視點的視訊處理，將雙眼相機所擷取的視訊轉換為立體視訊，其轉換步驟 包含高準確度的場景深度估測，以及對應的虛擬視點合成。第三部分，針對輸入為多視 點的視訊處理，將針對先進的多相機系統所擷取的視訊轉換為立體視訊，其轉換步驟亦 包含場景深度估測和虛擬視點合成。此部分可支援多視點的視訊，觀者可自由切換視點 於視訊中。本計畫目標應用為現今人類追求的高畫質視訊作處理。針對其所帶來的於高 運算複雜度和高記憶體存取之挑戰，發展適合平行處理實現的立體視訊處理的演算法， 將之轉換為適於硬體加速與排程的硬體設計，並且特別針對記憶體的大量存取做考量， 以達成即時運算。

With the popularity and maturity of high definition TV, the next generation home entertainment has now focused on the 3D video to obtain more vivid viewing experience with the fast development of stereoscopic display. However, 3D video faces the tougher challenges than the traditional 2D video does. First, a 2D video cannot display with stereoscopic effect just on the stereoscopic display. Current 2D to 3D video approach often uses the off-line method with limited stereoscopic effect. Second, a good stereoscopic video needs high precise depth map estimation and view synthesis, which consumes a lot of computational resources. Third, just similar to the high definition trend in 2D video, a high definition stereoscopic video not only brings higher computational burden but also consumes a lot of data transfer and thus puts a severe bandwidth demand to system bus and the memory system. Being aware above issues, this project aims to develop a 3D video design that consists of three parts spanned in three years. The first part considers the single view input case for traditional 2D TV contents. Thus, a 2D to 3D conversion will be developed with just 2D video information. The second part considers the two view input case from a commercial two view camera. Thus, a high accurate depth information and related view synthesis will be developed for high definition 3D display. The third part considers the multiple views input case from advanced multiple camera system for free viewpoint video. Thus, more information can be used to compute depth and view synthesis for wide angle viewpoint. The whole project is targeted to the high definition video, which is the mainstream of viewer’s experience but suffers sever computational complexity and especially heavy memory access. With above complexity in mind, this project is to develop the corresponding algorithms and associated implementation with special focus on the heavy memory access to achieve real time processing.