三維景物重建之多視圖矩陣分解法

Abstract

本研究計畫預計在三年期間內發展一系列的電腦視覺技術，來研發透過多張影像來進行場景的三 維稠密式幾何重建，而這些輸入影像可能是利用許多佈置於景物上方圓頂形架上之眾多相機所拍攝， 或是利用單一相機於場景中移動所拍攝。此外這些影像或視圖可以是已校正過或是未校正過。 在第一年的研究計畫中，將類似用架設在一圓頂上的多台相機對準室內同一物體拍攝，並利用這 些視圖來重建物體的三維幾何模型。在此系統中的相機皆事先經過校正，也就是說相機的內外部參數 皆為已知且固定不變。我們提出一個基於多於三張視圖矩陣分解理論建稠密式立體物體模型，而非僅 用兩張兩張影像所建之局部立體物體加以合併而已。首先使用我們所提出的楕圓式LoG 特徵擷取演算 法取得具有仿射不變性的特徵點。此方法可以不受不同視角拍攝所產生影像變形的影響。對於多張不 同影像中候選特徵點的比對是在仿射正規化的影像空間進行，而非在原本的影像空間比對。因此可避 免傳統以固定正四方形視窗之關聯係數計算測試所造成的失敗，尤其當兩拍攝相機之間基線較大時， 更是如此。下一步我們將用初始所擷取具有高精確度的特徵點得到一個＂初始邊界模型＂(IBEM)，以 用來做為場景近似幾何模型。接著利用多視圖的影像亮度一致性測試與可見度測試，用疊代或漸進方 式對IBEM 做進一步的提煉，最後即可得到精細的稠密式網格模型。此模型可用網格小平面區塊為單 位，一片接一片地貼上從適當視角影像中對應到的影像區塊的紋理。 第二年的研究計畫將利用單一相機來拍攝，並進行多視圖稠密式戶外場景的重建。在此假設相機 的內部參數是已知且在拍攝時固定不變。然而因為相機的外部參數是未知的，因此需要對外部參數進 行估算。在估算之前，必須先得到足夠數量的可靠對應點。經由我們對於具有仿射不變性所擷取的特 徵點中，可以得到一些唯一對應的特徵點。至於若需要其他更多的點對應，可以針對已得的唯一對應 特徵點找最近的特徵點用。利用這些特徵點可用多視圖矩陣分解法得到相機的外部參數與特徵點的三 維座標。為了排除可能在上述相機參數估測時發生只找到的區域性的最小值，將再利用模擬退火法搭 配直交實驗設計法，以前述方法的解當做初始解來進行統計搜尋，藉以得到全域的參數估測最佳解。 此時就能使用得到的可靠相機間極線幾何關係，解決戶外人工場景中大量的重覆性圖樣間的可能錯誤 匹配。最後則如同第一年計畫進行稠密式的物體幾何模型重建。 第三年提出的計畫則為利用未校正影像進行多視圖三維重建。以今日之手持數位攝影機進行變焦 拍攝十分普遍，取得的影像就是未校正過。我們嘗試分析相機內部參數無變動與有變動兩種影像對重 建工作精準度之影響。兩種影像都先用多視圖矩陣分解法求得投射空間的三維場景模型，再試圖用兩 種可能方法將三維模型由投射空間轉換至歐基理德空間。其中一種是利用場景知識如三組以上的明顯 互相垂直線條組合資訊來做，另一種是用自動校正技術來做。此外我們也要研究解決在三維重建過程 中會遇到的影像雜訊、遺失點與外來點等問題，以進而提昇三維重建模型的精準度。

The three-year research project addresses a series of computer vision techniques for reconstructing 3D dense scenes from plenty of images taken by either a large number of fixed cameras scattering around the scene or a single camera moving around the scene. In addition, the images can be either calibrated or uncalibrated. In the first-year project a dense reconstruction of a 3D indoor scene (or a target object) will be derived from multiple cameras with a fixture deployed in the shape of a dome. They aim at a common object, while capturing the images. In this setting all cameras are calibrated in advance, that is, their intrinsic and extrinsic camera parameters are all known and fixed. We propose an accurate dense scene reconstruction method based on multi-view factorization theory rather than by gluing pairwise stereo reconstructions. First of all, affine-invariant interest points are extracted by our proposed elliptical LoG detector which can resist against the view change caused by the different camera viewpoints. The correspondence point matching among the candidate interest points across the multiple views is conducted in an affine-normalized image space instead of the original image space. As a result, the failure common to the standard window-based correlation test on the point correspondence is avoided, in particular, in the wide baseline stereo setting. Next, to make a dense scene reconstruction possible we propose to obtain an initial boundary edge model (IBEM) based on the initial highly accurate interest points extracted. Then an iterative or progressive refinement of the IBEM is conducted based on the multi-view photo-consistency test and visibility test. A final refined mesh model obtained becomes a dense mesh model. The model can be mapped patch by patch with texture fetched from a corresponding image patch in the respective properly selected view. In the second-year project a multi-view dense outdoor scene reconstruction is conducted using a single camera, assuming its intrinsic camera parameters are known and fixed throughout the picture shooting. Since the camera extrinsic parameters are unknown in this camera set-up, we need to estimate these parameters. Before the estimation a sufficient number of reliable corresponding point pairs have to be obtained first. This is made possible through our affine-invariant interest point detector in which some uniquely matched point pairs can be found. Besides, more point pairs are obtained through the use of proximity measure on the interest points with respect to the already uniquely matched point pairs. The multi-view factorization method is used to derive the camera extrinsic parameters and 3D interest points. To deal with possible local minimum traps in the parameter estimation a global optimal solution to the parameter estimation the above solution is used as an initial solution in a stochastic search method making use of the simulated annealing procedure with the aid of the orthogonal experimental design for the solution space. Consequently, the reliable eipolar geometry relationships among the cameras can be obtained and the relationships will be used to disambiguate the similar structure patterns abundant in the outdoor man-made scene. Finally, a dense reconstruction can be done in the same way as given in the first-year project. The third-year project will address the multi-view 3D reconstruction using uncalibrated images. The uncalibrated images are quite common in the nowadays hand-held digital camera shooting during which the lens zooming function is likely activated. We shall study the effect of the change in the camera intrinsic parameters on the reconstruction accuracy. For the group of pictures captured with or without the change in the intrinsic parameters we apply the multi-view factorization method to obtain a projective 3D scene model, and then two possible ways of transforming the projective reconstruction to a Euclidean reconstruction will be proposed: one using the scene prior knowledge and one using the autocalibration. At the same time, we shall study how to deal with the image noise, missing points, and outliers as a whole during the multi-view reconstruction process so that more accurate reconstruction can be achieved.