利用腦磁圖進行大腦訊息處理之流形編碼與解碼

Abstract

人腦視覺編碼和解碼對於研究神經活動表示法扮演相當重要的角色，編碼是大腦將外界刺激或心理狀態轉換為神經活動的過程，反之，解碼的過程則是利用大腦的神經活動反向推估相對應之刺激或狀態。一旦能夠建立起模擬大腦運作的解碼或編碼模型，我們將可以對低階的視覺刺激圖樣或高階的複雜刺激影像（如人臉影像）進行解碼或編碼，藉此探討大腦的運作過程。這樣的編解碼過程通常需透過一種由神經活動層次表示法轉到高階感知層次表示法的轉換過程，此過程也是階層性人臉處理的基礎。許多低階的視覺處理模型已經於先前的研究中被提出，然而探討大腦如何處理高階的視覺資訊仍是個難解的問題。其中，先前的研究指出大腦的活動形成高維流形，且推測此即為人臉身分識別的基礎，但此論點尚未被證明也尚未被廣泛地應用於神經科學之研究上。 本論文包含三個研究，並設計出三種不同的視覺刺激材料和實驗來進行方法驗證。首先，於第一個研究中，我們提出一個大腦雙向解碼與編碼模型，此模型利用包含時序和空間資訊的腦磁波訊號建構出大腦神經活動的流形表示法。在我們提出的解碼流程中，首先利用光束集成法從腦磁波訊號估算大腦皮質活動，接著利用主要成分分析法擷取腦部活動時序上的主要成分 (temporal principal components, TPCs)。之後我們計算這些TPC在空間上的流形分佈，進而得到空間與時間上的主要成分 (spatiotemporal component, STC)。我們進一步建立起STC與從影像求得之小波轉換係數之間的線性對映關係。當此關係建立後，我們將能夠由腦部神經活動重建出視覺刺激影像。更進一步，由於此模型的可逆性，我們也能夠從視覺刺激影像預測出相對應之隨空間、時間變化的腦部神經活動。我們利用十一種不同棋盤格之組合方式所產生的視覺樣式當作視覺刺激。在此實驗中我們發現影像樣式在空間上的分佈資訊可以在流形空間中顯現出來。更甚，利用本模型重建出的影像樣式與真實的影像之間有高達0.71的空間相關程度，且預測出的腦部活動與真實的腦部活動之間也有0.89的高度相關程度 於第二個研究中，我們將流形解碼方法應用於大腦對於人臉影像處理的研究。我們利用包含不同人臉視角與不同注視角度的人臉影像當作視覺刺激材料，並藉由其產生的一系列單試驗腦部活動，建構出一個低維度的神經活動流形。於此建構出的流形空間，我們可以利用高階概念合成的特性預測包含不同人臉視角和不同注視方向的人臉影像，藉此探討人臉的階層性處理機制。由結果發現在枕葉人臉區和右側上層顳葉溝的M170時間成分，分別能夠準確地預測人臉視角和注視角度。 最後，於第三個研究中，我們基於先前提出的非監督式流形學習方法，開發出一套監督式的流形學習演算法，並進一步比較此方法和傳統非監督式與監督式學習法在預估人臉轉動角上的準確率。我們利用自然拍攝的人臉影像，套用我們開發的監督式學習演算法，發現大腦活動的低維空間可以顯示出人臉的旋轉角度，且準確率高於傳統的資料降維方法。 以上研究結果說明了除了大腦活動資料的空間分佈關係外，由腦磁波量測到的時間資訊在視覺處理上也扮演了重要的角色。藉由實驗所得到的結論，我們認為基於神經活動流形之分析方法可以被應用在神經科學的研究上，且可將其視為在人臉處理上一種有效的表示法。

Visual encoding and decoding are crucial aspects in investigating the neural representation of visual information in the human brain. Once a decoding and encoding model is constructed, it can be used to decode the low-level visual patterns or higher-level visual stimuli such as facial images. Such decoding requires the transformation from a neural representation to a perceptual representation, which lays essential foundation to hierarchical face processing. Previous studies have proposed models for low-level visual processing. However, how the brain process the high-level perception remains an unsolved question. There are three studies in the thesis and three corresponding experiments were conducted for each of the study in the thesis. In the first study, we proposed a bidirectional model for decoding and encoding of visual stimulus based on manifold representation of the temporal and spatial information extracted from magnetoencephalographic (MEG) data. In the proposed decoding process, principal component analysis is applied to extract temporal principal components (TPCs) from the visual cortical activity estimated by a beamforming method. The spatial distribution of each TPC is in a high-dimensional space and can be mapped to the corresponding spatiotemporal component (STC) on a low-dimensional manifold. Once the linear mapping between the STC and the wavelet coefficients of the stimulus image is determined, the decoding process can synthesize an image resembling the stimulus image. The encoding process is performed by reversing the mapping or transformation in the decoding model and can predict the spatiotemporal brain activity from a stimulus image. In our first experiments using visual stimuli containing eleven combinations of checkerboard patches, the information of spatial layout in the stimulus image was revealed in the embedded manifold. The correlation between the reconstructed and original images was 0.71 and the correlation map between the predicted and original brain activity was highly correlated to the map between the original brain activity for different stimuli (r=0.89). In the second study, we applied a decoding method based on the decoding and encoding model to face representations. A low-dimensional neural manifold was constructed using a set of single-trial brain activity data evoked by stimuli with basic face viewpoints and gaze directions. As a perceptual representation with synthesis property, this manifold was able to predict composite viewpoints and directions from brain activity. In the second experiments, when facial images with varying viewpoints and gaze-directions were used as the experimental stimuli, the M170 component in occipital face area and the right superior temporal sulcus gave accurate prediction for face viewpoints and gaze directions, respectively. In the third study, we proposed a supervised locally linear embedding method to construct the embedded manifold from brain activity, taking into account similarities between corresponding stimuli. In our experiments, photographic portraits were used as visual stimuli and brain activity was calculated from MEG data using a source localization method. The results of 10×10-fold cross-validation revealed a strong correlation between manifolds of brain activity and the orientation of faces in the presented images, suggesting that high-level information related to image content can be revealed in the brain responses represented in the manifold. These results suggest that the temporal component is important in visual processing and manifolds can well represent the information related to visual perception. In addition, the proposed neural manifold method can be used to construct an effective perceptual representation for face processing and is applicable to investigation into the inherent patterns of brain activity.