深層變異性轉移學習與分類

Abstract

轉移學習(transfer learning)是利用來源領域(source domain)的知識來強化目標領域(target domain)的學習。此篇論文提出深層轉移學習架構應用於模式分類(pattern classification)上，此架構以深層潛在變數模型(deep latent variable model)為基礎，表達潛在變數的隨機性。我們的目標是訓練出不受不同領域影響、且擁有大量類別標籤資訊的潛在特徵(latent features)，為了達成目標，我們提出一個新的潛在變數模型，模型中包含類別特徵與領域特徵的潛在變數，它可以描述潛在變數在領域調適(domain adapation)中的隨機性。在訓練的過程，我們導入變異性自動編碼器(variational auto-encoder)，一種隨機性的深層模型架構，我們想訓練出鑑別式模型(discriminative model)，其潛在特徵對不同類別具有鑑別性且與來源領域跟目標領域無關，但模型中類別與領域的依賴性(dependencies)存在著，他會造成轉移學習的困難並影響模型的正確率。 在此研究中，為了避免不想要的依賴性發生，我們在目標函數導入額外的最大均值差(maximum mean discrepancy, MMD)處罰項，經由訓練可以減少潛在特徵與來源領域跟目標領域的依賴性。除了使用最大均值差處罰項，我們可以利用競爭過程(adversarial process)訓練模型去匹配來源領域與目標領域的分布。此競爭網路(adversarial neural network)架構包含了生成模型(generative model)與鑑別模型，我們利用提出的潛在變數模型做為生成模型來產生潛在特徵，而鑑別模型則是判斷某潛在特徵是來自來源領域還是目標領域。訓練競爭網路的方式為一種兩玩家的極小化極大(minimax)賽局：我們訓練生成模型來使鑑別模型的錯誤最大化，使潛在特徵難以去分辨是來自來源領域還是目標領域，代表潛在特徵與領域無關。此賽局可以促使生成模型產生出與不同領域無關的潛在特徵。我們也利用半監督式學習模型(semi-supervised model)，將沒有標籤資料的標籤視為潛在變數，並訓練潛在特徵使之與類別標籤有高度關聯性， 在實現上，利用變異性貝式最佳化，我們同時進行特徵表示與模式分類的訓練。我們製作不同種類的模擬資料來驗證我們的模型在移除潛在特徵對不同領域的依賴性與保留對類別標籤的高度相關性上的效能，接者我們利用真實世界的資料：Office-Caltech dataset 與 Amazon reviews 來驗證我們的模型效能，他們涵蓋了不同種類的領域調適工作。

Transfer learning, which leverages the knowledge from source domains to enhance learning capability in a target domain, has been proven effective in various applications. This thesis presents a deep transfer learning framework for statistical pattern classification based on a deep latent variable model where the randomness of latent variables is faithfully reflected and compensated. Our goal is to estimate the latent features which are invariant across different domains and maximally discriminative among different classes. To fulfil this goal, a new latent variable model is introduced to reflect the stochastic behavior in latent features during a deep learning for domain adaptation. The latent variables in proposed model consist of those latent features corresponding to class labels and operating domains. A variational auto-encoder architecture is incorporated into such a stochastic deep model during inference. We would like to pursue a discriminative model where the underlying features are discriminative among different classes and at the same time these features are invariant to source and target domains. However, the dependencies between classes and domains do exist and shall cause the difficulty or inaccuracy in transfer learning via latent variable model. In this study, we prevent the unwanted dependencies by introducing an additional penalty term in training objective, which is called the maximum mean discrepancy. This term encourages independence or equivalently compensates the dependencies in distributions of latent features between source domain and target domain. Besides using the MMD penalty term, by an adversarial process, we can train the model to match the distribution of latent features of source domain and target domain. This adversarial network architecture includes a generative model, the proposed latent variables model, which generates the latent features and a discriminative model that determines whether a sample of latent features came from the domain distribution or the source distribution. The idea of training an adversarial network is to train the generative model to maximize the error of discriminative model, which means that the latent features are hard to classify into source domain or target domain, so the latent features are invariant to domains. This framework corresponds to a minimax two-player game. This competition can drive the generative model to generate the feature representation which is invariant to different domains. In proposed semi-supervised model, we encourage the latent features highly correlated to the label factor and treat the label of unlabeled data as latent variable. In the implementation, we conduct co-training for feature representation and pattern classification based on the stochastic gradient variational Bayes optimization. We create different type of simulated data to evaluate the performance of removing dependencies in different domains and preserving the maximal label information in transfer learning. Next, we evaluate our model by using real world data, the Office-Caltech dataset and Amazon reviews, which cover different tasks for learning the adaptation across different domains.