Browsing by Author "Lee, Chao-Hsi"

Now showing 1 - 4 of 4

Deep Unfolding for Topic Models
(2018-02-01)
Chien, Jen-Tzung
;
Lee, Chao-Hsi
;
電機工程學系
;
Department of Electrical and Computer Engineering
Deep unfolding provides an approach to integrate the probabilistic generative models and the deterministic neural networks. Such an approach is benefited by deep representation, easy interpretation, flexible learning and stochastic modeling. This study develops the unsupervised and supervised learning of deep unfolded topic models for document representation and classification. Conventionally, the unsupervised and supervised topic models are inferred via the variational inference algorithm where the model parameters are estimated by maximizing the lower bound of logarithm of marginal likelihood using input documents without and with class labels, respectively. The representation capability or classification accuracy is constrained by the variational lower bound and the tied model parameters across inference procedure. This paper aims to relax these constraints by directly maximizing the end performance criterion and continuously untying the parameters in learning process via deep unfolding inference (DUI). The inference procedure is treated as the layer-wise learning in a deep neural network. The end performance is iteratively improved by using the estimated topic parameters according to the exponentiated updates. Deep learning of topic models is therefore implemented through a back-propagation procedure. Experimental results show the merits of DUI with increasing number of layers compared with variational inference in unsupervised as well as supervised topic models.
DEEP UNFOLDING INFERENCE FOR SUPERVISED TOPIC MODEL
(2016)
Lee, Chao-Hsi
;
Chien, Jen-Tzung
;
電機工程學系
;
Department of Electrical and Computer Engineering
Conventional supervised topic model for multi-class classification is inferred via the variational inference algorithm where the model parameters are estimated by maximizing the lower bound of the logarithm of marginal likelihood function over input documents and labels. The classification accuracy is constrained by the variational lower bound. In this study, we aim to improve the classification accuracy by relaxing this constraint through directly maximizing the negative cross entropy error function via a deep unfolding inference (DUI). The inference procedure for class posterior is treated as the layer-wise learning in a deep neural network. The classification accuracy in DUI is accordingly increased by using the estimated topic parameters according to the exponentiated updates. Deep learning of supervised topic model is achieved through an error back-propagation algorithm. Experimental results show the superiority of DUI to variational Bayes inference in supervised topic model.
DIRICHLET MIXTURE ALLOCATION
(2016)
Chien, Jen-Tzung
;
Lee, Chao-Hsi
;
Tan, Zheng-Hua
;
電機工程學系
;
Department of Electrical and Computer Engineering
The topic model based on latent Dirichlet allocation relies on the prior statistics of topic proportionals for multinomial words. The words in a document are modeled as a random mixture of latent topics which are drawn from a single Dirichlet prior. However, a single Dirichlet distribution may not sufficiently characterize the variations of topic proportionals estimated from the heterogeneous documents. To deal with this concern, we present a Dirichlet mixture allocation (DMA) model which learns latent topics and their proportionals for topic and document clustering by using the prior based on a Dirichlet mixture model. Multiple Dirichlets pave a way to capture the structure of latent variables in learning representation from real-world documents covering a variety of topics. This paper builds a new latent variable model and develops a variational Bayesian inference procedure to learn model parameters consisting of mixture weights, Dirichlet parameters and word multinomials. Experiments on document representation show the merit of the proposed structural learning by increasing the number of Dirichlets in a DMA topic model.
Latent Dirichlet mixture model
(2018-02-22)
Chien, Jen-Tzung
;
Lee, Chao-Hsi
;
Tan, Zheng-Hua
;
電機工程學系
;
Department of Electrical and Computer Engineering
Text representation based on latent topic model is seen as a non-Gaussian problem where the observed words and latent topics are multinomial variables and the topic proportionals are Dirichlet variables. Traditional topic model is established by introducing a single Dirichlet prior to characterize the topic proportionals. The words in a text document are represented by a random mixture of semantic topics. However, in real world, a single Dirichlet distribution may not faithfully reflect the variations of topic proportionals estimated from the heterogeneous documents. To address these variations, we propose a new latent variable model where latent topics and their proportionals are learned by incorporating the prior based on Dirichlet mixture model. The resulting latent Dirichlet mixture model (LDMM) is constructed for topic clustering as well as document clustering. Multiple Dirichlets provide a solution to build structural latent variables in learning representation over a variety of topics. This study carries out the inference for LDMM according to the variational Bayes and the collapsed variational Bayes. Such an unsupervised LDMM is further extended to a supervised LDMM for text classification. Experiments on document representation, summarization and classification show the merit of structural prior in LDMM topic models. (C) 2017 Elsevier B. V. All rights reserved.