Browsing by Author "Kuo, Che-Yu"

Now showing 1 - 3 of 3

MARKOV RECURRENT NEURAL NETWORK LANGUAGE MODEL
(2019-01-01)
Chien, Jen-Tzung
;
Kuo, Che-Yu
;
電機工程學系
;
Department of Electrical and Computer Engineering
Recurrent neural network (RNN) has achieved a great success in language modeling where the temporal information based on deterministic state is continuously extracted and evolved through time. Such a simple deterministic transition function using input-to-hidden and hidden-to-hidden weights is usually insufficient to reflect the diversities and variations of latent variable structure behind the heterogeneous natural language. This paper presents a new stochastic Markov RNN (MRNN) to strengthen the learning capability in language model where the trajectory of word sequences is driven by a neural Markov process with Markov state transitions based on a K-state long short-term memory model. A latent state machine is constructed to characterize the complicated semantics in the structured lexical patterns. Gumbel-softmax is introduced to implement the stochastic backpropatation algorithm with discrete states. The parallel computation for rapid realization of MRNN is presented. The variational Bayesian learning procedure is implemented. Experiments demonstrate the merits of stochastic and diverse representation using MRNN language model where the overhead of parameters and computations is limited.
MARKOV RECURRENT NEURAL NETWORKS
(2018-01-01)
Kuo, Che-Yu
;
Chien, Jen-Tzung
;
電機工程學系
;
Department of Electrical and Computer Engineering
Deep learning has achieved great success in many real-world applications. For speech and language processing, recurrent neural networks are learned to characterize sequential patterns and extract the temporal information based on dynamic states which are evolved through time and stored as an internal memory. Traditionally, simple transition function using input-to-hidden and hidden-to-hidden weights is insufficient. To strengthen the learning capability, it is crucial to explore the diversity of latent structure in sequential signals and learn the stochastic trajectory of signal transitions to improve sequential prediction. This paper proposes the stochastic modeling of transitions in deep sequential learning. Our idea is to enhance latent variable representation by discovering the Markov state transitions in sequential data based on a K-state long short-term memory (LSTM) model. Such a latent state machine is capable of learning the complicated latent semantics in highly structured and heterogeneous sequential data. Gumbel-softmax is introduced to implement stochastic learning procedure with discrete states. Experimental results on visual and text language modeling illustrate the merit of the proposed stochastic transitions in sequential prediction with limited amount of parameters.
STOCHASTIC MARKOV RECURRENT NEURAL NETWORK FOR SOURCE SEPARATION
(2019-01-01)
Chien, Jen-Tzung
;
Kuo, Che-Yu
;
電機工程學系
;
Department of Electrical and Computer Engineering
Monaural source separation based on recurrent neural network is learned to characterize the sequential patterns in source signals based on dynamic states which are propagated through time. The hidden states are assumed to be deterministic along a single path where a shared long short-term memory (LSTM) is used. Such assumptions may not faithfully reflect the randomness and the variety of temporal features in mixed signals. To strengthen the capability of LSTM in source separation, we propose a stochastic Markov LSTM where the regression from the mixed signal to its source signals is learned with a stochastic indicator of Markov state which selects the state-dependent LSTM for signal separation at each time. A set of LSTMs is discovered to capture the structural diversity of temporal signals or the stochastic trajectory of state transitions for sequential prediction. A new state machine is constructed to learn the complicated latent semantics in heterogeneous and structural mappings between mixed signals and source signals. The Gumbel-softmax sampling is implemented in the backpropagation algorithm with discrete Markov states. Experiments on speech enhancement illustrate the merit of the proposed stochastic Markov LSTM in terms of short-term objective intelligibility measure of the separated speech.