This work tackles the problem of learning a set of language specific acousticunits from unlabeled speech recordings given a set of labeled recordings fromother languages. Our approach may be described by the following two stepsprocedure: first the model learns the notion of acoustic units from thelabelled data and then the model uses its knowledge to find new acoustic unitson the target language. We implement this process with the Bayesian SubspaceHidden Markov Model (SHMM), a model akin to the Subspace Gaussian Mixture Model(SGMM) where each low dimensional embedding represents an acoustic unit ratherthan just a HMM's state. The subspace is trained on 3 languages from theGlobalPhone corpus (German, Polish and Spanish) and the AUs are discovered onthe TIMIT corpus. Results, measured in equivalent Phone Error Rate, show thatthis approach significantly outperforms previous HMM based acoustic unitsdiscovery systems and compares favorably with the Variational Auto Encoder-HMM.
Quick Read (beta)
Bayesian Subspace Hidden Markov Model for Acoustic Unit Discovery
This work tackles the problem of learning a set of language specific acoustic units from unlabeled speech recordings given a set of labeled recordings from other languages. Our approach may be described by the following two steps procedure: first the model learns the notion of acoustic units from the labelled data and then the model uses its knowledge to find new acoustic units on the target language. We implement this process with the Bayesian Subspace Hidden Markov Model (SHMM), a model akin to the Subspace Gaussian Mixture Model (SGMM) where each low dimensional embedding represents an acoustic unit rather than just a HMM’s state. The subspace is trained on 3 languages from the GlobalPhone corpus (German, Polish and Spanish) and the AUs are discovered on the TIMIT corpus. Results, measured in equivalent Phone Error Rate, show that this approach significantly outperforms previous HMM based acoustic units discovery systems and compares favorably with the Variational Auto Encoder-HMM.
Bayesian Subspace Hidden Markov Model for Acoustic Unit Discovery
Lucas Ondel, Hari Krishna Vydana, Lukáš Burget, Jan Černocký
Brno University of Technology
Index Terms: Bayesian Inference, Hidden Markov Model, Subspace Model, Variational Bayes, Low-resource languages, Acoustic Unit Discovery
State-of-the-art Automatic Speech Recognition (ASR) systems rely upon very large amount of speech recordings paired with textual transcriptions. While this approach has proven to be very successful, it is however limited to the very few languages having enough resources to train an ASR system. Due to the cost of data collection and transcription, broadening the range of speech technologies to any language remains an unreachable objective. Parallel to the mainstream ASR, there has been a growing interest in the paradigm of unsupervised learning of speech . Unsupervised speech learning attempts to use machine learning techniques to extract various information (phonetic content, speaker identity, …) from unlabeled recordings. While this is considerably harder than standard ASR, solving this problem would have a considerable impact on the field by reducing the amount of human labour necessary to build a full fledged ASR pipeline. It is also important to emphasize that the linguistic diversity is diminishing worldwide. Many languages are now considered endangered and risk to disappear in a near future. Affordable speech technologies could be a precious tool to help linguists and communities to document and preserve these languages.
This work focuses on the specific task of acoustic unit discovery (AUD). Given a collection of unlabeled recordings in a specific language, the task is to learn a set of basic speech units (also called pseudo-phones) to describe the language. AUD algorithms have to solve three problems: to segment the speech, to cluster the segments into units and to infer how many units are necessary to describe the language. Several approaches have been proposed relying upon Bayesian non-parametric version of the Hidden Markov Model (HMM) [2, 3, 4]. An important recent extension of this model is the VAE-HMM [5, 6, 7] which combines the traditional HMM with Variational Auto Encoder . However, most of the AUD algorithms are prone to model speaker/channel or any non-phonetic variability. To address this issue, we propose the Bayesian Subspace HMM (SHMM). The SHMM is an HMM based AUD model in which the parameters of each unit is constrained to be in the phonetic subspace of the total parameter space. This restriction forces the AUD model to focus on the phonetic content of the speech signal and to ignore irrelevant information.
2.1 Standard Acoustic Unit Discovery
Let be the sequence of observed speech frames and be the set of acoustic units. is a sequence of variables indicating to which unit each speech frame is associated, and are model-dependent latent variables. We consider generative models for which the complete likelihood of the data factorizes as:
and the likelihood of a speech frame for a given unit is member of the exponential family of distribution:
where is the -dimensional vector of natural parameters corresponding to one acoustic unit, are the sufficient statistics and is the (log-)normalization constant of the density. Note that the nature of the model for the units (HMM, GMM, Linear Dynamical Model, …) will depend on the value of and the sufficient statistics . In this work we consider that each unit is modeled by an HMM with a GMM for each state’s emission but it can be replaced by any model satisfying Eq. 1 and Eq. 2. Previous works [2, 3, 6] use special cases of this model to perform the AUD. More precisely, one can understand AUD as finding a set of vectors such that the likelihood of the observation is maximized 11 1 These algorithm also learn the number of acoustic units needed to fit the data. This search is difficult because speech recordings encode many factors other than the phonetic information (speaker identity, emotions, environment, …) and the AUD algorithm may maximize the likelihood while modeling non-phonetic information.
2.2 Subspace HMM
To avoid the AUD model to capture non-phonetic information, we proposed the Subspace HMM (SHMM) which constrains the parameters of the acoustic units to live in the phonetic space. This model extends the unsupervised HMM by assuming that the phonetic information of a language is contained in a subspace of the total parameters space. Formally, it is defined as:
where is a differentiable function. We further refine this subspace model by introducing a prior over the subspace’s parameters:
As depicted in Fig. 0(b), the bases of span the subspace containing the phonetic variability. Since the parameters of the acoustic units are constrained to live in a low dimensional subspace, the AUD algorithm can be seen as finding the set of embeddings which maximizes the likelihood of the observations. By constraining the search in the phonetic subspace, we therefore force the algorithm to ignore non-phonetic source of variability.
Note that Subspace Gaussian Mixture Model , Joint Factor Analysis , Subspace Multinomial Model , etc. are special cases of Eq. 3. In fact, Eq. 3 is the general form of any subspace model for which the complete likelihood is a member of the exponential family of distributions. We denote Eq. 3 as the Generalized Subspace Model (GSM) of which the Subspace HMM, like other aforementioned models, is just a special instance. The graphical representation of the (GSM) is depicted in Fig. 0(a). To complete our definition of the SHMM, we need to specify the mapping from to the natural parameters space . In our setting, each unit is modeled by a HMM with a 3 states left-to-right topology and each state has a GMM emissions with Gaussian components with diagonal covariance matrix. For convenience, we introduce the vector which can be decomposed into three parts . is the vector of parameters (before the mapping ) associated with the th HMM state. further decomposes into where is the vector encoding the parameters of the mixture’s weights and and are the vectors encoding the parameters of the mean and covariance matrix of the jth Gaussian component, respectively. We set such that:
where is the elementwise exponential function. One could also include the transition probabilities of the HMM but we kept them as fixed parameters in this work.
2.3 Estimating the phonetic subspace
Unlike previous AUD algorithms, our model requires to specify the phonetic subspace (parameterized by and ) before searching the acoustic units. This is a ”chicken or egg” problem since we need the phonetic subspace to find the pseudo-phones of the language and we need to know the phones of a language to estimate the subspace. However, this problem can be alleviated by observing that many languages in the world have common phones. It is reasonable to believe that the phonetic subspace of language is well approximated by a phonetic subspace estimated from one or several other languages for which we have labeled data. Interestingly, this rationale naturally fits the Bayesian approach of the problem of AUD. Given unlabeled set of observation in a target language , previous Bayesian AUD algorithms try to estimate the inventory of (pseudo-)phones of the target language by estimating:
If we now assume the phonetic subspace to be estimated from the observations of another language with known inventory of phones , the problem can be reformulated as:
The term may be seem as some educated/informative prior which embeds the notion of phone into the AUD algorithm. This educated prior needs to be estimated as well which leads to a two steps procedure for the SHMM AUD algorithm. First, given the labeled data of one or several languages, the prior over the acoustic units is estimated. Informally speaking, we force the model to learn ”what is a phone”. Second the unlabeled data of the target language is clustered into pseudo-phones given the phonetic knowledge acquired by the model during the first step.
The two steps of the training (learning the prior and clustering the units) are carried out by optimizing the same objective function except that when estimating the prior, the acoustic unit transcription of each utterance is known. The presence or absence of the transcription will be reflected in . When there is no transcription, can be understood as a ”pseudo-phone” loop (see  for details) and when the transcription is known then is just the inference graph used for forced alignment in a traditional HMM based ASR system.
Since the estimation of the exact posterior of the model’s parameters is intractable, we use the Variational Bayes (VB) objective function to find an approximate posterior:
where denote the expectation w.r.t. the distribution and is the Kullback-Leibler divergence. Eq. 14 is not tractable for arbitrary distribution we therefore consider the restricted set of distributions with the following mean-field factorization and the given parameterization:
The parameters of the variational posterior over will depend on the type of the model of the acoustic unit. For the case of an HMM, this is the probability to be in a particular state given the sequence of observations. Under these restrictions the optimization reduces to:
Since we assume each unit to be modeled by an HMM, has an analytical solution which can be efficiently calculated using the forward-backward algorithm . has no analytical solution but can be found through a stochastic gradient ascent scheme. Noting that we have:
Finally, we approximate by using the so called re-parameterization trick introduced in :
In practice we use the ADAM optimizer  to update and we use samples to compute the empirical expectation. The parameters are re-estimated every updates of .
3.1 Data, Features and Metrics
We conducted our experiments with the TIMIT  database and 3 languages from the GlobalPhone corpus : German (GE), Polish (PO) and Spanish (SP). For each of the three GlobalPhone languages, we kept only 3000 randomly selected utterances. We used two sets of features: (i) the MFCC features concatenated with their first and second derivatives (ii) the Multi-Lingual bottleNeck (MBN) features trained on 17 Babel’s languages . The set of languages used to train the MBN features does not include English, German, Polish or Spanish. Both set of features were extracted at a rate of 100 Hz. For the case the MBN features, the audio signal was down-sampled to 8kHz.
We evaluated the different AUD algorithms in terms of phonetic segmentation and equivalent Phone Error Rate (eq. PER) ([17, 5]). For the phonetic segmentation we used the standard Recall, Precision and F-score measured against the timing provided in the TIMIT database with the 61 original phones. We tolerated boundary shifted by +- 2 frames (20 milliseconds). To compute the eq. PER, we mapped each acoustic unit to one of the 61 phones it overlaps the most with. Then, we reduced the reference transcription and proposed transcription to the 39 phone set  and computed the PER.
|Model||Features||Prior Language||Recall||Precision||F-score||eq. PER|
|HMM ||MFCC + +||None||-||-||-||65.4|
|VAE-HMM ||MFCC + +||None||-||-||-||58.9|
|VAE-BHMM ||log-mel FBANK + +||None||-||-||-||56.57|
|HMM||MFCC + +||None||66.47||57.81||61.84||64.92|
|SHMM||MFCC + +||GE||75.74||78.98||77.32||58.89|
|SHMM||MFCC + +||GE+PO||73.94||74.47||74.20||58.23|
|SHMM||MFCC + +||GE+PO+SP||75.03||74.00||74.51||56.91|
3.2 Estimating the phonetic subspace
First, we ran a controlled experiment to assess whether the SHMM is able to properly learn the phonetic subspace of a language. In this experiment, we used the MBN features and each HMM state had 8 Gaussian components. First, we trained a Bayesian HMM phone recognizer on the 48 phone set with a flat phonotactic language model on the traditional TIMIT training set (no SA* utterances) and decoded on the test set mapping the phones to the 39 phone set. This phone recognizer achieved 36.4 % Phone Error Rate (PER). This number is very high since we have removed crucial elements of the traditional ASR pipeline (language model, context-dependent phones, …) in order to evaluate the quality the acoustic model. For comparison, we trained a monophone system with a flat phonotactic language model using the Kaldi toolkit  which yielded 37.3 % PER. We then trained an SHMM based phone recognizer with varying subspace dimension using the same training and testing setup as the baseline HMM. We used the baseline model to provide the first estimate of which we modified so that all the Gaussian components within a state have equal responsibility. We pre-trained the subspace for 15000 updates before updating then we re-estimated after every 1000 updates of for 30 iterations.
Results, shown in Fig. 2, indicate that the SHMM is perfectly able to learn the phonetic subspace of a language by compressing the 3861-dimensional 22 2 3 states . 80 is the features dimension, accounts for the mean and the diagonal of the covariance matrix and 7 is the dimension of the per-state mixture weights. parameter space to a subspace as small as 30 dimensions and yet achieving the same PER as the HMM baseline.
3.3 Acoustic Unit Discovery
We now consider the case of unsupervised learning of speech where English is assumed to be a low-resourced language. In this setup, we use the complete TIMIT set (training, development and testing set including the SA* utterances) as the corpus from which to extract acoustic units. In this experiment, all the HMM/SHMM have 4 Gaussian components per state. Our baseline is the HMM based AUD system described in  and the VAE-(B)HMM based AUD system proposed in [5, 6]. We compare these baselines with 3 SHMM based AUD models for which the posterior of the phonetic subspace was estimated using: (i) German language (ii) German and Polish languages (iii) German, Polish and Spanish languages. For each case the phonetic subspace had 35, 70 and 100 dimensions respectively. Note that the choice of the languages and the order of combination was arbitrary and it is likely that choosing languages closely related to the target language would be beneficial. We considered all the phones of all the languages to be unique and didn’t merge them while estimating the subspace. The posteriors of the embeddings corresponding to the German, Polish and Spanish phones were discarded before the AUD clustering.
The results are presented in Table 1 and differ significantly depending on the input features. The SHMM always benefits from learning the phonetic subspace in terms of eq. PER. Interestingly, the baseline HMM fails to benefit from the MBN features as it underperforms compared to the HMM trained on MFCC features. The SHMM, thanks to its subspace, learns from other languages to fully exploit the discriminatively trained features. Regarding the segmentation evaluation, the SHMM better segments the speech compared to the simple HMM. However, we observe that using more than one language does not necessarily improves the segmentation. Also, contrary to the eq. PER, the MBN features does not seem to be ideal to get accurate segmentation.
Finally, we tried to label the TIMIT corpus with a HMM phone-recognizer (MBN features) trained on German, German and Polish and German, Polish and Spanish and we interpreted the output phones as acoustic units. For these 3 models the eq. PER was 61.22 % (GE), 66.47 % (GE+PO) and 71.96 % (GE+PO+SP). Contrary to the SHMM, this naive approach does not benefit from having more languages.
We proposed a new model for AUD: the Subspace HMM. Unlike other AUD models the SHMM is trained in a supervised fashion on one or several languages to learn the notion of ”phone”. This phonetic knowledge is encoded into a non-linear subspace of the total parameter space. Then, the SHMM searches a set of of acoustic units in this subspace which maximizes the likelihood of the observations of the target language. The SHMM outperforms the HMM based AUD and is competitive with the VAE-HMM. When using discriminatively trained features, the SHMM achieves 49.2 % equivalent PER on TIMIT whithout any supervision in the target language.
The work was supported by Czech National Science Foundation (GACR) project ”NEUREM3” No. 19-26934X, Czech Ministry of Interior project No. VI20152020025 ”DRAPAK”, and Czech Ministry of Education, Youth and Sports from the National Programme of Sustainability (NPU II) project ”IT4Innovations excellence in science - LQ1602”. This work was also supported by by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA) MATERIAL program, via Air Force Research Laboratory (AFRL) contract # FA8650-17-C-9118. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ODNI, IARPA, AFRL or the U.S. Government.
The authors would like to thank Kateřina Žmolíková for her corrections and insightful comments.
-  J. R. Glass, “Towards unsupervised speech processing,” in ISSPA. IEEE, 2012, pp. 1–4.
-  C. Lee and J. R. Glass, “A nonparametric bayesian approach to acoustic model discovery,” in ACL (1). The Association for Computer Linguistics, 2012, pp. 40–49.
-  L. Ondel, L. Burget, and J. Cernocky, “Variational inference for acoustic unit discovery,” in Procedia Computer Science, vol. 2016, no. 81. Elsevier Science, 2016, pp. 80–86.
-  L. Ondel, L. Burget, J. Černocký, and S. Kesiraju, “Bayesian phonotactic language model for acoustic unit discovery,” in Proceedings of ICASSP 2017. IEEE Signal Processing Society, 2017, pp. 5750–5754.
-  J. Ebbers, J. Heymann, L. Drude, T. Glarner, R. Haeb-Umbach, and B. Raj, “Hidden markov model variational autoencoder for acoustic unit discovery,” in Interspeech 2017, 18th Annual Conference of the International Speech Communication Association, Stockholm, Sweden, August 20-24, 2017, 2017, pp. 488–492. [Online]. Available: http://www.isca-speech.org/archive/Interspeech_2017/abstracts/1160.html
-  T. Glarner, P. Hanebrink, J. Ebbers, and R. Haeb-Umbach, “Full bayesian hidden markov model variational autoencoder for acoustic unit discovery,” in Interspeech. ISCA, 2018, pp. 2688–2692.
-  L. Ondel, P. Godard, L. Besacier, E. Larsen, M. Hasegawa-Johnson, O. Scharenborg, E. Dupoux, L. Burget, F. Yvon, and S. Khudanpur, “Bayesian Models for Unit Discovery on a very Low Resource Language,” in IEEE International Conference on Acoustics, Speech and Signal Processing, ser. ICASSP, Calgary, Canada, 2018. [Online]. Available: sources/Ondel18bayesian.pdf
-  D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” in ICLR, 2014.
-  D. Povey, A. Ghoshal, G. Boulianne, N. Goel, M. Hannemann, Y. Qian, P. Schwarz, and G. Stemmer, “The kaldi speech recognition toolkit,” in In IEEE 2011 workshop, 2011.
-  P. Kenny, “Joint factor analysis of speaker and session variability: Theory and algorithms,” Tech. Rep., 2005.
-  S. Kesiraju, L. Burget, I. Szőke, and J. Černocký, “Learning document representations using subspace multinomial model,” in Proceedings of Interspeech 2016. International Speech Communication Association, 2016, pp. 700–704.
-  L. R. Rabiner, “A tutorial on hidden markov models and selected applications in speech recognition,” in PROCEEDINGS OF THE IEEE, 1989, pp. 257–286.
-  D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in ICLR, 2015.
-  J. S. Garofolo, L. F. Lamel, W. M. Fisher, J. G. Fiscus, D. S. Pallett, and N. L. Dahlgren, “Darpa timit acoustic phonetic continuous speech corpus cdrom,” 1993.
-  T. Schultz, N. T. Vu, and T. Schlippe, “Globalphone: A multilingual text & speech database in 20 languages.” in ICASSP. IEEE, 2013, pp. 8126–8130.
-  R. Fer, P. Matejka, F. Grezl, O. Plchot, K. Vesely, and J. H. Cernocky, “Multilingually trained bottleneck features in spoken language recognition,” Computer Speech & Language, vol. 46, no. Supplement C, pp. 252 – 267, 2017.
-  H. Kamper, A. Jansen, and S. Goldwater, “A segmental framework for fully-unsupervised large-vocabulary speech recognition,” Computer Speech & Language, vol. 46, pp. 154–174, 2017.
-  H.-W. H. Kai-Fu Lee, “Speaker-independent phone recognition using hidden markov models,” vol. IEEE Transactions on accoustics, speech and signal processing, no. 37, pp. 1641–1648, 1989.