Exploiting Cross-Lingual Speaker and Phonetic Diversity for Unsupervised Subword Modeling

Abstract

This research addresses the problem of acoustic modeling of low-resource languages for which transcribed training data is absent. The goal is to learn robust frame-level feature representations that can be used to identify and distinguish subword-level speech units. The proposed feature representations comprise various types of multilingual bottleneck features (BNFs) that are obtained via multi-task learning of deep neural networks (MTL-DNN). One of the key problems is how to acquire high-quality frame labels for untranscribed training data to facilitate supervised DNN training. It is shown that learning of robust BNF representations can be achieved by effectively leveraging transcribed speech data and well-trained automatic speech recognition (ASR) systems from one or more out-of-domain (resource-rich) languages. Out-of-domain ASR systems can be applied to perform speaker adaptation with untranscribed training data of the target language, and to decode the training speech into frame-level labels for DNN training. It is also found that better frame labels can be generated by considering temporal dependency in speech when performing frame clustering. The proposed methods of feature learning are evaluated on the standard task of unsupervised subword modeling in Track 1 of the ZeroSpeech 2017 Challenge. The best performance achieved by our system is $9.7\%$ in terms of across-speaker triphone minimal-pair ABX error rate, which is comparable to the best systems reported recently. Lastly, our investigation reveals that the closeness between target languages and out-of-domain languages and the amount of available training data for individual target languages could have significant impact on the goodness of learned features.

I Introduction

State-of-the-art automatic speech recognition (ASR) systems have demonstrated fairly impressive performance in terms of word accuracy [1, 2]. This is mainly attributed to the advances of deep neural network (DNN) based acoustic models (AMs) and language models (LMs) [3, 4]. Typically a well-trained DNN-based AM requires hundreds to thousands of hours of transcribed speech. As a matter of fact, high-performance ASR systems are available only for major languages [5]. Even for resource-rich languages, preparing transcriptions for available training data is a time-consuming task that involves considerable human effort. For many languages in the world, very little or no transcribed speech is available [6], and conventional acoustic modeling techniques are simply not applicable.

Unsupervised speech modeling is the task of building sub-word or word-level AMs, when only untranscribed speech are available for training [7, 8, 9]. This is also known as the zero-resource problem, which has attracted increasing research interest in recent years. The Zero Resource Speech Challenge 2015 (ZeroSpeech 2015) [9] and 2017 (ZeroSpeech 2017) [6] precisely focused on unsupervised speech modeling. ZeroSpeech 2017 was organized to tackle two sub-problems, namely unsupervised subword modeling (Track 1) and spoken term discovery (STD) (Track 2). The present study addresses the Track 1 problem and aims to learn frame-level feature representation that is effective in identifying and discriminating subword-level units and robust to irrelevant factors, e.g., speaker and/or channel variation, emotion, etc. Robust feature representations obtained by learning from data have been found to be preferable to conventional spectral features like Mel-frequency cepstral coefficients (MFCCs) for downstream applications [10, 11].

DNN models are commonly adopted in frame-level feature learning for unsupervised subword modeling. A DNN model is typically trained using available speech data. The learned features are obtained either from a designated low-dimension hidden layer of the DNN, known as the bottleneck features (BNFs) [12], or from the softmax output layer, known as the posterior features or posteriorgram [13]. To facilitate supervised training of the DNN, target labels of training speech are needed. In zero-resource scenarios, the key problem is how to generate informative frame-level labels in the absence of speech transcription. One of the possible approaches is based on unsupervised clustering of training speech. The frame-level cluster indices can be used as target labels for DNN training [12, 11, 13]. Another approach seeks to use pre-trained out-of-domain ASR systems to tokenize untranscribed in-domain speech and hence each frame is assigned with an ASR senone label [5, 14]. Fully unsupervised [13] or weakly supervised [15, 16, 17] methods for DNN training were also reported in the research on acoustic modeling for low-resource languages.

The present study adopts the general framework of supervised DNN training for the purpose of extracting BNF as the learned feature representation. We attempt to improve the efficacy and performance of learned features along two directions. First, advanced unsupervised acoustic modeling techniques are explored to generate initial frame-level labels for supervised DNN training. Second, speaker adaptation techniques are applied to make input speech features more robust to speaker variation.

Dirichlet process Gaussian mixture model (DPGMM) is commonly used for clustering of unlabelled speech frames [18]. It demonstrated superior performance on the tasks in ZeroSpeech Challenges [19, 20]. However, DPGMM clustering, as well as other conventional clustering algorithms like $k$-means [21] and GMM [13], assumes that neighboring speech frames are independent of each other. This is obviously not in accordance with the nature of speech. To address this limitation, a full-fledged Gaussian mixture model-hidden Markov model (GMM-HMM) AM is trained to capture contextual information in speech. The transcriptions required for GMM-HMM training are initialized via DPGMM clustering. Following the terminology in [22], this model is referred to as DPGMM-HMM. We use the DPGMM-HMM AM to generate frame-level labels to support BNF representation learning. In [22], a similar approach was adopted for learning feature-space maximum likelihood linear regression (fMLLR) and posteriorgram features.

In unsupervised subword modeling, the outcome of frame clustering ideally comprises a set of clusters that correspond to phoneme-related speech units. The underlying assumption is that speech frames identified as the same phoneme should have homogeneous acoustic properties. In practice, speaker and environment variations would inevitably impact the reliability of frame clustering results. Our preliminary experiments showed that applying DPGMM typically results in an excessive number of fine-grained clusters. Similar observations were reported in [23, 24]. These over-fragmented clusters may adversely affect the effectiveness of unsupervised speech modeling. In this work we develop and apply a new algorithm to filter out infrequent labels in DPGMM clustering results, and experimentally validate its effectiveness.

In addition to the DPGMM-HMM labels, a different type of frame labels can be obtained using one or more out-of-domain ASR systems [5, 14]. While the DPGMM-HMM frame labels incorporate statistical information of the acoustic properties of target speech, the ASR senone labels leverage the phonetic information acquired from out-of-domain languages. We propose to exploit their complementarity in DNN based feature learning by applying the multi-task learning strategy [25].

Numerous studies have demonstrated the benefit of applying speaker adaptation on input features for unsupervised subword modeling [12, 26]. In the present study, we propose to exploit cross-lingual speech data in fMLLR-based speaker adaptation. Specifically, transcribed speech from a resource-rich language is used to train an out-of-domain ASR system. This system is then applied to the zero-resource target languages for estimating linear discriminant analysis (LDA), maximum likelihood linear transform (MLLT) and fMLLR transforms on conventional spectral features. We advocate that this approach is effective and practically desirable as transcribed speech data of resource-rich languages are relatively easy to access.

The remainder of this paper is organized as follows. Section II provides a review of related works on unsupervised subword modeling with untranscribed speech. In Section III, we provide detailed description on the proposed approaches to feature learning. Section IV introduces experimental design on ZeroSpeech 2017 development data. Section V discusses and analyzes experimental results. Section VI gives the conclusions.

II Related works

II-A Deep learning approaches to unsupervised subword modeling

A variety of DNN models have been investigated towards unsupervised subword modeling. They include multi-layer perceptron (MLP) [12], auto-encoder (AE) [13], correspondence AE (cAE), denoising AE (dAE) [27], variational AE (VAE) [28] and siamese network [29]. In terms of training strategies, these models can be classified into three categories, namely, supervised (MLP), unsupervised (AE, VAE, dAE) and weakly/pair-wise supervised (cAE, siamese network). Supervised DNN training requires frame-level labels for all training speech, which could be obtained either via a clustering process or exploiting out-of-domain resources. In [12, 11], DPGMM clustering was performed on conventional short-time spectral features of target speech, followed by multilingual DNN training to obtain the BNF representation. In [13], GMM-universal background model (GMM-UBM) was used to generate frame labels. A DNN was trained using these labels to generate BNF or posteriorgram representation. In [5, 14], language-mismatched ASR systems were utilized to decode the target speech, and frame labels were generated from the ASR decoding lattices. In [30], BNF representation was generated by applying multi-task learning with both in-domain and out-of-domain data [25]. The frame labels for out-of-domain data were obtained by HMM forced alignment, while the labels for in-domain data were from DPGMM clustering [12]. In [5, 31, 14], a DNN AM was trained with transcribed data of an out-of-domain language, and used to extract BNFs or posteriorgrams from target speech.

Unsupervised DNN training does not require any kind of target labels. For example, an AE model generates non-linear embeddings of input speech and meanwhile learn to reconstruct the same speech from the embeddings. Recently, weakly-supervised model training is studied extensively [15, 16, 17]. In the cAE model [27], a pair of speech segments that contain the same linguistic unit (word or subword) are used as the input and output for training, with the objective of minimizing the reconstruction error. In a siamese network, the input comprises two speech segments. The network is trained to determine whether the segments are from the same linguistic unit or not. These models were shown to achieve better performance than unsupervised models [27]. However, for zero-resource languages, such pair-wise knowledge may not be directly available.

II-B Unsupervised subword modeling without using DNN

There were numerous studies on unsupervised subword modeling without involving deep learning models. In these studies, clustering of short-time frame features is an important first step. After frame clustering, each cluster is represented by a learned probability distribution, and the cluster posteriorgram can be regarded as the learned representation for subword modeling. Frame clustering could be done straightforwardly by applying $k$-means [21], GMM [32] and DPGMM [19] algorithms. In [19], DPGMM clustering was applied to a zero-resource target language. An extension of this approach was reported in [20], where clustering was performed with fMLLR-based speaker-adapted features. In [32], GMM posteriorgram and HMM posteriorgram were compared, where the HMM was trained based on GMM-UBM clustering results.

To better retain temporal dependency in speech, frame clustering can be embodied in segment level. Initial segmentation of speech utterances could be obtained by hierarchical agglomerative clustering [33], or using language-mismatched phone recognizers [34, 35]. Subsequently a fixed-length feature vector is derived to represent each speech segment. Clustering of segment-level feature vectors was tackled using a range of algorithms, including vector quantization (VQ) [36], segmental GMM (SGMM) [37], spectral clustering [38] and graph clustering [39]. In [40], segmentation and clustering were integrated as a jointly optimized process.

The present study is on one hand largely based on DNN modeling of speech, and on the other hand incorporates the ideas of frame clustering (as the initial tokenization) [19], fMLLR-based speaker adaptation [20], and use of HMM to capture temporal dependency [32].

II-C Optimizing DPGMM clustering

DPGMM clustering has been shown to be a preferred method of frame labeling for unsupervised subword modeling [19, 20]. Nevertheless, one shortcoming of DPGMM is that it tends to produce over-fragmented speech units [23, 24]. Different approaches have been proposed to tackle this problem. In [23], DPGMMs were replaced by the Dirichlet process mixture of mixtures model (DPMoMM) to enable multi-modal cluster inference. In [24], small-sized clusters were merged based on low functional load [41, 42]. In our work, this problem is tackled by a label filtering algorithm.

DPGMM for frame labeling could be optimized at input feature level. Conventional spectral features like MFCC [19] and perceptual linear prediction (PLP) [26] were commonly used as the initial representations of target speech. Albeit straightforward, these features are considered sub-optimal for unsupervised subword modeling, as they contain a lot of irrelevant information such as speaker identity and emotion. Heck et al. [43, 26] found that fMLLR transforms can noticeably suppress speaker-related feature variation, and advocated the importance of speaker adaptation in the concerned task. To enable supervised estimation of fMLLRs, clustering results on spectral features were taken as pseudo transcriptions. Chen et al. [12] showed that vocal tract length normalization (VTLN) on top of spectral features contribute to generating more robust DPGMM frame labels. In our study, fMLLR features are estimated by exploiting an out-of-domain ASR system.

III Proposed System

The proposed system framework for unsupervised subword modeling of zero-resource languages is illustrated as in Fig. 1. It comprises three modules, namely, speaker-adapted feature extraction, unsupervised acoustic modeling, and multi-task BNF learning. Speech frames of the target language are first processed by an out-of-domain ASR system, where VTLN, LDA, MLLT and fMLLR transforms are estimated sequentially. The DPGMM clustering algorithm is applied to the fMLLR features of target speech. The resulted frame labels are post-processed by a label filtering algorithm and then used for context-dependent GMM-HMM (CD-GMM-HMM) acoustic modeling. The trained AMs forced align target speech to generate DPGMM-HMM alignments. Subsequently, an MTL-DNN is trained to generate BNFs for subword modeling. The training tasks of MTL include DPGMM-HMM alignment prediction and language-mismatched label prediction of multiple target languages. The language-mismatched labels are generated by multiple out-of-domain ASR systems.

The proposed system design emphasizes on leveraging speech data resources from out-of-domain languages [5, 14]. This is realized in the following aspects:

• Use out-of-domain data to perform fMLLR speaker adaptation on target speech.

• Use out-of-domain ASR systems to generate frame labels to facilitate multi-task DNN training.

• Use an out-of-domain DNN AM to extract BNFs.

Fig. 1: The proposed framework of unsupervised subword modeling.

III-A Speaker adaptation with out-of-domain data

For resource-rich languages, a large amount of transcribed and speaker-annotated speech data are readily available. We propose to utilize these out-of-domain data to model speaker variation in untranscribed speech of the target speech. A conventional CD-GMM-HMM AM is trained using the out-of-domain data. Based on this model, VTLN, LDA, MLLT and fMLLR transforms can be estimated. Subsequently, CD-GMM-HMM AMs with speaker adaptive training (CD-GMM-HMM-SAT) are trained and used to estimate fMLLR transforms for target speech utterances. It must be noted that the estimated fMLLR features of target speech could be directly used for subword modeling. They are expected to provide a better baseline than the conventional spectral features like MFCCs or PLPs.

III-B Frame labeling

III-B1 DPGMM clustering

DPGMM is a non-parametric Bayesian extension to GMM, where a Dirichlet process prior replaces the vanilla GMM. One advantage of DPGMM clustering is that the cluster number does not need to be pre-defined. Let us consider $M$ zero-resource target languages. For an utterance from the $i$-th language, the frame-level features are denoted as $\{{𝒙}_{\mathrm{𝟏}}^{𝒊}\mathbf{,}{𝒙}_{\mathrm{𝟐}}^{𝒊}\mathbf{,}\mathbf{\dots }\mathbf{,}{𝒙}_{𝑳}^{𝒊}\}$, where $L$ is the number of frames in the utterance. By applying DPGMM clustering, $K$ clusters are obtained and represented with $k$ Gaussian components. The frame labels $\{l_1^i,l_2^i,\mathrm{\dots },l_L^i\}$ are given as,

$$l_t^i=\underset{1\le k\le K}{\arg \max }\mathrm{Prob}(k|{𝒙}_{𝒕}^{𝒊}),$$ (1)

where $\mathrm{Prob}(k|{𝒙}_{𝒕}^{𝒊})$ denotes the posterior probability of ${𝒙}_{𝒕}^{𝒊}$ with respect to the $k$-th Gaussian component. The inference of DPGMM parameters can be performed using the algorithm as described in [18].

III-B2 Out-of-domain ASR decoding

Given a speech utterance in the target language, an out-of-domain ASR system can be applied to generate a sequence of phone-level or state-level labels [14]. The idea can be naturally extended to using multiple out-of-domain ASR systems and desirably providing a wide coverage of phonetic diversity. The outcome of ASR decoding depends on the relative weighting of AM and LM. In our work, the LM is assigned a very small weight, such that the acquired frame labels mainly reflect acoustic properties of the target speech being modeled.

III-C DPGMM label filtering

Fig. 2: Example of cluster size sorting.

For a specific target language, let us assume that $K$ Gaussian components (clusters) are obtained by DPGMM clustering. The frame labels are denoted as $l_1,l_2,\mathrm{\dots },l_N$ for an $N$-frame utterance. Let $c_k$ be the number of frames labeled as cluster $k$, i.e.,

$$c_k=\sum _{i=1}^N\mathrm{𝟙}(l_i=k),k\in \{1,2,\mathrm{\dots },K\},$$ (2)

where $\mathrm{𝟙}(\cdot )$ is the indicator function.

The elements in $\{c_1,c_2,\mathrm{\dots },c_K\}$ are sorted in descending order into $\{{\widehat{c}}_1,{\widehat{c}}_2,\mathrm{\dots },{\widehat{c}}_K|{\widehat{c}}_1\ge {\widehat{c}}_2\ge \mathrm{\dots }\ge {\widehat{c}}_K\}$. $m(\cdot )$ denotes the index mapping function, i.e.,

$${\widehat{c}}_k=c_{m(k)}.$$ (3)

Fig. 2 gives an example of cluster size sorting.

Let $P$ be the percentage of frame labels that we aim to retain. These frames are from $K_{cut}$ “dominant” clusters, where

$$K_{\mathrm{cut}}=\underset{K^{\prime }}{\arg \min }\frac{{\sum }_{k=1}^{K^{\prime }}{\widehat{c}}_k}{N}\ge P.$$ (4)

$𝒪$ denotes the collection of all frame labels that are removed, i.e.,

$𝒪$ $=\{l_i:l_i\in ℱ,i\in \{1,2,\mathrm{\dots },N\}\},$ (5)

where

$ℱ$ $=\{m(K_{\mathrm{cut}}+1),\mathrm{\dots },m(K)\}.$ (6)

$ℱ$ contains indices of $K-K_{cut}$ clusters that are the least frequent to occur. Frames assigned to these clusters are considered as outliers.

In the extreme case when $P$ is set to $1$, $ℱ$ and $𝒪$ will be empty sets. The smaller the value of $P$, the larger the proportion of filtered frame labels. The label filtering algorithm is summarized as in Algorithm III-C. DPGMM label filtering algorithm Input: $l_1,l_2,\mathrm{\dots },l_N$, $P$
Output: $𝒪$ [1] Calculate $c_k$ by Equation (2). Sort $\{c_1,c_2,\mathrm{\dots },c_K\}$ in descending order. Calculate $m(k)$ by Equation (3). Calculate $K_{\mathrm{cut}}$ by Equation (4) and $P$. Select a subset of $l_1,l_2,\mathrm{\dots },l_N$ as $𝒪$, by Equation (5)&(6).Frame labels that are removed.

III-D DPGMM-HMM acoustic modeling

Each DPGMM cluster can be regarded as a pseudo phone. The sequence of DPGMM frame labels (after filtering) can be converted into a pseudo transcription by collapsing neighboring duplicated labels, e.g., “1,3,3,3,7,10,10” $\to$ “1,3,7,10”. Based on the pseudo transcription, HMM acoustic modeling is done by following the standard supervised training pipeline, i.e., proceeding from monophone model training with uniform time alignment to context-dependent GMM-HMM (CD-GMM-HMM). The trained AM is used to produce time alignment information for DNN-based subword discriminative modeling (will be discussed in Section III-E). To be distinguished from the DPGMM frame labels, the frame labels obtained from the HMM forced alignment are referred to as DPGMM-HMM labels.

Although the DPGMM labels could be directly used for supervised DNN acoustic modeling [12, 14], we expect that DPGMM-HMM labels are more reasonable as they are derived with consideration on contextual dependency of speech.

III-E Multi-task learning for BNFs

The bottleneck feature (BNF) is a type of representation obtained from a designated low-dimension hidden layer of a DNN. In ASR applications, BNFs have been shown to provide a compact and phonetically-discriminative representation of input speech, and be effective in suppressing linguistically-irrelevant variations [44]. In the context of zero-resource speech modeling, BNFs have also been widely investigated [12, 5, 14, 17].

The proposed MTL-DNN is depicted in Fig 3. The DNN training involves a total of $M+N$ tasks, which involves $M$ zero-resource target languages and $N$ out-of-domain ASR systems. Each of the tasks is represented by a task-specific softmax output layer in the DNN. The hidden layers, including a low-dimension linear BN layer, are shared across all tasks. For the zero-resource language tasks, state-level or phone-level DPGMM-HMM labels are used as target labels. The decoding output from each of the out-of-domain ASR systems provides one set of frame-level labels for MTL.

For the MTL-DNN trained only on the $M$ target language tasks, the extracted BNFs are referred to as multilingual unsupervised BNFs (MUBNFs). When out-of-domain ASR tasks are added, the BNFs are named language-independent BNFs (LI-BNFs). In the case that only the out-of-domain ASR tasks are involved, the extracted BNFs are referred to as out-of-domain supervised BNFs (OSBNFs).

The DPGMM-HMM labels are obtained through statistical modeling of target speech. The ASR senone labels leverage the phonetic knowledge acquired from out-of-domain languages. It is expected that they would contribute complementarily in feature learning. Learning from speech of multiple languages would result in a language-independent BNF representation that is more generalizable to unknown languages.

For the shared-hidden-layer structure in the MTL-DNN, multi-layer perceptron (MLP) is commonly used [12, 13, 31, 14]. In this study, in addition to MLP, we investigate the use of long short-term memory (LSTM) [45] and bi-directional LSTM (BLSTM) [46], which were shown to perform better than MLP in conventional supervised acoustic modeling.

Fig. 3: MTL-DNN for extracting LI-BNF, MUBNF and OSBNF. The term “OOD” stands for out-of-domain.

On the other hand, BNF representation can also be obtained from the DNN AM pre-trained for a resource-rich language [5]. This is considered as a transfer learning approach [47]. This transfer learning BNF (TLBNF) is expected to further enrich the feature representation and will be jointly used with MUBNF, OSBNF and LI-BNF for subword modeling.

IV Experimental setup

IV-A Dataset and evaluation metric

Experiments are carried out with the development data of ZeroSpeech 2017 Track 1 [6]. The data covers three target languages, namely English, French and Mandarin. For each language, there are separate training set and test set of untranscribed speech. Speaker identity information is provided for the train sets but not available for the test sets. The test data are organized into subsets of different utterance lengths: $1$ second, $10$ second and $120$ second. Detailed information about the dataset are given as in Table I.

TABLE I: Development data of ZeroSpeech 2017 Track 1

	Training		Test
	Duration (hours)	# speakers	Duration (hours)
English	$45$	$60$	$27$
French	$24$	$18$	$18$
Mandarin	$2.5$	$8$	$25$

The evaluation metric adopted for ZeroSpeech 2017 Track 1 task is the ABX subword discriminability. Inspired by the match-to-sample task in human psychophysics, it is a simple method to measure the discriminability between two categories of speech units [9]. The basic ABX task is to decide whether $X$ belongs to $x$ or $y$, if $A$ belongs to $x$ and $B$ belongs to $y$, where $A$, $B$ and $X$ are three data samples, $x$ and $y$ are the two pattern categories concerned. The performance evaluation in ZeroSpeech 2017 is carried out on the triphone minimal-pair task. A triphone minimal pair comprises two triphone sequences, which have different center phones and identical context phones, for examples, “beg”-“bag”, “api”-“ati”. Discriminating triphone minimal pairs is a non-trivial task. The performance of a feature representation on the triphone minimal-pair ABX task is considered a good indicator of its efficacy in speech modeling [48].

Let $x$ and $y$ denote a pair of triphone categories. Consider three speech segments $A$, $B$ and $X$, where $A$ and $X$ belong to category $x$ and $Y$ belongs to $y$. The ABX discriminability of $x$ from $y$ is measured in terms of the ABX error rate $ϵ(x,y)$, which is defined as the probability that the distance of $A$ from $X$ is greater than that of $B$ from $X$, i.e.,

(7)

where $S(x)$ and $S(y)$ denote the sets of features that represent triphone categories $x$ and $y$, respectively. $d(\cdot ,\cdot )$ denotes the dissimilarity between two speech segments, which is computed by dynamic time warping (DTW) in our study. The frame-level dissimilarity measure used for DTW scoring is the cosine distance. Note that $ϵ(x,y)$ is asymmetric to $x$ and $y$. A symmetric form can be defined by taking average of $ϵ(x,y)$ and $ϵ(y,x)$. The overall ABX error rate is obtained by averaging over all triphone categories and speakers in the test set. A high ABX error rate means that the feature representation is not discriminative, and vice versa. Intuitively, the error rate should be no larger than $50\%$, as by random decision, the expectation of ABX error rate is $50\%$.

Two evaluation conditions were defined in ZeroSpeech 2017, namely within-speaker and across-speaker. In both conditions, the segments $A$ and $B$ to be evaluated are generated by the same speaker. In the within-speaker condition, segment $X$ is generated by the same speaker as $A$ and $B$; In the across-speaker condition, $X$ is generated by a speaker different from $A$ and $B$.

IV-B Out-of-domain ASR systems

Four out-of-domain ASR systems are utilized and investigated in our experiments. They cover the languages of Cantonese (CA), Czech (CZ), Hungarian (HU) and Russian (RU). The Cantonese ASR is trained with the CUSENT database [49]. The database contains $20,378$ training utterances from $34$ male and $34$ female speakers, with a total of $19.3$ hours of speech. The Kaldi toolkit [50] is used to train two versions of AMs: CD-GMM-HMM-SAT and DNN-HMM. DNN-HMM training labels are acquired from CD-GMM-HMM-SAT time alignment. The input features for CD-GMM-HMM-SAT are $40$-dimension fMLLRs, and the input features for DNN-HMM are fMLLRs with $\pm 5$ splicing. The fMLLR features are estimated during CD-GMM-HMM-SAT training. Specifically, VTLN is estimated towards $39$-dimension MFCCs+$\mathrm{\Delta }$+$\mathrm{\Delta }\mathrm{\Delta }$. The resulted features with $\pm 3$ splicing are used to estimate $40$-dimension LDA and MLLT. Finally, fMLLR transforms are estimated. MFCC features are computed using a $25$-ms Hamming window and a $10$-ms frame shift. Per-utterance cepstral mean variance normalization (CMVN) is applied to MFCCs. The DNN-HMM model for Cantonese is a $7$-layer MLP, with layer configuration $440$-$1024\times 5$-$40$-$1024$-$2462$. The dimension of the output layer is determined by the number of CD-HMM states modeled by CD-GMM-HMM-SAT. Hidden layers are activated with sigmoid function, except for the $40$-dimension linear BN layer. The network is trained to optimize the cross-entropy criterion. A syllable trigram LM trained with transcriptions of CUSENT training data is used during decoding. The LM is trained with SRILM [51].

The other three out-of-domain ASR systems are all phone recognizers developed by Brno University of Technology [52]. The recognizers adopt a $3$-layer MLP structure, in which the first two are sigmoid layers and the third is a softmax layer. They were trained with the SpeechDat-E databases [53]. The numbers of modeled phones in Czech, Hungarian and Russian are $45,61$ and $52$, respectively. The training data sizes are $9.7$, $7.9$ and $14.0$ hours, respectively. The cross-entropy criterion was used for MLP training.

IV-C Speaker adaptation of target speech

The Cantonese ASR system is used to perform fMLLR-based speaker adaptation of target speech on the $39$-dimension MFCC features in a two-pass procedure. In the first pass, input speech utterances are decoded in a speaker-independent manner, using unadapted features, from which initial fMLLR transforms are estimated. In the second pass, input speech are decoded with initial fMLLRs in a speaker-adaptive manner. After the decoding, final fMLLR transforms for target speech utterances are estimated. The dimension of fMLLR features is $40$.

IV-D DPGMM frame clustering and label filtering

Speech frames for different languages are clustered separately by the DPGMM algorithm based on the $40$-dimension fMLLR features. The implementation of DPGMM clustering is performed using an open-source tool developed by Chang et al. [18]. For the three target languages, namely English, French and Mandarin, the numbers of iterations of clustering were $120,200$ and $3000$ respectively. The numbers of iterations for English and French are determined by preliminary experiments. Specifically, the iterations for English ranging in $\{40,80,\mathrm{\dots },680\}$ and for French ranging in $\{40,80,\mathrm{\dots },400\}$ were tested. The optimal numbers of iterations were $120$ and $200$ respectively. For Mandarin, the number of iterations was empirically determined. The resulted numbers of DPGMM clusters for English, French and Mandarin are $1118,1345$ and $596$, respectively. Each frame is assigned a cluster label. Fig. 4 shows the results of clustering in the form of cumulative distribution function (CDF) for the three target languages. The clusters are sorted according to their cluster size in descending order. In other words, each point $(K_i,Q_i)$ on the CDF represents the proportion of frame labels $Q_i$ that the largest $K_i$ clusters cover.

For label filtering, we evaluated different thresholds on the percentage of preserved labels, with the value of $P$ ranging from $0.6$ to $0.95$, with the step size of $0.05$. After filtering, the frame-level label sequences are converted into pseudo transcriptions, for the training of DPGMM-HMM AMs (in Section IV-E).

Fig. 4: Clustering results in the form of cumulative distribution function for the three target languages. Clusters are sorted according to cluster size in descending order.

DPGMM clustering was also tested with MFCCs as input features. The numbers of iterations for MFCC clustering are $200,240$ and $3000$ for English, French and Mandarin respectively, and the resulted numbers of DPGMM clusters are $1554,1541$ and $381$.

IV-E DPGMM-HMM and MTL-DNN training

DPGMM-HMM AMs are trained from scratch with pseudo transcriptions. Different from the conventional $3$-state HMM topology, during DPGMM-HMM training we set $1$-state HMM for each pseudo phone. This prevents the problem of unsuccessful forced alignments, as the numbers of pseudo phones for target languages are significantly larger than the number of phones for a typical language. The input features for DPGMM-HMM are $40$-dimension fMLLRs estimated by the Cantonese ASR. The training procedure follows the standard pipeline as in Kaldi s5 recipe, i.e., starting from CI-GMM-HMM to CD-GMM-HMM, followed by VTLN and fMLLR-based SAT. After training, the numbers of CD-HMM states for English, French and Mandarin are $2818,2856$ and $2688$, respectively.

The MTL-DNN model is trained with all the three target zero-resource languages, from which BNFs are extracted and evaluated by the ABX subword discriminability task. There are two types of tasks for MTL, namely, DPGMM-HMM alignment prediction task and out-of-domain ASR label prediction task. In the first case, three tasks are included, i.e., frame alignments generated by DPGMM-HMM AMs, one for each target zero-resource language. In the second case, four tasks corresponding to Cantonese, Czech, Hungarian and Russian recognizers’ senone labels are included. The senone labels are generated by decoding with LM to AM weight ratio set to $0.001$. After MTL-DNN training, $40$-dimension HMM-LI-BNFs are extracted for the ABX task evaluation. Similarly, HMM-MUBNFs, extracted by MTL-DNN with DPGMM-HMM alignment tasks, and OSBNFs, extracted by MTL-DNN with one or more out-of-domain phone recognizers’ senone labels, are also evaluated by the ABX task. The dimensions of both HMM-MUBNFs and OSBNFs are $40$. As illustrated in Fig. 3, we defined several BNF representations according to the tasks included in MTL-DNN training. The configurations for (HMM-)MUBNF, OSBNF and (HMM-)LI-BNF are listed in Table II.

TABLE II: Configurations for (HMM-)MUBNF, OSBNF and (HMM-)LI-BNF representations

Task label from	DPGMM			DPGMM-HMM			CA	CZ	HU	RU
Train set	EN	FR	MA	EN	FR	MA	Pooling EN, FR and MA
MUBNF	✓	✓	✓
OSBNF1							✓
OSBNF2							✓	✓	✓	✓
LI-BNF1	✓	✓	✓				✓
LI-BNF2	✓	✓	✓				✓	✓	✓	✓
HMM-MUBNF				✓	✓	✓
HMM-LI-BNF1				✓	✓	✓	✓
HMM-LI-BNF2				✓	✓	✓	✓	✓	✓	✓

The MTL-DNN is implemented in three different model structures: MLP, LSTM and BLSTM. The input features are $40$-dimension fMLLRs spliced with context size $\pm 5$. The dimensions of shared hidden layers in the MLP are $440$-$1024\times 5$-$40$-$1024$. Sigmoid activation is used in all hidden layers except that the $40$ neurons in the BN layer use linear activation functions. The learning rate for MLP training is set at $0.008$ at the beginning, and halved when no improvement is observed on a cross-validation set. The mini-batch size is $256$. The LSTM model comprises $2$ LSTM layers with $320$-dimension cell activation vectors, and $1024$-dimension outputs. A $40$-dimension BN layer followed by a $1024$-dimension fully-connected (FC) layer is set on top of LSTMs. For the BLSTM model, there are $2$ pairs of forward and backward LSTM layers. Each bi-directional layer has $320$-dimension cell activation vectors and $512$-dimension outputs. A BN layer followed by an FC layer is set on top of BLSTMs, with the same configuration as in the LSTM. The activation function in (B)LSTMs is $\tanh$. The learning rate is $2\mathrm{e}-4$ initially, and halved under the same criteria as for MLP. The truncated back-propagation through time (BPTT) algorithm [54] is used to train (B)LSTM, with a fixed time step $T_{bptt}=20$. Note that the model parameters of LSTM and BLSTM structures were tuned in preliminary studies, while for MLP we follow the configuration of our previous study [14].

IV-F TLBNF generation

The TLBNFs for target zero-resource languages are generated by applying the Cantonese DNN-HMM AM as the feature extractor. During TLBNF extraction, all the parameters of the DNN-HMM are fixed. The fMLLR features for target languages are fed as inputs to the DNN-HMM till its BN layer to generate TLBNFs.

V Results and Discussion

TABLE III: ABX error rates ($\%$) on the baseline, our proposed methods and state of the art of ZeroSpeech 2017. MLP is adopted as the shared-hidden-layer structure. Label filtering is not applied.

		Within-speaker										Across-speaker
		English			French			Mandarin			Avg.	English			French			Mandarin			Avg.
		1s	10s	120s	1s	10s	120s	1s	10s	120s		1s	10s	120s	1s	10s	120s	1s	10s	120s
[baseline=(char.base)] [shape=circle,draw,inner sep=1.25pt] (char) 1;	MFCC Baseline [6]	$12.0$	$12.1$	$12.1$	$12.5$	$12.6$	$12.6$	$11.5$	$11.5$	$11.5$	$12.0$	$23.4$	$23.4$	$23.4$	$25.2$	$25.5$	$25.2$	$21.3$	$21.3$	$21.3$	$23.3$
	Out-of-domain fMLLR [14]	$8.0$	$8.2$	$7.3$	$10.3$	$10.3$	$9.1$	$9.3$	$9.3$	$8.4$	$8.9$	$13.4$	$12.0$	$11.3$	$17.2$	$15.8$	$14.8$	$10.7$	$10.2$	$9.4$	$12.8$
	Out-of-domain fMLLR [5]	$7.8$	$7.7$	$7.0$	$10.4$	$10.5$	$9.2$	$9.2$	$11.4$	$8.8$	$9.1$	$14.2$	$11.9$	$11.3$	$17.6$	$15.2$	$14.4$	$12.7$	$13.6$	$10.0$	$13.4$
[baseline=(char.base)] [shape=circle,draw,inner sep=1.25pt] (char) 2;	MUBNF0	$8.0$	$7.3$	$7.3$	$10.3$	$9.4$	$9.3$	$10.1$	$8.8$	$8.9$	$8.8$	$13.5$	$12.4$	$12.4$	$17.8$	$16.4$	$16.1$	$12.6$	$11.9$	$12.0$	$13.9$
	MUBNF	$7.4$	$6.9$	$6.3$	$9.6$	$9.0$	$8.1$	$9.8$	$8.8$	$8.1$	$8.2$	$10.9$	$9.5$	$8.9$	$15.2$	$13.0$	$12.0$	$10.5$	$8.9$	$8.2$	$10.8$
	OSBNF1	$7.2$	$7.1$	$6.3$	$10.2$	$9.7$	$8.7$	$9.1$	$8.6$	$7.6$	$8.3$	$10.0$	$9.7$	$8.6$	$13.9$	$13.4$	$11.6$	$9.0$	$8.4$	$7.5$	$10.2$
	OSBNF2	$6.8$	$6.7$	$5.9$	$9.5$	$9.2$	$8.3$	$9.7$	$8.9$	$8.0$	$8.1$	$9.5$	$9.2$	$7.9$	$13.1$	$13.0$	$11.3$	$9.4$	$8.7$	$7.9$	$10.0$
	LI-BNF1	$6.9$	$6.6$	$6.1$	$9.5$	$9.2$	$8.4$	$9.2$	$8.5$	$7.9$	$8.0$	$10.0$	$8.9$	$8.2$	$14.3$	$12.9$	$11.5$	$9.5$	$8.5$	$7.7$	$10.2$
	LI-BNF2	$6.6$	$6.4$	$5.7$	$9.1$	$9.3$	$8.2$	$9.5$	$8.7$	$8.1$	$8.0$	$9.4$	$8.7$	$7.8$	$13.4$	$12.7$	$11.0$	$9.3$	$8.6$	$7.7$	$9.8$
[baseline=(char.base)] [shape=circle,draw,inner sep=1.25pt] (char) 3;	HMM(S)-MUBNF	$7.2$	$6.7$	$6.3$	$9.7$	$9.2$	$8.3$	$10.4$	$9.2$	$8.5$	$8.4$	$10.2$	$9.3$	$8.6$	$14.5$	$13.0$	$11.9$	$10.7$	$9.2$	$8.4$	$10.6$
	HMM(P)-MUBNF	$7.1$	$6.6$	$6.2$	$9.4$	$9.1$	$7.8$	$9.9$	$8.8$	$8.2$	$8.1$	$10.4$	$9.2$	$8.7$	$14.5$	$12.7$	$11.7$	$10.4$	$8.9$	$8.2$	$10.5$
	HMM(P)-LI-BNF1	$6.8$	$6.3$	$5.8$	$9.1$	$8.7$	$7.8$	$9.1$	$8.5$	$7.6$	$7.7$	$9.7$	$8.7$	$8.0$	$13.7$	$12.3$	$11.1$	$9.7$	$8.4$	$7.6$	$9.9$
	HMM(P)-LI-BNF2	$6.6$	$6.4$	$5.7$	$9.2$	$8.8$	$8.1$	$9.2$	$8.6$	$7.9$	$7.8$	$9.3$	$8.7$	$7.8$	$13.0$	$12.4$	$11.0$	$9.5$	$8.5$	$7.7$	$9.8$
[baseline=(char.base)] [shape=circle,draw,inner sep=1.25pt] (char) 4;	TLBNF	$7.2$	$6.8$	$6.1$	$9.6$	$9.0$	$8.0$	$8.7$	$7.6$	$6.8$	$7.8$	$10.6$	$9.6$	$8.7$	$14.2$	$13.2$	$11.5$	$8.5$	$7.6$	$6.7$	$10.1$
	TLBNF+LI-BNF1	$7.0$	$6.6$	$6.0$	$9.3$	$8.8$	$7.9$	$8.6$	$7.5$	$6.7$	$7.6$	$10.3$	$9.3$	$8.4$	$13.9$	$12.9$	$11.4$	$8.5$	$7.6$	$6.7$	$9.9$
	TLBNF+LI-BNF2	$7.1$	$6.6$	$6.0$	$9.4$	$8.9$	$7.8$	$8.7$	$7.5$	$6.8$	$7.6$	$10.4$	$9.4$	$8.5$	$14.0$	$13.0$	$11.3$	$8.5$	$7.6$	$6.6$	$9.9$
	TLBNF+HMM(P)-LI-BNF1	$7.0$	$6.6$	$6.0$	$9.4$	$8.8$	$7.8$	$8.6$	$7.5$	$6.7$	$7.6$	$10.3$	$9.4$	$8.4$	$13.9$	$12.9$	$11.3$	$8.5$	$7.6$	$6.6$	$9.9$
	TLBNF+MUBNF+OSBNF1	$6.8$	$6.4$	$5.8$	$9.0$	$8.8$	$7.8$	$8.5$	$7.7$	$6.8$	$7.5$	$9.9$	$9.0$	$8.2$	$13.6$	$12.6$	$11.1$	$8.4$	$7.7$	$6.7$	$9.7$
	TLBNF+HMM(P)-MUBNF+OSBNF1	$6.8$	$6.4$	$5.7$	$8.8$	$8.7$	$7.5$	$8.4$	$7.5$	$6.8$	$7.4$	$10.0$	$9.0$	$8.2$	$13.6$	$12.6$	$11.1$	$8.4$	$7.6$	$6.7$	$9.7$
	TLBNF+HMM(P)-MUBNF+OSBNF2	$6.7$	$6.4$	$5.8$	$9.0$	$8.8$	$7.5$	$8.3$	$7.5$	$6.8$	$7.4$	$10.0$	$9.0$	$8.2$	$13.6$	$12.6$	$11.1$	$8.4$	$7.6$	$6.7$	$9.7$
	Heck et al. [20]	$6.9$	$6.2$	$6.0$	$9.7$	$8.7$	$8.4$	$8.8$	$7.9$	$7.8$	$7.8$	$10.1$	$8.7$	$8.5$	$13.6$	$11.7$	$11.3$	$8.8$	$7.4$	$7.3$	$9.7$
	Chorowski et al. [28]	$5.8$	$5.7$	$5.8$	$7.1$	$7.0$	$6.9$	$7.4$	$7.2$	$7.1$	$6.7$	$9.3$	$9.3$	$9.3$	$11.9$	$11.4$	$11.6$	$8.6$	$8.5$	$8.5$	$9.8$

Table III provides a master summary to facilitate performance comparison among different systems of feature representation learning. The methods are organized in four groups, marked by circled numerals [baseline=(char.base)] [shape=circle,draw,inner sep=1.25pt] (char) 1; to [baseline=(char.base)] [shape=circle,draw,inner sep=1.25pt] (char) 4; in the Table. The first group comprises a few relevant baseline and reference systems. The MFCC baseline system refers to the one, in which generic MFCC features are directly used in triphone minimal pair discrimination. The first out-of-domain fMLLR system comes from previous work [14], which used a Cantonese ASR system for fMLLR estimation. The second one used a Japanese ASR [5].

The second and third groups of systems all use multilingual BNF representations, which are learned by different methods as described in Section IV-E. DPGMM labels and DPGMM-HMM labels are applied in the the second group and the third group respectively. In the second group, MUBNF0 is learned using MFCC as input features for DPGMM clustering and MTL-DNN modeling. The other representations in these two groups are learned using fMLLRs as DNN input features. As described in Section IV-E and Table II, OSBNF1 and OSBNF2 are trained with out-of-domain ASR senone labels, and LI-BNF1 and LI-BNF2 are trained with both DPGMM labels and out-of-domain ASR senone labels. In the third group, “HMM(S)” and “HMM(P)” denote the use of state-level and phone-level HMM alignments respectively for label generation. The fourth group of systems are built on different combination of BNF features. The “+” sign is used to denote concatenation of two frame-level feature representations. The experimental results on all methods of BNF representation learning as shown in Table III are obtained by using the MLP structure in MTL-DNN. In addition, two representative systems that achieved very good performances in ZeroSpeech 2017 [20, 28] are also listed in the Table.

V-A Effect of out-of-domain speaker adaptation

The fMLLR features estimated with in-domain data were shown to perform significantly better than conventional spectral features in unsupervised subword modeling [26, 22]. In the present study, it has been shown that similar improvement could also be attained by performing speaker adaptation using an out-of-domain ASR system. Both out-of-domain fMLLR features in the first group of systems outperform the MFCC baseline consistently on all target languages. This improvement can be achieved without requiring any transcribed training data of the target language, which is highly desirable in the zero-resource scenario.

[5], the out-of-domain ASR system was trained on $240$ hours of Japanese speech. The experimental results in [14] show that using a Cantonese ASR system trained on only $19$ hours of speech could give a better performance in both within- and across-speaker conditions. The advantage is particularly significant when the target language is Mandarin.

V-B Effectiveness of multilingual BNFs

The following observations can be made on the performances of the learned multilingual BNF representations:

(1) BNF representations learned by MTL-DNN clearly outperform the respective input features to the DNN. MTL-DNN training with DPGMM labels is effective for both MFCC and fMLLR. The average ABX error rates achieved by MUBNF0 are $8.8\%$ and $13.9\%$ in the within-speaker and across-speaker conditions respectively, versus $12.0\%$ and $23.3\%$ attained by MFCC. For MUBNF representation, the relative performance improvements over fMLLR are $7.9\%$ and $15.6\%$ in the two test conditions. MUBNF outperforms MUBNF0 to a large extent, especially in the across-speaker test condition. This suggests that speaker adaptation at input feature level is a critical step in obtaining speaker-invariant BNF representations.

(2) The effectiveness of BNF can be further improved by training the MTL-DNN with additional out-of-domain ASRs’ senone labels. With the Cantonese ASR’s senone labels included as one of the training tasks, the LI-BNF1 representation reduces within-/across-speaker ABX error rates by absolute $0.2\%/0.6\%$ as compared to MUBNF. When the senone labels of Czech, Hungarian and Russian are added, the resulted LI-BNF2 representation shows a further improvement of absolute $0.4\%$ under the across-speaker condition. This shows that out-of-domain acoustic-phonetic knowledge provides complementary information to the in-domain clustering labels for feature learning. The performance gain of OSBNF2 over OSBNF1, as well as that of LI-BNF2 over LI-BNF1, confirm the benefit of exploiting a wider coverage of language resources.

The performance of OSBNF2 is inferior to OSBNF1 on Mandarin test set, but not on English and French. It is noted that OSBNF1 is learned by using the Cantonese ASR senone labels while OSBNF2 is learned by involving Cantonese and the other three European languages. Cantonese, being a Chinese dialect, is apparently closer to Mandarin than Czech, Hungarian and Russian in terms of acoustic-phonetic properties. The experimental results imply that the frame labels generated by involving highly-mismatched out-of-domain languages may be of low quality and not suitable for feature learning.

(3) As discussed in Section III-D, DPGMM-HMM labels are obtained by modeling temporal dependency of speech and DPGMM labels are determined with the assumption that neighboring speech frames are independent. Comparing the corresponding systems in the second and the third groups of Table III, it is noted that DPGMM-HMM labels perform slightly better than DPGMM labels. The ABX error rates attained with HMM(P)-MUBNF, HMM(P)-LI-BNF1 and HMM(P)-LI-BNF2 are about absolute $0.2\%$ - $0.3\%$ lower than those with MUBNF, LI-BNF1 and LI-BNF2 respectively, except for HMM(P)-LI-BNF2 under the across-speaker condition. This demonstrates that capturing temporal dependency in speech is beneficial to feature learning for subword modeling [22]. It is also noted that phone-level HMM alignments are better than state-level ones.

(4) Combining different types of BNF feature representations leads to further improvement of performance. Specifically, by concatenating HMM(P)-MUBNF, OSBNF1 and TLBNF, the best ABX error rates under both within-speaker and across-speaker conditions are achieved ($7.4\%$ and $9.7\%$). It is found that BNFs learned from in-domain unsupervised data (HMM(P)-MUBNF, OSBNF1) and learned via transfer learning (TLBNF) can be jointly used to compose an optimal feature representation that is better than any individual BNF.

The best performance attained in this study is competitive to the best submitted system for the ZeroSpeech 2017 challenge, which is based on the combination of multiple DPGMM posteriorgrams [20]. These posteriograms were generated with unsupervisedly estimated fMLLRs based on different implementation parameters. The combination of posteriorgrams led to $3.0\%$ and $3.3\%$ relative error rate reduction under the within-speaker and across-speaker conditions, compared to the use of single posteriorgram representation. In our work, concatenating the three aforementioned BNF representations results in $5.1\%$ and $4.0\%$ relative error rate reduction, as compared with the best system with single BNF. It must be noted that no out-of-domain transcribed speech was involved in the system of [20].

In a very recent work [28], vector quantized VAE (VQ-VAE) was applied to develop a system of unsupervised subword modeling. The reported average ABX error rate was $6.7\%$ for within-speaker condition, which is the best among all reported systems so far. For the across-speaker condition, our proposed systems with combined BNF features have slightly better performance than VQ-VAE ($9.8\%$). Our systems are found to be more effective on long utterances than VQ-VAE. In Table III, it is noted that the performance of VQ-VAE does not depend on utterance duration. For English and Mandarin, the ABX error rates are almost exactly the same between the cases of $1$s and $120$s. One possible reason is that the VQ-VAE system does not perform explicit utterance-level speaker normalization on input features. On the contrary, the BNF representations investigated in the study perform significantly better on longer utterances ($10$s & $120$s) than on $1$s ones. It is also noted that our systems are more effective for Mandarin in the across-speaker condition. This may be due to the use of Cantonese speech in feature learning. VQ-VAE may be over-fitting to Mandarin due to small data size [28].

V-C Effectiveness of label filtering

TABLE IV: Comparison of MTL-DNN shared-hidden-layer structures in feature representation learning of ZeroSpeech 2017.

		Within-speaker										Across-speaker
		English			French			Mandarin			Avg.	English			French			Mandarin			Avg.
		1s	10s	120s	1s	10s	120s	1s	10s	120s		1s	10s	120s	1s	10s	120s	1s	10s	120s
MUBNF	MLP	$7.4$	$6.9$	$6.3$	$9.6$	$9.0$	$8.1$	$9.8$	$8.8$	$8.1$	$\mathbf{8.2}$	$10.9$	$9.5$	$8.9$	$15.2$	$13.0$	$12.0$	$10.5$	$8.9$	$8.2$	$\mathbf{10.8}$
	LSTM	$7.4$	$7.1$	$6.8$	$10.0$	$9.5$	$8.7$	$10.4$	$9.5$	$8.7$	$8.7$	$10.4$	$9.6$	$9.0$	$14.6$	$13.3$	$12.3$	$10.9$	$9.3$	$8.6$	$10.9$
	BLSTM	$7.4$	$7.1$	$6.7$	$9.9$	$9.5$	$8.9$	$10.4$	$9.4$	$8.7$	$8.7$	$10.4$	$9.6$	$9.0$	$14.7$	$13.3$	$12.1$	$10.7$	$9.3$	$8.6$	$10.9$
HMM(P)-MUBNF	MLP	$7.1$	$6.6$	$6.2$	$9.4$	$9.1$	$7.8$	$9.9$	$8.8$	$8.2$	$\mathbf{8.1}$	$10.4$	$9.2$	$8.7$	$14.5$	$12.7$	$11.7$	$10.4$	$8.9$	$8.2$	$\mathbf{10.5}$
	LSTM	$7.2$	$6.8$	$6.4$	$9.9$	$9.4$	$8.7$	$10.4$	$9.5$	$8.8$	$8.6$	$10.0$	$9.3$	$8.6$	$14.3$	$13.1$	$11.8$	$10.7$	$9.3$	$8.6$	$10.6$
	BSLTM	$7.3$	$6.9$	$6.5$	$9.6$	$9.5$	$8.4$	$10.5$	$9.4$	$9.0$	$8.6$	$10.1$	$9.4$	$8.9$	$14.2$	$13.0$	$11.9$	$10.8$	$9.4$	$8.7$	$10.7$
HMM(P)-LI-BNF1	MLP	$6.8$	$6.3$	$5.8$	$9.1$	$8.7$	$7.8$	$9.1$	$8.5$	$7.6$	$\mathbf{7.7}$	$9.7$	$8.7$	$8.0$	$13.7$	$12.3$	$11.1$	$9.7$	$8.4$	$7.6$	$\mathbf{9.9}$
	LSTM	$6.7$	$6.6$	$5.9$	$9.5$	$9.4$	$8.2$	$9.6$	$8.9$	$7.9$	$8.1$	$9.6$	$9.1$	$8.1$	$14.1$	$13.3$	$11.6$	$10.2$	$9.1$	$8.0$	$10.3$
	BLSTM	$7.0$	$6.6$	$6.1$	$9.3$	$9.2$	$8.2$	$9.4$	$8.7$	$8.0$	$8.1$	$9.5$	$9.0$	$8.2$	$13.7$	$13.0$	$11.6$	$9.7$	$8.7$	$7.8$	$10.1$

The effectiveness of the proposed label filtering algorithm is evaluated with the HMM(P)-MUBNF representation, which is trained exclusively based on DPGMM-HMM labels, without involving out-of-domain speech data. Algorithm III-C requires one tunable parameter $P$, i.e., the percentage frame labels to be retained. The average ABX error rates attained with different values of $P$ are plotted as in Fig. 5. $P=1$ means that all labels are kept, which is the setting used to obtain the results in Table III.

Under both within-speaker and across-speaker conditions, the optimal values of $P$ are in the range of $0.7$ to $0.9$. That is, when on average about $10-30\%$ of the frame labels are removed, the ABX error rates could be slightly reduced. This indicates that indeed a certain portion of the labels are not reliable. However, if too many labels are removed, e.g., more than $30\%$, the system performance would degrade significantly, because some good labels are lost.

The proposed label filtering method is very simple in that only the occurrence counts of the labels are considered. Fig. 5 shows that this criterion is appropriate to a certain extent. However, there may exist infrequent subword units that are meaningful and crucial in conveying linguistic content. In [23, 24], it was suggested to reduce the number of DPGMM clusters without ignoring any frame labels. Since these studies were carried out on a different database, direct comparison of system performance can not be made.

Fig. 5: Average ABX error rates ($\%$) with respect to label filtering percentage over three zero-resource languages, in HMM(P)-MUBNF representation.

V-D Comparison of DNN model structures

For BNF feature learning with the MTL-DNN approach, DNN models other than MLP can be used. Table IV compares the system performances obtained by using MLP, LSTM and BLSTM. The feature representations being investigated include MUBNF, HMM(P)-MUBNF and HMM(P)-LI-BNF1, and label filtering is not applied.

Fig. 6: Average ABX error rates ($\%$) of HMM(P)-MUBNF representation over utterance lengths for each language. Left: Across-speaker; Right: Within-speaker.

It is noted that LSTM and BLSTM do not perform as well as MLP on all three types of BNF representations. Experiments were carried out with different parameter settings on LSTM and BLSTM, and the system performance remained largely unchanged. Fig. 6 gives the performances of HMM(P)-MUBNF learned by MLP, LSTM and BLSTM for each target language. For English (EN), different DNN structures have similar performance. For French (FR) and Mandarin (MA), the advantage of MLP over (B)LSTM is more prominent. This may be related to that the amount of training data for English is significantly greater than those for French and Mandarin. The advantage of LSTM and BLSTM over MLP in conventional supervised acoustic modeling has been widely recognized and attributed to the capability of capturing temporal characteristics of speech. With limited training data, the benefits of recurrent structures can not be fully exploited. In our systems, contextual information is incorporated via the use of DPGMM-HMM labels and its effectiveness has been demonstrated by the experimental results.

VI Conclusions

BNFs learned from multilingual speech data have been proven highly effective for acoustic modeling of spoken languages. In the case of low-resource languages, the challenge of lacking transcribed data could be translated into the problem of acquiring high-quality labels to facilitate supervised DNN training. Commonly used approaches to tackling this problem include applying clustering algorithms on short-time speech frames and leveraging a language-mismatched phone recognizer to decode input speech. In this paper, it has been demonstrated that learning of robust BNF representations could be achieved by joint contributions from a variety of techniques, including: (1) use of speaker adapted features; (2) considering temporal dependency in speech when performing frame clustering; (3) increasing phonetic diversity by involving multiple out-of-domain languages; (4) discarding unreliable frame labels in DNN training.

The proposed methods of feature learning have been evaluated on the standard task of unsupervised subword modeling in the ZeroSpeech 2017 Challenge. The experimental results have shown that effective speaker adaptation with untranscribed training data could be achieved by using an out-of-domain ASR system. Out-of-domain ASR systems from resource-rich languages can also be utilized to provide phonetically informed labels to support multi-task learning of BNFs, in conjunction with the learning tasks based on DPGMM-HMM clustering labels. Combining different types of BNFs by vector concatenation leads to further performance improvement. The best performance achieved by our proposed system is $9.7\%$ in terms of across-speaker triphone minimal-pair ABX error rate. It is equal to the performance of the best submitted system in the ZeroSpeech 2017 and better than other recently reported systems.

In principle, the proposed methods are expected to be effective for any combination of languages other than those in ZeroSpeech 2017. Nevertheless, our investigation has suggested that the closeness between target languages and out-of-domain languages and the amount of available training data for individual target languages might have significant impact on the goodness of learned features.