With the introduction of the variational autoencoder (VAE), probabilisticlatent variable models have received renewed attention as powerful generativemodels. However, their performance in terms of test likelihood and quality ofgenerated samples has been surpassed by autoregressive models withoutstochastic units. Furthermore, flow-based models have recently been shown to bean attractive alternative that scales well to high-dimensional data. In thispaper we close the performance gap by constructing VAE models that caneffectively utilize a deep hierarchy of stochastic variables and model complexcovariance structures. We introduce the Bidirectional-Inference VariationalAutoencoder (BIVA), characterized by a skip-connected generative model and aninference network formed by a bidirectional stochastic inference path. We showthat BIVA reaches state-of-the-art test likelihoods, generates sharp andcoherent natural images, and uses the hierarchy of latent variables to capturedifferent aspects of the data distribution. We observe that BIVA, in contrastto recent results, can be used for anomaly detection. We attribute this to thehierarchy of latent variables which is able to extract high-level semanticfeatures. Finally, we extend BIVA to semi-supervised classification tasks andshow that it performs comparably to state-of-the-art results by generativeadversarial networks.
Quick Read (beta)
BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling
With the introduction of the variational autoencoder (VAE), probabilistic latent variable models have received renewed attention as powerful generative models. However, their performance in terms of test likelihood and quality of generated samples has been surpassed by autoregressive models without stochastic units. Furthermore, flow-based models have recently been shown to be an attractive alternative that scales well to high-dimensional data. In this paper we close the performance gap by constructing VAE models that can effectively utilize a deep hierarchy of stochastic variables and model complex covariance structures. We introduce the Bidirectional-Inference Variational Autoencoder (BIVA), characterized by a skip-connected generative model and an inference network formed by a bidirectional stochastic inference path. We show that BIVA reaches state-of-the-art test likelihoods, generates sharp and coherent natural images, and uses the hierarchy of latent variables to capture different aspects of the data distribution. We observe that BIVA, in contrast to recent results, can be used for anomaly detection. We attribute this to the hierarchy of latent variables which is able to extract high-level semantic features. Finally, we extend BIVA to semi-supervised classification tasks and show that it performs comparably to state-of-the-art results by generative adversarial networks.
bayesnet \usetikzlibraryarrows \usetikzlibraryshapes.multipart \tikzset state/.style= rectangle split, rectangle split parts=2, rectangle split part fill=red!30,blue!20, rounded corners, draw=black, very thick, minimum height=2em, text width=3cm, inner sep=2pt, text centered,
One of the key aspirations in recent machine learning research is to build models that understand the world (Kingma, 2013; Rezende et al., 2014; Goodfellow et al., 2014; van den Oord et al., 2016c). Generative models are providing the means to learn from a plethora of unlabeled data in order to model a complex data distribution, e.g. natural images, text, and audio. These models are evaluated by their ability to generate data that is similar to the input data distribution from which they were trained on. The range of applications that come with generative models are vast, where audio synthesis (van den Oord et al., 2016a) and semi-supervised classification (Rasmus et al., 2015; Maaløe et al., 2016; Salimans et al., 2016) are examples hereof. Generative models can be broadly divided into explicit and implicit density models. The generative adversarial network (GAN) (Goodfellow et al., 2014) is an example of an implicit model, since it is not possible to procure a likelihood estimation from this model framework. The focus of this research is instead within explicit density models, for which a tractable or approximate likelihood estimation can be performed.
The three main classes of powerful explicit density models are autoregressive models (Larochelle & Murray, 2011; van den Oord et al., 2016c), flow-based models (Dinh et al., 2014, 2016; Kingma & Dhariwal, 2018), and probabilistic latent variable models (Kingma, 2013; Rezende et al., 2014; Mnih & Gregor, 2014). In recent years autoregressive models, such as the PixelRNN and the PixelCNN (van den Oord et al., 2016c; Salimans et al., 2017), have achieved superior likelihood performance and flow-based models have proven efficacy on large-scale natural image generation tasks (Kingma & Dhariwal, 2018). However, in the autoregressive models the runtime performance of generation is scaling poorly with the complexity of the input distribution. The flow-based models do not posses this restriction and do indeed generate visually compelling natural images when sampling close to the mode of the distribution. However, generation from the actual learned distribution is less compelling and the likelihood is far from the one of autoregressive models (Kingma & Dhariwal, 2018).
Probabilistic latent variable models possess intriguing properties that are different from the other classes of explicit density models. They are characterized by a posterior distribution over the latent variables of the model, derived from Bayes’ theorem, which is typically intractable and needs to be approximated. This distribution most commonly lies on a low-dimensional manifold that can provide insights into the internal representation of the data (Bengio et al., 2013). However, the latent variable models have largely been disregarded as powerful generative models due to blurry generations and poor likelihood performances on natural image tasks. Larsen et al. (2016); Dosovitskiy & Brox (2016), amongst others, attribute this tendency to the usage of a similarity metric in pixel space. Contrarily, we attribute it to the lack of overall model expressiveness for accurately modeling complex input distributions, as discussed in Zhao et al. (2017); Rezende & Viola (2018).
There has been much research into explicitly defining and learning more expressive latent variable models. Here, the complementary research into learning a covariance structure through a framework of normalizing flows (Rezende & Mohamed, 2015; Tomczak & Welling, 2016; Kingma et al., 2016) and the stacking of a hierarchy of latent variables (Burda et al., 2015a; Ranganath et al., 2016; Maaløe et al., 2016; Sønderby et al., 2016) have shown promising results. However, despite significant improvements, the reported performance of these models has still been inferior to their autoregressive counterparts. This has spawned a new class of explicit density models that adds an autoregressive component to the generative process of a latent variable model (Gulrajani et al., 2016; Chen et al., 2017). In this combination of model paradigms the latent variables can be viewed as merely a lossy representation of the input data and the model still suffers from the same issues as autoregressive models.
In this research we argue that latent variable models that are defined in a sufficiently expressive way can compete with autoregressive and flow-based models in terms of test log-likelihood and quality of the generated samples. We introduce the Bidirectional-Inference Variational Autoencoder (BIVA), a model formed by a deep hierarchy of stochastic variables that uses skip-connections to enhance the flow of information and avoid inactive units. To define a flexible posterior approximation, we construct a bidirectional inference network using stochastic variables in a bottom-up and a top-down inference path. We perform an in-depth analysis of BIVA and show (i) an ablation study that analyses the contributions of the individual novel components, (ii) that the model is able to improve on state-of-the-art results on benchmark image datasets, (iii) that a small extension of the model can be used for semi-supervised classification and performs comparably to current state-of-the-art models, and (iv) that the model, contrarily to other state-of-the-art explicit density models (Nalisnick et al., 2018), can be utilized for anomaly detection on complex data distributions.
2 Deep Learning and Variational Inference
The introduction of stochastic backpropagation (Paisley et al., 2012; Hoffman et al., 2013) and the variational auto-encoder (VAE) (Kingma, 2013; Rezende et al., 2014) has made approximate Bayesian inference and probabilistic latent variable models applicable to machine learning problems considering complex data distributions, e.g. natural images, audio, and text. The VAE is a generative model parameterized by a neural network and is defined by an observed variable that depends on a hierarchy of stochastic latent variables so that: . This is illustrated in Figure 1(a).
The distributions over the latent variables of the VAE are normally defined as Gaussians with diagonal covariance, whose parameters depend on the previous latent variable in the hierarchy (with the top latent variable ). The likelihood is typically a Gaussian distribution for continuous data, or a Bernoulli distribution for binary data.
In order to learn the parameters we seek to maximize the log marginal likelihood over a training set: . However, complex data distributions require an expressive model, which makes the above integral intractable. In order to circumvent this we use Variational Inference (Jordan et al., 1999), and introduce a posterior approximation , known as inference network or encoder, that is parameterized by a neural network . Using Jensen’s inequality we can derive the evidence lower bound (ELBO), a lower bound to the integral in the marginal likelihood which is a function of the variational approximation and the generative model :
The parameters and can be optimized by maximizing the ELBO with stochastic backpropagation and the reparameterization trick, which allows to use gradient ascent algorithms with low variance gradient estimators (Kingma, 2013; Rezende et al., 2014). As illustrated in Figure 1(b), in a VAE the variational approximation is factorized with a bottom-up structure, , so that each latent variable is conditioned on the variable below in the hierarchy. For ease of computation, all the factors in the variational approximation are typically assumed to be Gaussians whose mean and diagonal covariance are parameterized by neural networks.
Latent variable collapse in VAEs.
A deep hierarchy of latent stochastic variables will result in a more expressive model. However, the additional variables come at a price. As shown in Chen et al. (2017); Maaløe et al. (2017), we can rewrite the ELBO (eq. (1)):
From the above it becomes obvious that, during the optimization of the VAE, the top stochastic latent variables may have a tendency to collapse into the prior, i.e. , if the model is powerful enough. This is supported by empirical results in Sønderby et al. (2016); Bowman et al. (2015) amongst others. The tendency has limited the applicability of deep VAEs in problems with complex data distributions, and has pushed VAE research towards the extension of shallow VAEs with autoregressive models, that allow to capture a lossy representation in the latent space while achieving strong generative performances (Gulrajani et al., 2016; Chen et al., 2017). Another research direction has focused on learning more complex prior distributions through normalizing flows (Rezende & Mohamed, 2015; Tomczak & Welling, 2016; Kingma et al., 2016). Our research considers instead the original goal of building expressive models that can exploit a deeper hierarchy of stochastic latent variables while avoiding variable collapse.
The Ladder VAE (LVAE) (Sønderby et al., 2016) is amongst the first attempts towards VAEs that can effectively leverage multiple layers of stochastic variables. This is achieved by parameterizing the variational approximation with a bottom-up deterministic path followed by a top-down inference path that shares parameters with the top-down structure of the generative model: . A graphical representation of the LVAE inference network can be found in Figure 1(c). Thanks to the bottom-up path, all the latent variables in the hierarchy have a deterministic dependency on the observed variable , which allows data-dependent information to skip all the stochastic variables lower in the hierarchy (Figure 1(d)). The stochastic latent variables that are higher in the hierarchy will therefore receive less noisy inputs, and will be empirically less likely to collapse. Despite the improvements obtained thanks to the more flexible inference network, in practice LVAEs with a very deep hierarchy of stochastic latent variables will still experience variable collapse. In the next section we will introduce the Bidirectional-Inference Variational Autoencoder, that manages to avoid these issues by extending the LVAE in 2 ways: (i) adding a deterministic top-down path in the generative model and (ii) defining a factorization of the latent variables at each level of the hierarchy that allows to construct a bottom-up stochastic inference path.
3 Bidirectional-Inference Variational Autoencoder
In this section we will first describe the architecture of the Bidirectional-Inference Variational Autoencoder (Figure 3), and then provide the motivation behind the main ideas of the model as well as some intuitions on the role of each of its novel components. Finally, we will show how this model can be used for a novel approach to detecting anomalous data.
3.1 Model architecture
In BIVA, at each layer of the hierarchy we split the latent variable in two components, , which belong to a bottom-up (BU) and top-down (TD) inference path, respectively. More details on this will be given when introducing the inference network. The generative model of BIVA is illustrated in Figure 2(a). We introduce a deterministic top-down path that is parameterized with neural networks and receives as input at each layer of the hierarchy the latent variable . This is done by defining a fully convolutional model and concatenating , and along the features’ dimension. can therefore be seen as a deterministic variable that summarizes all the relevant information coming from the stochastic variables higher in the hierarchy, . The latent variables and are conditioned on all the information in the higher layers, and are conditionally independent given . The joint distribution of the model is then given by:
where are the parameters of the generative model. Each stochastic latent variable is parameterized by a Gaussian distribution with diagonal covariance, with one neural network for the mean and another neural network for the variance. Since the and variables are on the same level in the generative model and of the same dimensionality, we share all the deterministic parameters going to the layer below. See Figure 2(a) and Appendix A for details.
Bidirectional inference network.
Due to the non-linearities in the neural networks that parameterize the generative model, the exact posterior distribution is intractable and needs to be approximated. As for VAEs, we therefore define a variational distribution , that needs to be flexible enough to approximate as closely as possible the true posterior distribution. We define a bottom-up (BU) and a top-down (TD) inference path, which are computed sequentially when constructing the posterior approximation for each data point , see Figures 2(b) and 2(c). The variational distribution over the BU latent variables depends on the data and on all BU variables lower in the hierarchy, i.e. , where denotes all the parameters of the BU path. As seen in Figure 2(b), has a direct dependency only on the BU variable below, . The dependency on is achieved, similarly to the generative model, through a deterministic bottom-up path . The TD variables as well depend on the data and the BU variables lower in the hierarchy through the BU inference path, but also on all variables above in the hierarchy through the TD inference path in Figure 2(c). The variational approximation over the TD variables is thereby . Importantly, all the parameters of the TD path are shared with the generative model, and are therefore denoted as . The overall inference network can be factorized as follows:
where the variational distributions over the BU and TD latent variables are Gaussians whose mean and diagonal covariance are parameterized with neural networks that take as input the concatenation over the feature dimension of the conditioning variables (BIVA is fully convolutional). Training of BIVA is performed, as for VAEs, by maximizing the ELBO in eq. (1) with stochastic backpropagation and the reparameterization trick.
BIVA can be seen as an extension of the LVAE in which we (i) add a deterministic top-down path and (ii) apply a bidirectional inference network. We will now provide the motivation and some intuitions on the role of these two novel components, that will then be validated with the experiments of Section 4.
Deterministic top-down path.
Skip-connections represent one of the simplest yet most powerful advancements of deep learning in recent years. They allow to construct very deep neural networks, by better propagating the information throughout the model and reducing the issue of vanishing gradients. Skip connections form for example the backbone of deep neural networks such as ResNets (He et al., 2015), which have shown impressive performances on a wide range of classification tasks. Our goal in this paper is to build very deep latent variable models that are able to learn an expressive latent hierarchical representation of the data. In our experiments we however found that the LVAE still had difficulties in activating the top latent variables for deeper hierarchies. To limit this issue, we add skip connections among the latent variables in the generative model by adding the deterministic top-down path, that makes each variable depend on all the variables above in the hierarchy (see Figure 2(d) for a graphical representation). This allows a better flow of information in the model and thereby avoids the collapse of latent variables. A related idea was recently proposed by Dieng et al. (2018), that add skip connections among the neural network layers parameterizing a shallow VAE with a single latent variable.
The inspiration for the bidirectional inference network of BIVA comes from the work on Auxiliary VAEs (AVAE) by Ranganath et al. (2016); Maaløe et al. (2016). An AVAE can be viewed as a shallow VAE with a single latent variable and an auxiliary variable that increases the expressiveness of the variational approximation . By making the inference network depend on the stochastic variable , the AVAE adds covariance structure to the posterior approximation over the stochastic unit , since it no longer factorizes over its components , i.e. . As discussed in the following, by factorizing the latent variables at each level of the hierarchy of BIVA we are able to achieve similar results without introducing additional auxiliary variables in the model. To see this, we can focus for example on the highest latent variable . In BIVA, the presence of the variables makes the bottom-up inference path stochastic, as opposed to the deterministic BU path of the LVAE. While the conditional distribution still factorizes over the components of , due to the stochastic BU variables the marginal distribution over no longer factorizes, i.e. Therefore, the BU inference path enables the learning of a complex covariance structure in the higher TD stochastic latent variables, which is fundamental in the model to extract good high-level semantic features from the data distribution. Notice that in BIVA only will have a marginally factorizing inference network.
3.3 Anomaly detection with BIVA
Anomaly detection is considered to be one of the most important applications of explicit density models. However, recent empirical results suggest that these models are not able to distinguish between two clearly distinctive data distributions, as they can assign a higher likelihood to data points from a data distribution that is very different from the one the model was trained on (Nalisnick et al., 2018). Based on a thorough study, Nalisnick et al. (2018) state that the main issue is the fact that explicit density models have a tendency to capture low-level statistics, as opposed to the high-level semantics that are preferable when doing anomaly detection. We hypothesize that the latent representations in the higher layers of BIVA can capture the high-level semantics of the data and that these can be used for improved anomaly detection.
In the standard ELBO from eq. (1), the main contribution to the expected log-likelihood term is coming from averaging over the variational distribution of the lower level latent variables. This will thus emphasize low-level statistics. So in order to perform anomaly detection with BIVA we instead need to emphasize the contribution from the higher layers. We can achieve this with an alternative log-likelihood lower bound that partly replaces the inference network with the generative model. It will be a weaker bound than the ELBO, but it has the advantage that it explicitly uses the generative hierarchy of the stochastic variables. In the following we define the hierarchy of stochastic latent variables as with . Instead of using the variational approximation over all stochastic variables in the model, we use the prior distribution for the first layers and the variational appoximation for the others, i.e. .
The new ELBO becomes:
is the ELBO in eq. (1). As for the ELBO, we approximate the computation of with Monte Carlo integration. Sampling from can be easily performed by obtaining samples from the inference network, that are then used to sample from the conditional prior .
Due to the sampling from the prior, eq. (3) will generally return a worse likelihood approximation than the ELBO. Despite this, with high values of represents a useful metric for anomaly detection. By only sampling the top variables from the variational approximation, in fact, we are forcing the model to only rely on the high-level semantics encoded in the highest variables of the hierarchy when evaluating this metric, and not on the low-level statistics encoded in the lower variables.
BIVA is empirically evaluated by (i) an ablation study analyzing each novel component, (ii) likelihood and semi-supervised classification results on binary images, (iii) likelihood results on natural images, and (iv) an analysis of anomaly detection in complex data distributions. We employ a free bits strategy with (Kingma et al., 2016) for all experiments to avoid latent variable collapse during the initial training epochs. Trained models are reported with 1 importance weighted sample, , and 1000 importance weighted samples, (Burda et al., 2015b). We evaluate the natural image experiments by bits per dimension (bits/dim), , where , , denotes the height, width, and channels respectively. In the Appendix we give further details and samples for the image tasks and initial results for the model applied to text. For a detailed description of the experimental setup see Appendix C and the source code11 1 The source code will be released on Github..
4.1 Ablation Study
BIVA can be viewed as an extension of the LVAE from Sønderby et al. (2016) where we add (i) extra dependencies in the generative model ( and ) through the skip connections obtained with the deterministic top-down path and (ii) a bottom-up (BU) path of stochastic latent variables to the inference model. In order to evaluate the effects of each added component we define an LVAE with the exact same architecture as BIVA, but without the BU variables and the deterministic top-down path. Next, we define the LVAE+, where we add to the LVAE’s generative model the deterministic top-down path. It is therefore the same model as in Figure 3 but without the BU variables. Finally, we investigate a LVAE+ model with stochastic layers. This corresponds to the depth of the hierarchy of the BIVA inference model . If this model is competitive with BIVA then it is an indication that it is the depth that determines the performance. The ablation study is conducted on the CIFAR-10 dataset against the best reported BIVA with layers (Section 4.3), which means stochastic latent layers in the deep LVAE+.
Table 1 presents a comparison of the different model architectures. The positive effect of adding the skip connections in the generative models can be evaluated from the difference between the LVAE and LVAE+ results, for which there is close to a 0.2 bits/dim difference in the ELBO. Thanks to the more expressive posterior approximation obtained using its bidirectional inference network, BIVA improves the ELBO significantly w.r.t the LVAE+, by more than 0.3 bits/dim. Notice that a deeper hierarchy of stochastic latent variables in the LVAE+ will not necessarily provide a better likelihood performance, since the LVAE+ performs worse than the LVAE+ despite having significantly more parameters.
In Figure 5 we plot for LVAE, LVAE+ and BIVA the KL divergence between the variational approximation over each latent variable and its prior distribution, . This KL divergence is 0 when the two distributions match, in which case we say that the variable has collapsed, since its posterior approximation is not using any data-dependent information. We can see that while the LVAE is only able to utilize its lowest 7 stochastic variables, all variables in both LVAE+ and BIVA are active. We attribute this tendency to the deterministic top-down path that is present in both models, which creates skip-connections between all latent variables that allow to better propagate the information throughout the model.
4.2 Binary Images
We evaluate BIVA in terms of test log-likelihood on statically binarized MNIST (Salakhutdinov & Murray, 2008), dynamically binarized MNIST (LeCun et al., 1998) and dynamically binarized OMNIGLOT (Lake et al., 2013). The model parameterization and optimization parameters have been kept identical for all binary image experiments (see Appendix C). For each experiment on binary image datasets, we finetune each model by setting the free bits to until convergence in order to test the tightness of the ELBO.
To the best of our knowledge, BIVA achieves state-of-the-art results on statically binarized MNIST, outperforming other latent variable models, autoregressive models, and flow-based models (see Table 2). Finetuning the model with improves the ELBO significantly and achieves slightly better performance for the 1000 importance weighted samples. For dynamically binarized MNIST and OMNIGLOT, BIVA achieves similar improvements with (state-of-the-art) and respectively, see Appendix E for additional results and details.
BIVA can be easily extended for semi-supervised classification by adding a categorical variable to represent the class, as done in Kingma et al. (2014). We add a classification model to the inference network, and a class-conditional distribution to the generative model (see Appendix B for a detailed description). We train 5 different semi-supervised models on MNIST, each using a different set of just 100 randomly chosen and evenly distributed MNIST labels. Table 3 presents the classification results on the test set (mean and standard deviation over the 5 runs), that shows that BIVA achieves comparable performance to recent state-of-the-art results by generative adversarial networks.
|Results with autoregressive components|
|PixelCNN (van den Oord et al., 2016c)|
|DRAW (Gregor et al., 2015)|
|IAFVAE (Kingma et al., 2016)|
|PixelVAE (Gulrajani et al., 2016)|
|PixelRNN (van den Oord et al., 2016c)|
|VLAE (Chen et al., 2017)|
|Results without autoregressive components|
|Discrete VAE (Rolfe, 2017)|
|M1+M2 (Kingma et al., 2014)||% ()|
|VAT (Miyato et al., 2015)||%|
|CatGAN (Springenberg, 2015)||% ()|
|SDGM (Maaløe et al., 2016)||% ()|
|LadderNet (Rasmus et al., 2015)||% ()|
|ADGM (Maaløe et al., 2016)||% ()|
|ImpGAN (Salimans et al., 2016)||% ()|
|TripleGAN (Li et al., 2017)||% ()|
|SSLGAN (Dai et al., 2017)||% ()|
4.3 Natural Images
We trained and evaluated BIVA on 32x32 CIFAR-10, 32x32 ImageNet (van den Oord et al., 2016c), and another BIVA on 64x64 CelebA (Larsen et al., 2016). For the output decoding we employ the discretized logistic mixture likelihood from Salimans et al. (2017) (see Appendix C for more details). In Table 4 we see that for the CIFAR-10 dataset BIVA outperforms other state-of-the-art non-autoregressive models and performs slightly worse than state-of-the-art autoregressive models. Notice that BIVA has fewer parameters (18.48M) than PixelCNN++ (28.57M parameters, Salimans et al. (2017)). For the 32x32 ImageNet dataset BIVA achieves better performance than flow-based models, but the performance gap to the autoregressive models remains large (Table 5). This may be due to the added complexity (more categories) of the 32x32 ImageNet dataset, requiring an even more flexible model. More research should be invested in defining an improved architecture for BIVA that holds more parameters and thereby achieves better performances.
|Results with autoregressive components|
|ConvDRAW (Gregor et al., 2016)|
|IAFVAE (Kingma et al., 2016)|
|IAFVAE (Kingma et al., 2016)|
|GatedPixelCNN (van den Oord et al., 2016b)|
|PixelRNN (van den Oord et al., 2016c)|
|VLAE (Chen et al., 2017)|
|PixelCNN++ (Salimans et al., 2017)|
|Results without autoregressive components|
|NICE (Dinh et al., 2014)|
|DeepGMMs (van den Oord & Schrauwen, 2014)|
|RealNVP (Dinh et al., 2016)|
|DiscreteVAE++ (Vahdat et al., 2018)|
|GLOW (Kingma & Dhariwal, 2018)|
|Results with autoregressive components|
|ConvDRAW (Gregor et al., 2016)|
|PixelRNN (van den Oord et al., 2016c)|
|GatedPixelCNN (van den Oord et al., 2016b)|
|Results without autoregressive components|
|RealNVP (Dinh et al., 2016)|
|GLOW (Kingma & Dhariwal, 2018)|
Figure 4 shows generated samples from the prior of a BIVA trained on the CelebA dataset. From a visual inspection, the samples are far superior to previous natural image generations by latent variable models. We believe that previous claims stating that this type of model can only generate blurry images should be disregarded (Larsen et al., 2016). Rather the limited expressiveness/flexibility of previous models should be blamed. Additional samples from BIVA can be found in Figure 1 and Appendix E.
4.4 Does BIVA know what it doesn’t know?
We test the anomaly detection capabilities of BIVA replicating the most challenging experiment of Nalisnick et al. (2018). We train BIVA on the CIFAR-10 dataset, and evaluate eq. (3) for various values of both on the CIFAR-10 test set and on the SVHN dataset (Netzer et al., 2011). The results can be found in Table 6 and Figure 6, and are reported in terms of bits per dimension (lower is better). We see that for , corresponding to the standard ELBO, BIVA wrongly assigns lower values to data points from SVHN. This is in line with the results obtained with other explicit density models in Nalisnick et al. (2018), and shows that by using the standard ELBO the low-level image statistics prevail and the model is not able to correctly detect out-of-distribution samples. However, for higher values of , the situation is reversed. We take this as an indication that BIVA uses the high-level semantics inferred from the data to better differentiate between the CIFAR-10 and SVHN distributions.
|CIFAR-10 Test||SVHN Test|
In this paper we have introduced BIVA, that significantly improves performances over previously introduced probabilistic latent variable models and flow-based models. BIVA is able to generate natural images that are both sharp and coherent, to improve on semi-supervised classification benchmarks and, contrarily to other models, allows for anomaly detection using the extracted high-level semantics of the data.
On natural image generation tasks, the autoregressive models are still superior to BIVA. Future directions should include experiments on BIVA with more parameters and testing for better architectures. Furthermore, the model flexibility can be increased by defining more expressive stochastic latent variables, such as inverse autoregressive flows (Kingma et al., 2016). Finally, BIVA can easily be complemented by state-of-the-art autoregressive models.
- Bengio et al. (2013) Bengio, Y., Courville, A., and Vincent, P. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8), 2013.
- Bowman et al. (2015) Bowman, S., Vilnis, L., Vinyals, O., Dai, A., Jozefowicz, R., and Bengio, S. Generating sentences from a continuous space. arXiv preprint arXiv:1511.06349, 2015.
- Burda et al. (2015a) Burda, Y., Grosse, R., and Salakhutdinov, R. Importance Weighted Autoencoders. arXiv preprint arXiv:1509.00519, 2015a.
- Burda et al. (2015b) Burda, Y., Grosse, R., and Salakhutdinov, R. Accurate and conservative estimates of mrf log-likelihood using reverse annealing. In Proceedings of the International Conference on Artificial Intelligence and Statistics, 2015b.
- Chen et al. (2017) Chen, X., Kingma, D. P., Salimans, T., Duan, Y., Dhariwal, P., Schulman, J., Sutskever, I., and Abbeel, P. Variational Lossy Autoencoder. In International Conference on Learning Representations, 2017.
- Dai et al. (2017) Dai, Z., Yang, Z., Yang, F., Cohen, W. W., and Salakhutdinov, R. Good semi-supervised learning that requires a bad GAN. In Advances in Neural Information Processing Systems, 2017.
- Dieng et al. (2018) Dieng, A. B., Kim, Y., Rush, A. M., and Blei, D. M. Avoiding latent variable collapse with generative skip models. arXiv preprint arXiv:1807.04863, 2018.
- Dinh et al. (2014) Dinh, L., Krueger, D., and Bengio, Y. Nice: Non-linear independent components estimation. arXiv preprint arXiv:1410.8516, 2014.
- Dinh et al. (2016) Dinh, L., Sohl-Dickstein, J., and Bengio, S. Density estimation using real nvp. arXiv preprint arXiv:1605.08803, 2016.
- Dosovitskiy & Brox (2016) Dosovitskiy, A. and Brox, T. Generating images with perceptual similarity metrics based on deep networks. In Advances in Neural Information Processing Systems, 2016.
- Goodfellow et al. (2014) Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. In Advances in Neural Information Processing Systems. 2014.
- Gregor et al. (2015) Gregor, K., Danihelka, I., Graves, A., and Wierstra, D. Draw: A recurrent neural network for image generation. arXiv preprint arXiv:1502.04623, 2015.
- Gregor et al. (2016) Gregor, K., Besse, Fredric, R. D. J., Danihelka, I., and Wierstra, D. Towards conceptual compression. arXiv preprint arXiv:1604.08772, 2016.
- Gulrajani et al. (2016) Gulrajani, I., Kumar, K., Ahmed, F., Ali Taiga, A., Visin, F., Vazquez, D., and Courville, A. PixelVAE: A latent variable model for natural images. arXiv e-prints, 1611.05013, November 2016.
- He et al. (2015) He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. arXiv preprint arXiv:1512.03385, 2015.
- Hochreiter & Schmidhuber (1997) Hochreiter, S. and Schmidhuber, J. Long short-term memory. Neural computation, 9(8):1735–1780, 1997.
- Hoffman et al. (2013) Hoffman, M. D., Blei, D. M., Wang, C., and Paisley, J. Stochastic variational inference. Journal of Machine Learning Research, 2013.
- Jordan et al. (1999) Jordan, M. I., Ghahramani, Z., Jaakkola, T. S., and Saul, L. K. An introduction to variational methods for graphical models. Machine Learning, 37(2):183–233, November 1999.
- Kingma & Ba (2014) Kingma, D. and Ba, J. Adam: A Method for Stochastic Optimization. arXiv preprint arXiv:1412.6980, 12 2014.
- Kingma & Dhariwal (2018) Kingma, D. P. and Dhariwal, P. Glow: Generative flow with invertible 1x1 convolutions. In Advances in Neural Information Processing Systems, 2018.
- Kingma et al. (2014) Kingma, D. P., Rezende, D. J., Mohamed, S., and Welling, M. Semi-Supervised Learning with Deep Generative Models. In Proceedings of the International Conference on Machine Learning, 2014.
- Kingma et al. (2016) Kingma, D. P., Salimans, T., Jozefowicz, R., Chen, X., Sutskever, I., and Welling, M. Improved variational inference with inverse autoregressive flow. In Advances in Neural Information Processing Systems. 2016.
- Kingma (2013) Kingma, Diederik P; Welling, M. Auto-Encoding Variational Bayes. arXiv preprint arXiv:1312.6114, 12 2013.
- Lake et al. (2013) Lake, B. M., Salakhutdinov, R. R., and Tenenbaum, J. One-shot learning by inverting a compositional causal process. In Advances in Neural Information Processing Systems. 2013.
- Larochelle & Murray (2011) Larochelle, H. and Murray, I. The neural autoregressive distribution estimator. In Proceedings of the International Conference on Artificial Intelligence and Statistics, 2011.
- Larsen et al. (2016) Larsen, A. B. L., Sønderby, S. K., Larochelle, H., and Winther, O. Autoencoding beyond pixels using a learned similarity metric. In Proceedings of the International Conference on Machine Learning, 2016.
- LeCun et al. (1998) LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. Gradient-based learning applied to document recognition. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 2278–2324, 1998.
- Li et al. (2017) Li, C., Xu, K., Zhu, J., and Zhang, B. Triple generative adversarial nets. arXiv preprint arXiv:1703.02291, 2017.
- Maaløe et al. (2016) Maaløe, L., Sønderby, C. K., Sønderby, S. K., and Winther, O. Auxiliary Deep Generative Models. In Proceedings of the International Conference on Machine Learning, 2016.
- Maaløe et al. (2017) Maaløe, L., Fraccaro, M., and Winther, O. Semi-supervised generation with cluster-aware generative models. arXiv preprint arXiv:1704.00637, 2017.
- Miyato et al. (2015) Miyato, T., Maeda, S.-i., Koyama, M., Nakae, K., and Ishii, S. Distributional Smoothing with Virtual Adversarial Training. arXiv preprint arXiv:1507.00677, 7 2015.
- Mnih & Gregor (2014) Mnih, A. and Gregor, K. Neural variational inference and learning in belief networks. In Proceedings of the International Conference on Machine Learning, pp. 1791–1799, 2014.
- Nalisnick et al. (2018) Nalisnick, E., Matsukawa, A., Teh, Y. W., Gorur, D., and Lakshminarayanan, B. Do deep generative models know what they don’t know? arXiv preprint arXiv:1810.09136, 2018.
- Netzer et al. (2011) Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., and Ng, A. Y. Reading digits in natural images with unsupervised feature learning. In Deep Learning and Unsupervised Feature Learning, workshop at Neural Information Processing Systems 2011, 2011.
- Paisley et al. (2012) Paisley, J., Blei, D. M., and Jordan, M. I. Variational bayesian inference with stochastic search. In Proceedings of the International Conference on Machine Learning, pp. 1363–1370, 2012.
- Ranganath et al. (2016) Ranganath, R., Tran, D., and Blei, D. M. Hierarchical variational models. In Proceedings of the International Conference on Machine Learning, 2016.
- Rasmus et al. (2015) Rasmus, A., Berglund, M., Honkala, M., Valpola, H., and Raiko, T. Semi-supervised learning with ladder networks. In Advances in Neural Information Processing Systems, 2015.
- Rezende & Mohamed (2015) Rezende, D. J. and Mohamed, S. Variational Inference with Normalizing Flows. In Proceedings of the International Conference on Machine Learning, 2015.
- Rezende & Viola (2018) Rezende, D. J. and Viola, F. Taming vaes. arXiv preprint arXiv:1810.00597, 2018.
- Rezende et al. (2014) Rezende, D. J., Mohamed, S., and Wierstra, D. Stochastic Backpropagation and Approximate Inference in Deep Generative Models. arXiv preprint arXiv:1401.4082, 04 2014.
- Rolfe (2017) Rolfe, J. T. Discrete variational autoencoders. In Proceedings of the International Conference on Learning Representations, 2017.
- Salakhutdinov & Murray (2008) Salakhutdinov, R. and Murray, I. On the quantitative analysis of deep belief networks. In Proceedings of the International Conference on Machine Learning, 2008.
- Salimans & Kingma (2016) Salimans, T. and Kingma, D. P. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. In Proceedings of the International Conference on Neural Information Processing Systems, 2016.
- Salimans et al. (2015) Salimans, T., Kingma, D. P., and Welling, M. Markov chain monte carlo and variational inference: Bridging the gap. In Proceedings of the International Conference on Machine Learning, 2015.
- Salimans et al. (2016) Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., and Chen, X. Improved techniques for training gans. arXiv preprint arXiv:1606.03498, 2016.
- Salimans et al. (2017) Salimans, T., Karparthy, A., Chen, X., and Kingma, D. P. Pixelcnn++: Improving the pixelcnn with discretized logistic mixture likelihood and other modifications. arXiv preprint:1701.05517, 2017, 2017.
- Semeniuta et al. (2017) Semeniuta, S., Severyn, A., and Barth, E. A hybrid convolutional variational autoencoder for text generation. arXiv preprint arXiv:1702.02390, 2017.
- Shah et al. (2018) Shah, H., Zheng, B., and Barber, D. Generating sentences using a dynamic canvas, 2018.
- Sønderby et al. (2016) Sønderby, C. K., Raiko, T., Maaløe, L., Sønderby, S. K., and Winther, O. Ladder variational autoencoders. In Advances in Neural Information Processing Systems 29. 2016.
- Springenberg (2015) Springenberg, J. Unsupervised and semi-supervised learning with categorical generative adversarial networks. arXiv preprint arXiv:1511.06390, 2015.
- Tomczak & Welling (2016) Tomczak, J. M. and Welling, M. Improving variational auto-encoders using householder flow. arXiv preprint arXiv:1611.09630, 2016.
- Tran et al. (2016) Tran, D., Ranganath, R., and Blei, D. M. Variational Gaussian process. In Proceedings of the International Conference on Learning Representations, 2016.
- Vahdat et al. (2018) Vahdat, A., Macready, W. G., Bian, Z., Khoshaman, A., and Andriyash, E. DVAE++: discrete variational autoencoders with overlapping transformations. In Proceedings of the International Conference on Machine Learning, 2018.
- van den Oord & Schrauwen (2014) van den Oord, A. and Schrauwen, B. Factoring variations in natural images with deep gaussian mixture models. In Advances in Neural Information Processing Systems, 2014.
- van den Oord et al. (2016a) van den Oord, A., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., and Kavukcuoglu, K. Wavenet: A generative model for raw audio. arXiv preprint arXiv:1609.03499, 2016a.
- van den Oord et al. (2016b) van den Oord, A., Kalchbrenner, N., Vinyals, O., Espeholt, L., Graves, A., and Kavukcuoglu, K. Conditional image generation with pixelcnn decoders. arXiv preprint arXiv:1606.05328, 2016b.
- van den Oord et al. (2016c) van den Oord, A., Nal, K., and Kavukcuoglu, K. Pixel recurrent neural networks. arXiv preprint arXiv:1601.06759, 01 2016c.
- Zhao et al. (2017) Zhao, S., Song, J., and Ermon, S. Towards deeper understanding of variational autoencoding models. arXiv preprint arXiv:1702.08658, 2017.
- Zhu et al. (2015) Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urtasun, R., Torralba, A., and Fidler, S. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision, pp. 19–27, 2015.
Appendix A Detailed Model Description
The generative model has a top-down path going from through the intermediary stochastic latent variables to . Between each stochastic layer there is a ResNet block with layers set up similarly to (Salimans et al., 2017). Weight normalization (Salimans & Kingma, 2016) is applied in all neural network layers. In the generative model the BU and TD units are not distinguished so we write . We use to denote the neural network function (a function of generative model parameters ) of ResNet layer asssociated with stochastic layer . The feature maps are written as . The generative process can then be iterated as and :
where , denotes concatenation of feature maps in the convolutional network and hidden units in the fully connected network, and and are parameterized by neural networks. To complete the generative model is written in terms of and through a ResNet block .
The inference model consists of a bottom-up (BU) and top-down (TD) paths such that bottom-up stochastic units only receive bottom-up information whereas the top-down units receive both bottom-up and top-down information. The top-down path shares parameters with the generative model. For each stochastic latent variable in we use a ResNet block with layers and there are associated neural network functions , with parameters collectively denoted by . The deterministic feature map of layer is denoted by :
Appendix B Semi-Supervised Learning
When defining BIVA for semi-supervised classification tasks we follow the approach described for the M2 model in Kingma et al. (2014). In addition to BIVA, described in detail in Appendix A, we introduce a classification model in the inference model, where is the class variable, and a Categorical latent variable dependency in the generative model.
For the classification model we introduce another deterministic hierarchy with an equivalent parameterization as . We denote the hierarchy . The forward-pass is performed by:
where is a final densely connected neural network layer, of the same dimension as the number of categories, and a Softmax activation function. The inference model is thereby factorized by:
For each stochastic latent variable, , and the observed variable in the generative model, as well as the TD path of the inference model, we add a conditional dependency on a categorical variable :
Evidence lower bound.
In a semi-supervised learning problem, we have labeled data and unlabeled data which results in two formulations of the ELBO. The ELBO for labeled data points is given by:
Since the classification model is not included in the above definition of the ELBO we add a classification loss term (a categorical cross-entropy), equivalent to the approach in Kingma et al. (2014):
where is a hyperparameter that we define as in Maaløe et al. (2016). For the unlabeled data points, we marginalize over the labels:
The combined objective function over the labeled, , and unlabeled data points, , are thereby given by:
Appendix C Experimental Setup
All models are optimized using Adamax (Kingma & Ba, 2014) with a hyperparameter setting similar to the one used in (Kingma et al., 2016). They are trained with a batch-size of 48 where the binary image experiments are trained on a single GPU and the natural image experiments are trained on two GPUs (by splitting the batch in 2 and then taking the mean over the gradients). For evaluation we use exponential moving averages of the parameters space, similar to Kingma et al. (2016); Salimans et al. (2017).
Binary image architecture.
BIVA has layers. The neural networks are defined by , 64x5x5 (number of kernels x kernel width x kernel height) convolutional layers and an overall stride of 2. Neural networks are defined by four , 64x3x3 convolutional layers. The final neural network, , applies a stride of 2. All stochastic latent variables are densely connected layers of dimension for respectively. We apply a dropout rate of for both the deterministic layers in the generative as well as the inference model.
Natural image architecture (32x32).
BIVA has layers. The neural networks are defined by , 96x5x5 convolutional layers and an overall stride of 2. Neural networks are defined by , 96x3x3 convolutional layers. Neural networks and are defined with a stride of 2. All stochastic latent variables are parameterized by convolutional layers with feature maps for respectively. The kernel width and height of the stochastic latent variables are defined similarly to the dimension of the subsequent output after striding. We apply a dropout rate of in the deterministic layers of the inference model.
Natural image architecture (64x64).
BIVA has layers. The and neural networks are defined by , 64x7x7 and 64x5x5 convolutional layers respectively with a stride of 2 in each. Neural networks are defined by 64x3x3 convolutional layers. Neural network is defined with a stride of 2. Neural networks are defined by , 128x3x3 convolutional layers and network has a stride of 2. All stochastic latent variables are parameterized by convolutional layers with feature maps for respectively. The kernel width and height of the stochastic latent variables are defined similarly to the dimension of the subsequent output after striding. We apply a dropout rate of in the deterministic layers of the inference model.
Appendix D Initial Results on Text Generation Tasks
|Results with autoregressive components, no dropout|
|RNN-VAE (Bowman et al., 2015), , warmup|
|RNN-VAE (Bowman et al., 2015), , finetuned|
|Hybrid VAE (Semeniuta et al., 2017), , finetuned|
|BIVA L=7, , finetuned|
|Results without autoregressive components, no dropout|
|Hybrid VAE (Semeniuta et al., 2017), , finetuned|
|BIVA L=7 Finetuned,|
Optimizing generative models coupled with autoregressive models is a difficult task. Such coupling causes the posterior to collapse, and the latent variables are ignored. Nonetheless, autoregressive components remain a cornerstone of the generative models for text (Bowman et al., 2015; Semeniuta et al., 2017; Shah et al., 2018). In order to enforce the model to use the latent variable, previous efforts aim at weakening the decoder using powerful regularizing tricks, such as word dropout (Bowman et al., 2015). We investigate the use of BIVA in the context of sentence modelling without weakening the decoder. We show that it allows optimizing the latent variables more effectively, resulting in a higher measured KL when compared to the RNN-VAE (Bowman et al., 2015) and the Hybrid VAE (Semeniuta et al., 2017).
We use the Bookcorpus dataset (Zhu et al., 2015) of sentences of maximum 40 words, no preprocessing is performed and sentences are tokenized using the white spaces. We defined a vocabulary of words and filtered out the sentences that contain non-indexed tokens. We randomly sampled sentences for testing and used the remaining M sentences for training.
We couple BIVA with an LSTM decoder, using the output of the convolutional model as an input sequence for the auto-regressive model. We compare our model against a LSTM language model (Hochreiter & Schmidhuber, 1997), the RNN-VAE (Bowman et al., 2015), and the Hybrid VAE (Semeniuta et al., 2017), which couples a convolutional architecture with an LSTM decoder. We also perform experiments without using autoregressive components.
All LSTM models are paremterized by units and we use embeddings of dimension . This results in an RNN-VAE model with 23.7M parameters and we limit the other models to use the same total number of parameters. This results in using a limited number of stochastic layers for the BIVA and small a small number of kernels of .
We trained the models for epochs with an initial learning rate of using the Adamax optimizer. We used batches of size and used only one stochastic sample. We train all latent variable models using the free bits method from (Kingma et al., 2016) with an initial KL budget of nats distributed equally over the stochastic variables and we incrementally decrease the free bits value on plateau. We also train the RNN-VAE baseline using the deterministic warmup method (Bowman et al., 2015; Sønderby et al., 2016) for comparison.
Likelihood and latent variables usage.
We report the test set results in table 7 and test samples in 8 and reconstructions in table 9. While BIVA without the autoregressive decoder is not competitive with an LSTM language model, we observe that replacing the LSTM inference model by a BIVA model allows exploiting the latent space more actively, which results in a higher measured KL than the RNN-VAE and Hybrid VAE baselines.
|he said .||“ two .|
|i tried to think of something to say to him , but he was already on his way back to the house .||“ you do n’t have to do this . ”|
|it sounded as if he was going to say something .||the light from the lamp was dim , but the light was dim and the room was dark .|
|“ and that ’s why you ’re coming . ”||or a nuclear bomb , or something .|
|“ what ? ”||“ the baby ? ”|
|she swallowed .||“ you ’re not going to kill me . ”|
|“ i want you . ”||she was n’t going to .|
|glancing up , i saw the way he was staring at me with a look of pure hatred .||“ i guess we could have been more careful , ” he said .|
|i need a favor . ”||there are some things that are not good .|
|he did n’t .||“ you ’re a good man .|
|you ’re not dead .||i had n’t been able to get it out .|
|i stood , and he followed .||“ you ’re going to have to be careful , ” he said .|
|“ can i sit on the couch and talk ? ”||it ’s not a bad idea .|
|“ it was n’t until i was fifteen , i was n’t in the mood to be around .||he asked .|
|i looked down at my lap .||“ this is a bad idea , ” he said , his voice a little hoarse .|
|the smile disappeared .||“ i ’m sure he ’s in love with you .|
|it was hard to tell which one was more of a rock .||as he stepped out of the car , he saw the man standing in the doorway , his eyes wide and his face pale .|
|i ’m not sure it ’s a good idea .||.|
|the first two .||“ no .|
|he was there .||“ in the meantime , i need to get some sleep , ” i said .|
|“ all of you , ” joe said .||i was n’t .|
|he did n’t care if he was n’t a vampire .||did i want to talk to you ?|
|her mouth curved up , then she nodded .||“ i want to hear you say it . ”|
|just tell me what you want in the end .||the train was already in the driveway .|
|and again .||“ good .|
|the other man ’s voice was hoarse and ragged .||i just needed to get out of here , and i needed to get out of here .|
|i had n’t known that was a bad idea , but i had n’t been able to get it out of my head .||“ this is a good idea .|
|your brother is the most important thing to me .||“ hey . ”|
|you dont need to go to the police , right ?||she took a deep breath and let it out .|
|there was a long silence .||then he kissed her .|
|i looked up .||i felt a warm hand on my shoulder and a warm smile spread across my face .|
|he nodded , and he looked at me , and i could tell he was thinking about it .||“ he ’s dead . ”|
|“ hang on , baby .||at the time , i was going to have to get out of the house .|
|we had to be close to the city , and we could n’t afford to be here .||he was so close to the edge of the bed .|
|you know , it would be better if you were n’t so stupid . ”||“ i do n’t know .|
|excuse me ?||“ i do n’t have a choice . ”|
|you know how much i love you , too .||i know i ’m not going to let him touch me , but i do .|
|a woman ’s voice , a voice that was familiar .||i could n’t see the face of the man who ’d just been in the doorway .|
|i have a very important business to attend to , and i ’m going to have to make a decision .||in the end , we all know that we are not going to be able to get out of this .|
|they sat on the small wooden table in the center of the room .||“ yes .|
|“ it ’s fine . ”||“ what are you doing here ? ”|
|she felt a rush of relief .||so the only thing that mattered was that he was here .|
|maria , he says .||neither of them spoke .|
|what ?||from now on , you will be able to get out of here .|
|“ it does n’t seem like a lot to me , ” he said .||the thought of having to kill him made him want to kill her .|
|he ’d told her everything .||the other two were staring at me , their eyes wide .|
|“ she ’s in shock .||i did n’t want to be a part of it , but i was n’t going to let it go .|
|“ after all , ” he murmured , “ i ’m going to go get the rest of the stuff . ”||“ i do n’t want to talk about it .|
|and then , finally , she ’d done it .||she looked at him , her eyes wide .|
|her words were a whisper , but it was n’t enough .||“ that ’s what you ’re going to do .|
|“ a sad song , being sung alone in the basement . ”||“ it sounds like you ’ve been through a lot . ”||“ you ’re going to be a great father . ”|
|more often , though , wherever she sank , beck was there .||more than anything , she wanted to be with him .||in the end , we all knew what was going on .|
|he looked just about as pale as i had ever seen him .||he ’s still a lot more than a friend .||he was n’t going to let her go .|
|caleb turned and shoved him back as he took his true form .||he lifted me up , his arms still wrapped around my waist .||he was standing in the doorway , his hands folded in front of him .|
|i gasped , tried to pull away , squeezed my legs together .||i gasped , and he was n’t able to stop himself .||i felt my body tense , and i could n’t help but smile .|
|i agreed as i adjusted myself and sat heavily in my chair .||i tried to ignore it , but my eyes were still closed .||i did n’t want to be the one to tell him .|
|you bind me , UNK in darkness , though , in light .||he ’d decided to take her home , to make her feel safe .||he was more than willing to let her go .|
|they promise me things , ask me questions , whisper and plead .||they might be able to do something about it , but they do n’t .||“ we need to talk , ” he said , his voice low .|
|i glowed as i held the bear , almost bigger than me .||i started to close my eyes , but he was too strong .||i could n’t help but smile at the sight of her .|
|i wonder how much he pays them to be his guard dogs .||i had to admit that it was n’t a good idea .||i do n’t want to be a part of this .|
|“ hmmm , ” richard muttered , and headed up the path .||“ jesus , ” he said , his voice barely audible .||“ but you ’re going to be a father .|
|he was happy that he had found it in the UNK hall .||he was n’t going to be the one to go .||he was n’t sure if he was going to make it .|
|at the shack , at the condo , at the hangar . ”||at the moment , the only thing that mattered was that he was n’t alone .||he was staring at the floor , his eyes wide .|
|“ i ’d pop to go to the dance with you . ”||“ i ’d prefer to go to the hospital .||“ i ’m going to go to the bathroom .|
|someday , i ’ll share them with the rest of the world .||and now i have a lot of my own .||“ we ’re going to have to do something about it .|
|“ maybe i ’m not the right person for this one ” .||“ maybe we can get a little more of a ride . ”||“ i do n’t think you ’re going to be able to do that . ”|
|“ gin is my sister , and she ’s coming with me .||“ there ’s a chance i can get a little more sleep . ”||“ if you want to , i ’ll be there . ”|
|thick desire stormed her … along with a bittersweet curl of emotion .||the tension was gone , and he was n’t looking at me .||the air smelled of stale cigarette smoke .|
|they caused him to stagger back and drove him to the ground .||they had to be at the top of the hill .||he ’d found a way to get her to safety .|
|you ’re not much of a friar , friar , he says .||you ’re not supposed to be around here , are you ? ”||you ’re not going to be able to do that , are you ? ”|
Appendix E Additional Results