Abstract
In the era of generative AI, deep generative models (DGMs) with latentrepresentations have gained tremendous popularity. Despite their impressiveempirical performance, the statistical properties of these models remainunderexplored. DGMs are often overparametrized, non-identifiable, anduninterpretable black boxes, raising serious concerns when deploying them inhigh-stakes applications. Motivated by this, we propose an interpretable deepgenerative modeling framework for rich data types with discrete latent layers,called Deep Discrete Encoders (DDEs). A DDE is a directed graphical model withmultiple binary latent layers. Theoretically, we propose transparentidentifiability conditions for DDEs, which imply progressively smaller sizes ofthe latent layers as they go deeper. Identifiability ensures consistentparameter estimation and inspires an interpretable design of the deeparchitecture. Computationally, we propose a scalable estimation pipeline of alayerwise nonlinear spectral initialization followed by a penalized stochasticapproximation EM algorithm. This procedure can efficiently estimate models withexponentially many latent components. Extensive simulation studies validate ourtheoretical results and demonstrate the proposed algorithms' excellentperformance. We apply DDEs to three diverse real datasets for hierarchicaltopic modeling, image representation learning, response time modeling ineducational testing, and obtain interpretable findings.