Understanding VAEs in Fisher-Shannon Plane

  • 2018-07-10 15:47:59
  • Huangjie Zheng, Jiangchao Yao, Ya Zhang, Ivor W. Tsang
  • 3


In information theory, Fisher information and Shannon information (entropy)are respectively used to measure the ability in parameter estimation and theuncertainty among variables. The uncertainty principle asserts a fundamentalrelationship between Fisher information and Shannon information, i.e., the moreFisher information we get, the less Shannon information we gain, and viceversa. This enlightens us about the essence of the encoding/decoding procedurein \emph{variational auto-encoders} (VAEs) and motivates us to investigate VAEsin the Fisher-Shannon plane. Our studies show that the performance of thelatent representation learning and the log-likelihood estimation areintrinsically influenced by the trade-off between Fisher information andShannon information. To flexibly adjust the trade-off, we further propose avariant of VAEs that can explicitly control Fisher information inencoding/decoding mechanism, termed as Fisher auto-encoder (FAE). Throughqualitative and quantitative experiments, we show the complementary propertiesof Fisher information and Shannon information, and give a guide for Fisherinformation conditioning to achieve high resolution reconstruction, disentanglefeature learning, over-fitting/over-regularization resistance, etc.


Introduction (beta)



Conclusion (beta)