Large-scale pre-trained language models have achieved great success onnatural language generation tasks. However, it is difficult to control thepre-trained language models to generate sentences with the desired attributesuch as topic and sentiment, etc. Recently, Bayesian Controllable LanguageModels (BCLMs) have been shown to be efficient in controllable languagegeneration. Rather than fine-tuning the parameters of pre-trained languagemodels, BCLMs use external discriminators to guide the generation ofpre-trained language models. However, the mismatch between training andinference of BCLMs limits the performance of the models. To address theproblem, in this work we propose a "Gemini Discriminator" for controllablelanguage generation which alleviates the mismatch problem with a smallcomputational cost. We tested our method on two controllable languagegeneration tasks: sentiment control and topic control. On both tasks, ourmethod reached achieved new state-of-the-art results in automatic and humanevaluations.