Model-based reinforcement learning (MBRL) has shown its advantages insample-efficiency over model-free reinforcement learning (MFRL). Despite theimpressive results it achieves, it still faces a trade-off between the ease ofdata generation and model bias. In this paper, we propose a simple and elegantmodel-embedding model-based reinforcement learning (MEMB) algorithm in theframework of the probabilistic reinforcement learning. To balance thesample-efficiency and model bias, we exploit both real and imaginary data inthe training. In particular, we embed the model in the policy update and learn$Q$ and $V$ functions from the real data set. We provide the theoreticalanalysis of MEMB with the Lipschitz continuity assumption on the model andpolicy. At last, we evaluate MEMB on several benchmarks and demonstrate ouralgorithm can achieve state-of-the-art performance.