We present an algorithm to directly solve numerous image restoration problems(e.g., image deblurring, image dehazing, image deraining, etc.). These problemsare highly ill-posed, and the common assumptions for existing methods areusually based on heuristic image priors. In this paper, we find that theseproblems can be solved by generative models with adversarial learning. However,the basic formulation of generative adversarial networks (GANs) does notgenerate realistic images, and some structures of the estimated images areusually not preserved well. Motivated by an interesting observation that theestimated results should be consistent with the observed inputs under thephysics models, we propose a physics model constrained learning algorithm sothat it can guide the estimation of the specific task in the conventional GANframework. The proposed algorithm is trained in an end-to-end fashion and canbe applied to a variety of image restoration and related low-level visionproblems. Extensive experiments demonstrate that our method performs favorablyagainst the state-of-the-art algorithms.