This paper investigates the vision-based autonomous driving with deeplearning and reinforcement learning methods. Different from the end-to-endlearning method, our method breaks the vision-based lateral control system downinto a perception module and a control module. The perception module which isbased on a multi-task learning neural network first takes a driver-view imageas its input and predicts the track features. The control module which is basedon reinforcement learning then makes a control decision based on thesefeatures. In order to improve the data efficiency, we propose visual TORCS(VTORCS), a deep reinforcement learning environment which is based on the openracing car simulator (TORCS). By means of the provided functions, one can trainan agent with the input of an image or various physical sensor measurement, orevaluate the perception algorithm on this simulator. The trained reinforcementlearning controller outperforms the linear quadratic regulator (LQR) controllerand model predictive control (MPC) controller on different tracks. Theexperiments demonstrate that the perception module shows promising performanceand the controller is capable of controlling the vehicle drive well along thetrack center with visual input.