We propose Neural Actor (NA), a new method for high-quality synthesis ofhumans from arbitrary viewpoints and under arbitrary controllable poses. Ourmethod is built upon recent neural scene representation and rendering workswhich learn representations of geometry and appearance from only 2D images.While existing works demonstrated compelling rendering of static scenes andplayback of dynamic scenes, photo-realistic reconstruction and rendering ofhumans with neural implicit methods, in particular under user-controlled novelposes, is still difficult. To address this problem, we utilize a coarse bodymodel as the proxy to unwarp the surrounding 3D space into a canonical pose. Aneural radiance field learns pose-dependent geometric deformations and pose-and view-dependent appearance effects in the canonical space from multi-viewvideo input. To synthesize novel views of high fidelity dynamic geometry andappearance, we leverage 2D texture maps defined on the body model as latentvariables for predicting residual deformations and the dynamic appearance.Experiments demonstrate that our method achieves better quality than thestate-of-the-arts on playback as well as novel pose synthesis, and can evengeneralize well to new poses that starkly differ from the training poses.Furthermore, our method also supports body shape control of the synthesizedresults.