Abstract
This paper proposes an image-to-painting translation method that generatesvivid and realistic painting artworks with controllable styles. Different fromprevious image-to-image translation methods that formulate the translation aspixel-wise prediction, we deal with such an artistic creation process in avectorized environment and produce a sequence of physically meaningful strokeparameters that can be further used for rendering. Since a typical vectorrender is not differentiable, we design a novel neural renderer which imitatesthe behavior of the vector renderer and then frame the stroke prediction as aparameter searching process that maximizes the similarity between the input andthe rendering output. We explored the zero-gradient problem on parametersearching and propose to solve this problem from an optimal transportationperspective. We also show that previous neural renderers have a parametercoupling problem and we re-design the rendering network with a rasterizationnetwork and a shading network that better handles the disentanglement of shapeand color. Experiments show that the paintings generated by our method have ahigh degree of fidelity in both global appearance and local textures. Ourmethod can be also jointly optimized with neural style transfer that furthertransfers visual style from other images. Our code and animated results areavailable at \url{https://jiupinjia.github.io/neuralpainter/}.