Abstract
We investigate using reinforcement learning agents as generative models ofimages (extending arXiv:1804.01118). A generative agent controls a simulatedpainting environment, and is trained with rewards provided by a discriminatornetwork simultaneously trained to assess the realism of the agent's samples,either unconditional or reconstructions. Compared to prior work, we make anumber of improvements to the architectures of the agents and discriminatorsthat lead to intriguing and at times surprising results. We find that whensufficiently constrained, generative agents can learn to produce images with adegree of visual abstraction, despite having only ever seen real photographs(no human brush strokes). And given enough time with the painting environment,they can produce images with considerable realism. These results show that,under the right circumstances, some aspects of human drawing can emerge fromsimulated embodiment, without the need for external supervision, imitation orsocial cues. Finally, we note the framework's potential for use in creativeapplications.