We explore how to enable machines to model 3D shapes like human modelersusing deep reinforcement learning (RL). In 3D modeling software like Maya, amodeler usually creates a mesh model in two steps: (1) approximating the shapeusing a set of primitives; (2) editing the meshes of the primitives to createdetailed geometry. Inspired by such artist-based modeling, we propose atwo-step neural framework based on RL to learn 3D modeling policies. By takingactions and collecting rewards in an interactive environment, the agents firstlearn to parse a target shape into primitives and then to edit the geometry. Toeffectively train the modeling agents, we introduce a novel training algorithmthat combines heuristic policy, imitation learning and reinforcement learning.Our experiments show that the agents can learn good policies to produce regularand structure-aware mesh models, which demonstrates the feasibility andeffectiveness of the proposed RL framework.