Reinforcement learning has been found useful in solving optimal power flow(OPF) problems in electric power distribution systems. However, the use oflargely model-free reinforcement learning algorithms that completely ignore thephysics-based modeling of the power grid compromises the optimizer performanceand poses scalability challenges. This paper proposes a novel approach tosynergistically combine the physics-based models with learning-based algorithmsusing imitation learning to solve distribution-level OPF problems.Specifically, we propose imitation learning based improvements in deepreinforcement learning (DRL) methods to solve the OPF problem for a specificcase of battery storage dispatch in the power distribution systems. Theproposed imitation learning algorithm uses the approximate optimal solutionsobtained from a linearized model-based OPF solver to provide a good initialpolicy for the DRL algorithms while improving the training efficiency. Theeffectiveness of the proposed approach is demonstrated using IEEE 34-bus and123-bus distribution feeders with numerous distribution-level battery storagesystems.