The problem of Reinforcement Learning (RL) in an unknown nonlinear dynamicalsystem is equivalent to the search for an optimal feedback law utilizing thesimulations/ rollouts of the unknown dynamical system. Most RL techniquessearch over a complex global nonlinear feedback parametrization making themsuffer from high training times as well as variance. Instead, we advocatesearching over a local feedback representation consisting of an open-loopsequence, and an associated optimal linear feedback law completely determinedby the open-loop. We show that this alternate approach results in highlyefficient training, the answers obtained are repeatable and hence reliable, andthe resulting closed performance is superior to global state of the art RLtechniques. Finally, if we replan, whenever required, which is feasible due tothe fast and reliable local solution, allows us to recover global optimality ofthe resulting feedback law.