Researchers have demonstrated that Deep Reinforcement Learning (DRL) is apowerful tool for finding policies that perform well on complex roboticsystems. However, these policies are often unpredictable and can induce highlyvariable behavior when evaluated with only slightly different initialconditions. Training considerations constrain DRL algorithm designs in thatmost algorithms must use stochastic policies during training. The resultingpolicy used during deployment, however, can and frequently is a deterministicone that uses the Maximum Likelihood Action (MLA) at each step. In this work,we show that a direct random search is very effective at fine-tuning DRLpolicies by directly optimizing them using deterministic rollouts. Weillustrate this across a large collection of reinforcement learningenvironments, using a wide variety of policies obtained from differentalgorithms. Our results show that this method yields more consistent and higherperforming agents on the environments we tested. Furthermore, we demonstratehow this method can be used to extend our previous work on shrinking thedimensionality of the reachable state space of closed-loop systems run underDeep Neural Network (DNN) policies.