Reinforcement learning algorithms can show strong variation in performancebetween training runs with different random seeds. In this paper we explore howthis affects hyperparameter optimization when the goal is to findhyperparameter settings that perform well across random seeds. In particular,we benchmark whether it is better to explore a large quantity of hyperparametersettings via pruning of bad performers, or if it is better to aim for qualityof collected results by using repetitions. For this we consider the SuccessiveHalving, Random Search, and Bayesian Optimization algorithms, the latter twowith and without repetitions. We apply these to tuning the PPO2 algorithm onthe Cartpole balancing task and the Inverted Pendulum Swing-up task. Wedemonstrate that pruning may negatively affect the optimization and thatrepeated sampling does not help in finding hyperparameter settings that performbetter across random seeds. From our experiments we conclude that Bayesianoptimization with a noise robust acquisition function is the best choice forhyperparameter optimization in reinforcement learning tasks.