Since deep neural networks were developed, they have made huge contributionsto everyday lives. Machine learning provides more rational advice than humansare capable of in almost every aspect of daily life. However, despite thisachievement, the design and training of neural networks are still challengingand unpredictable procedures. To lower the technical thresholds for commonusers, automated hyper-parameter optimization (HPO) has become a popular topicin both academic and industrial areas. This paper provides a review of the mostessential topics on HPO. The first section introduces the key hyper-parametersrelated to model training and structure, and discusses their importance andmethods to define the value range. Then, the research focuses on majoroptimization algorithms and their applicability, covering their efficiency andaccuracy especially for deep learning networks. This study next reviews majorservices and toolkits for HPO, comparing their support for state-of-the-artsearching algorithms, feasibility with major deep learning frameworks, andextensibility for new modules designed by users. The paper concludes withproblems that exist when HPO is applied to deep learning, a comparison betweenoptimization algorithms, and prominent approaches for model evaluation withlimited computational resources.