Abstract
The quest for performant networks has been a significant force that drivesthe advancements of deep learning in recent years. While rewarding, improvingnetwork design has never been an easy journey. The large design space combinedwith the tremendous cost required for network training poses a major obstacleto this endeavor. In this work, we propose a new approach to this problem,namely, predicting the performance of a network before training, based on itsarchitecture. Specifically, we develop a unified way to encode individuallayers into vectors and bring them together to form an integrated descriptionvia LSTM. Taking advantage of the recurrent network's strong expressive power,this method can reliably predict the performances of various networkarchitectures. Our empirical studies showed that it not only achieved accuratepredictions but also produced consistent rankings across datasets -- a keydesideratum in performance prediction.