Abstract
The recent ground-breaking advances in deep learning networks ( DNNs ) makethem attractive for embedded systems. However, it can take a long time for DNNsto make an inference on resource-limited embedded devices. Offloading thecomputation into the cloud is often infeasible due to privacy concerns, highlatency, or the lack of connectivity. As such, there is a critical need to finda way to effectively execute the DNN models locally on the devices. This paperpresents an adaptive scheme to determine which DNN model to use for a giveninput, by considering the desired accuracy and inference time. Our approachemploys machine learning to develop a predictive model to quickly select apre-trained DNN to use for a given input and the optimization constraint. Weachieve this by first training off-line a predictive model, and then use thelearnt model to select a DNN model to use for new, unseen inputs. We apply ourapproach to the image classification task and evaluate it on a Jetson TX2embedded deep learning platform using the ImageNet ILSVRC 2012 validationdataset. We consider a range of influential DNN models. Experimental resultsshow that our approach achieves a 7.52% improvement in inference accuracy, anda 1.8x reduction in inference time over the most-capable single DNN model.