Abstract
Machine learning (ML), especially deep learning is made possible by theavailability of big data, enormous compute power and, often overlooked,development tools or frameworks. As the algorithms become mature and efficient,more and more ML inference is moving out of datacenters/cloud and deployed onedge devices. This model deployment process can be challenging as thedeployment environment and requirements can be substantially different fromthose during model development. In this paper, we propose a new ML developmentand deployment approach that is specially designed and optimized forinference-only deployment on edge devices. We build a prototype and demonstratethat this approach can address all the deployment challenges and result in moreefficient and high-quality solutions.