Unmanned Aerial Systems (UAS) are being increasingly deployed for commercial,civilian, and military applications. The current UAS state-of-the-art stilldepends on a remote human controller with robust wireless links to performseveral of these applications. The lack of autonomy restricts the domains ofapplication and tasks for which a UAS can be deployed. Enabling autonomy andintelligence to the UAS will help overcome this hurdle and expand its useimproving safety and efficiency. The exponential increase in computingresources and the availability of large amount of data in this digital era hasled to the resurgence of machine learning from its last winter. Therefore, inthis chapter, we discuss how some of the advances in machine learning,specifically deep learning and reinforcement learning can be leveraged todevelop next-generation autonomous UAS. We first begin motivating this chapterby discussing the application, challenges, and opportunities of the current UASin the introductory section. We then provide an overview of some of the keydeep learning and reinforcement learning techniques discussed throughout thischapter. A key area of focus that will be essential to enable autonomy to UASis computer vision. Accordingly, we discuss how deep learning approaches havebeen used to accomplish some of the basic tasks that contribute to providingUAS autonomy. Then we discuss how reinforcement learning is explored for usingthis information to provide autonomous control and navigation for UAS. Next, weprovide the reader with directions to choose appropriate simulation suites andhardware platforms that will help to rapidly prototype novel machine learningbased solutions for UAS. We additionally discuss the open problems andchallenges pertaining to each aspect of developing autonomous UAS solutions toshine light on potential research areas.