Abstract
Scientific computing is increasingly incorporating the advancements inmachine learning and the ability to work with large amounts of data. At thesame time, machine learning models are becoming increasingly sophisticated andexhibit many features often seen in scientific computing, stressing thecapabilities of machine learning frameworks. Just as the disciplines ofscientific computing and machine learning have shared common underlyinginfrastructure in the form of numerical linear algebra, we now have theopportunity to further share new computational infrastructure, and thus ideas,in the form of Differentiable Programming. We describe Zygote, a DifferentiableProgramming system that is able to take gradients of general programstructures. We implement this system in the Julia programming language. Oursystem supports almost all language constructs (control flow, recursion,mutation, etc.) and compiles high-performance code without requiring any userintervention or refactoring to stage computations. This enables an expressiveprogramming model for deep learning, but more importantly, it enables us toincorporate a large ecosystem of libraries in our models in a straightforwardway. We discuss our approach to automatic differentiation, including itssupport for advanced techniques such as mixed-mode, complex and checkpointeddifferentiation, and present several examples of differentiating programs.