Abstract
Organizations rely on machine learning engineers (MLEs) to operationalize ML,i.e., deploy and maintain ML pipelines in production. The process ofoperationalizing ML, or MLOps, consists of a continual loop of (i) datacollection and labeling, (ii) experimentation to improve ML performance, (iii)evaluation throughout a multi-staged deployment process, and (iv) monitoring ofperformance drops in production. When considered together, theseresponsibilities seem staggering -- how does anyone do MLOps, what are theunaddressed challenges, and what are the implications for tool builders? We conducted semi-structured ethnographic interviews with 18 MLEs workingacross many applications, including chatbots, autonomous vehicles, and finance.Our interviews expose three variables that govern success for a production MLdeployment: Velocity, Validation, and Versioning. We summarize common practicesfor successful ML experimentation, deployment, and sustaining productionperformance. Finally, we discuss interviewees' pain points and anti-patterns,with implications for tool design.