We analyze and reframe AI progress. In addition to the prevailing metrics ofperformance, we highlight the usually neglected costs paid in the developmentand deployment of a system, including: data, expert knowledge, human oversight,software resources, computing cycles, hardware and network facilities,development time, etc. These costs are paid throughout the life cycle of an AIsystem, fall differentially on different individuals, and vary in magnitudedepending on the replicability and generality of the AI solution. Themultidimensional performance and cost space can be collapsed to a singleutility metric for a user with transitive and complete preferences. Even absenta single utility function, AI advances can be generically assessed by whetherthey expand the Pareto (optimal) surface. We explore a subset of theseneglected dimensions using the two case studies of Alpha* and ALE. Thisbroadened conception of progress in AI should lead to novel ways of measuringsuccess in AI, and can help set milestones for future progress.