Recent work on fairness in machine learning has primarily emphasized how todefine, quantify, and encourage "fair" outcomes. Less attention has been paid,however, to the ethical foundations which underlie such efforts. Among theethical perspectives that should be taken into consideration isconsequentialism, the position that, roughly speaking, outcomes are all thatmatter. Although consequentialism is not free from difficulties, and althoughit does not necessarily provide a tractable way of choosing actions (because ofthe combined problems of uncertainty, subjectivity, and aggregation), itnevertheless provides a powerful foundation from which to critique the existingliterature on machine learning fairness. Moreover, it brings to the fore someof the tradeoffs involved, including the problem of who counts, the pros andcons of using a policy, and the relative value of the distant future. In thispaper we provide a consequentialist critique of common definitions of fairnesswithin machine learning, as well as a machine learning perspective onconsequentialism. We conclude with a broader discussion of the issues oflearning and randomization, which have important implications for the ethics ofautomated decision making systems.