Research at the intersection of machine learning and the social sciences hasprovided critical new insights into social behavior. At the same time, avariety of critiques have been raised ranging from technical issues with thedata used and features constructed, problematic assumptions built into models,their limited interpretability, and their contribution to bias and inequality.We argue such issues arise primarily because of the lack of social theory atvarious stages of the model building and analysis. In the first half of thispaper, we walk through how social theory can be used to answer the basicmethodological and interpretive questions that arise at each stage of themachine learning pipeline. In the second half, we show how theory can be usedto assess and compare the quality of different social learning models,including interpreting, generalizing, and assessing the fairness of models. Webelieve this paper can act as a guide for computer and social scientists aliketo navigate the substantive questions involved in applying the tools of machinelearning to social data.