Candidate generation is the first stage in recommendation systems, where alight-weight system is used to retrieve potentially relevant items for an inputuser. These candidate items are then ranked and pruned in later stages ofrecommender systems using a more complex ranking model. Since candidategeneration is the top of the recommendation funnel, it is important to retrievea high-recall candidate set to feed into downstream ranking models. A commonapproach for candidate generation is to leverage approximate nearest neighbor(ANN) search from a single dense query embedding; however, this approach thiscan yield a low-diversity result set with many near duplicates. As users oftenhave multiple interests, candidate retrieval should ideally return a diverseset of candidates reflective of the user's multiple interests. To this end, weintroduce kNN-Embed, a general approach to improving diversity in denseANN-based retrieval. kNN-Embed represents each user as a smoothed mixture overlearned item clusters that represent distinct `interests' of the user. Byquerying each of a user's mixture component in proportion to their mixtureweights, we retrieve a high-diversity set of candidates reflecting elementsfrom each of a user's interests. We experimentally compare kNN-Embed tostandard ANN candidate retrieval, and show significant improvements in overallrecall and improved diversity across three datasets. Accompanying this work, weopen source a large Twitter follow-graph dataset, to spur further research ingraph-mining and representation learning for recommender systems.