Heterogeneous ensembles that can aggregate an unrestricted number and varietyof base predictors can effectively address challenging prediction problems. Inparticular, accurate ensembles that are also parsimonious, i.e., consist of asfew base predictors as possible, can help reveal potentially useful knowledgeabout the target problem domain. Although ensemble selection offers a potentialapproach to achieving these goals, the currently available algorithms arelimited in their abilities. In this paper, we present several algorithms thatincorporate ensemble diversity into a reinforcement learning (RL)-basedensemble selection framework to build accurate and parsimonious ensembles.These algorithms, as well as several baselines, are rigorously evaluated ondatasets from diverse domains in terms of the predictive performance andparsimony of their ensembles. This evaluation demonstrates that ourdiversity-incorporated RL-based algorithms perform better than the others forconstructing simultaneously accurate and parsimonious ensembles. Thesealgorithms can eventually aid the interpretation or reverse engineering ofpredictive models assimilated into effective ensembles. To enable such atranslation, an implementation of these algorithms, as well the experimentalsetup they are evaluated in, has been made available athttps://github.com/GauravPandeyLab/lens-learning-ensembles-using-reinforcement-learning.