Reinforcement learning is important part of artificial intelligence. In thispaper, we review model-free reinforcement learning that utilizes the averagereward optimality criterion in the infinite horizon setting. Motivated by thesolo survey by Mahadevan (1996a), we provide an updated review of work in thisarea and extend it to cover policy-iteration and function approximation methods(in addition to the value-iteration and tabular counterparts). We present acomprehensive literature mapping. We also identify and discuss opportunitiesfor future work.