Abstract
We apply neural nets with ReLU gates in online reinforcement learning. Ourgoal is to train these networks in an incremental manner, without thecomputationally expensive experience replay. By studying how individual neuralnodes behave in online training, we recognize that the global nature of ReLUgates can cause undesirable learning interference in each node's learningbehavior. We propose reducing such interferences with two efficient inputtransformation methods that are geometric in nature and match well thegeometric property of ReLU gates. The first one is tile coding, a classicbinary encoding scheme originally designed for local generalization based onthe topological structure of the input space. The second one (EmECS) is a newmethod we introduce; it is based on geometric properties of convex sets andtopological embedding of the input space into the boundary of a convex set. Wediscuss the behavior of the network when it operates on the transformed inputs.We also compare it experimentally with some neural nets that do not use thesame input transformations, and with the classic algorithm of tile coding plusa linear function approximator, and on several online reinforcement learningtasks, we show that the neural net with tile coding or EmECS can achieve notonly faster learning but also more accurate approximations. Our resultsstrongly suggest that geometric input transformation of this type can beeffective for interference reduction and takes us a step closer to fullyincremental reinforcement learning with neural nets.