Central Pattern Generators (CPGs) are biological neural circuits capable ofproducing coordinated rhythmic outputs in the absence of rhythmic input. As aresult, they are responsible for most rhythmic motion in living organisms. Thisrhythmic control is broadly applicable to fields such as locomotive roboticsand medical devices. In this paper, we explore the possibility of creating aself-sustaining CPG network for reinforcement learning that learns rhythmicmotion more efficiently and across more general environments than the currentmultilayer perceptron (MLP) baseline models. Recent work introduces theStructured Control Net (SCN), which maintains linear and nonlinear modules forlocal and global control, respectively. Here, we show that time-sequencearchitectures such as Recurrent Neural Networks (RNNs) model CPGs effectively.Combining previous work with RNNs and SCNs, we introduce the Recurrent ControlNet (RCN), which adds a linear component to the, RCNs match and exceed theperformance of baseline MLPs and SCNs across all environment tasks. Ourfindings confirm existing intuitions for RNNs on reinforcement learning tasks,and demonstrate promise of SCN-like structures in reinforcement learning.