Abstract
The intrinsic high dimension of fluid dynamics is an inherent challenge tocontrol of aerodynamic flows, and this is further complicated by a flow'snonlinear response to strong disturbances. Deep reinforcement learning, whichtakes advantage of the exploratory aspects of reinforcement learning (RL) andthe rich nonlinearity of a deep neural network, provides a promising approachto discover feasible control strategies. However, the typical model-freeapproach to reinforcement learning requires a significant amount of interactionbetween the flow environment and the RL agent during training, and this hightraining cost impedes its development and application. In this work, we proposea model-based reinforcement learning (MBRL) approach by incorporating a novelreduced-order model as a surrogate for the full environment. The model consistsof a physics-augmented autoencoder, which compresses high-dimensional CFD flowfield snaphsots into a three-dimensional latent space, and a latent dynamicsmodel that is trained to accurately predict the long-time dynamics oftrajectories in the latent space in response to action sequences. The accuracyand robustness of the model are demonstrated in the scenario of a pitchingairfoil within a highly disturbed environment. Additionally, an application toa vertical-axis wind turbine in a disturbance-free environment is discussed inthe Appendix Based on the model trained in the pitching airfoil problem, werealize an MBRL strategy to mitigate lift variation during gust-airfoilencounters. We demonstrate that the policy learned in the reduced-orderenvironment translates to an effective control strategy in the full CFDenvironment.