When a natural language generation (NLG) component is implemented in areal-world task-oriented dialogue system, it is necessary to generate not onlynatural utterances as learned on training data but also utterances adapted tothe dialogue environment (e.g., noise from environmental sounds) and the user(e.g., users with low levels of understanding ability). Inspired by recentadvances in reinforcement learning (RL) for language generation tasks, wepropose ANTOR, a method for Adaptive Natural language generation forTask-Oriented dialogue via Reinforcement learning. In ANTOR, a natural languageunderstanding (NLU) module, which corresponds to the user's understanding ofsystem utterances, is incorporated into the objective function of RL. If theNLG's intentions are correctly conveyed to the NLU, which understands asystem's utterances, the NLG is given a positive reward. We conductedexperiments on the MultiWOZ dataset, and we confirmed that ANTOR could generateadaptive utterances against speech recognition errors and the differentvocabulary levels of users.