Abstract
We present an end-to-end framework for generating synthetic users forevaluating interactive agents designed to encourage positive behavior changes,such as in health and lifestyle coaching. The synthetic users are grounded inhealth and lifestyle conditions, specifically sleep and diabetes management inthis study, to ensure realistic interactions with the health coaching agent.Synthetic users are created in two stages: first, structured data are generatedgrounded in real-world health and lifestyle factors in addition to basicdemographics and behavioral attributes; second, full profiles of the syntheticusers are developed conditioned on the structured data. Interactions betweensynthetic users and the coaching agent are simulated using generativeagent-based models such as Concordia, or directly by prompting a languagemodel. Using two independently-developed agents for sleep and diabetes coachingas case studies, the validity of this framework is demonstrated by analyzingthe coaching agent's understanding of the synthetic users' needs andchallenges. Finally, through multiple blinded evaluations of user-coachinteractions by human experts, we demonstrate that our synthetic users withhealth and behavioral attributes more accurately portray real human users withthe same attributes, compared to generic synthetic users not grounded in suchattributes. The proposed framework lays the foundation for efficientdevelopment of conversational agents through extensive, realistic, and groundedsimulated interactions.