Abstract
A major obstacle in developing robust and generalizable smart home-basedHuman Activity Recognition (HAR) systems is the lack of large-scale, diverselabeled datasets. Variability in home layouts, sensor configurations, and userbehavior adds further complexity, as individuals follow varied routines andperform activities in distinct ways. Building HAR systems that generalize wellrequires training data that captures the diversity across users andenvironments. To address these challenges, we introduce AgentSense, a virtualdata generation pipeline where diverse personas are generated by leveragingLarge Language Models. These personas are used to create daily routines, whichare then decomposed into low-level action sequences. Subsequently, the actionsare executed in a simulated home environment called VirtualHome that weextended with virtual ambient sensors capable of recording the agentsactivities as they unfold. Overall, AgentSense enables the generation of rich,virtual sensor datasets that represent a wide range of users and home settings.Across five benchmark HAR datasets, we show that leveraging our virtual sensordata substantially improves performance, particularly when real data arelimited. Notably, models trained on a combination of virtual data and just afew days of real data achieve performance comparable to those trained on theentire real datasets. These results demonstrate and prove the potential ofvirtual data to address one of the most pressing challenges in ambient sensing,which is the distinct lack of large-scale, annotated datasets without requiringany manual data collection efforts.