Abstract
Standardized surveys scale efficiently but sacrifice depth, whileconversational interviews improve response quality at the cost of scalabilityand consistency. This study bridges the gap between these methods byintroducing a framework for AI-assisted conversational interviewing. Toevaluate this framework, we conducted a web survey experiment where 1,800participants were randomly assigned to AI 'chatbots' which use large languagemodels (LLMs) to dynamically probe respondents for elaboration andinteractively code open-ended responses to fixed questions developed by humanresearchers. We assessed the AI chatbot's performance in terms of codingaccuracy, response quality, and respondent experience. Our findings reveal thatAI chatbots perform moderately well in live coding even without survey-specificfine-tuning, despite slightly inflated false positive errors due to respondentacquiescence bias. Open-ended responses were more detailed and informative, butthis came at a slight cost to respondent experience. Our findings highlight thefeasibility of using AI methods such as chatbots enhanced by LLMs to enhanceopen-ended data collection in web surveys.