Abstract
As language technologies become widespread, it is important to understand howchanges in language affect reader perceptions and behaviors. Theserelationships may be formalized as the isolated causal effect of some focallanguage-encoded intervention (e.g., factual inaccuracies) on an externaloutcome (e.g., readers' beliefs). In this paper, we introduce a formalestimation framework for isolated causal effects of language. We show that acore challenge of estimating isolated effects is the need to approximate allnon-focal language outside of the intervention. Drawing on the principle ofomitted variable bias, we provide measures for evaluating the quality of bothnon-focal language approximations and isolated effect estimates themselves. Wefind that poor approximation of non-focal language can lead to bias in thecorresponding isolated effect estimates due to omission of relevant variables,and we show how to assess the sensitivity of effect estimates to such biasalong the two key axes of fidelity and overlap. In experiments onsemi-synthetic and real-world data, we validate the ability of our framework tocorrectly recover isolated effects and demonstrate the utility of our proposedmeasures.