Sender-receiver interactions, and specifically persuasion games, are widelyresearched in economic modeling and artificial intelligence, and serve as asolid foundation for powerful applications. However, in the classic persuasiongames setting, the messages sent from the expert to the decision-maker areabstract or well-structured application-specific signals rather than natural(human) language messages, although natural language is a very commoncommunication signal in real-world persuasion setups. This paper addresses theuse of natural language in persuasion games, exploring its impact on thedecisions made by the players and aiming to construct effective models for theprediction of these decisions. For this purpose, we conduct an online repeatedinteraction experiment. At each trial of the interaction, an informed expertaims to sell an uninformed decision-maker a vacation in a hotel, by sending hera review that describes the hotel. While the expert is exposed to severalscored reviews, the decision-maker observes only the single review sent by theexpert, and her payoff in case she chooses to take the hotel is a random drawfrom the review score distribution available to the expert only. The expert'spayoff, in turn, depends on the number of times the decision-maker chooses thehotel. We consider a number of modeling approaches for this setup, differingfrom each other in the model type (deep neural network (DNN) vs. linearclassifier), the type of features used by the model (textual, behavioral orboth) and the source of the textual features (DNN-based vs. hand-crafted). Ourresults demonstrate that given a prefix of the interaction sequence, our modelscan predict the future decisions of the decision-maker, particularly when asequential modeling approach and hand-crafted textual features are applied.