In crowd scenarios, reliable trajectory prediction of pedestrians requiresinsightful understanding of their social behaviors. These behaviors have beenwell investigated by plenty of studies, while it is hard to be fully expressedby hand-craft rules. Recent studies based on LSTM networks have shown greatability to learn social behaviors. However, many of these methods rely onprevious neighboring hidden states but ignore the important current intentionof the neighbors. In order to address this issue, we propose a data-drivenstate refinement module for LSTM network (SR-LSTM), which activates theutilization of the current intention of neighbors, and jointly and iterativelyrefines the current states of all participants in the crowd through a messagepassing mechanism. To effectively extract the social effect of neighbors, wefurther introduce a social-aware information selection mechanism consisting ofan element-wise motion gate and a pedestrian-wise attention to select usefulmessage from neighboring pedestrians. Experimental results on two publicdatasets, i.e. ETH and UCY, demonstrate the effectiveness of our proposedSR-LSTM and we achieves state-of-the-art results.