Bi-directional LSTMs are a powerful tool for text representation. On theother hand, they have been shown to suffer various limitations due to theirsequential nature. We investigate an alternative LSTM structure for encodingtext, which consists of a parallel state for each word. Recurrent steps areused to perform local and global information exchange between wordssimultaneously, rather than incremental reading of a sequence of words. Resultson various classification and sequence labelling benchmarks show that theproposed model has strong representation power, giving highly competitiveperformances compared to stacked BiLSTM models with similar parameter numbers.