Recent knowledge enhanced pre-trained language models have shown remarkableperformance on downstream tasks by incorporating structured knowledge fromexternal sources into language models. However, they usually suffer from aheterogeneous information alignment problem and a noisy knowledge injectionproblem. For complex reasoning, the contexts contain rich knowledge thattypically exists in complex and sparse forms. In order to model structuredknowledge in the context and avoid these two problems, we propose to unifystructure reasoning and language model pre-training. It identifies four typesof elementary knowledge structures from contexts to construct structuredqueries, and utilizes the box embedding method to conduct explicit structurereasoning along queries during language modeling. To fuse textual andstructured semantics, we utilize contextual language representations ofknowledge structures to initialize their box embeddings for structurereasoning. We conduct experiments on complex language reasoning and knowledgegraph (KG) reasoning tasks. The results show that our model can effectivelyenhance the performance of complex reasoning of both language and KGmodalities.