Abstract
The emergence of large-scale pre-trained language models, such as ChatGPT,has revolutionized various research fields in artificial intelligence.Transformers-based large language models (LLMs) have gradually replaced CNNsand RNNs to unify fields of computer vision and natural language processing.Compared with the data that exists relatively independently such as images,videos or texts, graph is a type of data that contains rich structural andrelational information. Meanwhile, natural language, as one of the mostexpressive mediums, excels in describing complex structures. However, existingwork on incorporating graph learning problems into the generative languagemodeling framework remains very limited. As the importance of large languagemodels continues to grow, it becomes essential to explore whether LLMs can alsoreplace GNNs as the foundation model for graphs. In this paper, we proposeInstructGLM (Instruction-finetuned Graph Language Model), systematically designhighly scalable prompts based on natural language instructions, and use naturallanguage to describe the geometric structure and node features of the graph forinstruction tuning an LLM to perform learning and inference on graphs in agenerative manner. Our method exceeds all competitive GNN baselines onogbn-arxiv, Cora and PubMed datasets, which demonstrates the effectiveness ofour method and sheds light on generative large language models as thefoundation model for graph machine learning.