Abstract
The emergence of large-scale pre-trained language models has revolutionizedvarious AI research domains. Transformers-based Large Language Models (LLMs)have gradually replaced CNNs and RNNs to unify fields of computer vision andnatural language processing. Compared with independent data samples such asimages, videos or texts, graphs usually contain rich structural and relationalinformation. Meanwhile, language, especially natural language, being one of themost expressive mediums, excels in describing complex structures. However,existing work on incorporating graph problems into the generative languagemodeling framework remains very limited. Considering the rising prominence ofLLMs, it becomes essential to explore whether LLMs can also replace GNNs as thefoundation model for graphs. In this paper, we propose InstructGLM(Instruction-finetuned Graph Language Model) with highly scalable prompts basedon natural language instructions. We use natural language to describemulti-scale geometric structure of the graph and then instruction finetune anLLM to perform graph tasks, which enables Generative Graph Learning. Our methodsurpasses all GNN baselines on ogbn-arxiv, Cora and PubMed datasets,underscoring its effectiveness and sheds light on generative LLMs as newfoundation model for graph machine learning. Our code is open-sourced athttps://github.com/agiresearch/InstructGLM.