Abstract
Despite recent community revelations about the advancements and potentialapplications of Large Language Models (LLMs) in understanding Text-AttributedGraph (TAG), the deployment of LLMs for production is hindered by its highcomputational and storage requirements, as well as long latencies during modelinference. Simultaneously, although traditional Graph Neural Networks (GNNs)are light weight and adept at learning structural features of graphs, theirability to grasp the complex semantics in TAG is somewhat constrained for realapplications. To address these limitations, we concentrate on the downstreamtask of node classification in TAG and propose a novel graph knowledgedistillation framework, termed Linguistic Graph Knowledge Distillation(LinguGKD), using LLMs as teacher models and GNNs as student models forknowledge distillation. It involves TAG-oriented instruction tuning of LLM ondesigned tailored prompts, followed by propagating knowledge and aligning thehierarchically learned node features from the teacher LLM to the student GNN inlatent space, employing a layer-adaptive contrastive learning strategy. Throughextensive experiments on a variety of LLM and GNN models and multiple benchmarkdatasets, the proposed LinguGKD significantly boosts the student GNN'spredictive accuracy and convergence rate, without the need of extra data ormodel parameters. Compared to teacher LLM, distilled GNN achieves superiorinference speed equipped with much fewer computing and storage demands, whensurpassing the teacher LLM's classification accuracy on some of benchmarkdatasets.