Abstract
Large language models (LLMs) acquire extensive knowledge during pre-training,known as their parametric knowledge. However, in order to remain up-to-date andalign with human instructions, LLMs inevitably require external knowledgeduring their interactions with users. This raises a crucial question: How willLLMs respond when external knowledge interferes with their parametricknowledge? To investigate this question, we propose a framework thatsystematically elicits LLM parametric knowledge and introduces externalknowledge. Specifically, we uncover the impacts by constructing a parametricknowledge graph to reveal the different knowledge structures of LLMs, andintroduce external knowledge through distractors of varying degrees, methods,positions, and formats. Our experiments on both black-box and open-sourcemodels demonstrate that LLMs tend to produce responses that deviate from theirparametric knowledge, particularly when they encounter direct conflicts orconfounding changes of information within detailed contexts. We also find thatwhile LLMs are sensitive to the veracity of external knowledge, they can stillbe distracted by unrelated information. These findings highlight the risk ofhallucination when integrating external knowledge, even indirectly, duringinteractions with current LLMs. All the data and results are publiclyavailable.