Abstract
Web crawl is a main source of large language models' (LLMs) pretraining data,but the majority of crawled web pages are discarded in pretraining due to lowdata quality. This paper presents Crawl4LLM, an efficient web crawling methodthat explores the web graph based on the preference of LLM pretraining.Specifically, it leverages the influence of a webpage in LLM pretraining as thepriority score of the web crawler's scheduler, replacing the standard graphconnectivity based priority. Our experiments on a web graph containing 900million webpages from a commercial search engine's index demonstrate theefficiency of Crawl4LLM in obtaining high-quality pretraining data. With just21% URLs crawled, LLMs pretrained on Crawl4LLM data reach the same downstreamperformances of previous crawls, significantly reducing the crawling waste andalleviating the burdens on websites. Our code is publicly available athttps://github.com/cxcscmu/Crawl4LLM.