Abstract
Long contexts of recent LLMs have enabled a new use case: asking models tofind security vulnerabilities in entire codebases. To evaluate modelperformance on this task, we introduce eyeballvul: a benchmark designed to testthe vulnerability detection capabilities of language models at scale, that issourced and updated weekly from the stream of published vulnerabilities inopen-source repositories. The benchmark consists of a list of revisions indifferent repositories, each associated with the list of known vulnerabilitiespresent at that revision. An LLM-based scorer is used to compare the list ofpossible vulnerabilities returned by a model to the list of knownvulnerabilities for each revision. As of July 2024, eyeballvul contains 24,000+vulnerabilities across 6,000+ revisions and 5,000+ repositories, and is around55GB in size.