Abstract
Dirbusting is a technique used to brute force directories and file names onweb servers while monitoring HTTP responses, in order to enumerate servercontents. Such a technique uses lists of common words to discover the hiddenstructure of the target website. Dirbusting typically relies on response codesas discovery conditions to find new pages. It is widely used in web applicationpenetration testing, an activity that allows companies to detect websitesvulnerabilities. Dirbusting techniques are both time and resource consuming andinnovative approaches have never been explored in this field. We hence proposean advanced technique to optimize the dirbusting process by leveragingArtificial Intelligence. More specifically, we use semantic clusteringtechniques in order to organize wordlist items in different groups according totheir semantic meaning. The created clusters are used in an ad-hoc implementednext-word intelligent strategy. This paper demonstrates that the usage ofclustering techniques outperforms the commonly used brute force methods.Performance is evaluated by testing eight different web applications. Resultsshow a performance increase that is up to 50% for each of the conductedexperiments.