Abstract
Recent developments in neural language models (LMs) have raised concernsabout their potential misuse for automatically spreading misinformation. Inlight of these concerns, several studies have proposed to detectmachine-generated fake news by capturing their stylistic differences fromhuman-written text. These approaches, broadly termed stylometry, have foundsuccess in source attribution and misinformation detection in human-writtentexts. However, in this work, we show that stylometry is limited againstmachine-generated misinformation. While humans speak differently when trying todeceive, LMs generate stylistically consistent text, regardless of underlyingmotive. Thus, though stylometry can successfully prevent impersonation byidentifying text provenance, it fails to distinguish legitimate LM applicationsfrom those that introduce false information. We create two benchmarksdemonstrating the stylistic similarity between malicious and legitimate uses ofLMs, employed in auto-completion and editing-assistance settings. Our findingshighlight the need for non-stylometry approaches in detecting machine-generatedmisinformation, and open up the discussion on the desired evaluationbenchmarks.