Abstract
We introduce an open-ended test grounded in algorithmic probability that canavoid benchmark contamination in the quantitative evaluation of frontier modelsin the context of their Artificial General Intelligence (AGI) andSuperintelligence (ASI) claims. Unlike other tests, this test does not rely onstatistical compression methods (such as GZIP or LZW), which are more closelyrelated to Shannon entropy than to Kolmogorov complexity and are not able totest beyond simple pattern matching. The test challenges aspects of AI, inparticular LLMs, related to features of intelligence of fundamental nature suchas synthesis and model creation in the context of inverse problems (generatingnew knowledge from observation). We argue that metrics based on modelabstraction and abduction (optimal Bayesian `inference') for predictive`planning' can provide a robust framework for testing intelligence, includingnatural intelligence (human and animal), narrow AI, AGI, and ASI. We found thatLLM model versions tend to be fragile and incremental as a result ofmemorisation only with progress likely driven by the size of training data. Theresults were compared with a hybrid neurosymbolic approach that theoreticallyguarantees universal intelligence based on the principles of algorithmicprobability and Kolmogorov complexity. The method outperforms LLMs in aproof-of-concept on short binary sequences. We prove that compression isequivalent and directly proportional to a system's predictive power and viceversa. That is, if a system can better predict it can better compress, and ifit can better compress, then it can better predict. Our findings strengthen thesuspicion regarding the fundamental limitations of LLMs, exposing them assystems optimised for the perception of mastery over human language.