Abstract

In this work, we identify the "2D-Cheating" problem in 3D LLM evaluation,where these tasks might be easily solved by VLMs with rendered images of pointclouds, exposing ineffective evaluation of 3D LLMs' unique 3D capabilities. Wetest VLM performance across multiple 3D LLM benchmarks and, using this as areference, propose principles for better assessing genuine 3D understanding. Wealso advocate explicitly separating 3D abilities from 1D or 2D aspects whenevaluating 3D LLMs. Code and data are available athttps://github.com/LLM-class-group/Revisiting-3D-LLM-Benchmarks

 

Quick Read (beta)