Existing graph layout algorithms are usually not able to optimize all theaesthetic properties desired in a graph layout. To evaluate how well thedesired visual features are reflected in a graph layout, many readabilitymetrics have been proposed in the past decades. However, the calculation ofthese readability metrics often requires access to the node and edgecoordinates and is usually computationally inefficient, especially for densegraphs. Importantly, when the node and edge coordinates are not accessible, itbecomes impossible to evaluate the graph layouts quantitatively. In this paper,we present a novel deep learning-based approach to evaluate the readability ofgraph layouts by directly using graph images. A convolutional neural networkarchitecture is proposed and trained on a benchmark dataset of graph images,which is composed of synthetically-generated graphs and graphs created bysampling from real large networks. Multiple representative readability metrics(including edge crossing, node spread, and group overlap) are considered in theproposed approach. We quantitatively compare our approach to traditionalmethods and qualitatively evaluate our approach using a case study andvisualizing convolutional layers. This work is a first step towards using deeplearning based methods to evaluate images from the visualization fieldquantitatively.