Graph Structure of Neural Networks

  • 2020-07-13 17:59:31
  • Jiaxuan You, Jure Leskovec, Kaiming He, Saining Xie
  • 56

Abstract

Neural networks are often represented as graphs of connections betweenneurons. However, despite their wide use, there is currently littleunderstanding of the relationship between the graph structure of the neuralnetwork and its predictive performance. Here we systematically investigate howdoes the graph structure of neural networks affect their predictiveperformance. To this end, we develop a novel graph-based representation ofneural networks called relational graph, where layers of neural networkcomputation correspond to rounds of message exchange along the graph structure.Using this representation we show that: (1) a "sweet spot" of relational graphsleads to neural networks with significantly improved predictive performance;(2) neural network's performance is approximately a smooth function of theclustering coefficient and average path length of its relational graph; (3) ourfindings are consistent across many different tasks and datasets; (4) the sweetspot can be identified efficiently; (5) top-performing neural networks havegraph structure surprisingly similar to those of real biological neuralnetworks. Our work opens new directions for the design of neural architecturesand the understanding on neural networks in general.

 

Quick Read (beta)

loading the full paper ...