Graphs are general and powerful data representations which can model complexreal-world phenomena, ranging from chemical compounds to social networks;however, effective feature extraction from graphs is not a trivial task, andmuch work has been done in the field of machine learning and data mining. Therecent advances in graph neural networks have made automatic and flexiblefeature extraction from graphs possible and have improved the predictiveperformance significantly. In this paper, we go further with this line ofresearch and address a more general problem of learning with a graph of graphs(GoG) consisting of an external graph and internal graphs, where each node inthe external graph has an internal graph structure. We propose a dualconvolutional neural network that extracts node representations by combiningthe external and internal graph structures in an end-to-end manner. Experimentson link prediction tasks using several chemical network datasets demonstratethe effectiveness of the proposed method.