Replacing Mobile Camera ISP with a Single Deep Learning Model

  • 2020-02-13 14:22:39
  • Andrey Ignatov, Luc Van Gool, Radu Timofte
  • 25

Abstract

As the popularity of mobile photography is growing constantly, lots ofefforts are being invested now into building complex hand-crafted camera ISPsolutions. In this work, we demonstrate that even the most sophisticated ISPpipelines can be replaced with a single end-to-end deep learning model trainedwithout any prior knowledge about the sensor and optics used in a particulardevice. For this, we present PyNET, a novel pyramidal CNN architecture designedfor fine-grained image restoration that implicitly learns to perform all ISPsteps such as image demosaicing, denoising, white balancing, color and contrastcorrection, demoireing, etc. The model is trained to convert RAW Bayer dataobtained directly from mobile camera sensor into photos captured with aprofessional high-end DSLR camera, making the solution independent of anyparticular mobile ISP implementation. To validate the proposed approach on thereal data, we collected a large-scale dataset consisting of 10 thousandfull-resolution RAW-RGB image pairs captured in the wild with the Huawei P20cameraphone (12.3 MP Sony Exmor IMX380 sensor) and Canon 5D Mark IV DSLR. Theexperiments demonstrate that the proposed solution can easily get to the levelof the embedded P20's ISP pipeline that, unlike our approach, is combining thedata from two (RGB + B/W) camera sensors. The dataset, pre-trained models andcodes used in this paper are available on the project website.

 

Quick Read (beta)

loading the full paper ...