Abstract
We propose the first deep learning approach for exemplar-based localcolorization. Given a reference color image, our convolutional neural networkdirectly maps a grayscale image to an output colorized image. Rather than usinghand-crafted rules as in traditional exemplar-based methods, our end-to-endcolorization network learns how to select, propagate, and predict colors fromthe large-scale data. The approach performs robustly and generalizes well evenwhen using reference images that are unrelated to the input grayscale image.More importantly, as opposed to other learning-based colorization methods, ournetwork allows the user to achieve customizable results by simply feedingdifferent references. In order to further reduce manual effort in selecting thereferences, the system automatically recommends references with our proposedimage retrieval algorithm, which considers both semantic and luminanceinformation. The colorization can be performed fully automatically by simplypicking the top reference suggestion. Our approach is validated through a userstudy and favorable quantitative comparisons to the-state-of-the-art methods.Furthermore, our approach can be naturally extended to video colorization. Ourcode and models will be freely available for public use.