Abstract
Blind deconvolution is a classical yet challenging lowlevel vision problemwith many realworld applications. Traditional maximum a posterior (MAP) basedmethods rely heavily on fixed and handcrafted priors that certainly areinsufficient in characterizing clean images and blur kernels, and usually adoptspecially designed alternating minimization to avoid trivial solution. Incontrast, existing deep motion deblurring networks learn from massive trainingimages the mapping to clean image or blur kernel, but are limited in handlingvarious complex and large size blur kernels. To connect MAP and deep models, wein this paper present two generative networks for respectively modeling thedeep priors of clean image and blur kernel, and propose an unconstrained neuraloptimization solution to blind deconvolution. In particular, we adopt anasymmetric Autoencoder with skip connections for generating latent clean image,and a fullyconnected network (FCN) for generating blur kernel. Moreover, theSoftMax nonlinearity is applied to the output layer of FCN to meet thenonnegative and equality constraints. The process of neural optimization canbe explained as a kind of "zeroshot" selfsupervised learning of thegenerative networks, and thus our proposed method is dubbed SelfDeblur.Experimental results show that our SelfDeblur can achieve notable quantitativegains as well as more visually plausible deblurring results in comparison tostateoftheart blind deconvolution methods on benchmark datasets andrealworld blurry images. The source code is available athttps://github.com/csdwren/SelfDeblur.
Quick Read (beta)
Neural Blind Deconvolution Using Deep Priors
Abstract
Blind deconvolution is a classical yet challenging lowlevel vision problem with many realworld applications. Traditional maximum a posterior (MAP) based methods rely heavily on fixed and handcrafted priors that certainly are insufficient in characterizing clean images and blur kernels, and usually adopt specially designed alternating minimization to avoid trivial solution. In contrast, existing deep motion deblurring networks learn from massive training images the mapping to clean image or blur kernel, but are limited in handling various complex and large size blur kernels. To connect MAP and deep models, we in this paper present two generative networks for respectively modeling the deep priors of clean image and blur kernel, and propose an unconstrained neural optimization solution to blind deconvolution. In particular, we adopt an asymmetric Autoencoder with skip connections for generating latent clean image, and a fullyconnected network (FCN) for generating blur kernel. Moreover, the SoftMax nonlinearity is applied to the output layer of FCN to meet the nonnegative and equality constraints. The process of neural optimization can be explained as a kind of ”zeroshot” selfsupervised learning of the generative networks, and thus our proposed method is dubbed SelfDeblur. Experimental results show that our SelfDeblur can achieve notable quantitative gains as well as more visually plausible deblurring results in comparison to stateoftheart blind deconvolution methods on benchmark datasets and realworld blurry images. The source code is available at https://github.com/csdwren/SelfDeblur.
1 Introduction
Blurry image  Xu & Jia [48] 
PanL0[28]  Sun et al. [41] 
PanDCP[30]  SelfDeblur 
Camera shake during exposure inevitably yields blurry images and is a longstanding annoying issue in digital photography. The removal of distortion from a blurry image, i.e., image deblurring, is a classical illposed problem in lowlevel vision and has received considerable research attention [2, 4, 20, 29, 56, 33, 30, 3, 11]. When the blur kernel is spatially invariant, it is also known as blind deconvolution, where the blurry image $\mathbf{y}$ can be formulated as,
$$\mathbf{y}=\mathbf{k}\otimes \mathbf{x}+\mathbf{n},$$  (1) 
where $\otimes $ denotes the 2D convolution operator, $\mathbf{x}$ is the latent clean image, $\mathbf{k}$ is the blur kernel, and $\mathbf{n}$ is the additive white Gaussian noise (AWGN) with noise level $\sigma $. It can be seen that blind deconvolution should estimate both $\mathbf{k}$ and $\mathbf{x}$ from a blurry image $\mathbf{y}$, making it remain a very challenging problem after decades of studies.
Most traditional blind deconvolution methods are based on the Maximum a Posterior (MAP) framework,
$(\mathbf{k},\mathbf{x})$  $=\mathrm{arg}\underset{\mathbf{x},\mathbf{k}}{max}\mathrm{Pr}(\mathbf{k},\mathbf{x}\mathbf{y}),$  (2)  
$=\mathrm{arg}\underset{\mathbf{x},\mathbf{k}}{max}\mathrm{Pr}\left(\mathbf{y}\mathbf{k},\mathbf{x}\right)\mathrm{Pr}\left(\mathbf{x}\right)\mathrm{Pr}\left(\mathbf{k}\right),$ 
where $\mathrm{Pr}\left(\mathbf{y}\mathbf{k},\mathbf{x}\right)$ is the likelihood corresponding to the fidelity term, and $\mathrm{Pr}\left(\mathbf{x}\right)$ and $\mathrm{Pr}\left(\mathbf{k}\right)$ model the priors of clean image and blur kernel, respectively. Although many priors have been suggested for $\mathbf{x}$ [2, 28, 56, 16] and $\mathbf{k}$ [56, 30, 20, 23, 41, 25, 34, 30, 50], they generally are handcrafted and certainly are insufficient in characterizing clean images and blur kernels. Furthermore, the nonconvexity of MAP based models also increases the difficulty of optimization. Levin et al. [20] reveal that MAPbased methods may converge to trivial solution of delta kernel. Perrone and Favaro [33] show that the success of existing methods can be attributed to some optimization details, e.g., projected alternating minimization and delayed normalization of $\mathbf{k}$.
Motivated by the unprecedented success of deep learning in lowlevel vision [12, 52, 13, 42, 24], some attempts have also been made to solve blind deconvolution using deep convolutional networks (CNNs). Given the training set, deep CNNs can either be used to extract features to facilitate blur kernel estimation [38, 1], or be deployed to learn the direct mapping to clean image for motion deblurring [51, 26, 43, 7]. However, these methods do not succeed in handling various complex and large size blur kernels in blind deconvolution. Recently, Ulyanov et al. [45] suggest the deep image prior (DIP) framework, which adopts the structure of a DIP generator network to capture lowlevel image statistics and shows powerful ability in image denoising, superresolution, inpainting, etc. Subsequently, Gandelsman et al. [6] combine multiple DIPs (i.e., DoubleDIP) for multitask layer decomposition such as image dehazing and transparency separation. However, DoubleDIP cannot be directly applied to solve blind deconvolution due to that the DIP network is designed to generate natural images and is limited to capture the prior of blur kernels.
In this paper, we propose a novel neural optimization solution to blind deconvolution. Motivated by the DIP network [45], an image generator network ${\mathcal{G}}_{x}$, i.e., an asymmetric Autoencoder with skip connections, is deployed to capture the statistics of latent clean image. Nonetheless, image generator network cannot well characterize the prior on blur kernel. Instead, we adopt a fullyconnected network (FCN) ${\mathcal{G}}_{k}$ to model the prior of blur kernel. Furthermore, the Softmax nonlinearity is deployed to the output layer of ${\mathcal{G}}_{k}$, and the nonnegative and equality constraints on blur kernel can then be naturally satisfied. By fixing the network structures (${\mathcal{G}}_{k}$ and ${\mathcal{G}}_{x}$) and inputs (${\mathbf{z}}_{k}$ and ${\mathbf{z}}_{x}$) sampled from uniform distribution, blind deconvolution is thus formulated as an unconstrained neural optimization on network parameters of ${\mathcal{G}}_{k}$ and ${\mathcal{G}}_{x}$. As illustrated in Fig. 2, given a blurry image $\mathbf{y}$, the optimization process can also be explained as a kind of ”zeroshot” selfsupervised learning [39] of ${\mathcal{G}}_{k}$ and ${\mathcal{G}}_{x}$, and our proposed method is dubbed SelfDeblur. Even though SelfDeblur can be optimized with either alternating optimization or joint optimization, our empirical study shows that the latter performs better in most cases.
Experiments are conducted on two widely used benchmarks [20, 19] as well as realworld blurry images to evaluate our SelfDeblur. Fig. 1 shows the deblurring results on a severe realworld blurry image. While the competing methods either fail to estimate large size blur kernels or suffer from ringing effects, our SelfDeblur succeed in estimating the blur kernel and generating visually favorable deblurring image. In comparison to the stateoftheart methods, our SelfDeblur can achieve notable quantitative performance gains and performs favorably in generating visually plausible deblurring results. It is worth noting that our SelfDeblur can both estimate blur kernel and generate latent clean image with satisfying visual quality, making the subsequent nonblind deconvolution not a compulsory choice.
Our contributions are summarized as follows:

•
A neural blind deconvolution method, i.e., SelfDeblur, is proposed, where DIP and FCN are respectively introduced to capture the priors of clean image and blur kernel. And the SoftMax nonlinearity is applied to the output layer of FCN to meet the nonnegative and equality constraints.

•
The joint optimization algorithm is suggested to solve the unconstrained neural blind deconvolution model for both estimating blur kernel and generating latent clean image, making the nonblind deconvolution not a compulsory choice for our SelfDeblur.

•
Extensive experiments show that our SelfDeblur performs favorably against the existing MAPbased methods in terms of quantitative and qualitative evaluation. To our best knowledge, SelfDeblur makes the first attempt of applying deep learning to yield stateoftheart blind deconvolution performance.
2 Related Work
In this section, we briefly survey the relevant works including optimizationbased blind deconvolution and deep learning based blind deblurring methods.
2.1 Optimizationbased Blind Deconvolution
Traditional optimizationbased blind deconvolution methods can be further categorized into two groups, i.e., Variational Bayes (VB)based and MAPbased methods. VBbased method [21] is theoretically promising, but is with heavy computational cost. As for the MAPbased methods, many priors have been suggested for modeling clean image and blur kernels. In the seminal work of [2], Chan et al. introduce the total variation (TV) regularization to model latent clean image in blind deconvolution, and motivates several variants based on gradientbased priors, e.g., ${\mathrm{\ell}}_{0}$norm [28] and ${\mathrm{\ell}}_{p}$norm [56]. Other specifically designed regularizations, e.g., ${\mathrm{\ell}}_{1}/{\mathrm{\ell}}_{2}$norm [16], patchbased prior [41, 25], lowrank prior [34] and dark channel prior [30, 50] have also been proposed to identify and preserve salient edges for benefiting blur kernel estimation. Recently, a discriminative prior [22] is presented to distinguish the clean image from a blurry one, but still heavily relies on ${\mathrm{\ell}}_{0}$norm regularizer for attaining stateoftheart performance. As for $\mathrm{Pr}(\mathbf{k})$, gradient sparsity priors [56, 30, 20] and spectral prior [23] are usually adopted. In order to solve the MAPbased model, several tricks have been introduced to the projected alternating minimization algorithm, including delayed normalization of blur kernel [33], multiscale implementation [16] and timevarying parameters [56].
After blur kernel estimation, nonblind deconvolution is required to recover the latent clean image with fine texture details [16, 28, 30, 41, 49, 23]. Thus, the priors should favor natural images, e.g., hyperLaplacian [15], GMM [55], nonlocal similarity [5], e.g., RTF [37], CSF [36] and CNN [17, 53], which are quite different from those used in blur kernel estimation. Our SelfDeblur can be regarded as a special MAPbased method, but two generative networks, i.e., DIP and FCN, are adopted to respectively capture the deep priors of clean image and blur kernel. Moreover, the joint optimization algorithm is effective to estimate blur kernel and generate clean image, making nonblind deconvolution not a compulsory choice for SelfDeblur.
2.2 Deep Learning in Image Deblurring
Many studies have been given to apply deep learning (DL) to blind deblurring. For example, DL can be used to help the learning of mapping to blur kernel. By imitating the alternating minimization steps in optimizationbased methods, Schuler et al. [38] design the deep network architectures for blur kernel estimation. By studying the spectral property of blurry images, deep CNN is suggested to predict the Fourier coefficients [1], which can then be projected to estimate blur kernel. In [40], CNN is used to predict the parametric blur kernels for motion blurry images.
For dynamic scene deblurring, deep CNNs have been developed to learn the direct mapping to latent clean image [51, 26, 43, 40, 18]. Motivated by the multiscale strategy in blind deconvolution, multiscale CNN [26] and scalerecurrent network [43] are proposed to directly estimate the latent clean image from the blurry image. The adversarial loss is also introduced for better recovery of texture details in motion deblurring [18]. Besides, by exploiting the temporal information between adjacent frames, deep networks have also been applied to video motion deblurring [27, 10, 31]. However, due to the severe illposedness caused by large size and complex blur kernels, existing DLbased methods still cannot outperform traditional optimizationbased ones for blind deconvolution.
Recently, DIP [45] and DoubleDIP [6] have been introduced to capture image statistics, and have been deployed to many lowlevel vision tasks such as superresolution, inpainting, dehazing, transparency separation, etc. Nonetheless, the DIP network is limited in capturing the prior of blur kernels, and DoubleDIP still performs poorly for blind deconvolution. To the best of our knowledge, our SelfDeblur makes the first attempt of applying deep networks to yield stateoftheart blind deconvolution performance.
3 Proposed Method
In this section, we first introduce the general formulation of MAPbased blind deconvolution, and then present our proposed neural blind deconvolution model as well as the joint optimization algorithm.
3.1 MAPbased Blind Deconvolution Formulation
According to Eqn. (1), we define the fidelity term as $\mathrm{log}\left(\mathrm{Pr}\left(\mathbf{y}\mathbf{k},\mathbf{x}\right)\right)={\parallel \mathbf{k}\otimes \mathbf{x}\mathbf{y}\parallel}^{2}$. And we further introduce two regularization terms $\mathrm{log}(\mathrm{Pr}(\mathbf{x}))=\varphi (\mathbf{x})$ and $\mathrm{log}(\mathrm{Pr}(\mathbf{k}))=\phi (\mathbf{k})$ for modeling the priors on latent clean image and blur kernel, respectively. The MAPbased blind deconvolution model in Eqn. (2) can then be reformulated as,
$($  $\mathbf{x},\mathbf{k})=\mathrm{arg}\underset{(\mathbf{x},\mathbf{k})}{\mathrm{min}}\parallel \mathbf{k}\otimes \mathbf{x}\mathbf{y}\parallel {}^{2}+\lambda \varphi (\mathbf{x})+\tau \phi (\mathbf{k})$  (3)  
$s.t\mathrm{.\hspace{0.25em}0}\le {x}_{i}\le 1,\forall i,{k}_{j}\ge 0,{\displaystyle {\sum}_{j}}{k}_{j}=1,\forall j,$ 
where $\lambda $ and $\tau $ are tradeoff regularization parameters. Besides the two regularization terms, we further introduce the nonnegative and equality constraints for blur kernel $\mathbf{k}$ [41, 56, 30, 33], and the pixels in $\mathbf{x}$ are also constrained to the range $[0,1]$.
Under the MAPbased framework, many fixed and handcrafted regularization terms have been presented for latent clean image and blur kernel [56, 30, 23, 41, 25, 34, 30, 50]. To solve the model in Eqn. (3), projected alternating minimization is generally adopted, but several optimization details, e.g., delayed normalization [33] and multiscale implementation [16], are also crucial to the success of blind deconvolution. Moreover, once the estimated blur kernel $\widehat{\mathbf{k}}$ is obtained by solving Eqn. (3), another nonblind deconvolution usually is required to generate final deblurring result,
$$\mathbf{x}=\mathrm{arg}\underset{\mathbf{x}}{\mathrm{min}}{\parallel \widehat{\mathbf{k}}\otimes \mathbf{x}\mathbf{y}\parallel}^{2}+\lambda \mathcal{R}(\mathbf{x}),$$  (4) 
where $\mathcal{R}(\mathbf{x})$ is a regularizer to capture natural image statistics and is quite different from $\varphi (\mathbf{x})$.
3.2 Neural Blind Deconvolution
Motivated by the success of DIP [45] and DoubleDIP [6], we suggest the neural blind deconvolution model by adopting generative networks ${\mathcal{G}}_{x}$ and ${\mathcal{G}}_{k}$ to capture the priors of $\mathbf{x}$ and $\mathbf{k}$. By substituting $\mathbf{x}$ and $\mathbf{k}$ with ${\mathcal{G}}_{x}$ and ${\mathcal{G}}_{k}$ and removing the regularization terms $\varphi (\mathbf{x})$ and $\phi (\mathbf{k})$, the neural blind deconvolution can be formulated as,
$\underset{({\mathcal{G}}_{x},{\mathcal{G}}_{k})}{\mathrm{min}}{\parallel {\mathcal{G}}_{k}({\mathbf{z}}_{k})\otimes {\mathcal{G}}_{x}({\mathbf{z}}_{x})\mathbf{y}\parallel}^{2}$  (5)  
$s.t\mathrm{.\hspace{0.25em}\hspace{0.25em}0}\le {({\mathcal{G}}_{x}({\mathbf{z}}_{x}))}_{i}\le 1,\forall i,$  
${({\mathcal{G}}_{k}({\mathbf{z}}_{k}))}_{j}\ge 0,{\displaystyle {\sum}_{j}}{({\mathcal{G}}_{k}({\mathbf{z}}_{k}))}_{j}=1,\forall j,$ 
where ${\mathbf{z}}_{x}$ and ${\mathbf{z}}_{k}$ are sampled from the uniform distribution, ${(\cdot )}_{i}$ and ${(\cdot )}_{j}$ denote the $i$th and $j$th elements.
However, there remain several issues to be addressed with neural blind deconvolution. (i) The DIP network [45] is designed to capture low level image statistics and is limited in capturing the prior of blur kernels. As a result, we empirically find that DoubleDIP [6] performs poorly for blind deconvolution (see the results in Sec. 4.1.2). (ii) Due to the nonnegative and equality constraints, the resulting model in Eqn. (5) is a constrained neural optimization problem and is difficult to optimize. (iii) Although the generative networks ${\mathcal{G}}_{x}$ and ${\mathcal{G}}_{k}$ present high impedance to image noise, the denoising performance of DIP heavily relies on the additional averaging over last iterations and different optimization runs [45]. Such heuristic solutions, however, both bring more computational cost and cannot be directly borrowed to handle blurry and noisy images.
In the following, we present our solution to address the issues (i)&(ii) by designing proper generative networks ${\mathcal{G}}_{x}$ and ${\mathcal{G}}_{k}$. As for the issue (iii), we introduce an extra TV regularizer and a regularization parameter to explicitly consider noise level in the neural blind deconvolution model.
Generative Network ${\mathrm{G}}_{\mathrm{x}}$. The latent clean images usually contain salient structures and rich textures, which requires the generative network ${\mathcal{G}}_{x}$ to have sufficient modeling capacity. Fortunately, since the introduction of generative adversarial network [8], dramatic progress has been made in generating high quality natural images [45]. For modeling $\mathbf{x}$, we adopt a DIP network, i.e., the asymmetric Autoencoder [35] with skip connections in [45], to serve as ${\mathcal{G}}_{x}$. As shown in Fig. 2, the first 5 layers of encoder are skip connected to the last 5 layers of decoder. Finally, a convolutional output layer is used to generate latent clean image. To meet the range constraint for $\mathbf{x}$, the Sigmoid nonlinearity is applied to the output layer. Please refer to the supplementary material for more architecture details of ${\mathcal{G}}_{x}$.
Generative Network ${\mathrm{G}}_{\mathrm{k}}$. On the one hand, the DIP network [45] is designed to capture the statistics of natural image but performs limited in modeling the prior of blur kernel. On the other hand, blur kernel $\mathbf{k}$ generally contains much fewer information than latent clean image $\mathbf{x}$, and can be well generated by simpler generative network. Thus, we simply adopt a fullyconnected network (FCN) to serve as ${\mathcal{G}}_{k}$. As shown in Fig. 2, the FCN ${\mathcal{G}}_{k}$ takes a 1D noise ${\mathbf{z}}_{k}$ with 200 dimensions as input, and has a hidden layer of 1,000 nodes and an output layer of ${K}^{2}$ nodes. To guarantee the nonnegative and equalitly constraints can be always satisfied, the SoftMax nonlinearity is applied to the output layer of ${\mathcal{G}}_{k}$. Finally, the 1D output of ${K}^{2}$ entries is reshaped to a 2D $K\times K$ blur kernel. Please refer to the supplementary material for more architecture details of ${\mathcal{G}}_{k}$.
Unconstrained Neural Blind Deconvolution with TV Regularization. With the above generative networks ${\mathcal{G}}_{x}$ and ${\mathcal{G}}_{k}$, we can formulate neural blind deconvolution into an unconstrained optimization form. However, the resulting model is irrelevant with the noise level, making it perform poorly on blurry images with nonnegligible noise. To address this issue, we combine both ${\mathcal{G}}_{x}$ and TV regularization to capture image priors, and our neural blind deconvolution model can then be written as,
$$\underset{{\mathcal{G}}_{k},{\mathcal{G}}_{x}}{\mathrm{min}}{\parallel {\mathcal{G}}_{k}({\mathbf{z}}_{k})\otimes {\mathcal{G}}_{x}({\mathbf{z}}_{x})\mathbf{y}\parallel}^{2}+\lambda \text{TV}({\mathcal{G}}_{x}({\mathbf{z}}_{x})).$$  (6) 
where $\lambda $ denotes the regularization parameter controlled by noise level $\sigma $. Albeit the generative network ${\mathcal{G}}_{x}$ is more powerful, the incorporation of ${\mathcal{G}}_{x}$ and another image prior generally is beneficial to deconvolution performance. Moreover, the introduction of the noise level related regularization parameter $\lambda $ can greatly improve the robustness in handling blurry images with various noise levels. In particular, we emperically set $\lambda =0.1\times \sigma $ in our implementation, and the noise level $\sigma $ can be estimated using the method introduced in [54].
3.3 Optimization Algorithm
The optimization process of Eqn. (6) can be explained as a kind of ”zeroshot” selfsupervised learning [39], where the generative networks ${\mathcal{G}}_{k}$ and ${\mathcal{G}}_{x}$ are trained using only a test image (i.e., blurry image $\mathbf{y}$) and no groundtruth clean image is available. Thus, our method is dubbed SelfDeblur. In the following, we present two algorithms for SelfDeblur, i.e., alternating optimization and joint optimization.
Alternating Optimization. Analogous to the alternating minimization steps in traditional blind deconvolution [56, 2, 30, 28, 41], the network parameters of ${\mathcal{G}}_{k}$ and ${\mathcal{G}}_{x}$ can also be optimized in an alternating manner. As summarized in Algorithm 3.3, the parameters of ${\mathcal{G}}_{k}$ are updated via the ADAM [14] by fixing ${\mathcal{G}}_{x}$, and vice versa. In particular, the gradient w.r.t. either ${\mathcal{G}}_{x}$ or ${\mathcal{G}}_{k}$ can be derived using automatic differentiation [32].
Joint Optimization. In traditional MAPbased framework, alternating minimization allows the use of projection operator to handle nonnegative and equality constraints and the modification of optimization details to avoid trivial solution, and thus has been widely adopted. As for our neural blind deconvolution, the model in Eqn. (6) is unconstrained optimization, and the powerful modeling capacity of ${\mathcal{G}}_{k}$ and ${\mathcal{G}}_{x}$ is beneficial to avoid trivial delta kernel solution. We also note that the unconstrained neural blind deconvolution is highly nonconvex, and alternating optimization may get stuck at saddle points [44]. Thus, joint optimization is more prefered than alternating optimization for SelfDeblur. Using the automatic differentiation techniques [32], the gradients w.r.t. ${\mathcal{G}}_{k}$ and ${\mathcal{G}}_{x}$ can be derived. Algorithm 3.3 summarizes the joint optimization algorithm, where the parameters of ${\mathcal{G}}_{k}$ and ${\mathcal{G}}_{x}$ can be jointly updated using the ADAM algorithm. Our empirical study in Sec. 4.1.1 also shows that joint optimization usually converges to better solutions than alternating optimization.
Both alternating optimization and joint optimization algorithms are stopped when reaching $T$ iterations. Then, the estimated blur kernel and latent clean image can be generated using $\mathbf{k}={\mathcal{G}}_{k}^{T}({\mathbf{z}}_{k})$ and $\mathbf{x}={\mathcal{G}}_{x}^{T}({\mathbf{z}}_{x})$, respectively. Benefited from the modeling capacity of ${\mathcal{G}}_{x}({\mathbf{z}}_{x})$, the estimated $\mathbf{x}$ is with visually favorable textures, and it is not a compulsory choice for our SelfDeblur to adopt another nonblind deconvolution method to generate final deblurring result.
[!t] \[email protected]@algorithmic[1] \RequireBlurry image $\mathbf{y}$ \EnsureBlur kernel $\mathbf{k}$ and clean image $\mathbf{x}$ \StateSample ${\mathbf{z}}_{x}$ and ${\mathbf{z}}_{k}$ from uniform distribution with seed 0. \State$\mathbf{k}={\mathcal{G}}_{k}^{0}({\mathbf{z}}_{k})$ \For$t=1\text{to}T$
$\mathbf{x}={\mathcal{G}}_{x}^{t1}\left({\mathbf{z}}_{x}\right)$ \StateCompute the gradient w.r.t. ${\mathcal{G}}_{k}$ \StateUpdate ${\mathcal{G}}_{k}^{t}$ using the ADAM algorithm [14] \State$\mathbf{k}={\mathcal{G}}_{k}^{t}({\mathbf{z}}_{k})$ \StateCompute the gradient w.r.t. ${\mathcal{G}}_{x}$ \StateUpdate ${\mathcal{G}}_{x}^{t}$ using the ADAM algorithm [14]
$\mathbf{x}={\mathcal{G}}_{x}^{T}\left({\mathbf{z}}_{x}\right)$, $\mathbf{k}={\mathcal{G}}_{x}^{T}({\mathbf{z}}_{k})$
[!ht] \[email protected]@algorithmic[1] \RequireBlurry image $\mathbf{y}$ \EnsureBlur kernel $\mathbf{k}$ and clean image $\mathbf{x}$ \StateSample ${\mathbf{z}}_{x}$ and ${\mathbf{z}}_{k}$ from uniform distribution with seed 0. \For$t=1\text{to}T$
$\mathbf{k}={\mathcal{G}}_{k}^{t1}\left({\mathbf{z}}_{k}\right)$ \State$\mathbf{x}={\mathcal{G}}_{x}^{t1}({\mathbf{z}}_{x})$ \StateCompute the gradients w.r.t. ${\mathcal{G}}_{k}$ and ${\mathcal{G}}_{x}$ \StateUpdate ${\mathcal{G}}_{k}^{t}$ and ${\mathcal{G}}_{x}^{t}$ using the ADAM algorithm [14]
$\mathbf{x}={\mathcal{G}}_{x}^{T}\left({\mathbf{z}}_{x}\right)$, $\mathbf{k}={\mathcal{G}}_{x}^{T}({\mathbf{z}}_{k})$
4 Experimental Results
In this section, ablation study is first conducted to analyze the effect of optimization algorithm and network architecture. Then, our SelfDeblur is evaluated on two benchmark datasets and is compared with the stateoftheart blind deconvolution methods. Finally, we report the results of SelfDeblur on several realworld blurry images.
Our SelfDeblur is implemented using Pytorch [32]. The experiments are conducted on a PC equipped with one NVIDIA Titan V GPU. Unless specially stated, the experiments follow the same settings, i.e., $T=5,000$, and the noises ${\mathbf{z}}_{x}$ and ${\mathbf{z}}_{k}$ are sampled from the uniform distribution with fixed random seed 0. Following [45], we further perturb ${\mathbf{z}}_{x}$ randomly at each iteration. The initial learning rate is set as $0.01$ and is decayed by multiplying 0.5 when reaching 2,000, 3,000 and 4,000 iterations. The source code, deep models and supplementary materials are available at https://github.com/csdwren/SelfDeblur.
4.1 Ablation Study
Ablation study is conducted on the dataset by Levin et al. [20], which is a popular blind deconvolution benchmark consisting of 4 clean images and 8 blur kernels. Using [54], the average estimated noise level of the blurry images in the dataset is $\sigma \approx 1\times {10}^{5}$. Thus we simply adopt $\lambda =1\times {10}^{6}$ on this dataset.
4.1.1 Alternating Optimization vs. Joint Optimization
We first evaluate the performance of SelfDeblur using alternating optimization (SelfDeblurA) and joint optimization (SelfDeblurJ). Table 1 reports the average PSNR and SSIM values. In terms of quantitative metrics, SelfDeblurJ significantly outperforms SelfDeblurA, demonstrating the superiority of joint optimization. Fig. 3 provides several failure cases of SelfDeblurA. It can be seen that SelfDeblurA may converge to delta kernel and worse solution, while SelfDeblurJ performs favorably on these cases. Therefore, joint optimization is adopted as the default SelfDeblur method throughout the following experiments.
Blurry images  SelfDeblurA  SelfDeblurJ 
SelfDeblurA  SelfDeblurJ 
30.53 / 0.8748  33.07 / 0.9313 
4.1.2 Network Architecture of ${\mathcal{G}}_{k}$
In this experiment, we compare the results by considering four kinds of network architectures: (i) SelfDeblur, (ii) DoubleDIP [6] (asymmetric Autoencoder with skip connections for both ${\mathcal{G}}_{x}$ and ${\mathcal{G}}_{k}$), (iii) SelfDeblur${}_{k}$ (removing the hidden layer from ${\mathcal{G}}_{k}$), and (iv) SelfDeblur${}_{k+}$ (adding an extra hidden layer for ${\mathcal{G}}_{k}$). From Table 2 and Fig. 4, SelfDeblur significantly outperforms DoubleDIP in estimating blur kernel and latent image. The result indicates that the DIP network is limited to capture the prior of blur kernel, and the simple FCN can be a good choice of ${\mathcal{G}}_{k}$. We further compare SelfDeblur with SelfDeblur${}_{k}$ and SelfDeblur${}_{k+}$. One can see that the FCN without hidden layer (i.e., SelfDeblur${}_{k}$) also succeeds in estimating blur kernel and clean image (see Fig. 4), but performs much inferior to SelfDeblur. Moreover, the threelayer FCN ()i.e., SelfDeblur${}_{k+}$) is superior to SelfDeblur${}_{k}$, but is inferior to SelfDeblur. To sum up, SelfDeblur is a good choice for modeling blur kernel prior.
SelfDeblur  SelfDeblur${}_{k}$  SelfDeblur${}_{k+}$  DoubleDIP  
PSNR  33.07  28.37  30.92  21.51 
SSIM  0.9313  0.8396  0.8889  0.5256 
SelfDeblur  SelfDeblur${}_{k}$  SelfDeblur${}_{k+}$  DoubleDIP 
4.1.3 Visualization of Intermediate Results
Using an image from the dataset of Levin et al. [20], Fig. 5 shows the intermediate results of estimated blur kernel and clean image at iteration $t$ $=$ $1,20,100,600,2,000\text{and}5,000$, along with the MSE curve for $\mathbf{k}$ and the PSNR curve for $\mathbf{x}$. When iteration $t=20$, the intermediate result of $\mathbf{x}$ mainly contains the salient image structures, which is consistent with the observation that salient edges is crucial for initial blur kernel estimation in traditional methods. Along with the increase of iterations, ${\mathcal{G}}_{x}$ and ${\mathcal{G}}_{k}$ begin to generate finer details in $\mathbf{x}$ and $\mathbf{k}$. Unlike traditional methods, SelfDeblur is effective in simultaneously estimating blur kernel and recover latent clean image when iteration $t\ge 20$, making the nonblind deconvolution not a compulsory choice for SelfDeblur.
4.2 Comparison with Stateofthearts
4.2.1 Results on dataset of Levin et al. [20]
Using the dataset of Levin et al. [20], we compare our SelfDeblur with several stateoftheart blind deconvolution methods, including Krishnan et al. [16], Levin et al. [20], Cho&Lee [4], Xu&Jia [48], Sun et al. [41], Zuo et al. [56] and PanDCP [30]. For SelfDeblur, $\lambda =1\times {10}^{6}$ is set for all the blurry images. Following [41, 56], we adopt the nonblind deconvolution method in [21] to generate final deblurring results. PSNR, SSIM [46] and Error Ratio [21] are used as quantitative metrics. And we also report the running time (sec.) of blur kernel estimation for each competing method. Our SelfDeblur is ran on an NVIDIA Titan V GPU, while the other methods are ran on a PC with 3.30GHz Intel(R) Xeon(R) CPU.
Blurry image  Zuo et al. ${}^{\mathrm{\Delta}}$[56]  Xu&Jia${}^{\mathrm{\Delta}}$[48]  SelfDeblur${}^{\mathrm{\Delta}}$  
Groundtruth  Sun et al. ${}^{\mathrm{\Delta}}$[41]  PanDCP${}^{\mathrm{\Delta}}$[30]  SelfDeblur 
PSNR  SSIM  Error Ratio  Time  
Known k${}^{\mathrm{\Delta}}$  34.53  0.9492  1.0000  — 
Krishnan et al. ${}^{\mathrm{\Delta}}$ [16]  29.88  0.8666  2.4523  8.9400 
Cho&Lee${}^{\mathrm{\Delta}}$ [4]  30.57  0.8966  1.7113  1.3951 
Levin et al. ${}^{\mathrm{\Delta}}$ [21]  30.80  0.9092  1.7724  78.263 
Xu&Jia${}^{\mathrm{\Delta}}$ [49]  31.67  0.9163  1.4898  1.1840 
Sun et al. ${}^{\mathrm{\Delta}}$ [41]  32.99  0.9330  1.2847  191.03 
Zuo et al. ${}^{\mathrm{\Delta}}$ [56]  32.66  0.9332  1.2500  10.998 
PanDCP${}^{\mathrm{\Delta}}$ [30]  32.69  0.9284  1.2555  295.23 
SelfDeblur${}^{\mathrm{\Delta}}$  33.32  0.9438  1.2509  — 
SelfDeblur  33.07  0.9313  1.1968  224.01 
Krishnan et al. [16]  Levin et al. [21]  Cho&Lee [4]  Xu&Jia [48]  Sun et al. [41]  Zuo et al. [56]  PanDCP [30]  SelfDeblur  
MSE  675.4  307.0  340.8  402.1  254.7  351.9  347.2  149.5 
MNC[9]  0.8261  0.8693  0.8635  0.8811  0.9305  0.9042  0.8936  0.9408 
Images  Cho&Lee${}^{\mathrm{\Delta}}$[4]  Xu&Jia${}^{\mathrm{\Delta}}$[48]  Xu et al. ${}^{\mathrm{\Delta}}$[49]  Michaeli et al. ${}^{\mathrm{\Delta}}$[25]  Perroe et al. ${}^{\mathrm{\Delta}}$ [33]  PanL0${}^{\mathrm{\Delta}}$[28]  PanDCP${}^{\mathrm{\Delta}}$[30]  SelfDeblur${}^{\mathrm{\Delta}}$  SelfDeblur 
Manmade  16.35/0.3890  19.23/0.6540  17.99/0.5986  17.43/0.4189  17.41/0.5507  16.92/0.5316  18.59/0.5942  20.08/0.7338  20.35/0.7543 
Natural  20.14/0.5198  23.03/0.7542  21.58/0.6788  20.70/0.5116  21.04/0.6764  20.92/0.6622  22.60/0.6984  22.50/0.7183  22.05/0.7092 
People  19.90/0.5560  25.32/0.8517  24.40/0.8133  23.35/0.6999  22.77/0.7347  23.36/0.7822  24.03/0.7719  27.41/0.8784  25.94/0.8834 
Saturated  14.05/0.4927  14.79/0.5632  14.53/0.5383  14.14/0.4914  14.24/0.5107  14.62/0.5451  16.52/0.6322  16.58/0.6165  16.35/0.6364 
Text  14.87/0.4429  18.56/0.7171  17.64/0.6677  16.23/0.4686  16.94/0.5927  16.87/0.6030  17.42/0.6193  19.06/0.7126  20.16/0.7785 
Avg.  17.06/0.4801  20.18/0.7080  19.23/0.6593  18.37/0.5181  18.48/0.6130  18.54/0.6248  19.89/0.6656  21.13/0.7319  20.97/0.7524 
Table 3 lists the average metrics of the competing methods. We report the results of SelfDeblur with two settings, i.e., the deblurring results purely by SelfDeblur and those using the nonblind deconvolution from [21], denoted as SelfDeblur${}^{\mathrm{\Delta}}$. In terms of PSNR and Error Ratio, SelfDeblur significantly outperforms the competing methods. As for average SSIM, SelfDeblur performs slightly inferior to Sun et al. and Zuo et al. By incorporating with nonblind deconvolution from [21], SelfDeblur${}^{\mathrm{\Delta}}$ can further boost quantitative performance and outperforms all the other methods. In terms of running time, SelfDeblur is timeconsuming due to the optimization of two generative networks, but is comparable with Sun et al. [41] and PanDCP [30].
We further provide the average PSNR and SSIM values for each blur kernel in Fig. 7. From the visual comparison in Fig. 6, the #4 blur kernel estimated by SelfDeblur is much closer to the groundtruth. As shown in the closeups, SelfDeblur and SelfDeblur${}^{\mathrm{\Delta}}$ can recover more visually favorable textures. We also note that both the performance gap and visual quality between SelfDeblur and SelfDeblur${}^{\mathrm{\Delta}}$ are not significant, and thus nonblind deconvolution is not a compulsory choice for our SelfDeblur.
Moreover, we evaluate the estimated kernels using two metrics, i.e., MSE best aligned to groundtruth blur kernel ${\mathbf{k}}^{gt}$ and maximum of normalized convolution (MNC) [9],
$$\text{MNC}=\mathrm{max}\left(\frac{\mathbf{k}\otimes {\mathbf{k}}^{gt}}{{\parallel \mathbf{k}\parallel}_{2}{\parallel {\mathbf{k}}^{gt}\parallel}_{2}}\right).$$ 
From Table 4, our SelfDeblur is much superior to the competing methods in terms of both the metrics.
4.2.2 Results on dataset of Lai et al. [19]
We further evaluate SelfDeblur on the dataset of Lai et al. [19] [19] consisting of 25 clean images and 4 large size blur kernels. The blurry images are divided into five categories, i.e., Manmade, Natural, People, Saturated and Text, where each category contains 20 blurry images. For each blurry image, the parameter $\lambda $ is set according to the noise level estimated using [54]. We compare our SelfDeblur with Cho&Lee [4], Xu&Jia[48], Xu et al. [49], Machaeli et al. [25], Perroe et al. [33], PanL$0$ [28] and PanDCP [30]. The results of competing methods except PanDCP [30] and ours are duplicated from [19]. The results of PanDCP [30] are generated using their default settings. Once the estimated blur kernel is obtained, nonblind deconvolution [15] is applied to the images of Manmade, Natural, People and Text, while [47] is used to handle Saturated images.
Blurry image  Xu&Jia${}^{\mathrm{\Delta}}$[48]  Perrone et al. ${}^{\mathrm{\Delta}}$[33]  SelfDeblur${}^{\mathrm{\Delta}}$  
Groundtruth  Michaeli et al. ${}^{\mathrm{\Delta}}$[48]  PanDCP${}^{\mathrm{\Delta}}$[33]  SelfDeblur 
Blurry image  Xu&Jia[48]  PanDCP[30]  SelfDeblur 
From Table 5, both SelfDeblur and SelfDeblur${}^{\mathrm{\Delta}}$ can achieve better quantitative metrics than the competing methods. In terms of image contents, our SelfDeblur outperform the other methods on any of the five categories. Fig. 9 shows the average PSNR and SSIM values for each blur kernel. From the results in Fig. 8, the blur kernel estimated by our SelfDeblur is more accurate than those by the competing methods, and the deconvolution result is with more visually plausible textures.
4.3 Evaluation on Realworld Blurry Images
Our SelfDeblur is further compared with Xu&Jia [48] and PanDCP [30] on several realworld blurry images. From Fig. 10, one can see that the blur kernels estimated by our SelfDeblur contain less noises, and the estimated clean images are with more visually plausible structures and textures. The kernel estimation errors by Xu&Jia and PanDCP are obvious, thereby yielding ringing artifacts in the estimated clean images. More results can be found in the supplementary materials.
5 Conclusion
In this paper, we proposed a neural blind deconvolution method, i.e., SelfDeblur. It adopts an asymmetric Autoencoder and a FCN to respectively capture the deep priors of latent clean image and blur kernel. And the SoftMax nonlinearity is applied to the output of FCN to meet the nonnegative and equality constraints of blur kernel. A joint optimization algorithm is suggested to solve the unconstrained neural blind deconvolution model. Experiments show that our SelfDeblur achieves notable performance gains over the stateoftheart methods, and is effective in estimating blur kernel and generating clean image with visually favorable textures.
References
 [1] (2016) A neural approach to blind motion deblurring. In European Conference on Computer Vision, pp. 221–235. Cited by: §1, §2.2.
 [2] (1998) Total variation blind deconvolution. IEEE Transactions on Image Processing 7 (3), pp. 370–375. Cited by: §1, §1, §2.1, §3.3.
 [3] (2019) Blind image deblurring with local maximum gradient prior. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1742–1750. Cited by: §1.
 [4] (2009) Fast motion deblurring. ACM Transactions on Graphics 28 (5), pp. 145. Cited by: §1, Figure 7, Figure 9, §4.2.1, §4.2.2, Table 3, Table 4, Table 5.
 [5] (2013) Nonlocally centralized sparse representation for image restoration. IEEE Transactions on Image Processing 22 (4), pp. 1620–1630. Cited by: §2.1.
 [6] (2019) ” Doubledip”: unsupervised image decomposition via coupled deepimagepriors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Cited by: §1, §2.2, §3.2, §3.2, §4.1.2.
 [7] (2019) Dynamic scene deblurring with parameter selective sharing and nested skip connections. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3848–3856. Cited by: §1.
 [8] (2014) Generative adversarial nets. In Advances in Neural Information Processing Systems, pp. 2672–2680. Cited by: §3.2.
 [9] (2012) Good regions to deblur. In European conference on computer vision, pp. 59–72. Cited by: §4.2.1, Table 4.
 [10] (2017) Online video deblurring via dynamic temporal blending network. In Proceedings of the IEEE International Conference on Computer Vision, pp. 4038–4047. Cited by: §2.2.
 [11] (2018) Normalized blind deconvolution. In European Conference on Computer Vision, Cited by: §1.
 [12] (2016) Accurate image superresolution using very deep convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1646–1654. Cited by: §1.
 [13] (2016) Deeplyrecursive convolutional network for image superresolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1637–1645. Cited by: §1.
 [14] (2015) Adam: a method for stochastic optimization. In International Conference on Learning Representations, Cited by: §3.3, §3.3, §3.3.
 [15] (2009) Fast image deconvolution using hyperLaplacian priors. In Advances in Neural Information Processing Systems, pp. 1033–1041. Cited by: §2.1, §4.2.2, Table 5.
 [16] (2011) Blind deconvolution using a normalized sparsity measure. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 233–240. Cited by: §1, §2.1, §2.1, §3.1, Figure 7, §4.2.1, Table 3, Table 4.
 [17] (2017) Learning to push the limits of efficient fftbased image deconvolution. In Proceedings of the IEEE International Conference on Computer Vision, pp. 4586–4594. Cited by: §2.1.
 [18] (2018) Deblurgan: blind motion deblurring using conditional adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8183–8192. Cited by: §2.2.
 [19] (2016) A comparative study for single image blind deblurring. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1701–1709. Cited by: §1, Figure 8, Figure 9, §4.2.2, §4.2.2, Table 5.
 [20] (2009) Understanding and evaluating blind deconvolution algorithms. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1964–1971. Cited by: §1, §1, §1, §2.1, Figure 6, Figure 7, §4.1.3, §4.1, §4.2.1, §4.2.1, Table 1, Table 3.
 [21] (2011) Efficient marginal likelihood optimization in blind deconvolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2657–2664. Cited by: §2.1, §4.2.1, §4.2.1, Table 3, Table 4.
 [22] (2018) Learning a discriminative prior for blind image deblurring. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6616–6625. Cited by: §2.1.
 [23] (2014) Blind image deblurring using spectral properties of convolution operators. IEEE Transactions on Image Processing 23 (12), pp. 5047–5056. Cited by: §1, §2.1, §2.1, §3.1.
 [24] (2019) On the convergence of learningbased iterative methods for nonconvex inverse problems. IEEE Transactions on Pattern Analysis and Machine Intelligence. Cited by: §1.
 [25] (2014) Blind deblurring using internal patch recurrence. In European Conference on Computer Vision, pp. 783–798. Cited by: §1, §2.1, §3.1, Figure 9, §4.2.2, Table 5.
 [26] (2017) Deep multiscale convolutional neural network for dynamic scene deblurring. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Vol. 1, pp. 3. Cited by: §1, §2.2.
 [27] (2019) Recurrent neural networks with intraframe iterations for video deblurring. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8102–8111. Cited by: §2.2.
 [28] (2017) ${L}_{0}$Regularized intensity and gradient prior for deblurring text images and beyond. IEEE Transactions on Pattern Analysis and Machine Intelligence 39 (2), pp. 342–355. Cited by: Figure 1, §1, §2.1, §2.1, §3.3, Figure 9, §4.2.2, Table 5.
 [29] (2018) Learning to deblur images with exemplars. IEEE Transactions on Pattern Analysis and Machine Intelligence. Cited by: §1.
 [30] (2018) Deblurring images via dark channel prior. IEEE Transactions on Pattern Analysis and Machine Intelligence 40 (10), pp. 2315–2328. Cited by: Figure 1, §1, §1, §2.1, §2.1, §3.1, §3.1, §3.3, Figure 10, Figure 6, Figure 7, Figure 9, §4.2.1, §4.2.1, §4.2.2, §4.3, Table 3, Table 4, Table 5.
 [31] (2017) Simultaneous stereo video deblurring and scene flow estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4382–4391. Cited by: §2.2.
 [32] (2017) Automatic differentiation in pytorch. In NIPS Autodiff Workshop: The Future of Gradientbased Machine Learning Software and Techniques, Cited by: §3.3, §3.3, §4.
 [33] (2014) Total variation blind deconvolution: the devil is in the details. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2909–2916. Cited by: §1, §1, §2.1, §3.1, §3.1, Figure 8, Figure 9, §4.2.2, Table 5.
 [34] (2016) Image deblurring via enhanced lowrank prior. IEEE Transactions on Image Processing 25 (7), pp. 3426–3437. Cited by: §1, §2.1, §3.1.
 [35] (2015) Unet: convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computerassisted Intervention, pp. 234–241. Cited by: §3.2.
 [36] (2014) Shrinkage fields for effective image restoration. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2774–2781. Cited by: §2.1.
 [37] (2013) Discriminative nonblind deblurring. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 604–611. Cited by: §2.1.
 [38] (2016) Learning to deblur. IEEE Transactions on Pattern Analysis and Machine Intelligence 38 (7), pp. 1439–1451. Cited by: §1, §2.2.
 [39] (2018) Zeroshot superresolution using deep internal learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3118–3126. Cited by: §1, §3.3.
 [40] (2015) Learning a convolutional neural network for nonuniform motion blur removal. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 769–777. Cited by: §2.2, §2.2.
 [41] (2013) Edgebased blur kernel estimation using patch priors. In IEEE International Conference on Computational Photography, pp. 1–8. Cited by: Figure 1, §1, §2.1, §2.1, §3.1, §3.1, §3.3, Figure 6, Figure 7, §4.2.1, §4.2.1, Table 3, Table 4.
 [42] (2017) Image superresolution via deep recursive residual network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Vol. 1, pp. 5. Cited by: §1.
 [43] (2018) Scalerecurrent network for deep image deblurring. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8174–8182. Cited by: §1, §2.2.
 [44] (2001) Convergence of a block coordinate descent method for nondifferentiable minimization. Journal of optimization theory and applications 109 (3), pp. 475–494. Cited by: §3.3.
 [45] (2018) Deep image prior. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9446–9454. Cited by: §1, §1, §2.2, §3.2, §3.2, §3.2, §3.2, §4.
 [46] (2004) Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing 13 (4), pp. 600–612. Cited by: §4.2.1.
 [47] (2014) Deblurring shaken and partially saturated images. International Journal of Computer Vision 110 (2), pp. 185–201. Cited by: §4.2.2, Table 5.
 [48] (2010) Twophase kernel estimation for robust motion deblurring. In European Conference on Computer Vision, pp. 157–170. Cited by: Figure 1, Figure 10, Figure 6, Figure 7, Figure 8, Figure 9, §4.2.1, §4.2.2, §4.3, Table 4, Table 5.
 [49] (2013) Unnatural l0 sparse representation for natural image deblurring. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1107–1114. Cited by: §2.1, §4.2.2, Table 3, Table 5.
 [50] (2017) Image deblurring via extreme channels prior. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4003–4011. Cited by: §1, §2.1, §3.1.
 [51] (2018) Dynamic scene deblurring using spatially variant recurrent neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2521–2529. Cited by: §1, §2.2.
 [52] (2017) Beyond a gaussian denoiser: residual learning of deep cnn for image denoising. IEEE Transactions on Image Processing 26 (7), pp. 3142–3155. Cited by: §1.
 [53] (2017) Learning deep CNN denoiser prior for image restoration. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Cited by: §2.1.
 [54] (2009) Scale invariance and noise in natural images. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2209–2216. Cited by: §3.2, §4.1, §4.2.2.
 [55] (2011) From learning models of natural image patches to whole image restoration. In Proceedings of the IEEE International Conference on Computer Vision, pp. 479–486. Cited by: §2.1.
 [56] (2016) Learning iterationwise generalized shrinkage–thresholding operators for blind deconvolution. IEEE Transactions on Image Processing 25 (4), pp. 1751–1764. Cited by: §1, §1, §2.1, §3.1, §3.1, §3.3, Figure 6, §4.2.1, Table 3, Table 4.