In this paper we propose a deep residual autoencoder exploitingResidual-in-Residual Dense Blocks (RRDB) to remove artifacts in JPEG compressedimages that is independent from the Quality Factor (QF) used. The proposedapproach leverages both the learning capacity of deep residual networks andprior knowledge of the JPEG compression pipeline. The proposed model operatesin the YCbCr color space and performs JPEG artifact restoration in two phasesusing two different autoencoders: the first one restores the luma channelexploiting 2D convolutions; the second one, using the restored luma channel asa guide, restores the chroma channels explotining 3D convolutions. Extensiveexperimental results on three widely used benchmark datasets (i.e. LIVE1,BDS500, and CLASSIC-5) show that our model is able to outperform the state ofthe art with respect to all the evaluation metrics considered (i.e. PSNR,PSNR-B, and SSIM). This results is remarkable since the approaches in the stateof the art use a different set of weights for each compression quality, whilethe proposed model uses the same weights for all of them, making it applicableto images in the wild where the QF used for compression is unkwnown.Furthermore, the proposed model shows a greater robustness thanstate-of-the-art methods when applied to compression qualities not seen duringtraining.