Understanding Deformable Alignment in Video Super-Resolution

  • 2020-09-15 17:55:06
  • Kelvin C. K. Chan, Xintao Wang, Ke Yu, Chao Dong, Chen Change Loy
  • 1

Abstract

Deformable convolution, originally proposed for the adaptation to geometricvariations of objects, has recently shown compelling performance in aligningmultiple frames and is increasingly adopted for video super-resolution. Despiteits remarkable performance, its underlying mechanism for alignment remainsunclear. In this study, we carefully investigate the relation betweendeformable alignment and the classic flow-based alignment. We show thatdeformable convolution can be decomposed into a combination of spatial warpingand convolution. This decomposition reveals the commonality of deformablealignment and flow-based alignment in formulation, but with a key difference intheir offset diversity. We further demonstrate through experiments that theincreased diversity in deformable alignment yields better-aligned features, andhence significantly improves the quality of video super-resolution output.Based on our observations, we propose an offset-fidelity loss that guides theoffset learning with optical flow. Experiments show that our loss successfullyavoids the overflow of offsets and alleviates the instability problem ofdeformable alignment. Aside from the contributions to deformable alignment, ourformulation inspires a more flexible approach to introduce offset diversity toflow-based alignment, improving its performance.

 

Quick Read (beta)

loading the full paper ...