Spatial convolutions are widely used in numerous deep video models. Itfundamentally assumes spatio-temporal invariance, i.e., using shared weightsfor every location in different frames. This work presents Temporally-AdaptiveConvolutions (TAdaConv) for video understanding, which shows that adaptiveweight calibration along the temporal dimension is an efficient way tofacilitate modelling complex temporal dynamics in videos. Specifically,TAdaConv empowers the spatial convolutions with temporal modelling abilities bycalibrating the convolution weights for each frame according to its local andglobal temporal context. Compared to previous temporal modelling operations,TAdaConv is more efficient as it operates over the convolution kernels insteadof the features, whose dimension is an order of magnitude smaller than thespatial resolutions. Further, the kernel calibration also brings an increasedmodel capacity. We construct TAda2D networks by replacing the spatialconvolutions in ResNet with TAdaConv, which leads to on par or betterperformance compared to state-of-the-art approaches on multiple video actionrecognition and localization benchmarks. We also demonstrate that as a readilyplug-in operation with negligible computation overhead, TAdaConv caneffectively improve many existing video models with a convincing margin. Codesand models will be made available athttps://github.com/alibaba-mmai-research/pytorch-video-understanding.