Pretrained contextualized language models such as BERT have achievedimpressive results on various natural language processing benchmarks.Benefiting from multiple pretraining tasks and large scale training corpora,pretrained models can capture complex syntactic word relations. In this paper,we use the deep contextualized language model BERT for the task of ad hoc tableretrieval. We investigate how to encode table content considering the structureand input length limit of BERT. We also propose an approach that incorporatesfeatures from prior literature on table retrieval and jointly trains them withBERT. In experiments on public datasets, we show that our best approach canoutperform the previous state-of-the-art method and BERT baselines with a largemargin under different evaluation metrics.