Florence: A New Foundation Model for Computer Vision

  • 2021-11-22 18:59:55
  • Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, Ce Liu, Mengchen Liu, Zicheng Liu, Yumao Lu, Yu Shi, Lijuan Wang, Jianfeng Wang, Bin Xiao, Zhen Xiao, Jianwei Yang, Michael Zeng, Luowei Zhou, Pengchuan Zhang
  • 94

Abstract

Automated visual understanding of our diverse and open world demands computervision models to generalize well with minimal customization for specific tasks,similar to human vision. Computer vision foundation models, which are trainedon diverse, large-scale dataset and can be adapted to a wide range ofdownstream tasks, are critical for this mission to solve real-world computervision applications. While existing vision foundation models such as CLIP,ALIGN, and Wu Dao 2.0 focus mainly on mapping images and textualrepresentations to a cross-modal shared representation, we introduce a newcomputer vision foundation model, Florence, to expand the representations fromcoarse (scene) to fine (object), from static (images) to dynamic (videos), andfrom RGB to multiple modalities (caption, depth). By incorporating universalvisual-language representations from Web-scale image-text data, our Florencemodel can be easily adapted for various computer vision tasks, such asclassification, retrieval, object detection, VQA, image caption, videoretrieval and action recognition. Moreover, Florence demonstrates outstandingperformance in many types of transfer learning: fully sampled fine-tuning,linear probing, few-shot transfer and zero-shot transfer for novel images andobjects. All of these properties are critical for our vision foundation modelto serve general purpose vision tasks. Florence achieves new state-of-the-artresults in majority of 44 representative benchmarks, e.g., ImageNet-1Kzero-shot classification with top-1 accuracy of 83.74 and the top-5 accuracy of97.18, 62.4 mAP on COCO fine tuning, 80.36 on VQA, and 87.8 on Kinetics-600.

 

Quick Read (beta)

loading the full paper ...