Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models

  • 2023-03-08 15:50:02
  • Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, Nan Duan
  • 210


ChatGPT is attracting a cross-field interest as it provides a languageinterface with remarkable conversational competency and reasoning capabilitiesacross many domains. However, since ChatGPT is trained with languages, it iscurrently not capable of processing or generating images from the visual world.At the same time, Visual Foundation Models, such as Visual Transformers orStable Diffusion, although showing great visual understanding and generationcapabilities, they are only experts on specific tasks with one-round fixedinputs and outputs. To this end, We build a system called \textbf{VisualChatGPT}, incorporating different Visual Foundation Models, to enable the userto interact with ChatGPT by 1) sending and receiving not only languages butalso images 2) providing complex visual questions or visual editinginstructions that require the collaboration of multiple AI models withmulti-steps. 3) providing feedback and asking for corrected results. We designa series of prompts to inject the visual model information into ChatGPT,considering models of multiple inputs/outputs and models that require visualfeedback. Experiments show that Visual ChatGPT opens the door to investigatingthe visual roles of ChatGPT with the help of Visual Foundation Models. Oursystem is publicly available at\url{}.


Quick Read (beta)

loading the full paper ...