PaLM 2 Technical Report

  • 2023-05-17 18:46:53
  • Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernandez Abrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan Botha, James Bradbury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christopher A. Choquette-Choo, Aakanksha Chowdhery, Clément Crepy, Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, Mark Díaz, Nan Du, Ethan Dyer, Vlad Feinberg, Fangxiaoyu Feng, Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastian Gehrmann, Lucas Gonzalez,
  • 44


We introduce PaLM 2, a new state-of-the-art language model that has bettermultilingual and reasoning capabilities and is more compute-efficient than itspredecessor PaLM. PaLM 2 is a Transformer-based model trained using a mixtureof objectives. Through extensive evaluations on English and multilinguallanguage, and reasoning tasks, we demonstrate that PaLM 2 has significantlyimproved quality on downstream tasks across different model sizes, whilesimultaneously exhibiting faster and more efficient inference compared to PaLM.This improved efficiency enables broader deployment while also allowing themodel to respond faster, for a more natural pace of interaction. PaLM 2demonstrates robust reasoning capabilities exemplified by large improvementsover PaLM on BIG-Bench and other reasoning tasks. PaLM 2 exhibits stableperformance on a suite of responsible AI evaluations, and enablesinference-time control over toxicity without additional overhead or impact onother capabilities. Overall, PaLM 2 achieves state-of-the-art performanceacross a diverse set of tasks and capabilities. When discussing the PaLM 2 family, it is important to distinguish betweenpre-trained models (of various sizes), fine-tuned variants of these models, andthe user-facing products that use these models. In particular, user-facingproducts typically include additional pre- and post-processing steps.Additionally, the underlying models may evolve over time. Therefore, one shouldnot expect the performance of user-facing products to exactly match the resultsreported in this report.


Quick Read (beta)

loading the full paper ...