A Systematic Characterization of Sampling Algorithms for Open-ended Language Generation

  • 2020-09-15 17:28:42
  • Moin Nadeem, Tianxing He, Kyunghyun Cho, James Glass
  • 2


This work studies the widely adopted ancestral sampling algorithms forauto-regressive language models, which is not widely studied in the literature.We use the quality-diversity (Q-D) trade-off to investigate three popularsampling algorithms (top-k, nucleus and tempered sampling). We focus on thetask of open-ended language generation. We first show that the existingsampling algorithms have similar performance. After carefully inspecting thetransformations defined by different sampling algorithms, we identify three keyproperties that are shared among them: entropy reduction, order preservation,and slope preservation. To validate the importance of the identifiedproperties, we design two sets of new sampling algorithms: one set in whicheach algorithm satisfies all three properties, and one set in which eachalgorithm violates at least one of the properties. We compare their performancewith existing sampling algorithms, and find that violating the identifiedproperties could lead to drastic performance degradation, as measured by theQ-D trade-off. On the other hand, we find that the set of sampling algorithmsthat satisfies these properties performs on par with the existing samplingalgorithms. Our data and code are available athttps://github.com/moinnadeem/characterizing-sampling-algorithms


Quick Read (beta)

loading the full paper ...