Abstract
Prompting with natural language instructions has recently emerged as apopular method of harnessing the capabilities of large language models. Giventhe inherent ambiguity present in natural language, it is intuitive to considerthe possible advantages of prompting with less ambiguous prompt styles, such asthe use of pseudo-code. In this paper we explore if prompting via pseudo-code instructions helpsimprove the performance of pre-trained language models. We manually create adataset of pseudo-code prompts for 132 different tasks spanning classification,QA and generative language tasks, sourced from the Super-NaturalInstructionsdataset. Using these prompts along with their counterparts in natural language,we study their performance on two LLM families - BLOOM and CodeGen. Ourexperiments show that using pseudo-code instructions leads to better results,with an average increase (absolute) of 7-16 points in F1 scores forclassification tasks and an improvement (relative) of 12-38% in aggregateROUGE-L scores across all tasks. We include detailed ablation studies whichindicate that code comments, docstrings, and the structural clues encoded inpseudo-code all contribute towards the improvement in performance. To the best of our knowledge our work is the first to demonstrate howpseudo-code prompts can be helpful in improving the performance of pre-trainedLMs.