Abstract
Large Language Models for code often entail significant computationalcomplexity, which grows significantly with the length of the input codesequence. We propose LeanCode for code simplification to reduce training andprediction time, leveraging code contexts in utilizing attention scores torepresent the tokens' importance. We advocate for the selective removal oftokens based on the average context-aware attention scores rather than averagescores across all inputs. LeanCode uses the attention scores of `CLS' tokenswithin the encoder for classification tasks, such as code search. It alsoemploys the encoder-decoder attention scores to determine token significancefor sequence-to-sequence tasks like code summarization. Our evaluation showsLeanCode's superiority over the SOTAs DietCode and Slimcode, with improvementsof 60% and 16% for code search, and 29% and 27% for code summarization,respectively.